Wayfair, a Fortune 500 company, has carved its niche as a leading online retailer for furniture and décor. But Wayfair isn't just another furniture store — they're harnessing cutting-edge generative AI to completely transform how you shop for your home. By leveraging AI, Wayfair personalizes product recommendations, enhances visualization tools, and streamlines the shopping experience to make it easier than ever to find the perfect furniture for your home.

How Wayfair uses AI for customer service

By integrating AI throughout the customer journey, Wayfair personalizes the shopping experience, streamlines decision-making, and offers a wider range of products that resonate with individual customer needs. This is achieved through leveraging:

• Generative AI for augmenting and improving customer service and sales interactions.
• AI-powered features to personalize search, communications, and customer understanding.

As a Fortune 500 company, Wayfair prioritizes maintaining top-tier customer experiences, which means customer support is a key touchpoint. Wayfair uses AI in several ways to enhance customer service and improve efficiency while maintaining its competitive edge in the market. The AI solutions employed at Wayfair support agents in:

• Providing quick and accurate responses to customer inquiries for more efficient and targeted assistance.
• Establishing high standards of service excellence and operational efficiency.

AI significantly enhances customer support at Wayfair by enabling human agents to handle customer inquiries efficiently. Wayfair’s AI models draft ideal responses based on customer questions, allowing agents to focus on refining and personalizing the answers. This approach not only saves time but also guarantees higher customer satisfaction, making Wayfair’s support system more responsive and effective.

As AI-enhanced capabilities become integrated into fundamental company workflows, it’s crucial they align with Wayfair’s high standards of service excellence and adhere to internal policies and regulations. This ultimately guarantees that AI complements, rather than compromises, the high-quality customer experience Wayfair is known for. For example, Wayfair’s AI needs to adhere to its return policy. Wayfair trains its models to understand and apply their return guidelines, making sure they can handle scenarios like processing a return request specific to the individual customer’s situation. This helps maintain consistency in customer interactions and ensures the AI aligns with Wayfair’s service standards.

Why Wayfair is partnering with Quotient for AI evaluation

Robust AI evaluation can cost a significant amount of time and money. This can be a blocker for enterprises that are trying to quickly ship effective generative AI solutions.

Since Wayfair prioritizes quality customer interactions, it’s important that they optimize their AI models for both accuracy and efficiency. Wayfair is partnering with Quotient to specifically address this need.
‍

After our collaboration with Quotient, we have a considerable opportunity to deploy customer-facing AI features that are high quality and policy-compliant. Other market solutions required us to create our own evaluation frameworks, which would have significantly set back our timeline

‍Graham Ganssle, Wayfair’s Head of AI for Customer Service.

Methodology

Quotient’s AI-backed Policy Compliance Evaluator rigorously assessed the compliance of Wayfair’s AI agent responses with internal customer support policies.

To validate accuracy, this tool was tested against a synthetic dataset of 2,300 violations tailored to Wayfair’s policies, generated through Quotient’s Evaluation Benchmark Generator with zero human feedback. At the Generator’s core lies Quotient’s in-house research, which has shown that small samples of human labels significantly improve the quality of synthetic data.

We used the following methodology:

1. We identified the relevant Wayfair policies for each real conversation
2. For each of those conversations and relevant policies, we generated multiple policy-violating LLM responses using the Evaluation Benchmark Generator
3. 70% of datapoints out of a 100 datapoint sub-sample were confirmed by Subject Matter Expert (SME) reviewers to be indeed policy-violating

Quotient’s Evaluation Benchmark Generator also provides the ability to factor in human feedback during the generation process, which yields datasets that are even more aligned to SME preferences.

Results

We evaluated the Policy Compliance Evaluator on the synthetic dataset of 2,300 policy violations. The results met Wayfair’s “highest expectations bar”:

• 85% True Positive Rate for identifying policy-violating responses
• 95% True Negative Rate for identifying policy-compliant LLM responses

“Quotient was the first product that was able to accurately and reliably test policy compliance of our AI agents. We’re now excited to continue to innovate with the confidence that all our AI solutions are meeting our high standards for customer solutions,” says Graham.

Using Quotient’s novel evaluation framework, Wayfair is able to confirm and improve the quality of their AI agents, and ship customer-facing features confidently, ultimately meeting customer needs more effectively.

Conclusion

AI has the power to transform customer service interactions, but evaluation is central to shipping AI products that can meet the needs of customers. By leveraging Quotient's AI Evaluation Platform, Wayfair has taken a significant step towards optimizing their AI for accuracy, efficiency, and adherence to their high standards. Quotient’s solution has the potential to eliminate most of the resource-intensive tasks that cripple enterprises from adopting generative AI at scale.

Quotient’s product can effectively address the unique needs of forward-thinking, AI-ready companies like Wayfair. Julia Neagu, CEO of Quotient, explains that partnering with Wayfair has helped ensure that Quotient’s products “meet the high standards and specific needs of Fortune 500 companies. In turn, Wayfair can build and ship AI solutions tailored to their unique customer and organizational requirements.” This collaboration also shows how large enterprises can harness AI-powered customer service solutions to deliver exceptional experiences for customers across industries. As Julia says, “Working closely with such a forward-thinking company has been instrumental in shaping our product to serve large enterprises effectively.”

Onwards 🚀

If you’re tackling similar challenges, we’d love to hear from you at contact@quotientai.co.