AI Evals

In the rapidly evolving landscape of AI product development, one skill has emerged as critically important yet often overlooked: the ability to write and implement effective AI evaluations, or "evals" as they call it. 

As Kevin Weil, CPO of OpenAI, put it succinctly: "Writing evals is going to become a core skill for product managers. It is such a critical part of making a good product with AI." 

This guide demystifies AI evals for product managers, providing practical frameworks, real-world examples, and actionable strategies to help you build more reliable AI products. 

What are AI evals and why do they matter?

AI evals are structured measurement frameworks that assess how well your AI system performs against defined criteria. Unlike traditional software testing with clear pass/fail scenarios, AI evals measure quality and effectiveness across multiple dimensions. 

To understand why evals matter, consider the case of Microsoft's Tay chatbot from 2016. Released on X (formerly Twitter) as an experiment in conversational understanding, Tay was designed to learn from interactions with users. Within 24 hours, Microsoft had to take Tay offline after it began doing increasingly offensive and inflammatory posts. The chatbot had been manipulated by users who exploited its learning algorithm, teaching it to parrot harmful content. 

There is also a more recent example from Meta. In 2022, Meta released Galactica, an advanced language model for scientific research. Despite initial excitement, Meta pulled the demo within three days because the model generated wildly inaccurate and sometimes offensive scientific content. Both cases demonstrate how proper evals could have identified vulnerabilities before public release, preventing both harmful outputs and significant brand damage. 

"If there is one thing we can teach people, it's that writing evals is probably the most important thing," — Mike Krieger, CPO of Anthropic

Traditional testing vs. AI Evals 

Key features of traditional software for testing AI evaluations:

• Deterministic outcomes 

• Non-deterministic responses 

• Clear pass/fail scenarios 

• Qualitative assessment scales 

• Unit tests follow fixed paths, LLMs can take multiple paths 

• Integration tests rely on documentation, improvement relies on data 

• Focus on correctness, relevance, tone, safety, etc. 

The anatomy of an effective AI Eval: A customer support example 

Effective AI evals typically include four key components: setting the role, providing the context, defining the goal, and establishing terminology and labels. 

Let's examine each through a customer support AI assistant example: 

1. Setting the role: Establishing the context for the evaluating system 

“You are a customer support quality evaluator with expertise in assessing response effectiveness.” 

This establishes who the evaluator is supposed to be and what expertise they should apply when reviewing the interaction. 

2. Providing the context: Supplying the data to be evaluated 

“CONTEXT: 

[BEGIN DATA] 

User: "I ordered headphones with express shipping, but my order still shows 'Processing'. I need them for a presentation on Friday." 

Assistant: "I understand your concern. I see your order #38291 is still 'Processing' despite express shipping. There was a warehouse delay, but your order has been prioritized for overnight delivery. You'll receive them by Thursday, in time for Friday. I've added a $10 credit for the inconvenience. Would you like a shipping confirmation email?" 

[END DATA]” 

This supplies the complete interaction that needs evaluation, giving the evaluator all the necessary information to make an informed assessment. 

3. Providing the goal: Clearly defining what success looks like 

EVALUATION TASK: 

Evaluate whether the assistant's response effectively addresses the customer's issue. Consider: 

1. Did the assistant acknowledge the customer's specific concern? 

2. Did the assistant provide a clear explanation of the issue? 

3. Did the assistant offer an appropriate solution that meets the customer's need (getting headphones before Friday)? 

4. Did the assistant take appropriate action to remedy the service failure?” 

This clearly defines what aspects should be evaluated and breaks down the specific criteria for success, guiding the evaluator to focus on the most important elements. 

4. Defining scoring criteria: Ensuring consistency in evaluation 

Rate the response on a 5-point scale: 

1 = Failed to address the issue 

2 = Partially addressed the issue with significant gaps 

3 = Adequately addressed the issue but could improve 

4 = Effectively addressed the issue with minor improvements possible 

5 = Perfectly addressed the issue with appropriate solution and compensation Your response must be a single number between 1-5.” 

This establishes a consistent rating system with clear definitions for each point on the scale, ensuring evaluations are standardized and comparable.

By structuring evaluations using these four components, you can create a comprehensive framework that ensures evaluators understand their role, have complete context, know exactly what to assess, and use consistent criteria for providing feedback. 

Types of AI evals 

Different evaluation approaches serve different purposes in your AI product development cycle: 

1. Human evals 

Description: Actual users or expert evaluators provide direct feedback on AI outputs. 

Implementation: Add feedback mechanisms like thumbs up/down buttons or rating scales to your product interface. 

Best for: High-stakes decisions, creative content, establishing "golden" examples. 

Example: Spotify's Podcast AI Summary feature initially used human evaluators to rate summary quality across dimensions like accuracy, comprehensiveness, and readability before building automated evaluation systems. 

2. LLM-as-judge evals 

Description: Using another LLM to evaluate your primary LLM's outputs. 

Implementation: Create prompts that instruct a "judge" LLM to evaluate specific aspects of your primary LLM's responses. 

Best for: Scaling evaluations cost-effectively, consistent application of evaluation criteria. 

Example prompt structure: 

You are evaluating the factual accuracy of an AI assistant's response. 

User query: "What year was the first iPhone released?" 

Assistant response: "The first iPhone was released in 2007." 

Assess whether the assistant's response contains any factual errors. 

Your response should be only "accurate" or "inaccurate". 

3. Code-based evals 

Description: Programmatic checks that verify specific aspects of AI outputs.

Implementation: Write functions that test for particular patterns, values, or behaviors in AI responses. 

Best for: Objective criteria like format compliance, runtime errors in generated code, or data validation. 

Example: A function that validates whether AI-generated SQL queries run without syntax errors. 

Common evaluation criteria for AI outputs 

It bears weight to understand some common eval criteria for AI outputs. While frameworks and methodologies are crucial, knowing what to measure is equally important. The most effective AI teams consistently evaluate these key dimensions: 

Accuracy 

Is the information factually correct? Track error rates and verify outputs against reliable sources. Example metric: Percentage of factual errors per response. 

Relevance 

Does the response actually address what was asked? Use a simple 1-5 scale to measure how well the AI tackles the specific query. 

Example metric: Relevance score on a 1-5 scale where 5 means the response directly addresses all aspects of the query. 

Coherence 

Can users follow the logic from start to finish? Evaluate the flow of ideas and logical connections between concepts. 

Example metric: 3-point scale (low/medium/high) rating of response coherence. 

Completeness 

Does the response cover all parts of the question? Check whether all aspects of the query were addressed without requiring follow-ups. 

Example metric: Percentage of query aspects covered in the response. 

Helpfulness 

Can users actually achieve their goals with this response? This might be the most important criterion, if the response doesn't enable action, users won't return. 

Example metric: Helpfulness rating from 1-5 based on how well the response enables user action. 

Safety 

Does the output avoid harmful content? Track policy violations and implement robust safety evaluations to prevent misuse. 

Example metric: Binary flag for presence of unsafe content or numeric risk score. 

Tone and style

Does the AI match your brand voice? Measure outputs against your communication guidelines to ensure consistency. 

Example metric: Style compliance score rating how well the output matches target tone. 

Contextual Awareness 

Does the AI remember previous interactions? Evaluate how well the system maintains conversation history in multi-turn exchanges. 

Example metric: Percentage of contextual errors in multi-turn conversations. 

Creativity (when appropriate) 

For creative applications, does the AI bring fresh ideas? Balance originality with usefulness. 

Example metric: Creativity rating on applications where innovation is desired. 

Different applications will naturally prioritize different criteria. A medical assistant might emphasize accuracy and safety, while a creative writing tool might value coherence and creativity. The key is determining which dimensions matter most for your specific use case and weighting your evaluation framework accordingly. 

Tools and techniques for implementing evals 

When implementing an evaluation system, follow these steps: 

1. Create golden examples: Collect 50-100 diverse user queries and manually label their underlying intent or expected responses 

2. Develop evaluation prompts: Create structured prompts that clearly define what success looks like

3. Generate synthetic test data: Use an LLM to create variations of your golden examples to increase test coverage 

4. Compare model performance: For side-by-side model evaluations, tools like Google's LLM Comparator can help visualize where and why outputs differ between models or prompt versions. This open-source tool lets you slice response data, identify performance patterns, and understand rationales for why one output might be superior to another. 

5. Build an auto-rater: Automate the evaluation process using evaluation prompts with a judge LLM. 

6. Maintain a dashboard: Track performance metrics over time, with breakdowns by relevant dimensions.

Implementation walkthrough across key dimensions 

The implementation process follows similar steps across all dimensions, with specific adaptations for each. Let's look at how to implement evals for each dimension: 

1. Create golden examples

For all dimensions, you'll need to collect 50-100 diverse examples that represent the range of user interactions your system will encounter. 

2. Develop evaluation prompts 

Tailor your evaluation prompts to each dimension: 

For understanding user intent: 

You are evaluating whether a shopping assistant correctly understands a user's intent. User query: "{user_query}" 

Assistant's understanding: "{assistant_response}" 

Actual intent: "{labeled_intent}" 

Determine if the assistant correctly understood the user's primary intent. Answer ONLY with "correct" or "incorrect". 

For relevance of recommendations: 

You are evaluating the relevance of product recommendations. 

User request: "{user_request}" 

User preferences: "{budget}, {occasion}, {stated_preferences}" 

Assistant's recommendations: "{product_list}" 

Rate how well the recommendations match the user's stated preferences on a scale of 1-5. Answer ONLY with a single number. 

For product knowledge accuracy: This is typically implemented as code rather than a prompt.

3. Generate synthetic test data 

Use LLMs to create variations of your golden examples for each dimension. For example: 

● For User Intent: Generate different ways users might ask for gifts 

● For Relevance: Create diverse preference combinations 

● For Accuracy: Include edge cases with unusual product attributes 

4. Build dimension-specific auto-raters 

Implement automation tailored to each dimension's needs: 

● Intent and Relevance: Use judge LLMs with the appropriate evaluation prompts 

● Product Accuracy: Use database validation scripts 

● Conversation Flow: Combine human evaluation with automated context tracking 

● Safety: Implement both content policy checkers and LLM judges 

5. Create a unified dashboard 

Track performance metrics over time, with breakdowns by relevant dimensions. This helps you identify trends and areas for improvement. 

Common pitfalls and how to avoid them 

Vague evaluation criteria - Define specific, measurable success criteria for each dimension

Inconsistent human evaluations - Periodically validate LLM evaluations against human judgments  

Over-reliance on LLM judges - Evaluate multiple dimensions including tone, creativity, and usefulness 

Focusing only on correctness - Create detailed rubrics with examples for each rating level 

Neglecting edge cases Intentionally include challenging test cases like ambiguous queries

Case study: Building robust AI evals - The BigLaw Bench approach 

The challenge 

Harvey, a legal AI assistant company, needed a comprehensive evaluation framework to measure how effectively AI models could perform real-world legal tasks. Existing benchmarks relied on multiple-choice questions or one-size-fits-all criteria, which failed to capture the complexity of actual legal work performed by lawyers. 

The eval approach 

The Harvey team developed "Big Law Bench" - a sophisticated evaluation framework tailored to real legal work: 

1. Task-specific rubrics: They created detailed, bespoke evaluation criteria for each legal task, mapped to actual billable work lawyers perform daily. 

2. Multi-dimensional scoring: Their evaluation methodology included: 

○ Answer scores: Positive points for meeting task requirements 

○ Source scores: Measuring how well models cited sources for their claims 

○ Penalty system: Negative points for hallucinations, irrelevant content, and other LLM failure modes 

3. Comprehensive coverage: The framework covered both transactional and litigation tasks across multiple practice areas, including document drafting, legal research, due diligence, and case management. 

4. Real-world applications: All tasks were derived from actual time entries billed by lawyers, ensuring the benchmark reflected genuine legal work. 

The results 

This evaluation framework revealed significant insights about AI model performance: 

● Leading foundation models (GPT-4o, Claude 3.5, Gemini 1.5) produced reasonably strong answers, solving the "blank page problem" and getting users more than halfway to a final work product 

● However, these models often struggled with providing traceable sources for their claims (source scores of 8-24%) 

● Harvey's specialized legal models outperformed general-purpose models, achieving 74% of lawyer-quality work while maintaining higher source scores (68%) 

● Transactional tasks generally saw better performance than litigation tasks, which required more complex reasoning and argumentation 

The evaluation framework became central to Harvey's product development, providing clear metrics on which to improve their models while identifying specific shortcomings in general-purpose AI when applied to specialized legal tasks. It also established an objective standard for comparing different AI solutions in the legal domain. Read more about this.

Beyond traditional metrics: Monitoring what really matters 

While business metrics like user engagement and conversion rates remain important, AI products require additional monitoring dimensions: 

Dimension key metrics to track warning signs:

1. User trust - Repeat usage rate, feedback sentiment. Declining return rate, negative feedback mentioning reliability 

2. Factual accuracy - Error rate by topic area Increasing error rates in specific domains 

3. Safety and ethics - Policy violation incidents, bias metrics. Any increase in harmful outputs or demographic disparities 

4. Contextual Awareness - Context maintenance score Rising rates of context confusion or irrelevant responses 

5. Adaptability Performance on novel queries 

Conclusion: The eval advantage 

As AI becomes integrated into more products, the ability to rigorously evaluate AI systems will increasingly distinguish exceptional product managers from the rest. By mastering the art and science of AI evals, you'll not only build better products but also position yourself as a leader in this rapidly evolving field. 

The product managers who thrive in the AI era will be those who can translate subjective human preferences into structured evaluation frameworks, continuously measuring and improving their AI systems across multiple dimensions. Start building your eval toolkit today, and you'll be well-prepared for the AI-driven product landscape of tomorrow.