AI products are measured differently from the products you’ve managed before. This glossary covers the metrics you need to evaluate performance, spot problems, and make decisions with your team. Not every metric applies to every product. Use this as a reference for the terms you encounter, not a list of everything you need to track. Many of these metrics only become meaningful when compared against a baseline. Where reference ranges exist, they are starting points, not targets. What matters is whether a metric is improving, degrading, or falling outside an acceptable range for your product.
Core Performance
These are the foundational metrics that tell you whether a model is doing its job. Some will feel familiar in name, but in AI these terms carry precise technical definitions that often differ from how they’re used in general conversation.
A content moderation model reviews user-submitted posts on a social platform and decides whether to remove them for policy violations. When it works well, legitimate posts stay up and harmful ones come down. When it gets things wrong, it either removes content it should have left alone or misses content it should have caught.
Accuracy The percentage of predictions the model gets right out of all predictions made. A correct decision means either removing a post that genuinely violated policy, or leaving up a post that was fine.
Say 1,000 posts come in. 200 violate policy and 800 are legitimate. If the model correctly removes the violating posts and correctly leaves up the legitimate ones, its accuracy is 100%.
The problem is that a model that never removes anything, leaving up all 200 violating posts, still gets 800 decisions right. That is 80% accuracy, while catching zero violations.
Now say 500 of the 1,000 posts violate policy and 500 are legitimate. A model that never removes anything now gets only 500 decisions right. That is 50% accuracy, and a number that low is impossible to ignore. Always check the distribution of your dataset before trusting an accuracy number.
Error rate The inverse of accuracy: the percentage of predictions the model gets wrong. In our example, 1,000 posts come in, 200 violate policy and 800 are legitimate. A model that never removes anything gets 800 decisions right and 200 wrong. That is 80% accuracy and a 20% error rate. At one million posts per day, 200,000 wrong decisions every day. That framing puts failure in terms stakeholders can feel.
Precision Of all the cases the model predicted as positive, the percentage that actually were. In content moderation, that means of all the posts the model flagged for removal, how many actually violated policy.
Say 1,000 posts come in. 200 violate policy and 800 are legitimate. A perfect model flags all 200 violating posts and nothing else. Every removal decision was correct. That is 100% precision.
Now say the model flags 300 posts for removal. Of those 300, only 150 actually violated policy. The other 150 were legitimate posts that should have stayed up. That is 50% precision. For every genuinely violating post the model caught, it also wrongly removed a legitimate one.
Precision matters most when false positives are costly. In content moderation, wrongly removing legitimate posts frustrates users and creates moderation appeals that cost time and trust.
Recall Of all the actual positive cases, the percentage the model correctly identified. In content moderation, that means of all the posts that actually violated policy, how many did the model catch.
Say 1,000 posts come in. 200 violate policy and 800 are legitimate. A perfect model catches all 200 violating posts. That is 100% recall.
Now say the model flags 300 posts for removal. Of those 300, 150 actually violated policy. That means 50 violating posts were never flagged and stayed up. The model caught 150 out of 200 violations. That is 75% recall.
Recall matters most when false negatives are costly. In content moderation, missing harmful content damages users and the platform’s reputation.
F1 score The harmonic mean of precision and recall: a single number that balances both. Using our example, the model had 50% precision and 75% recall. F1 score combines them into a single figure, in this case 60%.[^1] Notice that 60% is lower than both precision and recall individually. F1 penalizes imbalance between the two. A model with very high recall and very low precision, or the reverse, scores lower than one that balances both reasonably well. It is most useful when false positives and false negatives are both costly and your dataset is imbalanced, meaning one outcome is far more common than the other. It will not tell you which failure mode your product can tolerate more. That is a product decision you bring to the metric, not one the metric makes for you.
[^1]: F1 = 2 x (precision x recall) / (precision + recall). In this example: 2 x (0.50 x 0.75) / (0.50 + 0.75) = 0.60.
AUC-ROC (Area Under the Receiver Operating Characteristic Curve) A measure of how reliably a model’s confidence scores separate positive cases from negative ones. When the model reviews a post it produces a confidence score between 0 and 100%: how certain it is that the post violates policy. You then set a removal threshold: posts above it get removed, posts below it stay up. Where you set that threshold is a product decision that depends on which error your product can tolerate more: wrongly removing legitimate posts or missing violations.
That threshold decision is only meaningful if the model’s confidence scores are trustworthy. AUC-ROC measures how trustworthy those confidence scores are. An AUC-ROC score of 1.0 means violating posts consistently receive higher confidence scores than legitimate ones. A score of 0.5 means the confidence scores are so mixed the model is no better than random. Use it to evaluate and compare models before committing to one.
False positive rate / False negative rate The false positive rate is the percentage of negative cases the model incorrectly flagged as positive. The false negative rate is the percentage of positive cases the model missed. These are the terms stakeholders outside your technical team will most naturally reach for.
The false positive rate measures how often the model wrongly acts on something that does not warrant it. In our example, 1,000 posts come in. 800 are legitimate and should stay up. The model flags 300 posts for removal. Of those 300, 150 are legitimate posts that should have stayed up. That is 150 out of 800 legitimate posts, a false positive rate of roughly 19%: how often it cries wolf.
The false negative rate measures how often the model misses something it should have caught. Of the 200 violating posts, the model only caught 150. The other 50 were never flagged and stayed up. That is 50 out of 200 violating posts, a false negative rate of 25%: how often it misses something real.
These are related to precision and recall but measure from a different starting point. Precision and recall ask how the model performed on what it flagged and what existed. False positive and false negative rates ask how often it was wrong about each group: legitimate posts and violating posts.
Calibration Whether the model’s confidence scores reflect its real-world accuracy.
When the model flags a post it does not just say remove or keep. It also says how certain it is: 95% confident this violates policy, or 60% confident. That score exists because not every decision is equally clear cut. Some posts are obvious violations. Others are borderline.
A content moderator reviewing flagged posts would want to know which flags are near certain and which are borderline, so they can prioritize their queue. The confidence score is supposed to tell them that.
Calibration measures whether it actually does. If the model says 95% confident, it should be right about 95% of the time. If it is only right 60% of the time when it says 95% confident, the score is misleading and the moderator cannot rely on it.
Calibration matters most when confidence scores are shown to people making decisions based on them. A poorly calibrated model does not just produce wrong answers. It produces wrong answers that look certain.
Text and Generation Quality
These metrics apply to models that produce language: translations, summaries, answers, drafts, or any other text output. They measure how closely a model’s output matches a reference or human standard.
A support team uses an AI to summarize customer support tickets before they reach a human agent. Each ticket is a thread of messages documenting the customer’s issue and everything that has been tried to resolve it. The agent reads the AI’s summary instead of the full thread. A good summary covers the customer’s issue, what was already tried, and what the customer is asking for.
BLEU (Bilingual Evaluation Understudy) A measure of how closely a model’s text output matches a human-written reference, using n-gram matching: sequences of one, two, three, or four consecutive words. If a human summarizer writes “customer’s order was delayed; requesting refund” and the model writes the same, that is a high BLEU score. If the model writes “shipment experienced a delay and the customer would like their money back,” the score is lower even though the meaning is the same. BLEU is precision-focused: it checks how much of what the model wrote matches the reference, but does not penalize the model for leaving out important information. Scores range from 0 to 1. Even two competent human summarizers writing about the same ticket tend to score in the 0.6 to 0.7 range against each other, since they naturally use different words and phrasing. That makes 0.6 to 0.7 the practical upper bound rather than a target. Scores above 0.7 in a summarization context are worth questioning rather than celebrating. They often signal something has gone wrong in how the model was trained or evaluated. What counts as a good score depends on your task, domain, and baseline.
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) A measure of how much of a human-written reference appears in the model’s output. Gisting means summarization: capturing the essential meaning of a text. Where BLEU checks how much of the model’s output appears in the reference, ROUGE checks how much of the reference appears in the model’s output. If the human summary mentions that the customer threatened to cancel their account and the model’s summary does not, ROUGE penalizes that omission. Most commonly used for summarization tasks, where missing critical information is a more serious failure than including extra detail. Scores range from 0 to 1, with higher scores indicating better coverage of the reference. What counts as a good score depends on your task, domain, and baseline. ROUGE-L measures the longest common subsequence: the longest set of words that appear in the same order in both texts, even if other words fall between them. If the reference summary says “customer requested a full refund after two failed attempts” and the model writes “customer asked for refund after attempts,” ROUGE-L identifies “customer refund after attempts” as a match because those four words appear in the same order in both texts. It is generally the most informative ROUGE variant for assessing whether a summary captures the structure and flow of the source.
BERTScore A measure of how closely the meaning of a model’s output matches a human-written reference. Uses a language model to compare meaning rather than counting word matches. If the reference summary says “customer is threatening to cancel” and the model writes “customer indicated they may not continue their subscription,” BERTScore recognizes these as semantically equivalent. BLEU would score them lower because the words do not match. Scores range from 0 to 1, with higher scores indicating closer semantic alignment with the reference. BERTScore is generally considered to correlate more closely with human judgment than BLEU or ROUGE, though results vary by domain and task. What counts as a good score depends on your task, domain, and baseline. Evaluating with BERTScore has a higher computational cost than BLEU or ROUGE: it requires running a language model for every evaluation, which makes it slower to run at scale.
Perplexity Measures how fluently a language model generates text by tracking how well it predicts each word as it writes. A summary that meanders, repeats itself, or uses awkward constructions has higher perplexity. A crisp, natural summary has lower perplexity. Used primarily as an internal model quality signal during training and development. A model with high perplexity on support ticket content may need further training on that domain. Unlike most metrics here, perplexity does not require a reference text. It measures the model against itself.
Hallucination rate The percentage of outputs in which the model generates information that was not in the source. A hallucinated summary might say the customer has contacted support three times before when the ticket shows only one prior contact, or state that a refund was already issued when it was not. A human agent who acts on a hallucinated summary may make decisions based on information that was never true. Hallucination rate is one of the most PM-relevant metrics in generative AI: it directly affects user trust, and it can be invisible to users who lack the domain knowledge to catch errors. High hallucination rates are often a product-ending problem in enterprise or professional contexts. See also: Ethics section.
Robustness and Generalization
These metrics measure how well a model performs outside the clean, controlled conditions of development. Production environments are noisier and more variable than test datasets, and robustness metrics are where that gap shows up.
A mobile banking app uses an AI model to categorize transactions automatically: groceries, utilities, subscriptions, dining, and so on. Users see the categories in their spending dashboard. The model was trained on clean, structured transaction data. In production it receives merchant names with typos, truncated descriptions, foreign characters, and unusual formatting. Some users also try to manipulate their categorization to game spending reports or budgeting features.
Adversarial test accuracy The percentage of deliberately crafted inputs the model handles correctly despite an attempt to fool it. In our example, a user might alter a merchant name to make a restaurant transaction appear as a utility bill. A model with low adversarial test accuracy would be fooled and categorize the transaction as a utility. A model with high adversarial test accuracy categorizes it correctly as dining. Adversarial test accuracy matters most for products where users have an incentive to game the system.
Input noise tolerance The percentage of imperfect inputs the model handles correctly. In our example, merchant names arrive truncated, misspelled, or formatted differently depending on the bank or payment processor: “WHOLEFDS MKT #1234,” “WF MKT,” and “Whole Foods Market” may all refer to the same merchant. A model with low noise tolerance might categorize “WHOLEFDS MKT #1234” as groceries but “WF MKT” as uncategorized, even though both are the same store. A model with high noise tolerance recognizes all three as the same merchant and categorizes them consistently as groceries. Particularly important for products where you cannot control the quality of incoming data.
Output consistency The percentage of runs that produce the same result when given the same input. AI models can produce different outputs each time they run, even on identical inputs. In our example, a customer’s transaction from the same merchant is categorized as dining on Monday and entertainment on Thursday, with no change to the input. The customer’s spending totals across categories are now wrong, and they cannot trust the dashboard to reflect their actual habits. For products where users depend on consistent behavior over time, high output consistency is a functional requirement, not just a quality preference.
Unseen data performance A comparison between how well the model performs on data it was trained on versus data it has never seen. Your team measures the model using the same metric on both, whether that is accuracy, F1 score, or another measure relevant to the task. In our example, a new merchant that did not exist when the model was trained might be assigned to the wrong category or flagged as uncategorized because the model has no basis for a confident prediction. A large gap between training performance and new data performance indicates the model has overfit: it learned the training data too narrowly to generalize beyond it.
Prompt sensitivity The degree to which a model’s output changes when the wording of an input varies in a meaningful way. Where input noise tolerance measures robustness to corrupted or imperfect data, prompt sensitivity measures robustness to legitimate variation in how the same thing is described. In our example, “NETFLIX.COM” might be categorized as entertainment while “Netflix subscription” is categorized as subscriptions, even though both refer to the same service. For products where output quality depends on consistent model behavior, high prompt sensitivity is a reliability risk: minor variations in how data arrives produce meaningfully different results for the user.
Explainability
SHAP values SHapley Additive exPlanations. A method for explaining individual model predictions by quantifying how much each input feature contributed to a specific output. If a loan approval model rejects an application, SHAP values can show which factors drove that decision and by how much: income, credit history, employment status, and debt-to-income ratio. SHAP values are widely used as an explainability tool in production AI, and they are increasingly required in regulated industries where decisions must be auditable. You will encounter SHAP values when your team needs to explain model behavior to regulators, customers, or internal stakeholders, or when diagnosing why a model is making unexpected decisions.
Fairness
Fairness metrics measure whether a model treats different groups of people equitably.
A financial services company uses an AI model to assess credit applications. The model is trained on historical approval data and produces a score that determines whether an application is approved, referred for manual review, or declined. These decisions have direct financial consequences for applicants, and fair lending laws in many jurisdictions require that they be made without discrimination on the basis of race, gender, age, or other protected attributes.
Bias score A broad term for any measurement of systematic disparity in how a model performs across demographic groups: race, gender, age, language, or other protected attributes. Because bias score is a category of measurements rather than a single metric, the number only means something in context: what group comparison was made, what outcome was measured, and what your product considers an acceptable disparity. In our example, a bias score might surface that the model approves applications from one demographic group at a significantly lower rate than another, or that its error rate is higher for certain groups. A model can be unbiased by one measure and biased by another, which is why bias score is typically the entry point that leads to more specific fairness analysis rather than a final verdict on its own.
Demographic parity Whether the model produces positive outcomes at equal rates across groups, regardless of whether those outcomes are accurate. In practice it is measured as the gap in approval rates between groups. A 5% gap means one group is approved at a rate 5 percentage points higher than another. In our example, demographic parity would require that the model approves credit applications at the same rate across demographic groups. It does not require that the approvals be equally well-calibrated to actual creditworthiness, only that the approval rate be equal. Used in contexts where equal representation of outcomes is the primary goal, and where the underlying data may itself reflect historical discrimination that equal accuracy would perpetuate.
Equalized odds Whether the model’s true positive rate and false positive rate are equal across groups. In practice it is measured as the difference in true positive rates and false positive rates across groups. A 5% gap means one group’s rate is 5 percentage points higher than another’s. Unlike demographic parity, equalized odds accounts for actual creditworthiness: it requires that the model approve equally qualified applicants at the same rate, and decline equally unqualified applicants at the same rate, across demographic groups. In our example, a model satisfying equalized odds would not approve unqualified applicants from one group at a higher rate than another, nor would it decline qualified applicants from one group more often than another. More demanding than demographic parity because it requires equal true positive rates and equal false positive rates across groups, not just equal approval rates.
Ethics
Fairness covers equitable treatment across groups. Two other dimensions matter just as directly: harmful outputs and false information.
Toxicity score The percentage of outputs that contain harmful, offensive, or policy-violating content: hate speech, harassment, graphic content, or other material that violates product or platform standards. Toxicity scoring is typically automated using a classifier trained on human-labeled examples of harmful content. PMs working on consumer or enterprise products with open-ended text generation need a toxicity baseline before launch and continuous monitoring after, since model behavior can shift as usage patterns evolve.
Hallucination rate The percentage of outputs in which the model generates false, fabricated, or unsupported information. When a model does this confidently, it can cause real harm. Incorrect medical information, fabricated legal citations, and invented product capabilities are all examples. Hallucination rate is both a generation quality metric and an ethical risk metric, and in high-stakes domains it is often the latter framing that determines product viability. See also: Text and Generation Quality section.
Operational / Production
These metrics measure how an AI product behaves in production, under real load, with real users. They are where model quality intersects with infrastructure, cost, and user experience.
A B2B SaaS company builds an AI feature that generates first-draft responses to RFP questions. Sales reps submit a question and receive a draft they can edit and send to the prospect. The product draws on a knowledge base of product documentation and past RFP answers. Hundreds of reps use it across multiple time zones, with peak usage during business hours and at quarter end.
Model drift Degradation in a model’s performance over time, as the patterns it learned during training become less applicable to current inputs. Model drift is not measured directly. It shows up as a decline in whatever performance metric your team is already tracking, accuracy, F1 score, or another measure relevant to your product. In our example, the product evolves: new features ship, pricing changes, and the competitive landscape shifts. A model trained on last year’s documentation starts generating drafts that reference outdated capabilities or miss new ones. Model drift is gradual and often invisible without active monitoring. The model does not break; it just gets slowly worse. Detecting it requires tracking performance metrics over time against a reliable baseline.
Data drift A change in the statistical distribution of inputs the model receives in production, relative to what it was trained on. It is detected by monitoring the statistical properties of inputs over time and comparing them to what the model was trained on. In our example, data drift might occur as the company moves upmarket: enterprise RFPs ask different questions than the mid-market RFPs the model was trained on, with more emphasis on security, compliance, and integration. Data drift does not mean the model has changed. It means the world it is operating in has changed. Data drift often precedes model drift: when inputs change significantly enough, model performance degrades as a result.
Latency How long the model takes to respond. In our example, a rep submitting an RFP question mid-call with a prospect needs a draft in seconds, not minutes. Latency is typically reported as P50, P95, and P99: the response time at the 50th, 95th, and 99th percentile of requests. P50 is the median experience. P95 and P99 capture the tail, meaning the slowest responses affecting a small but real proportion of users. A product where P50 is 400ms but P99 is 8 seconds means half of reps get a response in under 400ms, but one in a hundred waits eight seconds. That one in a hundred could be the rep on a live call. A rep who hits a slow response at a critical moment loses trust in the product faster than the average response time would suggest. Higher accuracy often comes with higher latency, making this one of the most common model quality tradeoffs a PM will negotiate.
Throughput The number of requests the model can process per unit of time, typically per second, at a given latency threshold. In our example, quarter end brings a surge in RFP activity across the entire sales team simultaneously. Where latency measures speed for a single rep, throughput measures capacity across all of them. A model that responds in two seconds for one rep but degrades to thirty seconds under peak load has a throughput problem, not a latency problem. Throughput directly affects whether the product remains usable at quarter end.
Cost per inference / token cost The compute cost of running the model once on a single input. For LLM-based products, cost is typically measured at the token level, the unit most cloud AI vendors bill by, where a token is roughly four characters of text. A single query might cost very little. In our example, a detailed RFP response might consume thousands of tokens per query. At hundreds of reps submitting multiple queries per day, even a small per-query cost becomes significant and sits directly inside the product’s cost of goods sold. Cost per inference and cost per token are the metrics that connect model quality decisions to margin and pricing decisions.
Rejection / abstention rate The percentage of inputs for which the model declines to produce an answer, either because it detects low confidence, an out-of-scope query, or a safety guardrail. In our example, a rep asking about a competitor’s product or a question outside the knowledge base should receive a clear decline rather than a hallucinated draft. A model that can say “I don’t know” avoids a class of errors that overconfident models generate. The tradeoff is usability: a high abstention rate means reps frequently hit dead ends. Setting the right abstention threshold is a product decision about which failure mode is more harmful: wrong drafts or no drafts.
User satisfaction rate The percentage of AI interactions that users rate positively, typically through explicit signals like thumbs up/down, star ratings, or follow-up surveys. In our example, a rep who edits a draft heavily before sending is signaling something the model metrics may not capture: the draft was technically grounded but missed the tone, the level of detail, or the specific concern the prospect raised. User satisfaction rate captures what the model metrics cannot: whether the draft was actually useful for that rep, in that conversation, with that prospect.
RAG-Specific Metrics
RAG (Retrieval-Augmented Generation) is an architecture where a model’s responses are grounded in documents or data retrieved at query time, rather than relying solely on what the model learned during training. These metrics apply specifically to products built on RAG pipelines. If your product uses a standard language model without a retrieval component, these metrics do not apply.
An HR team deploys an internal knowledge assistant that employees use to ask questions about company policy, benefits, and procedures. When an employee asks a question, the assistant searches an internal document library, retrieves the most relevant policy documents, and generates an answer. A good response draws from the right documents, represents them accurately, and actually answers what the employee asked.
Retrieval precision Of all the documents the assistant retrieves when an employee asks a question, the percentage that are actually relevant to that question. If an employee asks “how many vacation days do I get in my first year?” and the assistant retrieves five documents, two of which are about vacation policy and three of which are about general HR procedures, retrieval precision is 40%. High retrieval precision means the assistant is surfacing useful material. Low precision means it is retrieving noise alongside the relevant material, increasing the chance the model draws from the wrong source and generates an inaccurate or irrelevant answer.
Retrieval recall Of all the documents in the library that are relevant to the employee’s question, the percentage that the assistant actually surfaces. If the vacation policy is split across two documents and the assistant retrieves only one, its recall is 50%. The employee asking about vacation days might get an accurate answer about their first-year entitlement but miss the accrual schedule for subsequent years, because the second document was never retrieved. Precision and recall pull in opposite directions in retrieval. Increasing one often decreases the other, and the right balance depends on whether missing relevant content or including irrelevant content is the more costly failure for your product.
Groundedness / Faithfulness Whether the assistant’s answer is supported by the retrieved documents and accurately represents what they say. A grounded response draws its claims from the retrieved content rather than from the model’s general training knowledge. A faithful response does not distort, overstate, or contradict what the source material says. Both matter: a response can draw from the right documents and still misrepresent them, and a response can accurately represent a document that was not the right source to begin with. These two terms are often used interchangeably across RAG evaluation frameworks. What matters in practice is that your evaluation covers both failure modes: the model ignoring its retrieved sources, and the model distorting them. In an HR context, either failure can mean an employee acts on incorrect policy information.
Answer relevance Whether the final response actually addresses what the employee asked. In practice it is measured as the percentage of responses that actually address what the employee asked. The assistant can retrieve the right documents, generate a grounded and faithful response, and still answer the wrong question. An employee asking “can I carry over unused vacation days?” needs a direct answer to that question, not a general summary of the vacation policy. Answer relevance is the end-to-end quality check that the other RAG metrics do not cover.
Agentic AI Metrics
Agentic AI systems take sequences of actions to complete tasks on a user’s behalf: calling tools, browsing the web, writing and executing code, and interacting with external services. These metrics apply to products built on agentic architectures. If your product uses a model to generate responses rather than take actions, these metrics are less relevant.
The scenario below uses the same HR setting as the RAG section, but a different system. Where the RAG knowledge assistant answers questions by retrieving documents, this agent takes action: it provisions access, books meetings, and updates systems. The distinction matters because the failure modes are different.
An HR team deploys an AI agent to handle employee onboarding. When a new hire is confirmed, the agent provisions software access, schedules orientation sessions, sends welcome emails, and updates the HR system. Each step requires a separate action. A failure at any point can leave a new employee without the tools or information they need on their first day.
Tool call success rate The percentage of individual actions the agent completes without error. Each software provisioning request, calendar booking, email send, and HR system update is a separate tool call. An agent that fails 10% of individual tool calls has a much higher overall failure rate when onboarding a single employee requires fifteen sequential steps, because each failure compounds on the ones before it. Tool call success rate helps isolate whether failures are coming from the model’s reasoning or from its execution.
Task completion rate The percentage of full onboarding sequences the agent completes successfully from start to finish. Where tool call success rate measures individual actions, task completion rate measures outcomes. An agent can have a high tool call success rate and a low task completion rate if it is completing individual steps correctly but making poor decisions about sequencing, such as sending the welcome email before access has been provisioned. Task completion rate is the primary metric for evaluating whether the agent is delivering value.
Instruction following rate The percentage of tasks where the agent executes the user’s actual request rather than a generic interpretation of it, and respects constraints and scope boundaries. Different roles require different software access: a new engineer needs different tools than a new sales rep. An agent with a low instruction following rate might provision the wrong tools, schedule with the wrong manager, or send a template email when a customized one was specified. This metric is distinct from task completion rate: an agent can complete an onboarding sequence while not following the role-specific instructions, or follow instructions precisely while failing to complete the sequence due to an external error.
Human preference rate / LLM-as-judge score How often HR reviewers, or a separate AI model acting as a judge, rate the agent’s onboarding communications and outputs as meeting quality standards. Human preference rate is used in offline evaluation to compare agent versions before shipping. LLM-as-judge uses a separate model to score outputs on defined quality criteria at scale, providing a faster and cheaper signal than full human review. Both are proxies for quality in contexts where there is no single correct answer, only better or worse ones.
