Skip to content

Ledger entryNo. 369037

LLM Evaluation

Also called: LLM evals · AI evals · Chatbot evaluation · LLM-as-a-judge

Definition
Offline testing of a bot or AI agent against a fixed set of real cases, scored by code checks, rubrics and LLM judges, before a prompt or model change ships.

Bots, flows & AI

Mode · ab test

Work it out

Does the new prompt really beat the old one?

14 / 14

Metric · run 1

Golden-set pass rate

Visitors per variant400 / 400

  • ACurrent prompt82.8%± 3.7 pts, 95% interval 79.0% to 86.5%
  • BNew prompt + refund toolWinner94.5%± 2.2 pts, 95% interval 92.3% to 96.7%
  • CSame prompt, smaller model79.5%± 4.0 pts, 95% interval 75.5% to 83.5%
  • Observed rate
  • 95% interval (± 1.96 × standard error)
  • True rate (unknown in a real test)
  • Where the leaders overlap

B (New prompt + refund tool) wins

Observed 94.5% (+14.2% vs A). Its 95% interval no longer overlaps any other variant, so the gap is very unlikely to be chance.

  • ACurrent promptControl

    Baseline in production today.

    331 of 400 converted

  • BNew prompt + refund tool

    Adds a refund-eligibility tool and an explicit reply-language rule.

    378 of 400 converted · +14.2% vs A

  • CSame prompt, smaller model

    Cheaper and faster, but skips tools more often.

    318 of 400 converted · −3.9% vs A

An offline regression run, not a customer test: each "visitor" is one graded golden-set case. Watch the intervals narrow — a 5-point lead can still sit inside the noise.

400 visitors per variant. B (New prompt + refund tool) wins. A: 82.8%, 95% interval 79.0% to 86.5%; B: 94.5%, 95% interval 92.3% to 96.7%; C: 79.5%, 95% interval 75.5% to 83.5%.

What is LLM evaluation?

LLM evaluation (often just "evals") is the practice of testing a chatbot or AI agent against a fixed collection of cases — a golden dataset — and scoring every answer, so a change to the system prompt, the model, the retrieval settings or the knowledge base can be judged before it reaches a customer. Each case pairs a realistic input, such as an anonymised WhatsApp message with its conversation history, with the behaviour you expect: the facts the reply must contain, the tool it should call, the language it should answer in, or the point at which it must hand over to a human.

Scoring uses three kinds of grader, cheapest first. Code checks verify anything deterministic — valid JSON, the right tool name and arguments, a reply under the length limit, the customer's language preserved, no phone number leaked. Rubric grading has people (or a model) rate qualities such as correctness, groundedness in the knowledge base and tone on a defined scale. LLM-as-a-judge uses a second model with that rubric to grade at scale, which only works once its scores have been checked against human labels.

Evaluation is offline and comparative. The same cases run against the current version (the baseline) and the candidate, several times each because outputs vary, and results are broken down by slice — intent, language, risk area — rather than reported as one number. That is what distinguishes it from an A/B test, which exposes real customers to two live variants and measures outcomes such as conversion or CSAT; evals decide whether a change is safe to put in front of customers at all.

Key takeaways

  • LLM evaluation runs a bot against a fixed golden set of real cases and scores the results before any change reaches customers.
  • Grade actions as well as words: the right tool, the right arguments, the right language, a handoff when one was needed.
  • Small sets are noisy — at a 90% pass rate, 50 cases give about ±8 points of uncertainty and 400 cases about ±3.
  • Calibrate an LLM judge against human labels before trusting its scores, and watch for length and position bias.
  • Evals are offline tests; A/B tests measure live customers. Run evals first, then A/B test what passes.

The figures

LLM evaluation: key facts and limits

Uncertainty at 90% pass, 50 cases
About ±8 points (95% interval)
Uncertainty at 90% pass, 200 cases
About ±4 points
Uncertainty at 90% pass, 400 cases
About ±3 points
Runs per case
3 or more, to expose non-determinism
Grader order
Code checks → rubric → calibrated LLM judge
Release gate
Thresholds met and no critical slice regresses

How does LLM evaluation work?

  1. Build the golden set from real traffic

    Sample a few hundred conversations from production with personal data masked, stratified by intent, language (English, Hindi, Hinglish, regional), and risk: refunds, abuse, prompt injection attempts, out-of-scope requests. Write the expected behaviour for each.

  2. Define graders, deterministic first

    Code checks for tool calls, schema validity, language and length; rubrics for correctness, groundedness and tone; an LLM judge only for what code cannot check, calibrated against human labels on a sample.

  3. Run baseline and candidate the same way

    Same cases, same retrieval snapshot, same settings, at least three runs per case to expose non-determinism. Record tokens and latency alongside quality.

  4. Compare with uncertainty, by slice

    Report pass rates with confidence intervals for every slice. A small overall gain can hide a large drop in one language or a safety category.

  5. Gate the release and keep growing the set

    Ship only when the candidate meets thresholds and no critical slice regresses. Add every production failure as a new case so the same mistake cannot return unnoticed.

In code

LLM evaluation in code

import json, math
 
def wilson(passed: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% confidence interval for a pass rate."""
    if n == 0:
        return (0.0, 0.0)
    p = passed / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return (c - h, c + h)
 
def grade(case: dict, out: dict) -> bool:
    exp = case["expected"]
    if exp.get("tool") and out.get("tool_call", {}).get("name") != exp["tool"]:
        return False                                   # wrong or missing action
    if out["language"] != case["language"]:
        return False                                   # replied in the wrong language
    if exp.get("handoff") and not out.get("handoff"):
        return False                                   # must escalate, did not
    verdict = judge(rubric=RUBRIC, case=case, reply=out["text"])  # LLM judge, calibrated
    return verdict["score"] >= 4                       # 1–5 scale
 
def run(bot, cases: list[dict], runs: int = 3) -> dict:
    by_slice: dict[str, list[bool]] = {}
    for case in cases:
        for _ in range(runs):                          # outputs vary: sample each case
            ok = grade(case, bot.respond(case["history"], case["message"]))
            by_slice.setdefault(case["slice"], []).append(ok)
    return {s: (sum(r) / len(r), wilson(sum(r), len(r))) for s, r in by_slice.items()}
 
cases = [json.loads(line) for line in open("golden_set.jsonl", encoding="utf-8")]
baseline, candidate = run(current_bot, cases), run(new_bot, cases)

Deterministic checks run first and short-circuit; the LLM judge only grades replies that already did the right thing. Results come back per slice with intervals.

Example: A model switch at a Bengaluru skincare brand

A Bengaluru D2C skincare brand wants to move its WhatsApp agent to a cheaper model. Its golden set has 320 cases across order tracking, product advice, returns and complaints, in English, Hindi and Hinglish, each run three times. Overall pass rate barely moves — 87% to 88% — but the Hinglish returns slice falls from 84% to 71%: the new model keeps replying in English and twice skips the refund-eligibility tool. The team adds an explicit language instruction and a tool-use example to the prompt, reruns, recovers the slice to 85%, and only then starts a small live A/B test on 10% of chats.

Costly mistakes

Common mistakes with LLM evaluation

  • Trusting an uncalibrated judge

    LLM judges tend to favour longer answers, the first option shown and text that resembles their own. Check agreement with human labels on a sample, randomise order and score against a written rubric.

  • One number for everything

    An overall pass rate hides slice regressions. Report per intent, language and risk category, and block releases on critical slices such as refunds, safety and handoff.

  • Testing words, not actions

    A fluent reply that skipped the order-lookup tool or invented a refund is a failure. Grade tool calls, arguments and handoff decisions explicitly.

  • Overfitting to the golden set

    Tuning prompts until a fixed set passes teaches the system the test. Keep a held-out set, refresh cases from production regularly and retire stale ones.

  • Reading noise as progress

    A 3-point gain on 60 cases is within the noise. Use confidence intervals, enough cases and repeated runs before calling a winner.

Why does LLM evaluation matter?

LLM systems fail quietly. A prompt tweak that improves English refund answers can break Hinglish order tracking; a cheaper model can start skipping the handoff on abusive messages; a new knowledge-base article can contradict an old one. None of that shows up in a few manual chats, because outputs vary run to run. A golden set and a regression gate turn "it seemed fine when I tried it" into a measured pass rate per slice, and give a team the confidence to change prompts and models weekly instead of fearing every release.

Best practices for LLM evaluation

  • Start with 50 hand-picked hard cases today; a small honest set beats none.
  • Include adversarial cases — prompt injection, abuse, requests for other customers' data.
  • Track cost and latency per case next to quality, so cheaper is never silently worse.
  • Run the suite automatically on every prompt, model or knowledge-base change.
  • Version the golden set with the prompt, so results are reproducible months later.

LLM Evaluation FAQs

What is LLM-as-a-judge?
It is using a second language model, given a scoring rubric, to grade another model's answers at scale. It is useful only after its scores have been compared with human judgements and its known biases checked.
How many test cases do I need to evaluate a chatbot?
Enough for the differences you care about to rise above noise. At a 90% pass rate, 50 cases leave about ±8 points of uncertainty and 400 cases about ±3, so start small but grow the set over time.
What is the difference between LLM evaluation and A/B testing?
Evaluation is offline: fixed cases, no customers involved, run before release. A/B testing is live: real customers see two variants and you measure outcomes such as conversion or CSAT.
What should a WhatsApp bot evaluation measure?
Correct and grounded answers, the right tool calls and arguments, replies in the customer's language and within length limits, safe refusals, and handoffs when a human is needed — broken down by intent and language.
How often should LLM evals run?
On every change to the prompt, model, retrieval settings or knowledge base, and on a schedule besides, because hosted models and your content both change underneath you.

See InfiQ pricing

Put LLM Evaluation to work

InfiQ is an official Meta Business Partner. Start a 7-day free trial — no card needed — and see this running on your own number.

Start free trialBrowse all 550 terms