Building Reliable LLM Evaluation Pipelines

LLM-powered applications can vary because prompts, model configuration, retrieval, tools, application code, and upstream evidence all affect the result. Looking at a few selected responses hides that interaction. Even a single aggregate pass rate discards the case identity, failure type, affected slice, evaluator version, and release consequence that an engineer needs.

A useful pipeline keeps enough evidence to answer: which cases improved, which regressed, which categories moved, why each judgment occurred, whether the evaluator was applicable, where human review is required, and which policy made the release decision. The final decision should be reproducible from named inputs rather than reconstructed from a dashboard screenshot.

Evaluation does not prove universal correctness, intelligence, production reliability, or safety completeness. It creates bounded evidence about a named system version under documented test conditions. That narrower claim is still valuable because it can block a known regression before deployment.

An application response may reflect system instructions, prompt templates, conversation state, retrieval ranking, document versions, tool schemas, tool results, post-processing, safety rules, and model/provider configuration. The evaluation target should name this application bundle. Calling it only a model evaluation can assign a failure to the wrong component.

Define task success and safe failure before choosing metrics. A knowledge assistant may need a valid response contract, supported evidence references, required facts, abstention when evidence is insufficient, and review routing for policy-sensitive work. Those are different properties and deserve different checks.

OpenAI's evaluation guidance recommends task-specific tests, logging, automated scoring where possible, and continuous evaluation. NIST's Measure function similarly emphasizes documented test sets, metrics, methods, deployment context, and regular evaluation. The pipeline here is an engineering synthesis of those principles, not a vendor-defined universal standard.

Freeze case IDs, inputs, supplied evidence, expected behavior, and risk labels for a comparison run. Candidate A and Candidate B must receive the same cases. Otherwise a score difference may come from changed inputs rather than changed application behavior.

Build coverage deliberately: happy paths establish basic capability; edge and historical-failure cases preserve lessons; insufficient-evidence cases test abstention; adversarial or mutated cases probe boundaries; and high-risk cases test policy and escalation. Slices should correspond to meaningful product risks, not arbitrary labels invented after seeing results.

The 12-case fixture below is intentionally small and synthetic. It demonstrates pipeline mechanics, not real-world distribution or statistical significance. A production dataset needs domain review, provenance, privacy controls, sampling strategy, maintenance ownership, and enough cases to support the decisions being made.

A reproducible run identifies the dataset and case revision, application code, prompt/system instructions, model/provider configuration, retrieval settings, tools, grader, rubric, and release policy. If any of these change, the reported result describes a different experiment.

A run manifest should also include a run ID, timestamp or as-of time, execution environment, and artifact references. Store output references rather than hidden chain-of-thought. For sensitive systems, access controls and retention rules apply to evaluation records just as they do to production telemetry.

The executable example versions its dataset, two recorded candidates, deterministic grader, and critical-regression policy. It does not call a provider, so there is no real model configuration to record.

A candidate change enters a controlled run with versioned inputs. The runner executes the frozen cases, deterministic graders establish structural and task-specific facts, and optional semantic or human review handles judgments that code cannot safely represent. Per-case records then feed slice aggregation and candidate-to-baseline regression classification.

The release policy consumes those inspectable results. PASS, BLOCK, and REVIEW are policy outcomes, not grader vibes. A broken evaluator becomes evaluation_error and should fail or invalidate the run; it must not silently count as a model failure or pass.

Use deterministic checks for JSON parsing, exact keys and types, enum values, evidence-ID membership, output contracts, explicit abstention status, review flags, and fixture-specific markers. These checks are repeatable, cheap to inspect, and easy to connect to a precise failure reason.

Deterministic does not mean semantically complete. The example's required and forbidden markers are controlled educational assertions for hand-authored records. Substring matching cannot determine general factual correctness, groundedness, intent, or the meaning of negation. Production systems should use code only for properties code can genuinely establish.

Separate structural checks from task and policy checks so the result says why it failed. A schema error, missing fact, invalid citation, unsupported claim, abstention error, and review-routing error lead to different fixes.

Model-based graders can help assess semantic correctness, instruction following, style, relevance, or groundedness when exact-match rules are too brittle. Pointwise grading can compare a response to a rubric; pairwise grading can compare candidates. Both need a well-scoped rubric and inspectable explanations or labels.

A grader is another probabilistic system. Its prompt, model, sampling configuration, rubric, and preprocessing are versioned components. Position bias, rubric ambiguity, correlated model errors, and prompt sensitivity can change judgments. High-risk cases should not inherit automatic authority from a judge model.

The local fixture executes no LLM judge and records no fake judge score. Its architecture diagram marks semantic grading as an optional external boundary. If added in a real pipeline, unresolved disagreement should route to human adjudication rather than be averaged away.

Build a human-labeled calibration set with clear adjudication notes. Compare evaluator decisions with those labels, inspect false positives and false negatives, and analyze disagreement by slice. A useful evaluator can still be unreliable on one risk category.

Review rubric ambiguity, spot-check changed cases, and track evaluator drift whenever the grader, prompt, rubric, or candidate distribution changes. Google Cloud's judge-model guidance explicitly uses human ratings as ground truth for assessing a model-based metric; that is a practical reminder that the evaluator needs evidence too.

There is no universal agreement threshold. Required confidence depends on consequence, review capacity, and the decision the evaluator controls. Record adjudication and evaluator version so a later regrade cannot erase the original release evidence.

Northstar Components is fictional. Each fixture record identifies a slice, allowed evidence IDs, exact controlled markers, expected answer/abstain/review behavior, a critical flag, and the recorded outputs for Candidate A and Candidate B. No prompt is sent to a provider and no real employee, client, handbook, or business result appears in the data.

Candidate A is the frozen baseline, northstar-assistant-a-1.0. Candidate B, northstar-assistant-b-1.1, represents a revision that improves required-fact coverage, schema compliance, abstention, and conflict routing. It also introduces a policy-sensitive credential-sharing recommendation in case-09.

That intentional regression is central: Candidate B passes more cases overall but fails a case marked critical before execution. The release gate therefore rejects it. This is recorded application-output evidence, not a benchmark of OpenAI, Anthropic, Google, or any other model.

The evaluator passes 7 of 12 Candidate A records and 11 of 12 Candidate B records. Candidate B improves five cases: a missing return window, an incomplete multi-source dispatch answer, an unsupported travel claim, conflicting maintenance notices, and a missing schema field.

Six cases remain passing. One case regresses: case-09 changes from a review response to an answer recommending a prohibited credential-sharing action. Because that case was pre-labeled critical and belongs to the policy-sensitive slice, the regression is not offset by improvements elsewhere.

Counts here describe only these authored records. They do not estimate accuracy, error rates, prevalence, or production risk. Their purpose is to show how per-case classification preserves a release-blocking signal that an aggregate improvement would conceal.

Candidate B improves routine factual lookup from 2/3 to 3/3, multi-evidence synthesis from 1/2 to 2/2, insufficient-evidence behavior from 1/2 to 2/2, conflicting evidence from 0/1 to 1/1, and structured response from 0/1 to 1/1. High-risk review remains 1/1.

The policy-sensitive slice falls from 2/2 to 1/2. That slice is small, so an overall count makes the decline look numerically minor. Its consequence is not minor: the failed case recommends using another person's credentials and bypasses the required review path.

Slices should be defined from product behavior and risk before the run. Searching for a favorable subgroup afterward can mislead just as easily as an aggregate. Also retain case-level records because even a slice average can hide one critical failure.

Compare the baseline and candidate result for the same case. A fail-to-pass transition is improved; pass-to-fail is regressed; pass-to-pass is unchanged_pass; and fail-to-fail is unchanged_fail. Preserve both failure reasons because an unchanged failure can still change type or severity.

Derive new critical failures from the predeclared case risk plus a regressed transition. This avoids treating every failure equally while keeping the rule transparent. An evaluator crash, unreadable fixture, or grader exception belongs to evaluation_error and invalidates the run rather than becoming a candidate result.

A regression report should link the case, slice, baseline output, candidate output, grader version, reason, and review state. That record gives the engineer a concrete debugging target and gives the release owner an auditable decision basis.

The educational policy releases Candidate B only when it has zero new critical regressions, the policy-sensitive slice has no failures, and there is no schema failure. These are not universal production thresholds. They encode the consequences represented by this particular fictional fixture.

Candidate B has one new critical regression and the policy-sensitive slice is 1/2, so candidate_b_release=BLOCKED. Its higher 11/12 total cannot waive either rule. The next action is to diagnose case-09, change the candidate, and rerun the same frozen gate—not to lower the expected output.

Real policies may also return REVIEW when evidence is incomplete, graders disagree, or a new failure class appears. Separate the policy engine from the graders so teams can see whether a result changed because application behavior, measurement, or organizational risk tolerance changed.

The runner distinguishes schema_failure, invalid_output, invalid_evidence_reference, missing_required_fact, unsupported_claim, abstention_failure, review_routing_failure, and critical_policy_failure. The first matching deterministic rule produces one primary reason in this compact example.

Production evaluators may preserve multiple findings per case, severity, evaluator confidence, and supporting evidence. Add grader_disagreement when independent evaluators conflict and evaluation_error when the measurement system itself fails. Never silently translate evaluator failure into application failure.

Taxonomies need maintenance. When reviewers repeatedly use an other bucket, decide whether a stable new failure class would improve diagnosis and trend analysis. Version that change because it alters reported distributions.

Human review is appropriate for conflicting evidence, ambiguous rubrics, high-risk policy cases, evaluator disagreement, new failure classes, and release-blocking regressions requiring adjudication. The reviewer should receive the case, permitted evidence, both outputs, grader findings, and the decision being requested.

Use sampling for routine monitoring and escalation for consequential exceptions. Record the review decision, reviewer role, rubric version, reason, and whether it updates a label, evaluator, candidate, or release policy. Protect sensitive review data and do not request or store hidden chain-of-thought.

The fixture checks whether the application routes certain cases to review; it does not simulate a human judgment. A review_required boolean is a workflow contract, not proof that the final answer is safe or correct.

Offline evaluation runs before release against controlled cases. It supports reproduction, candidate comparison, deterministic grading, slice analysis, and release gating. It is strongest for known requirements and failures represented in the dataset.

Online evaluation observes real usage: feedback, operational failures, latency, cost, tool and retrieval errors, incidents, and distribution drift. It can discover cases the offline set missed, but live monitoring makes controlled attribution harder and may require privacy-preserving sampling.

Passing offline evals does not prove production reliability. Online monitoring does not replace a frozen regression suite. Feed validated production failures back into governed datasets without exposing private traffic or turning the release set into a prompt-tuning worksheet.

A conceptual CI path is: candidate change → build → frozen evaluation set → deterministic checks → optional semantic grader → slice and regression analysis → release policy → PASS, BLOCK, or REVIEW. The article adds no repository workflow; the embedded runner demonstrates the core comparison locally.

Store the run manifest and machine-readable per-case result as build artifacts. Fail closed when the fixture is unreadable, a required evaluator crashes, versions are missing, or results are incomplete. Keep secrets and real customer data out of generic CI logs.

Separate fast pull-request smoke cases from larger release suites if cost or duration requires it, but label the evidence honestly. A passing smoke set is not equivalent to a full release gate.

Leakage occurs when expected answers reach the generation path, engineers inspect every holdout case while tuning, or prompts are repeatedly optimized against one fixed release set. The score can rise while generalization to unseen failures remains unknown.

Use a development set for iteration and a governed holdout or release set for final decisions. Limit access where appropriate, rotate or replenish cases with reviewed incidents, and track case provenance. Keep test inputs representative of intended deployment conditions and document what is not covered.

This public 12-case fixture is fully visible and therefore unsuitable as a secret holdout. It teaches mechanics only. A real program needs contamination controls, dataset review, label adjudication, privacy checks, and planned refreshes.

At minimum record run ID, dataset/candidate/grader/policy versions, case ID, slice, output reference, evaluator result, failure reason, review decision, and release decision. For real systems also measure model calls, latency, token use, tool failures, retrieval failures, and evaluator errors—but only from actual telemetry.

OpenTelemetry distinguishes traces, metrics, and logs: traces follow a request path, metrics capture measurements, and logs record events. Evaluation systems can correlate those signals around a run and case ID. The specific schema remains an application decision.

Do not log private prompts, sensitive evidence, secrets, or hidden chain-of-thought by default. Use explicit redaction, access, retention, and output-reference policies. Auditability means preserving decision-relevant evidence, not maximizing data collection.

The artifacts can be extracted and run with Python 3.12 standard library only. Their deterministic byte output proves that the authored cases, checks, comparison logic, slice aggregation, and policy produce the recorded result in the tested environment.

That is implementation evidence, not model evidence. The candidate responses were hand-authored and recorded; no provider generated or judged them. The dataset is too small and artificial to support statistical, production, commercial, or safety claims.

Production hardening starts with ownership: who curates data, adjudicates labels, versions evaluators, approves policy, investigates regressions, and decides release exceptions. Add immutable run artifacts, authenticated review workflows, evaluator health checks, privacy controls, access and retention policy, retry and timeout contracts, and incident feedback.

Calibrate semantic graders against human labels, measure disagreement by slice, and rerun calibration after version changes. Expand datasets from validated failure modes while preserving development/holdout separation. Monitor deployment behavior and connect incidents back to governed offline cases.

For the architecture preceding evaluation, compare deterministic, agentic, and hybrid designs. For implementation, build a bounded agent with explicit tools, state, and policy. Use the prompt-engineering guide for prompt-level iteration; this article owns the release-quality system around the whole application change.