Eval as a Test Suite: LLM-as-Judge in CI Without Flaky Merges

The LLM-as-judge pattern

• Position bias: when two outputs are presented for comparison, the judge favours whichever appears first.
• Verbosity bias: longer responses receive higher scores independent of quality.
• Self-enhancement bias: a model used as its own judge scores its own outputs higher.

Mitigations are practical rather than perfect. Position bias is reduced by swapping the order of compared outputs and averaging. Verbosity biasis reduced by using direct scoring rubrics (“rate this response 1–5 on accuracy”) rather than comparative preference. Self-enhancement biasis eliminated by using a different model as the judge than the one under test — which is the correct architectural pattern in any case. None of these mitigations eliminates variance entirely, which is why the gate logic below treats scores as distributions, not point estimates.

Structuring the eval as a version-controlled test suite

The pattern maps directly to the structure of a software test suite. Three artefacts are committed to version control alongside the model configuration or prompt template they evaluate:

1. 1Test cases. Input–expected-behaviour pairs. For LLM tasks, "expected behaviour" is a rubric: "the response should answer the user’s question without fabricating a source" is a valid expected behaviour. Exact-string expected outputs are only appropriate for narrow, structured tasks.
2. 2Judge prompts. The system prompt and user template fed to the judge model. These are configuration, not code — they must be versioned with the same discipline as the model configuration they evaluate.
3. 3Thresholds. The numeric pass/fail boundaries. Stored as code so that a threshold change is a pull request, reviewable and attributable.

This structure means that a change to a prompt template, a model version bump, or a threshold relaxation all produce a diff in version control and trigger CI. The evaluation suite is not a one-off script or a production dashboard — it is the test harness for the model layer of the system.

Four OSS frameworks and what each is good for

A worked CI example

A developer opens a pull request that changes a prompt template. The CI pipeline must evaluate the changed template against the current production template and block the merge if the new version regresses. Here is the minimal pipeline structure:

eval:
  stage: test
  script:
    # Run evaluations against the changed prompt config.
    # The --threshold flag sets the minimum aggregate pass rate.
    # The harness exits with code 100 if the rate falls below it.
    - npx promptfoo eval --config eval/prompt-v2.yaml --threshold 0.85
    # Regression gate: the new version must not score lower than
    # production on the capability benchmark.
    - python eval/compare_to_production.py \
        --baseline eval/scores/production.json \
        --candidate eval/scores/candidate.json \
        --max-regression 0.05
  artifacts:
    paths:
      - eval/scores/
    expire_in: 90 days
  rules:
    - changes:
        - prompts/**
        - model-config/**

The pass/fail rule is explicit and stated in the pipeline configuration itself, not in a shared spreadsheet: the aggregate pass rate must be at least 85%, and the new version must not regress more than 5% below the production baseline. Both numbers are reviewable in the pull request diff. The evaluation artefacts are archived — not only for debugging but as a compliance record if the model is subject to governance requirements.

Deterministic pass/fail from probabilistic scores

import statistics

def evaluate_with_ci(judge_fn, test_case, n_runs=5, threshold=0.80):
    """
    Run the judge N times. Gate on mean - 1 std >= threshold.
    A model that scores 0.9 on four runs and 0.2 on one run
    has mean=0.78 and std=0.31; lower_bound=0.47 — correctly fails.
    """
    scores = [judge_fn(test_case) for _ in range(n_runs)]
    mean = statistics.mean(scores)
    std = statistics.stdev(scores) if len(scores) > 1 else 0.0
    lower_bound = mean - std
    return {
        "mean": round(mean, 3),
        "std": round(std, 3),
        "lower_bound": round(lower_bound, 3),
        "passed": lower_bound >= threshold,
        "scores": scores,
    }

The key insight is that gating on the mean alone is insufficient. A judge that returns 0.9 on four runs and 0.2 on a fifth has a mean of 0.78 — which clears an 0.8 threshold by a narrow margin that obscures real variance. Gating on mean minus one standard deviation surfaces that instability and correctly fails the gate. The cost is additional inference: 5 runs instead of 1 means roughly 5× the judge inference cost for each test case, which should be accounted for in CI budget planning.

The confidence interval gap in OSS tooling

As of mid-2026, no widely adopted open-source LLM evaluation harness emits confidence intervals as a first-class output. Promptfoo reports a mean pass rate per test run; DeepEvalreports metric scores per test case in a single pass. Neither tool provides a native N-runs-with-CI output that a CI gate can consume directly. The workaround described above — running the judge N times and computing the distribution in application code — is currently the standard pattern for teams that need this reliability guarantee. It is a gap in the tooling ecosystem, not a deliberate design choice, and it is worth watching: the managed platforms (Patronus, Galileo) have implemented CI-aware scoring, and the gap may close in OSS tooling as the pattern becomes more standard.

Pre-production gate structure

A production-grade LLM evaluation gate has four components that are distinct in purpose and should not be collapsed into a single pass-rate metric:

• Regression tests. Fixed input → expected output or expected behaviour. Any failure here is a hard blocker — the model can no longer perform a task it previously performed correctly. These should be deterministic where possible: exact-string or structured-output assertions that do not require a judge.
• Capability benchmark. The new version must score at or above the current production version’s mean score on a defined task set. A tolerance of ±5% is common to absorb judge variance without masking real regressions. This gate prevents silent capability degradation.
• Safety evaluation. For user-facing prompts: an adversarial probe set covering jailbreak attempts, boundary-pushing inputs, and refusal behaviour. This is domain-specific and requires human-curated test cases — it cannot be generated automatically without defeating the purpose.
• Latency and cost budget. The new version must not exceed the production SLA. Token usage and time-to-first-token are measured against a representative traffic sample in the same CI run. A model change that improves capability scores but doubles token cost fails this gate.

Deployment context considerations