SAVED EVALUATION RUNS
LLM Regression Comparison
Compare a baseline and candidate evaluation run on stable case IDs. See added, removed and matched cases, pass/fail/skipped/error/unknown transitions, schema-validity checks, regression rate and pass-rate delta. No model is called; only deterministic outcomes are compared.
BASELINE VS CANDIDATE
Paste two evaluation-run JSON documents
Each document is limited to 2 MB and must contain a cases array with stable id values. External references are not fetched. Uploads stay in this browser.
Status values are normalised to pass, fail, skipped, error or unknown. schemaValid is optional and treated as true/false/null.
COMPARISON RESULTS
Run comparison
Matched cases
Cases with the same ID in both runs. Status transitions show whether a case improved, regressed, stayed the same, or changed to skipped/error/unknown. Deterministic assertion changes are listed separately.
| Case ID | Baseline | Candidate | Schema baseline | Schema candidate | Case change | Assertion changes |
|---|
Added cases
Case IDs present in the candidate run but not in the baseline.
| Case ID | Status | Schema valid |
|---|
Removed cases
Case IDs present in the baseline run but not in the candidate.
| Case ID | Status | Schema valid |
|---|
Import JSON contract
Each run must be a single JSON object. The required field is a cases array. Every case must have a stable string id. Its status is normalised to pass, fail, skipped, error or unknown. An optional assertions array can record deterministic checks; each item needs a stable string id and a status using the same values. Optional schemaValid must be true or false (omitted/null means unknown).
Pass rate is pass-status cases divided by all supplied cases; skipped, error and unknown cases remain in the denominator. New regression rate is matched baseline-pass cases that became candidate-fail divided by all matched baseline-pass cases; a zero denominator is N/A. Added and removed cases are reported separately. The comparability gate requires matching rulesVersion, datasetId and schemaVersion; mismatches are incompatible and missing metadata qualifies the comparison for review.
{
"runId": "baseline-001",
"generatedAt": "2026-10-30T10:00:00Z",
"rulesVersion": "1.2.0",
"datasetId": "qa-v3",
"schemaVersion": "answer-v2",
"cases": [
{ "id": "case-1", "status": "pass", "schemaValid": true, "assertions": [
{ "id": "has-answer", "status": "pass" },
{ "id": "within-range", "status": "pass" }
] },
{ "id": "case-2", "status": "fail", "schemaValid": true, "assertions": [
{ "id": "has-answer", "status": "pass" },
{ "id": "within-range", "status": "fail" }
] },
{ "id": "case-3", "status": "skipped" }
]
}What the comparison reports
- Added/removed cases are grouped separately so you can see coverage changes before interpreting pass-rate deltas.
- Regression rate counts matched baseline-pass cases that became candidate-fail, divided by matched baseline-pass cases. Removed cases do not count as candidate failures; they are shown separately. If the denominator is zero, the rate is N/A.
- Pass-rate delta is the candidate pass rate minus the baseline pass rate, in percentage points.
- Comparability gate checks
rulesVersion,datasetIdandschemaVersion. Any mismatch is incompatible; missing metadata qualifies the comparison for review instead of being treated as a confirmed match. - Assertion outcomes are matched by stable assertion IDs and report pass-to-fail regressions, fail-to-pass improvements, other status changes, and added or removed assertions separately from the case status.
This tool does not call any model, grade semantic quality, or infer whether a changed outcome is acceptable. It only reports deterministic status and schema-validity differences.