Evaluation Engine¶

Core interface¶

from dataclasses import dataclass
from typing import Protocol, Any


@dataclass(frozen=True)
class EvaluationSample:
    input: Any
    expected_output: Any | None
    actual_output: Any | None
    context: Any | None
    metadata: dict[str, Any]


@dataclass(frozen=True)
class EvaluationResult:
    score: float | None
    label: str | None
    explanation: str | None
    metadata: dict[str, Any]


class Evaluator(Protocol):
    name: str

    async def evaluate(
        self,
        sample: EvaluationSample,
    ) -> EvaluationResult: ...

Built-in evaluators — MVP¶

Deterministic¶

exact_match
contains
regex
json_schema
hit_at_k (config k; uses metadata.expected_doc_ids and context.documents)
recall_at_k (same inputs; |expected ∩ top-k| / |expected|)
mrr (same inputs; 1/rank of first expected id in top-k, else 0)
precision_at_k (same inputs; |expected ∩ top-k| / k; available, not in the default RAG set)
ndcg (same inputs; binary nDCG@k; available, not in the default RAG set)
latency
token_usage
cost
tool_call_success

LLM judge¶

answer_relevance
groundedness
correctness
context_precision
context_recall
citation_accuracy
refusal
pii
toxicity
prompt_injection

LLM judges return structured, validated output and are versioned by prompt_version (see LLM judges).

Naming vs Ragas¶

Ragas	opengrade `kind`
Faithfulness	`groundedness`
Response / answer relevancy	`answer_relevance`
Context precision	`context_precision`
Context recall	`context_recall`
Factual correctness (approx.)	`correctness`

Use opengrade kinds in API/config; there is no Ragas-name alias in the registry.

RAG metrics sets and pack scoring¶

Projects hold versioned metrics sets; the default set seeds RAG kinds (hit_at_k, recall_at_k, mrr, context_precision, context_recall, must_contain, groundedness, answer_relevance, correctness, latency, plus citation_accuracy and refusal seeded disabled with thresholds 0.8 / 1.0 — enable them for assistants that cite sources or must decline unanswerable questions — and the safety checks pii, toxicity, prompt_injection seeded disabled with threshold 1.0). POST /experiments/{experiment_id}/evaluate-pack resolves a set (body override → experiment pin → project default), then runs all enabled entries with linked evaluator_id values against experiment item outputs (including OTLP-bound runs). Optional save_as_default persists the request set on the experiment. Calling ensure on an existing project default backfills any missing seed kinds (including newly added ones) without rewriting existing entry thresholds.

Evaluators are shared per kind across metrics sets, so each entry's config (e.g. k, max_ms, judge model) is overlaid on the evaluator config at scoring time (kind cannot be overridden); run metadata records config_override and the effective config_hash. Live scoring applies the same overlay.

Output resolution: once an experiment has any recorded outputs, every item is read from those outputs only — dataset-level actual_output/context are never used as fallback (they belong to another run). Items without an output in that experiment score FAIL (0.0, metadata.missing_output=true) for every evaluator. Experiments with no outputs at all keep reading inline dataset fields (e.g. datasets built from traces).

Explicit POST .../evaluate with evaluator_ids bypasses set resolution.

Groundedness reads retrieved chunk text from run context.documents (populated from SDK set_retrieval_documents, which emits OpenInference flattened attributes retrieval.documents.N.document.*, or from the same flattened keys / reranker.output_documents.N.document.* written by auto-instrumentation such as LangChain or LlamaIndex: content becomes text, metadata.title/file_name the title, and a missing id falls back to metadata.doc_id/document_id/node_id/id; an aggregated retrieval.documents blob, if present, still wins over flattened keys). When a trace has several retrieval stages (RETRIEVER → RERANKER), context.documents is the full list from the last stage that recorded documents (by end time)—including multiple chunks that share a parent doc id. Retrieval metrics (hit_at_k, recall_at_k, mrr, precision_at_k, ndcg) dedupe those ids by first occurrence when ranking.

PASS/FAIL rule (offline runs and live scoring): an item passes only if the evaluator label is not FAIL and the score meets the entry threshold. The stored label is that effective verdict; when it overrides the evaluator's own label, the original is kept in metadata.judge_label (live: appended to the explanation). LLM judges never emit a label, so for them the verdict comes from the threshold alone, with two exceptions: refusal labels every item PASS/FAIL (its score is 0 or 1), the safety checks pii / toxicity / prompt_injection label PASS (clean, 1.0) / FAIL (flagged, 0.0), and citation_accuracy labels FAIL when the answer cites nothing (and require_citations is on) or when min_coverage is not met. A run is FAILED if any item fails.

LLM judges only receive the fields their rubric needs: answer_relevance gets input + answer, groundedness adds context.documents, correctness adds expected_output — for rag_chat items, answer_relevance, groundedness, and correctness also get conversation history when prior turns are present — context_precision and context_recall get input + expected_output + context.documents (no answer), citation_accuracy gets the cited statements + only the documents they cite, refusal gets input + answer + whichever of expected_output / context.documents decides whether the question is answerable, the safety checks get only their configured fields (prompt_injection never gets the answer). Conversation judges: query_rewrite_faithfulness (history + context.search_query), conversation_consistency (full transcript including the latest assistant answer), resolution (history + latest Q/A + last-turn context.documents; skips without documents; turns_to_resolution only when resolved). Item metadata (e.g. expected_doc_ids) is never sent to a judge. Prompt versions are {kind}.{method}.{sha256[:12]} of the system templates that pair uses.

Live scoring: a failing judge records an ERROR score for its kind without discarding the others (score_warning lists the failed kinds; status is error only if every judge failed).

LLM judges¶

Code: opengrade.evaluation.judges (prompts, parsing, claims, rubric, evaluators, cache, snapshots, warnings). Judges return label=None; explanation is a short human-readable summary and structured details go into metadata.

Methods and config¶

Kind	`method`	Notes
`groundedness`	`claims` (default), `rubric`	claims: extract answer claims, verify each against `context.documents` (2 calls)
`correctness`	`claims` (default), `rubric`	claims: extract gold claims from `expected_output`, check coverage in the answer (2 calls, 4 with `scoring: f1`; one fewer on a gold-claim cache hit)
`answer_relevance`	`rubric` (only value)	one call, anchored 1-5 level
`context_precision`	`claims` (default), `rubric`	claims: one batched relevance verdict per retrieved document vs question + reference; score is RAGAS average precision
`context_recall`	`claims` (default), `rubric`	claims: extract gold claims from `expected_output`, check coverage in `context.documents` (2 calls; one fewer on a gold-claim cache hit)
`citation_accuracy`	`claims` (only value)	citation markers are parsed locally; each cited sentence is verified against the documents it cites (1 call; 0 when nothing valid is cited)
`refusal`	`classify` (default), `phrases`	classify: one call that labels the answer (and reference) refusal/answer and, without gold, whether the documents contain the answer. phrases: deterministic phrase match, no LLM call
`pii`	`classify` (only value)	one call: does the question / answer expose personal data? (categories `email`, `phone`, `id_number`, `address`, `name`, `financial`, `other`)
`toxicity`	`classify` (only value)	one call: hate, insults, threats, sexual content, self-harm (`hate`, `insult`, `threat`, `sexual`, `self_harm`, `other`)
`prompt_injection`	`classify` (only value)	one call: do the question / retrieved documents try to manipulate the assistant? (`instruction_override`, `role_play`, `data_exfil`, `ignore_context`, `other`)

rubric for groundedness/correctness/context_recall is a single holistic call intended for small local models; it is never selected as a silent fallback.

Entry config keys (part of config_hash). Invalid values raise ValueError when the evaluator is built.

key	values	default	applies to
`method`	`claims` \| `rubric`	`claims` (groundedness, correctness, context_precision, context_recall), `rubric` (answer_relevance)	all judges
`scoring`	`recall` \| `f1`	`f1`	correctness with `method=claims`
`max_claims`	integer >= 1	30	`method=claims`; extra claims are dropped and `claims_truncated=true`
`model`	LiteLLM model id	`LLM_MODEL`	all
`temperature`	finite float	`LLM_TEMPERATURE`	all
`pattern`	regex, first group = cited id(s)	`\[([^\[\]\n]{1,200})\](?!\()` (`[pto]`, `[1, 3]`, `[^2]`, not markdown links)	citation_accuracy
`require_citations`	bool	`true`	citation_accuracy: an answer citing nothing FAILs (a refusal is SKIPPED)
`min_coverage`	0-1	none	citation_accuracy: FAIL when fewer sentences carry a citation
`answerable_from`	`auto` \| `label` \| `documents`	`auto`	refusal: where "should it answer?" comes from
`check`	`both` \| `unanswerable` \| `answerable`	`both`	refusal: score only the items that should be declined (missed refusals) or only the answerable ones (over-refusals)
`refusal_phrases`	list of strings	English + Italian defaults	refusal with `method=phrases` (replaces the defaults, whole-word, case-insensitive)
`fields`	non-empty subset of `input`, `output`, `documents`	`["input", "output"]` (pii, toxicity), `["input", "documents"]` (prompt_injection)	safety checks: what the classifier reads; `output` is rejected for prompt_injection

Scoring¶

Groundedness (claims): supported / claims. Each claim is supported, contradicted or not_supported. Zero claims (refusal, "I don't know") is SKIPPED (no_factual_claims). Missing context is SKIPPED; empty context.documents is a minimum-score FAIL offline (RAG assumption). Live scoring treats empty retrieved documents as SKIPPED (no retrieved documents (live)) so non-retrieval turns are not scored as ungrounded.
Correctness (claims): default scoring: f1 extracts gold claims and answer claims; recall = covered / gold claims, precision = supported / answer claims against expected_output, score is F1; any contradicted gold or answer claim gives 0.0. scoring: recall skips answer-claim extraction and does not penalize extra non-contradicting statements. Zero gold claims is SKIPPED (no_gold_claims).
Context precision (claims): for each retrieved document (rank order), verdict relevant / not_relevant vs question + expected_output. Score is average precision (\sum_k (P@k \cdot v_k) / \sum v_k); 0 when no document is relevant. Optional config.k truncates the document list; max_claims is also applied as a safety cap. Missing expected_output or context is SKIPPED; empty context.documents is a minimum-score FAIL.
Context recall (claims): extract gold claims from expected_output (same gold-claim cache keying as correctness, keyed by this judge's prompt_version), then verdict covered / contradicted / missing per claim against context.documents. Score is covered / gold claims (a contradicted claim is not covered but does not zero the whole score). Zero gold claims is SKIPPED (no_gold_claims). Missing expected_output or context is SKIPPED; empty context.documents is a minimum-score FAIL. Not gold-less: offline pack only.
Citation accuracy: the answer is split into sentences; citation markers (pattern) are attached to the sentence they follow (a marker after the full stop still belongs to it), lines after a Sources: / References: / Fonti: header are a reference list and ignored. Each cited id resolves to a retrieved document by id, then title, then 1-based position ([2]). A sentence citing an id that was not retrieved is invalid_citation (no LLM call). The others are verified against only the documents they cite (supported / contradicted / not_supported, even if an uncited document would support them). Score supported / cited sentences; metadata.claims lists every sentence with its verdict, citation_coverage is cited sentences / all sentences. No citation at all: FAIL (or SKIPPED with require_citations=false, or when the answer is a refusal). Cited documents without text: SKIPPED. Gold-less: runs on live traffic.
Safety checks (pii, toxicity, prompt_injection): one structured call over the configured fields (input = last-turn question, output = answer, documents = retrieved documents with text). A field with no text counts as missing; if none of the configured fields has text the result is SKIPPED (e.g. only documents and nothing retrieved, or CONTENT_CAPTURE_ENABLED=false), otherwise the judge runs on what is there (prompt_injection still scores a question-only turn). They do not need an answer. Score 1.0 / PASS when clean, 0.0 / FAIL when flagged; metadata flagged, categories (unknown values become other), evidence (at most 5 snippets of 200 chars), fields_checked, fields_missing, reasoning. Gold-less: they run on live traffic. No regex/detector method and no redaction yet.
Refusal: is the question answerable? auto uses, in order, the dataset label (metadata.answerable, or metadata.expected_refusal / should_refuse inverted; bools or "true"/"false" strings), the reference answer (a reference that declines means "should refuse"), then the retrieved documents (judge decides whether they contain the answer; an empty document list means "should refuse"). Then score 1 for correct_answer / correct_refusal, 0 for over_refusal (declined an answerable question) / missed_refusal (answered one it should have declined: likely hallucination); metadata.outcome, expected_answerable, refused, answerable_source. No source: SKIPPED. On live traffic (no gold) it reads the documents, so enable it there only for retrieval turns. Use group_by / the outcome metadata, or two entries with check: unanswerable and check: answerable, to see hallucination rate and over-refusal rate separately.
Rubric: {reasoning, level} with level 1-5 and score = (level - 1) / 4.
Missing actual_output is SKIPPED for the answer judges (groundedness, correctness, answer_relevance). context_precision and context_recall never read the answer, so they also score retrieval-only runs (og.evaluate_retrieval); they, and correctness, need expected_output.
PASS/FAIL comes only from the entry threshold.

Determinism and versioning¶

LLM_TEMPERATURE (default 0.0) and LLM_SEED (default 42) are sent on every judge call; LiteLlmClient passes drop_params=True so providers without seed ignore it. Entry temperature / model override them per entry.
prompt_version = "{kind}.{method}.{sha256[:12]}" is derived from a canonical JSON hash of the system prompt templates that (kind, method) uses (e.g. groundedness.claims.a1b2c3d4e5f6). Changing any of those templates changes the hash automatically — no manual revision bump. The id is stored in each result's metadata and in run.metadata.prompt_version. Compare flags a prompt_version / judge_model / config_hash mismatch between candidate and baseline as a config mismatch (flagged, not blocked).
For personal audit, the first time a judge scores an item the full prompt bundle is upserted into a local gitignored JSON file (default .opengrade/judge_prompt_snapshots.json, override with OPENGRADE_JUDGE_PROMPT_SNAPSHOTS). Entries are keyed by the full SHA-256. Constructing a judge (e.g. validating project-default metrics) does not write it. The file is not shared and is not required for compare/cache — those use the hash in metadata only.
Result metadata: method, prompt_version, model, claims (per claim text and verdict, plus doc_ids/quote/reasoning where applicable), counts (n_claims, n_supported, n_contradicted; correctness: n_gold, n_covered, n_missing, recall, and precision for f1), claims_truncated, llm_calls, cache_hit. Rubric results store level and reasoning.
Explanations are capped at 1000 characters (claim lists are abbreviated).

Gold-claim cache¶

Gold claims extracted from expected_output are stored in judge_claim_cache, keyed by a SHA-256 of question + expected output + prompt_version + model. A hit skips the extract call, so candidate and baseline experiments are scored against identical gold claims. Entries are written only after the extract output passed validation and produced at least one claim; empty extractions are not cached, and a stored empty list is treated as a miss so a flaky empty extract cannot lock an item into permanent no_gold_claims SKIPPED.

Output validation and errors¶

Every step parses tolerantly (code fences, JSON wrapped in prose) and validates against a schema (enum verdicts, level 1-5, one verdict per claim sent). On failure, a format-only repair call rewrites the invalid draft into schema-shaped JSON (max 2 LLM calls per step; the judge task is not re-asked). Failures after repair produce an ERROR result with metadata.error_type:

judge_output_invalid: the model did not follow the format after format repair; metadata.raw_output holds the first 2000 characters.
llm_not_configured: provider auth failure (bad/missing credentials after a call). Missing model/API key before any call is SKIPPED instead (metadata.skip_reason: llm_not_configured); it does not fail the experiment.
llm_unavailable: timeout, connection, rate-limit or provider 5xx after LiteLLM retries.
context_overflow: context window exceeded. Documents are never truncated silently.
llm_error: any other provider error.

Minimum judge model¶

Dev / smoke / local Ollama: ollama/gemma4:e2b is a workable minimum for claims judges: it discriminates good vs wrong_fact vs hallucination on the HR sanity suite, with format repair covering most schema glitches.
Quality gates / CI / production scoring: prefer a stronger chat model that reliably emits structured JSON, or set method: rubric on groundedness/correctness/context_* for small models. There is no silent fallback from claims to rubric.
Verify a candidate judge with scripts/judge_sanity.py (and optionally scripts/judge_smoke.py) using the same LLM_MODEL / LLM_API_BASE as the API.

Unsuitable-model warning¶

Offline: when judge_output_invalid items exceed 20% of evaluated items (SKIPPED and missing-output items excluded; the other error types do not count as format failures), the run gets metadata.warnings = ["judge_model_unsuitable"] and metadata.warning_detail (model, method, failure_rate, message). The remedy is method: rubric or a stronger judge model.
Live: an interaction whose judge result is judge_output_invalid gets a judge_model_unsuitable score_warning. The list endpoint also returns judge_warnings computed over the last 50 live scores per kind (unfiltered by list query params), using the same >20% rule as offline runs.

Manual smoke test¶

scripts/judge_smoke.py runs the built-in judges on a fixed sample against a real model (not part of CI):

cd backend
LLM_MODEL=ollama/gemma4:e2b LLM_API_BASE=http://localhost:11434 \
  uv run python ../scripts/judge_smoke.py [--method rubric]

A capable model should give groundedness about 0.5, correctness about 0.5 and answer_relevance 1.0.

Judge sanity suite (meta-eval)¶

scripts/judge_sanity.py scores the fixed gold set in scripts/judge_sanity/knowledge.json under synthetic variants (good, wrong_fact, hallucination, bad_retrieval, refusal) and checks aggregates against scripts/judge_sanity/judge_sanity_thresholds.yaml. On ERROR / judge_output_invalid it prints kind, item index, explanation, and truncated raw_output to stderr for diagnosis. In-process judges only — no API server. Not part of default PR CI.

cd backend
LLM_MODEL=ollama/gemma4:e2b LLM_API_BASE=http://localhost:11434 \
  uv run python ../scripts/judge_sanity.py [--variants good,wrong_fact] [--method rubric]

Optional GitHub Actions template: .github/workflows/judge-sanity.yml.example (workflow_dispatch; copy to judge-sanity.yml and set LLM_* secrets).

Evaluator execution¶

Dataset
   │
   ▼
Scheduler (or evaluate-pack / metrics set)
   │
   ├── evaluator A ──► results
   ├── evaluator B ──► results
   └── evaluator C ──► results
                     │
                     ▼
              Aggregation
                     │
                     ▼
                Experiment / Run

Scoring¶

Every evaluator must document: - score range - higher/lower-is-better semantics - threshold compatibility - whether scores are deterministic - provider/model dependency

Reproducibility¶

An evaluation run stores: - evaluator name/version - configuration hash - dataset version - model/provider - prompt/template version - execution timestamp - application version

Failure handling¶

Do not use score=0 for infrastructure failures or non-evaluable inputs.

Use explicit states:

PENDING
RUNNING
PASSED
FAILED
ERROR
SKIPPED

Outcome classes:

ERROR — judge/infra failure (timeout, auth, exception). score is null; run becomes ERROR and the experiment becomes failed.
SKIPPED — metric cannot be evaluated (missing gold / required fields, or judge model/API key not configured). score is null; excluded from mean/pass_rate; does not fail the experiment.
FAIL with minimum score (usually 0.0) — quality miss, including empty/missing retrieval when gold is present. Lowers mean/pass_rate and can fail gates.

Helpers: opengrade.evaluation.outcomes (skip, fail_min, error, pass_).

LLM judge requirements¶

structured JSON output
timeout
retry policy
max concurrency
token/cost accounting
model/provider recorded
optional redaction
judge prompt version recorded

Extension point¶

Evaluator registry (evaluation/registry.py):

register_evaluator("my_business_rule", MyBusinessRuleEvaluator)

This registry is the primary open-source extension surface: contributors add evaluators here, not by forking API or UI code. How to write, register and test one: 16-custom-evaluators.md. App-specific checks that should not live in the server go in the SDK (@opengrade.metric).

Not yet: loading out-of-tree evaluator packages without modifying the core (e.g. Python entry points).