Metrics and judges¶
Custom metrics computed in your process, and LLM judges run by the server. Built-in server-side evaluators are described in Metrics and judges.
metric ¶
metric(fn: Callable[..., Any]) -> LocalMetric
metric(*, name: str | None = None, threshold: float | None = None, version: int | str | None = None, config: Mapping[str, Any] | None = None, requires: Iterable[str] = ()) -> Callable[[Callable[..., Any]], LocalMetric]
metric(fn: Callable[..., Any] | None = None, /, *, name: str | None = None, threshold: float | None = None, version: int | str | None = None, config: Mapping[str, Any] | None = None, requires: Iterable[str] = ()) -> LocalMetric | Callable[[Callable[..., Any]], LocalMetric]
Turn a function into a LocalMetric.
@opengrade.metric or @opengrade.metric(threshold=0.5, version=2, requires=("documents",)).
version is stored as config["version"].
LocalMetric
dataclass
¶
LocalMetric(fn: Callable[..., Any], name: str | None = None, threshold: float | None = None, config: Mapping[str, Any] = dict(), requires: tuple[str, ...] = ())
A Python metric scored in-process, in experiments and in production.
fn(sample) gets a Sample; fn(item, output) (two required positional
parameters) gets the dataset item and the output. fn may be async.
threshold sets PASS/FAIL (default: 0 fails, anything else passes). Bump
config (or version in @opengrade.metric) when the code changes: compare
then reports a config mismatch instead of a misleading delta. requires lists
Sample fields that must be non-empty; when one is missing the result is SKIPPED.
Sample
dataclass
¶
Sample(input: Any = None, output: Any = None, expected: Any = None, documents: list[dict[str, Any]] = list(), context: dict[str, Any] = dict(), metadata: dict[str, Any] = dict(), trace_id: str | None = None, item_id: str | None = None)
What a metric sees: one experiment item or one production call.
In an experiment expected comes from the dataset item; in production it is
None. documents are the retrieved documents (set_retrieval_documents).
Score
dataclass
¶
Score(value: float | None = None, label: str | None = None, explanation: str | None = None, metadata: dict[str, Any] = dict())
A metric result. label overrides the threshold rule only to fail/skip/error.
metadata
class-attribute
instance-attribute
¶
LlmJudge
dataclass
¶
LlmJudge(name: str, criteria: str, scale: Literal['1-5', 'binary'] = '1-5', levels: Mapping[int, str] | None = None, fields: Sequence[str] = ('input', 'output'), examples: Sequence[Mapping[str, Any]] = tuple(), threshold: float | None = 0.5, live: bool = False, model: str | None = None, temperature: float | None = None)
A judge described in plain language.
criteria: what a good answer looks like (the judge's instructions).scale:"1-5"(score 0, 0.25, ... 1) or"binary"(pass = 1, fail = 0).levels: optional descriptions per level for"1-5"({5: "...", 1: "..."}).fields: what the judge sees, frominput,output,expected,documents,metadata.outputis required. A judge that readsexpectedonly runs on test sets.examples: up to 5 graded examples ({"output": ..., "level": 2}or{"output": ..., "verdict": "fail"}, optionalinput/expected/reasoning).threshold: pass mark (0..1). Default 0.5.live: also score production traffic.model/temperature: judge model override (default: serverLLM_MODEL).
examples
class-attribute
instance-attribute
¶
register ¶
register(client: Client | None = None, project_id: str | None = None) -> EvaluatorResponse
Create the evaluator on the server, or reuse the one with the same config.
A changed config creates a new version (scores from different criteria are not
compared as if they were the same metric). The latest version of a live judge
is the one applied to production traffic.
preview ¶
preview(*, output: Any, input: Any = None, expected_output: Any = None, documents: list[dict[str, Any]] | None = None, metadata: dict[str, Any] | None = None, client: Client | None = None, project_id: str | None = None) -> EvaluatorPreviewResponse
Grade one sample now (nothing is stored). Returns score, explanation, prompt.