Experiments¶
Run an app over a test set, score it, compare it with a baseline.
Call these as methods of your Opengrade instance: og.evaluate, og.start_experiment,
og.sweep, og.evaluate_retrieval. They take exactly the arguments documented below (type
checkers see the same signatures), run with og activated and use og.client unless you pass
client=. The module-level functions use the instance activated with og.activate().
evaluate ¶
evaluate(*, dataset: DatasetLike, task: Callable[[DatasetItemResponse], Any], metrics: Sequence[Metric] | None = None, metrics_set_id: str | None = None, name: str | None = None, params: Mapping[str, Any] | None = None, baseline: str | ExperimentResponse | ExperimentResult | None = None, max_concurrency: int = 1, project: str | None = None, version: str | None = None, git: bool | Mapping[str, Any] = True, task_type: str | None = None, app_config_id: str | None = None, app_config_alias: str | None = None, raise_on_error: bool = False, timeout: float = 600.0, repeats: int = 1, group_by: str | None = None, og: Any = None, client: Client | None = None) -> ExperimentResult
Run task on every dataset item, score it and return the result.
task(item) gets a DatasetItemResponse (item.input,
item.expected_output, item.metadata) and returns the output. Calls to
opengrade.set_retrieval_documents inside it feed retrieval metrics. A task that
raises is recorded as an item error (unless raise_on_error).
repeats runs the task that many times per item, to measure non-deterministic
pipelines: scores are averaged per item and flaky items are reported. group_by
(an item metadata field) adds per-group means and baseline deltas to the result.
start_experiment ¶
start_experiment(name: str | None = None, *, dataset: DatasetLike, metrics: Sequence[Metric] | None = None, metrics_set_id: str | None = None, params: Mapping[str, Any] | None = None, baseline: str | ExperimentResponse | ExperimentResult | None = None, project: str | None = None, version: str | None = None, git: bool | Mapping[str, Any] = True, task_type: str | None = None, app_config_id: str | None = None, app_config_alias: str | None = None, repeats: int = 1, group_by: str | None = None, og: Any = None, client: Client | None = None) -> Experiment
Create an experiment and return it for a manual loop.
metrics=None scores with the project's default metrics set; a list scores
with exactly those server kinds (plus local callables). Pass metrics_set_id
to reuse an existing set (e.g. cloned from another experiment) instead of
resolving kinds. baseline is an experiment id, "alias:<name>" (the
experiment a promoted alias points to), a name (latest wins) or a previous
result. With an app config the server snapshots it as the experiment's params
(exp.params), replacing params.
git=True records the commit the code runs at (local checkout, else CI env),
so a compare shows which revisions candidate and baseline came from; it also
becomes version (short SHA) when none is given. Pass a dict to set it
explicitly, or False to skip.
repeats runs every item that many times in run_task (pass repeat= to
exp.run in a manual loop). Scores average per item, so the comparison with the
baseline still pairs items, and items whose verdict changes between repeats are
counted as flaky.
group_by names a dataset-item metadata field ("category", "meta.lang"):
the result then also reports means and baseline deltas per group value.
Experiment
dataclass
¶
Experiment(*, client: Client, experiment: ExperimentResponse, items: list[DatasetItemResponse], metrics_set_id: str | None, score_on_server: bool, local_metrics: list[LocalMetric], baseline_id: str | None, repeats: int = 1, group_by: str | None = None, og: Opengrade | None = None)
A live experiment: iterate items(), wrap each in run(item), then finish.
Usable as a context manager — a clean exit calls finish() and stores the
outcome on .result; an exception skips scoring.
params
property
¶
Stored model config: params, or the app config snapshot when one was bound.
log_params ¶
Merge params into the stored model config and return the merged config.
Allowed until the experiment is scored, and not for experiments created from an
app config: the server answers 409 (OpengradeAPIError) in both cases.
log_metrics ¶
Upsert experiment-level scalars (MLflow-style: one final value per key).
They show up in the summary, are compared with the baseline and can gate a
release (experiment_metrics: {cost_usd: {max: 2}}). Returns all of them.
run_task ¶
run_task(task: Callable[[DatasetItemResponse], Any], *, max_concurrency: int = 1, raise_on_error: bool = False) -> None
Run task(item) on every item inside run(item), logging its return value.
With repeats > 1 every item runs that many times (one trace and output each).
run ¶
run(item: DatasetItemResponse | str, *, repeat: int = 0) -> Iterator[ItemRun]
Run one item: opens a root span bound to (experiment, item) and records output.
Inside the block call run.log(output) or opengrade.set_output(...);
opengrade.set_retrieval_documents is captured too. repeat (0-based) tells
repeated runs of the same item apart; running the same pair again replaces it.
finish ¶
finish(*, timeout: float = 600.0, bind_timeout: float = 30.0, poll_interval: float = 1.0) -> ExperimentResult
Write outputs, score with the metrics set, compare with the baseline.
ExperimentResult
dataclass
¶
ExperimentResult(experiment: ExperimentResponse, url: str, items: list[ItemRun], evaluation: EvaluateResponse | None = None, server_summary: ExperimentSummaryResponse | None = None, comparison: ExperimentCompareResponse | None = None, baseline_id: str | None = None)
Outcome of a finished experiment: server scores, local scores, baseline delta.
server_summary
class-attribute
instance-attribute
¶
by_metric
property
¶
Evaluator name → per-item results from the last evaluate().
by_item
property
¶
Dataset item id → evaluator name → result from the last evaluate().
local_metrics
property
¶
In-process mean per local metric; with repeats each item weighs once.
metrics
property
¶
Mean score per metric as aggregated by the platform.
Local metrics are submitted too, so they appear here under their name; the in-process mean is only a fallback when the submission failed.
experiment_metrics
property
¶
Experiment-level scalars: log_metrics values plus the SDK's opengrade.* ones.
groups
property
¶
Mean score per group value and metric (evaluate(group_by=...)).
flaky
property
¶
Per metric: items whose verdict changed between repeats (empty without repeats).
summary ¶
Metric table with baseline deltas (ANSI colors on a TTY; plain otherwise).
ItemRun
dataclass
¶
ItemRun(item: DatasetItemResponse, repeat: int = 0, output: Any = None, has_output: bool = False, error: str | None = None, trace_id: str | None = None, latency_ms: float | None = None, scores: dict[str, float | None] = dict(), metric_errors: dict[str, str] = dict(), results: dict[str, Score] = dict(), documents: list[dict[str, Any]] | None = None, context: dict[str, Any] = dict(), url: str | None = None, trace_url: str | None = None, _explicit_output: bool = False)
One dataset item executed inside Experiment.run().
scores
class-attribute
instance-attribute
¶
metric_errors
class-attribute
instance-attribute
¶
results
class-attribute
instance-attribute
¶
results: dict[str, Score] = field(default_factory=dict)
log ¶
log(output: Any = _MISSING, *, documents: Sequence[Mapping[str, Any]] | None = None, context: Mapping[str, Any] | None = None) -> None
Record the item's output (and optionally retrieved documents / extra context).
An explicit log(output) wins over values captured from opengrade.set_output.
log_params ¶
Experiment.log_params on the active experiment.
log_metrics ¶
Experiment.log_metrics on the active experiment.
DatasetColumnError ¶
DatasetColumnError(*, missing: Sequence[str], found: Sequence[str], row_index: int | None = None, path: str | Path | None = None)
Bases: ValueError
Raised when a dataset row or CSV header is missing required columns.
sweep ¶
sweep(*, dataset: DatasetLike, task: Callable[[DatasetItemResponse, dict[str, Any]], Any] | None = None, retriever: Callable[[DatasetItemResponse, dict[str, Any]], Any] | None = None, grid: Mapping[str, Sequence[Any]] | None = None, variants: Sequence[Mapping[str, Any]] | None = None, metrics: Sequence[Metric] | None = None, name: str | None = None, params: Mapping[str, Any] | None = None, baseline: str | ExperimentResponse | ExperimentResult | None = None, k: int = 5, max_concurrency: int = 1, repeats: int = 1, group_by: str | None = None, project: str | None = None, version: str | None = None, git: bool | Mapping[str, Any] = True, task_type: str | None = None, raise_on_error: bool = False, timeout: float = 600.0, og: Any = None, client: Client | None = None) -> SweepResult
Run one experiment per configuration on the same test set and rank them.
grid={"model": ["gemma", "llama"], "top_k": [3, 5]} runs every combination;
variants=[{...}, {...}] runs an explicit list. task(item, params) (full pipeline,
scored like evaluate) or retriever(item, params) (retrieval only, scored like
evaluate_retrieval with cutoff k) receives each configuration's params.
Every experiment stores its configuration (plus params shared by all, and
params["sweep"] = name) as its params, so the UI's Compare view can filter the sweep,
plot any two metrics against each other and show what changed between two runs.
Configurations run one after another (max_concurrency applies within each run).
A configuration that fails to run is reported in result.errors (or raised with
raise_on_error=True); the others still run.
SweepResult
dataclass
¶
SweepResult(name: str, runs: list[SweepRun], url: str, metric_names: list[str] = list())
All runs of a sweep. leaderboard() ranks them, url opens them side by side.
metric_names
class-attribute
instance-attribute
¶
run_metric_names
property
¶
Experiment-level metrics on any run: SDK ones first, then log_metrics keys.
leaderboard ¶
leaderboard(metric: str | None = None, *, lower_is_better: bool | None = None) -> list[SweepRun]
Runs sorted best first on metric (default: the first scored metric).
Direction follows the metric name (latency/cost/error → lower is better) unless
lower_is_better is given. Runs without a value come last.
best ¶
best(metric: str | None = None, *, lower_is_better: bool | None = None) -> SweepRun | None
The top run of leaderboard(metric), or None when nothing was scored.
to_records ¶
One flat dict per run (params, metric means, run metrics, ids): ready for pandas.
summary ¶
Plain-text leaderboard: one row per configuration, best first.
SweepRun
dataclass
¶
SweepRun(params: dict[str, Any], result: ExperimentResult | None = None, error: str | None = None)
One configuration of a sweep: its params and the experiment it produced.
value ¶
Mean of a scored metric, or an experiment metric (opengrade.latency_p95_ms).
evaluate_retrieval ¶
evaluate_retrieval(*, dataset: DatasetLike, retriever: Callable[..., Any], k: int = 5, metrics: Sequence[Metric] | None = None, name: str | None = None, params: Mapping[str, Any] | None = None, baseline: str | ExperimentResponse | ExperimentResult | None = None, max_concurrency: int = 1, project: str | None = None, version: str | None = None, git: bool | Mapping[str, Any] = True, doc_id: Callable[[Any], Any] | None = None, raise_on_error: bool = False, timeout: float = 600.0, repeats: int = 1, group_by: str | None = None, client: Client | None = None) -> ExperimentResult
Score only the retrieval step of a RAG pipeline: no answer is generated.
retriever(item) gets the dataset item (or the question text when its first
parameter is named query/question/text/q) and returns the ranked
documents: dicts with id and text, strings, LangChain Document or
LlamaIndex NodeWithScore. Gold ids come from the item's expected_doc_ids.
Default metrics are hit@k, recall@k and MRR at cutoff k (no LLM, cheap enough to
compare chunk sizes, embedding models or rerankers on every change). Add
"context_precision" / "context_recall" to judge relevance and coverage with the
LLM judge (needs document text and expected_output). Metrics that grade an answer
are rejected. The experiment is stored with params["eval_mode"] = "retrieval" and
the retrieval latency as opengrade.latency_p50_ms / _p95_ms.
set_retrieval_documents ¶
Record retrieved documents on the current span as OpenInference flattened attrs.
Each document is a dict or an object with id and optionally text (or
content / page_content), score, title, metadata. Like the rest
of tracing this never raises: a document without an id is skipped with a warning.
set_search_query ¶
Record the standalone / rewritten retrieval query for conversation judges.
Sets opengrade.retrieval.query on the current span and captures it into the
experiment / production context as search_query.
Chat test cases¶
Helpers for rag_chat datasets, where item.input is {"history": [...], "question": "..."}.
Recipe: Evaluate a chat assistant.
chat_input ¶
chat_input(item: object) -> RagChatInput
Return parsed chat input from a dataset item or raw input value.
Accepts item.input shaped as {"history": [...], "question": "..."}
or a plain string question (empty history).
RagChatInput ¶
ChatTurn ¶
bind_evaluation ¶
Bind the current span to an experiment dataset item (and repeat, when > 0).