Control plane client¶
HTTP client for projects, datasets, experiments, metrics, alerts and live runs. Client groups its methods by resource (client.datasets, client.experiments, ...); each resource is documented below.
Client ¶
Client(base_url: str | None = None, *, timeout: float = 30.0, transport: BaseTransport | None = None, api_key: str | None = None, project_id: str | None = None)
Platform control-plane client (projects, datasets, experiments, …).
api_key is fixed at construction (pass explicitly or via Opengrade.client).
client.projects ¶
client.datasets ¶
create ¶
create(project_id: str | None = None, *, name: str, version: int = 1, description: str | None = None, task_type: str = 'rag_qa') -> DatasetResponse
list ¶
list(project_id: str | None = None, *, task_type: str | None = None, name: str | None = None, limit: int | None = None, offset: int = 0) -> list[DatasetResponse]
add_item ¶
add_item(dataset_id: str, *, input: str | RagChatInput, expected_output: str | None = None, actual_output: str | None = None, context: RagItemContext | None = None, metadata: RagItemMetadata | None = None, source_trace_id: str | None = None, source_span_id: str | None = None) -> DatasetItemResponse
create_with_items ¶
create_with_items(project_id: str | None = None, *, name: str, items: list[dict[str, Any]], version: int = 1, description: str | None = None, task_type: str = 'rag_qa') -> DatasetResponse
ensure_items ¶
Resume uploading until the dataset has len(items) rows; return stored count.
import_items ¶
import_items(dataset_id: str, *, path: str | Path | None = None, file: tuple[str, bytes] | None = None, format: str | None = None, content_type: str | None = None) -> ImportDatasetItemsResponse
client.experiments ¶
create ¶
create(project_id: str | None = None, *, name: str, dataset_id: str, model_config: dict[str, Any] | None = None, version: str | None = None, git: dict[str, Any] | None = None, baseline_experiment_id: str | None = None, app_config_id: str | None = None, app_config_alias: str | None = None, metrics_set_id: str | None = None) -> ExperimentResponse
list ¶
list(project_id: str | None = None, *, name: str | None = None, limit: int | None = None, offset: int = 0) -> list[ExperimentResponse]
set_alias ¶
set_alias(alias: str, experiment_id: str, *, project_id: str | None = None, note: str | None = None, force: bool = False) -> ExperimentAliasResponse
alias_history ¶
alias_history(alias: str, *, project_id: str | None = None, limit: int = 50) -> list[ExperimentAliasEventResponse]
list_outputs ¶
list_outputs(experiment_id: str, *, keys: list[tuple[str, int]] | None = None) -> list[ExperimentItemOutputResponse]
upsert_outputs ¶
Write per-item outputs; keys left out of an item keep their stored value.
submit_client_results ¶
submit_client_results(experiment_id: str, metrics: list[dict[str, Any]]) -> list[EvaluationRunResponse]
Store scores computed client-side: one evaluation run per metric.
Each metric is {"name", "results": [{"dataset_item_id", "score", "label",
"explanation", "metadata"}], "threshold"?, "config"?}. Bump config when
the metric code changes so compare flags it instead of reporting a delta.
log_metrics ¶
Upsert experiment-level scalars (last value per key wins); returns all of them.
update_params ¶
Merge params into model_config.
The server answers 409 once an evaluation has started or the experiment has evaluation runs, or when it was created from an app config.
evaluate_pack ¶
evaluate_pack(experiment_id: str, *, metrics_set_id: str | None = None, save_as_default: bool = False, poll_interval: float = 1.0, timeout: float = 600.0) -> EvaluateResponse
Start pack scoring (202) and poll until the experiment finishes.
summary ¶
summary(experiment_id: str, *, run_ids: list[str] | None = None, evaluator_ids: list[str] | None = None, group_by: str | None = None) -> ExperimentSummaryResponse
Aggregates per evaluator; group_by (item metadata key) adds per-group rows.
compare ¶
compare(experiment_id: str, baseline_id: str, *, evaluator_ids: list[str] | None = None, candidate_run_ids: list[str] | None = None, baseline_run_ids: list[str] | None = None, group_by: str | None = None) -> ExperimentCompareResponse
Candidate vs baseline; group_by (item metadata key) adds per-group deltas.
client.metrics_packs ¶
client.metrics_sets ¶
client.evaluators ¶
Project evaluators (server metrics). Custom LLM judges: config.kind="custom_judge".
Prefer opengrade.LlmJudge for judges: it creates/reuses the evaluator for you.
create ¶
create(project_id: str | None = None, *, name: str, type: str, config: dict[str, Any], version: int = 1) -> EvaluatorResponse
preview ¶
preview(project_id: str | None = None, *, config: dict[str, Any], output: Any, input: Any = None, expected_output: Any = None, documents: list[dict[str, Any]] | None = None, metadata: dict[str, Any] | None = None) -> EvaluatorPreviewResponse
Run config on one sample without saving anything (calls the judge model).
client.alerts ¶
Server-side alert rules on production metrics (checked every minute by the server).
create ¶
create(project_id: str | None = None, *, name: str, metric: str, threshold: float, operator: str = 'gt', window_minutes: int = 60, min_samples: int | None = None, score_name: str | None = None, score_source: str | None = None, webhook_url: str | None = None, webhook_format: str = 'json', repeat_minutes: int | None = None, enabled: bool = True, environment: str | None = None, version: str | None = None, condition: str = 'threshold', baseline_minutes: int | None = None) -> AlertRuleResponse
metric: error_rate, latency_p50_ms/p90/p99, traffic, cost_usd, total_tokens,
needs_review, score_mean or score_fail_rate (with score_name).
condition:
"threshold": the metric vsthreshold."change"(drift): relative change vs the previousbaseline_minutes(default: one window).threshold=-0.2, operator="lt"fires on a 20% drop."anomaly": z-score vs the past windows of the previousbaseline_minutes(default: 24 windows).threshold=3, operator="abs"fires on a 3-sigma spike or drop.
operator: "gt", "lt" or "abs" (either direction; change/anomaly).
update ¶
Change the given fields; None clears score_name/score_source/webhook_url/
repeat_minutes.
client.live_runs ¶
submit ¶
submit(project_id: str | None = None, *, question: str, answer: str, documents: list[dict[str, Any]] | None = None, metadata: dict[str, Any] | None = None, external_id: str | None = None, metrics_set_id: str | None = None) -> LiveInteractionResponse
list ¶
list(project_id: str | None = None, *, judge_status: str | None = None, search: str | None = None, failed_only: bool = False, since: str | None = None, until: str | None = None, metadata: list[str] | None = None, limit: int = 50, offset: int = 0) -> ListLiveInteractionsResponse
promote ¶
promote(interaction_id: str, *, dataset_id: str, expected_output: Any, expected_doc_ids: list[str] | None = None) -> DatasetItemResponse
client.trace_scores ¶
log ¶
Upsert scores on traces: {"trace_id", "name", "value"?, "label"?,
"explanation"?, "threshold"?, "source"?, "author"?, "metadata"?} (max 500).
AsyncClient ¶
AsyncClient(base_url: str | None = None, *, timeout: float = 30.0, transport: AsyncBaseTransport | None = None, api_key: str | None = None, project_id: str | None = None)
Async platform control-plane client (projects, live_runs, trace_scores).
api_key defaults to opengrade.set_api_key() / OPENGRADE_API_KEY.
trace_scores
instance-attribute
¶
AppConfigClient ¶
App config registry calls. Obtained via Client(...).app_configs /
og.client.app_configs.
create_app_config ¶
create_app_config(project_id: str, name: str, *, prompt: dict[str, Any] | None = None, model: dict[str, Any] | None = None, retrieval: dict[str, Any] | None = None, description: str | None = None) -> AppConfigResponse
list_app_configs ¶
list_app_configs(project_id: str, *, name: str | None = None, latest: bool = False) -> list[AppConfigResponse]