Evaluation Records¶

Reference

Surface: Captured evaluation records and evidence bindings.

Stability: Public core contract for captured evaluation.

Use this page when: Implementing or reviewing captured evidence.

Captured comparisons use neutral evaluation record contracts. A record keeps its stable ID, input, expected value, output, score provenance, metadata, and error state. Baseline and subject records must use the same complete schedule.

The public request contract is invarlock/evaluation-request-v2. Deterministic and recorded-score captured comparisons use directory pack invarlock/evidence-pack-v2 and an external invarlock/evidence-verification-receipt-v3. The pack has one manifest signature, not a second signed record envelope. verify checks recipient-owned policy, complete-run and normalized-request digests, and signer identity. report is a deterministic presentation step and does not confer verification authority.

The captured comparison binds bindings.baseline, bindings.subject, and bindings.policy. Metric results use baseline_mean and subject_mean; absolute policy bounds use subject_minimum and subject_maximum. The captured SDK helper is compare_runs(baseline=..., subject=..., policy=...), with no alternate role keywords or wire-field aliases. These captured fields do not rename native historical reports or third-party exports. Native pack v1 and receipt v1/v2 contracts are unchanged. Selecting comparison.metric: judge instead publishes judge evidence with its own measurement and recipient trust contracts; see judge measurements.

For the exact JSON schemas, see the files under contracts/ and their synced copies under src/invarlock/_data/contracts/.

Record contracts¶

Schema	Wire identifier	Meaning
`evaluator_export.schema.json`	`invarlock/evaluator-export-v1`	Source-specific native payload with explicit evaluator version and run ID
`evaluation_run.schema.json`	`invarlock/evaluation-run-v1`	Complete canonical run with records and score provenance
`evaluation_case_set.schema.json`	`invarlock/evaluation-case-set-v1`	Independently reviewed IDs, inputs, expected values and metadata
`comparison_policy.schema.json`	`invarlock/comparison-policy-v1`	Metric, slice, sample, interval and subject-bound requirements
`multi_metric_comparison.schema.json`	`invarlock/multi-metric-comparison-v1`	Deterministically replayed paired results and three-way decision
`normalized_captured_request.schema.json`	`invarlock/evaluation-request-v2`	Path-free portable request identity stored inside the captured pack

Runs require format, source (name, version), run_id, artifact_digest, source_digest, score_provenance, and records. Every record has id, input, expected, output, scores, metadata, error, and context. Null output or an explicit error is retained, not silently dropped. Likelihood scoring can use a null output when valid reference-continuation facts are present. Matching IDs must bind matching input, reference and metadata. Missing required scorer facts produce insufficient_evidence, rather than a pass computed over survivors. Duplicate IDs or changed paired facts are integration errors.

artifact_digest attributes the run to a supplied artifact identity; it does not prove inference. source_digest binds raw imported bytes when applicable and is part of the complete run. Canonical run_digest covers the whole normalized run, including provenance and all records, not just its model or schedule. Do not substitute a raw export checksum for a complete-run pin.

load_run supports invarlock, jsonl, evaluator-native-json, evaluator-json, inspect-json, lm-eval-samples, promptfoo-jsonl, and langfuse-json. These adapters normalize recorded outputs, never register a runtime provider or import execution authority. Original upstream bytes and their source_digest remain unchanged. A canonical invarlock run supplies its own metadata; external adapters require explicit source, run ID and artifact or hosted service identity, with approved score provenance when recorded metrics are used. External captured request sources accept artifact_digest: null with the complete service_identity descriptor. Canonical adapter: invarlock runs carry their own descriptor and reject source identity overrides. The external parser also binds the original file bytes through source_digest.

Dedicated evaluator exports¶

For an existing pipeline, save its native cases or results as JSON and use adapter: evaluator-native-json in the captured request. Dedicated mappings cover all 19 maintained ecosystems. Your pipeline keeps its current evaluator environment; evaluation, verification and reporting run in a separate core InvarLock environment without the upstream SDK. No common export envelope or SDK-side InvarLock installation is required; the file follows its evaluator's documented JSON profile.

For example, use Ragas's own explicit serializer after retaining your SingleTurnSample objects and their planned IDs:

import json

rows = [{"id": case_id, "sample": sample.model_dump(mode="json")}
        for case_id, sample in captured_samples]
if [row["id"] for row in rows] != planned_ids:
    raise ValueError("capture differs from the complete planned schedule")
with open("subject-native.json", "x", encoding="utf-8") as stream:
    json.dump(rows, stream, ensure_ascii=False, allow_nan=False)

captured_samples and the independently frozen planned_ids belong to your workflow. Export the baseline in the same way, preserving failed cases. Declare the raw file, actual SDK version, run ID and evaluated model identity in each captured request source:

path: subject-native.json
adapter: evaluator-native-json
source: {name: ragas, version: "0.4.3"}
run_id: subject-campaign
artifact_digest: "<actual model artifact digest>"

Use the version actually captured. For hosted models, declare artifact_digest: null and the complete service descriptor. Pin an independently reviewed expected_case_set_digest in policy to reject missing or changed scheduled cases, and an approved expected_run_digest on each source when required. Raw-file digests bind the imported bytes; the source name and version remain explicit attribution rather than proof of execution.

When the evaluator SDK and InvarLock dependency requirements are compatible, the shared Python exporter also accepts native SDK objects directly:

from importlib.metadata import version
from invarlock.engine import export_evaluator_result, evaluator_input_capabilities

run = export_evaluator_result(
    "ragas",
    [{"id": case_id, "sample": sample} for case_id, sample in captured_samples],
    "subject-export.json",
    expected_ids=planned_ids,
    source_version=version("ragas"),
    run_id="subject-campaign",
    artifact_digest=model_artifact_digest,
)
print(evaluator_input_capabilities(run))

This convenience API checks complete case membership before writing, refuses an existing destination, and returns a validated canonical run. Its output is an invarlock/evaluator-export-v1 envelope consumed with adapter: evaluator-json, using the same source/run/model declarations. For hosted models pass artifact_digest=None, service_identity=identity. Both adapters use the same source-specific mapping and downstream scorers.

Keep pinned MLflow 3.14.0 in a separate environment: it requires cryptography<49, while core InvarLock requires cryptography>=50. Export the original prediction table as JSON and use evaluator-native-json; no core dependency change is needed. SDK serializer smoke tests run from source with minimal test dependencies and do not establish SDK/core co-installation.

The native shape recipes cover Harness, Inspect, Promptfoo, DeepEval, Ragas, LightEval, Hugging Face Evaluate, Pydantic Evals, AutoEvals, OpenEvals, MLflow, Garak, OpenAI Evals, Phoenix Evals, Langfuse, Opik, Azure AI Evaluation, Evidently and TruLens. With compatible co-installation, Inspect EvalLog and Langfuse ExperimentResult objects can be passed directly to the Python exporter. The raw-file route uses Inspect's JSON log or bare public Langfuse ExperimentResult fields, including item_results; it does not require a Langfuse-specific InvarLock envelope. Original outputs can be captured before upstream grading. Scalar SDK wrappers accept an optional metric_result; no extra evaluator call is required to use InvarLock's scorers. Reference fields may be omitted for reference-free judge tasks where the native profile permits them; exact match and NLL still require a reference. Structured outputs remain JSON for compatible policies; text-only scorers do not stringify them. Aggregate-only reports still need the original per-case prediction table or attempts.

Per-case string metadata supplies policy slices. Optional metadata.invarlock_scores selects named finite numeric observations; native per-case numeric metrics are also retained where the profile defines them. Recorded-score policies require explicit score_provenance at export/import and matching accepted_provenance in policy. Numeric metrics do not become likelihoods or judge calls. Preserve actual continuation facts under metadata.invarlock_likelihood for NLL, and supply the separate complete judge measurement contract for judging. See scorer selection and policy requirements.

Langfuse experiments¶

langfuse-json imports the explicit invarlock/langfuse-export-v1 capture produced from a Langfuse Python experiment. It does not parse arbitrary dashboard exports. The maintained exporter uses langfuse==4.14.1; the installed parser, verification and reporting need no Langfuse SDK, account or network access.

Preserve a stable item ID and the complete independently selected case list. The exporter rejects missing or duplicate results instead of accepting only successful tasks. Structured inputs require an explicit text projection for judge scoring. Exact match needs original answers and references; NLL requires explicit continuation measurements; judging requires a recipe and complete judge measurements. Numeric Langfuse scores alone do not establish those facts.

Follow the Langfuse handoff example for capture, source identities, failed records and scorer-specific requirements.

Hosted service identity¶

A canonical run may include service_identity only with artifact_digest: null. Local artifact runs retain their existing digest contract and need no new field. The service descriptor is closed and requires every field below; unavailable observed model or exposed revision values are explicitly null.

Field	Accepted value and meaning
`kind`	Exactly `hosted_service`
`provider`, `service`, `deployment`, `requested_model`	Nonempty strings, at most 512 characters each, describing the requested endpoint
`observed_model`, `exposed_revision`	Nonempty strings of at most 512 characters, or `null`; source-reported observations
`configuration`	JSON object with at most 100 properties containing relevant non-secret settings
`configuration_digest`	`invarlock.engine.digest(configuration)`
`harness`	Closed object with nonempty `name` and `version` (at most 512 characters each), plus SHA-256 `source_digest`
`observation_window`	Closed object with `started_at` and `ended_at`, ordered UTC timestamps using `YYYY-MM-DDTHH:MM:SS[.microseconds]Z`

The complete descriptor is limited to 1 MiB. Unknown fields, an inconsistent configuration digest, invalid timestamps or a reversed window are rejected. Identity strings reject ASCII control characters and all-whitespace values. Fractional seconds, when present, contain one through six digits; the start may equal the end. invarlock.engine.make_run and capture_evaluator_run accept artifact_digest=None, service_identity=identity. The configuration digest is computed over the configuration object; the service identity digest is computed over the complete descriptor. run_digest additionally binds all records and run provenance. These are different identities and cannot substitute for one another. The public invarlock.engine.validate_service_identity(identity) validates the descriptor. invarlock.engine.evaluated_subject_digest(run) returns the hosted descriptor digest or, for a local run, its explicit artifact digest; it does not fall back from a malformed hosted identity. Neither helper authorizes the subject or replaces a complete-run pin.

For example, given a reviewed descriptor and explicitly mapped records:

from invarlock.engine import capture_evaluator_run, digest

identity["configuration_digest"] = digest(identity["configuration"])
run = capture_evaluator_run(
    rows,
    source={"name": "service-capture", "version": "1"},
    run_id="subject-campaign",
    artifact_digest=None,
    service_identity=identity,
)

The variables identity and rows must contain the complete descriptor and actual captured facts. Canonical source bytes and complete-run anchors authenticate what was supplied; a service label, revision string or digest of the descriptor does not identify immutable model weights or attest execution. A hosted run uses the captured request and verification path. It does not register a native runtime provider or require provider access during offline replay. See the requalification guide for campaign preparation and interpretation.

Capture and explicit input projections¶

invarlock.engine.capture_evaluator_run accepts explicitly mapped per-case rows from any evaluator, with its actual source name/version, run ID and supplied artifact digest, or artifact_digest=None with a hosted service_identity. evaluator_input_capabilities reports usable counts and unavailable IDs for exact_match, normalized_nll_per_utf8_byte and judge. Availability describes input facts; it grants neither qualification nor proof of an upstream scorer execution.

For a structured input, input_projection may select an existing string with {"kind": "json-pointer", "pointer": "/input/question"}. Pointers must be rooted at /input or /context and select a string. The capture retains the original input and context plus the exact projection binding. Paired runs must agree on the original input and projection configuration as well as the selected text. Objects and arrays are never implicitly converted to text. External adapter sources may declare the same projection in a captured request. Canonical adapter: invarlock runs already contain their bindings and reject source overrides, including a second projection.

Set comparison.metric to exact_match, normalized_nll_per_utf8_byte or judge to select a scorer explicitly. Exact match and NLL must agree with the single metric in the comparison policy. Omit this selector for an ordinary multi-metric comparison. Judge uses a separate recipe and retained-call path.

Captured reference likelihoods¶

A record may additionally retain likelihood with these closed fields:

Field	Meaning
`basis`	Exactly `reference_continuation`
`logprob_sum`	Finite, zero or negative natural-log probability sum of the reference continuation
`token_count`	Positive integer count of measured reference tokens
`utf8_byte_count`	Positive integer equal to the reference string's UTF-8 byte length
`input_digest`, `reference_digest`	Canonical digests of the original input and reference
`artifact_digest`, `source`	Exact evaluated artifact and evaluator identity from the run; artifact is `null` for hosted runs
`service_identity_digest`	Required for hosted likelihoods: canonical digest of the complete run service descriptor; absent for local artifacts
`configuration_digest`, `tokenizer_digest`	Pins for the likelihood measurement configuration and tokenizer

These facts must originate from an actual likelihood measurement. Generated answer probabilities and summary losses have different semantics. With an input projection, input_digest still identifies the preserved original input.

The NLL policy metric uses direction: lower, unit: nats_per_utf8_byte, aggregation: mean and ratio_max in place of maximum_regression. Its configuration requires configuration_digest, baseline_tokenizer_digest and subject_tokenizer_digest. For hosted NLL, the policy has one shared configuration_digest: both service descriptors and each likelihood row must match it. Different service configurations are not admitted by this profile; separate model, observation-window and tokenizer identities may still differ. The score per record is -logprob_sum / utf8_byte_count; the comparison is the ratio of subject and baseline means, with the same paired resampling method as native NLL. Interval width is measured in ratio units. Reference and configuration bindings and the declared tokenizer identities are checked before scoring. Replay does not execute the model or tokenizer, so their measurement claims remain source assertions. The retained Mistral 7B sentinel contains fresh generation and reference-likelihood measurements across all 19 maintained evaluator profiles, both import routes and two model artifacts. The Harness likelihood reference is a smaller six-pair, same-model CPU control. Each reference qualifies its recorded profile; synthetic contract tests and exact-match results alone do not establish likelihood measurement support.

Judge selection uses the separate judge recipe and retained measurement contract through the same captured request. It does not reinterpret a recorded scalar score as a native judge trial. prompt.reference_mode: per_case explicitly adds the frozen string reference to each judge request as a separate field. Omitted or none preserves the reference-free behavior. The reference never becomes part of the evaluated model input. See judge measurements.

Policy and identity¶

Policies contain 1..16 metrics and up to 16 metadata slices in addition to the overall scope. Each metric selects a unique name, kind, closed configuration, direction (higher/lower), unit, mean aggregation, minimum count, maximum regression and maximum interval width; NLL replaces maximum regression with ratio_max. Optional absolute limits on the subject mean are subject_minimum and subject_maximum.

Built-in captured kinds are exact_match, normalized_match, numeric_tolerance, json_exact, json_fields, token_f1, and normalized_nll_per_utf8_byte. Exact match and normalized NLL share native scorer semantics while retaining captured-input assurance. normalized_match and token_f1 pin unicode_version; a recipient with an incompatible Unicode environment refuses replay without issuing a receipt. recorded instead selects a score_key and exact accepted_provenance (kind, source, version, unit, rubric_digest). The verifier authenticates attributed values and paired arithmetic; it does not reproduce or authenticate the original scoring process.

An independent planned case set can be pinned in policy as expected_case_set_digest. Use freeze_case_set, case_set_digest, and validate_run_case_set to require the reviewed schedule, not merely agreement between two possibly incomplete captures.

normalize_captured_request strips physical source/output paths and binds each adapter, complete run digest, explicit expected_run_digest when authored, source overrides, and policy digest. captured_request_digest hashes those canonical portable bytes. An explicit run pin is semantic and is not dropped even when equal to the derived run digest. Relocating a request tree changes no portable identity; changing a source, provenance, policy or pin can change it. Use these SDK helpers, not a handwritten YAML/JSON hash recipe.

Pack boundary¶

Captured pack v2 has exactly manifest.json, checksums.sha256, request.json, inputs/policy.json, records/baseline.json, records/subject.json, and reports/evaluation.report.json. Signed packs also have manifest.signature.json; authentication: unsigned_local packs have no signature and a null signing-key fingerprint. The signature uses the existing invarlock/evidence-pack-signature-v1 envelope over the canonical manifest. There is no second signed comparison envelope, embedded verification receipt, or implicit legacy migration.

The normalized request, policy, records and report are canonical JSON with a final LF. The checksum ledger uses fixed payload paths. comparison_id hashes the canonical object containing kind: captured, the normalized request digest, both run digests and the policy digest; it identifies the comparison inputs. Extra files/directories, path traversal, symlinks, duplicate keys, non-finite numbers and byte-limit violations are rejected. Payloads must be regular files; verification checks a stable snapshot and validates their digests. Publication is atomic and no-clobber. Keep receipts and rendered outputs outside the immutable pack. See capacity, reports and receipt scopes, and the signed trust-profile flow.