Skip to content

Decision semantics

Assurance note

In plain language: For native exact match and NLL, InvarLock recomputes scores from authenticated paired records, derives the paired interval, and applies the policy to the conservative bound. When the policy qualifies sample size, precision, or exact-match side accuracy, those checks must pass too. A favorable point value cannot override a threshold, count, width, or side- accuracy failure.

Question: How does each scoring contract turn its measurements and uncertainty into a replayable policy decision?

Decision use: Use these definitions to review thresholds, reproduce canonical report arithmetic, and interpret a pass or fail at the boundary.

Evidence: Pinned local source identity, canonical schedule, provider observations, verifier-derived paired scores, exact policy bytes, independent artifact and schedule anchors, and the metric-specific paired interval recorded in the report.

Choose the decision contract

The selected evidence family determines the arithmetic and the meaning of its result labels. Native judge selection uses the bounded judge contract, even though it shares the main commands with native exact match and NLL.

Evidence family Decision basis Meaning when the result does not pass
Native exact match, NLL or deterministic extension Authenticated provider records, the metric-specific paired interval and configured qualification checks fail means the policy was not met; it does not by itself establish degradation
Captured deterministic, likelihood or recorded scores Every declared metric and slice, with conservative comparison bounds and optional subject-mean bounds regression means a bound check failed; missing facts, too few records or excessive width alone yield insufficient_evidence
Bounded judge, including native judge Complete planned trials, declared units, bounded-score intervals and the judge analysis policy regression requires an entirely adverse interval after count and precision checks; a boundary-straddling interval is inconclusive

A combined evidence set requires both component acceptances and preserves their statistical scopes. It does not create a new joint confidence guarantee. The detailed native derivation follows; captured and judge rules appear under Captured comparison and bounded judge decisions.

The native exact-match, normalized-NLL and deterministic extension contract makes one decision over one authenticated, ordered, finite schedule. It reports a point comparison and a verifier-replayed paired interval. Exact match uses a paired Newcombe 95% effect-size interval. Normalized NLL and an authorized scorer extension use deterministic paired resampling over the authenticated schedule.

The notation, preconditions and equations through the deterministic-extension section apply to that native contract, including its 10,000-record limit. Captured comparison capacity and judge plan limits are separate.

New evaluations under that contract emit invarlock/comparison-report-v3. Strict verification continues to replay v2 reports without side-accuracy qualification and v1 reports with their original exact-match interval method, so signed historical evidence keeps its original meaning.

Arithmetic and replay

Native and captured numerical comparisons use Python float arithmetic for means, intervals and policy comparisons. On the supported conventional Python builds this is binary64 arithmetic. The exact operation order, interval method, resampling stream, interpolation and whether threshold equality passes are part of replay; an algebraically equivalent expression need not round identically.

Native NLL and deterministic-extension resampling use their schedule-bound SHA-256 index generation. Captured continuous-score comparisons use a separate SHAKE-256 stream with rejection sampling. Both use math.fsum for relevant sums and linear interpolation between ordered bootstrap draws. These methods are distinct; substituting one stream, summation algorithm or percentile rule would change the numerical contract. Exact match uses the versioned paired Newcombe method described below, not either bootstrap.

For native exact-match and scorer-extension point deltas, the implementation sums per-pair subject - baseline differences with math.fsum, divides by the record count, then multiplies by 100.0. Subtracting the separately computed side means is algebraically equivalent but can round differently. Use the native report builder when reproducing canonical values.

Captured point deltas subtract the separately rounded side means. For large common scores, that subtraction can lose a small paired change: baseline [2**54, 0] and subject [2**54, 1] have an exact mean change of 0.5, while the captured point delta is 0.0. Captured continuous intervals already use paired differences; the point delta does not override their policy checks. Moving the subtraction inside the sum is not universally exact either, so an arithmetic change must preserve the historical replay contract.

The optional contribution recipe can separately reconcile exact retained-score contributions, the omitted remainder and the gap to the original report's total. Its rounding_residual field is that arithmetic gap; reconciliation alone does not prove rounding was its cause. First validate the complete source workflow, metric scope, weights and numeric units. The recipe neither replaces canonical values nor grants acceptance or verification authority.

A fixed seed or binary64 dtype alone does not promise identical results across all Python builds, math libraries or numerical backends. Preserve the package, Python and platform identities when investigating a replay discrepancy near a threshold. Do not relax comparison tolerances or round a report until it passes. The optional NumPy diagnostics have their own numerical method identifiers and remain outside these acceptance calculations. Switching from covariance to the explicit smaller-Gram method can change a strict reference-edge count through rounding. Preserve the method and environment; the reported edge distance is not a numerical error bound. See diagnostic limits.

Bounded judge measurements use a different arithmetic path: exact rational aggregation and controlled Decimal calculations with outward-rounded interval endpoints. Their wire numbers and method are documented in the judge reference. This distinction matters when auditing rounding; a float-based reconstruction is not a replacement for judge replay.

Explore hypothetical exact-match edits

The optional Python helper below asks how many subject outcomes would need to flip to change a current native v3 exact-match verdict, with the baseline, policy and record count fixed:

from invarlock.exact_match_sensitivity import exact_match_sensitivity

baseline = [1] * 75 + [0] * 25
subject = [1] * 72 + [0] * 3 + [1] * 3 + [0] * 22
advisory = exact_match_sensitivity(
    baseline, subject, policy={"delta_min_pp": -10.0}, max_changes=8
)
assert advisory["status"] == "exact"
assert advisory["minimum_changes"] == 4

The policy argument contains exactly the native metrics.exact_match fields. Count, interval width and side-accuracy qualification still apply. Even an improved subject outcome can widen an interval enough to fail a width gate.

exact means all smaller edit distances were ruled out and a witness was replayed through the native report builder. subject_flip_indices are zero-based positions in the supplied input order. lower_bound means only distances through checked_through_changes were completely searched; minimum_changes stays null. If all possible distances were checked without a witness, no_flip_possible applies to this fixed edit model. A search stopped partway through a distance does not claim that distance was ruled out.

The default search examines at most 2,048 candidate tables through eight edits. The supported upper bounds are 5,000 tables, 32 edits and 10,000 paired records. The input digest binds the supplied outcomes and policy, not their provenance. Keep this advisory separate from signed evidence and the official decision. It does not authenticate outcomes, estimate rerun risk, support historical v1/v2 arithmetic, or apply to captured multi-slice, continuous or judge comparisons.

Notation

Symbol Meaning
\(n\) Number of scheduled records; \(1 \le n \le 10{,}000\)
\(i\) Schedule position, \(i \in \{1, \ldots, n\}\)
\(B\), \(S\) Baseline and subject sides
\(y_i\) Scheduled expected output for record \(i\)
\(\hat{y}_{X,i}\) Authenticated output text for side \(X\)
\(L_{X,i}\) Authenticated sum of target-token log probabilities
\(u_i\) Positive UTF-8 byte count of \(y_i\)
\(w_i\) Positive target token count used only for a derived perplexity interpretation
\(x_{X,i}\) Verifier-derived record score for side \(X\)
\(\bar{x}_X\) Arithmetic mean of side \(X\) record scores
\(\tau_{\Delta}\) Policy floor exact_match.delta_min_pp
\(\tau_A\) Optional exact-match floor minimum_side_accuracy for each side mean
\(\tau_{N}\) Policy ceiling normalized_nll_per_utf8_byte.ratio_max
\(\tau_{S}\) Policy floor scorer_extension.delta_min_pp
\([q_L,q_U]\) Metric-specific paired interval
\(n_{\min}\) Optional policy minimum minimum_record_count
\(w_{\max}\) Optional policy maximum interval width in percentage points or ratio units

Shared preconditions

Replay proceeds only when:

  1. the normalized request, source identity, and canonical schedule are bounded and digest-authenticated;
  2. record IDs are unique and both observations have exactly the schedule IDs in schedule order;
  3. each observed input_sha256 matches the corresponding scheduled input;
  4. every selected observation record has status: ok;
  5. provider, artifact, observation, schedule, runtime, report, and request bindings are internally consistent;
  6. the schedule and both providers declare the selected task and provider collection metric;
  7. the verifier's independently supplied baseline artifact, subject artifact, canonical schedule and both runtime digests match the bound identities;
  8. the evidence signature authenticates against the independently supplied evidence-signer fingerprint, and any request-digest anchor matches; that request anchor is required when either provider is llama_cpp; and
  9. the verifier's policy bytes have the same digest as the policy bound into the signed evidence.

Failure of a precondition is a verification error, not a poor metric score. InvarLock does not impute, discard, reorder, or average around an invalid record.

Exact-match percentage-point delta

For each side and scheduled record:

\[ x_{X,i} = \mathbf{1}\!\left[\hat{y}_{X,i}=y_i\right]. \]

Equality is literal Unicode string equality. It does not trim whitespace, case-fold, normalize, parse an answer, or compare token sequences.

The side means and point comparison are:

\[ \bar{x}_X = \frac{1}{n}\sum_{i=1}^{n}x_{X,i}, \qquad \Delta_{\mathrm{pp}} = 100\left(\bar{x}_S-\bar{x}_B\right). \]

The report records comparison.value = \(\Delta_{\mathrm{pp}}\). The policy is:

{
  "resolved_policy": {
    "metrics": {
      "exact_match": {
        "delta_min_pp": -2.0,
        "maximum_interval_width_pp": 10.0,
        "minimum_record_count": 400,
        "minimum_side_accuracy": 0.5
      }
    }
  }
}

The verdict uses the interval lower bound:

\[ \operatorname{pass}_{\mathrm{EM}} \iff q_L \ge \tau_{\Delta} \land n \ge n_{\min} \land (q_U-q_L) \le w_{\max} \land \bar{x}_B \ge \tau_A \land \bar{x}_S \ge \tau_A. \]

The boundary is inclusive. delta_min_pp is an absolute percentage-point floor, not a relative percentage and not a floor applied only to the point estimate. The count and width terms apply only when the policy supplies both coupled fields; without them, the threshold-only policy remains valid. The two side-accuracy terms apply only when minimum_side_accuracy is present. That optional finite value lies from 0 through 1 and prevents a passing relative comparison between two sides whose absolute exact-match accuracy is below the policy floor.

For every pair, InvarLock also classifies the two binary outcomes:

  • baseline pass and subject fail: a regression;
  • baseline fail and subject pass: an improvement;
  • both pass; or
  • both fail.

If \(r\) is the regression count and \(g\) is the improvement count, then \(\Delta_{\mathrm{pp}}=100(g-r)/n\). The report records all four counts, the number of discordant pairs \(r+g\), and the exact two-sided McNemar probability conditional on those discordant pairs. The McNemar result describes evidence of asymmetry between regressions and improvements; it does not control the acceptance verdict.

The exact-match interval is a paired Newcombe hybrid-score 95% interval for the subject-minus-baseline effect. Current v3 reports use the continuity-corrected method:

{
  "method": "newcombe_hybrid_score_paired_v2",
  "scope": "paired_binary_outcomes",
  "interval_mass": 0.95,
  "lower": -1.8,
  "upper": 0.7
}

The policy reads lower; the point delta and McNemar probability cannot override a lower bound below delta_min_pp.

For a signed v1 report, strict replay instead requires newcombe_hybrid_score_paired_v1 and reconstructs the legacy result exactly. It does not apply the v2 calculation to old evidence.

Normalized NLL per UTF-8 byte

For each side and record, the provider performs teacher-forced scoring of the authenticated expected continuation and supplies finite \(L_{X,i}\) and positive byte count \(u_i\). Prompt tokens are excluded from the target loss. The verifier requires the byte count to equal the UTF-8 length of the authenticated scheduled target, then derives:

\[ x_{X,i} = -\frac{L_{X,i}}{u_i}. \]

Every derived value must be finite and nonnegative. The side means use equal record weight:

\[ \bar{x}_X = \frac{1}{n}\sum_{i=1}^{n}x_{X,i}. \]

The baseline mean must be positive. The point ratio is:

\[ R_{\mathrm{NLL}} = \frac{\bar{x}_S}{\bar{x}_B}. \]

The policy is:

{
  "resolved_policy": {
    "metrics": {
      "normalized_nll_per_utf8_byte": {
        "maximum_interval_width_ratio": 0.05,
        "minimum_record_count": 400,
        "ratio_max": 1.05
      }
    }
  }
}

The verdict uses the interval upper bound:

\[ \operatorname{pass}_{\mathrm{NLL}} \iff q_U \le \tau_N \land n \ge n_{\min} \land (q_U-q_L) \le w_{\max}. \]

This statistic is a ratio of arithmetic means of per-record normalized scores. It is not pooled NLL, a byte-weighted mean, token-weighted mean, or perplexity. Each selected record contributes one score regardless of target length. It measures expected-continuation likelihood regression under the authenticated prompt, target, provider, and runtime; it is not a general model-quality score.

The count and ratio-width terms apply only when their coupled policy fields are present. Their pass does not establish that the schedule represents a broader population.

Worked normalized-NLL example

Suppose two authenticated records yield:

Record Baseline \(x_{B,i}\) Subject \(x_{S,i}\)
1 \(-(-5)/10 = 0.5\) \(-(-5.5)/10 = 0.55\)
2 \(-(-20)/20 = 1.0\) \(-(-22)/20 = 1.1\)

Then \(\bar{x}_B=0.75\), \(\bar{x}_S=0.825\), and the point ratio is \(1.1\). That point fails a ratio_max of 1.05; even when a point ratio is below the ceiling, the canonical verdict still fails if its interval upper bound exceeds 1.05.

Derived perplexity interpretation

For a normalized-NLL comparison, the verifier may also derive a token-weighted perplexity ratio when both sides bind one matching authenticated tokenizer contract and report the same positive target token count \(w_i\) for every pair. It derives token-normalized record NLL:

\[ x_{X,i} = -\frac{L_{X,i}}{w_i}. \]

The token-weighted mean NLL and side perplexity are:

\[ \mu_X^{(w)} = \frac{\sum_{i=1}^{n}w_i x_{X,i}}{\sum_{i=1}^{n}w_i}, \qquad P_X = \exp\!\left(\mu_X^{(w)}\right). \]

The derived ratio is:

\[ R_{\mathrm{PPL}} = \frac{P_S}{P_B} = \exp\!\left(\mu_S^{(w)}-\mu_B^{(w)}\right). \]

The canonical derived_measurements.perplexity_ratio object records the authenticated likelihood basis, method, tokenizer digest, total target-token count, both side perplexities, and the ratio. If tokenizer contracts differ, token counts are unavailable or unequal, or exponentiation is non-finite, the object records a deterministic unavailable reason instead.

This interpretation is not a selectable metric. It has no policy threshold, confidence interval, or verdict authority. Acceptance continues to use the normalized-NLL ratio and its schedule-resampling interval.

Normalized-NLL paired resampling

The report's uncertainty object is fixed:

{
  "method": "paired_percentile_bootstrap_sha256_v1",
  "scope": "authenticated_schedule",
  "interval_mass": 0.95,
  "replicates": 2048,
  "lower": 0.98,
  "upper": 1.04
}

Let \(d\) be the 32 bytes represented by the canonical schedule's lowercase SHA-256. For replicate \(r \in \{0,\ldots,2047\}\) and draw \(j \in \{0,\ldots,n-1\}\), InvarLock computes:

\[ h_{r,j} = \operatorname{SHA256} \left(d\,\|\,\operatorname{u64be}(r)\,\|\,\operatorname{u64be}(j)\right), \]

and selects index:

\[ k_{r,j} = \operatorname{u64be}\!\left(h_{r,j}[0:8]\right) \bmod n. \]

The baseline and subject normalized-NLL scores at \(k_{r,j}\) are drawn together. InvarLock recomputes the complete ratio for each replicate, sorts the 2,048 values, and uses linearly interpolated percentiles at \(0.025\) and \(0.975\). When \(n=1\), the interval is the point value.

This construction provides:

  • deterministic replay without a platform PRNG;
  • pairing preservation under every draw;
  • a versioned algorithm and fixed replicate count; and
  • a conservative bound that directly controls the policy verdict.

It does not provide random-sample provenance. The SHA-256-derived index stream makes replay deterministic; it does not make the original selected schedule a probability sample.

Authorized deterministic scorer extension

A request may select one closed scorer_extension binding instead of a built-in metric. For every side and record, the provider authenticates exactly the scheduled expected output, output text, and output digest. The explicitly authorized scorer replays those facts and returns:

\[ 0 \le x_{X,i} \le 1, \]

where larger values are always better. The scorer does not own aggregation or policy arithmetic. The core computes:

\[ \bar{x}_X = \frac{1}{n}\sum_{i=1}^{n}x_{X,i}, \qquad \Delta_{S,\mathrm{pp}} = 100\left(\bar{x}_S-\bar{x}_B\right). \]

The independent policy pins the exact scorer identity and configuration:

{
  "resolved_policy": {
    "metrics": {
      "scorer_extension": {
        "scorer_id": "example.token_f1",
        "scorer_version": "1.0.0",
        "descriptor_sha256": "5555555555555555555555555555555555555555555555555555555555555555",
        "configuration_sha256": "6666666666666666666666666666666666666666666666666666666666666666",
        "delta_min_pp": -2.0,
        "maximum_interval_width_pp": 10.0,
        "minimum_record_count": 400
      }
    }
  }
}

The scorer runs twice during replay and both canonical results must match. The core then applies the fixed 2,048-replicate paired schedule-resampling method defined above to the paired unit-interval values. The verdict is:

\[ \operatorname{pass}_{S} \iff q_L \ge \tau_S \land n \ge n_{\min} \land (q_U-q_L) \le w_{\max}. \]

The core already ships invarlock.normalized_match, invarlock.numeric_tolerance, invarlock.json_fields, invarlock.json_exact and invarlock.token_f1 through this extension contract. A separately supplied scorer, such as VQA answer normalization, needs explicit authorization. See the scorer reference. The scorer-extension v1 contract does not admit SQL or code execution, model-based semantic similarity, network services, externally assigned ratings, external models, or LLM judges. Those sources do not execute through the deterministic extension contract.

The native judge scorer uses a separate bounded measurement decision contract. It retains and replays the declared judge requests, responses, attempts, parsing, independent-unit aggregation, interval, and recipient policy. It also accepts captured evaluator answers and retained judge calls. See judge measurements for those acceptance rules; no live judge calls occur inside deterministic-extension replay.

For the native exact-match, NLL and extension metrics above, the sample controls are optional but indivisible. Exact match and scorer extensions use maximum_interval_width_pp, whose value must be positive and no greater than 200. Normalized NLL uses positive maximum_interval_width_ratio. minimum_record_count is an integer from 1 through 10,000. The canonical v3 report records the minimum, maximum, observed values, units, individual results, and combined sample_qualification.passed.

Captured comparison and bounded judge decisions

Captured exact-match and normalized-NLL requests use the invarlock/multi-metric-comparison-v1 contract. Its binary metrics use paired Newcombe intervals in score units; continuous delta metrics use paired_mean_shake256_percentile_v1, which differs from the native SHA-256 resampling method. Captured NLL reuses the native mean-ratio arithmetic and paired_percentile_bootstrap_sha256_v1, seeded from the sorted scope's retained case facts rather than a native provider schedule. Captured policy evaluates every metric and slice; its marginal intervals do not provide simultaneous family-wide coverage.

Captured recorded metrics can apply policy to upstream scores, including judge or other externally assigned ratings with explicit approved provenance and rubric binding. Replay recomputes aggregation and decision arithmetic, not the upstream judgment. Missing facts or insufficient count produce insufficient_evidence; a violated regression or subject bound produces regression; an otherwise acceptable but overly wide interval produces insufficient_evidence. The captured regression label is a policy-gate result: its conservative bound can fail even when the interval straddles the allowed-degradation boundary. It does not claim that the entire interval demonstrates adverse change. Absolute subject bounds apply to the observed subject mean, not a confidence bound on that mean. A bound violation takes precedence over excessive width; missing measurements or an unmet minimum count are checked before either. Across metric/slice rows, any regression takes precedence over insufficient_evidence. All rows must pass for the comparison to pass. See the captured policy contract.

Selecting judge instead uses fixed-benchmark-hoeffding-v1 over bounded, declared units under an assumption of independence across those units. Replay checks grouping and arithmetic; it cannot establish that independence. Replay averages repetitions within cases and cases within units before calculating equal-unit means. Repetitions and additional cases in one unit do not increase the inference sample size. The Hoeffding bound targets the fixed benchmark's average expected score under the declared independence assumption; it is not the schedule-composition bootstrap described above. Bonferroni adjustment uses the declared family size and error budget, including published advisory intervals. Constant observed ratings still have positive interval width unless the declared support itself is constant.

Judge preflight also reports an advisory precision forecast from the declared scale, error budget, comparison-family size and independent-unit count. It assumes complete trials and independent units; it does not predict acceptance or change execution readiness. Additional repetitions in the same units do not improve this forecast. See judge preflight for guaranteed-width counts and unattainable-width limits.

Judge gates distinguish pass, regression and insufficient_evidence. Within each judge gate, minimum-unit and maximum-width checks take precedence: a failure yields insufficient_evidence. Once those checks pass, an interval entirely beyond tolerated degradation establishes regression; an interval that straddles the decision boundary remains inconclusive. An optional absolute subject bound uses the subject interval with the same decision rules, unlike the captured subject-mean check above. Incomplete scheduled trials yield insufficient_evidence without complete-case inference. Across required judge gates, regression takes precedence over insufficient evidence. decision_role: required controls the conjunction of metrics; advisory metrics cannot independently authorize recipient acceptance. Optional reference-label studies assess the judge's usefulness but are not inputs required by runtime replay. See judge statistics.

Verifier-owned replay

For native pack-v1 evidence, the evidence signer cannot supply an accepted aggregate directly. Verification:

  • reconstructs the canonical schedule and record order;
  • derives exact-match or normalized-NLL record scores from authenticated facts, or requires an explicitly authorized scorer and replays it twice;
  • derives paired exact-match counts, exact McNemar probability, and the Newcombe interval, or the normalized-NLL or scorer-extension schedule-resampling interval;
  • derives a perplexity interpretation only when tokenizer and token-count facts are comparable;
  • reconstructs canonical paired records;
  • recomputes side means, point comparison, the selected interval, threshold application, optional count/width and exact-match side-accuracy qualification, and verdict; and
  • requires canonical equality with the stored report.

The implementation is in build_comparison_report, and independent replay is in verify_comparison_evidence.

Finite-schedule interpretation

For the native comparisons derived above, a pass means that the policy's conservative bound cleared its threshold and any configured count, width, and exact-match side-accuracy requirements passed for the authenticated records and runtime configuration. It does not establish:

  • equality, equivalence, or non-inferiority in a broader population;
  • representativeness of the selected prompts or targets;
  • statistical power for a separately defined hypothesis;
  • safety, robustness, fairness, or general model quality;
  • execution attestation from a runtime-image digest; or
  • immunity to repeated tuning or best-run selection bias.

Where population inference is necessary, define and precommit a sampling and statistical protocol outside InvarLock and retain that evidence separately. Do not extend either paired interval beyond the authenticated schedule without those assumptions.

References