Skip to content

v0.2.0 — Consistent testing and baseline rules

Status: implemented locally; package publication and hosted Actions acceptance are pending. No tag or push is performed by the implementation task. See the verification record for actual environments and limitations.

Delivered behavior

Area Change
Evaluation CLI and Python coordinate selection, scoring, policy, and reference comparison through one module; top_k is strictly enforced during scoring (returned_count and evaluated_count recorded)
Baseline quality Default promotion requires every evaluated question to pass; explicit aggregate and per-question threshold settings are validated
Comparison Added/removed questions, changed requirements/settings, and lost/gained required evidence are visible
Compatibility Recorded matching/retrieval/threshold contracts (longprobe-lexical-v2) determine strict comparability; incompatible suites or pre-v2 references have no like-for-like aggregate delta
Baseline identity Requested/resolved labels, reference timestamp, and comparison status are recorded and displayed
HTTP validation Valid empty retrieval arrays are evaluated as quality misses; missing paths, non-list results, or malformed items fail as unrelaxable execution errors (invalid_response, http_error, timeout, transport_error)
Pytest Configured default/custom adapters, session report reuse, and explicit automatic enforcement work without private user assignments
Automation Parsed JSON invocations emit one stdout object; command failures have documented categories
CI & Privacy Shared CI outputs (PR comments, annotations, default longprobe-result.json) redact sensitive question text and chunk evidence by default, keeping verdicts and safe counts intact

Upgrade checklist

  1. Back up your SQLite baseline file with writers closed.
  2. Review goldens.yaml and approve only verified questions/evidence.
  3. Inspect threshold policy, including any per-question overrides.
  4. Run longprobe check --no-compare to inspect current quality independently.
  5. Explicitly save a reviewed eligible reference with longprobe baseline save --label latest.
  6. Update JSON consumers and pytest usage to the documented interfaces.
  7. Configure a successful trusted default-branch baseline run/artifact for reusable CI comparisons.

Behavior changes

  • CLI and Python baseline promotion now use the same default gate. An average above threshold does not compensate for a failed question. Select baseline.quality_gate: aggregate explicitly only when that is the intended policy.
  • Per-question thresholds override invocation/global thresholds. Unknown IDs fail against the full loaded suite.
  • Python retains a defensive candidate with its evaluation-time policy. Later mutation of the returned report or configuration does not weaken it; rerun to adopt new settings.
  • Removed/added evaluated questions and changed contracts make strict whole-suite comparison incompatible. Equal-recall evidence loss is a regression.
  • Existing unversioned reports load and list without rewriting, but missing historical settings make strict comparison unverified. Independent baseline saving remains available.
  • latest remains literal. Saving another label does not update it. Explicit missing references fail; optional default absence is visible.
  • check --baseline LABEL selects a required reference. check --no-compare skips comparison and cannot be combined with that explicit label.
  • JSON includes schema version 2, full result contracts, policy/quality, baseline identity, and comparison metadata. Existing aggregate keys and the diff alias remain available. A handled input/execution error uses a separate safe error envelope.
  • Exit codes are 0 for success/nonblocking quality, 1 for blocking quality, 2 for invalid/empty/incompatible setup, and 3 for execution failures. Parser-level usage errors retain ordinary stderr/code 2 behavior.
  • longprobe report PATH renders a validated saved report without retrieval. Rendering does not itself re-enforce the report's quality gate.
  • longprobe_report is the supported pytest fixture. --longprobe-fail-threshold now enforces selected questions and comparison quality rather than relying on an aggregate description and private config state.
  • Reusable workflow baseline restoration is explicit. Missing restoration inputs run unbaselined; configured invalid/missing artifacts fail. Comments can be skipped for restricted PR permissions without changing the verdict.

No SQL table migration is required. New report JSON fields are stored in existing rows only when explicitly saving/replacing a label. Old executables can discard metadata they do not understand if used to replace labels; preserve backups before downgrading.

Unchanged scope and limitations

Requires Python 3.11 or newer; the validated interpreter matrix remains 3.11 and 3.12. This release retains strict configuration, empty-evaluation rejection, stale-state invalidation, optional-SDK lazy imports, and safe golden-file writes.

Normal evaluation does not wire embedding conversion into vector-only Pinecone/Qdrant flows. Semantic matching remains word-frequency cosine similarity. HTTP adapter responses are strictly validated and transport/response errors are classified operationally without exposing raw response bodies. Explain does not calculate rank/score deltas, and watch still responds only to golden-file changes.

Golden suites test selected reviewed examples, not all possible questions or every live user query. Generated/captured evidence still needs human review. Baseline promotion and golden approval remain explicit; there is no new manager sign-off application.

The older changelog's named @longprobe_check and historical rank/distance explanation claims do not describe supported v0.2.0 interfaces. Use the documented report fixture and missing-evidence explanation behavior.

See CLI reference, baseline management, Python interface, pytest, and CI/CD.