Skip to content

v0.2.0 — Local implementation and verification record

Item Result
Verification date October 19, 2026, Asia/Karachi
Candidate version 0.2.0
Functional source Commits through 0f18987; release preparation files accompany this record
Starting checkout 0882240, package version 0.1.3, initially clean
Local disposition Implementation and required local behavior checks pass
Hosted disposition GitHub-hosted CI, real artifact transport, and publication remain pending the maintainer's manual push/release actions
External changes No push, tag, package publication, production golden update, production baseline promotion, or real PR comment was performed

Verified results

Environment/check Actual result
Python 3.12.14, full suite with integration enabled 374 passed, 15 skipped
Python 3.11.16, installed candidate, full suite with integration enabled 374 passed, 15 skipped
Optional Chroma cases 15 skipped in each environment because the optional Chroma SDK is absent
Source and changed/new Python lint Passed
Formatting of changed/new Python files Passed
Actionlint 1.7.12 on changed workflows and CI examples Passed
Local workflow JavaScript with API stubs 8 passed; verifies trusted-run selection and create/update/denied-comment paths without GitHub writes
Lockfile metadata check Passed: uv lock --check --offline; no dependency-version upgrade was required
Wheel and source archive build Passed using the configured Hatchling backend
Base-only wheel install outside checkout Passed on Python 3.12.14
Installed CLI HTTP workflow and local CI helper Passed against the synthetic localhost retriever
High-average baseline rejection in installed wheel 80% mean with one failed question rejected; existing reference preserved
Saved-report rendering with configuration absent Passed; no evaluation/configuration needed
Startup side effects No startup service connection or project-data writes observed under guarded smoke check
Runtime/distribution/source-archive identity 0.2.0; Requires-Python: >=3.11
Strict MkDocs build Passed with MkDocs 1.6.1 and Material 9.7.7

The only initial baseline-suite error was the execution sandbox rejecting a localhost socket. The controlled HTTP tests passed when run with the required local-socket permission. This was an environment restriction, not a product failure.

The Python 3.12 development environment used pytest 9.1.1, pytest-asyncio 1.4.0, Typer 0.27.1, Rich 15.0.0, PyYAML 6.0.3, NumPy 2.4.4, and Requests 2.33.1. The separately installed Python 3.11 candidate also passed with Typer 0.27.3, NumPy 2.4.6, and Requests 2.34.2. The base-only wheel environment had no pytest, generation SDK, framework SDK, or vector-store SDK installed.

Requirement evidence

Paths below are relative to the repository root.

Outcome Implementation Executable evidence
EVAL-01: Shared evaluation core/evaluation.py, Python facade, CLI delegation, shared adapter construction tests/unit/test_evaluation.py; CLI/Python settings/results parity and zero-call invalid selections
EVAL-02: Baseline gates core/policy.py, strict config overrides, defensive candidate snapshots test_evaluation_policy.py, test_cli_baseline_quality_gate.py, test_evaluation.py; high-average rejection, explicit aggregate mode, drafts, mutation/stale state, invalid labels
EVAL-03: Pytest Configured adapter fixture, required selected config, session report, opt-in enforcement tests/integration/test_pytest_plugin.py; 11 isolated user projects/settings cases, without private config assignments
EVAL-04: Comparable differences Versioned contracts, unified reporter/store differences, legacy preservation test_report_serialization.py, test_evaluation_policy.py, test_diff.py; changed coverage/requirements/settings, equal-recall swaps, reordered inputs, malformed/legacy data
EVAL-05: Reference identity Explicit required labels, visible optional default absence, no implicit promotion test_evaluation.py, test_cli_json.py; selected label, timestamp/status, missing required label before retrieval
EVAL-06: Automation Shared payload reader/writer, error envelopes, exit categories, read-only report command test_cli_json.py; stdout/stderr separately parsed, quality relaxation, operational errors, saved report without config
EVAL-07: CI Version-aware installation, artifact validation/restoration, one subprocess evaluation, saved feedback, final verdict test_ci.py, tests/integration/test_ci_workflow.py, test_workflow_scripts.py; real local HTTP call counter and local API stubs
Retained v0.1.3 behavior Strict configuration, empty rejection, lazy SDKs, safe files, stale-state invalidation Existing configuration/startup/file-safety/empty-selection tests and extended controlled workflow

Artifact restore tests verify read-only validation and preservation of an existing destination when the artifact is missing, malformed, legacy, has multiple databases, has the wrong label/path, or is incompatible. Local CI tests prove one retrieval per evaluated question while annotations and Markdown are derived from the same JSON report.

Version 0.2.0 deliberately changes baseline quality defaults, compatibility handling, report fields, CLI exits, and pytest threshold semantics. See the migration notes and baseline guide.

Implementation choices

  • evaluate() returns the existing ProbeReport with comparison/quality metadata rather than adding a second public result wrapper. This preserves the documented facade return type.
  • check --no-compare makes intentionally unbaselined checks explicit, including CI without restoration inputs.
  • Workflow helpers live in the installed package, so a reusable workflow does not depend on helper files existing in the caller repository.
  • Pytest uses public stash storage and opt-in session enforcement instead of private configuration attributes or synthetic test items.
  • Source archives include workflow files referenced by packaged regression tests.

Commit sequence

Commit Main change
d61fd31 Versioned contracts and shared legacy-preserving serialization
98a7b63 Shared baseline policy and structural/evidence differences
a5efff6 Shared evaluation, adapter construction, and defensive Python promotion
14a4428 Clean JSON, failure categories, explicit reference, saved-report rendering
94f2b4a Configured pytest execution and supported report fixture
f916741 One-run CI and explicit trusted artifact restoration
f00bb46 Contradictory-report rejection and required selected pytest config
e836b45 Formatting and verification cleanup
47b6252 Executed workflow trust/comment-permission checks
0f18987 Unusable baseline labels and missing CI input categorization

The release-preparation commit contains the final version/lockfile, migration documentation, CI test selection, and installed-artifact verifier. No commits were pushed or amended as part of implementation.

Reproduction

From an installed source checkout:

uv sync --locked --dev --python 3.12
uv run pytest tests/ --run-integration -q
uv run ruff check src/
uv lock --check --offline
uv build

Use a separate Python 3.11 environment for the second matrix result. The local HTTP cases need permission to bind localhost. Workflow JavaScript tests need Node; they are explicitly marked as unavailable if Node is absent.

The actual artifact build used a temporary environment containing Hatchling and the Python build tool:

python -m build --no-isolation

For a base-only installed-artifact check:

uv venv /tmp/longprobe-v020-wheel --python 3.12
uv pip install --python /tmp/longprobe-v020-wheel/bin/python dist/longprobe-0.2.0-py3-none-any.whl
/tmp/longprobe-v020-wheel/bin/python -I examples/verify_v020_release.py

The verifier checks the installed package outside the checkout, optional-SDK/test-dependency absence, startup behavior, actual HTTP CLI operations, baseline rejection/preservation, report rendering, local CI feedback, distribution identity, source archive contents, and hashes.

Workflow validation used the official actionlint 1.7.12 Linux binary after verifying its release checksum. The downloaded archive SHA-256 was 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8. Actionlint and all temporary verification environments remain outside the repository.

Verified artifacts

These identify the files used by the final clean-install verifier, not hypothetical future rebuilds.

Artifact SHA-256
longprobe-0.2.0-py3-none-any.whl 72732cc1ac06c89c54e51789161eb214bf52da3d48b3262f38604d7756d0b245
longprobe-0.2.0.tar.gz 39b89a1e6d1c8ccdf9c90c840826a18b518bd5a5b096d2a96b18879b3554c206

The source archive includes workflow files needed by the packaged CI-script tests. Verification records are repository documentation rather than inputs to the source archive, avoiding a self-referential artifact hash.

Release identity and remaining checks

Read-only publication checks on October 19, 2026 returned PyPI versions 0.1.0 and 0.1.1, and public tags v0.1.0 and v0.1.1. The proposed 0.2.0 number was unused at that time. Recheck availability before actual tagging/publication. Sources: PyPI metadata, repository tags.

Before public release, the maintainer still needs to:

  1. Push the local commits and inspect hosted CI results.
  2. Verify real artifact download/upload and workflow token permissions on GitHub, using a reviewed default-branch baseline.
  3. Confirm publication/version identity and follow the project's tagging/release procedure.

Cloud retrieval, paid generation, the full all-extras environment, Windows/macOS filesystem behavior, and real GitHub PR publication were not exercised. Optional SDK behavior outside the changed evaluation contract remains subject to the documented limitations. Local API stubs establish workflow-script logic, not real hosted permission acceptance.