v0.2.0 — Local implementation and verification record¶
| Item | Result |
|---|---|
| Verification date | October 19, 2026, Asia/Karachi |
| Candidate version | 0.2.0 |
| Functional source | Commits through 0f18987; release preparation files accompany this record |
| Starting checkout | 0882240, package version 0.1.3, initially clean |
| Local disposition | Implementation and required local behavior checks pass |
| Hosted disposition | GitHub-hosted CI, real artifact transport, and publication remain pending the maintainer's manual push/release actions |
| External changes | No push, tag, package publication, production golden update, production baseline promotion, or real PR comment was performed |
Verified results¶
| Environment/check | Actual result |
|---|---|
| Python 3.12.14, full suite with integration enabled | 374 passed, 15 skipped |
| Python 3.11.16, installed candidate, full suite with integration enabled | 374 passed, 15 skipped |
| Optional Chroma cases | 15 skipped in each environment because the optional Chroma SDK is absent |
| Source and changed/new Python lint | Passed |
| Formatting of changed/new Python files | Passed |
| Actionlint 1.7.12 on changed workflows and CI examples | Passed |
| Local workflow JavaScript with API stubs | 8 passed; verifies trusted-run selection and create/update/denied-comment paths without GitHub writes |
| Lockfile metadata check | Passed: uv lock --check --offline; no dependency-version upgrade was required |
| Wheel and source archive build | Passed using the configured Hatchling backend |
| Base-only wheel install outside checkout | Passed on Python 3.12.14 |
| Installed CLI HTTP workflow and local CI helper | Passed against the synthetic localhost retriever |
| High-average baseline rejection in installed wheel | 80% mean with one failed question rejected; existing reference preserved |
| Saved-report rendering with configuration absent | Passed; no evaluation/configuration needed |
| Startup side effects | No startup service connection or project-data writes observed under guarded smoke check |
| Runtime/distribution/source-archive identity | 0.2.0; Requires-Python: >=3.11 |
| Strict MkDocs build | Passed with MkDocs 1.6.1 and Material 9.7.7 |
The only initial baseline-suite error was the execution sandbox rejecting a localhost socket. The controlled HTTP tests passed when run with the required local-socket permission. This was an environment restriction, not a product failure.
The Python 3.12 development environment used pytest 9.1.1, pytest-asyncio 1.4.0, Typer 0.27.1, Rich 15.0.0, PyYAML 6.0.3, NumPy 2.4.4, and Requests 2.33.1. The separately installed Python 3.11 candidate also passed with Typer 0.27.3, NumPy 2.4.6, and Requests 2.34.2. The base-only wheel environment had no pytest, generation SDK, framework SDK, or vector-store SDK installed.
Requirement evidence¶
Paths below are relative to the repository root.
| Outcome | Implementation | Executable evidence |
|---|---|---|
| EVAL-01: Shared evaluation | core/evaluation.py, Python facade, CLI delegation, shared adapter construction |
tests/unit/test_evaluation.py; CLI/Python settings/results parity and zero-call invalid selections |
| EVAL-02: Baseline gates | core/policy.py, strict config overrides, defensive candidate snapshots |
test_evaluation_policy.py, test_cli_baseline_quality_gate.py, test_evaluation.py; high-average rejection, explicit aggregate mode, drafts, mutation/stale state, invalid labels |
| EVAL-03: Pytest | Configured adapter fixture, required selected config, session report, opt-in enforcement | tests/integration/test_pytest_plugin.py; 11 isolated user projects/settings cases, without private config assignments |
| EVAL-04: Comparable differences | Versioned contracts, unified reporter/store differences, legacy preservation | test_report_serialization.py, test_evaluation_policy.py, test_diff.py; changed coverage/requirements/settings, equal-recall swaps, reordered inputs, malformed/legacy data |
| EVAL-05: Reference identity | Explicit required labels, visible optional default absence, no implicit promotion | test_evaluation.py, test_cli_json.py; selected label, timestamp/status, missing required label before retrieval |
| EVAL-06: Automation | Shared payload reader/writer, error envelopes, exit categories, read-only report command | test_cli_json.py; stdout/stderr separately parsed, quality relaxation, operational errors, saved report without config |
| EVAL-07: CI | Version-aware installation, artifact validation/restoration, one subprocess evaluation, saved feedback, final verdict | test_ci.py, tests/integration/test_ci_workflow.py, test_workflow_scripts.py; real local HTTP call counter and local API stubs |
| Retained v0.1.3 behavior | Strict configuration, empty rejection, lazy SDKs, safe files, stale-state invalidation | Existing configuration/startup/file-safety/empty-selection tests and extended controlled workflow |
Artifact restore tests verify read-only validation and preservation of an existing destination when the artifact is missing, malformed, legacy, has multiple databases, has the wrong label/path, or is incompatible. Local CI tests prove one retrieval per evaluated question while annotations and Markdown are derived from the same JSON report.
Version 0.2.0 deliberately changes baseline quality defaults, compatibility handling, report fields, CLI exits, and pytest threshold semantics. See the migration notes and baseline guide.
Implementation choices¶
evaluate()returns the existingProbeReportwith comparison/quality metadata rather than adding a second public result wrapper. This preserves the documented facade return type.check --no-comparemakes intentionally unbaselined checks explicit, including CI without restoration inputs.- Workflow helpers live in the installed package, so a reusable workflow does not depend on helper files existing in the caller repository.
- Pytest uses public stash storage and opt-in session enforcement instead of private configuration attributes or synthetic test items.
- Source archives include workflow files referenced by packaged regression tests.
Commit sequence¶
| Commit | Main change |
|---|---|
d61fd31 |
Versioned contracts and shared legacy-preserving serialization |
98a7b63 |
Shared baseline policy and structural/evidence differences |
a5efff6 |
Shared evaluation, adapter construction, and defensive Python promotion |
14a4428 |
Clean JSON, failure categories, explicit reference, saved-report rendering |
94f2b4a |
Configured pytest execution and supported report fixture |
f916741 |
One-run CI and explicit trusted artifact restoration |
f00bb46 |
Contradictory-report rejection and required selected pytest config |
e836b45 |
Formatting and verification cleanup |
47b6252 |
Executed workflow trust/comment-permission checks |
0f18987 |
Unusable baseline labels and missing CI input categorization |
The release-preparation commit contains the final version/lockfile, migration documentation, CI test selection, and installed-artifact verifier. No commits were pushed or amended as part of implementation.
Reproduction¶
From an installed source checkout:
uv sync --locked --dev --python 3.12
uv run pytest tests/ --run-integration -q
uv run ruff check src/
uv lock --check --offline
uv build
Use a separate Python 3.11 environment for the second matrix result. The local HTTP cases need permission to bind localhost. Workflow JavaScript tests need Node; they are explicitly marked as unavailable if Node is absent.
The actual artifact build used a temporary environment containing Hatchling and the Python build tool:
For a base-only installed-artifact check:
uv venv /tmp/longprobe-v020-wheel --python 3.12
uv pip install --python /tmp/longprobe-v020-wheel/bin/python dist/longprobe-0.2.0-py3-none-any.whl
/tmp/longprobe-v020-wheel/bin/python -I examples/verify_v020_release.py
The verifier checks the installed package outside the checkout, optional-SDK/test-dependency absence, startup behavior, actual HTTP CLI operations, baseline rejection/preservation, report rendering, local CI feedback, distribution identity, source archive contents, and hashes.
Workflow validation used the official actionlint 1.7.12 Linux binary after verifying its release checksum. The downloaded archive SHA-256 was 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8. Actionlint and all temporary verification environments remain outside the repository.
Verified artifacts¶
These identify the files used by the final clean-install verifier, not hypothetical future rebuilds.
| Artifact | SHA-256 |
|---|---|
longprobe-0.2.0-py3-none-any.whl |
72732cc1ac06c89c54e51789161eb214bf52da3d48b3262f38604d7756d0b245 |
longprobe-0.2.0.tar.gz |
39b89a1e6d1c8ccdf9c90c840826a18b518bd5a5b096d2a96b18879b3554c206 |
The source archive includes workflow files needed by the packaged CI-script tests. Verification records are repository documentation rather than inputs to the source archive, avoiding a self-referential artifact hash.
Release identity and remaining checks¶
Read-only publication checks on October 19, 2026 returned PyPI versions 0.1.0 and 0.1.1, and public tags v0.1.0 and v0.1.1. The proposed 0.2.0 number was unused at that time. Recheck availability before actual tagging/publication. Sources: PyPI metadata, repository tags.
Before public release, the maintainer still needs to:
- Push the local commits and inspect hosted CI results.
- Verify real artifact download/upload and workflow token permissions on GitHub, using a reviewed default-branch baseline.
- Confirm publication/version identity and follow the project's tagging/release procedure.
Cloud retrieval, paid generation, the full all-extras environment, Windows/macOS filesystem behavior, and real GitHub PR publication were not exercised. Optional SDK behavior outside the changed evaluation contract remains subject to the documented limitations. Local API stubs establish workflow-script logic, not real hosted permission acceptance.