Changelog¶
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
[0.2.0] - Unreleased¶
Added¶
- Versioned evaluation contracts and shared JSON/SQLite report validation; legacy rows remain readable without automatic rewriting.
- Explicit per-question thresholds, baseline quality policy, reference identity, and coverage/requirement/evidence differences.
check --baseline LABEL,check --no-compare, JSON baseline-save output, and read-onlyreport PATHrendering.- Supported session-scoped
longprobe_reportpytest fixture and automatic opt-in quality enforcement. - Trusted baseline artifact restoration and one-evaluation CI feedback with retained diagnostics.
Changed¶
- CLI and Python share evaluation and baseline promotion; the default requires every evaluated approved question to pass.
- Equal-recall loss of required evidence is a regression. Changed contracts/coverage and metadata-free references cannot certify strict comparison.
- Parsed JSON commands emit one stdout document; handled failures use safe error envelopes and documented exit categories.
- Reusable CI honors package version selection and only relaxes completed quality failures when explicitly configured.
Migration¶
- Back up old baseline databases and explicitly establish reviewed new-format references; no SQL table migration is required.
- Global threshold overrides preserve explicit per-question thresholds. Aggregate promotion requires an explicit policy selection.
- Adopt
longprobe_reportinstead of private pytest config assignments; old named-decorator/rank-delta changelog claims are not implemented interfaces. - See v0.2.0 release notes for format, exit-code, fixture, and baseline changes. Hosted CI/publication remain pending.
[0.1.3] - Local reliability milestone (unpublished)¶
Fixed¶
- Runtime version now derives from installed package metadata; base startup keeps optional SDKs lazy and missing SDK calls give installation guidance.
- Invalid CLI configuration fails explicitly. Strict shapes, scalar types/ranges, and active environment credentials are validated before service calls; deliberate Python default configuration remains supported.
- Empty evaluations fail across the scorer, Python facade, and CLI; failed reruns invalidate older reports for baseline saving.
- Golden writes validate content and use same-directory atomic replacement, preserving existing data on ordinary preparation/write failures.
- Auto-capture validates existing goldens before generation and never resets an unreadable or invalid set. Initialization preserves files unless forced.
Added¶
- Explicit
generate --output PATH --overwritefor question-list replacement; incompatible output/auto-capture flags fail preflight. - Controlled HTTP workflow and failure-injection regression tests across supported Python versions.
- Local HTTP quick-start fixture, accurate configuration examples, and recovery/known-limitations documentation.
Compatibility¶
- Minimum Python version is now 3.11; Python 3.10 support and its CI job have been removed.
- Empty selections and invalid settings that previously passed or fell back to defaults now fail.
- Use an installed editable checkout for development; the runtime version requires distribution metadata.
- No baseline database/schema migration. Embedding integration, baseline-policy unification, default pytest fixture repair, and general JSON-output cleanup remain deferred.
[0.1.2] - 2026-08-11¶
Historical development entry. Evaluation, baseline, pytest, and CI consistency was completed locally in the v0.2.0 candidate; hosted acceptance remains pending.
Added¶
- Draft vs. Approved Status Schema: Added
statusfield (draft/approved) toGoldenQuestion. Auto-captured questions start indraftstate so unvetted questions do not pollute regression suites. - Interactive TUI Editor (
longprobe edit): Terminal editor using Rich to browse, filter, edit, and approve draft golden questions into trusted ground truth. - Pytest Plugin (
pytest-longprobe): Early pytest integration. For v0.2.0 use thelongprobe_reportfixture; there is no supported@longprobe_checkdecorator. - Strict Baseline Quality Gating (
longprobe baseline save): Prevents saving low-recall or degraded probe reports as trusted baselines (--threshold), evaluating strictly on approved questions. - Two-Baseline Comparison (
longprobe diff <a<b>): Compare any two historical snapshot baselines side-by-side with--thresholdquality verification. - Diagnostic Summaries (
longprobe explain): Human-readable missing-evidence analysis. Rank deltas and distance drops are not implemented. - CLI
--include-draftsOption: Flag oncheckand livediffto evaluate draft questions on demand.
0.1.1 - 2026-05-06¶
Added¶
- Professional MkDocs documentation site with Material theme
- Three live demo GIFs showcasing key features:
- Test RAG Retrieval (quick validation)
- Monitor RAG Quality (detailed analysis)
- Detect Regressions (baseline comparison)
- Comprehensive documentation structure:
- Getting Started guides (Installation, Quick Start, Configuration)
- User Guide (Golden Questions, Match Modes, CLI Reference, Python API, Baseline Management)
- Integration guides (Vector Stores, Pytest, CI/CD, LangChain, LlamaIndex)
- Demo pages with detailed explanations
- API Reference
- Contributing guide
- GitHub Actions workflow for automatic documentation deployment
- Custom CSS for improved documentation UI
- Professional README with logo, badges, and demo GIFs
Changed¶
- Updated README with centered logo and GitHub raw URLs for images
- Improved documentation navigation with collapsible sidebar sections
- Enhanced logo visibility in documentation header
Fixed¶
- Logo display issues in documentation
- Asset paths for GitHub Pages compatibility
0.1.0 - 2026-05-05¶
Added¶
- Initial release of LongProbe
- Golden question management with YAML configuration
- Three match modes: ID, text, and semantic similarity
- Baseline storage and regression detection using SQLite
- CLI with multiple output formats (table, JSON, GitHub Actions)
- Pytest plugin integration for test suite integration
- Retriever adapters:
- ChromaDB direct adapter
- Pinecone direct adapter
- Qdrant direct adapter
- LangChain retriever wrapper
- LlamaIndex retriever wrapper
- HTTP/REST API adapter with configurable mapping
- Question generation with LLM support (OpenAI, Anthropic, Gemini, Ollama)
- Auto-capture functionality to build golden sets from live retrievers
- Watch mode for continuous testing during development
- GitHub Actions reusable workflow for CI/CD integration
- Comprehensive test suite with 172 unit tests
- Rich CLI output with color-coded tables and progress indicators
- Diff reporting showing exactly which chunks were lost/gained
- Per-question recall scoring with configurable thresholds
- Support for question tags and metadata
- Environment variable expansion in configuration files
- Document parsing for multiple formats (TXT, MD, CSV, JSON, PDF, DOCX, PPTX, XLSX)
Documentation¶
- Comprehensive README with quick start guide
- CLI reference documentation
- Python API examples
- Configuration reference
- Match mode explanations
- GitHub Actions integration examples
- Example configuration files and golden sets