LongProbe

Sub-second RAG regression testing for production pipelines

---

Overview

"Did my last commit break retrieval?" — now you know in seconds.

LongProbe is a sub-second RAG regression harness. Define your Golden Questions once, run longprobe check on every commit, and get an exact diff of which document chunks were lost in your latest change — before your users notice.

Think pytest --watch for your RAG pipeline.

Why LongProbe?

Every RAG developer faces the same silent killer: you refactor chunking strategy, upgrade LangChain, or add a new document — and your retrieval silently degrades. DeepEval and RAGChecker are heavyweight evaluation frameworks meant for batch analysis, not fast regression checks in a dev loop.

LongProbe gives you instant feedback:

• ⚡ Sub-second checks on small golden sets
• 🔍 Exact diffs showing which chunks were lost/gained
• 📊 Recall scores with per-question breakdown
• 💾 Baseline tracking to catch regressions over time
• 🧪 pytest integration for existing test suites
• 🔌 Pluggable adapters for any vector store

Quick Example

# Install
pip install longprobe

# Initialize
longprobe init

# Define your golden questions in goldens.yaml
# Configure your vector store in longprobe.yaml

# Run checks
longprobe check

# Save baseline
longprobe baseline save --label v1.0

# Compare after changes
longprobe diff --baseline v1.0

Part of the Long Suite

LongProbe is part of the EnDevSols Long Suite of RAG tools:

• LongParser - Document ingestion and chunking
• LongTrainer - RAG chatbot framework
• LongTracer - Hallucination detection
• LongProbe - Retrieval regression testing ← You are here

Together they cover the full RAG pipeline from ingestion to production monitoring.

Features

Core Capabilities

• ⚡ Sub-second checks on small golden sets
• 📋 Golden Questions + Required Chunks defined in simple YAML
• 🔍 Three match modes: exact ID, text substring, semantic similarity
• 📊 Recall Score with per-question breakdown
• 🔄 Regression diff: exactly which chunks were lost/gained
• 💾 SQLite baseline store: compare against any previous run

Developer Experience

• 🧪 pytest plugin: integrate into existing test suites
• 🖥️ Beautiful CLI with Rich tables, JSON, and GitHub Actions output
• 👀 Watch mode: auto re-run on file changes
• 🏗️ CI/CD ready: fails pipeline on regression

Integrations

• 🔌 Pluggable adapters: LangChain, LlamaIndex, Chroma, Pinecone, Qdrant
• 🌐 HTTP adapter: test any RAG API
• 🐍 Python API: programmatic access to all features

Next Steps

• Quick Start

  ---

  Get up and running in 5 minutes

  Quick Start

• User Guide

  ---

  Learn how to define golden questions and configure LongProbe

  User Guide

• Demos

  ---

  See LongProbe in action with live demos

  View Demos

• API Reference

  ---

  Detailed API documentation for Python integration

  API Docs

Community & Support

• GitHub Issues: Report bugs or request features
• Discussions: Ask questions and share ideas
• Contributing: Contribution guidelines

License

LongProbe is released under the MIT License.