CLI reference¶
Use longprobe COMMAND --help for implemented options. Default paths are goldens.yaml and longprobe.yaml.
| Command | Purpose |
|---|---|
init |
Create example files, preserving existing files unless --force |
check |
Evaluate selected questions with shared scoring/comparison rules |
capture |
Query supplied questions and select evidence; --auto creates drafts |
generate PATH |
Generate questions from documents; auto-capture creates draft cases |
edit |
Review status, question text/mode, approve drafts, and save |
baseline save |
Fresh approved-only evaluation and explicit policy-gated promotion |
baseline list |
List snapshots at --db-path |
baseline delete --label LABEL |
Confirm deletion of a snapshot at --db-path |
diff --baseline LABEL |
Compare a live run with an explicit reference |
diff LABEL_A LABEL_B |
Compare stored target B against reference A |
report REPORT_PATH |
Render saved JSON without configuration or retrieval |
explain QUESTION_ID |
Rerun evaluation and explain missing evidence |
watch |
Poll the golden file and rerun after changes |
Evaluation and output¶
longprobe check --goldens tests/goldens.yaml --config longprobe.yaml
longprobe check --threshold 0.9 --top-k 10
longprobe check --tag critical --tag billing
longprobe check --include-drafts
longprobe check --baseline release-a
longprobe check --no-compare
longprobe check --output json > result.json
longprobe report result.json --output github
longprobe baseline save --label latest --output json
Multiple tags require all specified tags. Empty suites and empty selections fail. Drafts are excluded by default and cannot be promoted. Global threshold overrides do not override explicit per-question settings.
check --baseline LABEL requires the label to exist. Optional automatic latest absence is visible in the report; no historical safety is implied. --no-compare deliberately disables comparison and cannot be combined with --baseline.
JSON and exit codes¶
check, diff, baseline save, and report support table, json, and github output. Parsed JSON-mode invocations emit one complete JSON object on stdout, including handled command errors. Notices and diagnostics go to stderr.
Success/evaluation payloads include existing aggregate/result fields, schema_version, recorded contracts/policy, quality, baseline_identity, comparison, and a status. diff remains an alias for the comparison object for automation compatibility. Error envelopes contain schema_version, status: error, and error.code/error.message.
| Code | Meaning |
|---|---|
0 |
Successful operation, or an explicitly nonblocking completed quality failure |
1 |
Completed evaluation/comparison or baseline quality failure |
2 |
Invalid input/setup, empty selection, missing explicit reference, or incompatible comparison |
3 |
Operational or unexpected execution failure |
A failed quality decision stays recorded as failed even when scoring.fail_on_regression: false allows a check to exit 0. That setting does not relax setup/execution failures or incompatible comparisons. Explicit diff gates quality; baseline promotion always applies its own policy.
Parser-level usage errors that prevent establishing a valid invocation use normal CLI stderr and code 2, without promising a JSON object. Watch and interactive authoring remain human-oriented. report renders completed data and does not re-enforce the saved quality verdict; malformed input still fails.
Authoring¶
longprobe capture --question "What is the refund policy?" --auto
longprobe capture --questions-file questions.txt --match-mode id
longprobe generate ./documents --num-questions 20 --auto-capture
longprobe edit
Install generation dependencies and configure the provider/model before generation. Review required evidence in YAML; the editor does not expose every field. Automatic capture is candidate evidence, not independently verified truth. Generating questions does not guarantee complete document coverage.
Question-list export is separate:
longprobe generate ./documents --output questions.yaml
longprobe generate ./documents --output questions.yaml --overwrite
--overwrite requires --output, and output mode cannot be combined with --auto-capture. Invalid existing golden files fail preflight and are never silently reset.
Baselines and watch¶
Saving another label does not update literal latest. New reports enforce contract compatibility; see baseline management for old reference migration and intentional rebaselining.
Watch runs after golden-file changes, not on startup or configuration/remote index changes. Failed iterations are reported and watching continues. Explain lists missing evidence and retrieved IDs; it does not calculate rank or distance deltas.