Architecture & Internals

LongCache combines local embedding, a process-local FAISS index, and a response storage backend.

                     ┌─────────────────────────────┐
                     │       Incoming Query        │
                     │  "How do I cancel my plan?" │
                     └──────────────┬──────────────┘
                                    │
                                    ▼
                     ┌─────────────────────────────┐
                     │        LocalEmbedder        │
                     │   BAAI/bge-small-en-v1.5    │
                     │  (384-dim, L2-normalized)   │
                     └──────────────┬──────────────┘
                                    │
                                    ▼
                     ┌─────────────────────────────┐
                     │       SimilarityIndex       │
                     │  FAISS (IndexFlatIP / HNSW) │
                     └──────────────┬──────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    │ Cosine Similarity ≥ Threshold?│
                    └───────┬───────────────┬───────┘
                       YES  │               │ NO
                            ▼               ▼
                   ┌────────────────┐  ┌──────────────────┐
                   │   Cache HIT    │  │    Cache MISS    │
                   │ Return Payload │  │ Call LLM API     │
                   └────────────────┘  │ Save to Backend  │
                                       └──────────────────┘

---

1. Local Embedder

LongCache includes a singleton embedding manager LocalEmbedder using sentence-transformers and BAAI/bge-small-en-v1.5.

• Vector Dimension: 384 dimensions.
• L2 Normalization: Vectors are unit-normalized upon generation so that inner product equals cosine similarity ($A \cdot B = \cos(\theta)$).
• Zero API Call: Vector computations happen locally after model weights have been downloaded. Measure embedding latency on your hardware.

2. FAISS Similarity Index

LongCache utilizes FAISS (Facebook AI Similarity Search):

• Flat Index (IndexFlatIP): Exhaustive inner product search. Best for collections up to ~50,000 queries per namespace where 100% exact recall is required.
• HNSW Index (IndexHNSWFlat): Hierarchical Navigable Small World graphs for ultra-large datasets (100k+ entries) where logarithmic search complexity is needed.

3. Composite Cache Scoping (Anti-Poisoning)

In production, returning an answer generated by gpt-4o for a prompt that was designed for claude-3-5-sonnet (or varying system prompts/temperatures) can break downstream logic.

LongCache prevents cross-context contamination by compounding metadata into composite keys:

$$\text{Composite Scope} = \text{Hash}(\text{namespace} + \text{model} + \text{system_prompt} + \text{temperature})$$

Entries are strictly isolated within their corresponding scope.

4. Eviction & TTL Management

1. TTL (Time to Live): Each cache entry has an explicit timestamp. Expired entries are skipped during lookup. Search continues to another eligible candidate, and background maintenance removes expired storage entries.
2. LRU / Size Eviction: When a namespace reaches max_entries, oldest accessed entries are purged.

Index maintenance in v0.1.1

Lookup skips candidates outside the requested namespace, expired entries, and missing backend entries. Rebuilds load all live namespaces and preserve the configured flat or HNSW index. HNSW is approximate and can miss eligible neighbors; measure recall for your workload. Redis persistence does not synchronize FAISS indexes between processes.