Design Decisions

Compression Improves Reasoning

The central thesis: compressed, structured context produces better model reasoning than raw conversation dumps. When a model receives 60K tokens of curated summaries organized by topic, it performs better than when it receives 60K tokens of raw chat history that includes noise, repetition, and irrelevant tangents.

Conversation text has extremely low information density. Most turns contain phatic exchanges, restated context, debugging dead ends, and scaffolding that served a purpose in the moment but adds noise later. Compaction strips this while preserving the semantic core.

The benchmarks confirm it: virtual-context answers 95 of 100 LongMemEval questions correctly, against 33 of 100 for a full-context baseline on the same reader model.

Two-Tagger Rationale

The embedding tagger is safe. It runs locally, costs nothing per call, executes in milliseconds, and produces deterministic results. If the LLM tagger fails or hallucinates tags, the embedding tagger still anchors retrieval.

The LLM tagger is rich. It understands context, catches implicit topics (“I’m worried about the deadline” → project-timeline), and generates natural vocabulary.

Running both costs one small-model call per turn but provides both retrieval reliability and vocabulary richness. The split is the default: retrieval.inbound_tagger_type is embedding out of the box and can be set to llm to route the inbound path through the LLM tagger instead. To cut cost further, tag_generator.type: "keyword" replaces LLM tagging of completed turns with keyword extraction; retrieval still works, with less vocabulary richness.

Sync-First Processing

Inbound must be synchronous. The model needs context before it can respond. Tagging, retrieval, and assembly must complete before the request is forwarded.

Completion can be asynchronous. After the response streams back, the background thread handles response tagging, index updates, compaction, and fact extraction. The user is already reading the response.

Each new request waits for the previous turn’s completion work on the same conversation. Users take seconds between turns, so the wait is rarely noticeable. The inbound tagger is an embedding model by default precisely to keep that synchronous path off the LLM.

The REST path has no completion signal of its own. A hosting service calls prepare before its model call and ingest after it, as two separate requests, and the ingest may arrive late or never. That asymmetry is why prepare persists the user half of the turn immediately, why ingest reconciles rather than appends, and why every write is idempotent against redelivery. The proxy gets completion for free by sitting on the response stream; the REST surface has to earn the same guarantees through reconciliation.

Tag Preservation Through Compaction

When segments are compacted, their tag assignments are preserved: the summary inherits the tags of the original turns, so a topic stays findable under the same name after compression.

The tag vocabulary is not frozen, though. Overloaded tags are split into narrower subtags, near-duplicate tags are canonicalized through the alias table, and queries against an old name still resolve to its successors. Stability comes from the alias chain, not from immutability.

This is why tag quality matters at assignment time: tags are the primary index. A bad tag persists until the vocabulary lifecycle corrects it, and everything stored under it in the meantime inherits the damage.

Chain Collapse Over Truncation

Many systems handle tool-heavy conversations by truncating old tool results. This is lossy and unpredictable.

Chain collapse replaces tool exchanges with compact stubs that include a restore reference. The model can see that information exists and recover it on demand via vc_restore_tool. Nothing is lost; it’s just paged out.

This mirrors virtual memory: pages are swapped to disk and faulted back in on access.

No SDK Dependencies

Virtual-context operates as a proxy, not a library. Point your API calls at 127.0.0.1:5757 instead of api.anthropic.com, and everything works. No client changes, no framework lock-in.

Format Detection Over Configuration

The proxy auto-detects whether a request uses the Anthropic, OpenAI Chat, OpenAI Responses, or Gemini API format. Detection uses structural signals (field names, URL paths, model name prefixes) rather than content heuristics.

Greedy Set Cover for Assembly

Segments sorted by retrieval score, added in order until budget is full. Optimal in practice (within a constant factor of the theoretical best) and fast (single pass, O(n)).

Demand-Paged Context

OS Virtual Memory                    virtual-context
-----------------                    ---------------
Physical RAM            =  Context window
Disk / swap             =  SQLite (segments, facts, summaries)
Page tables             =  TurnTagIndex (per-turn topic tracking)
Page faults             =  vc_expand_topic (demand paging)
Page eviction (LRU)     =  Compaction (topic-aware eviction)
Working set             =  Active paging depths per tag
Address space           =  Full conversation history (unbounded)
Memory protection       =  Bleed gating (topic-shift isolation)

The model sees recent turns at full fidelity, retrieved summaries for relevant topics, topic hints listing what else is available, and tool definitions for paging in more. The engine manages the page table, page frames, and eviction.

Fact Supersession Over Versioning

Facts use supersession (new fact invalidates old) rather than versioning (keep all versions). “User lives in LA” replaces “User lives in NYC.”

For cases where history matters, the fact’s when field and underlying conversation segments preserve the timeline. Supersession cleans up the active fact set, not the historical record.