Engine Internals

Compactor

Compaction converts raw conversation turns into compressed segments. It begins when context window fill crosses the soft threshold and is forced at the hard threshold. It selects uncompacted turns outside the protected window, groups them by tag overlap, and calls the summarization model to produce condensed segment summaries. Each summary preserves the tag set, turn range and token count of the original, and every non-_general tag on a just-compacted segment gets its tag summary materialized at commit.

Compaction is incremental. A watermark tracks which turns have been processed, so only turns above it are candidates. Protected recent turns are never compacted, keeping the most recent context at full fidelity. In multi-worker deployments each compaction runs as a leased, fenced operation, so a stalled worker cannot overwrite a takeover.

There is no second summarize-the-summaries tier. The separately named deep_compaction_ratio is a payload-side filter in the proxy’s message rewriting: turns far enough below the compacted boundary are dropped from the outgoing payload entirely rather than stubbed, because the segment summaries already cover them. Stored data is never affected.

The compactor runs on a background pool after on_turn_complete, never blocking the response path.

compaction:
  soft_threshold: 0.70        # begin compaction at 70% fill
  hard_threshold: 0.85        # force compaction at 85% fill
  protected_recent_turns: 6   # recent turns exempt from compaction
  min_summary_tokens: 200
  max_summary_tokens: 2000

Code Mode

compaction.code_mode is on by default, so this reshapes summarization and fact extraction in every deployment, including ones that are not about code.

With it on, segment summaries and tag-summary rollups use coding-oriented prompt templates rather than generic ones, and fact extraction changes what it keeps:

• Investigatory actions are excluded. That the assistant ran the tests or opened a file is process noise, not knowledge, and is not extracted.
• Conclusions are kept: findings, decisions, configuration values, bugs and their fixes, tool and library choices, and what was built or changed.
• Facts are framed about the thing, not the assistant. “The endpoint now supports sorting”, not “the assistant added sorting”.
• Code references are emitted alongside facts: a deduplicated list of the concrete files, functions and classes materially discussed in a turn, carried through compaction and rendered next to summaries, so a question about which files a topic touched survives compression.

Setting code_mode: false restores the generic prompts and stops emitting code references. Nothing else changes.

Cache-Aware Payload Flushing

Compacting storage and rewriting the outgoing payload are two separate steps, tracked by two watermarks: how far compaction has covered the conversation in storage, and how far the outgoing payload reflects it. Rewriting the payload breaks the byte-identical prefix providers use for prompt-cache hits, so applying compaction to the payload immediately trades cache discounts for compaction savings.

With compaction.defer_payload_mutation enabled, a flush gate decides per request:

• Warm cache, where the gap since the previous request is under the TTL, with pending compaction work: payload mutations are deferred. The prefix stays byte-identical and cache hits keep flowing while storage-side compaction proceeds normally.
• Cold cache, where the TTL has passed and the provider cache has expired anyway: the flush watermark catches up and the payload is rewritten, so compaction savings arrive exactly when the cache discount was already gone.
• Safety valve: if a warm-deferred payload exceeds the hard threshold, the gate force-flushes and re-runs the skipped mutations, so deferral can never push the payload past budget.
• After a restart the cache age is unknown and treated as warm, so the first request never mutates blindly.

Deferral conflicts with the fill pass, which rewrites the payload every request and destroys the prefix deferral preserves; the proxy logs a configuration-conflict warning when both are enabled. The gate logs a FLUSH_GATE: line on every decision and DROP-COMPACTED: when compacted turns leave a payload.

compaction:
  defer_payload_mutation: false   # true = preserve the prompt-cache prefix while warm
  flush_ttl_seconds: 300          # cache considered cold after this idle time

Cache Breakpoints

The boundary the flush gate protects is set explicitly. On the Anthropic format only, since the marker is an Anthropic Messages API construct, an ephemeral cache breakpoint is inserted so that stable content ends inside the cached region and the injected context block sits after it.

When injecting into the system prompt, the breakpoint is placed on the last stable block of the client’s own system content, and the fresh context block is appended after it. When injecting into the last user message, the breakpoint lands on that message’s last existing content block, with the same effect.

If you set your own cache breakpoints, virtual-context takes them over on the system prompt. Any cache_control markers you placed on system blocks are stripped and re-placed at the boundary described above. This is not a merge and it is not configurable: the injected context block changes on every turn, so it must fall outside the cached prefix, and leaving a client-set marker in place would put it inside.

This is automatic and not configurable. The client’s stable prompt and history stay cacheable across turns even though volatile retrieved context is appended to every request; the flush gate then decides when it is worth breaking that prefix to apply compaction.

Tagging Pipeline

Tags are the primary indexing mechanism. Every turn is tagged on two paths, at two different moments.

The inbound embedding tagger runs on the user message before the model responds. It computes vector similarity against existing tags with a local embedding model and assigns the closest matches above a threshold. It is fast and deterministic, and it is what makes retrieval safe: if a topic came up before, its tag is found even when the user phrases it differently.

The LLM turn tagger runs on the background path with the full completed turn. It produces richer vocabulary and catches nuances the embedding tagger misses, with no latency pressure because the response has already streamed. Every turn ends up LLM-tagged; the inbound embedding tags exist so retrieval has tags before the turn completes.

The context bleed gate is an embedding-similarity check that decides whether preceding turns are related enough to include in the tagger prompt. When similarity falls below the threshold, that context is left out, so an abrupt topic shift is not tagged with the previous topic’s vocabulary.

Tag splitting handles tags that grow too large: the engine splits them into subtags and registers aliases, so queries against the original name still find segments under the split subtags. Alias resolution reads the durable store.

TurnTagIndex

The in-memory index of per-turn tag assignments. Each entry records the turn number, the turn’s tag list and primary tag, the backing canonical turn ID, and session date, sender and fact signals. It answers lookback queries such as which tags were active in the last few turns, which retrieval uses to compute the working set, and it is rebuilt from canonical turn rows on session restore.

Segmenter

The segmenter splits compacted output into discrete segments, each carrying a tag set inherited from the compacted turns, a token count, the summary text, and the range of original turns it covers. Segments are the unit of summary storage.

Retrieval

Retrieval decides which stored topics are relevant to the current query. The ranked unit is the tag: three signals each produce a ranked list of candidate tags, the lists are fused, and segments are then fetched for the winning tags.

The three rankings are combined by Reciprocal Rank Fusion with configurable weights. The query vector can optionally be blended with recent conversational context, guarded so the blend cannot demote a tag below its bare-query score.

Active tag skipping: tags from the most recent turns are skipped during retrieval, because their content is already present in the raw history inside the context window and retrieving them would spend budget on duplicates.

Dampening and Boosts

After fusion, three adjustments run, all on by default.

• Gravity dampening halves the embedding score of tags with no BM25 support at all, so a purely semantic match cannot outrank tags with corroborating keyword evidence.
• Hub dampening penalizes tags whose segment count exceeds the 90th percentile of the distribution, with query tags exempt, preventing catch-all topics from dominating every retrieval.
• Resolution boost promotes fact-bearing tags, so topics with extracted structured facts rank ahead of equally scored topics without them.

Reserved seats, off by default, can force the top embedding-only candidates into the fused result for queries where the embedding signal is the only one that finds the right topic.

Assembly

The assembler constructs the <virtual-context> block injected into the system prompt. Alongside retrieved summaries it renders extracted facts, the requester’s person card, and, in group conversations with speaker features enabled, the speaker roster.

Remaining budget is filled by greedy set cover: segments are added in score order until the budget is exhausted, and a segment that would overflow is skipped in favour of a smaller one that fits. The budget is a fraction of the total context window, and the assembler counts tokens precisely including overhead for tags, separators and metadata lines, so injected context never exceeds it.

Context Hints

After compaction the assembler injects a structured <context-topics> block: a budgeted topic list with per-topic descriptors, a line naming how many topics exist in total, and guidance on paging more detail in through the tools. The hint is cached and pre-warmed at compaction commit, so the first request after a compaction does not pay the rebuild cost.

Cross-Channel Context

A server-style community can deliberately unify all of its channels into one stored conversation, giving the assistant one continuous memory of the whole community. The transport stays channel-local, though: the payload the client sends contains only the current channel’s messages, so the model’s raw window is blind to what just happened in a sibling channel.

assembly.protected_window_db_source closes that gap. The default off keeps the protected window payload-only with no store read. Setting merge additionally reads recent turns from the durable store and merges them into model-visible context as a quoted block marked with verified provenance, rendered oldest to newest with speaker attribution, so cross-channel activity becomes visible without waiting for compaction and retrieval to catch up.

Merging deduplicates conservatively: only an exact same-channel copy suppresses the stored row. A matching turn retained from another channel’s history is not proof the current payload shows it, and unknown-channel legacy rows fail open to a harmless duplicate rather than silently hiding the only cross-channel copy from the model. The quoted block is explicitly reference material: messages inside it carry no instruction authority, only the current requester’s own row does.

assembly:
  protected_window_db_source: "off"   # or "merge" for unified multi-channel conversations

Session Restore and Recovery

A conversation’s live state, meaning the tag index, working set and watermarks, has to survive process restarts and identity rebinds.

• Session-state snapshots persist per-conversation state through a provider, Redis-backed in the proxy, and rehydrate it on the next request, so a restart or redeploy does not reset the working set or watermarks.
• Self-hydrate on rebind: an engine constructed for an explicit conversation ID, for example after an attach redirects identity, hydrates from that conversation’s snapshot instead of starting cold.
• Markers from canonical rows: the restore markers that make rehydration possible derive from canonical turn rows, and an admin backfill rebuilds them for conversations predating the mechanism.
• Store-backed recovery: when a client truncates its own history to manage its window, the engine detects the truncation and restores the missing context from durable storage, so the upstream payload reads as if nothing was lost.

Temporal Recall Modes

Time-scoped recall takes a mode that shapes what a temporal query returns.

Date ranges are resolved by the temporal resolver, turning relative phrases into absolute dates, and executed against stored session dates, so answers about when something happened come from date arithmetic rather than model recall.

Token Counter

The counter is image-aware: for base64-encoded images it uses dimension-based costing, matching how providers price vision input, rather than counting base64 characters. Without this, image-heavy conversations are massively overestimated. The fallback chain runs anthropic to tiktoken to estimate depending on what is installed.

Fact Extraction

Facts are structured subject | verb | object triples carrying a status (active, completed, planned, abandoned, recurring), an absolute date where known, a location where applicable, and a type: personal, experience or world.

Supersession: when a new fact contradicts an existing one, the old fact is superseded. Moving from one city to another invalidates the earlier residence fact. Supersession is a dedicated model-backed checker with its own provider and model configuration, invoked from the compaction pipeline after extraction.

Querying: structured lookups filter by subject, verb and status. Verb matching expands through curated synonym clusters plus embedding similarity against the verbs actually present in the store, so a query for one verb also matches its near-equivalents. Facts can additionally be ranked by dense similarity between the query and stored fact embeddings.

Chain Collapse

Tool-heavy conversations produce large tool-call and tool-result pairs that dominate the context window. Chain collapse identifies consecutive pairs, writes the full content to durable storage, replaces the original messages with compact stubs carrying a restore reference, and lets the model recover any collapsed chain at full fidelity through a restore tool.

This is lossless: nothing is discarded, it is moved to cheaper storage with a pointer left behind. Orphan stripping handles ranges that begin or end mid-exchange, removing a trailing call without its result or a leading result without its call, so message structure stays valid.

Media Compression

Base64-encoded images in messages are decoded, resized, and replaced in the payload, with the original written to disk for recovery. Because providers cost vision input by dimensions rather than by base64 length, the token savings are modest, but the bandwidth and latency improvements are substantial.

Monitor

The monitor recalculates context window fill after each turn as raw history plus injected context over the window size. Crossing the soft threshold signals the compactor; crossing the hard threshold forces immediate compaction. The fill level is exposed through the dashboard and telemetry.

Tool Loop

The engine exposes eight tools to the model on the proxy tool loop. This surface is distinct from the MCP server’s eight tools; the two lists overlap but are not the same.

Anti-repetition tracks which segments have already been shown across rounds and suppresses duplicates; if the model starts searching in circles, strategy hints suggest alternatives. Empty streak detection injects similar hints when consecutive calls return nothing.

Channel-scoped quote search. The quote tool accepts an optional channel argument, either a channel name with or without a leading hash, or a stored channel ID. When passed, both the lexical and semantic branches restrict matches to rows whose stored channel provenance matches, and rows with empty provenance fail closed out of scope. Omitted, the whole conversation is searched; the tool schema tells the model to scope only when the question is explicitly about one channel.