In modern LLM applications, state management is often the most significant engineering bottleneck. Traditional stateless APIs force developers to pass the entire conversation history back and forth on every turn. As context windows grow, this approach introduces unsustainable token usage, high costs, and unacceptable user-facing latency. To build premium, responsive, and truly intelligent AI companions and tools, we must implement robust persistent memory architectures.
The Stateless Latency Penalty
When query latency climbs above 1.5 seconds, user engagement plummets. In conversational systems, a significant portion of this latency is driven by prompt processing (prefill) times for large context structures. If a system passes a raw chat history of 10,000 tokens on every message, the LLM engine must re-evaluate those tokens repeatedly. This stateless approach scales linearly in cost and quadratically in process complexity.
"State management isn't just about preserving variables; it is the infrastructure that turns a model call into a production-ready application."
Architecting a Multi-Tier Memory System
To deliver sub-200ms latency patterns, we need a tiered memory architecture analogous to traditional hardware caches. Instead of relying solely on the LLM's dynamic context window, we divide memory into three distinct, structured layers:
- L1: Local Context Buffer (Working Memory) — The most recent 3–5 dialogue turns, kept instantly available in the local application state for high-fidelity responses.
- L2: Semantic Cache (Short-term Recall) — An in-memory vector database containing recently processed user intents and model decisions to bypass model queries entirely for repeated questions.
- L3: Vector Database & Graph Storage (Long-term Memory) — A persistent vector store (such as Supabase pgvector) coupled with metadata tags, providing semantic search across the entire user history.
Implementing Semantic Caching
Before any prompt is dispatched to an LLM provider, we vectorize the incoming user query and run a cosine similarity query against our local semantic cache. If a match is found with a confidence threshold above 0.94, we serve the cached response instantly. This saves downstream compute cycles and cuts response times down to under 50ms.
// Example schema setup in Supabase pgvector
CREATE TABLE semantic_cache (
id uuid DEFAULT gen_random_uuid() PRIMARY KEY,
query_text text NOT NULL,
embedding vector(1536),
response_text text NOT NULL,
created_at timestamp WITH time zone DEFAULT timezone('utc'::text, now())
);Dynamic Context Window Compaction
For dialogue histories that exceed the active context window size, we run background summation routines. When the conversation buffer approaches 80% of our maximum target size, an asynchronous process triggers: it isolates the oldest 30% of the conversation, calls a lightweight LLM to summarize the core topics and entity relations, saves these summaries as a persistent profile in L3, and prunes the raw message history.
By constantly compressing the conversational baseline and retrieving relevant facts on-demand using vector similarity searches, we maintain an active context window that is compact, highly relevant, and extremely fast to process.