Agent Memory
Architecture
Context windows are finite. Agent work isn't. Every LLM agent — from coding assistants to autonomous researchers — hits the same wall: how do you give an agent durable, searchable, editable memory that persists across sessions without drowning it in tokens?
This document presents an optimized flat-file + hybrid search architecture, benchmarked against alternatives and deployed in production. The industry is converging on a single insight: the best memory is text the agent can read AND write.
The Optimized Memory Stack
A flat-file hierarchy paired with hybrid search gives agents both structure and retrieval. No databases, no graph stores — just files the agent owns and a search index it can query.
TOOLS.md, SOUL.md
0.6 / 0.4 weighting
context window
docs, entities
- MEMORY.md — curated pointers
- USER.md — user preferences
- TOOLS.md — capabilities ref
- SOUL.md — agent persona
- Daily logs (today + yesterday)
- Reference documents
- Entity files
- Session transcripts
- memory/YYYY-MM-DD.md
- memory/reference/*.md
- memory/entities/*.md
- Pre-compaction flush
- sqlite-vec for vectors
- BM25 full-text index
- Local embeddings (Qwen3)
- Hybrid score merging
What the Research Shows
Five findings from benchmarks, papers, and production systems that shaped this architecture.
File-Only Memory Competes
The Letta benchmark showed that filesystem-only memory achieves 74% on LoCoMo, a major long-conversation memory benchmark — competitive with far more complex systems using vector databases and graph stores.
74% LoCoMoIndustry Convergence
Anthropic's Memory Tool, Claude Code's CLAUDE.md, and ChatGPT's persistent memory all independently converged on the same pattern: file-based memory the agent can read and write.
Hybrid Search Wins
Combining BM25 keyword search with vector similarity consistently outperforms either approach alone. Keyword search catches exact terms vectors miss; vectors capture semantic meaning keywords can't.
BM25 + VectorMemory Consolidation
The SimpleMem paper demonstrated that memory consolidation — periodically summarizing and restructuring stored memories — yields a 64% improvement over Claude-Mem's raw append approach.
+64% vs Claude-MemReadable + Writable
The industry is converging on a principle: the optimal memory format is text the agent can read AND write. Not opaque embeddings. Not locked databases. Plain text files with semantic search layered on top.
Plain text winsThe Optimal Configuration
What we benchmarked and found works best. Every parameter here was tested against alternatives.
Hybrid Search Weights
0.6 vector · 0.4 BM25 text. This ratio consistently outperformed pure vector or pure keyword search across diverse query types.
Score Threshold
minScore: 0.25 — low enough to catch tangentially relevant memories, high enough to filter noise. Tuned against real-world recall queries.
Local Embeddings
Qwen3 Embedding 0.6B — 1024 dimensions, runs locally via LM Studio. Free, fast, competitive with commercial embedding APIs. Zero API cost.
Index Workspace Files
Don't just index memory/ — adding TOOLS.md, USER.md, SOUL.md boosted recall scores 87–113%. The agent's own config files are some of its most-needed context.
Session Transcript Indexing
Index past session transcripts for cross-session recall. Questions asked weeks ago become searchable, enabling true long-term conversational memory.
sqlite-vec Acceleration
Use sqlite-vec for fast vector queries. Single-file database, no server process, sub-millisecond lookups. Pairs naturally with the flat-file approach.
MMR: Disable for Small Corpora
Maximal Marginal Relevance (diversity dedup) adds overhead with negligible benefit on corpora under ~500 files. Enable only when you have enough content for redundancy to matter.
Temporal Decay: Disable
Time-based score decay sounds theoretically appealing but produces negligible improvements on small corpora. Skip it until your corpus grows past ~500 files.
Implementation Guides
Four paths to production-grade agent memory — from generic frameworks to specific tools.
Memory Hierarchy
Memory Hierarchy:
├── MEMORY.md # Curated long-term pointers (always loaded)
├── memory/YYYY-MM-DD.md # Daily logs (append-only, read today+yesterday)
├── memory/reference/ # Detailed docs (loaded on demand via search)
└── memory/entities/ # Optional: one file per key entity
Core Principles
- Use any embedding model — OpenAI
text-embedding-3-small, Nomic Embed, or a local GGUF model. The architecture is embedding-agnostic. - Implement hybrid search: combine vector similarity with BM25 keyword matching for best-of-both retrieval.
- Merge scores with:
finalScore = 0.6 * vectorScore + 0.4 * textScore - Agent reads memory at session start, writes decisions and facts during the session.
- Pre-compaction flush: always remind the agent to save durable memories before context compaction discards working memory.
- Just-in-time context retrieval — don't load everything. Search on demand, read targeted snippets.
Score Merging Implementation
function hybridSearch(query, corpus) {
const vectorResults = vectorSearch(query, corpus);
const bm25Results = bm25Search(query, corpus);
// Merge with weighted combination
const merged = mergeResults(vectorResults, bm25Results, {
vectorWeight: 0.6,
textWeight: 0.4,
minScore: 0.25,
});
return merged.filter(r => r.score >= 0.25);
}
4 Complementary Memory Systems
Claude Code (the CLI/IDE agent) ships with four layered memory systems that work together — from developer-authored project context to fully autonomous learned memory.
1. CLAUDE.md — Project Memory
# Priority hierarchy (highest → lowest):
~/.claude/CLAUDE.md # User-level (all projects)
./CLAUDE.md # Project root
./.claude/CLAUDE.md # Project config dir
../CLAUDE.md # Parent directory (inherited)
- Written by the developer, read by main agent + all subagents.
- Put: project conventions, architecture decisions, coding standards, key context.
- Keep focused — conventions not tutorials. Reference files, don't inline large content.
- Higher-priority files override lower ones in the hierarchy.
2. Auto-Memory
- Claude Code automatically saves useful context between sessions — things it learns about your codebase, preferences, and patterns.
- Per-project, per-user scope. No cross-project leakage.
- Main agent only — subagents don't see auto-memories.
- Managed via the
/memorycommand in Claude Code CLI.
3. /memory Command
- Opens an editor to manually review and edit Claude's stored memories.
- Use to curate what Claude remembers about your project — delete noise, add important context, restructure.
- Think of it as direct access to Claude's "notebook" for this project.
4. Subagent Persistent Memory (v2.1.33+)
---
name: code-reviewer
description: Reviews code for quality and best practices
tools: Read, Write, Edit, Bash
model: sonnet
memory: user
---
You are a code reviewer.
Update your memory with patterns and conventions you discover.
Memory Scopes
| Scope | Location | Git-tracked | Shared | Best For |
|---|---|---|---|---|
user |
~/.claude/agent-memory/<name>/ |
✕ No | ✕ No | Cross-project knowledge |
project |
.claude/agent-memory/<name>/ |
✓ Yes | ✓ Yes | Team-shared project knowledge |
local |
.claude/agent-memory-local/<name>/ |
✕ No | ✕ No | Personal project knowledge |
- First 200 lines of
MEMORY.mdinjected into subagent system prompt at startup. - Agent has Read/Write/Edit access to its memory directory.
- If MEMORY.md exceeds 200 lines, agent moves details to topic files in its memory dir.
Best Practices
- Keep CLAUDE.md focused: conventions, not tutorials. It's a cheat sheet, not a manual.
- Use
memory: userfor subagents that should learn across projects — code reviewers, style enforcers, test writers. - Combine skills (static knowledge) + memory (dynamic, learned over time) for the most capable agents.
- Reference files instead of inlining large content — keep CLAUDE.md lean.
- Update CLAUDE.md as the project evolves — treat it as living documentation, not a write-once config.
Memory Tool
from claude_agent_sdk import query, ClaudeAgentOptions, AgentDefinition
# Enable the Memory Tool (file-based, you own storage)
message = client.messages.create(
model="claude-opus-4-6",
tools=[{"type": "memory_20250818", "name": "memory"}],
messages=[...]
)
Subagent with Persistent Memory
# Subagent with persistent memory
agents={
"researcher": AgentDefinition(
description="Research agent with persistent memory",
tools=["Read", "Write", "Glob", "Grep"],
model="sonnet",
memory="user", # Persists across projects
),
}
Context Editing
# Context Editing (84% token reduction in 100-turn tests)
response = client.beta.messages.create(
model="claude-opus-4-6",
context_management={
"edits": [
{"type": "clear_tool_uses_20250919"},
{"type": "clear_thinking_20251015",
"keep": {"type": "thinking_turns", "value": 2}}
]
}
)
Key Concepts
- Memory Tool: Claude makes CRUD calls against your storage backend — files, a database, or S3. You control persistence.
- CLAUDE.md: Project-level persistent memory files. Loaded automatically, editable by the agent.
- Subagent memory scopes:
user(cross-project),project(team-shared),local(personal per-agent). - Context Editing + Memory Tool together: +39% performance improvement in complex multi-turn sessions.
Configuration
// ~/.openclaw/openclaw.json
{
agents: {
defaults: {
memorySearch: {
provider: "openai", // OpenAI-compatible endpoint
remote: {
baseUrl: "http://localhost:1234/v1/", // LM Studio
apiKey: "lm-studio",
batch: { enabled: false }
},
fallback: "none",
model: "text-embedding-qwen3-embedding-0.6b",
query: {
minScore: 0.25,
hybrid: {
enabled: true,
vectorWeight: 0.6,
textWeight: 0.4,
candidateMultiplier: 4,
mmr: { enabled: false },
temporalDecay: { enabled: false }
}
},
experimental: { sessionMemory: true },
sources: ["memory", "sessions"],
extraPaths: ["TOOLS.md", "USER.md", "SOUL.md", "AGENTS.md"]
}
}
}
}
Agent Capabilities
memory_search: Semantic recall — query past memories with natural language, returns ranked results with hybrid BM25 + vector scoring.memory_get: Targeted read — fetch specific memory files by path when you know what you need.- Pre-compaction flush: Automatic reminder triggers the agent to save durable memories before context compaction discards working memory.
- Session transcript indexing: Past conversation turns are indexed and searchable, enabling cross-session memory without explicit logging.
- QMD backend (experimental): BM25 + vectors + reranking sidecar for advanced retrieval pipelines.
Automated Extraction + Consolidation
# Automated extraction + consolidation pipeline
# Runs as: python -m memory_pipeline extract|consolidate|status
# Daily: extract novel facts from sessions
$ python -m memory_pipeline extract
# → Scans sessions, embeds chunks, compares against memory
# → Sends novel chunks to Opus for fact extraction
# → Writes to memory/YYYY-MM-DD.md
# Weekly: consolidate and deduplicate
$ python -m memory_pipeline consolidate --apply
# → Clusters similar memories, identifies duplicates
# → Opus merges and resolves contradictions
# → Promotes high-value entries to MEMORY.md
Benchmark Results
Before and after our optimization pass — indexing workspace files and tuning hybrid search.
What NOT to Do
Common mistakes that add complexity without improving agent memory quality.
✕ Don't add a graph database
For single-user or small-team systems, graph databases (Neo4j, etc.) add massive infrastructure complexity with minimal benefit. Flat files + hybrid search cover the same ground for <100 entities.
✕ Don't over-engineer retrieval
Simple hybrid search (BM25 + vectors) beats complex multi-stage pipelines with rerankers, query expansion, and hypothetical document embeddings. Add complexity only when simple approaches measurably fail.
✕ Don't rely on "mental notes"
Anything not written to disk is lost on context compaction or session end. If the agent learns something durable, it must write it to a file. "I'll remember that" is a lie — context windows don't survive restarts.
✕ Don't load entire memory files
Search first, read targeted snippets. Loading a 50KB daily log burns context that could hold useful work. Use memory_search to find the right section, then read just that section.
✕ Don't enable temporal decay early
Time-based score decay has negligible effect on corpora under ~500 files. It sounds smart but adds complexity without measurable improvement until your memory corpus is large enough for recency to matter.
State-of-the-Art Comparison
How this architecture compares to major memory systems across key dimensions.
| Feature | This Architecture | ChatGPT | Claude API | Mem0 | Letta |
|---|---|---|---|---|---|
| Storage type | Flat files + SQLite | Opaque cloud | Files (your infra) | Vector DB + graph | Files + SQLite |
| Search method | ✓ Hybrid BM25+vector | Proprietary | Tool-based | ✓ Vector + graph | ✓ Archival search |
| Auto-extraction | ✓ Hybrid pipeline | ✓ Automatic | ◐ Agent-driven | ✓ Automatic | ✓ Automatic |
| Consolidation | ✓ Weekly automated | ✓ Background | ✕ Manual | ✓ Automatic | ✓ Automatic |
| Graph relations | ✕ Not needed | ✕ None | ✕ None | ✓ Entity graph | ✕ None |
| Human-editable | ✓ Plain markdown | ✕ Opaque | ✓ Your files | ✕ DB records | ◐ JSON files |
| Local / private | ✓ Fully local | ✕ Cloud-only | ◐ API calls | ◐ Self-hostable | ✓ Self-hostable |
| Infrastructure | ✓ Zero (files only) | N/A (hosted) | Low (API) | Medium (DB + API) | Low (server) |
Automated Memory Pipelines
Two automated pipelines handle the tedious work of memory extraction and consolidation — embeddings do cheap bulk filtering, LLMs do precise understanding.
Daily Extraction Pipeline
Scans session transcripts (JSONL), embeds chunks via Qwen3 0.6B, compares against existing memory embeddings to find novel information (cosine sim < 0.60). Sends novel chunks to Claude Opus for fact extraction — decisions, preferences, configs, key facts.
~15-30s per runWeekly Consolidation Pipeline
Loads all memory embeddings from SQLite. Builds pairwise similarity matrix, clusters near-duplicates (cosine > 0.85). Identifies promotion candidates from daily logs. Sends clusters to Opus for intelligent merging and contradiction resolution.
--apply or --dry-runWhy Hybrid (Embeddings + LLM)?
Pure embeddings can't distinguish "loves React" from "hates React" — both embed similarly. Pure LLM is expensive for scanning large corpora. Hybrid: embeddings do cheap bulk filtering, LLM does precise understanding on filtered candidates.
Same arch as Mem0Extraction Flow
- Scan session transcripts (JSONL files) for the target date
- Parse user/assistant turns, strip metadata, chunk into ~400-token segments
- Embed each chunk via local embedding model (Qwen3 Embedding 0.6B, 1024 dims)
- Compare against all existing memory embeddings — find chunks with
max_similarity < 0.60 - Send novel chunks to Claude Opus for fact extraction with category prefixes:
[decision],[preference],[fact],[config] - Write extracted facts to daily memory log at
memory/YYYY-MM-DD.md - Key insight: Embeddings handle SELECTION (fast, free, deterministic). LLM handles UNDERSTANDING (smart, accurate).
Consolidation Flow
- Load all memory chunk embeddings from SQLite (memory source only, excludes session transcripts)
- Build pairwise similarity matrix, filter same-file adjacent chunk overlaps
- Cluster near-duplicates (cosine > 0.85) using greedy clustering
- Identify promotion candidates — daily log entries not yet captured in MEMORY.md (sim < 0.70)
- Send clusters to Claude Opus for intelligent merging, contradiction resolution, summarization
- Generate consolidation report to
memory/reference/consolidation-YYYY-MM-DD.md - Auto-promote high-value entries to MEMORY.md with
--applyflag
Trigger Architecture
# When each pipeline runs:
Session Start → extract (background, ~15-30s)
4:30 AM daily → memory-hygiene (archive, validate, budget)
5:00 AM daily → memory-extraction (scan sessions → extract facts)
Sunday 7:30 AM → memory-consolidation (cluster → dedup → promote)
# Key principle:
# Every session start triggers extraction from the previous session.
# Daily and weekly crons are the safety net.
Pipeline Usage
# Daily: extract novel facts from sessions
$ python -m memory_pipeline extract
# → Scans sessions, embeds chunks, compares against memory
# → Sends novel chunks to Opus for fact extraction
# → Writes to memory/YYYY-MM-DD.md
# Weekly: consolidate and deduplicate
$ python -m memory_pipeline consolidate --apply
# → Clusters similar memories, identifies duplicates
# → Opus merges and resolves contradictions
# → Promotes high-value entries to MEMORY.md
# Check corpus health
$ python -m memory_pipeline status
Key References
Papers, benchmarks, and codebases that informed this architecture.