LLM Knowledge Base Architecture: The Karpathy Pattern

A new architectural pattern for giving LLMs durable, auditable, and self-healing access to proprietary knowledge — without the complexity of vector databases.

In a widely-shared post on X, Andrej Karpathy — former Director of AI at Tesla, OpenAI co-founder, and now an independent AI researcher — described an approach he calls “LLM Knowledge Bases.” The pattern offers an alternative to the dominant Retrieval-Augmented Generation (RAG) paradigm by treating the LLM itself as a “full-time research librarian” that actively compiles, lints, and interlinks Markdown (.md) files.

By directing a significant portion of token throughput into the manipulation of structured knowledge rather than boilerplate code, the pattern represents a meaningful evolution of the “Second Brain” concept: one that is self-healing, auditable, and entirely human-readable.

Beyond RAG

For the past three years, the dominant paradigm for grounding LLMs in proprietary data has been Retrieval-Augmented Generation (RAG). In a standard RAG setup, documents are chunked, converted into vector embeddings, stored in a specialized database, and retrieved via similarity search at query time.

The Karpathy approach rejects this complexity for mid-sized datasets. Instead, it relies on the LLM’s increasing ability to reason over structured text directly.

The architecture functions in three distinct stages:

1. Data Ingest

Raw materials — research papers, GitHub repositories, datasets, and web articles — are dumped into a raw/ directory. Karpathy uses tools like the Obsidian Web Clipper to convert web content into Markdown, ensuring images are stored locally so the LLM can reference them through vision capabilities.

2. The Compilation Step

This is the core innovation. Instead of merely indexing the files, the LLM compiles them. It reads the raw data and writes a structured wiki: generating summaries, identifying key concepts, authoring encyclopedia-style articles, and — crucially — creating backlinks between related ideas.

This compilation step is where the LLM functions as a librarian rather than a search engine. The output is human-readable, navigable, and editable.

3. Active Maintenance (Linting)

The system isn’t static. Periodic “health checks” or “linting” passes scan the wiki for inconsistencies, missing data, and new connections. As one community member observed, the result is “a living AI knowledge base that actually heals itself.”

By treating Markdown files as the source of truth, this pattern avoids the “black box” problem of vector embeddings. Every claim made by the AI can be traced back to a specific .md file that a human can read, edit, or delete. This connects directly to dual-readability principles.

The Three Pillars of the Pattern

Markdown as the Source of Truth

The choice of Markdown is deliberate. It is the most LLM-friendly and compact data format, and it ensures the knowledge base is not locked into any specific vendor. If the application disappears, the files remain readable in any text editor. This is the “file-over-app” philosophy that has gained traction in tools like Obsidian.

The LLM as Active Maintainer

Unlike traditional knowledge management, where the human curates and the system indexes, the Karpathy pattern inverts the relationship. As Karpathy himself summarized: “You rarely ever write or edit the wiki manually; it’s the domain of the LLM.”

The LLM is not just a reader of the knowledge base. It is its primary author and maintainer.

Structure Over Similarity

Where RAG relies on semantic similarity (nearest-neighbor search in vector space), the Karpathy pattern relies on explicit structure — backlinks, indices, and curated summaries. This produces a knowledge graph that is navigable rather than retrievable. See agentic markdown programming for the related concept of Markdown as an instruction layer.

Multi-Agent Extension: The Compound Loop

Architectural breakdowns by community members extend the Karpathy pattern to multi-agent systems — what some have called the “Swarm Knowledge Base.” The core challenge in a multi-agent swarm is that one hallucination can compound and “infect” the collective memory.

The proposed solution is a dedicated Quality Gate: an independent supervisor model (such as the Hermes model trained by Nous Research for structured evaluation) that scores and validates every draft article before it is promoted to the “live” wiki.

This creates a Compound Loop:

1. Agents dump raw outputs into a working area.
2. The compiler organizes them into structured drafts.
3. The Quality Gate validates the drafts.
4. Verified briefings are fed back to agents at the start of each session.

The result is that agents never “wake up blank” — they begin every task with a filtered, high-integrity briefing of everything the collective has learned.

Contamination Mitigation

Steph Ango (co-creator of Obsidian) raised an important refinement: keep the personal “vault” clean and let the agents play in a “messy vault,” only bringing over the useful artifacts once the agent-facing workflow has distilled them.

This is contamination mitigation — preventing the LLM’s experimental work from polluting the canonical knowledge store. It establishes a clear boundary between agent scratch space and the human-trusted record.

Scaling and Performance

A common critique of non-vector approaches is scalability. Karpathy notes that at the scale of approximately 100 articles and 400,000 words, the LLM’s ability to navigate via summaries and index files is more than sufficient. Traditional RAG infrastructure often introduces more latency and “retrieval noise” than it solves at this scale.

For a departmental wiki, a personal research project, or any focused domain, the structured approach can outperform vector retrieval on both speed and traceability.

The pattern also supports an “ephemeral wiki” concept: a temporary, focused mini-knowledge-base spawned for a specific task, then dissolved once the work is complete. This points toward a future where users don’t just “chat” with an AI — they spawn task-specific research environments.

Comparison: Vector RAG vs. Markdown Wiki

The vector DB approach is like a massive, unorganized warehouse with a very fast forklift driver. You can find anything, but you don’t know why it’s there or how it relates to the pallet next to it. The Markdown Wiki is like a curated library with a head librarian who is constantly writing new books to explain the old ones.

When To Use The Karpathy Pattern

The pattern is well-suited for:

• Mid-sized, high-signal corpora (roughly 100 to 10,000 documents).
• Research and writing workflows where traceability matters.
• Personal knowledge management and “Second Brain” systems.
• Departmental or domain-specific wikis with clear ownership.
• Multi-agent systems where a Quality Gate can validate outputs before they enter shared memory.

It is less appropriate for:

• Massive document collections (millions of records).
• Real-time, dynamic data streams that can’t be compiled into stable artifacts.
• Fully unstructured retrieval where similarity is the only relevant signal.

For systems at the boundary, a hybrid approach — Markdown source of truth with an optional vector layer for breadth retrieval — may combine the best of both. See RAG for the complementary pattern.

The End State: Synthetic Data and Fine-Tuning

Karpathy’s exploration points toward an ultimate destination: as the wiki grows and the data becomes “purer” through continuous LLM linting, it becomes the perfect training set.

Instead of the LLM reading the wiki in its context window at query time, the user can eventually fine-tune a smaller, more efficient model on the wiki itself. The LLM would then “know” the knowledge base in its own weights — turning a personal research project into a custom, private intelligence. This connects to the broader fine-tuning discussion.

Why The Pattern Matters

For the individual researcher, the Karpathy pattern means the end of the “forgotten bookmark” — knowledge compounds rather than scattering.

For the enterprise, it represents the transition from a “raw/ data lake” to a “compiled knowledge asset.” Most organizations currently drown in unstructured data: Slack logs, internal wikis, PDF reports that no one has the time to synthesize. A “Karpathy-style” enterprise layer wouldn’t just search these documents. It would actively author a living organizational intelligence that updates in real-time.

The pattern’s deepest claim is philosophical: that the LLM’s role is shifting from one-shot conversational partner to autonomous archive. As Karpathy put it, the wiki is “the domain of the LLM.” The user owns the data; the AI is a highly sophisticated editor that visits the files to perform work.

We are entering the era of LLM-maintained memory.