Build an LLM Knowledge Base That Actually Compounds
Full system, exact prompts, real configuration, and honest about where it breaks.
The Problem You Already Know You Have
You have knowledge scattered everywhere. Articles saved in 4 apps. Bookmarks from 2023 you’ll never revisit. Notes from meetings in a folder you forgot existed.
When you ask AI a question about your stuff, it starts from zero every time. Upload docs, ask a question, get an answer. Next session? Forgotten everything. That’s how ChatGPT file uploads, NotebookLM, and most RAG systems work. Zero accumulation.
The Idea (60 Seconds)
Instead of the AI searching your raw files every time, the AI reads your sources once and compiles a structured wiki. Summaries, cross-references, connections between ideas, contradictions flagged. All maintained by the AI. All in simple markdown files.
Next time you ask a question, the AI doesn’t dig through raw documents. It reads the wiki it already built. The connections are already there. Every new source you add makes the wiki richer. Every question you ask can get filed back in. Knowledge compounds instead of resetting.
No database. No embeddings. No vector store. Just folders and text files.
Why This Setup, Not The Others
Three things make this particular stack worth your time:
Factory Droid reads and writes local files natively. No copy-paste. No uploading. The AI operates directly on your filesystem reading PDFs, creating wiki pages, updating the index, all in one pass.
OpenRouter gives you any model. I run
glm-5.1through OpenRouter. You can swap to Claude, GPT-4o, Gemini, Llama, or any model that drops next week. One config change. No vendor lock-in on the intelligence layer.Obsidian renders the wiki as it’s built. Graph view, backlinks, search, YAML properties, all work automatically because the AI writes standard Obsidian-compatible markdown. You see the knowledge base grow in real time.
What You Need
Factory Droid - AI coding agent that reads/writes local files (factory.ai)
OpenRouter API key - model gateway that lets you use any LLM (openrouter.ai)
Obsidian - markdown editor with wiki-link support (obsidian.md)
10+ source documents on a topic you care about
30 minutes for initial setup, then 10 minutes per source after that
No special software beyond these three. No accounts beyond these two. No plugins to install.
Step 1: Install and Configure Factory Droid (5 Minutes)
Install Droid:
# macOS / Linux
curl -fsSL https://docs.factory.ai/install.sh | bash
Authenticate:
droid login
Add Your Custom Model via OpenRouter
This is how you run any model through Droid, not just the default ones. I use glm-5.1. You can use whatever OpenRouter supports.
Edit ~/.factory/settings.json (create it if it doesn’t exist):
{
"customModels": [
{
"model": "z-ai/glm-5.1",
"displayName": "GLM-5.1 [OpenRouter]",
"baseUrl": "https://openrouter.ai/api/v1",
"apiKey": "YOUR_OPENROUTER_API_KEY",
"provider": "generic-chat-completion-api",
"maxOutputTokens": 65536
}
]
}
To use a different model, change the model field to any model ID from openrouter.ai/models. The displayName is just what shows up in the Droid UI.
To select your custom model in a session:
droid --model "z-ai/glm-5.1"
Or type /model inside a running Droid session to switch on the fly.
Step 2: Create the Folder Structure (2 Minutes)
Create this anywhere on your machine:
my-knowledge-base/
├── raw/ # Your source material. AI reads but never modifies.
│ └── assets/ # Images, screenshots, diagrams
├── wiki/ # AI-maintained wiki. You read. AI writes.
├── outputs/ # Reports, analyses, answers from queries
└── AGENTS.md # The schema file that makes this whole thing work
Three folders, one file. If you’re spending more than 2 minutes here, you’re overthinking it.
Step 3: Write Your Schema File (The Step Everyone Skips)
The schema is the difference between a generic chatbot and a disciplined wiki maintainer. It tells your AI what the knowledge base is about, how to organize it, and what to do when you add sources, ask questions, or run maintenance.
As you can also see, I have multiple knowledge domains in my structure. You don’t have to setup multi-domain folders, one will do just fine. If that’s the case, just remove or rename what you don’t want.
Save this as AGENTS.md in your project root:
# Knowledge Base Schema
## Identity
This is a personal knowledge base about [YOUR TOPIC HERE].
Maintained by an LLM agent. The human curates sources and asks questions. The LLM does everything else.
## Architecture
- raw/ contains immutable source documents. NEVER modify files in raw/.
- wiki/ contains the compiled wiki. The LLM owns this directory entirely.
- wiki/architecture/ -- Enterprise and solution architecture
- wiki/resilience-ops/ -- Resilience, operations, SRE
- wiki/data-ai/ -- Data platforms, ML, AI
- wiki/security/ -- Security, IAM, data protection
- wiki/software-engineering/ -- Software design, practices, CI/CD
- wiki/ai-automation/ -- AI for business process automation
- wiki/index.md -- Master index of all pages by domain
- wiki/log.md -- Append-only chronological record
- Cross-cutting pages (e.g. contradictions-and-tensions.md) live at wiki/ root
- Each domain folder has a home.md landing page listing its pages
- outputs/ contains generated reports, analyses, and query answers.
## Wiki Conventions
- Every topic gets its own .md file in the appropriate domain folder under wiki/
- Every wiki file starts with YAML frontmatter:
---
title: [Topic Name]
created: [Date]
last_updated: [Date]
source_count: [Number of raw sources that informed this page]
status: [draft | reviewed | needs_update]
---
- After frontmatter, a one-paragraph summary
- Use [[topic-name]] for internal links between wiki pages
- Every factual claim cites its source: [Source: filename.md]
- When new info contradicts existing content, flag explicitly:
> CONTRADICTION: [old claim] vs [new claim] from [source]
## Index and Log
- wiki/index.md lists every page by domain with a one-line description
- wiki/log.md is append-only chronological record
- Log entry format: ## [YYYY-MM-DD] action | Description
(Actions: ingest, query, lint, update)
## Ingest Workflow
When processing a new source:
1. Read the full source document
2. Discuss key takeaways with user
3. Create or update a summary page in the appropriate wiki/ domain folder
4. Update wiki/index.md and the domain's home.md
5. Update ALL relevant entity and concept pages across the wiki
6. Add backlinks from existing pages to new content
7. Flag any contradictions with existing wiki content
8. Append entry to wiki/log.md
9. A single source should touch 10-15 wiki pages
## Query Workflow
When answering a question:
1. Read wiki/index.md first to find relevant pages
2. Read all relevant wiki pages
3. Synthesize answer with [Source: page-name] citations
4. If answer reveals new insights, offer to file it back into wiki/
5. Save valuable answers to outputs/
## Lint Workflow (Monthly)
Check for:
- Contradictions between pages
- Stale claims superseded by newer sources
- Orphan pages with no inbound links
- Concepts mentioned but never explained
- Missing cross-references
- Claims without source attribution
Output: wiki/lint-report-[date].md with severity levels
## Focus Areas
[List 3-5 topics this knowledge base covers]
Customise three things before saving:
The [YOUR TOPIC HERE] line -- make it specific (”Enterprise Architecture for Financial Services” not just “Architecture”)
The domain folders -- rename/add/remove to match your topics (I have 6 domains; you might have 3 or 10)
The Focus Areas -- list the 3-5 domains this KB covers
This file is read by Droid at the start of every session. It’s the single most important file in the entire system.
Step 4: Fill Your Raw Folder (10 Minutes of Dumping, Zero Organising)
Open raw/ and dump everything in:
Copy-paste articles into
.mdor.txtfilesExport notes from whatever app you’re using now
Save screenshots and diagrams to
raw/assets/Drop in PDFs (Droid can extract text from them)
Paste in research papers, competitor breakdowns, internal docs
Dump bookmarks you’ve been hoarding for months
Don’t organise it. Don’t rename anything. Don’t clean it up. That’s the AI’s job.
If you have PDFs, Droid will handle extraction automatically. The Anthropic PDF skill (github.com/anthropics/skills/tree/main/skills/pdf) uses pdfplumber and pypdf under the hood. If Droid doesn’t have these installed, it will install them as part of the ingest. No manual setup needed.
The Obsidian Web Clipper browser extension converts any web article to markdown in one click. Set a hotkey to pull all images locally so the AI can reference them.
The goal is volume. Not perfection.
Step 5: Run Your First Ingest
Open your project in Droid:
cd my-knowledge-base
droid
Then paste this prompt:
INGEST PROMPT (single source):
Read the schema in AGENTS.md. Then process [FILENAME] from raw/. Read it fully, discuss key takeaways with me, then: create a summary page in the appropriate wiki/ domain folder, update wiki/index.md and the domain's home.md, update all relevant concept and entity pages across the wiki, add backlinks, flag any contradictions, and append to wiki/log.md. Use the PDF skill ([github.com/anthropics/skills/tree/main/skills/pdf](https://github.com/anthropics/skills/tree/main/skills/pdf) to read the PDFs or convert them to md format.
Start with one source at a time. Read the summaries. Check the updates. Guide the AI on what to emphasise. This produces dramatically better results than batch-processing everything at once.
What happens during an ingest:
Droid reads the full source document from
raw/It discusses key takeaways with you (this is your quality gate)
It creates a summary page in the right domain folder
It creates cross-cutting concept pages that connect to existing content
It updates the index and domain home page
It adds backlinks from existing pages to the new content
It flags contradictions with existing wiki content
It appends a log entry to
wiki/log.md
A single good source will touch 10-15 wiki pages. That’s the compounding in action.
For PDFs specifically, add this to your ingest prompt:
Use the PDF skill (pdfplumber/pypdf) to extract text from the PDF before processing.
For batch ingest (less supervised, use after you trust the system):
Read AGENTS.md. Process all unprocessed files in raw/ sequentially. For each: create summary in the appropriate domain folder, update index and home.md, update relevant pages, add backlinks, flag contradictions, log the ingest. Proceed automatically.
After 5-10 sources, your wiki/ folder will have an index, a log, domain home pages, and 15-30 interconnected pages. That’s when things click.
Step 6: Set Up Obsidian (3 Minutes)
Your wiki is already Obsidian-compatible. It uses markdown files, [[wiki-links]], and YAML frontmatter. You just need to point Obsidian at it.
If Obsidian is already installed:
Open Obsidian
Click “Open folder as vault”
Select your wiki/ folder
Done
If you need to install Obsidian:
# Linux (Flatpak)
flatpak install flathub md.obsidian.Obsidian
# macOS
brew install --cask obsidian
# Or download from obsidian.md
What works immediately in Obsidian:
[[wiki-links]] - all cross-page links are clickable and navigable
Graph view - click the graph icon to see the interconnected page structure
Backlinks panel - right sidebar shows which pages link to the current page
YAML frontmatter - properties like title, status, source_count appear in the properties panel
Search - Ctrl+Shift+F for global search across all pages
Folder navigation - domain folders show up in the file explorer
The vault opens on index.md as the landing page with the full catalogue of all wiki pages by domain.
Step 7: Start Querying Your Knowledge Base
Once you have 10+ wiki pages, the system becomes genuinely useful.
QUERY PROMPT:
Read wiki/index.md. Based on what's in the knowledge base, answer: [YOUR QUESTION]. Cite which wiki pages informed your answer. If this reveals new connections worth preserving, create a new page in the appropriate wiki/ domain folder and update the index.
Questions that extract the most value:
“What are the three biggest gaps in this knowledge base?”
“Which sources disagree with each other, and on what?”
“What should I research next based on what’s here?”
“Write a 500-word briefing on [topic] using only wiki content”
“What connections exist between [concept A] and [concept B]?”
“What contradictions or tensions exist across the sources?”
The critical loop: good answers should be filed back into the wiki. A comparison, an analysis, a connection you discovered. These compound just like ingested sources do. Every question makes the next answer better.
Step 8: Run Monthly Health Checks
This is the step nobody does. It’s the step that prevents the whole system from slowly rotting.
LINT PROMPT:
Run a full health check on wiki/ per the lint workflow in AGENTS.md. Output to wiki/lint-report-[date].md with severity levels. Suggest 3 articles to fill the biggest knowledge gaps.
Why this matters: when the AI writes something slightly wrong and you save it back, the next answer builds on the wrong thing. Two months later, you have five pages reinforcing the same error. Health checks catch this before it snowballs.
One check per month. Ten minutes of your time. Non-negotiable if you want the system to stay trustworthy.
Step 9: Let It Compound
After 4-6 weeks of consistent use, you’re not just searching notes. You’re querying a structured knowledge system that understands connections between your sources better than you do.
Three ways to accelerate the compounding:
File exploration outputs back. When the AI generates a comparison or analysis you find valuable, save it into wiki/ or outputs/. Your own explorations and queries always add up in the knowledge base.
Add visual outputs. Have the AI render answers as markdown tables, charts, or slide decks. These become reusable assets, not throwaway chat messages.
Version control everything. Your wiki is just markdown files. Initialize a git repo. You get full history, branching, and the ability to undo anything the AI messes up.
cd my-knowledge-base
git init
git add .
git commit -m "Initial knowledge base"
Where This System Breaks
This is a nascent pattern, not a finished product. Karpathy himself called it “a hacky collection of scripts.” Here’s what you need to know:
Context Window Ceiling
The wiki works at ~100 articles and ~400K words. But even 128K-token context windows only hold ~96K words. The AI reads selectively through the index, which means it can miss things. Research shows LLMs suffer from “lost in the middle” effects where information in the centre of long inputs gets deprioritised. Your query results will have blind spots. Accept this.
Error Compounding
The AI writes a wiki page with a subtle mistake. You query against it. The mistake enters your answer. You file that answer back. Now two pages reinforce the same error. Monthly linting helps, but the AI doing the linting has the same blind spots as the AI that made the error. This is the single biggest risk.
Hallucination Doesn’t Disappear
The wiki approach reduces hallucination because the AI grounds answers in your sources. But it doesn’t eliminate it. The AI can still synthesise connections that don’t exist in the source material. And because the wiki looks authoritative (clean markdown, cross-references, citations), you’re more likely to trust incorrect information. Don’t.
Cost Isn’t Zero
Every ingest, every query, every lint check costs tokens. A single source that touches 10-15 pages can run $1-2 in API calls with frontier models. Cheaper than a research assistant. Not free. Using OpenRouter with cost-efficient models like glm-5.1 helps, but it’s still not zero.
It Doesn’t Scale to Enterprise
The index-file approach works without RAG at ~100 articles. At 10,000+ sources, this pattern breaks. The index grows too large. Consistency across thousands of pages becomes impossible. You’ll need the infrastructure this system was designed to avoid. Know the ceiling.
What To Do About The Breakpoints
Problem Mitigation Error compounding Monthly lint checks. Cross-check critical claims manually. Never trust blindly on high-stakes decisions. Context limits Keep each wiki focused on one domain. Multiple domains? Multiple knowledge bases. Cost Use frontier models for ingest and complex queries. Cheaper models for simple updates. Hallucination The schema requires source citations on every claim. If a page makes a claim without [Source: filename], linting flags it. Scale Accept this is a personal tool, not enterprise infrastructure. If you outgrow it, that’s a good problem. Model bias Swap models in OpenRouter with one config change. Re-lint after switching to catch interpretation differences.
Your Complete Prompt Library
Every prompt from this guide, collected in one place:
SCHEMA: Copy the full AGENTS.md template from Step 3.
INGEST (one source):
Read the schema in AGENTS.md. Then process [FILENAME] from raw/. Read it fully, discuss key takeaways with me, then: create a summary page in the appropriate wiki/ domain folder, update wiki/index.md and the domain's home.md, update all relevant concept and entity pages across the wiki, add backlinks, flag any contradictions, and append to wiki/log.md.
INGEST (batch, less supervised):
Read AGENTS.md. Process all unprocessed files in raw/ sequentially. For each: create summary in the appropriate domain folder, update index and home.md, update relevant pages, add backlinks, flag contradictions, log the ingest. Proceed automatically.
QUERY:
Read wiki/index.md. Answer: [QUESTION]. Cite wiki pages. If this answer is worth preserving, offer to file it as a new wiki page in the appropriate domain folder.
LINT:
Run a full health check on wiki/ per the lint workflow in AGENTS.md. Output to wiki/lint-report-[date].md with severity levels. Suggest 3 articles to fill gaps.
EXPLORE:
Read wiki/index.md and identify the 5 most interesting unexplored connections between existing topics. For each, explain what insight it might reveal and what source would help confirm it.
BRIEF:
Based on everything in wiki/, write a 500-word executive briefing on [TOPIC]. Cite sources. Structure it as: current state, key tensions, open questions, recommended next steps.
CONTRADICTIONS:
Read all wiki pages and identify every place where guidance in one page conflicts with, undermines, or creates tension with guidance in another page. Categorise as: explicit contradiction, implicit tension, acknowledged trade-off, or vague guidance. Rate severity as high/medium/low.
Go Build It
The difference between bookmarking Karpathy’s gist and benefiting from it is one afternoon.
Pick your topic. Create the folders. Copy the schema. Drop in what you already have. Run your first ingest.
Then do it again tomorrow with another source. And next week with five more.
The wiki gets smarter every time. That’s the whole point.
Three folders. One schema. One custom model. An AI that does the grunt work you’d never do yourself.
Stop collecting bookmarks. Start compiling knowledge.