Three Months of my LLM-wiki: A Follow-Up and an Update

2026-07-19 · Source: AI on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Research Methodology & Innovation · Depth: Intermediate, medium

Summary

The author provides a three-month follow-up on their LLM Research Wiki, a knowledge base built on Andrej Karpathy's pattern and maintained by Claude Code. The wiki nearly doubled in size, growing from 185 to 367 pages, with cross-references increasing from 9 to 12 per page. This growth exposed a design flaw: the LLM agent's "LINT" workflow struggled with mechanical checks at scale, becoming slow and token-expensive. Inspired by projects like Tome and Engram, the author developed "wiki lint", a small, dependency-free tool. This "hands layer" handles deterministic tasks such as resolving broken links, identifying orphan pages, checking index-disk consistency, and validating frontmatter. The "wiki lint" tool successfully identified 10 orphan pages and 9 unindexed pages in the live wiki, which were subsequently corrected. The author retained specialized features like richer page types and epistemic markers for humanities research. The open template is now called Zissa Wiki and includes this new linter.

Key takeaway

For Research Scientists or AI Engineers building LLM-maintained knowledge bases, you should explicitly separate the LLM's interpretive role from mechanical validation tasks. Implement dedicated, deterministic tools like "wiki lint" to handle checks such as broken links, orphan pages, and index consistency. This approach ensures scalability and accuracy, preventing silent data rot as your knowledge base grows, allowing your LLM agent to focus on higher-value synthesis and judgment.

Key insights

LLMs excel at interpretive tasks but require deterministic tools for mechanical validation in large, evolving knowledge bases.

Principles

• Separate interpretive tasks for agents, mechanical tasks for tools.
• Externalize data-shaped rules into configuration files.
• Deterministic tools scale better for knowledge base validation.

Method

Implement a "hands layer" tool to offload deterministic knowledge base checks from LLM agents. This tool performs link resolution, orphan page identification, index-disk comparison, and frontmatter validation using a configuration file.

In practice

• Use "wiki lint" for deterministic knowledge base health checks.
• Implement configuration files for wiki page types and rules.
• Integrate automated checks as a final step in wiki sessions.

Topics

• LLM Research Wiki
• Knowledge Management
• AI Agents
• Data Validation
• Zissa Wiki
• Andrej Karpathy Pattern

Code references

• chicken-noodle-chris/tome
• jeromeetienne/engram

Best for: AI Engineer, Research Scientist, Software Engineer

Related on AIssential

• Building my own LLM-Wiki Research Team — Treat notes like code and compile them into a persistent, compounding markdown wiki using an LLM Agent to avoid rediscovering insights.
• When the Judge Is Wrong — LLM-as-judge is unreliable for structured tasks, even with ground truth or source documents.
• ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs — Parametric tool retrieval in LLMs often shows a knowledge-retrieval dissociation, failing on realistic queries despite high benchmark scores.
• Give Your AI Unlimited Updated Context — An LLM-maintained personal wiki creates a persistent, compounding knowledge artifact, eliminating repetitive context setup.
• How to Build a Powerful LLM Knowledge Base — LLM-powered knowledge bases automate information capture and retrieval, significantly enhancing decision-making and operational efficiency.
• Can LLMs Learn and Apply Multi-Level Modelling Semantics? A First Empirical Study — LLMs reliably transcribe explicit modelling requirements but struggle with implicit multi-level semantics and inferring unstated structural safeguards.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI on Medium.