Why Notion’s biggest AI breakthrough came from simplifying everything

2026-01-02 · Source: VentureBeat · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

Notion AI's engineering team, led by Ryan Nystrom, achieved a "step function improvement" in their V3 productivity software by simplifying their approach to LLMs and agentic AI. Initially, they experimented with complex code generation and schemas but found dramatically improved model performance by pivoting to simple prompts, human-readable markdown formats for Notion pages, and minimal abstraction. This re-wired approach enabled the release of customizable AI agents, which have become Notion's most successful AI tool to date. The team also learned the importance of context restraint, identifying a 100,000 to 150,000 token limit as a "sweet spot" to avoid performance degradation and confusion. Furthermore, they advocate for a curated menu of tools rather than an exhaustive list to prevent decision paralysis for the AI model.

Key takeaway

For AI Architects and Machine Learning Engineers building agentic systems, your focus should shift from complex data modeling to simplifying inputs and toolsets. Embrace human-readable formats like markdown for LLM interactions and invest in robust middleware for translation, rather than over-engineering prompts or fine-tuning models. Prioritize internal validation and user adoption over early cost optimization or extensive evals to find true product-market fit.

Key insights

Simplifying AI interactions with human-readable formats and minimal abstraction dramatically improves model performance and user adoption.

Principles

• Channel the model: use APIs as intended, with plain English.
• Less context is better for LLM performance and accuracy.
• Curate tools to avoid overwhelming AI agents with choices.

Method

Represent complex data structures (like Notion pages) as simple markdown. Focus engineering effort on a robust middleware layer for translation between native application structures and LLM-native formats like markdown or JSON.

In practice

• Prioritize internal dogfooding to validate AI features.
• Use manual context addition (e.g., @mentions) for user control.
• Optimize costs and performance only after achieving product-market fit.

Topics

• Notion AI
• Agentic AI
• LLM Prompt Engineering
• Context Window Management
• AI Middleware

Best for: AI Architect, Machine Learning Engineer, Entrepreneur, AI Engineer, AI Product Manager, MLOps Engineer

Related on AIssential

• How to build AI product sense — Context engineering optimizes LLM agent performance by managing context growth through offloading, reduction, isolation, and layered action spaces.
• From Coder to Manager: Navigating the Shift to Agentic Engineering with Notion Co-Founder Simon Last — Notion is evolving into an agent-centric platform, shifting from human-direct work to human-managed AI agents.
• 10 Most Important AI Concepts Explained Simply — Understanding core AI concepts demystifies its behavior and enhances effective tool utilization.
• 4 Things Everyone Must Know About LLMs and Agentic AI — Effective LLM and agentic AI development prioritizes model selection, task simplification, context management, and structural prompting over complex prompts.
• We Don’t Have an AI Problem. We Have a Context Problem. (And MCP Exposes It) — Unstructured context, not model limitations, is the primary bottleneck in scaling and maintaining LLM-based systems.
• Notion’s Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs of Notion — Notion's AI development prioritizes user collaboration and iterative refinement over raw model capabilities, adapting to evolving LLM features.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by VentureBeat.