Assemble Each RAG Generation Prompt from a Base Prompt Plus the Rules Each Question Needs

2026-07-05 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, extended

Summary

This article, part 8B of the Enterprise Document Intelligence series, details a structured approach to RAG generation prompt assembly. It introduces a "dispatcher" mechanism that dynamically composes system prompts from a fixed BASE and specific fragments, tailored to the ParsedQuestion's expected_answer_shape and GenerationBrief. This method prevents "mega-prompts" by adding only necessary constraints for answer shapes like "amount" or "date," and cross-cutting rules for formatting or disambiguation. User prompts include the original question, keywords, and labeled passage lines, with a critical GLOBAL_LINE rule for accurate source citation. The system also supports FieldExtraction[T] for granular, per-field evidence in complex schemas and integrates dynamic few-shot examples, retrieved by similarity, to address recurring model errors in formatting or extraction. Crucially, it mandates persisting the full raw LLM response for comprehensive audit trails and debugging.

Key takeaway

For AI Engineers building enterprise RAG systems, adopt a modular prompt assembly strategy. Dynamically compose system prompts from a base and specific fragments based on question type, rather than using a single, complex mega-prompt. This improves auditability and reduces token waste. Implement per-field evidence wrappers and persist full raw LLM responses to ensure traceability and simplify debugging of future answer discrepancies. Consider integrating dynamic few-shot examples to address recurring model formatting or extraction issues.

Key insights

Dynamic prompt composition with fragments and per-field evidence ensures auditable, precise RAG generation, avoiding monolithic prompts.

Principles

• Prompt composition should be dynamic and modular.
• Anchor every claim to source line numbers.
• Persist full raw LLM responses for audit.

Method

A dispatcher reads a ParsedQuestion, picks the schema, composes the system prompt from a fixed BASE plus fragments, builds the user prompt, calls the model, and stores the full trace. This avoids monolithic prompts.

In practice

• Use GLOBAL_LINE rule for accurate citations.
• Implement FieldExtraction[T] for granular evidence.
• Employ dynamic few-shot examples for specific errors.

Topics

• RAG Generation
• Prompt Engineering
• System Prompt Design
• ParsedQuestion Schema
• Field-Level Evidence
• Dynamic Few-Shot Learning
• LLM Audit Trails

Code references

• doc-intel/notebooks-vol1

Best for: AI Engineer, Machine Learning Engineer, Prompt Engineer

Related on AIssential

• Dispatching the Parsed RAG Question: Chunk Strategy, Model Tier, Activations, Audit — Effective RAG dispatch dynamically adapts chunking, model choice, and activations based on parsed questions and document profiles for auditability and cost efficiency.
• Context Engineering for RAG : The Four Typed Inputs Behind Every RAG Answer — Context engineering systematically assembles typed inputs for LLMs, enhancing RAG auditability and cost-efficiency.
• LangChain Series #3: Prompts Explained — Prompt Templates, Chat Prompts, Dynamic Prompting… — LangChain transforms basic prompts into dynamic, structured, and reusable components for robust LLM applications.
• Derivation Prompting: A Logic-Based Method for Improving Retrieval-Augmented Generation — Derivation Prompting uses logic-inspired rule application to enhance RAG generation, reducing LLM hallucinations and improving interpretability.
• Zuckerberg Is Cloning Himself? — Enhancing LLM performance requires strategic prompting, external knowledge integration, and agentic workflows, moving beyond base model limitations.
• The Open Source RAG Stack: A Complete Guide to Building Retrieval-Augmented Generation Systems — The open-source RAG stack offers a modular, customizable approach to building context-rich AI systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.