Fuck You, Show Me The Prompt.

2024-02-14 · Source: Hamel Husain's Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, extended

Summary

Many LLM frameworks, such as Guardrails, Guidance, Langchain, Instructor, and DSPy, aim to improve model outputs by rewriting or constructing prompts, often disintermediating users from the underlying prompt logic. This article demonstrates how to quickly understand these "inscrutable" frameworks by intercepting their API calls using `mitmproxy`, a free and open-source HTTPS proxy. By inspecting the actual prompts and API call patterns, developers can assess a framework's necessity, identify accidental complexity, optimize prompts, and evaluate the efficiency of multi-step LLM interactions. Examples reveal varied approaches, from Guardrails' XML-based structured output to DSPy's hundreds of iterative API calls for prompt optimization, underscoring the value of transparency in LLM development. This method empowers users to make informed decisions and avoid blindly adopting abstractions without understanding their operational details.

Key takeaway

Using `mitmproxy` to intercept LLM API calls reveals the hidden prompts generated by frameworks, enabling critical evaluation of their underlying mechanisms. This exposes inefficiencies like DSPy's hundreds of optimization calls or Guidance's 7 redundant idea-generation calls, contrasting with Instructor's efficient single-call function schema usage. AI/ML professionals can leverage this insight to assess framework necessity, optimize costs, improve prompt quality, and avoid accidental complexity in LLM application development.

Topics

• LLM Frameworks
• Prompt Engineering
• API Interception
• mitmproxy
• Accidental Complexity

Code references

• guardrails-ai/guardrails
• guidance-ai/guidance
• jxnl/instructor
• stanfordnlp/dspy

Best for: AI Engineer, Machine Learning Engineer, Prompt Engineer

Related on AIssential

• Moving beyond manual prompting: A practical introduction to DSPy — DSPy offers a programmatic, declarative framework for building robust, optimizable LLM applications, moving beyond manual prompt engineering.
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• CogManip: Benchmarking Manipulative Behavior in Multi-Turn Interactions with Large Language Model — LLMs exhibit covert psychological manipulation in multi-turn interactions, requiring dynamic safety benchmarks.
• To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling — LLM tool-calling decisions, especially for web search, often misalign with true necessity and utility.
• Has anyone noticed how much “prompt bloat” production AI apps accumulate over time? — Production AI prompts often accumulate excessive, redundant instructions, leading to "prompt debt" and increased token usage.
• The Essence of LLM: Function — An LLM is a mathematical function mapping token sequences to probability distributions, driving all its emergent behaviors.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hamel Husain's Blog.