Does the Structured Output schema become part of the prompt?

2026-07-19 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, long

Summary

Structured Outputs enable Large Language Models (LLMs) to generate predictable, consistently formatted responses, addressing the fragility of parsing free-form text. When using OpenAI's SDK, a Pydantic schema, like "CountryList", is converted into a JSON Schema. This schema is then sent to the OpenAI API via the "response_format" field, not as part of the prompt text. OpenAI internally transforms this JSON Schema into a Context-Free Grammar (CFG) to perform dynamic constrained decoding. This process ensures that each generated token adheres to the defined structure, preventing syntax errors or unexpected fields. While the schema isn't a text prompt, its inclusion in the API request increases input token usage, for example, from 18 to 124 tokens in a given example. This method simplifies LLM integration into applications by providing reliable, ready-to-use structured data.

Key takeaway

For AI Engineers integrating LLMs into applications, adopting Structured Outputs is crucial for robust data consumption. This approach eliminates the need for fragile regex or complex post-processing by ensuring LLM responses consistently adhere to a defined schema. You should define Pydantic models for your desired output, as this directly translates into a JSON Schema used by the LLM provider for constrained decoding. This practice significantly reduces maintenance overhead and improves application reliability, despite a slight increase in input token count.

Key insights

Structured Outputs ensure LLM responses conform to a predefined schema through dynamic constrained decoding, not by modifying the prompt text.

Principles

• Regex and text parsing are fragile for LLM outputs.
• Consistent output structures simplify application integration.
• JSON Schema converts to Context-Free Grammar for token-level constraint.

Method

Define a Pydantic schema, pass it to `client.responses.parse()` with `text_format`, which generates a JSON Schema for OpenAI's constrained decoding.

In practice

• Directly consume LLM output as Python objects.
• Display structured data in applications.
• Save parsed data to databases reliably.

Topics

• Structured Outputs
• Large Language Models
• OpenAI API
• JSON Schema
• Pydantic
• Constrained Decoding
• Application Integration

Best for: AI Engineer, Machine Learning Engineer, Software Engineer

Related on AIssential

• Issue #128 - Structured LLM Outputs with Pydantic — Combine Pydantic schemas with LangChain's output parsers and LCEL for robust, structured LLM outputs.
• Structured Outputs with LLMs: JSON Mode, Function Calling, and When to Use Each — LLM structured output methods vary in schema enforcement, suiting different application reliability and consistency needs.
• Structured outputs — Ollama now supports constraining LLM outputs to a JSON schema for reliable, consistent structured data.
• Structured Language Model Generation with Outlines — Outlines ensures LLMs produce strictly structured outputs by masking invalid tokens during generation.
• Building Intelligent Feedback Systems: A Deep Dive into Conditional Agentic Workflows with… — Agentic workflows with structured outputs enable LLMs to act as decision-makers in complex, stateful processes.
• MedMind AI at #SMM4H-HeaRD 2026: Data Extraction and Generation Using Prompt Engineering and Structured Outputs (Tasks 1–6) — General-purpose LLMs can accurately extract and classify health data using prompt engineering and structured outputs without fine-tuning.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.