Structured Outputs with LLMs: JSON Mode, Function Calling, and When to Use Each

2026-06-18 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

This article details three distinct methods for generating structured, machine-readable outputs from Large Language Models (LLMs): JSON Mode, Function Calling, and Structured Outputs. JSON Mode is the simplest, ensuring a valid JSON object but offering no guarantees on its internal structure or field consistency. Function Calling, a more advanced approach, allows developers to define a specific schema for the output, though minor deviations can still occur. It also serves as a foundation for agentic AI workflows, enabling LLMs to decide and trigger external actions. Structured Outputs, a stricter variant of Function Calling, guarantees exact schema adherence through constrained decoding, preventing any deviations. This method, available on GPT-4o and later models, ensures reliability for production systems, despite potential performance impacts and schema limitations.

Key takeaway

For AI Engineers building applications that rely on consistent, machine-readable LLM outputs, carefully select your output method. If your system requires strict data integrity for downstream processing or agentic actions, you should prioritize Structured Outputs with `strict: true` for guaranteed schema adherence, especially with GPT-4o. For simpler data extraction where schema flexibility is acceptable, JSON Mode can suffice. Avoid Function Calling without `strict: true` for critical data, as it may introduce schema deviations.

Key insights

LLM structured output methods vary in schema enforcement, suiting different application reliability and consistency needs.

Principles

• JSON Mode ensures valid JSON, not structural schema.
• Function Calling defines schema, but allows deviations.
• Structured Outputs strictly enforces schema via constrained decoding.

Method

JSON Mode requires `response_format={\"type\": \"json_object\"}` and a system prompt. Function Calling/Structured Outputs define a JSON schema for tools, then use `tools` and `tool_choice` parameters, with `strict: true` for the latter.

In practice

• Use JSON Mode for basic machine-readable responses.
• Employ Function Calling for agentic workflows or flexible schema.
• Choose Structured Outputs for critical, schema-guaranteed data extraction.

Topics

• LLM Structured Outputs
• JSON Mode
• Function Calling
• OpenAI API
• Schema Enforcement
• Agentic AI Workflows

Best for: AI Engineer, MLOps Engineer, Software Engineer

Related on AIssential

• Structured outputs — Ollama now supports constraining LLM outputs to a JSON schema for reliable, consistent structured data.
• Structured Outputs vs. Function Calling: Which Should Your Agent Use? — Structured outputs define data shape; function calling dictates application control flow and external interactions.
• The Structured Output Benchmark (SOB) - validates both JSON parse and value accuracy [R] — The Structured Output Benchmark (SOB) measures JSON value accuracy, revealing a significant gap between schema compliance and factual correctness.
• Get Vision LLMs to Follow Your Rules: Prompt-Guided JSON Formatting — Prompt-guided JSON formatting enables LLMs to generate structured data with specific, user-defined output formats.
• When the Judge Is Wrong — LLM-as-judge is unreliable for structured tasks, even with ground truth or source documents.
• TOON: Beyond JSON for LLMs — TOON offers a token-efficient structured data format specifically designed to optimize LLM context window usage over traditional JSON.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.