Lost in Space: Finding the Right Tokens for Structured Output

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Research on structured output for large language models reveals that output format significantly impacts downstream performance in tasks like annotation and classification. The study, which tested four popular LLM families with five output formats across four common NLP benchmarks, found that models achieve highest accuracy when guided to use conventional formats, such as single letters for multiple-choice questions or real numbers for numerical predictions. Crucially, performance consistently improved by 5%-10% when models were prompted to return tokens incorporating leading whitespace, with smaller models showing the most significant gains. This improvement is attributed to leading whitespace helping models circumvent structural deficiencies inherent in subword token representations. The findings offer best practices for researchers employing language models as zero-shot classifiers with structured output.

Key takeaway

For Machine Learning Engineers designing zero-shot classifiers with structured output, prioritize conventional output formats like single letters or real numbers. You should also explicitly guide your models to return tokens that include leading whitespace, as this can boost performance by 5%-10%, particularly for smaller models, by addressing subword token representation issues. This simple adjustment can significantly enhance accuracy and reliability in your LLM applications.

Key insights

LLM structured output performance improves significantly with conventional formats and leading whitespace due to subword token representation benefits.

Principles

• Conventional formats enhance LLM accuracy.
• Leading whitespace improves token representation.
• Smaller models benefit most from whitespace.

Method

The study tested four LLM families with five output formats on four NLP benchmarks, analyzing performance changes when sampling tokens according to grammar-guided structured output.

In practice

• Use letters for multiple-choice output.
• Use real numbers for numerical prediction.
• Incorporate leading whitespace in output tokens.

Topics

• Structured Output
• Large Language Models
• Zero-Shot Classification
• Tokenization
• Subword Representations
• NLP Benchmarks

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Using a Local LLM as a Zero-Shot Classifier — Locally hosted LLMs can effectively perform zero-shot classification on short, semantically complex texts where traditional methods fail.
• Inside LLMs Part 1: How Large Language Models Read, Encode, and Position Every Word You Write |… — LLMs transform text into numerical representations via tokenization, embeddings, and positional encoding.
• The Structured Output Benchmark (SOB) - validates both JSON parse and value accuracy [R] — The Structured Output Benchmark (SOB) measures JSON value accuracy, revealing a significant gap between schema compliance and factual correctness.
• Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study — Tokenizer efficiency and zero-shot performance vary significantly across foundation models on Ukrainian legal text.
• Can AI Save India’s Dying Languages? | Building a Tulu LLM Without Training Data Ft. Prathamesh — Structured prompting and custom romanization enable LLMs to generate low-resource languages without extensive training data.
• Command R+ — The AI field faces challenges in LLM evaluation, zero-shot translation, and a dynamic job market.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.