Asking language models how to represent data for fine-tuning

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

The paper "Asking language models how to represent data for fine-tuning" by Singh et al. (SURGeLLM 2026) addresses the challenge of choosing optimal data formats for fine-tuning language models on structured data. The authors demonstrate three key findings. First, data format choice significantly impacts learning efficiency even after fine-tuning, indicating models do not simply adapt to any format. Second, pre-trained models can autonomously suggest suitable candidate formats by auto-completing partial prompts, reducing reliance on human intuition. Third, and most critically, the performance of a base model on different formats reliably predicts its post-fine-tuning performance: the format that performs best before fine-tuning remained among the top candidates after fine-tuning in 16 out of 18 settings across three data structure types, three models, and six tasks. This allows format selection via inference, eliminating costly trial-and-error fine-tuning.

Key takeaway

For Machine Learning Engineers optimizing language model fine-tuning for structured data, you should prioritize evaluating data representation formats using base model inference. This approach allows you to reliably predict post-fine-tuning performance, as the best-performing format before fine-tuning often remains superior. By leveraging pre-trained models to suggest candidate formats and then testing them via inference, you can avoid expensive and time-consuming trial-and-error fine-tuning runs, significantly streamlining your development process.

Key insights

Base model performance on data formats reliably predicts post-fine-tuning efficacy, enabling inference-based format selection.

Principles

• Data format choice is crucial for fine-tuning efficiency.
• Pre-trained models can suggest optimal data formats.
• Base model performance predicts fine-tuned performance.

Method

A three-step approach: confirm format importance, use pre-trained models to suggest formats via prompt auto-completion, then select formats based on base model inference performance.

In practice

• Evaluate data formats on base models before fine-tuning.
• Use prompt auto-completion to generate format candidates.
• Prioritize formats performing well in zero-shot inference.

Topics

• Language Models
• Fine-tuning
• Structured Data
• Data Representation
• Prompt Engineering
• Inference Optimization

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Supervised Fine Tuning | Day 0 — SFT transforms rambling base models into instruction-following assistants using curated prompt/response data.
• Predicting Causal Effects from Natural Language Queries using Structured Representations — Finetuning and structured representations significantly enhance LLM performance in predicting causal effects from natural language.
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data — Structured output formats and careful source data selection are key to high-quality synthetic pretraining data.
• The Hardest Part of Fine-Tuning Isn’t the Training — The most challenging aspect of NLP fine-tuning is effective label design and meticulous data annotation, not the model training itself.
• HARP: Efficient Data Selection for Finetuning Large Language Models — HARP efficiently selects finetuning data for LLMs by hierarchically evaluating representative subsets and inferring utilities.
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models — EBFT fine-tunes language models by matching sequence-level features, improving performance over next-token prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.