Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

2026-07-23 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Advanced, extended

Summary

A study evaluating Qwen3.5 instruction-tuned models (0.8B to 27B parameters) reveals that task competence and instruction following are distinct abilities, especially in small language models (SLMs). Researchers tested models across multiple-choice question answering, sentiment classification, and mathematical question answering, using both standard and instruction-conflicting "non-standard" prompts. They introduced the Instruction-Following Failure Rate (IFFR) metric to quantify instruction-following failures. Results show that smaller models (e.g., 0.8B, 2B) often achieve high standard accuracy, demonstrating task competence, but routinely ignore non-standard instructions, leading to IFFR values approaching 100%. In contrast, larger models (e.g., 9B, 27B) exhibit a significant reduction in IFFR, indicating improved instruction adherence while maintaining task competence. This highlights that gains in task capability do not automatically ensure reliable control over model behavior.

Key takeaway

For machine learning engineers deploying instruction-tuned models, especially smaller ones, you must move beyond standard accuracy metrics. Your evaluation should explicitly test instruction adherence by using prompts that conflict with typical task objectives. Relying solely on high standard accuracy can mask significant instruction-following failures, leading to unpredictable model behavior in production. Implement metrics like Instruction-Following Failure Rate (IFFR) to ensure your models reliably follow user requests, even when those requests deviate from conventional task performance.

Key insights

Task competence and instruction following are distinct abilities, with small models often failing to adhere to conflicting instructions despite task proficiency.

Principles

• Task competence does not automatically provide reliable control over model behavior.
• Reporting only standard accuracy hides instruction-following failures.
• Instruction adherence generally improves with model scale, but can be non-monotonic.

Method

Evaluate models using standard and non-standard (task-conflicting) instructions across multiple tasks, scoring against original ground truth, and calculating Instruction-Following Failure Rate (IFFR).

In practice

• Implement non-standard instructions (e.g., "select incorrect option", "opposite sentiment", "twice the answer").
• Calculate IFFR to isolate instruction-following failures from task competence.

Topics

• Instruction Following
• Small Language Models
• Model Evaluation Metrics
• Task Competence
• Qwen3.5
• Instruction-Following Failure Rate

Code references

• farajimahdieh/language-model-task-competence-vs-instruction-following

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models — Task competence and instruction following are distinct abilities, with standard accuracy hiding instruction-following failures.
• Building AdvancedIF: Evolving Instruction Following Beyond IFEval and “Avoid the Letter C” — Current instruction-following benchmarks often prioritize programmatic verifiability over real-world utility, hindering AI development.
• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models — LLM instruction adherence and recall degrade significantly with increasing instruction count and context length, with format effects being model-specific.
• 10 Ways to Make LLMs Follow Instructions Consistently Instead of Randomly Ignoring Them — LLM instruction inconsistency stems from identifiable factors, not randomness, requiring targeted prompting fixes.
• Self-Preference Is Weak or Absent in Verifiable Instruction-Following Revision: A Four-Model Test Under Genuine Authorship — LLMs show no detectable self-preference when revising their own text based on verifiable instruction-following.
• WildIFEval: Instruction Following in the Wild — LLMs struggle with multi-constraint instructions, especially length, highlighting a need for improved instruction-following.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.