CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

2026-07-17 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

CRAFT is a novel method designed to diagnose weak Large Language Model (LLM) capabilities and generate targeted fine-tuning data, moving beyond simple performance measurement to identify *why* a model fails. It transforms any rubric-based evaluation dataset into a model-specific diagnosis by treating each grading criterion as a capability probe. CRAFT extracts capability descriptions from prompt-rubric pairs, clusters these into a hierarchical capability tree, scores the target model at each node, and dynamically selects low-performing nodes where failures are clearest. This diagnosis then guides the generation of specific supervised fine-tuning data. Comparative evaluations against prompt-level EvalTree clustering and untargeted random generation, using four open-source models across finance and legal domains and 13 held-out benchmarks, demonstrated CRAFT's effectiveness. It achieved the strongest finance domain average for all four models and was strongest for three of four models in the legal domain, remaining competitive for the fourth.

Key takeaway

For Machine Learning Engineers developing domain-specific LLMs, CRAFT offers a precise method to enhance model performance. Instead of broad evaluations, you should implement rubric-based diagnostics to pinpoint exact capability weaknesses. This approach allows you to generate highly targeted fine-tuning data, directly addressing identified failures and leading to measurably better models, particularly in specialized fields like finance and legal.

Key insights

CRAFT diagnoses LLM weaknesses at the rubric criterion level to generate targeted fine-tuning data, improving model performance.

Principles

• Evaluation should diagnose *why* models fail.
• Rubric criteria serve as capability probes.
• Hierarchical clustering clarifies failure granularity.

Method

CRAFT extracts capability descriptions from prompt-rubric pairs, clusters them into a hierarchical tree, scores the model at nodes, and selects low-performing nodes to direct targeted data generation.

In practice

• Apply rubric-based evaluations for deeper LLM diagnostics.
• Generate fine-tuning data based on specific capability gaps.
• Improve domain-specific LLM performance (e.g., finance, legal).

Topics

• LLM Evaluation
• Targeted Fine-tuning
• Capability Diagnosis
• Rubric-based Assessment
• Hierarchical Clustering
• Supervised Fine-tuning Data

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction — Augmented LLM-generated rubrics can achieve near-human baseline performance for paper reproduction evaluation, despite inherent biases.
• Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA — Diagnosing small LLMs before fine-tuning predicts performance changes and prevents knowledge degradation in critical QA domains.
• Cluster-Aware Dual-Level Test Specification Generation for Large-Scale Automotive Software Requirements — A "Cluster-then-Summarize" pipeline automates scalable, context-aware test specification generation for automotive software requirements.
• The Poisoned Chalice of LLM Evaluation Report — Data contamination in LLM evaluation for software engineering requires robust, generalizable white-box membership inference methods.
• LLMSurgeon: Diagnosing Data Mixture of Large Language Models — LLMSurgeon diagnoses LLM pretraining data mixtures from generated text, enabling post-hoc auditing without direct data access.
• Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking — Document set quality, considering inter-document interactions, is critical for LLM generation in RAG.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.