LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance

2026-04-21 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

A study investigated the interpretive behaviors of Large Language Models (LLMs) when applied to automated code compliance, focusing on how different fine-tuning strategies and model scales affect attribution patterns. Researchers used a perturbation-based attribution analysis to compare full fine-tuning (FFT), Low-Rank Adaptation (LoRA), and quantized LoRA fine-tuning across various LLM parameter sizes. The findings indicate that FFT generates statistically distinct and more focused attribution patterns compared to parameter-efficient methods like LoRA. Additionally, as LLM scale increases, models prioritize numerical constraints and rule identifiers in text, though performance gains in semantic similarity for generated computer-processable rules plateaued for models exceeding 7 billion parameters. This research aims to enhance the explainability of LLMs for critical, regulation-based tasks within the Architecture, Engineering, and Construction (AEC) industry.

Key takeaway

For AI Engineers developing LLMs for automated code compliance in regulated industries like AEC, understanding attribution patterns is critical. Your choice of fine-tuning strategy, particularly between full fine-tuning and LoRA, directly impacts model interpretability. Prioritize full fine-tuning if explainability and focused attribution are paramount, and carefully evaluate the performance gains of models larger than 7B parameters, as semantic similarity improvements may plateau.

Key insights

Fine-tuning strategies and model scale significantly alter LLM interpretive behavior for code compliance.

Principles

• FFT yields more focused attribution patterns than LoRA.
• Larger LLMs prioritize numerical and rule identifiers.
• Semantic performance plateaus beyond 7B parameters.

Method

A perturbation-based attribution analysis was used to compare interpretive behaviors of LLMs across full fine-tuning, LoRA, and quantized LoRA, examining the impact of varying model parameter sizes.

In practice

• Consider FFT for clearer LLM attribution.
• Evaluate LoRA for parameter efficiency.
• Benchmark models larger than 7B for diminishing returns.

Topics

• LLM Attribution Analysis
• Automated Code Compliance
• Fine-tuning Strategies
• Model Scale
• Low-Rank Adaptation

Best for: AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance — Fine-tuning strategies and model scale significantly alter LLM interpretive behavior in code compliance tasks.
• A Practical Guide to LLM Fine Tuning — PEFT methods like LoRA and QLoRA efficiently adapt LLMs by updating minimal parameters, preserving base model knowledge.
• Understanding post-training of LLMs: SFT — Post-training guides LLM behavior and expression, not knowledge, primarily via SFT and preference optimization.
• Making LLMs tell you how confident they really are through probe-targeted fine tuning.[R] — LLMs internally know their confidence but need fine-tuning to express it verbally, overcoming RLHF's bias.
• Build an LLM from Scratch 6: Finetuning for Classification — Fine-tuning pre-trained LLMs for specific classification tasks is efficient and effective, even with smaller models.
• Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment — CLAIR enables robust federated LoRA fine-tuning by identifying shared low-rank structures and contaminated clients from local model differences.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.