LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance

2026-04-16 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Research on large language models (LLMs) for automated code compliance typically focuses on performance, but this study investigates how training decisions influence their interpretive behavior. The paper uses a perturbation-based attribution analysis to compare LLMs across various fine-tuning strategies, including full fine-tuning (FFT), Low-Rank Adaptation (LoRA), and quantized LoRA. It also examines the impact of different LLM parameter sizes. Findings indicate that FFT yields attribution patterns that are statistically distinct and more concentrated than those from parameter-efficient methods. Additionally, as model scale increases, LLMs prioritize numerical constraints and rule identifiers in text, though performance gains in semantic similarity for generated rules plateau for models larger than 7B parameters. This work aims to enhance the transparency of LLMs for critical, regulation-based tasks in the Architecture, Engineering, and Construction industry.

Key takeaway

For AI Engineers developing LLMs for regulatory compliance, understanding how fine-tuning and model scale affect interpretive behavior is crucial. Full fine-tuning (FFT) offers more focused attribution, which can be vital for explainability in critical applications. Be aware that performance gains in semantic similarity may plateau for models larger than 7B, suggesting an optimal scale for resource allocation.

Key insights

Fine-tuning strategies and model scale significantly alter LLM interpretive behavior in code compliance tasks.

Principles

• FFT creates more focused attribution patterns.
• Larger LLMs prioritize specific textual elements.

Method

A perturbation-based attribution analysis was used to compare LLM interpretive behaviors across full fine-tuning, LoRA, and quantized LoRA, and varying model scales.

In practice

• Consider FFT for clearer LLM attribution.
• Evaluate LLM scale beyond 7B for diminishing returns.

Topics

• LLM Attribution Analysis
• Automated Code Compliance
• Fine-tuning Strategies
• Model Scale
• Low-Rank Adaptation

Best for: AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• LLM attribution analysis across different fine-tuning strategies and model scales for automated code compliance — Fine-tuning strategies and model scale significantly alter LLM interpretive behavior for code compliance.
• I created an LLM trained solely on Jeffrey Epsteins emails to see how messed up it becomes :) — Fine-tuning an LLM on a controversial, unannotated dataset can reveal unique behavioral shifts and spark technical debate.
• Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance — LLM-generated explanations often lack epistemic faithfulness, but a new method improves it via attention-level guidance.
• Understanding post-training of LLMs: SFT — Post-training guides LLM behavior and expression, not knowledge, primarily via SFT and preference optimization.
• 5 Fun Papers That Explain LLMs Clearly — Five foundational papers collectively explain the core mechanisms of modern LLMs, from architecture to knowledge integration.
• Exploring Extrinsic and Intrinsic Properties for Effective Reasoning with Code Interpreter — Stronger LLM Code Interpreter reasoning correlates with specific "crucial tokens" and cognitive behaviors like verification and backtracking.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.