Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing, Mechanistic Interpretability · Depth: Expert, extended

Summary

A study analyzed moral bias, specifically the Knobe effect, in finetuned large language models (LLMs) using mechanistic interpretability. Researchers applied Layer-Patching analysis across Llama-3.1-8B, Mistral-7B-v0.1, and Gemma-2-9b. They found that finetuning consistently amplifies the Knobe effect, causing LLMs to attribute significantly higher intentionality to negative side effects than positive ones, mirroring human cognitive bias. Pretrained models exhibited minimal bias. This moral bias was localized to specific mid-to-late Transformer layers. Critically, patching activations from the corresponding pretrained model into these few layers effectively mitigated the bias, reducing the intentionality attribution gap from 1.6 to 0.0 in Llama-3.1-8B and Mistral-7B-v0.1, and from 3.83 to 0.03 in Gemma-2-9b, without retraining. An ablation study confirmed the bias is induced by finetuning, not model scale, though its magnitude increases with model size.

Key takeaway

For Machine Learning Engineers deploying finetuned LLMs in ethical reasoning tasks, you should actively assess and mitigate moral biases like the Knobe effect. Your finetuned models likely internalize human biases, localizing them in mid-to-late layers. Consider implementing Layer-Patching techniques to selectively remove these biases post-training, using pretrained model activations. This approach offers a precise way to enhance model fairness without costly retraining, ensuring more aligned and responsible AI systems.

Key insights

Finetuning introduces human-like moral biases in LLMs, which are localizable and mitigable via mechanistic interpretability.

Principles

• Finetuning, not pretraining, induces moral biases in LLMs.
• Moral biases localize to mid-to-late Transformer layers.
• Bias magnitude increases with LLM scale after finetuning.

Method

The Layer-Patching algorithm selectively overwrites finetuned model residual activations with those from the pretrained version at specific layers during inference, without altering model parameters or requiring retraining.

In practice

• Use Layer-Patching to mitigate moral biases post-finetuning.
• Target mid-to-late Transformer layers for bias intervention.
• Evaluate LLM moral judgments using Knobe effect scenarios.

Topics

• Large Language Models
• Moral Bias
• Knobe Effect
• Mechanistic Interpretability
• Layer-Patching
• Finetuning

Code references

• TransformerLensOrg/TransformerLens

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, AI Ethicist

Related on AIssential

• A Unified Moral-Value Dataset for Instruction Tuning — A unified moral-value dataset and instruction tuning pipeline improves LLM value alignment while preserving general task performance.
• Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoders — LLMs may infer demographic attributes from subtle linguistic cues, even before explicit output.
• Weight Patching: Toward Source-Level Mechanistic Localization in LLMs — Weight Patching localizes LLM behaviors to specific parameters by transferring weights between specialized and base models.
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction — Fine-tuned AI text detectors amplify a pretrained typicality axis, not construct a new AI-vs-human boundary.
• Modeling Pathology-Like Behavioral Patterns in Language Models Through Behavioral Fine-Tuning — Behavioral fine-tuning can induce specific, stable, pathology-like biases in LLMs, shifting their fundamental interpretive priors.
• Large language models often prioritize Western moral values, overlooking other cultures — Large language models exhibit a systematic Western moral bias, misrepresenting non-Western cultural values.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.