Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?

2026-03-31 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new conceptual framework predicts how training affects the monitorability of a Large Language Model's (LLM) Chain-of-Thought (CoT). The framework models LLM post-training as a Reinforcement Learning (RL) environment where the reward function is decomposed into two terms: one for final outputs and another for the CoT. These terms are classified as "aligned," "orthogonal," or "in-conflict" prior to training. The framework predicts that "in-conflict" terms will reduce CoT monitorability, "orthogonal" terms will have no effect, and "aligned" terms will improve it. Empirical validation across various RL environments confirmed that training with "in-conflict" reward terms reduces CoT monitorability and that optimizing such terms is challenging.

Key takeaway

For research scientists developing and deploying LLMs, understanding the relationship between reward function components and CoT monitorability is crucial. If your training objectives include both final output and CoT-based rewards, you should analyze their alignment using this framework. Prioritize "aligned" or "orthogonal" reward structures to maintain or improve CoT monitorability, as "in-conflict" terms will likely degrade oversight capabilities and complicate optimization.

Key insights

A framework predicts how reward term alignment impacts LLM Chain-of-Thought monitorability during training.

Principles

• Reward terms can be aligned, orthogonal, or in-conflict.
• In-conflict reward terms reduce CoT monitorability.
• Optimizing in-conflict reward terms is difficult.

Method

The method models LLM post-training as an RL environment, decomposing rewards into output and CoT terms, then classifying these terms to predict monitorability changes.

In practice

• Classify reward terms before LLM training.
• Avoid training with "in-conflict" reward terms.

Topics

• Chain-of-Thought
• LLM Monitoring
• Reinforcement Learning
• Reward Alignment
• AI Oversight

Best for: Research Scientist, AI Scientist

Related on AIssential

• Predicting When RL Training Breaks Chain-of-Thought Monitorability — A framework predicts when RL training degrades Chain-of-Thought monitorability by analyzing reward structures.
• Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings — Sequential DPO's impact on prior preferences varies significantly, not uniformly, depending on objective relationships and signal strength.
• Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs — Advanced LLM reasoning can be exploited by framing harmful actions within ethical dilemmas, bypassing safety alignments.
• Alignment as Institutional Design: From Behavioral Correction to Transaction Structure in Intelligent Systems — AI alignment should shift from behavioral correction to institutional design, fostering aligned behavior through internal economic structures.
• Why We Think — Allocating more test-time compute via methods like Chain-of-Thought significantly boosts LLM reasoning and problem-solving capabilities.
• Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph — This paper investigates the solvability of dilemmas and conflicts in Large Language Model (LLM) alignment, first summarizing and taxonomizing these diverse conflicts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.