Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

2026-07-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Code-MUE is a novel black-box framework designed to measure the uncertainty of Code Large Language Models (LLMs) by addressing the critical syntax-semantics gap in existing methods. Unlike approaches relying on textual similarity, Code-MUE grounds uncertainty in observable runtime behavior through execution-based Semantic Interaction Graphs. It quantifies global semantic diversity by calculating the Von Neumann entropy of the solution space. A large-scale empirical study involving eight state-of-the-art LLMs demonstrated Code-MUE's strong negative correlation with functional correctness, achieving a Spearman's correlation up to -0.98. This significantly outperforms lexical and embedding-based baselines, enabling robust risk detection and selective prediction in practical software engineering workflows.

Key takeaway

For Machine Learning Engineers deploying Code LLMs in critical software systems, you should integrate Code-MUE to quantify model uncertainty based on observable execution behavior, moving beyond superficial text metrics. This enables robust risk detection and selective prediction, significantly improving reliability and mitigating potential functional, security, or safety consequences stemming from LLM stochasticity.

Key insights

Code-MUE quantifies Code LLM uncertainty by analyzing the semantic diversity of execution-based outputs, bridging the syntax-semantics gap.

Principles

• Code uncertainty requires semantic, not just syntactic, evaluation.
• Execution-based analysis reveals true code behavior diversity.
• Von Neumann entropy quantifies solution space diversity.

Method

Code-MUE constructs Semantic Interaction Graphs from the runtime behavior of multiple LLM-generated code variants. It then calculates the Von Neumann entropy of this solution space to quantify global semantic diversity and, consequently, model uncertainty.

In practice

• Apply Code-MUE for robust risk detection.
• Implement selective prediction for Code LLMs.
• Evaluate LLM code outputs via runtime behavior.

Topics

• Code LLMs
• Uncertainty Quantification
• Semantic Analysis
• Execution-based Testing
• Software Engineering
• Large Language Models

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs — Code-MUE quantifies LLM uncertainty in code by executing multiple samples and analyzing their runtime behavior via Semantic Interaction Graphs.
• Code Is More Than Text: Uncertainty Estimation for Code Generation — Code generation uncertainty estimation requires specialized methods accounting for code's unique properties, outperforming natural language-derived baselines.
• Uncertainty Quantification for LLM Function-Calling — Adapting Uncertainty Quantification methods to the unique structure of LLM function-calling outputs significantly enhances their performance.
• A Mechanistic Lens on Semantic Conflicts: Using Activation Patching to Understand LLM Behavior — LLMs often prioritize misleading semantic cues over executable code, impacting correctness in software engineering tasks.
• From Brewing to Resolution: Tracing the Internal Lifecycle of Code Reasoning in LLMs — LLMs' code reasoning involves an internal "brewing" phase before diverging into distinct resolution outcomes, revealing hidden failure modes.
• Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework — LLM trustworthiness hinges on evaluating reasoning fidelity, not just final answer agreement, a gap GraphEVAL addresses.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.