Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

2026-01-26 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

Code-MUE, introduced in 2026, is a black-box framework designed to measure uncertainty in Code Large Language Models (LLMs) by analyzing execution-based Semantic Interaction Graphs. It addresses the critical syntax-semantics gap in code by grounding uncertainty in observable runtime behavior, rather than relying on superficial textual similarity or inaccessible internal model states. A large-scale empirical study across eight state-of-the-art LLMs demonstrated Code-MUE's strong negative correlation with functional correctness, achieving Spearman's correlation up to -0.98. The framework significantly outperformed lexical and embedding-based baselines, enabling robust risk detection and selective prediction in practical workflows with AUROC scores exceeding 0.85 on multiple benchmarks like HumanEval and MBPP. It quantifies global semantic diversity using the Von Neumann entropy of the solution space.

Key takeaway

For Machine Learning Engineers deploying Code LLMs, Code-MUE offers a critical black-box method to assess code reliability. You can use its uncertainty scores to automatically reject high-risk generations, improving system safety. Additionally, its centrality-based re-ranking allows you to select the most functionally consistent solution from multiple candidates without needing ground-truth test oracles. This enhances deployment confidence and reduces manual validation effort.

Key insights

Code-MUE quantifies LLM uncertainty in code by executing multiple samples and analyzing their runtime behavior via Semantic Interaction Graphs.

Principles

• Code's executability grounds semantic uncertainty in observable behavior.
• Functional correctness correlates strongly with execution-based semantic consensus.

Method

Code-MUE samples programs, synthesizes hybrid test inputs, constructs a Semantic Interaction Graph from execution traces, then computes Von Neumann entropy from its spectrum.

In practice

• Use Code-MUE for robust risk detection in LLM-generated code.
• Employ centrality-based re-ranking to select optimal code candidates.

Topics

• Code LLMs
• Uncertainty Quantification
• Execution-based Testing
• Semantic Interaction Graphs
• Von Neumann Entropy
• Risk Assessment
• Selective Prediction

Code references

• hnurxn/Code-Uncertainty

Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs — Code-MUE quantifies Code LLM uncertainty by analyzing the semantic diversity of execution-based outputs, bridging the syntax-semantics gap.
• Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty — Structural uncertainty quantifies LLM logical reasoning consistency by measuring the stability of self-preference-induced rankings over sampled solutions.
• CodeChat-Eval: Evaluating Large Language Models in Multi-Turn Code Refinement Dialogues — LLMs significantly degrade in functional correctness during multi-turn code refinement, especially with semantic or additive changes.
• The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation — Incidental prompt cues systematically steer LLM algorithm selection, creating an "invisible lottery" over code quality beyond functional correctness.
• Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs — LLM uncertainty estimation in multilingual settings is significantly impacted by reasoning language and model scale.
• Vibe-Coding: Feedback-Based Automated Verification with no Human Code Inspection, a Feasibility Study — Precise, fine-grained feedback is critical for reliable LLM-generated code verification in runtime-adaptive systems.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.