Towards Functional Correctness of Large Code Models with Selective Generation

2026-06-16 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

This paper introduces a novel approach to enhance the functional correctness of large code models by addressing code hallucination. The authors propose a "selective code generator" that leverages automatically generated unit tests via dynamic code analysis tools, specifically fuzzing methods like Atheris. This system defines "α-code entailment" to measure functional correctness, where a generated code snippet is considered correct if it satisfies a high probability (1-α) of matching the true code's input-output behavior. The learning algorithm for this selective generator provides a theoretical guarantee to control the false discovery rate (FDR-CE) of hallucinated code. The method, termed FuzzEval for evaluation, demonstrates superior performance over open and closed code generators, including GPT-4o, Gemini 1.5 Pro, and DeepSeek R1, across various datasets and programming languages, showing improved controllability of hallucination and selection efficiency.

Key takeaway

For AI Engineers developing or deploying code generation models, integrate selective generation to enhance functional correctness and trustworthiness. Leverage dynamic code analysis tools like fuzzing for automatic unit test generation, establishing a robust "α-code entailment" framework. This controls code hallucination rates with theoretical guarantees, improving reliability and reducing risks. Prioritize selecting appropriate scoring functions for optimal calibration.

Key insights

Code hallucination in LLMs can be controlled by a selective generator using fuzzing-generated unit tests and "α-code entailment".

Principles

• Code's executable nature enables automatic unit test generation.
• "α-code entailment" defines functional correctness via probabilistic execution.
• Selective generation can control hallucination rates with theoretical guarantees.

Method

A selective generator learns a selection function s(x) = 1(f(x, G(x)) >= τ) by minimizing τ while bounding the FDR-CE using fuzzing-generated unit tests and a binomial tail bound.

In practice

• Use fuzzing tools to generate extensive unit tests for code evaluation.
• Implement "α-code entailment" to quantify functional correctness.
• Apply selective generation to filter out uncertain code outputs.

Topics

• Code Generation
• Large Language Models
• Code Hallucination
• Fuzzing
• Unit Test Generation
• Selective Prediction
• Functional Correctness

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Coding After Coders: The End of Computer Programming as We Know It — Automated testing in software development uniquely tethers AI agents to reality, mitigating hallucination risks.
• Code Is More Than Text: Uncertainty Estimation for Code Generation — Code generation uncertainty estimation requires specialized methods accounting for code's unique properties, outperforming natural language-derived baselines.
• Evidence Graph Consistency in Retrieval-Augmented Generation: A Model-Dependent Analysis of Hallucination Detection — Graph-based RAG hallucination detection is model-dependent, reversing for GPT-class models due to differing hallucination styles.
• Claude Opus 4.8: Capabilities and Reactions — Comprehensive AI model evaluation requires diverse benchmarks, user feedback calibration, and understanding model "welfare" for accurate assessment.
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can actively verify and debug competitive programming code, but current inference costs limit practical deployment.
• Why wait until the end to realize your model’s code won’t actually run? — Code generation requires dynamic, incremental reasoning, not just upfront planning, due to emergent complexity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.