Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q

2018-08-06 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Expert, long

Summary

A new study benchmarks four Large Language Models (LLMs) on repairing 67 real Qiskit defects from the Bugs4Q dataset across six pinned Qiskit releases: 0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, and 2.3.1. The research reveals that quantum benchmarks like Bugs4Q are susceptible to "silent label inversion," where benchmark entries become invalid without explicit errors as Qiskit versions evolve. Evaluating GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini, the study found GPT-5.4 achieved the highest pass@10 at 48.8%, with GPT-5.4-mini at 47.3%, GPT-5o-mini at 30.3%, and GPT-4o-mini at 22.6%. Performance peaked at Qiskit 0.45.0 and significantly declined after the Qiskit 1.0 transition. Crucially, 64% of observed successful repairs occurred on benchmark entries that were invalid for the target version, and many failures were due to API incompatibility rather than logical repair errors. The authors release a re-validated, version-pinned Bugs4Q benchmark.

Key takeaway

For Machine Learning Engineers evaluating Large Language Models for quantum program repair, you must adopt version-pinned evaluation protocols. Your benchmark results will be unreliable if you don't account for Qiskit version changes, which cause silent label inversions and API incompatibilities. Ensure you track both repair correctness and environmental executability. Consider developing multi-version-aware prompting strategies to improve model robustness across different Qiskit releases.

Key insights

Quantum program repair benchmarks require version-pinned validation to accurately assess LLM capabilities due to silent label inversion and API evolution.

Principles

• Benchmark validity is version-dependent.
• Silent label inversion can invalidate benchmarks.
• Environment compatibility impacts repair success.

Method

The study constructed a deduplicated 67-defect Bugs4Q benchmark, authored executable tests, and applied a version-pinned validation protocol across six Qiskit releases to evaluate LLM repair candidates.

In practice

• Pin Qiskit versions for consistent testing.
• Track executability alongside repair logic.
• Investigate multi-version-aware prompting.

Topics

• Large Language Models
• Quantum Program Repair
• Qiskit
• Bugs4Q Benchmark
• API Compatibility
• Benchmark Validation

Code references

• slrrla/APR-new-experiment
• Z-928/Bugs4Q-Framework
• Z-928/Bugs4Q-NA

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• On the Reproducibility of Quantum Software Defect Datasets: A Case Study of Bugs4Q — Quantum software defect datasets face severe reproducibility degradation due to rapid ecosystem evolution, often requiring source-level patches.
• Statevector-Referenced Geometry Survival of a Four-Qubit ZZ Quantum Kernel on IBM Quantum Hardware: A Fixed-Subset Diagnostic Across Three Execution Configurations — Quantum kernel fidelity on hardware does not guarantee task relevance; hardware distortion dominates.
• Qiskit Code Migration with LLMs — Taxonomy-guided RAG with LLMs significantly enhances Qiskit code migration accuracy and reduces hallucinations.
• Large-Language-Model Discovery of Quantum LDPC Codes through Structured Concept Evolution — Large language models can discover complex quantum LDPC codes by evolving structured algebraic concepts.
• Benchmarking Instance-Dependent Label Noise with Controlled Corruptions — CILN generates explicit, controllable instance-dependent label noise via input corruptions, revealing new algorithm failure modes.
• MPC-Patch-Bench: Security-Aware LLM Code Patch for Multi-Party Computation — Evaluating LLM code repair for MPC requires specialized benchmarks addressing cryptographic safety and numerical fidelity.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.