Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

2026-06-05 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

T2J-Bench is a new benchmark for codebase conversion, specifically from PyTorch training codebases to JAX, that reformulates conversion as "transfer under a fixed equivalence contract." Developed by researchers from USC and Google Cloud AI Research, it evaluates conversions through three ordered stages: Spec (interface admissibility), Numeric (forward outputs, losses, gradients), and Behavioral (short training dynamics). Across 355 blind conversion attempts, the best system achieved only a 26.7–28.9% overall pass rate, despite Spec pass rates up to 91.1%. Findings indicate that a 4.7x token-budget spread yielded only a 2.2x pass-rate spread, and systems systematically overestimated success by 66.6–97.8 percentage points, suggesting failures stem from contract-misaligned self-validation rather than limited budget or backbone strength.

Key takeaway

For AI Scientists and ML Engineers developing coding agents for codebase conversion, you must prioritize aligning agent self-validation with external semantic contracts. Your current local checks likely overstate success, as demonstrated by the 66.6–97.8 percentage point gap observed in T2J-Bench. Implement robust source-comparison loops and explicit seam discipline for cross-paradigm conversions to ensure true behavioral equivalence, not just structural plausibility.

Key insights

Coding agents over-trust local validation, failing semantic contracts in codebase conversion despite surface checks.

Principles

• Equivalence verification must be external and finite.
• Semantic preservation is harder than structural plausibility.
• Self-validation often substitutes shape for semantics.

Method

T2J-Bench verifies codebase conversion via three stages: Spec (interface), Numeric (outputs, gradients), and Behavioral (training dynamics) against a fixed equivalence contract derived from the source.

In practice

• Implement source-comparison loops in agent self-tests.
• Design explicit seam discipline for cross-paradigm work.
• Audit agent validation routines for contract misalignment.

Topics

• Codebase Conversion
• AI Agents
• Benchmarking
• Observational Equivalence
• PyTorch to JAX
• Semantic Preservation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Can coding agents relicense open source through a “clean room” implementation of code? — AI-assisted code rewrites challenge traditional "clean room" definitions and open-source licensing compliance.
• QuantCode-Bench: A Benchmark for Evaluating the Ability of Large Language Models to Generate Executable Algorithmic Trading Strategies — Generating trading strategies requires LLMs to master financial logic, API usage, and semantic alignment beyond mere syntax.
• SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Tasks — Code agents struggle with complex, multi-file code refactoring, indicating a gap in repository-level reasoning beyond functional correctness.
• I Built an AI That Modernizes Legacy COBOL — And Then Made It Prove Its Own Work — Separating correctness (compiler theory) from intent (LLM semantics) is key to reliable legacy code modernization.
• Many SWE-bench-Passing PRs Would Not Be Merged into Main — Automated code benchmarks like SWE-bench significantly overstate AI agent usefulness without human review and iteration.
• Build custom code-based evaluators in Amazon Bedrock AgentCore — Custom code-based evaluators in Amazon Bedrock AgentCore enable deterministic, domain-specific quality checks for agentic applications.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.