A semantic mutation metric for metamorphic relation adequacy in scientific computing programs

2026-05-17 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new metric, the Semantic Mutation Score (SMS), is proposed to enhance metamorphic relation adequacy in scientific computing programs. Addressing the test-oracle problem, SMS overcomes the limitation of classical Mutation Score (MS) by incorporating five domain-semantic operators: Conservation Erosion, Operator Substitution, Hyperparameter, Trajectory Flip, and Structural Injection. The research demonstrates SMS's backward compatibility, showing it degenerates to MS in a characterized limit. Evaluation involved a 12-PUT x 5-MP design across four single-output float-to-float classes (numeric, probabilistic, surrogate, machine-learning), using a three-layer attribution classifier. Results indicate a medium-effect range for Cliff's delta, with cross-source LLM pooling not significantly altering the effect. Crucially, AST-level overlap between LLM-generated and default cosmic-ray syntactic mutants was small, revealing that Hyperparameter, Structural Injection, and Trajectory Flip classes are unreachable by default first-order syntactic configurations.

Key takeaway

For Software Engineers or ML Engineers developing and testing scientific computing programs, traditional syntactic mutation testing may overlook critical domain-specific faults. You should consider adopting the Semantic Mutation Score (SMS) to achieve more comprehensive test adequacy. This metric, with its five domain-semantic operators, helps identify fault classes like Hyperparameter and Trajectory Flip that are unreachable by default first-order syntactic methods, ensuring greater reliability in your scientific applications.

Key insights

Semantic Mutation Score (SMS) offers a domain-semantic adequacy metric for metamorphic testing in scientific computing, addressing syntactic mutation limitations.

Principles

• Domain semantics are crucial for testing scientific code.
• SMS is backward-compatible with classical MS.
• Syntactic mutations miss specific semantic fault types.

Method

A 12-PUT x 5-MP design, using a three-layer attribution classifier, evaluated SMS across numeric, probabilistic, surrogate, and machine-learning program classes, comparing LLM-generated mutants to syntactic pools.

In practice

• Apply SMS for robust test adequacy in scientific computing.
• Explore domain-semantic operators for mutation testing.
• Utilize LLM-generated mutants to uncover new fault classes.

Topics

• Metamorphic Testing
• Mutation Testing
• Scientific Computing
• Software Engineering
• Test Adequacy
• Semantic Mutation Score
• Large Language Models

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Software Engineer

Related on AIssential

• MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis — Functional coupling between methods can automate metamorphic relation construction and MTC generation, addressing the test oracle problem.
• XDomainBench: Diagnosing Reasoning Collapse in High-Dimensional Scientific Knowledge Composition — LLMs exhibit systematic reasoning collapse in complex, interactive scientific knowledge composition tasks.
• STMutants: A Mutation Testing Dataset for Structured Text Programs in Industrial Automation — STMutants provides the first public mutation testing benchmark for IEC 61131-3 Structured Text, enabling reproducible research and LLM evaluation for PLC software.
• A Set-Theoretic Approach to Detecting Logic Bugs in DBMS Inner Join Optimizations — INNER/NATURAL JOINs are semantically equivalent to set intersection operations, forming a basis for logic bug detection.
• Reference-Free Evaluation of Taxonomies — Reference-free metrics for taxonomy quality can assess robustness and logical adequacy without gold standards.
• DySem: Uncovering Dynamic Semantic Components via Multilingual Consensus for Calculating Semantic Textual Similarity — DySem uses multilingual consensus to find dynamic, sample-specific semantic dimensions for improved STS in LLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.