Articulating Assumptions in AI-Generated Scientific Analyses through Task Decomposition

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Science & Research — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences, Physical Sciences & Chemistry · Depth: Expert, extended

Summary

A multi-agent framework called "quantity grounded semantic differencing" addresses transparency and reproducibility issues in LLM-generated scientific analysis code. This framework decomposes the code generation process into distinct agents responsible for helper selection, code generation, execution-based repair, implementation tracing, and post-generation critique. It also introduces a Golden Axe Oracle module to inspect and clarify ambiguities in initial user instructions before code generation. Validated on representative collider physics analyses within the LHCO domain, the modular task decomposition significantly enhances both transparency and reliability. This approach enables substantially smaller models, such as Qwen family 14B and 32B parameter models, to execute the complete workflow effectively, a notable improvement over previous methods requiring approximately 70B parameter models.

Key takeaway

For AI Scientists or Research Engineers focused on generating reliable scientific analysis code with LLMs, you should adopt a multi-agent framework that explicitly decomposes code generation, execution, tracing, and validation. This approach, especially with pre-generation ambiguity clarification and post-generation semantic critique, significantly improves reproducibility and allows for the use of smaller, more accessible LLMs (e.g., 14B parameter models), reducing computational overhead and increasing auditability of scientific workflows.

Key insights

Multi-agent task decomposition improves LLM-generated scientific code transparency and enables smaller models.

Principles

• Decompose complex LLM tasks into auditable, specialized stages.
• Explicitly clarify user ambiguities before code generation.
• Separate domain-specific knowledge from generic orchestration logic.

Method

The framework uses specialized agents for helper selection, code generation, execution-based repair, implementation tracing, and post-generation critique, complemented by an ambiguity clarification module.

In practice

• Utilize a helper selector to guide LLM code generation.
• Implement a tracer to reconstruct "as-implemented" specifications.
• Employ an oracle module to pre-emptively resolve prompt ambiguities.

Topics

• LLM Code Generation
• Multi-Agent Systems
• Scientific Computing
• High Energy Physics
• Code Transparency
• Prompt Engineering
• LHCO Domain

Code references

• AHamamd150/LLM_For_scientific_Codes

Best for: AI Scientist, Research Scientist, AI Engineer

Related on AIssential

• From Data to Theory: Autonomous Large Language Model Agents for Materials Science — Autonomous LLM agents can perform end-to-end scientific theory development, from equation selection to code execution and validation.
• InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis — Procedurally generated benchmarks offer controlled, scalable evaluation for LLM scientific reasoning and abstention.
• GenoMAS: A Multi-Agent Framework for Scientific Discovery via Code-Driven Gene Expression Analysis — GenoMAS automates gene expression analysis using a multi-agent LLM framework that balances structured workflows with adaptive, code-driven problem-solving.
• Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning — LLM agents struggle with complex, multi-stage scientific reasoning tasks like chemical cost estimation, even with tool access.
• Battery-Sim-Agent: Leveraging LLM-Agent for Inverse Battery Parameter Estimation — LLM agents can reframe inverse scientific problems as reasoning tasks, outperforming traditional black-box optimization.
• Towards Generalist Agents for Accelerating Scientific Discovery171 — LLMs can accelerate scientific discovery by generating and refining hypotheses within agentic, iterative workflows.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.