CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

2026-07-11 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Research Methodology & Innovation · Depth: Expert, extended

Summary

The CausalDS benchmark introduces a novel approach to evaluating causal reasoning in agentic data-science workflows, addressing gaps in existing benchmarks by integrating symbolic causal reasoning, realistic data analysis, and advanced tool use. It generates synthetic "scenes" from Structural Causal Models (SCMs), observational data, and natural-language stories, incorporating an observation layer to vary data difficulty and empirically grounding its composition. Tasks span Pearl's three rungs (associational, interventional, counterfactual), requiring data science coding, uncertainty quantification, and epistemic abstention for non-identifiable queries. Initial evaluation of six LLMs on a 100-scene exam revealed Claude Opus 4.8 as the top performer with a CausalDSScore of 0.278. Models showed significant differentiation in uncertainty quantification (95% ATE interval coverage 20.0%–71.4%) and abstention (18.8%–75.0%), with frontier models generally demonstrating superior epistemic skills and tool-use efficiency.

Key takeaway

For AI Scientists and Machine Learning Engineers evaluating or deploying LLMs for complex data-science tasks involving causality, you must assess models beyond basic content correctness. Prioritize models that demonstrate strong epistemic skills, such as accurately quantifying uncertainty and recognizing when a causal claim is not identifiable. Additionally, evaluate their tool-use efficiency and interaction style, as these factors are crucial for reliable causal analysis and avoiding "causal parrot" failures in real-world applications.

Key insights

CausalDS benchmarks LLM data-science agents on integrated causal reasoning, data analysis, and tool use, including abstention for non-identifiable queries.

Principles

• Causal reasoning requires integrated symbolic, quantitative, and epistemic skills.
• Abstaining on non-identifiable queries is a critical, distinct skill.
• Observation layers can vary data-analysis difficulty without altering causal structure.

Method

CausalDS generates scenes from sampled SCMs, synthesizes observational data, maps nodes to domain variables with natural-language stories, and derives tasks across Pearl's hierarchy, scoring deterministically with abstention.

In practice

• Evaluate LLMs on uncertainty quantification and abstention.
• Design benchmarks with varied data observation difficulties.
• Assess agent tool-use efficiency and interaction style.

Topics

• Causal Inference
• Large Language Models
• Data Science Agents
• Benchmarking
• Structural Causal Models
• Uncertainty Quantification
• Epistemic Abstention

Code references

• andleb/causalds
• y0-causal-inference/y0

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• CausalDS: Benchmarking Causal Reasoning in Data-Science Agents — CausalDS benchmarks LLM agents on causal reasoning, data science, and tool use across Pearl's three rungs.
• Do you agree with Judea that learning from data is not everything? [D] — Purely data-driven machine learning is mathematically limited in inferring causation from correlation.
• Instrumented data for causal scientific machine learning — Instrumented data integrates mechanistic models and uncertainty into each datum for causal scientific machine learning.
• CausalDS: Benchmarking Causal Reasoning in Data-Science Agents — CausalDS benchmarks LLM agents' causal reasoning, data science, and tool use across Pearl's three rungs.
• Breaking the Chain: A Causal Analysis of LLM Faithfulness to Intermediate Structures — LLMs' apparent faithfulness to intermediate reasoning structures is fragile; they often fail to update predictions after causal intervention.
• The Future of Agentic Data Science — Agentic data science, by integrating decision science and robust causal/Bayesian methods, makes rigorous analysis accessible and verifiable.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.