Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

2022-04-27 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

DataGovBench is a new benchmark designed to evaluate Large Language Models (LLMs) in real-world data analysis scenarios, addressing limitations of existing benchmarks that focus on small, single tables. Derived from governmental open data, DataGovBench features 178 unique datasets with an average of 210K rows and 18 columns, including multi-tabular structures (over 36%) and external knowledge (over 57%). It comprises two tasks: Table QA, involving 211 complex decomposable questions (totaling 414 individual questions) requiring textual or visual answers, and Table Insight, which evaluates expert-level insight generation from 6 datasets. Comprehensive experiments with leading LLMs, both standalone and with agentic frameworks like Answer Agent and Insight Agent, reveal significant performance gaps, with top scores remaining below 0.4 for Table QA and below 0.5 for Table Insight, indicating current LLM systems are far from real-world demands.

Key takeaway

For AI Scientists and ML Engineers developing LLM-based data analysis agents, this benchmark highlights critical performance deficiencies in handling real-world data complexities. Your focus should shift towards improving narrative-level reasoning and accurate fact retrieval from large, multi-tabular datasets with rich metadata and external knowledge. Utilize DataGovBench to rigorously evaluate and advance agents capable of both complex query answering and proactive insight generation, moving beyond simple fact retrieval.

Key insights

Current LLMs, even with agentic support, significantly underperform on real-world complex data analysis tasks.

Principles

• Real-world data analysis requires handling large, multi-tabular datasets with external knowledge.
• Agentic frameworks are crucial for improving LLM performance on complex data analysis tasks.
• Feature type-specific table serialization enhances LLM understanding of large tables.

Method

DataGovBench construction involves curating large open datasets, annotating QA pairs via an LLM-human-in-the-loop pipeline, and compiling ground-truth insights from expert reports. The Answer Agent uses serialization, self-correcting code generation, and VLM/MLLM reflection.

In practice

• Implement agentic frameworks to enhance LLM capabilities for complex data analysis.
• Utilize feature type-specific table serialization for efficient LLM processing of large tables.
• Focus LLM development on narrative-level reasoning and accurate fact retrieval from complex data.

Topics

• Large Language Models
• Data Analysis Benchmarking
• Agentic AI Systems
• Table Question Answering
• Exploratory Data Analysis
• DataGovBench

Code references

• SoHasegawa/datagovbench
• ckan/ckanapi
• streamlit/streamlit

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning — LLMs excel at orchestrating graph algorithms via tools but struggle with compositional multi-hop graph reasoning.
• DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems — Multimodal, multi-query benchmarks are crucial for evaluating LLM-based data science agents on real-world, open-ended problems.
• How do One Big Table and AI fit together — Context engineering is crucial for AI to deliver consistent, low-latency, and cost-effective answers.
• ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments — ScholarQuest benchmark reveals LLM-based search agents significantly underperform in academic literature exploration, despite outperforming baselines.
• TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering — LLMs struggle with implicit predictive reasoning in tabular data, often misinterpreting intent as simple retrieval.
• Towards Verifiable Agentic Data Science: Solving Irregular TSQA Via Tool-Grounded Reasoning — IRTS-ToolBench bridges the gap in evaluating LLMs on real-world irregular time series data by offering a standardized benchmark.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.