TSAQA: Time Series Analysis Question And Answering Benchmark

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

TSAQA is a new unified benchmark designed to expand task coverage and evaluate diverse temporal analysis capabilities for time series question answering. This benchmark integrates six distinct tasks, ranging from conventional analysis like anomaly detection and classification to advanced functions such as characterization, comparison, data transformation, and temporal relationship analysis. The dataset comprises 210,000 samples spanning 13 domains and utilizes diverse formats, including true-or-false, multiple-choice, and a novel puzzling format. Zero-shot evaluation revealed that TSAQA remains challenging for current Large Language Models, with the best-performing commercial model, Gemini-2.5-Flash, achieving only 65.08% average accuracy. While instruction tuning improved open-source models, LLaMA-3.1-8B still showed significant room for improvement, indicating that TSAQA extends beyond general-purpose LLMs and also evaluates language-capable time series foundation models.

Key takeaway

For machine learning engineers developing time series question answering systems, you should integrate the TSAQA benchmark into your evaluation pipeline. This new benchmark highlights significant gaps in current LLM capabilities for temporal analysis, even for models like Gemini-2.5-Flash and LLaMA-3.1-8B. Your focus should shift towards specialized time series foundation models or advanced instruction tuning to achieve robust performance on diverse analytical tasks.

Key insights

TSAQA provides a comprehensive benchmark for time series QA, revealing current LLM limitations in temporal analysis.

Principles

• Current LLMs struggle with time series QA.
• Diverse tasks are crucial for temporal analysis evaluation.
• Specialized benchmarks extend beyond general-purpose LLMs.

Method

TSAQA integrates 6 diverse time series analysis tasks, using 210k samples across 13 domains with TF, MC, and puzzling formats, evaluated via zero-shot and instruction tuning.

In practice

• Evaluate LLMs on complex time series analysis tasks.
• Benchmark time series foundation models (TSFMs).
• Access the dataset on Hugging Face for research.

Topics

• Time Series Analysis
• Question Answering
• LLM Benchmarking
• Time Series Foundation Models
• Temporal Data Analysis
• Anomaly Detection

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning — LLM agents struggle with multi-turn, decision-oriented time series analysis due to memory, uncertainty, and domain reasoning gaps.
• Beyond Tokenization: Direct Timestep Embedding and Contrastive Alignment for Time-Series Question Answering — Direct timestep embedding and contrastive alignment overcome LLM tokenization issues for time-series data, improving TSQA performance.
• Representing Time Series as Structured Programs for LLM Reasoning — Representing time series as structured symbolic programs enables LLMs to reason effectively without fine-tuning.
• Harnessing Generalist Agents for Contextualized Time Series — TimeClaw's agentic harness framework enables LLMs to perform robust, contextualized time series reasoning by providing native temporal runtime support.
• Gemini tops benchmarks, again — Hardware-accelerated LLMs offer significant speed advantages over traditional inference methods, despite potential initial quality trade-offs.
• Beyond Tokenization: Direct Timestep Embedding and Contrastive Alignment for Time-Series Question Answering — Direct timestep embedding and contrastive alignment overcome LLM tokenization limits for time-series data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.