DSMentor: Curriculum-Guided Inference with Online Memory for Data-Science LLM Agents

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, quick

Summary

DSMentor is an inference-time optimization framework designed to improve large language model (LLM) agent performance on complex data science tasks by implementing curriculum learning. This framework addresses the common issue of LLM agents overlooking task order during inference. DSMentor, guided by a mentor and supported by an online long-term memory, organizes problems by difficulty, retains prior experiences, and leverages them to guide subsequent reasoning. Extensive experiments on the DSEval and QRData benchmarks demonstrate its effectiveness. When used with Claude-3.5-Sonnet, DSMentor improves pass rates by up to 5.2% over baseline agents and achieves an 8.8% gain compared to GPT-4 utilizing Program-of-Thoughts prompting, validating curriculum-based inference strategies for LLM agents.

Key takeaway

For Machine Learning Engineers developing LLM agents for complex data science problems, DSMentor demonstrates that curriculum-guided inference with online memory significantly improves code generation pass rates. If your agents struggle with task order or retaining prior experiences, you should explore implementing a similar framework. This approach, which organizes problems by difficulty and leverages past solutions, offers a concrete path to enhance LLM agent reliability and performance on benchmarks like DSEval and QRData.

Key insights

Curriculum learning and online memory significantly boost LLM agent performance on data science tasks.

Principles

• Task order impacts LLM agent performance.
• Curriculum learning enhances LLM problem-solving.
• Online memory improves reasoning guidance.

Method

DSMentor applies curriculum learning to organize data science problems by difficulty, retaining prior experiences in online memory to guide subsequent LLM reasoning.

In practice

• Apply curriculum learning to LLM agent inference.
• Integrate online memory for experience retention.
• Benchmark LLM agents on DSEval and QRData.

Topics

• LLM Agents
• Curriculum Learning
• Online Memory
• Data Science
• DSEval Benchmark
• Claude-3.5-Sonnet

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, Data Scientist

Related on AIssential

• Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents — HORMA uses a hierarchical, file-system-like memory to enable efficient, detailed context retrieval for LLM agents in long-horizon tasks.
• SciR: A Controllable Benchmark for Scientific Reasoning in LLMs — SciR is a new benchmark controlling scientific reasoning difficulty for LLMs via extraction and inference axes.
• DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems — Multimodal, multi-query benchmarks are crucial for evaluating LLM-based data science agents on real-world, open-ended problems.
• AI Trends 2026: OpenClaw Agents, Reasoning LLMs, and More with Sebastian Raschka - #762 — LLM progress is shifting from raw model scaling to reasoning-focused post-training, inference-time techniques, and better tool integration.
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective — EvoMemBench systematically evaluates LLM agent memory across scope and content, revealing current limitations and specialized strengths.
• RaMem: Contextual Reinstatement for Long-term Agentic Memory — RaMem prevents "context collapse" in LLM agent memory by reinstating original episodic context to validate retrieved fragments.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.