Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Expert, quick

Summary

An open-source framework has been introduced to evaluate AI agents powered by open-weight large language models (LLMs) for data preparation in governance-restricted research. This framework addresses the challenge of using LLMs with sensitive personal data, where transmitting data to cloud-based services is prohibited. It comprises a curated ground-truth dataset, including cleaning scripts for six sweeps of data from a British cohort study, task definitions like category harmonization and multi-wave merging, and automated routines for evaluating LLM-produced R code and outputted data. Benchmarking 31-35B parameter models on consumer-grade hardware across 20 data preparation tasks (creating 102 variables) showed an "average task completion" up to 87.9%. These results indicate a viable path for AI-assisted data preparation using locally deployable open-weight LLMs in settings with strict data governance.

Key takeaway

For research scientists or data scientists handling sensitive personal data under strict governance, you should consider locally deploying open-weight large language models for data preparation. This approach bypasses restrictions on transmitting data to cloud services, offering a secure and effective alternative. Models in the 31-35B parameter range, running on consumer-grade hardware, demonstrated up to 87.9% task completion, indicating their readiness for real-world application in such environments. Explore the RRBench framework to evaluate and integrate these agentic coding solutions into your workflows.

Key insights

Open-weight LLMs running locally can effectively automate data preparation for sensitive data in governance-restricted research settings.

Principles

• Data governance restricts cloud LLM use.
• Local open-weight LLMs enable sensitive data processing.
• Agentic LLM evaluation needs ground-truth and automation.

Method

Evaluate open-weight LLM agents using a framework with ground-truth cleaning scripts, defined data preparation tasks like harmonization and merging, and automated routines to assess generated R code and outputted data.

In practice

• Deploy open-weight LLMs for sensitive data preparation.
• Use 31-35B parameter models on consumer-grade hardware.
• Adopt the RRBench framework for agent evaluation.

Topics

• Agentic Coding
• Open-weight LLMs
• Data Preparation
• Data Governance
• Longitudinal Studies
• LLM Evaluation

Code references

• UCL-ARC/RRBench

Best for: AI Engineer, Machine Learning Engineer, CTO, AI Scientist, Research Scientist, Data Scientist

Related on AIssential

• Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks — Open-weight LLMs deployed locally offer a viable path for AI-assisted data preparation in governance-restricted research.
• Agentic coding without the cloud: evaluating open-weight large language models on longitudinal data preparation tasks — Open-weight LLMs offer a viable path for AI-assisted data preparation in governance-restricted research settings.
• [AINews] GLM-5.2: the top Frontend Coding model in the world, IndexShare for Speculative Decoding — Open-weight models can achieve frontier-level coding and agentic performance with efficient long-context handling.
• Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities — Current LLMs, even with agentic support, significantly underperform on real-world complex data analysis tasks.
• MiniMax M3 vs GLM-5.2 vs Kimi K3: which open-weight model should you actually self-host for agentic — Leaderboard benchmarks often mislead real-world self-hosting decisions for agentic coding models.
• RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications — Agentic coding product evaluation demands native non-English task specifications and rigorous trajectory auditing for model substitution.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.