TAG: A Lightweight Framework for Test-Driven Agentic Artifact Generation

2026-06-26 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Cybersecurity & Data Privacy · Depth: Advanced, extended

Summary

A lightweight framework named TAG is introduced for test-driven agentic generation of structured artifacts using Large Language Models (LLMs), addressing reliability challenges in production deployments. This framework, deployed at Microsoft Sentinel, operates on the principle "LLMs generate, we validate." It features test-driven generation where LLMs refine outputs based on indicative error messages, deterministic tests for programmatic checks (e.g., schema, syntax), and LLM-based tests for semantic evaluation (e.g., intent alignment). Crucially, LLM-based judges are calibrated against human expert decision distributions to ensure professional-grade validation. The framework is demonstrated on security domain artifacts like KQL query generation, MITRE ATT&CK mapping, and entity mapping, achieving a 35.6% end-to-end yield from 931 hypotheses, with KQL generation being the primary bottleneck.

Key takeaway

For AI Engineers building production systems that rely on LLM-generated structured artifacts, this framework offers a robust approach to ensure reliability. Implement a test-driven generation loop with both deterministic and expert-calibrated LLM-based semantic tests. Prioritize actionable error messages and consider multi-stage pipelines to isolate faults and progressively filter artifacts, significantly improving the quality and deployability of LLM outputs.

Key insights

A test-driven framework validates LLM-generated artifacts through iterative refinement and expert-calibrated semantic judges.

Principles

• LLMs generate, humans validate.
• Test failures drive LLM refinement.
• Calibrate LLM judges to human experts.

Method

An agent uses tools to generate artifacts, which are then validated by a test suite. Failed tests provide error messages for iterative self-correction, with LLM judges calibrated to expert decisions.

In practice

• Generate KQL queries.
• Map to MITRE ATT&CK.
• Create entity schemas.

Topics

• LLM Agents
• Test-Driven Development
• Artifact Generation
• LLM Evaluation
• Microsoft Sentinel
• Security Automation

Best for: AI Architect, AI Scientist, Research Scientist, Machine Learning Engineer, AI Engineer, AI Security Engineer

Related on AIssential

• Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automation — Multi-agent LLMs can automate testbench generation, creating high-quality data for efficient Verilog code generation.
• Building an AI Dream Analysis Engine, Part 2: Designing a Production-Ready LLM Pipeline — Production-ready LLM applications require a multi-stage pipeline, not just direct API calls, to ensure reliability and structured outputs.
• StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows — Artifact-centered evaluation reveals workflow failures LLM agents miss in structural engineering.
• Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios — LLM agents struggle with end-to-end software generation, often preferring monolithic designs over modularity.
• LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle — Active, specification-aware fuzzing with an agentic oracle effectively uncovers deep semantic bugs in LLM agent frameworks.
• Understanding AI Agents: What Actually Works When Building AI Products — Building robust AI products requires a structured approach addressing LLM limitations through wrapping layers, rigorous evaluation, and continuous operational feedback.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.