Scenario Generation for Testing of Autonomous Driving Systems Using Real-World Failure Records

2026-06-30 · Source: Artificial Intelligence · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Advanced, quick

Summary

A new scenario generation pipeline is proposed for testing Autonomous Driving Systems (ADS). It leverages categorical and contextual information from historical real-world failure records in natural language format. This modular, LLM-based approach creates synthetic scenarios compatible with specific system testing constraints. The method was successfully applied to generate diverse scenarios for autonomous navigation testing on the Metadrive simulator, utilizing NHTSA ADS crash records. The pipeline accurately and diversely generates scenarios, incorporating 4 road types, 3 non-ego vehicle movement types, and on-road anomalies like working zones. This approach revealed interesting system failures within a limited testing budget of 20 scenarios, demonstrating its efficacy in pre-deployment failure discovery.

Key takeaway

For MLOps Engineers or AI Engineers developing autonomous driving systems, integrating this LLM-based scenario generation pipeline can significantly enhance pre-deployment testing. By leveraging real-world failure records, your team can discover critical system failures more efficiently and diversify test cases beyond purely mathematical models. This approach optimizes your testing budget, ensuring robust system behavior with fewer manually designed templates.

Key insights

A modular LLM-based pipeline generates diverse ADS test scenarios from real-world failure records.

Principles

• Historical failure records are reliable sources for scenario generation.
• Modular LLM-based generation adapts to testing constraints.
• Diverse scenario elements enhance failure discovery.

Method

The approach uses LLMs to synthetically generate scenarios from categorical and contextual natural language information in historical failure records, ensuring compatibility with system testing constraints.

In practice

• Generate diverse test scenarios for autonomous navigation.
• Utilize NHTSA ADS crash records for real-world data.
• Integrate 4 road types and 3 non-ego vehicle movements.

Topics

• Autonomous Driving Systems
• Scenario Generation
• LLM Applications
• Simulation Testing
• Failure Analysis
• Metadrive Simulator
• NHTSA Crash Records

Code references

• anjaliParashar/crash2scenario

Best for: NLP Engineer, AI Scientist, Research Scientist, Robotics Engineer, MLOps Engineer, AI Engineer

Related on AIssential

• RiskFlow: Fast and Faithful Safety-Critical Traffic Scenario Generation — RiskFlow generates realistic safety-critical traffic scenarios rapidly by transforming action sequences in a single forward pass, avoiding iterative denoising.
• SPHINX: First Explain, Then Explore — Adversarial scenario generation should diagnose policy failures using explainable AI, then explore targeted improvements.
• ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving — ScenePilot generates critical, physically solvable autonomous driving scenarios that expose autonomy stack failures.
• Nobody Is QA Testing Their LLM Apps (That's Going to Be a Problem) — Testing probabilistic AI systems demands a shift from deterministic assertions to statistical quality guarantees.
• RiskFlow: Fast and Faithful Safety-Critical Traffic Scenario Generation — RiskFlow efficiently generates realistic safety-critical traffic scenarios by transforming Gaussian action sequences in a single pass with output-space guidance.
• Your Agent Has a Genome: Sequence-Level Behavioral Analysis and Runtime Governance of LLM-Powered Autonomous Agents — Encoding agent actions into XEPV sequences enables quantitative behavioral analysis and runtime governance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.