Automatic Testing of Interacting Autonomous Vehicles

2016-01-01 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

EVITA, an innovative approach for automatic testing of interacting autonomous vehicles (AVs), addresses the limitations of single-AV scenario-based testing. Current methods overlook critical behaviors emerging from multi-AV interactions, a significant challenge given the >1,400 AVs from >80 manufacturers testing in the USA and 16,000 licenses in China. EVITA utilizes multi-objective optimization to generate diverse, safety-critical scenarios while minimizing complexity. Evaluated against DoppelTest across six road networks using FrenetiX and Baidu Apollo simulators, EVITA consistently triggers a greater variety of AV interactions, identifies more diverse collisions, and exposes more traffic rule violations, including speeding and traffic light infractions. Furthermore, it generates simpler scenarios with fewer AVs and maintains high efficiency, producing 200-400 valid scenarios in 3 CPU-hours for FrenetiX.

Key takeaway

For AV Test Engineers developing robust autonomous driving systems, relying solely on single-AV scenario testing is insufficient and risks overlooking critical emergent behaviors. You should integrate multi-objective optimization approaches like EVITA into your test generation pipeline. This will enable you to systematically uncover a wider range of safety-critical interactions and traffic rule violations with simpler, more debuggable scenarios, significantly enhancing the thoroughness and efficiency of your validation efforts.

Key insights

Multi-objective optimization can systematically generate diverse, safety-critical scenarios by testing emergent behaviors from interacting autonomous vehicles.

Principles

• Multi-objective optimization balances test coverage and scenario complexity.
• Emergent behaviors from multi-AV interactions reveal unique safety issues.
• Interaction maps quantify and classify trajectory changes for diversity.

Method

EVITA employs NSGA-II, a multi-objective genetic algorithm, to evolve scenarios. It mutates and recombines AV planning problems and traffic light configurations, optimizing for interaction count, diversity, and minimal AVs.

In practice

• Implement interaction testing to expose emergent multi-AV behaviors.
• Leverage multi-objective search for efficient scenario generation.
• Utilize CommonRoad for standardized scenario definition and simulation.

Topics

• Autonomous Vehicles
• Scenario-Based Testing
• Multi-Objective Optimization
• Interaction Testing
• AV Safety
• Genetic Algorithms

Code references

• TUM-AVS/Frenetix-Motion-Planner
• ApolloAuto/apollo
• fabiocavaleri/EVITA
• Software-Aurora-Lab/DoppelTest

Best for: Research Scientist, AI Scientist, Robotics Engineer, MLOps Engineer

Related on AIssential

• Collaborative Multi-Agent Testing for Emergent Failure Discovery in Autonomous Driving Systems — Collaborative multi-agent testing with a shared blackboard improves emergent failure discovery in Autonomous Driving Systems.
• ScenePilot: Controllable Boundary-Driven Critical Scenario Generation for Autonomous Driving — ScenePilot generates critical, physically solvable autonomous driving scenarios that expose autonomy stack failures.
• EvoEye: Self-Evolving Runtime Monitoring for Autonomous Driving Systems — Self-evolving runtime monitors for ADSs improve hazard detection by iteratively learning from targeted, error-revealing scenario executions.
• Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing — Ensemble-based simulation during search-based testing effectively identifies simulator-agnostic failures in ADAS.
• This AI Car Drove Through a Tokyo Typhoon — General-purpose AI models can achieve robust zero-shot generalization in complex real-world autonomous driving scenarios.
• Artificial Intelligence for Modeling and Simulation of Mixed Automated and Human Traffic — AI methods are crucial for accurately simulating complex mixed human and autonomous traffic behaviors.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.