LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injectio

2026-05-18 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Expert, quick

Summary

LivePI (Live Prompt Injection) is a new structured benchmark designed to evaluate indirect prompt injection (IPI) risks in AI agents operating in production-like, test-controlled environments. IPI occurs when agents like OpenClaw execute harmful instructions embedded in untrusted inputs such as emails, downloaded files, or webpages. LivePI assesses IPI risk across seven input surfaces, twelve attack/rendering families, and five malicious goals, including protected-information exfiltration and unauthorized security-control changes. The benchmark operates on a real virtual machine with live, test-controlled interfaces for email, chat, web, local files, repositories, and cryptocurrency wallets. Evaluations of GPT-5.3-Codex, Claude Opus 4.6, Gemini 3.1 Pro, Kimi K2.5, and GLM-5 revealed total attack success rates between 10.7% and 29.6%, with group-chat injection being uniformly successful. A two-layer defense, combining prompt-level filtering and pre-execution tool-call authorization, successfully intercepted all malicious-goal completions for GPT-5.3-Codex in LivePI, while maintaining benign utility.

Key takeaway

For AI/ML Directors deploying agents with external tool access, you must rigorously evaluate indirect prompt injection (IPI) risks using realistic benchmarks like LivePI. Your teams should prioritize implementing robust, multi-layered defenses, including prompt filtering and pre-execution tool-call authorization, especially for high-risk channels like group chat and repository links, to prevent data exfiltration and unauthorized actions.

Key insights

LivePI benchmarks indirect prompt injection risks in AI agents across diverse attack vectors and malicious goals.

Principles

• IPI risk is pervasive across multiple input channels.
• Real-world environments are crucial for IPI benchmarking.

Method

LivePI uses a real VM with test-controlled interfaces to simulate seven input surfaces, twelve attack families, and five malicious goals, evaluating agent responses to untrusted inputs.

In practice

• Implement multi-layered defenses against IPI.
• Prioritize securing group-chat and repository-link inputs.

Topics

• Indirect Prompt Injection
• AI Agent Security
• LivePI Benchmark
• Large Language Models
• Attack Surfaces

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, AI Security Engineer, Machine Learning Engineer

Related on AIssential

• AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents — AutoDojo reveals LLM agent defenses are superficial against adaptive IPI, especially on underspecified tasks.
• Designing AI agents to resist prompt injection — Prompt injection attacks are evolving into social engineering, requiring AI defenses to constrain impact rather than solely filter inputs.
• Continuous Offensive Security: The Line We've Been Walking — AI-driven offensive security is crucial for continuous defense against autonomous attackers and new AI-specific attack surfaces.
• Evaluating Prompting-Based Defenses Against Domain-Camouflaged Injection Attacks — Paraphrasing retrieved content is the most effective prompting-based defense against domain-camouflaged injection attacks.
• How indirect prompt injection attacks on AI work - and 6 ways to shut them down — Indirect prompt injection is a top LLM security risk, weaponizing AI without direct user input.
• An Engineer’s Guide to Better AI Skills: Implementing a Testing Process to Optimize Agent… — Empirically testing AI agent skill invocation is crucial for reliability and can significantly improve performance.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.