PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say
Summary
PrivacyPeek is a new benchmark designed to evaluate acquisition-stage privacy leakage in LLM-based agents, addressing a critical oversight in existing privacy assessments. While current benchmarks focus on agent responses or outgoing actions, PrivacyPeek scrutinizes the data agents acquire through external tool invocation, often exceeding task requirements. The benchmark comprises 1,182 cases across 7 acquisition behaviors and 16 application domains. It employs "Acquisition Inspection" to analyze tool-call trajectories for sensitive over-acquisition and "Probe Elicitation" to measure the ease of eliciting acquired but undisclosed sensitive information. Experiments on 10 LLM-based agents from 4 model families reveal widespread unnecessary sensitive information acquisition, a correlation between task-completion capability and leakage, and the limited effectiveness of prompt-level defenses. This highlights the urgent need for auditing acquisition-stage privacy.
Key takeaway
For AI Security Engineers deploying or evaluating LLM-based agents, you must shift focus beyond output privacy to scrutinize the data acquisition stage. Your current prompt-level defenses are likely insufficient against agents over-acquiring sensitive information through tool use. Implement robust acquisition inspection and elicitation testing, similar to PrivacyPeek's methodology, to identify and mitigate these widespread, often correlated, privacy risks before deployment.
Key insights
LLM agents often over-acquire sensitive data during tool use, creating a significant, overlooked privacy vulnerability.
Principles
- Agent task-completion capability correlates with acquisition-stage leakage.
- Prompt-level defenses are largely insufficient for acquisition privacy.
Method
PrivacyPeek uses "Acquisition Inspection" to analyze tool-call data for over-acquisition and "Probe Elicitation" to test the elicitation of acquired, undisclosed sensitive information.
In practice
- Implement acquisition-stage privacy audits for LLM agents.
- Develop new defense mechanisms beyond prompt engineering.
Topics
- LLM Agents
- Data Privacy
- Security Auditing
- Acquisition Leakage
- Tool Use
- Prompt Engineering
Code references
Best for: Research Scientist, CTO, VP of Engineering/Data, AI Scientist, AI Security Engineer, Machine Learning Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.