15 Privacy Questions Every AI Builder is Asking

2026-03-16 · Source: Vanishing Gradients · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Software Development & Engineering · Depth: Advanced, extended

Summary

Katharine Jarmul, a Privacy in ML/AI Expert and author of "Practical Data Privacy," addresses 15 critical privacy questions for AI builders deploying LLMs, agents, and multimodal systems. The discussion emphasizes that system prompts and agent harnesses should be considered public by default, and "privacy observability" is crucial. Key topics include defining privacy across legal, social, and technical lenses, major concerns like murky data origins and purpose limitation, and various privacy controls from basic sanitization to advanced cryptographic methods. The article also covers evaluating privacy engineering through testing and observability, maturing practices for international regulations, the importance of red teaming, and the nuanced role of federated learning. Builders are urged to recognize AI's intimate nature, view privacy as a spectrum, and embrace the field's technical challenges.

Key takeaway

For AI Engineers building LLMs and agent systems, you must proactively integrate privacy engineering from the outset. Assume your system prompts and agent harnesses are public, and prioritize data flow mapping and privacy observability. Implement layered controls like PII redaction, dynamic routing, and algorithmic guardrails. Actively red team your systems and foster a culture of privacy champions to uncover vulnerabilities early, ensuring compliance and user trust.

Key insights

Technical privacy in AI demands proactive engineering, assuming system prompts and agent harnesses are public by default.

Principles

• Privacy is subjective and contextual.
• System prompts are public by default.
• Privacy is a grayscale, not a binary.

Method

Start by mapping data flows and establishing privacy observability. Implement layered guardrails, dynamic routing, and data minimization, progressing to advanced cryptographic solutions.

In practice

• Audit application logs for unexpected PII.
• Deploy Microsoft Presidio for PII redaction.
• Host internal hackathons to red team AI systems.

Topics

• Privacy Engineering
• Large Language Models
• AI Agents
• Privacy Observability
• Red Teaming
• Data Governance
• Differential Privacy

Code references

• linexjlin/GPTs
• torch-uncertainty/torch-uncertainty
• meta-llama/PurpleLlama

Best for: AI Engineer, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• Presentation: Busting AI Myths and Embracing Realities in Privacy & Security — Effective AI privacy and security demand proactive, continuous organizational effort, not reliance on external fixes or single-solution myths.
• How to Implement Privacy by Design in Tech — Privacy by Design shifts privacy decisions to early design stages, making them choices, not remediation.
• Introducing OpenAI Privacy Filter — OpenAI's Privacy Filter offers context-aware PII detection and redaction via a small, efficient, open-weight model.
• Can your AI agent remember your secrets without the cloud ever seeing them? — Edge LLM agents face privacy risks by transmitting sensitive personal data to the cloud for advanced functions.
• Privacy, AI, and Tech Works Worth Your Attention — AI's pervasive integration demands a re-evaluation of privacy laws, governance models, and individual rights, particularly concerning data usage and accountability.
• The Day I Realized Our LLM Was Quietly Logging Patient SSNs — Data protection for LLMs is an architectural problem requiring a robust redaction gate, not just instruction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Vanishing Gradients.