I Tried to Break an AI’s Security — Here’s Everything I Learned as a Complete Beginner

2026-07-19 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy, Software Development & Engineering · Depth: Novice, long

Summary

A developer embarked on a learning journey into AI security, specifically prompt injection, starting with the free browser game Gandalf by Lakera. This game, which has logged nearly 9 million interactions from over 200,000 people, challenges users to trick an AI into revealing a secret password. Moving beyond the game, the developer built a simplified prompt injection lab using Spring Boot and a local Llama 3.2 model via Ollama. This personal project explored four defense levels: no protection, a stronger system prompt, an output filter for literal leaks, and an input filter with a keyword blocklist. The most effective defense found was an architectural change where the LLM never knew the secret, shifting verification to deterministic backend code. The project highlighted that prompt injection is a language problem, system prompts are not security boundaries, and layered filters have blind spots.

Key takeaway

For software engineers building with LLMs, securing your applications against prompt injection requires more than just system prompts or simple filters. You should prioritize architectural design patterns that prevent the LLM from ever knowing sensitive secrets. Handle critical verification and sensitive data access in deterministic backend code, completely outside the model's context, to mitigate the inherent language-based vulnerabilities of LLMs. This approach offers robust protection where filters alone will inevitably fail.

Key insights

The core vulnerability in prompt injection lies in language itself, making traditional security controls insufficient.

Principles

• System prompts define policy, not security enforcement.
• Layered input/output filters have inherent blind spots.
• Architectural design, not filtering, secures secrets from LLMs.

Method

Build a local prompt injection lab by installing Ollama, pulling `llama3.2`, and running a Spring Boot app. Implement and test layered defenses, including system prompts, input/output filters, and architectural separation of secrets.

In practice

• Run open-source LLMs locally with Ollama for free testing.
• Implement output filters to block literal secret disclosures.
• Architect applications to keep sensitive secrets out of LLM context.

Topics

• Prompt Injection
• LLM Security
• Ollama
• Spring Boot
• System Prompts
• Architectural Security

Code references

• NehaKhann/ai-security-gandalf

Best for: AI Security Engineer, AI Student, Software Engineer

Related on AIssential

• Prompt Injection: The #1 AI Security Threat Every Developer Must Understand — Prompt injection is the top AI security threat, exploiting LLMs by overriding system instructions with malicious user input.
• I Tried to Break an AI’s Security — Here’s Everything I Learned as a Complete Beginner — Prompt injection exploits language, making system prompts and filters insufficient for robust AI security.
• Today’s stupid prompt injection: just write like the AI — LLMs' inability to distinguish text roles by source, only by style, enables effective prompt injection.
• The 3 Invisible Risks Every LLM App Faces (And How to Guard Against Them) — LLM applications introduce unique security risks requiring specialized guardrails beyond traditional software security.
• The Language of Security: How Prompt Syntax Shapes Secure Code Generation in Open LLMs — Prompt syntax, specifically constituent type and position, fundamentally shapes the security of LLM-generated code.
• What is Prompt Injection? — LLM agents process all input uniformly, making them vulnerable to adversarial text that overwrites intended instructions.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.