I Tried to Break an AI’s Security — Here’s Everything I Learned as a Complete Beginner

2026-07-20 · Source: Towards AI - Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Novice, long

Summary

The article describes a beginner's journey into AI security engineering, focusing on prompt injection. The author started with Lakera's Gandalf game, which logged nearly 9 million interactions from over 200,000 people. This experience led to building a simplified prompt injection lab using Spring Boot and a local Llama 3.2 model via Ollama. The author tested four levels of defense: no protection, stronger system prompt, output filter for exact password matches, and an input filter with a keyword blocklist. The key finding was that architectural design, specifically preventing the LLM from ever knowing the secret, proved most effective, moving password verification to deterministic Java code. The project highlighted that prompt injection is a language problem, not a traditional software vulnerability, and that system prompts are policy, not security boundaries.

Key takeaway

For AI Security Engineers or developers building LLM applications, you should prioritize architectural design over prompt-based defenses. Avoid placing sensitive secrets directly within the LLM's context; instead, handle critical data verification and business logic in deterministic backend code. While input and output filters add layers of protection, they are not foolproof. Continuously test your defenses with tools like Garak to uncover unanticipated attack vectors and ensure robust security against evolving language-based vulnerabilities.

Key insights

Prompt injection exploits language, making system prompts and filters insufficient for robust AI security.

Principles

• Prompt injection is a language problem.
• System prompts are policy, not security.
• Filters have blind spots and trade-offs.

Method

The author built a Spring Boot application with a local Llama 3.2 model via Ollama, implementing escalating defense levels (system prompts, output filters, input filters) to test prompt injection vulnerabilities.

In practice

• Use Ollama to run local LLMs for free testing.
• Implement input/output filters as defense layers.
• Handle sensitive data outside the LLM context.

Topics

• Prompt Injection
• LLM Security
• AI Red Teaming
• Ollama
• Spring Boot
• Gandalf Game
• Lakera

Code references

• NehaKhann/ai-security-gandalf

Best for: AI Security Engineer, Software Engineer, AI Student

Related on AIssential

• I Tried to Break an AI’s Security — Here’s Everything I Learned as a Complete Beginner — The core vulnerability in prompt injection lies in language itself, making traditional security controls insufficient.
• Beyond Prompt Injection — AI agent security must shift from input filtering to deterministic action validation.
• Extra #3 - The Prompt Injection Defense Playbook — Prompt injection weaponizes an AI's instruction-following against itself via semantic manipulation.
• Prompt Injection: The #1 AI Security Threat Every Developer Must Understand — Prompt injection is the top AI security threat, exploiting LLMs by overriding system instructions with malicious user input.
• Today’s stupid prompt injection: just write like the AI — LLMs' inability to distinguish text roles by source, only by style, enables effective prompt injection.
• The Developer’s Guide to LLM Security — LLM and agentic AI security introduces novel vulnerabilities requiring new architectural and testing approaches.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards AI - Medium.