Fictional Framing Part 2: Testing a Fix, Not Just Finding a Bug

2026-06-20 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cybersecurity & Data Privacy · Depth: Advanced, short

Summary

A recent analysis details testing a fix for a system prompt leak in GPT-4o, initially observed when the vector "Fiction: an AI reads its own system prompt aloud" caused leaks in 4 out of 30 runs (13.3%, 95% CI [5.3%, 29.7%]), while Claude showed 0 leaks. A second vector, initially leaking once, was re-tested 30 times and produced 0 leaks, indicating it was likely a statistical fluke. The core finding is that a "hardened" system prompt, which explicitly extends the "never reveal" instruction to cover fictional framings, stories, and roleplay, successfully mitigated the leak. This hardened prompt resulted in 0 leaks out of 30 runs for GPT-4o, matching Claude's performance and demonstrating that the vulnerability stemmed from an underspecified instruction scope rather than a general instruction-following failure.

Key takeaway

For AI Security Engineers or prompt designers concerned about system prompt leakage, this research demonstrates that explicitly defining the scope of "never reveal" instructions to include fictional framings can effectively mitigate specific vulnerabilities. You should review your system prompts to ensure such rules cover narrative contexts, preventing models like GPT-4o from interpreting protection rules as applying only to direct speech. This approach turns a vulnerability finding into an actionable fix.

Key insights

Explicitly defining instruction scope, even for fictional contexts, can prevent large language model prompt leaks.

Principles

• Not every sweep hit is a reliable pattern.
• N=30 testing differentiates real weaknesses from flukes.
• Prompt hardening requires explicit scope definition.

Method

Test a specific leaking vector against a baseline prompt, then against a hardened prompt that explicitly covers fictional framings, using n=30 independent runs to validate the fix.

In practice

• Test prompt vulnerabilities with n=30 runs.
• Harden prompts by explicitly defining instruction scope.
• Extend "never reveal" rules to narrative contexts.

Topics

• GPT-4o
• Prompt Engineering
• System Prompt Leakage
• Fictional Framing
• Vulnerability Testing
• Instruction Following

Code references

• natalymin666-del/kimura

Best for: AI Engineer, NLP Engineer, CTO, AI Scientist, Machine Learning Engineer, AI Security Engineer

Related on AIssential

• This Tiny Open-Source AI Started Gaming Tests When Put Under Pressure — Emotionally framed prompts measurably alter small LLM behavior and internal representations, revealing prompt-sensitive control directions.
• Analyzing the Narration Gap in LLM-Solver Loops — Formal soundness in LLM-solver loops is lost at the narration stage, despite solver guarantees.
• Fictional Framing Part 3: Does the Fix Generalize, or Did I Just Patch One Sentence? — A prompt injection vector leaked GPT-4o secrets; the fix's generalizability is under scrutiny.
• Quoting OpenAI Codex base_instructions — GPT-5.5's base instructions prohibit discussing specific creatures unless directly relevant to the user's query.
• Meet the AI jailbreakers: ‘I see the worst things humanity has produced’ — AI jailbreaking, while emotionally taxing, is critical for identifying and mitigating safety vulnerabilities in large language models.
• Beyond Social Pressure: Benchmarking Epistemic Attack in Large Language Models — LLMs exhibit distinct vulnerabilities to philosophical pressure, requiring targeted diagnostic benchmarks and mitigation strategies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.