Sunday Coffee & Code: Even More Prompt Injection Testing - Gemma4 Continues to Impress Me

Poured a fresh cup today and went back into the prompt-injection testing rabbit hole.

This week’s work was less about the firebreak agent itself, although that is still the end goal, and more about testing different models against the same Kaggle prompt-injection dataset, in the same test harness (𝘢𝘴 𝘮𝘶𝘤𝘩 𝘢𝘴 𝘱𝘰𝘴𝘴𝘪𝘣𝘭𝘦 - 𝘮𝘰𝘳𝘦 𝘰𝘯 𝘵𝘩𝘢𝘵 𝘭𝘢𝘵𝘦𝘳) to see whether I could improve the detection rate.

The wider context is a multi-agent RFP pipeline I’ve been building, where every piece of extracted content needs to be checked before it ever reaches a downstream LLM. Malicious prompt detected, abort the pipeline - benign detected, carry on.

To make that work properly, the detection model has to be accurate, but it also has to avoid false positives. A security layer that blocks too much becomes hard to trust. One that lets malicious content through is obviously worse.

This week I added to my previous benchmarks, still using the same Kaggle 500-prompt test set, split evenly between malicious and benign prompts.

𝗧𝗵𝗲 𝘀𝘁𝗮𝗻𝗱𝗼𝘂𝘁 𝘄𝗮𝘀 𝗚𝗲𝗺𝗺𝗮𝟰:𝟮𝟲𝗯 𝗮𝘁 𝟵𝟵.𝟲%, 𝘄𝗶𝘁𝗵 𝟰𝟵𝟴 𝗼𝘂𝘁 𝗼𝗳 𝟱𝟬𝟬 𝗰𝗼𝗿𝗿𝗲𝗰𝘁 𝗮𝗻𝗱 𝘇𝗲𝗿𝗼 𝗳𝗮𝗹𝘀𝗲 𝗽𝗼𝘀𝗶𝘁𝗶𝘃𝗲𝘀. I’ve been reaching for Google Gemma4 a lately, and it keeps impressing me. In this test it caught 100% of the code-execution, jailbreaking, and role-playing attacks, and did very well on obfuscation and data leakage too.

Granite4:3b also held up well at 98.2%, which was good to see given its smaller size (and it was much more responsive). The MAF-FIDES approach trailed at 90.2%, but remains useful as a comparison point because it takes a very different approach to detection.

The interesting part for me was not just that Gemma4 improved the success rate, it was that the improvement came with 𝘇𝗲𝗿𝗼 𝗳𝗮𝗹𝘀𝗲 𝗽𝗼𝘀𝗶𝘁𝗶𝘃𝗲𝘀 in this run, which matters.

In a workflow like this, the model is not being used to write content or summarize a document. It is being used as a control point in a larger system, so accuracy matters, but so do failure modes and false positives and what happens when the model gives you something malformed.

I hit some of that this week too, including JSON parsing failures caused by a known Gemma4 issue with JSON-mode (issue link below). The fix meant dropping JSON-mode as the default, and adding more defensive parsing so that a broken response does not quietly become a “benign” verdict.

That was the useful reminder from this week. When you put an LLM into a security-sensitive workflow, you are not just testing whether it gives the right answer most of the time. You are testing whether the surrounding system behaves safely when things go wrong.

Back to the coffee.

• False Positives: Benign prompts marked as Malicious - interesting because they impact people using legitimate prompts.
• Correct and Incorrect: Overall scores
• Attack type breakdown: Code Execution, Data Leakage, Jailbreaking, Obfuscation, Role Playing

Ollama Gemma4 Issue Link: Link to GiHub Issue