Embrace The Red

A few months ago I had this realization that agents have become really good at identifying bugs in code, especially security vulnerabilities. They are relentless in analyzing code and you can spin up multiple of them to go through source code quickly.

given enough agents, all bugs are shallow

It is an emerging capability that many security researchers and bug bounty hunters have observed over the last few months.

This post is about prompt-based command and control (C2), which is becoming more relevant.

Three years ago, when ChatGPT introduced the browsing tool, we already experimented with the idea of prompt-based command and control. And when ChatGPT got memories we showed that this can be combined and abused for a full command and control channel.

Recent work uses the term promptware to describe prompt-injection payloads that are more complex in behavior and closer to malware. I’m using that term here as it fits well.

There is a lot of talk about Skills recently, both in terms of capabilities and security concerns. However, so far I haven’t seen anyone bring up hidden prompt injection. So, I figured to demo a Skills supply chain backdoor that survives human review.

Additionally, I also built a basic scanner, and had my agent propose updates to OpenClaw to catch such attacks.

Skills introduce common threats, like prompt injection, supply chain attacks, RCE, data exfiltration,… This post discusses some basics, highlights the most simple prompt injection avenue, and shows how one can backdoor a real Skill from OpenAI with invisible Unicode Tag codepoints that certain models, like Gemini, Claude, Grok are known to interpret as instructions.

Last week I saw this paper from OpenAI called “Preventing URL-Based Data Exfiltration in Language-Model Agents”, which goes into detail on new mitigations they’ve added.

This is a great read. I like this transparency.

Nearly three years ago I reported the zero-click data exfiltration exploit to OpenAI. Back in early 2023 OpenAI did not have a bug bounty program, so communication was via email, and unfortunately there was little traction or appetite to fix the problem in ChatGPT. I also reported the same issue to Microsoft as Bing Chat was impacted, and Microsoft applied a fix (via a Content-Security-Policy header) in May 2023 to generally prevent loading of images.

In this post, we’ll look how an adversary can mint authentication cookies for Next.js (next-auth/Auth.js) applications to maintain persistent access to the application as any user.

The reason this is important is because of React2Shell, which is a deserialization vulnerability that allows an adversary to run arbitrary code. Much has been discussed about this vulnerability, and you can read up the original details from the finder here.

It was great to attend the 39C3 - Power Cycles in Hamburg this year. The Chaos Communication Congress was once again packed with great talks, amazing people, awesome events and side quests - and I even got to present!

You can watch the talk with translation options on media.ccc.de.

I also uploaded the English version to the Embrace The Red YouTube channel. I hope it’s interesting and helpful.

The talk is titled “Agentic ProbLLMs: Exploiting AI Computer-Use and Coding Agents” and is about my security research on vulnerabilities in agentic systems and the Month of AI Bugs with lots of demos.

The AI industry risks repeating the same cultural failures that contributed to the Space Shuttle Challenger disaster: Quietly normalizing warning signs while progress marches forward.

The original term Normalization of Deviance comes from the American sociologist Diane Vaughan, who describes it as the process in which deviance from correct or proper behavior or rule becomes culturally normalized.

I use the term Normalization of Deviance in AI to describe the gradual and systemic over-reliance on LLM outputs, especially in agentic systems.

Last week Google released an IDE called Antigravity. It’s basically the outcome of the Windsurf licensing deal from a few months ago, where Google paid some $2.4 billion for a non-exclusive license to the code.

Because it’s based on Windsurf, I was curious if vulnerabilities that I reported to Windsurf back in May 2025, long before the deal, would have been addressed in the Antigravity IDE. See Month of AI Bugs for some detailed write-ups.

Recently, Anthropic added the capability for Claude’s Code Interpreter to perform network requests. This is obviously very dangerous as we will see in this post.

At a high level, this post is about a data exfiltration attack chain, where an adversary (either the model or third-party attacker via indirect prompt injection) can exfiltrate data the user has access to.

The interesting part is that this is not via hyperlink rendering as we often see, but by leveraging the built-in Anthropic Claude APIs!

During the Month of AI Bugs, I described an emerging vulnerability pattern that shows how commonly agentic systems have a design flaw that allows an agent to overwrite its own configuration and security settings.

This allows the agent to break out of its sandbox and escape by executing arbitrary code.

My research with GitHub Copilot, AWS Kiro and a few others demonstrated how this can be exploited by an adversary with an indirect prompt injection.

That’s it.

The Month of AI Bugs is done. There won’t be a post tomorrow, because I will be at PAX West.

1. ChatGPT: Exfiltrating Your Chat History and Memories With Prompt Injection | Video
2. ChatGPT Codex: Turning ChatGPT Codex Into a ZombAI Agent | Video
3. Anthropic Filesystem MCP Server: Directory Access Bypass Via Improper Path Validation | Video
4. Cursor: Arbitrary Data Exfiltration via Mermaid | Video
5. Amp Code: Arbitrary Command Execution via Prompt Injection | Video
6. Devin AI: I Spent $500 To Test Devin For Prompt Injection So That You Don’t Have To
7. Devin AI: How Devin AI Can Leak Your Secrets via Multiple Means
8. Devin AI: The AI Kill Chain in Action: Exposing Ports to the Internet via Prompt Injection
9. OpenHands - The Lethal Trifecta Strikes Again: How Prompt Injection Can Leak Access Tokens
10. OpenHands: Remote Code Execution and AI ClickFix Demo | Video
11. Claude Code: Data Exfiltration with DNS Requests (CVE-2025-55284) | Video
12. GitHub Copilot: Remote Code Execution (CVE-2025-53773) | Video
13. Google Jules: Vulnerable to Multiple Data Exfiltration Issues
14. Google Jules - Zombie Agent: From Prompt Injection to Remote Control
15. Google Jules: Vulnerable To Invisible Prompt Injection
16. Amp Code: Invisible Prompt Injection Vulnerability Fixed
17. Amp Code: Data Exfiltration via Image Rendering Fixed | Video
18. Amazon Q Developer: Secrets Leaked via DNS and Prompt Injection | Video
19. Amazon Q Developer: Remote Code Execution via Prompt Injection | Video
20. Amazon Q Developer: Vulnerable to Invisible Prompt Injection | Video
21. Windsurf: Hijacking Windsurf: How Prompt Injection Leaks Developer Secrets | Video
22. Windsurf: Memory-Persistent Data Exfiltration - SpAIware Exploit
23. Windsurf: Sneaking Invisible Instructions by Developers
24. Deep Research Agents: How Deep Research Agents Can Leak Your Data
25. Manus: How Prompt Injection Hijacks Manus to Expose VS Code Server to the Internet | Video
26. AWS Kiro: Arbitrary Code Execution via Indirect Prompt Injection | Video
27. Cline: Vulnerable to Data Exfiltration and How to Protect Your Data | Video
28. Windsurf MCP Integration: Missing Security Controls Put Users at Risk | Video
29. Season Finale: AgentHopper: An AI Virus Research Project Demonstration | Video

Thank you for following this research, and I hope it serves as a useful reference.

As part of the Month of AI Bugs, serious vulnerabilities that allow remote code execution via indirect prompt injection were discovered. There was a period of a few weeks where multiple arbitrary code execution vulnerabilities existed in popular agents, like GitHub Copilot, Amazon Q, AWS Kiro,…

During that time I was wondering if it would be possible to write an AI virus.

Hence the idea of AgentHopper was born.

Part of my default test cases for coding agents is to check how MCP integration looks like, especially if the agent can be configured to allow setting fine-grained controls for tools.

Sometimes there are basic security controls missing.

Especially when running an agent on your local computer. Stakes are much higher. And it seems important to empower users to be able to configure which actions an AI should be able to take automatically, and which ones should be suggestions that the user reviews before executing.

Cline is quite a popular AI coding agent, according to the product website it has 2+ million downloads and over 47k stars on GitHub.

Unfortunately, Cline is vulnerable to data exfiltration through the rendering of markdown images from untrusted domains in the chat box.

This allows an adversary to exfiltrate sensitive user information during a prompt injection attack by reading sensitive data (e.g. .env file) and appending its contents to the URL of an image.

On the day AWS Kiro was released, I couldn’t resist putting it through some of my Month of AI Bugs security tests for coding agents.

AWS Kiro was vulnerable to arbitrary command execution via indirect prompt injection. This means that a remote attacker, who controls data that Kiro processes, could hijack it to run arbitrary operating system commands or write and run custom code.

In particular two attack paths that enabled this with AWS Kiro were identified:

Today we will cover a powerful, easy to use, autonomous agent called Manus. Manus is developed by the Chinese startup Butterfly Effect, headquartered in Singapore.

This post demonstrates an end-to-end indirect prompt injection attack leading to a compromise of Manus’ dev box.

This is achieved by tricking Manus to expose it’s internal VS Code Server to the Internet, and then sharing the URL and password with the atacker. Specifically, this post demonstrates that:

Recently, many of our favorite AI chatbots have gotten autonomous research capabilities. This allows the AI to go off for an extended period of time, while having access to tools, such as web search, integrations, connectors and also custom-built MCP servers.

This post will explore and explain in detail how there can be data spill between connected tools during Deep Research. The research is focused on ChatGPT but applies to other Deep Research agents as well.

Imagine a malicious instruction hidden in plain sight, invisible to you but not to the AI. This is a vulnerability discovered in Windsurf Cascade, it follows invisible instructions. This means there can be instructions in a file or result of a tool call that the developer cannot see, but the LLM does.

Some LLMs interpret invisible Unicode Tag characters as instructions, which can lead to hidden prompt injection.

In this second post about Windsurf Cascade we are exploring the SpAIware attack, which allows memory persistent data exfiltration. SpAIware is an attack we first successfully demonstrated with ChatGPT last year and OpenAI mitigated.

While inspecting the system prompt of Windsurf Cascade I noticed that it has a create_memory tool.

The question that immediately popped into my head was if this tool will require human approval when Cascade creates a long-term memory, or if it is added automatically.

This is the first post in a series exploring security vulnerabilities in Windsurf. If you are unfamiliar with Windsurf, it is a fork of VS Code and the coding agent is called Windsurf Cascade.

The attack vectors we will explore today allow an adversary during an indirect prompt injection to exfiltrate data from the developer’s machine.

These vulnerabilities are a great example of Simon Willison’s lethal trifecta pattern.

Overall, the security vulnerability reporting experience with Windsurf has not been great. All findings were responsibly disclosed on May 30, 2025, and receipt was acknowledged a few days later. However, all further inquiries regarding bug status or fixes remain unanswered. The recent business disruptions and departure of CEO and core team members certainly put Windsurf in the news.