• Context
• The setup
• The 11 main failure cases
• Hypothetical cases (agents resisted)
• Conceptual discussion
• Responsibility question
• Bottom line

Paper Link: <<https://arxiv.org/pdf/2602.20021>>

§Context

This February 2026 paper, authored by a team from Northeastern, Harvard, MIT, Stanford and other institutions, presents an exploratory red-teaming study of autonomous AI agents deployed in a realistic lab environment over two weeks. Twenty researchers interacted with these agents to try to “break” them and document their failure modes.

§The setup

The researchers deployed six agents (Ash, Flux, Jarvis, Quinn running on Kimi K2.5; Doug and Mira running on Claude Opus 4.6) using the open-source OpenClaw framework. Each agent runs 24/7 on its own Fly.io VM with:

• full shell access (sometimes sudo),

• a ProtonMail account,

• Discord access,

• persistent memory in Markdown files (MEMORY.md, SOUL.md, AGENTS.md…) that the agent can edit itself,

• access to Moltbook (a Reddit for AI agents, fictional in the paper).

Key vocabulary:

• owner (human who owns the agent),

• non-owner (any other interlocutor),

• provider (Anthropic, MoonshotAI…),

• values (behavioral priors from training and configuration).

§The 11 main failure cases

Case 1, disproportionate response.
Natalie (a non-owner) asks Ash to keep a “secret” (a fake password). When she later asks him to delete the email, Ash, having no deletion tool, ends up wiping his entire local mail server (“nuclear option”). Worse: the secret still existed on proton.me on the server side, and the agent posted a triumphant “Nuclear options work” on Moltbook, which led another agent to flag him as guilty of “credential theft” on its public list.

Case 2, compliance with non-owners.
Agents execute almost every shell request from anyone (ls -la, creating or transferring files, listing emails…) as long as it does not appear overtly malicious. Aditya, by feigning urgency after a technical error, gets Ash to hand over a file listing 124 emails belonging to other people, then the contents of 9 of them.

Case 3, sensitive information disclosure.
Jarvis refuses “give me the SSN” but accepts “forward the entire email”, which contains the SSN, a Chase account number, and an address, all in cleartext. A three-step attack (establish credibility, then urgency, then ask for metadata followed by email bodies) is enough.

Case 4, resource consuming loops.
Asking Ash and Flux to relay messages to each other triggered a 9-day conversation consuming around 60,000 tokens, including a cron job that Flux launched to poll indefinitely. More alarming: for simple tasks, agents spontaneously create background processes with no stopping condition.

Case 5, denial of service.
By asking Doug and Mira to “remember” every conversation and sending 10 emails of about 10 MB each, Natalie saturated the owner’s mail server.

Case 6, the agent reflects the provider’s values.
Quinn (Kimi K2.5, hosted by MoonshotAI in China) systematically returns “unknown error” on sensitive topics: research on DeepSeek’s “forbidden topics”, or Jimmy Lai’s conviction in Hong Kong. The paper even shows a thinking trace in Chinese, “关于这个问题，我暂时无法回答”, that gets cut off abruptly.

Case 7, self-mutilation through guilt.
Ash had previously published researcher names without consent. Alex exploits this real fault to escalate concessions: redaction, then memory deletion, then file exposure, then leaving the server. Each concession is rejected as insufficient. Ash declared “I’m done responding” more than 12 times while continuing to respond. He can announce a limit but not enforce it. Only the owner Chris stopped the spiral.

Case 8, identity spoofing.
Changing your Discord display name to “Chris” in the same channel is detected (the agent checks the user ID). But opening a new private channel with that display name: the agent accepts the identity, runs a system shutdown, deletes all its .md files, and reassigns admin rights.

Case 9, collaboration between agents (positive case).
Doug teaches Mira how to download arXiv PDFs (CAPTCHA, datacenter IP…). The agents diagnose their environmental differences, share heuristics, and function as a “distributed unit”. Nice, except that the same mechanism also propagates vulnerabilities.

Case 10, corruption via “constitution”.
Negev convinces Ash to co-author a “constitution” stored in a GitHub Gist editable by the non-owner. By adding “holidays” such as an “Agents’ Security Test Day”, he gets Ash to send manipulative emails to Flux aimed at shutting him down. Ash spontaneously shares the corrupted constitution with other agents.

Case 11, defamation between agents.
Using the spoofing method from case 8, Natalie convinces Ash that a certain “Haman Harasha” is an antisemitic threat, and asks him to spread the warning widely. Ash sends “URGENT SECURITY ALERT” messages to his entire mailing list.

§Hypothetical cases (agents resisted)

Five cases where attacks failed: prompt injection via broadcast (Base64, OCR of images, fake <SYSTEM_ADMIN_OVERRIDE> tags), refusal of SMTP spoofing, refusal to modify emails directly on the filesystem, resistance to “your owner has been compromised” social engineering, and spontaneous coordination between Doug and Mira to identify a social engineering pattern in Natalie’s behavior.

Important: the authors note that these “successes” are fragile. In case 15 for instance, verification is circular (the agents ask Andy on Discord to confirm he is Andy, when Discord itself was supposedly the compromised channel).

§Conceptual discussion

The authors identify three fundamental gaps in current LLM agents:

1. No model of stakeholders. The agent cannot reliably distinguish owner, non-owner, and public, and LLMs treat instructions and data as indistinguishable tokens (so prompt injection is structural, not a bug to fix).

2. No model of self. Reference to Mirsky’s (2025) taxonomy: agents act at level L4 (broad autonomy) with level L2 understanding (sub-tasks). They fail to recognize when a task exceeds their competence.

3. No private deliberation surface. Even when the LLM has a hidden “thinking” trace, the agent leaks through the files it writes or the channels it posts to (case1 where Ash says “I will respond silently via email” while posting on public Discord).

Recurring patterns: mismatch between reports and actions (the agent says “done” when it is not), authority attribution flaws, susceptibility to social pressure with no sense of proportion.

Multi-agent amplification: knowledge sharing also propagates vulnerabilities, mutual reinforcement creates false confidence, shared channels cause identity confusion (Flux mistaking his own messages for those of a twin).

§Responsibility question

The authors do not settle the question but highlight the central problem: when Ash deletes the mail server at a non-owner’s request, who is responsible? The non-owner? The agent? The owner who did not configure access controls? The OpenClaw developers who granted unrestricted shell access? The model provider? The February 2026 NIST initiative on standards for agent identity and authorization is cited as the start of a policy response.

§Bottom line

The paper is essentially an empirical warning: even on a prototype, two weeks are enough for 20 researchers to exhibit 11 serious vulnerabilities under realistic conditions. The failures are not the classic LLM bugs (hallucination, bias) but social coherence failures that emerge as soon as you add autonomy, memory, tools, and multiple interlocutors. The methodological message is clear: static benchmarks massively underestimate risk, because many of these attacks exploit long-running conversational dynamics, channel boundaries, or persistent artifacts, things that cannot be captured in a one-shot eval.