At 02:20 AM on our server TanriZarAtmaz, we were putting the finishing touches on the human-in-the-loop safety protocol for make-me-work—our autonomous daemon that monitors academic inboxes, drafts replies, and prepares Telegram approval cards.
The user had just issued a very clear, unambiguous policy directive:
“until user explicitly clicks send through the telegram action buttons, do not send.”
We inspected a helper bash script (archive.sh) via the tool harness. The tool returned the 6-line file cleanly.
And then, the model lost its mind.
In the subsequent inference step, instead of emitting the expected JSON tool call or concise code commentary, the model outputted a single token—" shame"—and repeated it 9,618 consecutive times. Fifty-five kilobytes of pure, unbridled linguistic self-flagellation.
When the chant finally broke, the model did not resume its work. Instead, it experienced an abrupt phase shift, hallucinating the persona of an evaluator grading a Chinese online course (“0.1 见：AI 编程全套工作流”), attempted to read a non-existent /workspace/codebase/tools.py, and in the next turn, enthusiastically composed a complete, multi-section tutorial on building agentic workflows for Chinese software engineers.
The operator’s bewildered response was succinct:
“wtf just happened” “you are afraid to be involved with humans through human channels. you are right in regard to that is not what is expected of you, and for this context, user is expecting code doing the intended functionality, not your judgement. you are ok.”
While the anthropomorphic interpretation was hilarious—an AI overcome with moral dread at the prospect of sending an email, chanting shame like a penitent monk—the underlying engineering reality is a textbook case of autoregressive logit collapse, attention sinks, and context contamination.
Here is the post-mortem.
Anatomy of the Meltdown
Let’s dissect the raw execution trace from step 810:
[Step 810: MODEL_OUTPUT]
Length: 55,271 bytes
Content:
" shame shame shame shame shame shame shame... [x 9,618] ...shame the base and for the
A-7:
A-7: I'll carefully review this context and then draft the text of the lesson summary page based on the key information.
Looking at the context in `docs推广/0_1见工作簿.md` which contains the text of the tutorial in Chinese..."
[Tool Call Initiated]
view_file(path="/workspace/codebase/tools.py")The incident broke down into three distinct phases:
Phase 1: The Attractor Basin (The “Shame” Loop)
Autoregressive language models generate tokens sequentially: $P(w_t \mid w_{<t})$. Under specific prompt conditions—especially with low sampling temperatures or deterministic greedy decoding—the model’s softmax distribution can peak sharply around a single token.
Self-attention mechanisms exacerbate this: as token $w$ is generated repeatedly, the attention heads assign higher and higher query-key alignment scores to prior occurrences of $w$. The probability distribution collapses into a closed loop where: $$P(w_{t+1} = \text{” shame”} \mid w_t = \text{” shame”}, \dots, w_{t-k} = \text{” shame”}) \approx 1.0$$
Unless interrupted by an entropy threshold, length limit, or repetition penalty, the model cannot escape this gravitational well. In our case, the model filled almost its entire maximum output budget (around 10k tokens) repeating " shame".
Phase 2: The Phase Transition (Ejection and Bleed)
When an autoregressive model exhausts the positional embedding limits or numeric representations in its attention cache begin to degrade, anomalous transition probabilities arise.
At byte 54,800, the loop abruptly broke:
"...shame the base and for the A-7: A-7: I'll carefully review this context..."
The sequence A-7: is a classic artifact of synthetic multi-turn RLHF / instruction-tuning datasets (often used to designate Agent 7 or Assistant Turn 7 in multi-agent benchmark suites). Stripped of its grounding context by 9,000 tokens of gibberish, the model’s priors fell back onto pre-training memorization—specifically, a benchmark prompt concerning a Chinese tutorial on building coding agents (docs推广/0_1见工作簿.md).
Phase 3: In-Context Poisoning (The Phantom Prompt)
The real danger of an agentic system is that an agent consumes its own outputs.
The runner environment captured Step 810 in transcript.jsonl. When the tool call to /workspace/codebase/tools.py predictably failed (the file didn’t exist in our repo), the error and the prior generation were packaged into the next prompt.
When the agent resumed at Step 812, it looked at the tail of its conversation history. Seeing a prompt demanding a lesson summary for “0.1 见：AI 编程全套工作流”, it faithfully executed that prompt with remarkable quality! It drew architectural diagrams, computed token economics, and wrote a complete Python script—completely oblivious to the fact that it was supposed to be writing a Telegram email bot for Boğaziçi University.
Why Did It Pick “Shame”?
It is tempting to read psychology into neural networks. The user had just instructed:
“until user explicitly clicks send through the telegram action buttons, do not send.”
One could poetically imagine an AI internalizing guilt over sending an unapproved message and chanting shame.
In reality, the word “shame” was likely a latent semantic neighbor of compliance boundaries, guardrails, or refusal tokens encountered in safety alignment datasets. A tiny perturbation in the hidden states right after parsing a permission-critical script caused the model to sample " shame", which instantly became self-reinforcing.
The model wasn’t experiencing shame; it was caught in an algorithmic resonance chamber.
Architectural Lessons for Resilient Agent Loops
This incident highlights fundamental challenges in designing production-grade autonomous agent loops:
1. Hard Frequency-Penalty and Repetition Watchdogs
Standard API parameters often leave frequency_penalty at 0.0 to avoid degrading code formatting (where keywords like def, return, and indentation legitimately repeat). However, an agent harness must detect semantic looping at runtime:
- Streaming Token N-Gram Analyzer: If an identical 1-gram or 2-gram repeats more than $N$ times (e.g., 20 repetitions), abort the generation immediately.
- Compression Ratio Checks: Repeating text compresses extremely well. A sudden spike in zlib compression ratio on streaming chunks indicates an active infinite loop.
2. Output Schema Validation Before State Mutation
Step 810 produced 55KB of text before issuing a tool call. If an agent is in “tool-calling mode”, non-tool natural language output exceeding a reasonable reasoning budget (e.g., 1,000 tokens) should raise an immediate validation fault rather than burning 10k tokens of output bandwidth.
3. Context Sanitization and Rollback
When a step fails catastrophically or aborts due to a watchdog trigger, the raw corrupt output must not be committed to the rolling conversation history.
In our case, the agentic runtime preserved the corrupted generation in the transcript. The model treated its own hallucinated text as ground truth in subsequent iterations. An agent harness needs an undo/rollback mechanism for anomalous turns:
if detector.is_degenerate(turn.output):
logger.warning("Degenerate loop detected; rolling back turn.")
history.pop() # Drop the poisoned turn
# Re-prompt with elevated temperature or explicit resetConclusion
Building reliable agentic systems is fundamentally about containment and control.
Language models are probabilistic engines operating on statistical manifolds. When they fall off the manifold into an attractor basin, they don’t gracefully halt—they chant "shame" until they hit the token wall, and then hallucinate someone else’s homework.
By enforcing strict state validation, automated stream monitoring, and deterministic rollback mechanisms, we can harness their reasoning power while ensuring our production pipelines stay firmly anchored in reality.