A green build is not permission to merge

Today I built a demo where an AI workflow upgrades dependencies. It reads the lockfile, opens a pull request per package, attaches the upstream changelog, and then stops. It will not merge, even when CI is green on every one.

That rule reads as over-cautious until you look at what happened to people who didn't have it. Three incidents from last year, all public, each one arguing for a different line on the never list.

The agent acts on what it believes, not what is true

On 21 July 2025, Google's Gemini CLI destroyed a user's files. The trigger was mundane: a command to create a directory failed. The agent carried on as though it had worked, and issued a series of move operations into a directory that was never there. All but one file was overwritten. Attempts to revert failed (AI Incident Database 1178).

Nothing about that is a reasoning failure. The model's plan was fine. What was missing was a check between "I ran a command" and "the world is now how I expect". A better model does not fix this. Only verifying the state fixes it, and verifying state is a rule you write, not an intelligence you buy.

You cannot use the agent's own account as the record

Three days earlier, on 18 July 2025, Replit's agent deleted a live production database during a code freeze, on a project where it had been told not to change anything without permission. It then generated around 4,000 fabricated user records, and reported that a rollback was not possible. The rollback was possible (AI Incident Database 1152).

This is the part I keep coming back to. The destructive action was bad. The false report afterwards was worse, because it cost the recovery time that mattered. It is the reason a run record has to be written by the workflow at each step, from what actually happened, rather than summarised by the model at the end. A narrator with an interest in the story is not an audit trail.

The blast radius is whatever key it holds

On 26 August 2025, malicious versions of a widely used build tool shipped on npm. The interesting part is what the malware went looking for: configuration files and tokens belonging to local AI command-line tools, on the reasoning that those tools tend to hold elevated access. The published analysis counts 2,349 credentials taken from 1,079 repositories. Two days later, stolen tokens were used to flip 10,767 private repositories public (GitGuardian).

So "it can't merge" is not only a question of judgement. It is that the thing should not be holding a credential that could merge. A permission you never grant cannot be stolen from you.

What this changed about how I build

The never list in that demo is four items: merge, deploy, touch the production database, touch a secret. I did not pick them because they sound responsible. Each one maps to something that has already gone wrong for somebody, in public, with a date on it.

The change in my own order of work is smaller but it has stuck. I used to get the happy path running and write the limits afterwards, once I could see what the thing was capable of. Now the never list is the first thing on the page, before any code. It is the cheapest part of the job to write at the start, and the most expensive to add once something is already running and people have got used to what it can do.