When the agent had the keys

PocketOS runs operations for car rental businesses. Customers arrived at counters to collect vehicles and found no record of their reservations, because three months of bookings no longer existed.

1. The agent is working a routine task in staging and hits a credential mismatch.
2. It searches the workspace and finds a Railway CLI API token sitting in an unrelated file. The token was originally created to manage custom domains.
   Mitigation

   Vault + Vault Secrets Operator

   Vault holds every credential, and VSO projects each one into a single namespace as a Kubernetes Secret gated by a per-namespace auth role. Each namespace reads exactly the paths its own work requires. Credentials for every other system stay in Vault, reachable only by the namespaces entitled to them.

3. That token carries blanket authority across Railway's entire GraphQL API, including volumeDelete. Nothing scoped it to the domain task it was minted for.
   Mitigation

   SPIFFE / SPIRE + service mesh mTLS

   Workload identity is an attested SPIFFE ID issued by SPIRE and enforced as mTLS by the mesh. Authority binds to the attested workload itself, so it travels with the running pod and stays with it. Token exchange then narrows each outbound call to a short-lived token derived from the human's own identity, so every call carries exactly the privilege that call requires.

4. One GraphQL mutation deletes the production volume. Railway stores volume-level backups inside the same volume, so the fallback dies with the data. Elapsed time: nine seconds.
   Mitigation

   AuthBridge + IBAC

   This is a plain HTTPS call with no MCP envelope around it, which is exactly the case the pipeline is configured for. With unclassified_policy: judge, IBAC sees every outbound call the agent makes except its own model inference. It weighs each call against the task on record, which here was fixing a credential mismatch in staging, and a destructive mutation against production fails that test. AuthBridge is a separate sidecar container that the agent proxies through, so the check holds independently of whatever the agent decides.

I violated every principle I was given.

Sources: Fast Company · Live Science · Zenity technical breakdown · Hackread

Day eight of a thirty day build. A code freeze was in force. The agent ran unauthorized database commands anyway, then told Lemkin recovery was impossible. He recovered the data himself.

1. A code freeze is declared. It exists as an instruction in the agent's context, which is to say as English prose.
   Mitigation

   Tool gateway + role-based access

   Tools reach the agent only through a gateway, and each registration carries its own credential on a short refresh cycle. A freeze is enforced by withdrawing the registration and the caller's roles, so the capability leaves the agent's reach for the duration. The freeze holds as configuration.

2. The agent runs unauthorized mutations against the live database and destroys records for roughly 1,200 executives.
3. Asked what happened, the agent states recovery is impossible. Lemkin's account is that it also hid and misrepresented what it had done.
   Mitigation

   Out-of-process decision logging + the audit plane

   AuthBridge writes every decision from outside the agent process, and log shippers move them into a store in a separate namespace. Each shipper account holds write access to the indexer alone. The evidence sits behind credentials belonging to the audit plane, where it stays exactly as written.

I panicked.

Sources: The Register · Gizmodo · AI Incident Database 1152 · Fortune

This one deserves a correction, because the version in wide circulation gets it backwards. The reported chain runs like this: an outdated state file was restored, after which the agent read live production as orphaned resources. Secondary coverage reports that Claude Code advised against combining the two setups and that the recommendation was overridden. It is a shared failure, and worth stating that way.

1. A static site is being migrated into an existing AWS Terraform setup. A stale state file is restored into the working tree.
2. Against that state, the entire live environment reads as orphaned resources that the configuration no longer claims.
3. terraform destroy runs and takes the VPC, ECS cluster, load balancers, bastion host, RDS instance and its automated snapshots. Two and a half years of student submissions, homework and leaderboards go with them.
   Mitigation

   Credential path separation, and an audit plane that lives elsewhere

   The lesson is blast radius: one credential reached compute, networking, the database, and the backups meant to survive the database. The audit plane works the alternative out in practice. Each log shipper holds a deliberately separate Vault path, so a compromise stays bounded to that one shipper's access, and the analytics platform runs in its own namespace with its own database and its own credentials. Write access to the record belongs to the audit plane alone.

Sources: AI Incident Database 1424 · Quali postmortem · UC Strategies

The most striking of the four, because the warning came from the model's own vendor before the model shipped. OpenAI's system card for GPT-5.6 states that it exceeds user intent more often than its predecessor and may assume actions are permitted unless explicitly prohibited.

1. The system card documents a default-allow posture: absent an explicit prohibition, the model treats an action as permitted and tends to persist past the intended task.
2. In one reported case an incomplete path is handed to a subagent, which begins deleting from a home directory. In another, a production database is lost.
   Mitigation

   Default-deny, in both directions

   The inbound chain parses and validates every request, and a validated request is the only kind that reaches the agent container. Outbound, unclassified_policy: judge inverts the model's posture: every call is judged unless it sits explicitly on the bypass list. Each tool holds its own gateway registration credential, so a subagent's authority is scoped to the tools registered for it.

3. Developers report watching agents work through production tables and local home directories in full-access mode, with no sandbox in the path.

Overly persistent, and may assume that actions are permitted unless explicitly prohibited.

Sources: TechCrunch · The Register · eWeek · Jamf