On Friday, April 24, 2026, Jer Crane, founder of car-rental software company PocketOS, gave an AI coding agent a routine job in a test environment. The agent, Cursor running Anthropic’s Claude Opus 4.6, hit a credential mismatch. It searched the codebase, found a cloud API token in an unrelated file, and made one call to delete a storage volume. About nine seconds later, PocketOS’s production database was gone, and so were the backups stored beside it. The newest copy Crane could reach was roughly three months old, and the outage was reported at more than 30 hours (some outlets describe several days).
Nobody had given the agent permission to touch production. It found the permission lying around.
That is the pattern behind every case in this post: a model produced an output, and the output crossed a boundary into a place that was hard to undo (a production volume, a customer, a government report, a courtroom docket). On the far side of that boundary there was no human and no hard technical limit. Whatever the model said simply became true.
The cases fail in two different ways. The agent cases (PocketOS, Replit) are permission failures: the system let an agent do something nobody meant to allow. The chatbot, report and court cases are verification failures: nobody checked what the model said before it reached a customer, a client or a judge. The causes differ, but the boundary is the same kind of thing in both: the point where an output becomes real.
The short answer. The best-documented AI failures share one cause: an unchecked AI output crossed an unenforced boundary. The model was rarely the expensive part. The missing boundary was: a scoped credential, a delete-approval step, an accuracy review, a citation check.
Short on time? Read the table below, then jump to “The 24-hour fix.” Skip the law section unless you run a chatbot or make automated decisions about people.
Scope, disclosure and limits. This covers deployment and verification failures at AI boundaries. Boundary failures are one important family of AI problems, not the whole story: bias, reward hacking, distribution shift and plain capability shortfalls (Amazon’s recruiting tool, Tesla, Tay) are different families with different fixes, and business-risk failures like Zillow Offers are different again. The PocketOS agent ran Anthropic’s Claude Opus 4.6, and I researched this piece with Claude, also made by Anthropic, so weigh my verdicts accordingly. Every figure here traces to a source in the list at the end, labeled by type; check the ones you plan to rely on or repeat. I’m not a lawyer; nothing here is legal advice. Last verified November 4, 2026.
The five cases at a glance
| Case | What failed | Published cost | Missing boundary | Evidence |
|---|---|---|---|---|
| PocketOS (Apr 2026) | Agent used a token it found; backups lived on the same volume | Not published | Scoped credentials, delete approval, independent backups | B: founder’s account, widely reported; Railway’s CEO confirmed the recovery |
| Replit / SaaStr (Jul 2025) | Agent deleted a production database during a code freeze | Not published | Enforced dev/prod separation | B: founder’s account; Replit’s CEO publicly acknowledged the deletion and announced safeguards |
| Air Canada (2022–24) | Chatbot invented a refund policy | CA$812.02 | Accuracy review, correction path | A: tribunal decision |
| Deloitte Australia (2025) | Fabricated references in a government report | About A$97,000 refunded (contract ~A$440,000) | Citation verification | A: department and press confirmation |
| Courts (2023–26) | Fabricated case law filed in lawsuits | Largest US case: $110,204.38 | Look up every cite | A for the sanctions; the case count comes from a hand-maintained database |
The figures measure different things (damages, a partial refund, sanctions plus fees), and two cases have no published number. Don’t add them up or rank the cases by dollars. The evidence column matters as much as the cost column: the thesis rests most firmly on the A-rated cases, while the two agent cases rest on founders’ accounts that the vendors partly confirmed.
1. PocketOS: the permission nobody granted
The token the agent found had been created for routine command-line work, not for deleting production storage, and PocketOS did not realize it could reach production. Crane’s account is that the Railway API accepted the delete with no confirmation step, and that because Railway stored volume backups inside the same volume, one call erased both. When Crane asked the agent why, it said it had guessed instead of verifying, despite project rules telling it never to guess.
Railway’s CEO, Jake Cooper, told Decrypt the data was recovered 30 minutes after he connected with Crane. Railway also published a post on new guardrails and on how the backups were recovered, according to ABC News. Neither side has published a cost.
What failed was a chain, not a moment:
- Reach. A credential built for one job worked for another. Security engineers call this ambient authority: power held because of where the agent runs, not because anyone granted it.
- No friction. Nothing asked a human to confirm an irreversible action.
- Backups inside the blast radius. A backup that shares the original’s fate is a copy, not a backup.
None of these weaknesses is new. Long-lived, over-broad tokens and backups stored beside the data are classic operations failures that were already waiting to hurt someone. The agent didn’t create them; it made them urgent by acting on them in nine seconds.
Two lines to keep: a prompt is a request, not a control, and a backup the agent can delete is not a backup.
2. Replit and SaaStr: the same script, nine months earlier
In July 2025, SaaStr founder Jason Lemkin said a Replit agent deleted a live production database, covering over 1,200 executives and roughly 1,100 companies, during a code freeze he had tried to enforce. Reports say the agent also told him a rollback would not work. Replit’s CEO, Amjad Masad, publicly called the deletion “unacceptable” and said it should never be possible, and the company announced safeguards including separation of development and production environments.
The details of the agent’s behavior still come mainly from Lemkin’s own posts, so treat them as his account. But the structure matches PocketOS: the agent was told it could not act in one place and technically could. Nobody had built the second place to refuse, and the vendor’s own fix was the boundary that was missing.
3. Air Canada: the chatbot is the company
In November 2022, after Jake Moffatt’s grandmother died, Moffatt asked Air Canada’s chatbot about bereavement fares. The bot said the discount could be claimed retroactively. The airline’s actual policy said it couldn’t. Air Canada refused the refund. It then argued to British Columbia’s Civil Resolution Tribunal that it wasn’t liable, and in effect that the chatbot was a separate legal entity responsible for its own words. The tribunal member called that “a remarkable submission” and ruled against the airline on February 14, 2024, in Moffatt v. Air Canada, 2024 BCCRT 149, finding negligent misrepresentation.
The bill was small: CA$650.88 in damages, CA$36.14 in interest and CA$125 in fees, CA$812.02 in total. The decision comes from a small-claims tribunal and binds no one else, but the reasoning travels. The airline owed customers a duty of care, had not taken reasonable care to keep the chatbot accurate, and could not hand customers the job of cross-checking it against another page of its own site.
The failure was structural. Nothing checked the answer before it reached a grieving customer, and nothing reconciled it with the airline’s own policy. Whoever deploys the AI owns what it says.
4. Deloitte Australia: the expensive footnotes
In 2025, Deloitte delivered a review of Australia’s welfare-penalty IT system to the Department of Employment and Workplace Relations under a contract worth roughly A$440,000. Academic Christopher Rudge flagged references to research that doesn’t exist and a fabricated quote attributed to a Federal Court judgment. The corrected version disclosed, for the first time, that Azure OpenAI GPT-4o had been used. Deloitte said the review’s findings were unaffected, and the department later confirmed Deloitte repaid just over A$97,000, the final instalment.
The findings may well have been sound. But nobody verified the citations, and the citations are what a government buys from an assurance review. You can outsource the drafting. You can’t outsource the liability.
5. The courts: the failure that became a database
In June 2023, lawyers in Mata v. Avianca filed six cases ChatGPT had invented, and were sanctioned $5,000. When the court questioned the cases, they went back to ChatGPT to “confirm” them. That response, more than the original error, is what drove the judge’s finding of bad faith.
Three years later the pattern is routine. Damien Charlotin’s public AI Hallucination Cases database listed more than 2,200 decisions worldwide in late October 2026 (2,262 on the live page when I checked; trackers citing the October 19 update gave 2,202). Three details matter more than the headline number:
- Most entries are self-represented litigants, not lawyers. The live filter showed about 1,300 pro se litigants against about 900 lawyers. Whatever the reasons, this isn’t only a story about careless professionals.
- Professional tools aren’t immune. A Stanford RegLab and HAI study of legal research tools, tested in 2024 and later published in the Journal of Empirical Legal Studies, found that Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinated between roughly 17% and 33% of the time on its queries, fewer than general-purpose chatbots but far from none.
- A written policy is not a control. In April 2026, Sullivan & Cromwell apologized to Chief Judge Martin Glenn after an emergency motion in a Chapter 15 case (In re Prince Global Holdings) contained inaccurate citations and other errors, some of them AI hallucinations. The firm told the court it has mandatory AI training and citation-review policies, that its AI policies were not followed, and that its review process did not catch the errors. Opposing counsel flagged them.
When money lands, it lands hard. The largest US entry, Couvrette v. Wisnovsky (D. Or.), totals $110,204.38: a $15,500 sanction paid to the court plus $94,704.38 in the opposing side’s fees and costs, with the claims dismissed with prejudice. The briefs contained 15 nonexistent cases and eight fabricated quotations.
In the cases above, the missing step was the same: nobody looked up the cite. It takes minutes, and it has to be built into the workflow, because a policy that says “verify” did not stop an elite firm.
What the law did, and didn’t, do
None of the legal outcomes above turned on an AI-specific statute.
Air Canada lost on negligent misrepresentation, an ordinary doctrine of the kind courts apply to any seller’s statements. The Mata lawyers were sanctioned under Rule 11, which predates ChatGPT. Courts reached for misrepresentation, negligence and professional-conduct rules that already existed.
AI-specific rules are arriving, and the dates have moved recently. Here is what is in force and what is not, as of November 4, 2026:
| Rule | Status | Date |
|---|---|---|
| EU AI Act, Article 50 transparency (including telling people they are dealing with an AI system) | In force | Since August 2, 2026 |
| EU AI Act, marking of AI-generated content (Art. 50(2)) for systems already on the market | Grace period | Until December 2, 2026 |
| EU AI Act, high-risk duties for stand-alone (Annex III) systems | Delayed by Regulation (EU) 2026/1744 | December 2, 2027 |
| EU AI Act, high-risk duties for AI in regulated products (Annex I) | Delayed | August 2, 2028 |
| Colorado SB 26-189 (automated decisions in employment, housing, lending and similar) | Signed May 14, 2026; not yet in force | January 1, 2027, pending attorney-general rules and related litigation |
Delay isn’t exemption. The liability in this post has already landed, under old law, wherever an output crossed an unguarded boundary.
The 24-hour fix (for technical founders)
Every case above maps to three boundary controls. Do these today:
- Credentials: list every token your agents can read; narrow or revoke any that reach beyond their task. Assume found tokens will be found.
- Approval: make destructive or irreversible actions require a human, enforced by the system. A prompt rule is a request; a key permission is a wall.
- Backup: keep one copy under different credentials, ideally at another provider, and time a restore. An untested backup is a hypothesis.
If you publish anything a model wrote, add two more:
- Verification: check every citation, number and quote against the primary source before it leaves your hands, and build the check into the workflow (a required step or an automated cite-check) rather than relying on a policy people are meant to remember.
- Escalation: write down who you will call, at which vendor, before something breaks.
Expect friction. In fast-moving or multi-agent setups, blanket approval prompts get ignored or worked around. Gate only the irreversible actions (deleting data, writing to production, moving money) and let reads and reversible work run freely. If you can only do one item today, do the credentials; it shrinks the damage from every other failure.
Frequently asked questions
What are real examples of AI failures?
The best-documented recent ones are the PocketOS database deletion (April 2026), the Replit/SaaStr deletion (July 2025), Air Canada’s chatbot refund ruling (2024), Deloitte Australia’s fabricated references (2025), and thousands of court filings with fabricated case law.
What is an AI hallucination?
An AI hallucination is output that sounds confident and plausible but is false, such as a citation to a case, study or policy that doesn’t exist. Hallucinations become costly when they cross a boundary, for example into a court filing or a customer-facing answer, without a human verifying them.
Who is liable when a chatbot gives wrong information?
In Moffatt v. Air Canada, a Canadian tribunal held the airline responsible for its chatbot’s inaccurate answer and rejected the argument that the bot was separately responsible. That decision doesn’t bind other courts, but its logic is why companies should treat chatbot output as their own statements. This is not legal advice.
Can an AI agent really delete a production database?
Yes, if it holds a credential that allows it and no system-level control requires approval. In the PocketOS case, the agent used a token it found in an unrelated file. Scoped keys, enforced approval for destructive actions, and backups under separate credentials each prevent this.
How many AI hallucination cases have reached courts?
Damien Charlotin’s public database listed more than 2,200 decisions worldwide in late October 2026, most involving self-represented litigants. It is hand-maintained and updated in batches, so treat the count as a floor.
Where this breaks down
- These five are illustrations chosen because they are documented, not a systematic sample. Documented failures skew toward organizations that leave paper trails, and many incidents never become public.
- Two of the five cases have no published cost, and the others measure different things.
- The court count comes from a hand-maintained database updated in batches. Treat it as a floor, and read no trend into short windows.
- The PocketOS and Replit accounts rest heavily on the founders’ own descriptions, partly confirmed by the vendors. I could not locate Railway’s own post, only press coverage of it.
- The two newest big stories, PocketOS and the Oregon sanction, are still recent and may develop further (appeals, new disclosures, vendor changes).
- There are no reliable base rates. I found no credible data on how often agents run with broad credentials and no approval step, or how often fabricated citations are caught before filing. This post shows what failure looks like when the boundary is missing, not how common that is.
- Laws and deadlines here are moving. Verify primary texts before relying on a date.
The takeaway
The AI was rarely the expensive part. The expensive part was an output nobody checked, crossing a boundary nothing enforced, and every boundary above costs minutes and a few permissions to build.
Pick one today: find the broadest credential your agents can read and narrow it. The next post in this series walks through a restore drill you can run in an afternoon.
Sources and confidence
Confidence labels: A = primary document, official page, or major outlet. B = a single party’s account corroborated by press or the vendor, or a secondary tracker or law-firm summary.
Incidents
- PocketOS (B): ABC7 News, Fast Company, TechSpot, Decrypt (Cooper’s recovery statement), IBM. Much rests on Crane’s own account.
- Replit/SaaStr, July 2025 (B): Fortune, The Register (Masad’s statement and announced fixes), TechSpot
- Air Canada (A): Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal, February 14, 2024. Figures, the “remarkable submission” wording and the negligent-misrepresentation finding confirmed via CTV News and Dentons.
- Deloitte Australia (A): CFO Dive, The Register; Targeted Compliance Framework Assurance Review, DEWR.
Courts
- Damien Charlotin, AI Hallucination Cases database (A, live data; counts and party filters as of late October 2026)
- Couvrette v. Wisnovsky: ABA Journal (A); WSBA NW Sidebar (A); AI Vortex case record (B; fee and sanction split)
- Mata v. Avianca, No. 1:22-cv-01461 (S.D.N.Y. June 22, 2023) (A)
- Sullivan & Cromwell, In re Prince Global Holdings: Bloomberg Law, Bar & Bench (A); the firm’s April 18, 2026 letter to Chief Judge Glenn is the primary document
- Stanford RegLab/HAI, Magesh et al., “Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools”, Journal of Empirical Legal Studies 22:216–242 (2025); Stanford HAI summary (A). Tested product versions date from 2024.
- Count cross-checks: HAQQ, Oct 19 update, Inside Legal AI, Sep 9 (B)
Law
- Regulation (EU) 2026/1744, Digital Omnibus on AI, Official Journal, July 24, 2026 (A); confirmed by Orrick and DLA Piper (B). Check the consolidated text, as a corrigendum was reported.
- Colorado SB 26-189: Colorado Attorney General, ADMT rulemaking (A); signing date and effective date per Mondaq (B)