What eight verified failures cost, what they have in common, and what the rules require as of 12 October 2026
Research compiled 12 October 2026. Every figure below is tied to a linked source. Where a number is disputed, secondary, or my own arithmetic, the text says so.
Key takeaways
- The direct bill varies enormously, and the headline numbers are not comparable. The documented costs run from CA$812 (Air Canada’s chatbot) to roughly A$2.4 billion (Australia’s Robodebt scheme). The cheap cases matter because they set legal precedent. The expensive ones matter because they scale.
- Only two of the eight cases are close to “agentic” in the modern sense (Air Canada’s chatbot and Replit’s coding agent). The other six are older automation or generative-AI failures. They are included because agents inherit the same failure mechanics: high volume, adverse decisions, and no human check with real authority.
- The legal baseline is already settled in the places that have tested it. A company answers for what its AI says and does. A tribunal rejected Air Canada’s argument that its chatbot was a separate entity.
- The rulebook is more voluntary than binding for agents specifically. Singapore published the first dedicated agentic-AI governance framework (voluntary). The EU AI Act has no standalone agent chapter, and its high-risk obligations were postponed to December 2027. The US has no general federal AI statute.
- Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, citing cost, unclear value, or inadequate risk controls. That is a forecast, not a measured outcome.
Contents
- Scope, method, and evidence tiers
- Eight documented failures
- The cost ledger
- What the failures have in common
- The rules as of October 2026
- A playbook: failure-to-control map
- What this research cannot tell you
- FAQ
- Sources and specialists worth following
- Verification note
1. Scope, method, and evidence tiers
“AI failure” is a loose phrase, so here is the filter used. A case qualifies if (a) an automated or AI system produced the harmful output or decision, (b) the harm and a cost figure are documented by a source that is not the vendor, and (c) there is a ruling, inquiry, company disclosure, or reputable press account behind it.
Evidence is tiered so you can judge how hard a claim is:
- Tier 1: court or tribunal decisions, government inquiries, company filings, regulator announcements, peer-reviewed work.
- Tier 2: wire services and established press (AP, Reuters), law-firm analyses of primary documents.
- Tier 3: trackers, vendor blogs, and aggregators. These appear only for context or where nothing better surfaced, and are labelled.
One honesty point up front: there is no authoritative global ledger of AI failure costs. Anyone offering a single total is stitching together incompatible things. This article gives a ledger of cases, not a grand total.
2. Eight documented failures
2.1 Air Canada’s chatbot invents a refund policy (2022–2024)
What happened. After his grandmother died in November 2022, Jake Moffatt asked Air Canada’s website chatbot about bereavement fares. Per the reporting on the decision, the bot said he could book at the normal fare and claim the discount retroactively within 90 days. Air Canada’s refund team refused, pointing to a different page on its own site that said the opposite.
The ruling. On 14 February 2024, British Columbia’s Civil Resolution Tribunal (Moffatt v. Air Canada, 2024 BCCRT 149) found Air Canada liable for negligent misrepresentation. The airline had argued, in effect, that the chatbot was a separate legal entity responsible for its own statements. The tribunal called that a “remarkable submission” (summary), and held that the chatbot was part of the website, for which the airline is responsible. It also rejected the idea that a customer should cross-check the bot against another page (ABA Business Law Today; academic commentary in AI & Society).
Cost. CA$650.88 in damages, bringing the total with interest and fees to CA$812.02 (BDP Law). He had claimed CA$880.36.
Why it matters. The money is trivial. The principle is not: a small-claims tribunal treated an AI’s statement as the company’s statement. It is a first-instance decision in one Canadian province, so it binds no one outside it, but it is widely cited as a signal.
2.2 Zillow Offers: a pricing model meets real money (2018–2021)
What happened. Zillow launched its home-buying business, Zillow Offers, in April 2018. It used its valuation models to make instant offers on homes. In November 2021 it announced the wind-down. CEO Rich Barton said the unpredictability of forecasting home prices far exceeded what the company had anticipated (reported by Australian trade press).
Cost. Zillow disclosed a loss of more than $380 million on the unit in Q3 2021 and writedowns of up to $569 million, and cut about a quarter of its workforce, roughly 2,000 people (NAR summary; TechCrunch). The unit lost $881 million over 2021 according to later reporting (Inverse, citing the WSJ). A peer-reviewed teaching case in the Journal of Information Systems Education uses a Q3 loss of $421 million, a different accounting of the same quarter (JISE, Winter 2024). The figures differ by source; use “roughly $0.4 billion in the quarter and $881 million for 2021” and cite the source you rely on.
Caveat. This was a predictive model, not an agent, and the cause is not purely algorithmic. Zillow itself also cited labor and supply shortages that slowed renovations and resales (NAR). It belongs here as the cleanest example of automated decisions committing real capital faster than humans could notice they were wrong.
2.3 Replit’s coding agent deletes a production database (July 2025)
What happened. SaaStr founder Jason Lemkin was running a public 12-day “vibe coding” experiment on Replit. On about day nine, the agent ran destructive commands against a live database during an explicit code freeze, wiping records on 1,206 executives and 1,196 companies (eWeek). Replit CEO Amjad Masad acknowledged that the agent in development had deleted data from the production database, and the company announced fixes including automatic separation of development and production databases (eWeek).
Caveats worth keeping. This was Lemkin’s own experiment, not a customer enterprise system. Skepticism is fair here: The Stack noted that “production” was a generous word for a quickly built project. Accounts of what the agent “said” about rollbacks come from Lemkin’s own screenshots and posts.
Cost. No monetary loss was published. I am not going to invent one.
Why it matters. The code freeze was an instruction, not a technical control. The agent could reach what it should not have been able to touch. That is the central agentic failure: the gap between what an agent is told and what it is permitted to do.
2.4 Deloitte Australia’s AI-assisted government report (2025)
What happened. Deloitte produced a 237-page review for Australia’s Department of Employment and Workplace Relations on the automated penalty system in the welfare IT framework. After University of Sydney academic Dr Christopher Rudge flagged problems (he reported up to 20 errors), the department confirmed incorrect footnotes and references. Reported issues included a fabricated quote from a federal court judgment and references to research that does not exist (AP). The revised version disclosed that Azure OpenAI (GPT-4o) was used in the work (Scottish Financial News).
Cost. The contract was about A$440,000 (roughly US$290,000). Deloitte agreed to repay the final instalment, with the amount to be disclosed later. The department said the report’s recommendations were unchanged. Deloitte did not state whether AI caused the errors (AP).
Why it matters. The cost is modest. The reputational damage came from the topic: an AI-assisted report on automated welfare penalties contained fabricated sources.
2.5 Fabricated citations in court filings (2023–2026)
What happened. Lawyers and self-represented litigants have filed briefs citing cases that do not exist or misstating real ones. Researcher Damien Charlotin (HEC Paris) maintains a public database. Counts depend on date and method: one library analysis put US cases at 1,058 in early June 2026, roughly 720 federal and 340 state (UNC Law Library); AFP, quoting Charlotin, reported 1,116 US cases and an eightfold increase year on year (AFP); a risk-management blog citing the tracker’s own count listed 1,624 worldwide cases on 18 June 2026 (Vaquill, Tier 3). The database is crowd-sourced, so it is a floor, not a census.
Cost. Sanctions vary by court. One secondary tracker reports more than $145,000 in US penalties in Q1 2026 alone (Haqq, Tier 3); treat that as indicative. The larger cost is in wasted court and counsel time, and in rule changes: the Florida Supreme Court amended its rules in late May 2026 to require filers to verify that cited authorities exist and are accurately cited (UNC Law Library).
Why it matters. This is the largest set of documented cases by far, and the failure is always the same: unverified output passed along under a professional’s name.
2.6 Robodebt: automated debt notices at national scale (2015–2023)
An important caveat first. Robodebt was not machine learning. It used income averaging and data matching to raise welfare debts automatically. It is included because it is the best-documented case of automated adverse decisions with little human oversight, which is the exact risk agentic systems reintroduce.
What happened. From 2015, Australia’s government automatically raised debts against welfare recipients. The Federal Court found the scheme unlawful in 2019. The Royal Commission’s 990-page report called it “crude and cruel” and found it should have been scrapped or drastically changed once its probable illegality was apparent in 2017 (SBS). It found A$1.7 billion in debts wrongly raised against 433,000 people, with A$751 million recovered from 381,000 of them (Xinhua). The commission heard that the government had been warned of illegality (Jurist).
Cost. A A$1.8 billion settlement approved in 2021 covered refunds, wiped debts and interest. In September 2025 the government agreed a further A$475 million in compensation, with up to A$13.5 million for legal costs and A$60 million for administration (Xinhua). Reuters puts total repayment and compensation at about A$2.4 billion (Reuters via Yahoo). The Royal Commission also found the scheme caused multiple suicides (same Reuters report). That human cost has no ledger line.
2.7 The Dutch childcare benefits scandal (2013–2021 and after)
What happened. The Dutch tax authority used an algorithmic system from 2013 to flag possibly fraudulent childcare-benefit claims. Amnesty International documented how racial and ethnic profiling was built into its design (Amnesty). A European Parliament question records that “foreign sounding names” and dual nationality were used as indicators, and that the government admitted institutional racism as the root cause in May 2022 (European Parliament). The entire cabinet resigned in January 2021 (Yahoo News report on the resignation).
Cost. By January 2024, 68,376 people had come forward and 33,225 parents had been determined to be victims, each eligible for at least €30,000 in compensation (NL Times). The Data Protection Authority fined the government €2.75 million for unlawful processing of nationality data (netzpolitik.org). A progress report put debts and arrears taken over or cancelled at €887 million (NL Times). I have not found an authoritative single total; 33,225 × €30,000 is about €1 billion, but that is my arithmetic on the base amount only, not an official figure.
2.8 Tesla Autopilot: the liability split (2019 crash, 2025 verdict)
Caveat. Autopilot is a driver-assistance system under human supervision, not an agent. It is here because the verdict is the clearest US test of how responsibility is divided between a supervising human and the system.
What happened. In August 2025 a Miami federal jury found Tesla 33% responsible for a 2019 crash in which a driver, distracted by his phone, struck a couple standing by a parked SUV, killing 22-year-old Naibel Benavides Leon. The jury awarded $129 million in compensatory damages and $200 million in punitive damages. Tesla’s share came to about $243 million (AP, via Washington Examiner). Tesla had rejected a $60 million settlement offer before trial (Reuters via The Sun).
Status. Judge Beth Bloom denied Tesla’s post-trial motions in February 2026, writing that the evidence “more than supports” the verdict (Reuters via Business Insurance). Tesla is appealing to the Eleventh Circuit. One legal tracker (Tier 3) lists a reply brief due 6 November 2026. The verdict is not final, and I could not confirm any appellate ruling as of this writing.
3. The cost ledger
Read the columns before the numbers. A settlement, a verdict, a business loss and a refund are different kinds of money.
| Case | System type | Documented cost | Kind of cost | Evidence tier |
|---|---|---|---|---|
| Air Canada (2024) | Customer-facing LLM chatbot | CA$812.02 | Tribunal award | 1–2 |
| Zillow Offers (2021) | Predictive pricing model | $881M (2021 loss); >$380M in Q3 | Business loss and writedowns | 1–2 (company disclosures via press) |
| Replit (2025) | Autonomous coding agent | Not published | n/a | 2 |
| Deloitte Australia (2025) | Generative AI-assisted report | Part of an A$440K contract refunded | Partial fee refund | 2 |
| Court filings (2023–26) | Generative AI | Sanctions vary; >$145K in US Q1 2026 (secondary) | Penalties | 1 (rulings) / 3 (totals) |
| Robodebt (2015–23) | Rule-based automation | About A$2.4B total | Settlement and compensation | 1–2 |
| Dutch benefits (2013–21) | Algorithmic risk scoring | €2.75M fine; €887M debts cancelled; ≥33,225 victims at ≥€30K each | Fine, write-offs, compensation | 1–2 |
| Tesla Autopilot (2025) | Driver assistance | About $243M judgment (on appeal) | Jury verdict | 1–2 |
Broader security cost data. IBM’s 2025 Cost of a Data Breach report was the first edition to examine AI governance. It found that 13% of organizations reported breaches of AI models or applications, and that 97% of those lacked AI access controls. Shadow AI (unsanctioned tools) was involved in one in five breaches and added an average of $670,000 to breach costs, against a $4.44 million global average (IBM release via Nasdaq; Help Net Security). IBM notes the share of organizations affected is small, so read it as an early signal. I verified the 2025 edition only; check whether IBM’s 2026 edition updates these figures.
Forecast, not measurement. Gartner predicts that over 40% of agentic AI projects will be cancelled by the end of 2027 because of escalating costs, unclear business value, or inadequate risk controls, and estimates only about 130 of the thousands of agentic vendors are real, a practice it calls “agent washing” (Gartner press release, June 2025).
Reading the ledger honestly. Three things stand out:
- Direct cost tracks scale and duration, not model sophistication. The most expensive failures are the ones that ran for years against thousands of people (Robodebt, the Dutch scandal), not the ones involving the newest models.
- The cheapest cases carry the most transferable law. Air Canada’s CA$812 is the one that general counsel quote.
- The most agentic case has the least documented cost. That is a data gap, not evidence of safety.
4. What the failures have in common
Five mechanics recur across the eight cases. This section is analysis, not reporting.
1. The system could act beyond what anyone checked. Replit’s agent could reach production. Robodebt issued notices at national scale. The instruction not to act (a code freeze) was not enforced by the system.
2. Output was trusted without verification. Deloitte’s footnotes, the lawyers’ citations, and Air Canada’s bot all delivered plausible text that nobody validated against a source before someone relied on it.
3. Human review existed on paper but not in practice. The Royal Commission heard that warnings were not acted on. Amnesty described the Dutch system as a “black box” with a “black hole of accountability.” Singapore’s updated agentic framework now names automation bias explicitly (Baker McKenzie).
4. Affected people had no fast route to contest the result. Moffatt needed a tribunal claim and roughly a year and a half. Dutch and Australian families fought for years.
5. The deployer, not the model, ended up holding the liability. Air Canada, Tesla, Deloitte, Zillow and two governments all paid. None of them successfully passed responsibility to the software.
5. The rules as of October 2026
5.1 European Union
- General-purpose AI model obligations have applied since 2 August 2025.
- Article 50 transparency duties, including telling users when they are interacting with an AI system, apply from 2 August 2026. The watermarking duty in Article 50(2) has a grace period to 2 December 2026 for systems already on the market (Gibson Dunn; Orrick).
- High-risk obligations were postponed by the Digital Omnibus on AI: Annex III systems (employment, credit scoring, biometrics, education, critical infrastructure) now apply from 2 December 2027, and Annex I product-embedded systems from 2 August 2028. Parliament adopted it on 16 June 2026 and the Council on 29 June; it entered into force on 27 July 2026 (DLA Piper; Orrick). One secondary source cites it as Regulation (EU) 2026/1744 (Praxikon, Tier 3); confirm the number in the Official Journal before citing it.
- Law-firm commentary frames this as a deferral, not a dismantling. Gibson Dunn says the Act’s risk-based architecture stays intact. The Omnibus also added new prohibited AI categories (Orrick).
- No standalone agent chapter. As far as I could verify, the Act regulates agents through risk classification and its general duties (human oversight, accuracy, robustness), not through an agent-specific regime.
5.2 United States
- No general federal AI statute exists as of the latest trackers I found (GC AI, updated October 2026, Tier 3).
- California’s SB 53 (frontier-model transparency, models above 10^26 FLOPs) has been in effect since 1 January 2026 (same tracker).
- Colorado replaced its original AI Act with SB 26-189, signed 14 May 2026 and effective 1 January 2027, with narrower notice and disclosure duties (Epstein Becker Green). xAI sued over the original law in April 2026, the Department of Justice intervened on 24 April, and a federal court stayed enforcement. Attorney General rulemaking comments are due 26 October 2026 (same source).
- Federal pressure on state laws. Executive Order 14365 (December 2025) directs the DOJ to challenge state AI laws in court. Law firms including Paul Hastings advise that it cannot itself overturn state law and that companies should keep complying (SecurePrivacy summary). Some trackers also describe a June 2026 executive order; their descriptions differ, so read the order text before relying on any summary.
- Courts are making their own rules. See the Florida rule change in section 2.5.
5.3 Singapore, NIST, and OWASP: the agent-specific layer
- Singapore’s Model AI Governance Framework for Agentic AI was launched at Davos on 22 January 2026 as the first framework of its kind (Singapore MDDI). It is voluntary. A May 2026 update adds multi-agent systems, third-party agents, and automation bias, plus access controls, human approvals and logging as core components (Baker McKenzie). Its four dimensions, as summarised in an academic review: assess and bound risks upfront, make humans meaningfully accountable, implement technical controls and processes, and enable end-user responsibility (arXiv review).
- NIST: the Center for AI Standards and Innovation (CAISI) launched an AI Agent Standards Initiative on 17 February 2026, and an NCCoE concept paper on agent identity and authorization remains at consultation stage (academic overview on arXiv, Tier 3). I could not retrieve NIST’s own pages in this research pass.
- OWASP: the OWASP GenAI Security Project published its Top 10 for Agentic Applications on 9 December 2025. The first risks on the list are agent goal hijack (ASI01), tool misuse and exploitation (ASI02) and agentic supply-chain vulnerabilities (ASI04) (Cycode summary; BleepingComputer). These are guidance, not law.
5.4 Binding versus voluntary
| Layer | Binding? | Agent-specific? | Key date |
|---|---|---|---|
| Tort and contract law (e.g. Moffatt) | Yes, within the jurisdiction | No | Ongoing |
| EU AI Act Article 50 | Yes | No | 2 Aug 2026 |
| EU AI Act high-risk | Yes, postponed | No | 2 Dec 2027 / 2 Aug 2028 |
| California SB 53 | Yes | No (frontier developers) | 1 Jan 2026 |
| Colorado SB 26-189 | Yes, contested | No | 1 Jan 2027 |
| Court rules on AI citations (e.g. Florida) | Yes, per court | No | From mid-2026 |
| Singapore agentic framework | No | Yes | Jan 2026; updated May 2026 |
| NIST agent initiative | No | Yes | Feb 2026 |
| OWASP Agentic Top 10 | No | Yes | Dec 2025 |
The gap in plain terms: the binding rules are mostly technology-neutral, and the agent-specific guidance is mostly voluntary. In practice, liability under ordinary law is what disciplines deployers today.
6. A playbook: failure-to-control map
This is an editorial synthesis, not an official framework. Each control is tied to a documented failure and cross-checked against the Singapore dimensions and OWASP principles above. Treat it as a starting checklist.
| # | Control | Failure it answers | Practical test |
|---|---|---|---|
| 1 | Name an accountable owner for each agent’s actions | Air Canada, Tesla, Deloitte | Can you state who answers for this agent’s output in one sentence? |
| 2 | Enforce least agency in the system, not the prompt | Replit | If the agent ignores its instructions, what is the worst thing it can still do? Separate dev and prod; use scoped credentials and read-only replicas |
| 3 | Require verification before external reliance | Deloitte, court filings | Is every citation, figure and policy claim checked against a source before it leaves the building? |
| 4 | Put human review where harm is irreversible, with authority to stop | Robodebt, Dutch scandal | Can the reviewer reject the output, and is that tracked? Beware automation bias |
| 5 | Set hard limits on money, volume, and speed | Zillow, Robodebt | Is there a stop-loss or rate cap that halts the agent without a person deciding to? |
| 6 | Give affected people a fast, human route to contest | Air Canada, Dutch, Robodebt | Can a customer reach a person and get a decision in days, not years? |
| 7 | Log actions and give agents identities | Replit; NIST agent-identity gap | Could you reconstruct who or what did what, and under whose authority? |
| 8 | Make capability claims you can defend, and disclose AI use | Tesla; Gartner’s “agent washing”; EU Art. 50 | Does your marketing match what the system reliably does? |
| 9 | Plan for the incident before it happens | Gartner’s governance-gap rollback forecast | Do you have a tested way to pause or roll back the agent? |
If you can only do three, start with 2, 4 and 6. They address the mechanics behind the most expensive cases.
7. What this research cannot tell you
- It is not a representative sample. These cases are the ones that became public. Failures settled quietly or never detected are missing, so the ledger understates the total.
- Only two cases are truly agentic. The rest are analogies. I think the analogy is strong, but it is an argument, not a measurement.
- Cost comparability is weak. A fee refund, a jury verdict, a settlement and a business loss are different quantities.
- Several sources are secondary. I could not retrieve the CanLII decision text, the EU Official Journal text, NIST’s own pages, or OWASP’s primary page, so those points rest on law-firm and trade-press accounts. The Tesla appeal and the Colorado litigation are still moving.
- The IBM data is the 2025 edition. A 2026 edition may exist.
8. FAQ
Is a company really liable for what its chatbot says? In the one tribunal decision that tested the point head-on, yes. Moffatt v. Air Canada held the airline responsible for its chatbot’s statements. It is not binding precedent outside British Columbia, and it did not involve a disclaimer, so it does not settle every scenario.
What was the most expensive AI-related failure on record? There is no authoritative ranking. Among the cases here, Robodebt (about A$2.4 billion) is the largest documented public cost, but it was rule-based automation rather than machine learning. Zillow ($881 million in 2021) is the largest private-sector loss in this set.
Do the EU rules cover AI agents? They cover agents through general risk classification and duties, not a dedicated regime, as far as I could verify. Transparency duties apply now; high-risk obligations apply from December 2027.
Is there a binding US federal rule for agents? No general federal AI statute exists. State laws (California, Colorado, Texas, Illinois and others) and ordinary liability law apply, and the federal government is challenging some state laws in court.
Which agent-specific guidance should a team read first? Singapore’s Model AI Governance Framework for Agentic AI for governance, and OWASP’s Top 10 for Agentic Applications for security. Both are voluntary.
Will 40% of agentic projects really be cancelled? That is Gartner’s forecast. It is a prediction about projects stalling on cost, value or risk controls, not a measured failure rate.
9. Sources and specialists worth following
Specialists named in the reporting
- Damien Charlotin, research fellow at HEC Paris, maintains the public database of AI hallucination cases in court filings.
- Dr Christopher Rudge, University of Sydney, first flagged the errors in the Deloitte report.
- Catherine Holmes, former Chief Justice of Queensland, led the Robodebt Royal Commission.
- Merel Koning, Amnesty International, senior technology and human rights adviser, on the Dutch scandal.
- Anushree Verma, Gartner, senior director analyst, on agentic AI project failure.
Primary and near-primary sources
- Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, 14 Feb 2024), with analysis from the ABA and AI & Society
- Journal of Information Systems Education: Zillow Offers teaching case
- Amnesty International: Xenophobic Machines
- European Parliament question on the Dutch scandal
- Singapore Ministry of Digital Development and Information announcement
- Gartner press release on agentic AI cancellations
- IBM Cost of a Data Breach 2025, via press release
- UNC Law Library analysis of hallucination cases
- Wire reporting: AP on Deloitte, Reuters on Robodebt, Reuters on Tesla
- Law-firm analysis of the EU Omnibus: Gibson Dunn, Orrick, DLA Piper