Zillow Offers and 7 More Verified AI and Algorithm Failures

Last verified: October 6, 2026. Every figure below was checked against a filing, a regulator, a tribunal decision, or at least two independent outlets. Where sources disagree, the disagreement is stated rather than smoothed over.

Most “AI gone wrong” lists share two flaws. They lump very different technologies together, and they repeat numbers nobody has checked. This one tries to fix both. It sorts each case by what actually failed, and it flags the claims that tend to get exaggerated.

The short version

  • Eight cases, 2012 to 2026, across finance, housing, retail, government, aviation, and consulting.
  • Only some are “AI” in the machine-learning sense. Others are plain software, rule-based automation, or software that worked exactly as designed and still caused harm.
  • In nearly every case, the failure was organizational: weak testing, no human check, ignored warnings, or a business model that assumed the world would hold still.
  • The costs range from a $12 million regulatory penalty to Australia’s A$1.8 billion Robodebt settlement (followed by A$475 million in additional compensation approved in 2026) and a fallen government.

First, what counts as an “AI failure”?

Calling every entry “AI” makes for a punchy headline and a wrong one. Sorting the cases into four tiers is more useful, and more honest.

TierWhat it meansCases in this article
Classical softwareNo learning involved; a deployment or control failureKnight Capital (2012)
Rule-based automationFixed logic applied at scale, with no human reviewRobodebt (2015–2019)
Machine learning and generative AILearned or generated outputs that were wrong, biased, or inventedDutch childcare benefits model, Amazon recruiting tool, Zillow Offers, Air Canada chatbot, Deloitte report
Algorithmic coordinationThe software did its job; the alleged harm is market conductRealPage (2024–2026; alleged, under active litigation)

Two classification notes. Zillow’s pricing technology is described publicly as ML-based, but Zillow has not published the exact model behind its Offers purchasing decisions, so “pricing algorithm” is the safest label. The Dutch system is described in secondary sources as a self-learning model, which is how it is labeled here.

Case files, in chronological order

1. Knight Capital (August 2012): the glitch that cost a company

Tier: classical software.

On August 1, 2012, a software installation error at Knight Capital sent erroneous orders into the market at the open. According to the SEC, the firm’s router sent more than 4 million orders in the first 45 minutes while trying to fill just 212 customer orders, trading more than 397 million shares. [S1]

The number, explained. Knight reported a realized pre-tax loss of approximately $440 million in its own filings. [S2] The SEC’s later order puts the loss at more than $460 million. [S1] Both are legitimate. The first is the company’s accounting figure and the second is the regulator’s. Use “about $440 million by the company’s count, over $460 million by the SEC’s.”

What actually failed. The SEC charged that Knight lacked adequate controls and procedures for code deployment and testing, and the firm paid a $12 million penalty. [S1] This was a controls failure around a deployment, not a model that “went rogue.” It earns a place here because it shows the pattern that recurs below: automation multiplies a small error into a company-threatening one faster than humans can react.

2. The Dutch childcare benefits scandal (2013–2021): when risk scoring ruined families

Tier: machine learning (self-learning risk model).

In 2013, the Dutch tax authority introduced a self-learning risk classification model to flag suspected childcare-benefit fraud. [S3] The model treated non-Dutch nationality as a risk factor, so a person without exclusively Dutch nationality was considered suspect by default. [S4]

The human cost is what makes this case different from the corporate ones. Roughly 26,000 families were wrongly accused of fraud, and the entire Dutch cabinet resigned in January 2021. [S5] Exact victim counts vary by source, with some saying “tens of thousands,” so attribute the figure rather than asserting it. The Dutch data protection authority later fined the tax administration €2.75 million in December 2021 for discriminatory processing of nationality data, followed by a further €3.7 million fine in April 2022. [S4][S6]

Why it matters for business readers. The model’s inputs were chosen by people. The discrimination was designed in rather than learned by accident, and it ran for years because few people with authority questioned the output.

3. Amazon’s recruiting tool (2014–2018): learning the wrong lesson from history

Tier: machine learning.

Reuters reported in October 2018 that Amazon had been building résumé-screening software since 2014. By 2015, the company realized it was not rating technical candidates in a gender-neutral way. [S7] The model had been trained on ten years of résumés, most from men. It penalized résumés containing the word “women’s” and downgraded graduates of two all-women’s colleges. [S7][S8]

Amazon edited the system to neutralize those specific terms, but could not guarantee it would not find other discriminatory proxies. The project was disbanded. [S8]

A claim to handle carefully. Reuters’ sources said recruiters looked at the tool’s recommendations but never relied on them alone. Amazon has said the tool was never used by its recruiters to evaluate candidates. [S9] The accurate framing is “built, tested, and abandoned,” not “deployed and discriminating at scale.”

4. Robodebt (2015–2019, with legal fallout through 2026): automation without a human in the loop

Tier: rule-based automation. Not AI.

Australia’s Robodebt scheme automatically sent debt notices to welfare recipients when income data from the tax office did not match their reported earnings. Under the earlier process, a Centrelink officer checked a debt before a notice went out. The new scheme dropped that check and relied on averaging income across 12 months. [S10] In 2019 the Federal Court found that averaging could not be used on its own to establish debts. [S10]

The government settled a class action for A$1.8 billion, approved by the Federal Court in June 2021. [S11] The settlement covered repayments, wiping of outstanding debts, and legal costs. The 2023 Royal Commission called the scheme a costly failure of public administration “in both human and economic terms.” [S11]

The bill kept growing. The A$1.8 billion headline was not the end. After the Royal Commission, Gordon Legal appealed the first settlement seeking further compensation. [S34][S35] A second class action settlement worth about A$475 million in additional compensation, shared among roughly 125,000 registered group members, was approved by the Federal Court on June 23, 2026. [S34][S36] The reported package also covers about A$60 million in administration costs and A$13.5 million in applicants’ legal costs. [S35] Under Gordon Legal’s published timetable, fixed payments are expected from about October 2026 to February 2027, with individually assessed claims running to 2028. [S34][S35] In September 2026, the government announced legislation to exempt the payments from tax, income testing and compensation recovery arrangements. [S37]

How to cite the cost. Do not add the A$1.8 billion and the A$475 million into one tidy total. The first figure was mostly refunds and the wiping of unlawful debts, with about A$112 million in compensation payments to group members. [S35] The second is new compensation for harm. Describe them as two separate settlements and let readers see the difference, or the number will mislead.

Number traps. Sources cite either 433,000 or 443,000 affected people. Say “more than 430,000.” Roughly A$751 million in unlawful debts was repaid by about 381,000 people before the scheme ended. [S12]

Why it belongs here. Robodebt shows that you do not need machine learning to cause algorithmic harm. A rigid rule, applied at scale and with the human check removed, is enough.

5. Zillow Offers (2021): the pricing model that overpaid

Tier: pricing algorithm (ML-based valuation tech).

Zillow’s board decided on November 2, 2021 to wind down Zillow Offers, its home-flipping business. The company cited home pricing unpredictability, capacity constraints, and operational problems made worse by the pandemic and labor and supply-chain conditions. [S13] Zillow expected the wind-down to cut about 25% of its workforce and to run into the second half of 2022. [S13]

The financial damage:

  • $304.4 million inventory write-down in Q3 2021 alone, because the company bought homes at higher prices than its estimates of what it could sell them for. [S14]
  • $407.9 million write-down for the full year 2021. [S13]
  • $881 million full-year loss attributed to the algorithm-driven Offers business. [S15]
  • A further $240–265 million in expected Q4 losses on homes still to be bought. [S16][S17]

Two nuances that fact-checkers will care about.

  1. Zillow’s own framing is forecasting failure, not “the model broke.” Founder Rich Barton said the unpredictability of home-price forecasting far exceeded what the company had anticipated, and that scaling Offers would bring too much earnings and balance-sheet volatility. [S18]
  2. Zillow did not abandon its valuation technology. Barton said the company would continue refining the Zestimate. [S18] The business died; the tool survived.

6. Air Canada’s chatbot (2022–2024): who owns a chatbot’s words?

Tier: machine learning and generative AI.

In November 2022, a customer asked Air Canada’s chatbot about bereavement fares after his grandmother died. The chatbot told him he could apply for the reduced fare retroactively after booking. The airline’s actual policy did not allow that. [S19] When he applied, the airline refused and offered a coupon instead. [S20]

On February 14, 2024, the British Columbia Civil Resolution Tribunal ruled in his favor (Moffatt v. Air Canada, 2024 BCCRT 149). [S19] Air Canada had argued the chatbot was effectively a separate legal entity responsible for its own actions. The tribunal rejected that, reasoning that the chatbot is part of the airline’s website and the airline is responsible for everything on it. [S21]

The number, explained. Reports vary because the award had several parts. The total payable was CAD $812.02, covering the refund, interest, and tribunal fees. [S22] If you want a single headline figure, say “about CAD $650 in damages, plus interest and fees.”

Context. This was a tribunal ruling, not a court decision, which limits its precedential weight. It is still widely cited as an early test of liability for AI-generated customer information. [S21]

7. RealPage (2024–2026): alleged algorithmic coordination under active litigation

Tier: algorithmic coordination.

This case is different in kind, and it is not a “failure” case in the way the others are. The allegation is not that RealPage’s rent-pricing software malfunctioned. It is that the software worked as designed, using landlords’ confidential data to nudge them toward similar pricing. RealPage denies any wrongdoing. [S23]

Where it stands as of October 6, 2026.

  • Federal settlement: The DOJ announced the settlement in November 2025. RealPage paid no damages and admitted no wrongdoing. Under the deal, nonpublic data used to train the model must be at least a year old. [S24] A North Carolina federal judge signed off on the settlement on May 19, 2026. [S25]
  • State claims continue: On October 1, 2026, a federal judge said the DOJ settlements did not address the core of the monopolization and information-sharing claims, which would move forward under a group of state attorneys general. [S26] Ten states had declined to join the federal deal. [S24] A report dated October 4 describes the judge ruling the remaining state claims not moot. [S27]
  • Private litigation: Landlord Mid-America Apartment Communities disclosed a class-action settlement in January 2026, payable in two installments of $26.5 million. It was subject to court approval when disclosed. [S28]

8. Deloitte Australia (2025): the report with invented citations

Tier: generative AI.

Deloitte was commissioned by Australia’s Department of Employment and Workplace Relations to review a welfare compliance framework. The 237-page report, published in July 2025, was found to contain a fabricated quote from a federal court judgment and references to nonexistent research papers. [S29] A Sydney University researcher, Chris Rudge, said he found up to 20 errors. [S29][S30]

The revised version disclosed that a generative AI tool (Azure OpenAI GPT-4o) was used in producing the report. [S30] Deloitte agreed to repay the final installment of its AU$440,000 (about US$290,000) contract. [S29][S31] The firm said the matter was resolved with the client and said the AI-generated errors did not affect the report’s substance or recommendations. [S32]

A wording trap. Deloitte did not publicly attribute the errors to AI. [S30] The accurate phrasing is “an AI-assisted report contained fabricated citations,” not “Deloitte blamed AI.”

What the eight cases share

The patterns below are analysis rather than reported fact, so treat them as arguments.

1. The failure is usually organizational, not technical. Knight lacked deployment controls. Robodebt removed the human check. Deloitte did not catch fabricated references before delivery. Zillow’s own account points to forecasting limits and operations, not a single broken component.

2. Models inherit the past. Amazon’s tool learned a decade of male-dominated hiring. Zillow’s pricing leaned on a market that then shifted. Both extrapolated confidently from a world that had stopped existing.

3. Human discretion is the first thing removed and the last thing restored. Robodebt’s defining change was dropping manual review. The Dutch system flagged families who then had to prove their innocence. The cheapest safeguard, a person who can say “this looks wrong,” is the one most often engineered out.

4. Liability stays with whoever deploys the tool. Air Canada tried to hand responsibility to its chatbot and lost. Deloitte’s refund is the commercial version of the same principle.

5. Scale turns small errors into balance-sheet events. Knight’s error ran for 45 minutes. Zillow’s overpayment compounded across thousands of purchases. Speed and volume are the point of automation, and also the reason mistakes compound.

6. Fixes tend to be partial. Amazon patched specific terms but could not rule out other proxies. Air Canada’s chatbot appeared to be taken offline after the ruling, according to press observation rather than an official statement. [S33]

An unpopular take. The “AI is dangerous” framing lets organizations off the hook. In most of these cases the technology did roughly what it was built to do. The decisions that failed were human ones: what data to use, what to automate, what to skip testing, and who was allowed to object.

A pre-deployment checklist drawn from these cases

Use this as a starting point, not a guarantee.

  1. Who can override it? Name the person or role with authority to reject the system’s output, and confirm they have time to do it.
  2. What does it learn from? Identify any historical pattern in the training data you would not want to repeat.
  3. What does it use as a proxy? Check whether inputs such as nationality, schools, or zip codes stand in for protected characteristics.
  4. What is the worst 45 minutes? Estimate the loss if it misbehaves at full speed before anyone notices, and add a kill switch sized to that.
  5. Who answers for its words? Assume you are liable for anything the system tells a customer.
  6. Who reviews AI-assisted deliverables? Require a human to verify every citation, quote, and number before it ships.
  7. What happens when the market changes? Define the condition under which you stop trusting the model.

Frequently asked questions

Was Zillow’s failure caused by AI? Zillow’s pricing technology is described as ML-based, but the company’s own explanation emphasizes forecasting uncertainty and operational limits alongside the pandemic environment. “The algorithm was solely to blame” is not supported by the filings. [S13]

Is Knight Capital an AI failure? No. It was a software deployment failure in an automated trading system. [S1] It is included because it shows how automation amplifies errors.

Was Amazon’s recruiting tool ever used to hire people? Reports say recruiters saw its recommendations but did not rely on them alone, and Amazon says the tool was never used by its recruiters to evaluate candidates. [S9]

Can a company be held responsible for its chatbot’s mistakes? In Moffatt v. Air Canada, a Canadian tribunal said yes, rejecting the argument that a chatbot is a separate legal entity. It was a tribunal ruling, not a binding court precedent. [S21]

Did RealPage lose its case? Not exactly. It settled with the DOJ without paying damages or admitting wrongdoing, while state-level claims continue. [S24][S26]

Sources

Primary and regulatory sources are listed first within each case where available.

Knight Capital

  • [S1] SEC press release 2013-222, “SEC Charges Knight Capital With Violations of Market Access Rule”: https://www.sec.gov/news/press-release/2013-222
  • [S2] Knight Capital Group, Form 8-K (August 2, 2012): https://www.sec.gov/Archives/edgar/data/0001060749/000119312512332176/d391111dex991.htm

Dutch childcare benefits scandal

Amazon recruiting tool

Robodebt

Zillow Offers

  • [S13] Zillow Group Form 10-K, FY2021: https://www.sec.gov/Archives/edgar/data/1617640/000161764022000013/z-20211231.htm
  • [S14] Zillow Group Form 10-Q, Q3 2021: https://www.sec.gov/Archives/edgar/data/1617640/000161764021000087/z-20210930.htm
  • [S15] National Mortgage Professional, “More Job Cuts Coming At Zillow”: https://nationalmortgageprofessional.com/news/more-job-cuts-coming-zillow
  • [S16] Daily Hive, “Zillow axes home-flipping business after losing $381 million”: https://dailyhive.com/seattle/zillow-offers-shutting-down
  • [S17] Inman, November 2, 2021: https://inman.com/2021/11/02/zillow-to-shut-down-ibuying-program-zillow-offers
  • [S18] Inman, same article, for Rich Barton’s statements on forecasting and the Zestimate.

Air Canada

RealPage

  • [S23] RealPage statement via National Mortgage Professional: https://nationalmortgageprofessional.com/news/realpage-settles-justice-department-software-steering-suit
  • [S24] Associated Press via Minnesota Lawyer, November 26, 2025: https://minnlawyer.com/2025/11/26/doj-realpage-rent-pricing-algorithm-antitrust-settlement/
  • [S25] Law360, “NC Judge OKs DOJ, RealPage Deal In Antitrust Suit,” May 19, 2026: https://www.law360.com/amp/articles/2479431
  • [S26] MLex, October 1, 2026: https://www.mlex.com/mlex/antitrust/articles/2533012
  • [S27] Hoodline, October 4, 2026: https://hoodline.com/2026/10/federal-judge-lets-rent-fixing-case-against-realpage-and-landlords-proceed
  • [S28] Mid-America Apartment Communities Form 8-K, January 26, 2026: https://www.sec.gov/Archives/edgar/data/912595/000119312526027241/maa-20260126.htm

Deloitte Australia