Banner Default Image

Amazon, Facebook internet outage: Verizon blamed for 'cascading catastrophic failure'

almost 5 years ago by Lucy Cinder

Amazon, Facebook internet outage: Verizon blamed for 'cascading catastrophic failure'


A Verizon error resulted in a "cascading catastrophic failure" that triggered widespread internet outages on Monday, affecting Cloudflare, Amazon, Facebook, and others. 

Website accelerator firm Cloudflare described the event as a "small heart attack" that left websites that rely on it, such as voice and text chat app Discord, unreachable from many parts of the internet for about two hours. And, according to Cloudflare, it was completely avoidable.  

The outage was due to a so-called route leak from a Verizon customer. In Cloudflare's case, this meant traffic that normally goes through Verizon and Level 3 Communications to reach Cloudflare instead went through a metal maker called Allegheny Technologies, Pennsylvania-based ISP DQE, and Cogent Communications onwards to Cloudflare. 

Allegheny and DQE's networks weren't up to the task of such a massive spike in traffic.

At the heart of yesterday's outage was the Border Gateway Protocol, which networks use to share information about what routes to take. Cloudflare says DQE incorrectly announced routes from its network to its customer, Allegheny. 

That routing information was passed on to Verizon, which "proceeded to tell the entire internet about these 'better' routes," explained Cloudflare engineer Tom Strickx.

"The leak should have stopped at Verizon. However, against numerous best practices outlined below, Verizon's lack of filtering turned this into a major incident that affected many internet services such as Amazon, Linode, and Cloudflare."  

Banner Default Image

Latest Jobs