How four rotten packets broke CenturyLink's network for 37 hours, knackering 911 calls, VoIP, broadband
.jpg)
A handful of bad network packets triggered a massive chain reaction that crippled the entire network of US telco CenturyLink for roughly a day and a half.
This is according to the FCC's official probe into the December 2018 super-outage, during which CenturyLink's broadband internet and VoIP services fell over and stayed down for a total of 37 hours. This meant subscribers couldn't, among other things, call 911 over VoIP at the time – which is a violation of FCC rules, and triggered a formal investigation.
"This outage was caused by an equipment failure catastrophically exacerbated by a network configuration error," America's communications regulator said in its summary of its inquiry, published yesterday.
"It affected communications service providers, business customers, and consumers who directly or indirectly relied upon CenturyLink’s transport services, which route communications traffic from various providers to locations across the country, resulting in extensive disruptions to phone service, including 911 calling."
CenturyLink has six long-haul networks that make up the backbone of its digital empire, interconnecting regions of America. These networks use Infinera-built nodes to switch packets over high-speed optic fiber: data flowing into each node is directed to other nodes, ultimately pumping VoIP, regular internet traffic, and more, across the nation as needed.
Each dodgy packet would arrive at a node, get rejected and be passed along a chain of filters until it was injected into a management channel and handed to all connecting nodes. Here's a flow diagram, courtesy of the FCC, showing how the corrupted packets ended up being forwarded on to all neighbouring nodes, and so on and so on, producing a growing chain reaction of corrupted packets.
"Due to the packets’ broadcast destination address, the malformed network management packets were delivered to all connected nodes. Consequently, each subsequent node receiving the packet retransmitted the packet to all its connected nodes, including the node where the malformed packets originated," the FCC said in its report.
"Each connected node continued to retransmit the malformed packets across the proprietary management channel to each node with which it connected because the packets appeared valid and did not have an expiration time. This process repeated indefinitely."
As you might imagine, the exponentially growing storm of packets soon overwhelmed CenturyLink's optic-fibre backbone, causing regular traffic to stop flowing: VoIP phones stopped working, internet access slowed to a halt, and so on. Folks in New Orleans were first to spot their connections stalling, at roughly 0356 EST on December 27.
Here is where things went from really, really bad to terrible: the nodes along the fiber network were so flooded, they could not be reached by their administrators to troubleshoot the issue. It wasn't until some 15 hours later the techies could finally track down the single errant node in Colorado responsible for sparking the deluge, not that replacing it helped. The packet tsunami was still washing back and forth, knocking nodes over.
"At 2102 on December 27, CenturyLink network engineers identified and removed the module that had generated the malformed packets," the report noted. "The outage, however, did not immediately end; the malformed packets continued to replicate and transit the network, generating more packets as they echoed from node to node."
It would be another three hours before CenturyLink's network admins could begin to get through to the other nodes and get them to kill off the spread of bad packets. It took until 1130 on December 28 to get visibility of the network back, and it wasn't until 2336 that all nodes had been restored. On December 29, just after midday, CenturyLink finally declared the crisis over.
"The event caused a nationwide voice, IP, and transport outage on CenturyLink’s fiber network. CenturyLink estimates that 12,100,108 calls were blocked or degraded due to the incident," the FCC said.
"Where long-distance voice callers experienced call quality issues, some customers received a fast-busy signal, some received an error message, and some just had a terrible connection with garbled words."
The outage also knackered local governments and telcos that relied on the CenturyLink network for portions of their services. State governments in Illinois, Kansas, Minnesota, and Missouri all had portions of their networks down for roughly 36 hours thanks to CenturyLink, and phone services sold by Comcast, Verizon, TeleCommunication Systems, General Dynamics IT, and West Safety Services – including 911 call centers – saw connectivity interrupted for some or all of the outage period.
As to what can be done to prevent similar failures, the FCC is recommending CenturyLink and other backbone providers take some basic steps, such as disabling unused features on network equipment, installing and maintaining alarms that warn admins when memory or processor use is reaching its peak, and having backup procedures in the event networking gear becomes unreachable.
"Currently, CenturyLink is in the process of updating its nodes’ Ethernet policer to reduce the chance of the transmission of a malformed packet in the future," the report notes. "The improved ethernet policer quickly identifies and terminates invalid packets, preventing propagation into the network. This work is expected to be completed in Fall, 2019."
source theregister
Industry: Unified Communications & Telecommunications

Latest Jobs
-
- Identity Channel Partner Manager | London
- London
- N/A
-
Identity Channel Partner Manager | London Location: South East UK (commutable to London) We are working with a Cyber Security business who are looking for a Channel Partner Manager to drive and grow relationships across their identity ecosystem. Prior experience working within VARs, distributors, vendors or resellers in the identity space is essential. You must have experience working with technologies such as CyberArk, Sailpoint, Okta etc Responsibilities will include, but not be limited to: Build, maintain and develop strong relationships with channel partners. Work closely with partner sales teams to support growth drive sales opportunities. Identify and onboard new partners while strengthening existing partnerships. Act as the key point of contact for all channel-related activity. If you are an experienced channel professional, with experience in the Identity space and are ready for your next challenge, apply today.
-
- Service Architect- DACH regions
- Germany
- Upto €110,000 plus bonus and benefits
-
Lead Service Architect with the authority and experience to take control of complex, multi-million-euro outsourcing bids. This role is about leading the Service/ solutioning effort, bringing structure to chaos, and driving the entire bid team to deliver winning proposals. The company area a global managed services business working with enterprise and public sector clients, across Cloud, End-User Computing, Digital Workplace, Service Desk, and Network Infrastructure. What You’ll Do: Lead Service/ solution design from qualification to contract. Control bid teams — architects, pricing, delivery, and SMEs. Break down RFPs/RFIs into actionable, costed, client-ready solutions. Present internally and to clients at decision-maker level. Run solution workshops, own the architecture, and shape the financial model. You’ll Need: Experience working as a Service architect, Service Manager or Customer Success Manager R Gravitas to lead and drive teams through high-stakes bids. Deep knowledge of managed services delivery and commercial models. Strong technical grasp: Cloud, Security, EUC, Unified Comms, Service Desk, and more. Experience leading deals across onshore, offshore, and hybrid delivery models.
-
- Deal Architect- DACH region
- Germany
- Upto €110,000 plus bonus and benefits
-
Lead Deal Architect with the authority and experience to take control of complex, multi-million-euro outsourcing bids. This role is about leading the solutioning/ Service effort, bringing structure to chaos, and driving the entire bid team to deliver winning proposals. The company is a global managed services business providing solutions to enterprise and public sector clients, across Cloud, End-User Computing, Digital Workplace, Service Desk, and Network Infrastructure. What You’ll Do: Lead the deal from qualification to contract. Control bid teams — architects, pricing, delivery, and SMEs. Break down RFPs/RFIs into actionable, costed, client-ready solutions. Present internally and to clients at decision-maker level. Run solution workshops, own the architecture, and shape the financial model. Be responsible for the service Wrap and ensuring the Service meets clients requirements You’ll Need: A back ground with IT Services Experience in a similar type of role, for example: Deal, Service, or Solution Architect in ICT outsourcing. Gravitas to lead and drive teams through high-stakes bids. Deep knowledge of managed services delivery and commercial models. Strong technical knowledge: Cloud, Security, EUC, Unified Comms, Service Desk, and more. Experience leading deals across onshore, offshore, and hybrid delivery models.
-
- Pre Sales Lead- IT Services
- Germany
- Upto €100,000 plus benefits
-
As the Pre-Sales Lead (Sales Engineer/ Solution Architect) you will drive large-scale ICT managed services and outsourcing deals (from €0.5M to €20M+). You'll work directly with Business Development and clients to design high-impact solutions across Cloud (Azure, IaaS, SaaS, PaaS), EUC, Unified Comms, Security (SIEM, PAM), Networks, and Smart Workplaces. What You’ll Do: Lead the end-to-end pre-sales cycle — from RFI/RFP to contract. Design innovative, client-specific solutions with technical & commercial impact. Present at CxO level and steer proposal strategies & financial models. Collaborate closely with Portfolio, Service Desk, Field, and Digital Workplace teams. Support deal shaping with strong knowledge of ITIL, SIAM, Automation, and cost analysis. What You’ll Bring: Have strong experience in pre-sales or solution architecture. Experience with €M+ managed service deals. Deep technical expertise in modern ICT stack and enterprise IT services. Strong German (C1) and English communication skills. Certifications: ITIL v3/v4 required; SIAM, ISO20000 desirable.