How four rotten packets broke CenturyLink's network for 37 hours, knackering 911 calls, VoIP, broadband
.jpg)
A handful of bad network packets triggered a massive chain reaction that crippled the entire network of US telco CenturyLink for roughly a day and a half.
This is according to the FCC's official probe into the December 2018 super-outage, during which CenturyLink's broadband internet and VoIP services fell over and stayed down for a total of 37 hours. This meant subscribers couldn't, among other things, call 911 over VoIP at the time – which is a violation of FCC rules, and triggered a formal investigation.
"This outage was caused by an equipment failure catastrophically exacerbated by a network configuration error," America's communications regulator said in its summary of its inquiry, published yesterday.
"It affected communications service providers, business customers, and consumers who directly or indirectly relied upon CenturyLink’s transport services, which route communications traffic from various providers to locations across the country, resulting in extensive disruptions to phone service, including 911 calling."
CenturyLink has six long-haul networks that make up the backbone of its digital empire, interconnecting regions of America. These networks use Infinera-built nodes to switch packets over high-speed optic fiber: data flowing into each node is directed to other nodes, ultimately pumping VoIP, regular internet traffic, and more, across the nation as needed.
Each dodgy packet would arrive at a node, get rejected and be passed along a chain of filters until it was injected into a management channel and handed to all connecting nodes. Here's a flow diagram, courtesy of the FCC, showing how the corrupted packets ended up being forwarded on to all neighbouring nodes, and so on and so on, producing a growing chain reaction of corrupted packets.
"Due to the packets’ broadcast destination address, the malformed network management packets were delivered to all connected nodes. Consequently, each subsequent node receiving the packet retransmitted the packet to all its connected nodes, including the node where the malformed packets originated," the FCC said in its report.
"Each connected node continued to retransmit the malformed packets across the proprietary management channel to each node with which it connected because the packets appeared valid and did not have an expiration time. This process repeated indefinitely."
As you might imagine, the exponentially growing storm of packets soon overwhelmed CenturyLink's optic-fibre backbone, causing regular traffic to stop flowing: VoIP phones stopped working, internet access slowed to a halt, and so on. Folks in New Orleans were first to spot their connections stalling, at roughly 0356 EST on December 27.
Here is where things went from really, really bad to terrible: the nodes along the fiber network were so flooded, they could not be reached by their administrators to troubleshoot the issue. It wasn't until some 15 hours later the techies could finally track down the single errant node in Colorado responsible for sparking the deluge, not that replacing it helped. The packet tsunami was still washing back and forth, knocking nodes over.
"At 2102 on December 27, CenturyLink network engineers identified and removed the module that had generated the malformed packets," the report noted. "The outage, however, did not immediately end; the malformed packets continued to replicate and transit the network, generating more packets as they echoed from node to node."
It would be another three hours before CenturyLink's network admins could begin to get through to the other nodes and get them to kill off the spread of bad packets. It took until 1130 on December 28 to get visibility of the network back, and it wasn't until 2336 that all nodes had been restored. On December 29, just after midday, CenturyLink finally declared the crisis over.
"The event caused a nationwide voice, IP, and transport outage on CenturyLink’s fiber network. CenturyLink estimates that 12,100,108 calls were blocked or degraded due to the incident," the FCC said.
"Where long-distance voice callers experienced call quality issues, some customers received a fast-busy signal, some received an error message, and some just had a terrible connection with garbled words."
The outage also knackered local governments and telcos that relied on the CenturyLink network for portions of their services. State governments in Illinois, Kansas, Minnesota, and Missouri all had portions of their networks down for roughly 36 hours thanks to CenturyLink, and phone services sold by Comcast, Verizon, TeleCommunication Systems, General Dynamics IT, and West Safety Services – including 911 call centers – saw connectivity interrupted for some or all of the outage period.
As to what can be done to prevent similar failures, the FCC is recommending CenturyLink and other backbone providers take some basic steps, such as disabling unused features on network equipment, installing and maintaining alarms that warn admins when memory or processor use is reaching its peak, and having backup procedures in the event networking gear becomes unreachable.
"Currently, CenturyLink is in the process of updating its nodes’ Ethernet policer to reduce the chance of the transmission of a malformed packet in the future," the report notes. "The improved ethernet policer quickly identifies and terminates invalid packets, preventing propagation into the network. This work is expected to be completed in Fall, 2019."
source theregister
Industry: Unified Communications & Telecommunications

Latest Jobs
-
- Account Manager - IT Services
- Germany
- €90000 plus OTE and Car
-
Are you a deal closer with a hunter mindset? Do you know how to uncover business pain points, and turn them into long-term digital transformation partnerships? Our Client are growing their sales force across Germany and looking for an ambitious, straight-talking Account Manager to take the lead on new client acquisition. You’ll focus on mid-sized to large enterprises across Germany helping to shape their digital future with tailored IT solutions in Workplace, Cloud, and Security. • Drive Growth: Own the full sales cycle for new business across your region. • Solution Sell: Build bespoke offers in Security, Digital Workplace and Cloud solutions • Build Relationships: Establish a solid pipeline through smart prospecting, marketing-driven leads, and your own network. • Represent a brand known for trust, delivery, and tech excellence—with 4,000 employees globally and a growing team within Germany. What You Bring • Proven new logo sales experience in the IT services space (not hardware!) • Deep knowledge in one or more of: Cybersecurity, Digital Workplace, or Cloud • Confidence to lead enterprise deals and pitch directly to senior stakeholders • Fluent German and good English skills Sind Sie ein Abschlussprofi mit Hunter-Mentalität? Wissen Sie, wie man geschäftliche Pain Points identifiziert und in langfristige Partnerschaften zur digitalen Transformation verwandelt? Unser Kunde baut derzeit sein Vertriebsteam in ganz Deutschland aus und sucht eine ambitionierte, ehrliche Persönlichkeit als Account Manager, die den Lead bei der Neukundengewinnung übernimmt. Ihr Fokus liegt auf mittelständischen bis großen Unternehmen in Deutschland, denen Sie mit maßgeschneiderten IT-Lösungen in den Bereichen Workplace, Cloud und Security den Weg in die digitale Zukunft ebnen. Ihre Aufgaben • Wachstum vorantreiben: Verantwortung für den gesamten Vertriebszyklus im Neugeschäft Ihrer Region. • Lösungsorientierter Vertrieb: Entwicklung individueller Angebote in den Bereichen Security, Digital Workplace und Cloud-Lösungen. • Beziehungen aufbauen: Aufbau einer stabilen Pipeline durch gezielte Ansprache, marketinggenerierte Leads und Ihr eigenes Netzwerk. • Marke repräsentieren: Werden Sie Teil eines Unternehmens mit 4.000 Mitarbeitenden weltweit und einem stark wachsenden Team in Deutschland – bekannt für Vertrauen, Verlässlichkeit und technologische Exzellenz. Was Sie mitbringen • Nachgewiesene Erfahrung in der Neukundenakquise im Bereich IT-Services (kein Hardwarevertrieb!) • Fundiertes Wissen in mindestens einem der Bereiche: Cybersecurity, Digital Workplace oder Cloud • Selbstbewusstes Auftreten im Umgang mit Enterprise-Deals und Entscheidungsträgern auf Top-Level • Verhandlungssichere Deutschkenntnisse und gute Englischkenntnisse
-
- Senior SOC Analyst Level 3. Microsoft Security stack | Ability to achieve SC Clearance
- London
- To attract the right person
-
Job Title: Senior SOC Analyst Level 3. Microsoft Security stack | Ability to achieve SC Clearance Location: Hybrid remote | London / Berkshire Overview: Senior SOC Analyst Level 3 to join a specialist Managed Security Services business. You will be responsible for advanced threat hunting / triage, incident response etc with a strong focus on the Microsoft Security Stack. Key Responsibilities: Lead and resolve complex security incidents / escalations Conduct advanced threat hunting using the Microsoft Security Stack. Build, optimise and maintain workbooks, rules, analytics etc. Correlate data across Microsoft 365 Defender, Azure Defender and Sentinel. Perform root cause analysis and post-incident reporting. Aid in mentoring and upskilling Level 1 and 2 SOC analysts. Required Skills & Experience: The ability to achieve UK Security Clearance (SC) – existing clearance ideal. (Sorry no visa applications) Current experience working with a SOC environment Microsoft Sentinel: Development and tuning of custom analytic rules. Workbook creation and dashboarding. Automation using Playbooks and SOAR integration. Kusto Query Language (KQL): Writing complex, efficient queries for advanced threat hunting and detection. Correlating data across key tables (e.g., SignInLogs, SecurityEvent, OfficeActivity, DeviceEvents). Developing custom detection rules, optimising performance, and reducing false positives. Supporting Sentinel Workbooks, Alerts, and Playbooks through advanced KQL use. Deep understanding of incident response, threat intelligence and adversary techniques (MITRE ATT&CK framework). Strong knowledge of cloud and hybrid security, particularly within Azure. Additional Requirements: Must hold or be eligible to achieve a minimum of Security Clearance (SC) level. Nice to have certifications (e.g., SC-200, AZ-500, GIAC) are desirable. Strong problem-solving and analytical skills. Excellent communication for clear documentation and team collaboration. Please follow Wheaton’s Law.
-
- New Business Sales Hunter | Cyber Security (UK Based)
- London
- To attract the right person
-
New Business Sales Hunter needed | Cybersecurity (UK Based) Are you looking for uncapped commission, a fun and sociable team that drives success with no politics? If so...You must Be UK based - and able to achieve UK SC clearance. (sorry no visas) Have a demonstrable history of sales success in Cyber Security Follow Weatons law. The role: Seeking a proven New Business Sales Hunter to join an established, successful and expanding cyber security firm. New business focused - £1m GP year one target (ramped). Sell a blend of security services & professional services. Ideal experience selling some or all of the following Cyber strategy & risk management Managed detection & response (MDR) Penetration testing Compliance & audit support You: Strong cybersecurity/IT services sales track record. Confident selling into mid-market & enterprise. UK based - London commutable 1x per week. Hunter mindset, full sales cycle ownership. Don't just send an email to apply give me a call on 07884666351
-
- CyberArk Architect
- London
- Upto £110,000 plus bonus and benefits
-
Are you ready to lead from the front and drive innovation in the Identity & Access Management (IAM) space? We’re looking for a seasoned CyberArk Architect who has CDE-CPC ideally or experience with privilege Cloud, someone who can lead with vision, execute with precision, and inspire teams to deliver excellence. As a key leader in our organisation, you’ll bring your strong business acumen and a technology-focused, innovative mindset to the table. You’ll be driving strategic initiatives, shaping transformation programs, and empowering teams to think big and deliver even bigger. Acting as a subject matter expert in CyberArk Leading strategic transformations in: Identity Governance Privileged Access Management (PAM) Access Management Customer Identity and Access Management (CIAM) Building and maintaining strong, collaborative relationships within the team Communicating clearly and confidently — both written and verbal — to deliver updates, raise potential issues, and share insights If you are interested in the above position we are looking for people with: deep expertise and a successful track record in IAM strategy, delivery, or assurance with CyberArk Hold relevant certifications such as CDE in Privileged Cloud or Guardian Have experience in a client-facing role (preferred, but not essential) Thrive in a hybrid working environment and are available to work from our or client London office three days a week Lead with clarity, communicate with impact, and adapt quickly to changing priorities