Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Downer
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Source: datacenterdynamics
Industry: Data Centre News
Latest Jobs
-
- Information Security Manager- Global Sporting Brand. UK. £100,000
- London
- £100.000
-
REFCH8265 Identifier Project Information Security Manager- Global Sporting Brand. UK. £100,000 A unique and exclusive opportunity to DCL Search to provide leadership and guidance Information and IT Security practices to one of the most recognised sporting brands in the world. You will be the envy of your colleagues, friends and peers as you take the lead in developing and implementing a security strategy. You must have a blend of knowledge across information security and technical security and be able to build internal and external stakeholder relationships. To coin a well known phrase, you should be a player manager. You don’t need to be currently hands on configuring firewalls, monitoring SIEM alerts, but maybe you have in the past. Ideally you will have come from a technical background as you will be closing be working with technical teams. Skills should include, but not be limited to: Managing / developing to Incident response plans. Information Security Risk Management / compliance. Security awareness Driving remediation plans to address vulnerabilities etc. Hybrid working. Up to £100,000 + benefits.
-
- Lead Information and Cyber Security Specialist, Financial Services. Exclusive to DCL Search
- London
- 85,000
-
Consultative approach with experience engaging with internal stakeholders providing advice and guidance across information security policies and standards into projects and programmes. Risk identification / Assessment / Management across people and process. ISO27001. Open mindedness to take on projects and programmes that will involve advising, scoping, refining, improving technical security control relating to best practice. Preferred experience; PCI DSS ISA or consultative experience within security Payment card industry. Information Security / technical security controls within Financial Services. Risk Assessment / management across technical controls. Technical Security background. Experience within secure by design and the technical security controls relating to projects / programmes. iSO27001 Lead Implementer / Auditor. CISA, CISM, CISSP. 2 days a fortnight in London- or more if you want.. Hybrid reworking.
-
- Cyber Security Associate, Financial Services. Exclusive to DCL Search
- London
- £40,000
-
Exclusive Cyber Security Associate needed within a forward thinking financial services business head quartered in London. DCL Search have been engaged on an Identifier Project to attract the very best cyber talent to this business. Influence the cyber security capability and direction within the business. Learn new skills working within a collaborative team. Grow as a security professional. ROLE Triaging and troubleshooting security alerts at a level 1 / level 2 capacity. Reviewing security change management requests. Managing and use of security tooling such as; Endpoint management Vulnerability management Patch management CASB Experience with the following tools is desirable. ZOHO Desktop Central (Endpoint Management) Splunk (SIEM) Qualys CASB (Microsoft) Microsoft Azure Varonis DatAdvantage ADAudit Plus Sonicwall, Paloalto, Dark Trace, Cloudflare, Cisco Umbrella, Microsoft defender.
-
- Senior Cyber Security Engineer, Financial Services. Exclusive to DCL Search
- London
- 70000
-
Exclusive Senior Cyber Security Engineer needed within a forward thinking financial services business head quartered in London. DCL Search have been engaged on an Identifier Project to attract the very best cyber talent to this business. Influence the cyber security capability and direction within the business. Learn new skills working within a collaborative team. Grow as a security professional. ROLE Day to day operations, management and scalability of existing cyber security systems Managing of and maturing security tooling such as; SIEM Endpoint Management Firewall Patch Management CASB Vulnerability management. Triaging and troubleshooting security alerts. Improve tooling, reducing false positives. Reviewing, approving, escalating security change management requests. Implementing new cyber security systems. Ideal technical experience · Vulnerability Management: Qualys · Endpoint Management: ZOHO Desktop Central · Forcepoint: CASB, DLP, webs security, email security. · SIEM (Splunk) · Firewalls: Sonicwall, Palo Alto · Endpoint Microsoft Defender · Appreciation of ISO27001, GDPR, PCI, etc 2 days a fortnight in London- or more if you want.. Hybrid reworking.