Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Downer
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Source: datacenterdynamics
Industry: Data Centre News
Latest Jobs
-
- Data Protection Officer (DPO)/ Risk Manager
- London
- 65000 plus benefits
-
Data Protection Officer with a strong focus on risk management is needed for this expanding Paytech business. You will work closely with the CISO to help understand how the data is being processed within the business and identify the business risk associated with this. this role will be both internally and externally focused As the business processes payments across the globe, they require someone with knowledge of the different data regulations, GPPR, CCPA. This role is focused on helping to identify potential business risks associated with the data usage and ensuring that issues are flagged to the relevant teams within the business.
-
- Security Monitoring and Vulnerability Analyst, End User. PCI Level 1 Service Provider.
- Unknown
- 50000
-
CH7843 Security Monitoring and Vulnerability Analyst, End User. Level 1 Service Provider. £50,000 London Security Monitoring and Vulnerability Analyst needed to join an Level 1 Service Provider. The Security Analyst will be responsible monitoring, configuring, fine tuning and generally improving the security tool capability. Specific experience with Tripwire Log Center and Tripwire Enterprise is highly desirable. Other security tools experience should include the managing and monitoring of firewalls, Anti Phishing, AV, vulnerability management, IAM etc. Current experience with Vulnerability management and penetration testing is highly desirable. Specifically the ability to effectively manage 3rd party pen tests. Hands on penetration testing is a very nice to have. You will be working within a specialist security team reporting to the CISO. Experience working within an end user environment within financial services is highly desirable. Flexible location. This is an exclusive role to DCL Search & Selection. https://calendly.com/chris-holt/arranged-call-with-chris-holt-soc-role-clone
-
- Internal Security Auditor, Level 1 Service Provider (ISO27001)
- London
- Upto 65,000 plus benefits
-
Internal Security Auditor ISO 27001, PCI, needed to join a Cyber team within this expanding Fintech business. The Internal Security Auditor will have end to end responsibility for planning, delivering, remediating any findings etc. Experience working within financial services is highly desirable. This Is a great time to join a newly formed and growing Cyber team within a rapidly expanding fintech, that is taking a major share of its market. We are looking for someone with experience, (but not to be limited to) a mix of Information Security standards, frameworks, audit principles, controls / policies and the management and use of the technical tooling etc. ISO 22301, ISO 27001, NIST Cybersecurity Framework etc An ideal candidate will be working within an end user environment with a cyber consultancy background. Experience taking a company through accreditation is highly desirable Experience managing internal stakeholders, technical teams and external third parties essential Flexible working, but with the ability to get into London. This is an exclusive role to DCL Search & Selection.
-
- DevOps Engineer with IdAM
- London
- Upto £80,000 plus benefits
-
We are ooking for an DevOps engineer, idealy with IdAM (identity access Management) experience, this is a senior role for someone that can be the lead hands on person on a project. Your role will be to work on the deployment project implementing the solution into the exsiting application so will be used to connect an applications into mulipe 3rd party appliactions. We Would look at someone who has done DevOps with Security and can cross train into IdaM, but preference would be given to someone with the IdAM experience this is a great opportunity to join a consultancy that work on some truely amazing and differnet solutions