Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Downer
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Source: datacenterdynamics
Industry: Data Centre News
Latest Jobs
-
- IAM Product/ Project Manager
- Belgium
- Upto €80,000 plus benefits
-
IAM product/project manager is need for this expanding service provider to help develop their IAM (CIAM) strategy roadmap. This role will have two main functions, to work between the front end digital team and the backend IAM development team to ensure that the current deployment of the solution runs smoothly and is fit for purpose, the Second function is to look at the business’ future digital offerings and to understand how the IAM/CIAM solution will develop with the new digital strategy, you will be looking at the future technology and the ensuring the IAM solution is fit for purpose. The business is an agile environment and you will require agile experience. This is a great opportunity to help shape a key product within the future digital strategy of this expanding service provider. We are looking for someone with both IAM and strong product management experience Project management experience would be beneficial. If you are interested speak to Robert Anderton on 0044 (0) 7957 493501 and he will be able to discuss the role in more details IAM product/project manager is needed for this expanding service provider to help develop their IAM (CIAM) strategy roadmap. This role will have two main functions, to work between the front end digital team and the backend IAM development team to ensure that the current deployment of the solution runs smoothly and is fit for purpose, the second function is to look at the business’ future digital offerings and to understand how the IAM/CIAM solution will develop with the new digital strategy, you will be looking at the future technology and the ensuring the IAM solution is fit for purpose. The business is an agile environment and you will require agile experience. This is a great opportunity to help shape a key product within the future digital strategy of this expanding service provider. We are looking for someone with both IAM and strong product management experience Project management experience would be beneficial. If you are interested then speak to Robert Anderton on 0044 (0) 7957 493501 and he will be able to discuss the role in more details
-
- Cyber Incident Manager, Proactive planning and management. SC
- London
- N/A
-
Cyber Incident Manager, Proactive planning and management. Cyber incident Manager needed to join a large and complex business to help them prepare for a cyber related incidents. SC clearance will be required. Current or the ability to achieve. This role does not require specific current hands on technical Incident response experience, but this background would give a distinct advantage. The role has two key functions: to help the business prepare for an event and to steer them through when / if that happens. The Cyber Incident response managers role will include, but not be limited to; working with internal stakeholders to develop a security incident management plan along with its and supporting policies. Developing plans and implementing strategies on how incidents are detected, reported, assessed and responded to. Engaging with leadership teams both internal and external, proactively mapping out this large business to identify and engage the various other stakeholders and their teams. Build out and document incident scenarios and their processes, ensure incident management procedures are updated, playbooks and key training etc. You should have experience working with both internal teams and external suppliers. The role will also focus on liaising with the various teams to ensure the security incident response plan is delivered effectively. CCIM, GCIH, CIPR (NCSC-Certified Cyber Incident Planning & Response) Looking to interview immediately.
-
- SOC Manager. SC Clearance. Immediate opportunity.
- Unknown
- N/A
-
Permanent SOC Manager. SC cleared / clearable, London / Birmingham. SOC Manager needed to replace a SOC contractor I placed into a client who is due to complete their assignment at the end of March. The ability to achieve SC clearance is essential. Looking for someone that is a blend of strategic stakeholder engagement with strong technical skills. The role will sit in a relatively new SOC environment. The position is to setup, implementation and management of resources to help with the initial and on-going stages of a new SOC. Experience engaging with and managing client stakeholder relationships as well as 3rd party relationships is critical. The role will involve; setting up, implementing and fine tuning the various initial stages of a SOC environment. Experience establishing and building out technical process / operational capability, managing of technical teams (analysts, engineers and architects, creation of policy / playbooks, fine turning is key. SPLUNK is the tooling of choice… Interviewing immediately. Set up a call with me today on https://calendly.com/chris-holt/arranged-call-with-chris-holt-soc-manager-role Direct contact details Chris.Holt@dclsearch.com or 07884666351
-
- Security engineer. Financial Services. UK. Permanent
- Unknown
- N/A
-
CH7863 Security engineer. End User . Financial Services Security Engineer needed to monitor and manage a security suite of tools within an End User environment. The Security Engiener will be responsible monitoring, configuring, fine tuning, incident management and generally improving the security tool capability. Specific experience with CyberArk, Tripwire Log Center and Tripwire Enterprise is highly desirable). Current experience with Vulnerability management and penetration testing is highly desirable. Specifically the ability to effectively manage 3rd party pen tests. You will be working within a specialist security team reporting to the CISO. Experience working within an end user environment within financial services is highly desirable. Flexible location. This is an exclusive role to DCL Search & Selection. To book a call please use my Calendy link https://calendly.com/chris-holt/arranged-call-with-chris-holt-soc-role-