Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Downer
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Source: datacenterdynamics
Industry: Data Centre News
Latest Jobs
-
- Contact 12 month- Security Operations- Crowdstrike Falcon Insight EDR / Analyst.
- United Kingdom
- Dependent on experience
-
Security Operations engineer / Analyst with Crowdstrike Falcon Insight EDR experience for a 12 month contract. Experienced Contractor with Crowdstrike Falcon Insight: Endpoint detection and Response (EDR) experience needed - 12 month rolling project. Implementation, configuration and Analyst experience needed with Crowdstrike Falcon Insight: (EDR) Migration project- relocating capability internationally. technically implementing, configuration of that that migration and then transition to BAU role monitoring. DCL Search exclusive associate Project.
-
- SailPoint Consultant
- London
- Upto £75,000 plus benefits
-
SailPoint Consultant is needed for an expanding Financial Service business, this is an exciting time to join the Business as they are in the Process of deploying both IAM and PAM solutions and this consultant will form a key part of the IAM team Location can be flexible but would require the individual to come into the London office a couple of times a month for team meetings and face to face project reviews Duties include · Engage in the Identity & Access Management project to deliver SailPoint IdentityNow and Privileged Access Management · On-board applications and users into IAM tools and customise or configure integrations as required · Regularly review, secure and recertify privileged roles in applications, databases and operating systems · Implement least privilege, just-in-time access, password rotation and vaulting wherever possible · Migrate application authentication to Single Sign-On through the use of SAML and OAuth · Implement and enforce the use of MFA where possible, focusing on critical applications and risky sign-ins · Provide technical support to Centrify and SailPoint users Key experience required: Previous experience with SailPoint, including integrating and deploying into a business, onboarding users and applications, supporting users and performing manual administration tasks. Experience with SAML and OAuth to migrate applications to Single Sign-on. If you are interested in hearing more please reach out to me for more information
-
- Centrify Consultant
- London
- Upto £75,000 plus benefits
-
A Privileged Access Management Consultant is needed for an expanding Financial Service business, this is an exciting time to join the Business as they are in the Process of deploying a Centrify PAM solution,, this consultant will form a key part of the team Location can be flexible but would require the individual to come into the London office a couple of times a month for team meetings and face to face project reviews Duties include · On-board applications and users into PAM tools and customise or configure integrations as required · Regularly review, secure and recertify privileged roles in applications, databases and operating systems · Implement least privilege, just-in-time access, password rotation and vaulting wherever possible · Migrate application authentication to Single Sign-On through the use of SAML and OAuth · Implement and enforce the use of MFA where possible, focusing on critical applications and risky sign-ins · Provide technical support to Centrify users You would also gain expsoure with the IAM toolset as part of an Identity Access deployment. Key experience required: Previous experience with a PAM tool (Centrify would be an added bonus but not essential) including integrating and deploying into a business, onboarding users and applications, supporting users and performing manual administration tasks. Experience with SAML and OAuth to migrate applications to Single Sign-on. If you are interested in hearing more please reach out to me for more information
-
- SOC team lead- Deputy SOC manager - Managed Security Services, Bradford. Exclusive
- Bradford
- £70,000 +
-
SOC team lead- Deputy SOC Manager - Managed Cyber Security Services, Bradford. Exclusive Identifier project. Technical team lead needed to join a Managed Cyber Security Services business. The role will be a hands on lead role and technical escalation point for the team. You will also be responsible for leading, mentoring, growing and developing the team. You will be the deputy SOC manager and be involved in the strategic growth of the capability. A managed security services background is essential, specifically within a managed security operations capability. Current hands on support experience across Firewall, SIEM, Incident Response is essential.