Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Downer
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Source: datacenterdynamics
Industry: Data Centre News
Latest Jobs
-
- Data Privacy Lead. Client Facing London. Permanent.
- London
- N/A
-
Data Privacy Lead. Client Facing London. Permanent. London based client facing. Must be eligible to undergo UK Security Clearance (SC) Key Responsibilities: Lead and support client facing data privacy projects. Assess compliance, define and deliver strategic projects / implement privacy solutions. Manage project teams and develop business opportunities. Required Experience: Experience in data protection and privacy standards. Background in consulting. Skills and Qualifications: Business consulting experience IAPPPrivacy Manager / Privacy Technologist Location Greater London UK based role. Not able to provide VISA sponsorship.
-
- VOIP / SIP App Developer. Contract. SIP | VOIP experience needed. SC Cleared Outside IR35 Contract. London
- London
- OUTSIDE IR35
-
SIP | VOIP Developer. SC Cleared Contract. London Looking for a SC Cleared SIP / VOIP Developer to develop an application that interacts with a set of voice and video signalling API’s You will also work on developing in-house applications, browser plugins and automated tooling to support secure communication systems. Responsibilities Develop an application that will manage number mapping and associated identities using commercial SBC API’s. Develop new user-facing features using React.js or other modern JavaScript frameworks. Build reusable components and front-end libraries for future use. Collaborate with the design team to translate UI/UX design wireframes into code. Work closely with backend developers to integrate front-end code with server-side logic. Conduct code reviews and provide constructive feedback to team members. Stay up-to-date on emerging technologies and industry trends to continuously improve our front-end development practices. Troubleshoot and debug issues that arise during development and in production environments. Maintain high coding standards and practices and ensure code is well-documented. Requirement Experience of developing specialist applications using REST API’s. Good knowledge of Go, Java and Python (open to alternative combinations of languages). Proficiency in front-end languages and frameworks such as HTML, CSS, JavaScript, React.js, etc. Strong understanding of web standards, responsive design, and cross-browser compatibility. Experience with version control systems such as Git. Knowledge of RESTful APIs and asynchronous request handling. Familiarity with UI/UX design principles and tools.
-
- Senior Data Privacy Consultant. Client Facing | London
- London
- N/A
-
Senior Data Privacy Consultant. Client Facing | London Senior Data Privacy Consultant needed for a key client facing opportunity. Must be willing to undergo SC Security Clearance. Hybrid role- onsite with customer / office 2-3 days a week. London Key Responsibilities: Lead and support client facing data privacy projects. Assess compliance, define and deliver strategic projects / implement privacy solutions. Manage project teams and develop business opportunities. Required Experience: Experience in data protection and privacy standards. Background in consulting. Skills and Qualifications: Business consulting experience IAPP Privacy Manager / Privacy Technologist Location Greater London UK based role. Not able to provide VISA sponsorship.
-
- Security Analyst - Internal role. London commutable. Permanent
- London
- N/A
-
Security Analyst - Internal role. London commutable opportunity. Operational Security - Investigate, escalate and proactively work to ensure household name remains protected. Project Security - Coordinate, log change requests with project delivery teams to meet security requirements Policy / compliance - work with team to aid in uplifting these as and where needed This role is role to investigate, escalate and proactively work to protect a globally recognised brand. You must have current hands on operational analytical security experience with Microsoft technology stack Someone with a SOC Analyst / security engineering background would be well suited. This position will join a small team and would suit someone that has broad experience across the security threat landscape. Experience / knowledge across industry GRC standards such NIST, ISO27001 etc very advantageous and a priority. You will work across multiple teams proactively working to secure the business. Must be able to commute to Central London 3 days a week. Visa sponsorship not available Apply today to find out more.