Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Industry: Data Centre News
- ISO27001 Information Security Consultant
- Up to £60,000
Information Security Consultant with ISO27001 audit and advisory experience is needed for a client facing opportunity with a Cyber Security company in London. Experience with ISO27001 is essential. Activities of the role will include, but not be limited to providing advice to clients, Gap analysis, Risk assessment, analysis, ISO27001 Audits. Experience taking a client through to iso 27001 certification is highly desirable. This Cybersecurity consultancy, who are dedicated to improving and investing in their client's businesses and employees careers, are looking for a security consultant due to expansion. All the training and development will be provided to help them specialise into the PCI industry / Security advisory industry. Ideal certifications ISO27001 Lead Auditor, ISO 27001 Lead implementer, PCI ISA. Aspiring PCI QSA. Other certifications such as CISSP, CISM or CISA are beneficial to have but not required. The ability to SC Clearance is essential. MUST be UK based and realistically able to commute to London. Structured career path, technical training, diverse and interesting clients available. (ISO70001 Lead Auditor, ISO 27001 Lead implementer, PCI ISA. Aspiring PCI QSA, ISO27001 Information Security Consultant) Contact me on email@example.com or 07884666351 or 02086634030 Ref: CH7514
- Google Cloud Data Engineer
- Up to £650 Per Day
Google Cloud Data Engineer London Up to £650 Per Day Duration: 3 months (Potential to extend) We are currently working with a leading Google Cloud partner who are currently looking for a Google Cloud Data Engineer in London. The Google Cloud Data Engineer will be responsible for a new, on-site project (start to finish) designing and implementing a data cataloguing platform using Google Cloud. Current Experience Required Google Cloud Data Analytics (Data Engineering, Data Mining, Data Cataloguing etc.) Cloud PUB / SUB Ref: PG7512
- Professional Services Security Engineer
- United Kingdom
Professional Services Security Engineer with current checkpoint experience is needed for the UK focused client facing implementation/migration, configuration position. The role will be utilising the latest versions of Checkpoint, so someone accredited with either CCSA or CCSE, on at least version R80 is ideal. The Professional Services Security Engineer must have current technical implementation experience using Checkpoint, however, I would look at someone with strong firewalling experience around other vendors such as Palo Alto and Fortinet. Being a multi-vendor professional services business, there is scope for this person to receive training and experience within other vendors. This is a UK wide role, the company in question has 2 offices across the UK, however, there is scope for this person to be home based when not on client site. Vendor training and exposure actively promoted.
- eDiscovery / Forensic Consultant, London, £65,000
Senior eDiscovery / forensic consultant needed to join a business is recognised for helping top tier clients across eDiscovery, Forensics, Incident Response, Advisory etc. Known the for quality, consistency of work throughout the world. This individual MUST be London based, client facing with deep technical hands on experience with eDiscovery / forensic tools, techniques and best practice. Hands on experience using Relativity is essential. The position is split between engaging with client stakeholders to provide consultancy, technical engaging to identify, preserve, collect, process, review and produce electronically stored information in litigation and manage / provide support for the other internal business functions. This will include, but not be limited to; manging client engagements, collecting / processing data within Relativity, delivering / providing guidance customisation on reports, advising clients. Any of the following certifications are highly desirable. • Relativity Certified Administrator (RCA) • Relativity Processing Specialist • Relativity Analytics Specialist Travel to client site will be involved. Fluency in multiple European languages is highly desirable. All details kept in the strictest of confidence. Contact me on Chris.firstname.lastname@example.org 07884666351 or 02086634030