Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Industry: Data Centre News
- Network Security Engineer
- €550 a day
German- based contract opportunity This is an onsite based position, we would need the Network Security engineer to be able to work on the client site 5 days a week Seeking an experienced Network Security Engineer for a leading technology company. Strong expertise in firewall/IPS solutions, proxy solutions, and certificate management is required. Good hands-on experience in networking and web-related technologies necessary. Strong problem-solving skills and the ability to work under pressure are essential. we are looking for a Network Security Engineer with the following experience: · Expertise in Administration, Management & Troubleshooting of Firewall / IPS solutions / Proxy solutions/Certificate Management Solutions · Good Hands-on Experience on security devices (PaloAlto/ /McAfee Proxy/CISCO ISE/Certificate Management) · Good Hands-on Experience in Networking with skills of switching, routing & wireless Technologies · Familiarity with web related technologies (Web applications, Web Services, Service Oriented Architectures) and of network/web related protocol · Configuration of NAT / PAT, firewall policies, profiling, objects, AD-Integration, backup – restore · Knowledge of Subnetting TCP/IP Communication, VLSM Configuration of VLAN VTP · Configuration of Routing Protocols e.g. RIPv1 & v2, OSPF, EIGRP, BGP Knowledge of standard and extended ACL 12 month contract
- IAM Consultant
- Upto £110,000 depending on level of position
Identity Access Management (IAM) Consultant Location: Germany We are seeking an experienced IAM consultants in Germany. we are looking for people from consultant through to Architect, The ideal candidate will have previous IAM deployment experience and be fluent in German. Key responsibilities: Design and implement IAM solutions for clients Provide expertise on industry best practices and standards Troubleshoot and resolve IAM-related issues Work closely with clients to understand their business requirements and provide solutions to meet those needs Qualifications: Previous deployment experience with IAM solutions Fluency in German Strong understanding of IAM technologies and principles Excellent communication and project management skills If you are an experienced IAM consultant with a strong track record of delivering successful projects, please apply today.
- ForgeRock Consultant
- Upto €85000 plus benefits
ForgeRock deployment consultant is needed for this expanding IT Services business within Spain, to act as their ForgeRock technical lead, Responsibilities include: High level and low level design, Scoping the techical needs of the project design, configure, develop and test the forgeRock deployment. We are looking for a strong IAM consultant ideally with ForgeRock experience, Must have strong Oauth 2.0, SAML and API experience
- IAM Consultant
- Upto €85000 plus benefits
An Identity & Access Management Consultant is needed for an expanding IT Security consultancy, based in France. (Remote role with monthly office meet-ups) The Identity & Access Management Consultant will be responsible for the technical design and implementation of Identity & Access Management/IAM products for a wide variety of clients. Deliver bespoke end-to-end consultancy service to our clients, from gathering requirements through to implementation. Work in a close team designing, developing, and implementing first-class IAM solutions. Manage client relationships, working closely with key stakeholders to continually evaluate business requirements and ensure the highest quality solution delivery. If you are interested we are looking for an individual with Previous experience working within the IAM or CIAM field is essential, Strong knowledge with SAML and Oauth and ideally OpenID Previous experience from any of these technologies: One Identity, SailPoint, Saviynt, Ubisecure, Ping Identity, would be advantageous