Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Industry: Data Centre News
- Healthcare Business Development Manager
- Up to £60,000 Base + UNCAPPED Earnings
Healthcare Business Development Manager We are currently working with a multi-vendor IT solutions provider who are looking for a Business Development Manager who will be responsible for selling into the Healthcare Industry in a new business focussed position. The Healthcare Business Development Manager will have Current/Recent experience working for an IT managed services business/solutions provider. Experience delivering £150,000+ GP a year Current/Recent experience winning new healthcare accounts (all accounts won are kept) Flexible working is provided and also uncapped earnings. Apply for more information or call Peter Georgiou on 02086634030. Unfortunately, our client are unable to provide sponsorship so candidates must be UK based (commutable to London). Ref PG7577
- Cyber Incident Response specialist
- Up to £75,000 Base
Cyber Incident Response specialist is needed to join a global consultancy whose cyber business unit are continuing to their investment in the growth of their team. The Cyber Incident Response specialist role is client-facing that will join an award-winning team that deliver varied, interesting and often challenging work to a wide range of prestigious clients. The Cyber Senior Incident Response MUST have current experience taking a client through the complete IR / triage process and have a blend of both technical and commercial (identifying and developing new business opportunities within a client) Proactive Incident response, forensics and Ediscovery experience is a MUST. An individual must be London commutable and happy to travel, often internationally. Key attributes should also include; stakeholder engagement, mentoring of team members, a collaborative working style. Technical experience must include; demonstrable experience within an cyber incident response, Forensic, cyber etc. Additional certifications could / should include GIAC certified (Intrusion analyst, incident handler, forensic handler) Any of the following are very desirable also CREST Certified Network Intrusion Analyst (CCNIA) CREST Certified Host Intrusion Analyst (CCHIA) CREST Certified Malware Reverse Engineer (CCMRE) CREST Practitioner Intrusion Analyst (CPIA) Career development and the opportunity to influence, apply today for more information or call Chris Holt on 07884666351 or 02086634030 or email email@example.com Unfortunately, our client are unable to provide sponsorship for this opportunity. Candidates must be UK based. Ref: CH7578
- Sales Engineer (Telecoms, Ethernet, SDH, MPLS, IP)
- Up to €75,000 + Commission
Sales Engineer / Presales Consultant is needed for this Global Tier 1 carrier. You will be working with Enterprise customers helping to design solutions that solve your their business needs. You will be responsible for working alongside sales providing presales technical consultancy around my client's solutions base. You will be responsible for providing support for new business opportunities in terms of responding to RFIs & RFPs, understanding customer network requirements, high-level network architecture & design (including supplier selection on a global basis) and technical handover to network implementation teams. This is a great opportunity to join a global player who are growing their France based teams. You will require a successful track record in the telecommunications arena ideally from a global tier 1 ISP or network provider, with a demonstrable track record in designing complex enterprise solutions. A Sales Engineer needs to be technically astute and has had experience in the design, presentation, and implementation of Wide Area Networks (WAN). They need to understand a range of Layer 1, 2, and 3 technologies (Ethernet, SDH, MPLS, IP, etc) and build a solution based on the best technology to meet a customer’s requirements. In addition, they should have an understanding and experience in supplementary telecommunications services such as VoIP, Video Conferencing, Cisco and Riverbed hardware, and Security If you have any questions about this role, give us a call on 0044208 663 4030 or contact/send your CV to firstname.lastname@example.org Ref: RA7275
- Partner Manager (Network and UCC Services)
- Up to €100,000 plus €60,000 OTE, car allowance and benefits
We are looking for an experienced Partner Manager / Channel Manager to work for a global service provider to sell their Global SIP and Network Services to Enterprises and corporates through the partner market. This is a hunter role, and looking for an experienced salesperson who has knowledge of selling to and through the SI, Cisco and Microsoft Resellers. You will be responsible for Own and formulate a Sales and execution plan to get to quarterly targets Generate Leads from the partners Work with the Partner salesperson to help close deals. Acquire new partners and Deliver Quarterly numbers through the partners Work with Partner Marketing to create and run joint Partner plans/events to drive sales You will need to have experience in selling at least one of these services, global SIP, UCC and SD-WAN into and through the channel of, European Sis, Cisco or Microsoft resellers. Hunters who can identify key decision-makers in the targeted accounts Should be credible for partner to trust the person with leads and opportunities Drive to work through leads to closure along with the partner salesperson Ref: RA7277