Facebook, Instagram, WhatsApp suffer global outage
Facebook has suffered one of the most sustained outages in the its history. The cause of the 14-hour problem remains unknown, and issues continue with Facebook, Instagram, WhatsApp and Messenger.
The trouble began around 9:00 PDT on 13 March and continues to affect some services today (14 March). The cause remains a mystery but appears to be with the application software. A tweet from NBC journalist Raj Mathai blaming "database overload" has no corroboration, and earlier reports pointing to a leak of BGP Internet routing data (similar to the cause of last year's Google Cloud outage) have been ruled out.
Reports from DownDetector suggested that the problem was global, and peaked between 9:00 and 15:00 PDT for all three services, before tailing off, though there are still residual reports of problems with all three applications.
Facebook announced two years ago that it was moving data from its WhatsApp acquisition from IBM's cloud to its own data centers. The fact that WhatsApp failed in step with the mother ship could be evidence that project was successfully completed.
"We're aware that some people are currently having trouble accessing the Facebook family of apps," Facebook said in a tweet. "We're working to resolve the issue as soon as possible." The company added that "the issue is not related to a DDoS attack."
Facebook has faced disruption before, but not on the same scale. In 2015, it went down twice in one week, but each outage was less than one hour, and the service only had 1.5 billion monthly users at the time, compared with today's 2.3 billion. The previous year, in 2014, a botched software update took it out for 2.5 hours, and in 2010, a database problem disabled it. It has been down for a longer period, but that was in 2008 when the site had less than 150 million users.
Respected network firm Netscout has scotched earlier reports of a BGP error. Last night, various outlets reported that Netscout had found evidence of a leak of BGP routing data, but Roland Dobbins, a Netscout principal engineer, told Ars Technica's Dan Goodin that this was an internal "miscommunication" which resulted in an erroneous email being sent to journalists:
Dan Goodin @dangoodin001
Replying to @dangoodin001
Roland Dobbins, principal engineer at Netscout's Assert team, says he has no data whatsoever to support that claim that a BGP leak is the cause of today's Facebook or Instagram outages. "There was an internal miscomm here," he says of the email PR people sent to reporters.
The outages have also affected Facebook's ad-buying system, several brand marketers have tweeted about the issue. Facebook said that it is investigating the overall impact of the outage "including the possibility of refunds for advertisers."
2019 sales estimates put Facebook's daily ad revenue at $250 million/£189m, so any downtime for ad sales will be costly.
Network monitoring company ThousandEyes said: “The cause would appear to be internal rather than a network or Internet delivery issue - for example we saw '500 internal server errors' from Facebook. Given the sheer scale and continuous changes that these web scale providers are constantly making to their applications and infrastructure, sometimes things break as a result of these changes, even in the most capable hands.
"When investigating Facebook’s issues today, we’re not seeing any BGP changes that are affecting connectivity, packet loss or latency. Since Facebook uses its own backbone network, it’s not clear/we don’t have insight as to how an external transit route issue would cause a disruption within the internal Facebook network.”
In an effort to minimize the chances of an outage, Facebook has a team working on 'Project Storm,' which stress-tests data centers with various tests and drills, including turning off a data center entirely.
Industry: Data Centre News
- Head of CSOC (Cyber Security Operations Centre), London. Greenfield opportunity
CH7928 Identifier Project Head of CSOC (Cyber Security Operations Centre), London. Greenfield UNIQUE opportunity to establish, manage and grow a world class CSOC. Headquartered in the UK, the CSOC will grow and deliver cyber services into an international and underdeveloped market. As a greenfield site the Head of CSOC will be the first person on the ground to build a capability and team around them. The role will include, but not be limited to; Setting up and establishing a CSOC monitoring service using IBM QRadar and Palo Alto CORTEX XSOAR. Develop roadmap for future CSOC capabilities. Establish, develop processes, playbooks and use cases. Recruit, Build and train a specialist team. Build relationships with key clients- onboard and scale. The successful candidate must have experience working within / leading a technical Cyber Security monitoring capability. Technical experience working with IBM QRadar and Palo Alto CORTEX XSOAR will be highly beneficial, although other SIEM / Security Orchestration, Automation and Response tools are welcome as the skillset is transferable. This brings significant opportunity and welcomes creativity / best practice. Frequent travel to London office will be required. Chris.firstname.lastname@example.org 07884666351
- Aruba Data Centre Network Engineer
- Upto £500 a day
**Outside of IR35 contract** Aruba Data Centre Network engineer is required to start ASAP. You will be responsible for Data Center and Campus Network - Design and implementation of Aruba CX Switches, We are looking for an experienced data centre engineer who is happy to work remotely and autonomously Experience you will need to have, Solid Networking experience with good knowledge of -Routing & Switching - MP-BGP EVPN VXLAN, MPLS/VPLS, EIGRP, BGP, OSPF and Multicast, Experience with Aruba switches ideally the CX range Ideally SC cleared or able to become SC Cleared. This will be a 3 month rolling contract
- Network Engineer-
- 65000 plus bonus
An exciting opportunity has arisen with a global financal business that are looking for an experienced Network engineer with strong Firewall experience in particular Checkpoint You will be responsible for help to support and manage the network infrastructure both LAN/ WAN And VPNs and will have a focus on network secuirty (Checkpoint firewall and Security controls) This is a multi-Site environment with few thousand end users, As the business is growing so is the network so great opportunity to join an expanding business that will help cross-train you in both cloud and onsite solutions. They are looking for a strong network engineer with a mix of expereience o Experience with Azure networking managing VNETs, ExpressRoute, VPNs and other Azure network features such as NSG/MS loadbalancer is required. Working with Check Point firewalls in Azure is desirable. o In-Depth understanding of networking fundamentals including: Network topology, Data Centre Infrastructure systems, Layer 2/Layer 3 Switching, VLANs, Routing, VPNs, High availability protocols. o Demonstrable hands-on experience with Meraki Switches and wireless infrastructure, Arista software, Forcepoint Proxy (would be a strongly desirable), Citrix Netscaler, and Forescout CounterACT Location can be flexible, they are looking for someone who would be happy to do a hybrid working model 3 days at home 2 days in the office.
- Technical Cyber Trainer, United Kingdom
Technical Cyber Trainer, United Kingdom Cyber trainer needed to build a new cyber training capability to develop a team of international security analysts. Technical Cyber training experience is a MUST. Any experience delivering technical Blue team based training is highly desired. This is a UK based role. Able to travel to London on occasion- when required. Agile, flexible working. A technical background is highly desirable and prefered. The role Aid in the setup, fine tuning and on-going customisation of Technical Cyber Security’s training platform. CyberBit experience desirable. Design, develop and deliver the internal syllabus to train employees with low cybersecurity knowledge to become analysts. Manage, monitor, support and assess learners. Fine tuning training and bespoke development. Manage, monitor and assess SOC analysts. Support employees through their career. The ability to deliver remote training is essential. This is a UK based role. Able to travel to London on occasion- when required. Agile working. Apply today to find out more information. Chris.Holt@dclsearch.com +447884666351