Google suffers Sunday outage, impacts Cloud, YouTube, Gmail and more
Google Cloud experienced widespread issues on Sunday, June 2, impacting the search giant's own services, as well as that of its cloud customers.
The intermittent outage, which has since been resolved, was blamed on "high levels of network congestion."
Networking not working
Google services like YouTube, Nest and Gmail, as well as Cloud customers like Snapchat, Shopify, Vimeo and Discord, were impacted by the problem, which began around 12:15 Pacific time.
"We are experiencing high levels of network congestion in the eastern USA, affecting multiple services in Google Cloud, G Suite and YouTube. Users may see slow performance or intermittent errors," the company said on its status page at the time.
While the congestion occurred in the US, its impact was felt globally and was described by network monitoring company ThousandEyes as a "large scale" outage
The problem was resolved as of 4:00 pm Pacific time, with Google promising to "conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits."
In a statement, the company apologized for the inconvenience and thanked customers for their "patience and continued support." It added: "Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better."
The outage which, among other things, meant that Nest users could not control their thermostats, comes after several major disruptions have impacted the largest cloud companies in recent years, highlighting the difficulty of building a resilient service, even with enormous resources.
Just last year saw Google Cloud go down due to a BGP error, Microsoft Azure be knocked out by a lightning strike, and Amazon Web Services be disrupted by a “power event.”
Update: Google's VP of 24x7, Benjamin Treynor Sloss, said in a blog post: "In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighbouring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows, much as urgent packages may be couriered by bicycle through even the worst traffic jam.
"Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage. The Google teams were keenly aware that every minute which passed represented another minute of user impact, and brought on additional help to parallelize restoration efforts."
- Head of Penetration Testing
- United Kingdom
Head of Penetration Testing needed to join a security consultancy that are delivering client facing penetration testing services around Web app and Infrastructure. Looking for someone hands on that is able to manage a highly skilled technical team of testers. 50-60% of the time is expected to be hands on, other duties will include, but not be limited to; leading and managing the day to day running of the team, mentoring, team upskill, recruitment, reporting, escalation, process improvement etc. Flexible location although south east is preferred. Anyone with Check / CREST experience is highly desirable. MUST be able to achieve SC clearance. UK based role. All details kept in confidence.
- CONTRACT SOC Manager. London / Birmingham. URGENT Immediate role.
REF7847 Contract SOC Manager. SC cleared, London / Birmingham. Initial 3 month Contract. SOC Manager needed to for an URGENT 3-4 month CONTRACT. SC clearance is essential. The project is to aid in the setup, implementation and management of resources to help with the initial stand up stages of a new SOC within a greenfield site. This is a short term contract role whilst a permanent hire is brought on over the coming 3 to 4 months. Experience engaging with and managing client stakeholder relationships as well as 3rd party relationships is critical. The role will involve; setting up, implementing and fine tuning the various initial stages of a SOC environment. Experience establishing and building out technical process / operational capability, managing of technical teams (analysts, engineers and architects, creation of policy / playbooks, fine turning is key. SPLUNK is the tooling of choice… Interviewing immediately. Set up a call with me today on https://calendly.com/chris-holt/arranged-call-with-chris-holt-remote-soc-role Direct contact details Chris.Holt@dclsearch.com or 07884666351
- SPLUNK Level 3 SOC Consultant, SIEM Splunk, London / Birmingham
REF CH7825 Level 3 SOC Consultant, SIEM Splunk, London / Birmingham £55,000 + Level 3 SOC Consultant, SIEM SPLUNK needed. Security Clearance. Permanent role Level 3 SOC Consultant, SIEM SPLUNK needed to join a public sector client. The ability to achieve SC clearance is essential. MUST have experience working with SPLUNK ideally to an Advanced Power User level. Splunk Enterprise Security (ES) knowledge and hands on experience highly desirable. The role will include, but not be limited to; managing and handling incidents end to end, supporting and mentoring level 1 / level 2 staff, supporting the SOC manager in the delivery of the SOC roadmap, engaging with the client stakeholders (other technical teams) as and where needed, use case development, advanced search and reporting etc. The individual MUST currently be living in the UK and be able to achieve UK security clearance. (SC) This is a permanent role To arrange a call with Chris Holt use this calendy link https://calendly.com/chris-holt/arranged-call-with-chris-holt-remote-soc-role Chris.Holt@dclsearch.com
- Aspiring Cyber Partner. Business lead, market maker.
Aspiring Cyber Partner (management consultancy) with Cyber specialism into Healthcare, Utilities and or Public Sector. Working with new and existing clients to help them solve, transform or evolve their cyber capabilities. MUST have; A proven management consultancy background in cyber. A history of identifying and closing new business opportunities. Currently Revenue generating / must be able to demonstrate recent wins. Client facing to board level with international businesses. Team leadership / mentoring experience. Extensive cyber industry experience. Digital transformation, Start-up environments etc. Experienced presenter at industry events, to be the public face of a business / capability. Breadth of knowledge across Cyber security. Service definition / creation. Would consider a senior director with experience delivering the above looking to step up. All conversations kept in confidence. To arrange a discreet call book a time to speak in my diary via https://calendly.com/chris-holt/cyber-partner-call Chris.Holt@dclsearch.com