Google suffers Sunday outage, impacts Cloud, YouTube, Gmail and more
Google Cloud experienced widespread issues on Sunday, June 2, impacting the search giant's own services, as well as that of its cloud customers.
The intermittent outage, which has since been resolved, was blamed on "high levels of network congestion."
Networking not working
Google services like YouTube, Nest and Gmail, as well as Cloud customers like Snapchat, Shopify, Vimeo and Discord, were impacted by the problem, which began around 12:15 Pacific time.
"We are experiencing high levels of network congestion in the eastern USA, affecting multiple services in Google Cloud, G Suite and YouTube. Users may see slow performance or intermittent errors," the company said on its status page at the time.
While the congestion occurred in the US, its impact was felt globally and was described by network monitoring company ThousandEyes as a "large scale" outage
The problem was resolved as of 4:00 pm Pacific time, with Google promising to "conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits."
In a statement, the company apologized for the inconvenience and thanked customers for their "patience and continued support." It added: "Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better."
The outage which, among other things, meant that Nest users could not control their thermostats, comes after several major disruptions have impacted the largest cloud companies in recent years, highlighting the difficulty of building a resilient service, even with enormous resources.
Just last year saw Google Cloud go down due to a BGP error, Microsoft Azure be knocked out by a lightning strike, and Amazon Web Services be disrupted by a “power event.”
Update: Google's VP of 24x7, Benjamin Treynor Sloss, said in a blog post: "In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighbouring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows, much as urgent packages may be couriered by bicycle through even the worst traffic jam.
"Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage. The Google teams were keenly aware that every minute which passed represented another minute of user impact, and brought on additional help to parallelize restoration efforts."
- ISO27001 Information Security Consultant
- Up to £60,000
Information Security Consultant with ISO27001 audit and advisory experience is needed for a client facing opportunity with a Cyber Security company in London. Experience with ISO27001 is essential. Activities of the role will include, but not be limited to providing advice to clients, Gap analysis, Risk assessment, analysis, ISO27001 Audits. Experience taking a client through to iso 27001 certification is highly desirable. This Cybersecurity consultancy, who are dedicated to improving and investing in their client's businesses and employees careers, are looking for a security consultant due to expansion. All the training and development will be provided to help them specialise into the PCI industry / Security advisory industry. Ideal certifications ISO27001 Lead Auditor, ISO 27001 Lead implementer, PCI ISA. Aspiring PCI QSA. Other certifications such as CISSP, CISM or CISA are beneficial to have but not required. The ability to SC Clearance is essential. MUST be UK based and realistically able to commute to London. Structured career path, technical training, diverse and interesting clients available. (ISO70001 Lead Auditor, ISO 27001 Lead implementer, PCI ISA. Aspiring PCI QSA, ISO27001 Information Security Consultant) Contact me on email@example.com or 07884666351 or 02086634030 Ref: CH7514
- Google Cloud Data Engineer
- Up to £650 Per Day
Google Cloud Data Engineer London Up to £650 Per Day Duration: 3 months (Potential to extend) We are currently working with a leading Google Cloud partner who are currently looking for a Google Cloud Data Engineer in London. The Google Cloud Data Engineer will be responsible for a new, on-site project (start to finish) designing and implementing a data cataloguing platform using Google Cloud. Current Experience Required Google Cloud Data Analytics (Data Engineering, Data Mining, Data Cataloguing etc.) Cloud PUB / SUB Ref: PG7512
- Professional Services Security Engineer
- United Kingdom
Professional Services Security Engineer with current checkpoint experience is needed for the UK focused client facing implementation/migration, configuration position. The role will be utilising the latest versions of Checkpoint, so someone accredited with either CCSA or CCSE, on at least version R80 is ideal. The Professional Services Security Engineer must have current technical implementation experience using Checkpoint, however, I would look at someone with strong firewalling experience around other vendors such as Palo Alto and Fortinet. Being a multi-vendor professional services business, there is scope for this person to receive training and experience within other vendors. This is a UK wide role, the company in question has 2 offices across the UK, however, there is scope for this person to be home based when not on client site. Vendor training and exposure actively promoted.
- eDiscovery / Forensic Consultant, London, £65,000
Senior eDiscovery / forensic consultant needed to join a business is recognised for helping top tier clients across eDiscovery, Forensics, Incident Response, Advisory etc. Known the for quality, consistency of work throughout the world. This individual MUST be London based, client facing with deep technical hands on experience with eDiscovery / forensic tools, techniques and best practice. Hands on experience using Relativity is essential. The position is split between engaging with client stakeholders to provide consultancy, technical engaging to identify, preserve, collect, process, review and produce electronically stored information in litigation and manage / provide support for the other internal business functions. This will include, but not be limited to; manging client engagements, collecting / processing data within Relativity, delivering / providing guidance customisation on reports, advising clients. Any of the following certifications are highly desirable. • Relativity Certified Administrator (RCA) • Relativity Processing Specialist • Relativity Analytics Specialist Travel to client site will be involved. Fluency in multiple European languages is highly desirable. All details kept in the strictest of confidence. Contact me on Chris.firstname.lastname@example.org 07884666351 or 02086634030