Google suffers Sunday outage, impacts Cloud, YouTube, Gmail and more
![Data Centre](/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBeFdFRGc9PSIsImV4cCI6bnVsbCwicHVyIjoiYmxvYl9pZCJ9fQ==--27ee9b3db655b53167a2bb85e410c22fe9fa3ea2/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaDdCem9MWm05eWJXRjBTU0lJYW5CbkJqb0dSVlE2QzNKbGMybDZaVWtpRFRjMU1IZzBOVEJlQmpzR1ZBPT0iLCJleHAiOm51bGwsInB1ciI6InZhcmlhdGlvbiJ9fQ==--9a8cb233bbd899661209fac1218cb930366c2398/berlin_busy_street_capital_2253938.jpg)
Google Cloud experienced widespread issues on Sunday, June 2, impacting the search giant's own services, as well as that of its cloud customers.
The intermittent outage, which has since been resolved, was blamed on "high levels of network congestion."
Networking not working
Google services like YouTube, Nest and Gmail, as well as Cloud customers like Snapchat, Shopify, Vimeo and Discord, were impacted by the problem, which began around 12:15 Pacific time.
"We are experiencing high levels of network congestion in the eastern USA, affecting multiple services in Google Cloud, G Suite and YouTube. Users may see slow performance or intermittent errors," the company said on its status page at the time.
While the congestion occurred in the US, its impact was felt globally and was described by network monitoring company ThousandEyes as a "large scale" outage
The problem was resolved as of 4:00 pm Pacific time, with Google promising to "conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits."
In a statement, the company apologized for the inconvenience and thanked customers for their "patience and continued support." It added: "Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better."
The outage which, among other things, meant that Nest users could not control their thermostats, comes after several major disruptions have impacted the largest cloud companies in recent years, highlighting the difficulty of building a resilient service, even with enormous resources.
Just last year saw Google Cloud go down due to a BGP error, Microsoft Azure be knocked out by a lightning strike, and Amazon Web Services be disrupted by a “power event.”
Update: Google's VP of 24x7, Benjamin Treynor Sloss, said in a blog post: "In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighbouring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows, much as urgent packages may be couriered by bicycle through even the worst traffic jam.
"Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage. The Google teams were keenly aware that every minute which passed represented another minute of user impact, and brought on additional help to parallelize restoration efforts."
![Banner Default Image](https://www.dclsearch.com/rails/active_storage/blobs/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBdytMRGc9PSIsImV4cCI6bnVsbCwicHVyIjoiYmxvYl9pZCJ9fQ==--683221fba4088f48e5f9c99e2719b73064c09cee/banner-default.jpg)
Latest Jobs
-
- Network & Security Consultant
- Spain
- Upto €54000 per year and benefits
-
Senior Network & Security Engineer to join a Managed Network & Security Team in Europe. In this critical role, you will: Play a pivotal role in managing and securing network infrastructure across datacenters, customer connections, and on-premise deployments. Proactively monitor network and security devices, analyse incidents, and implement solutions to ensure optimal performance and security. Collaborate with colleagues and customers to troubleshoot issues, troubleshoot outages, and implement effective resolutions. Lead and participate in network system installations for new facilities and expansions. Develop and maintain network infrastructure procedures, recommend technical strategies, and propose improvements to enhance network capabilities. Stay up-to-date on the latest network and security technologies and trends. Work as part of a collaborative international team, contributing to team presentations and knowledge sharing. To be successful, you'll need: Proven expertise in Cisco network solutions (CCNP R&S/Sec/Wireless preferred)for both BAU and project work. In-depth knowledge of network security principles and experience with Fortinet firewalls. Experience deploying and managing large, complex network infrastructure (routing, switching, wireless, security). Solid understanding of ITIL v3 framework for incident, change, and problem management. Excellent troubleshooting skills with experience using Wireshark or similar protocol analysers. Strong communication and teamwork skills, with the ability to work independently and collaborate effectively.
-
- Security Analyst - Internal role. London commutable. £50,000
- London
- £50,000
-
Security Analyst - Internal role. London commutable opportunity. Operational Security - Investigate, escalate and proactively work to ensure household name remains protected. Project Security - Coordinate, log change requests with project delivery teams to meet security requirements Policy / compliance - work with team to aid in uplifting these as and where needed This role is role to investigate, escalate and proactively work to protect a globally recognised brand. You must have current hands on operational analytical security experience with Microsoft technology stack Someone with a SOC Analyst / security engineering background would be well suited. This position will join a small team and would suit someone that has broad experience across the security threat landscape. Experience / knowledge across industry GRC standards such NIST, ISO27001 etc would be advantageous. You will work across multiple teams proactively working to secure the business. Must be able to commute to Central London 3 days a week. Visa sponsorship not available Apply today to find out more.
-
- Network & Security Consultant
- Romania
- €54000 plus benefits
-
Senior Network & Security Engineer to join a Managed Network & Security Team in Europe. In this critical role, you will: Play a pivotal role in managing and securing network infrastructure across datacenters, customer connections, and on-premise deployments. Proactively monitor network and security devices, analyse incidents, and implement solutions to ensure optimal performance and security. Collaborate with colleagues and customers to troubleshoot issues, troubleshoot outages, and implement effective resolutions. Lead and participate in network system installations for new facilities and expansions. Develop and maintain network infrastructure procedures, recommend technical strategies, and propose improvements to enhance network capabilities. Stay up-to-date on the latest network and security technologies and trends. Work as part of a collaborative international team, contributing to team presentations and knowledge sharing. To be successful, you'll need: Proven expertise in Cisco network solutions (CCNP R&S/Sec/Wireless preferred) for both BAU and project work. In-depth knowledge of network security principles and experience with Fortinet firewalls. Experience deploying and managing large, complex network infrastructure (routing, switching, wireless, security). Solid understanding of ITIL v3 framework for incident, change, and problem management. Excellent troubleshooting skills with experience using Wireshark or similar protocol analysers. Strong communication and teamwork skills, with the ability to work independently and collaborate effectively.
-
- Network & Security Consultant
- Hungary
- Upto €54000 per year and benefits
-
Senior Network & Security Engineer to join a Managed Network & Security Team in Europe. In this critical role, you will: Play a pivotal role in managing and securing network infrastructure across datacenters, customer connections, and on-premise deployments. Proactively monitor network and security devices, analyse incidents, and implement solutions to ensure optimal performance and security. Collaborate with colleagues and customers to troubleshoot issues, troubleshoot outages, and implement effective resolutions. Lead and participate in network system installations for new facilities and expansions. Develop and maintain network infrastructure procedures, recommend technical strategies, and propose improvements to enhance network capabilities. Stay up-to-date on the latest network and security technologies and trends. Work as part of a collaborative international team, contributing to team presentations and knowledge sharing. To be successful, you'll need: Proven expertise in Cisco network solutions (CCNP R&S/Sec/Wireless preferred) for both BAU and project work. In-depth knowledge of network security principles and experience with Fortinet firewalls. Experience deploying and managing large, complex network infrastructure (routing, switching, wireless, security). Solid understanding of ITIL v3 framework for incident, change, and problem management. Excellent troubleshooting skills with experience using Wireshark or similar protocol analysers. Strong communication and teamwork skills, with the ability to work independently and collaborate effectively.