go.dev/_content/solutions/google/sitereliability.md - website - Git at Google

 ---
 title: "Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go"
 layout: solution
 company: Google Site Reliability Engineering (SRE)
 logoSrc: sitereliability.svg
 heroImgSrc: go_sitereliability_case_study.png
 series: Case Studies
 quote: |
   Google’s Site Reliability Engineering team has a mission to protect, provide for, and progress the software and systems behind all of Google’s public services — Google Search, Ads, Gmail, Android, YouTube, and App Engine, to name just a few — with an ever-watchful eye on their availability, latency, performance, and capacity.

   They shared their experience building core production management systems with Go, coming from experience with Python and C++.
 authors:
   - Pierre Palatin, Site Reliability Engineer
 ---

 Google runs a small number of very large services. Those services are powered
 by a global infrastructure covering everything a developer needs: storage
 systems, load balancers, network, logging, monitoring, and much more.
 Nevertheless, it is not a static system—it cannot be. Architecture evolves,
 new products and ideas are created, new versions must be rolled out, configs
 pushed, database schema updated, and more. We end up deploying changes to our
 systems dozens of times per second.

 Because of this scale and critical need for reliability, Google pioneered Site
 Reliability Engineering (SRE), a role that many other companies have since adopted.
 “SRE is what you get when you treat operations as if it’s a software problem.
 Our mission is to protect, provide for, and progress the software and systems
 behind all of Google’s public services with an ever-watchful eye on their
 availability, latency, performance, and capacity.”
 — [Site Reliability Engineering (SRE)](https://sre.google/).

 {{backgroundquote `
   quote: |
     Go promised a sweet spot between performance and readability that neither of
     the other languages [Python and C++] were able to offer.
 `}}

 In 2013-2014, Google’s SRE team realized that our approach to production
 management was not cutting it anymore in many ways. We had advanced far beyond
 shell scripts, but our scale had so many moving pieces and complexities that a
 new approach was needed. We determined that we needed to move toward a
 declarative model of our production, called "Prodspec", driving a dedicated
 control plane, called "Annealing".

 When we started those projects, Go was just becoming a viable option for
 critical services at Google. Most engineers were more familiar with Python
 and C++, either of which would have been valid choices. Nevertheless, Go
 captured our interest. The appeal of novelty was certainly a factor of
 course. But, more importantly, Go promised a sweet spot between performance
 and readability that neither of the other languages were able to offer. We
 started a small experiment with Go for some initial parts of Annealing and
 Prodspec. As the projects progressed, those initial parts written in Go found
 themselves at the core. We were happy with Go—its simplicity grew on us, the
 performance was there, and concurrency primitives would have been hard to
 replace.

 {{backgroundquote `
   quote: |
     Now the majority of Google production is managed and maintained by our systems
     written in Go.
 `}}

 At no point was there ever a mandate or requirement to use Go, but we had no
 desire to return to Python or C++. Go grew organically in Annealing and
 Prodspec. It was the right choice, and thus is now our language of choice.
 Now the majority of Google production is managed and maintained by our systems
 written in Go.

 The power of having a simple language in those projects is hard to overstate.
 There have been cases where some feature was indeed missing, such as the
 ability to enforce in the code that some complex structure should not be
 mutated. But for each one of those cases, there have undoubtedly been tens or
 hundred of cases where the simplicity helped.

 {{backgroundquote `
   quote: |
     Go’s simplicity means that the code is easy to follow, whether it is to spot
     bugs during review or when trying to determine exactly what happened during a
     service disruption.
 `}}

 For example, Annealing impacts a wide variety of teams and services meaning
 that we relied heavily on contributions across the company. The simplicity of
 Go made it possible for people outside our team to see why some part or another
 was not working for them, and often provide fixes or features themselves. This
 allowed us to quickly grow.

 Prodspec and Annealing are in charge of some quite critical components. Go’s
 simplicity means that the code is easy to follow, whether it is to spot bugs
 during review or when trying to determine exactly what happened during a
 service disruption.

 Go performance and concurrency support have also been key for our work. As our
 model of production is declarative, we tend to manipulate a lot of structured
 data, which describes what production is and what it should be. We have large
 services so the data can grow large, often making purely sequential processing
 not efficient enough.

 We are manipulating this data in many ways and many places. It is not a matter
 of having a smart person come up with a parallel version of our algorithm. It
 is a matter of casual parallelism, finding the next bottleneck and
 parallelising that code section. And Go enables exactly that.

 As a result of our success with Go, we now use Go for every new development for
 Prodspec and Annealing.

 In addition to the Site Reliability Engineering team, engineering teams across
 Google have adopted Go in their development process. Read about how the
 [Core Data Solutions](/solutions/google/coredata/),
 [Firebase Hosting](/solutions/google/firebase/), and
 [Chrome](/solutions/google/chrome/) teams use Go to build fast, reliable,
 and efficient software at scale.
	---
	title: "Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go"
	layout: solution
	company: Google Site Reliability Engineering (SRE)
	logoSrc: sitereliability.svg
	heroImgSrc: go_sitereliability_case_study.png
	series: Case Studies
	quote: \|
	Google’s Site Reliability Engineering team has a mission to protect, provide for, and progress the software and systems behind all of Google’s public services — Google Search, Ads, Gmail, Android, YouTube, and App Engine, to name just a few — with an ever-watchful eye on their availability, latency, performance, and capacity.

	They shared their experience building core production management systems with Go, coming from experience with Python and C++.
	authors:
	- Pierre Palatin, Site Reliability Engineer
	---

	Google runs a small number of very large services. Those services are powered
	by a global infrastructure covering everything a developer needs: storage
	systems, load balancers, network, logging, monitoring, and much more.
	Nevertheless, it is not a static system—it cannot be. Architecture evolves,
	new products and ideas are created, new versions must be rolled out, configs
	pushed, database schema updated, and more. We end up deploying changes to our
	systems dozens of times per second.

	Because of this scale and critical need for reliability, Google pioneered Site
	Reliability Engineering (SRE), a role that many other companies have since adopted.
	“SRE is what you get when you treat operations as if it’s a software problem.
	Our mission is to protect, provide for, and progress the software and systems
	behind all of Google’s public services with an ever-watchful eye on their
	availability, latency, performance, and capacity.”
	— [Site Reliability Engineering (SRE)](https://sre.google/).

	{{backgroundquote `
	quote: \|
	Go promised a sweet spot between performance and readability that neither of
	the other languages [Python and C++] were able to offer.
	`}}

	In 2013-2014, Google’s SRE team realized that our approach to production
	management was not cutting it anymore in many ways. We had advanced far beyond
	shell scripts, but our scale had so many moving pieces and complexities that a
	new approach was needed. We determined that we needed to move toward a
	declarative model of our production, called "Prodspec", driving a dedicated
	control plane, called "Annealing".

	When we started those projects, Go was just becoming a viable option for
	critical services at Google. Most engineers were more familiar with Python
	and C++, either of which would have been valid choices. Nevertheless, Go
	captured our interest. The appeal of novelty was certainly a factor of
	course. But, more importantly, Go promised a sweet spot between performance
	and readability that neither of the other languages were able to offer. We
	started a small experiment with Go for some initial parts of Annealing and
	Prodspec. As the projects progressed, those initial parts written in Go found
	themselves at the core. We were happy with Go—its simplicity grew on us, the
	performance was there, and concurrency primitives would have been hard to
	replace.

	{{backgroundquote `
	quote: \|
	Now the majority of Google production is managed and maintained by our systems
	written in Go.
	`}}

	At no point was there ever a mandate or requirement to use Go, but we had no
	desire to return to Python or C++. Go grew organically in Annealing and
	Prodspec. It was the right choice, and thus is now our language of choice.
	Now the majority of Google production is managed and maintained by our systems
	written in Go.

	The power of having a simple language in those projects is hard to overstate.
	There have been cases where some feature was indeed missing, such as the
	ability to enforce in the code that some complex structure should not be
	mutated. But for each one of those cases, there have undoubtedly been tens or
	hundred of cases where the simplicity helped.

	{{backgroundquote `
	quote: \|
	Go’s simplicity means that the code is easy to follow, whether it is to spot
	bugs during review or when trying to determine exactly what happened during a
	service disruption.
	`}}

	For example, Annealing impacts a wide variety of teams and services meaning
	that we relied heavily on contributions across the company. The simplicity of
	Go made it possible for people outside our team to see why some part or another
	was not working for them, and often provide fixes or features themselves. This
	allowed us to quickly grow.

	Prodspec and Annealing are in charge of some quite critical components. Go’s
	simplicity means that the code is easy to follow, whether it is to spot bugs
	during review or when trying to determine exactly what happened during a
	service disruption.

	Go performance and concurrency support have also been key for our work. As our
	model of production is declarative, we tend to manipulate a lot of structured
	data, which describes what production is and what it should be. We have large
	services so the data can grow large, often making purely sequential processing
	not efficient enough.

	We are manipulating this data in many ways and many places. It is not a matter
	of having a smart person come up with a parallel version of our algorithm. It
	is a matter of casual parallelism, finding the next bottleneck and
	parallelising that code section. And Go enables exactly that.

	As a result of our success with Go, we now use Go for every new development for
	Prodspec and Annealing.

	In addition to the Site Reliability Engineering team, engineering teams across
	Google have adopted Go in their development process. Read about how the
	[Core Data Solutions](/solutions/google/coredata/),
	[Firebase Hosting](/solutions/google/firebase/), and
	[Chrome](/solutions/google/chrome/) teams use Go to build fast, reliable,
	and efficient software at scale.