PerformanceMonitoring.md - wiki - Git at Google


            
            ---

            
            title: PerformanceMonitoring

            
            ---


            The Go project monitors the performance characteristics of the Go implementation

            
            as well as that of subrepositories like golang.org/x/tools.


            ## Benchmarks


            `golang.org/x/benchmarks/cmd/bench` is the entrypoint for our performance tests.

            
            For Go implementations, this runs both the

            
            [Sweet](https://golang.org/x/benchmarks/sweet) (end-to-end benchmarks)

            
            and [bent](https://golang.org/x/benchmarks/cmd/bent) (microbenchmarks)

            
            benchmarking suites.


            For the `golang.org/x/tools` project, it runs the repository's benchmarks.


            These benchmarks can all be invoked manually, as can `cmd/bench`, but using both

            
            Sweet and bent directly will likely offer a better user experience.

            
            See their documentation for more details.


            ## Performance testing principles


            ### Change with the times


            Our set of benchmarks is curated.

            
            It is allowed to change over time.

            
            Sticking to a single benchmark set over a long period of time can easily land

            
            us in a situation where we're optimizing for the wrong thing.


            ### Always perform a comparison


            We never report performance numbers in isolation, and only relative to some

            
            baseline.

            
            This strategy comes from the fact that comparing performance data taken far

            
            apart in time, even on the same hardware, can result in a lot of noise that

            
            goes unaccounted for.

            
            The state of a machine or VM on one day is likely to be very different than

            
            the state of a machine or VM on the next day.


            We refer to the tested version of source code as the "experiment" and the

            
            baseline version of source code as the "baseline."


            ## Presubmit


            Do you have a Gerrit change that you want to run against our benchmarks?


            Select a builder containing the word `perf` in the "Choose Tryjobs" dialog that

            
            appears when selecting a [SlowBot](/wiki/SlowBots).


            There are two kinds of presubmit builders for performance testing:


            - `perf_vs_parent`, which measures the performance delta of a change in isolation.

            
            - `perf_vs_tip`, which measures the performance delta versus the current

            
              tip-of-tree for whichever repository the change is for.

            
              (Remember to rebase your change(s) before using this one!)


            There's a third special presubmit builder for the tools repository as well which

            
            contains the string `perf_vs_gopls_0_11`.

            
            This measures the performance delta versus the `release-branch-gopls.0.11` branch

            
            of the tools repository.


            ## Postsubmit


            The [performance dashboard](http://perf.golang.org/dashboard) provides

            
            continuous monitoring of benchmark performance for every commit that is made to

            
            the main Go repository and other subrepositories.

            
            The dashboard, more specifically, displays graphs showing the change in certain

            
            performance metrics (also called "units") over time for different benchmarks.

            
            Use the navigation interface at the top of the page to explore further.


            The [regressions page](https://perf.golang.org/dashboard/?benchmark=regressions)

            
            displays all benchmarks in order of biggest regression to biggest improvement,

            
            followed by all benchmarks for which there is no statistically clear answer.


            On the graphs, red means regression, blue means improvement.


            ### Baselines


            In post-submit, the baseline version for Go repository performance tests is

            
            automatically determined.

            
            For performance tests against changes on release branches, the baseline is always

            
            the latest release for that branch (for example, the latest minor release for

            
            Go 1.21 on `release-branch.go1.21`).

            
            For performance tests against tip-of-tree, the baseline is always the latest

            
            overall release of Go.

            
            This is indicated by the name of the builder that produces these benchmark

            
            esults, which contains the string `perf_vs_release`.

            
            What this means is that on every minor release of Go, the baseline shifts.

            
            These baseline shifts can be observed in the [per-metric view](#per-metric-view).


            Performance tests on subrepositories typically operate against some known

            
            long-term fixed baseline.

            
            For the tools repository, it's the tip of the `release-branch-gopls.0.11`

            
            branch.


            ### Per-metric view


            Click on any graph's performance metric name to view a more detailed timeline

            
            of the performance deltas for that metric.


            ![Image displaying which link to click to reach the per-metric

            
            page.](images/performance-monitoring-per-metric-link.png)


            This view is particularly useful for identifying the culprit behind a regression

            
            and for pinpointing the source of an improvement.


            Sometimes, a performance change happens because a benchmark has changed or

            
            because the baseline version being used as changed.

            
            This view also displays information about the baseline versions and the version

            
            of `golang.org/x/benchmarks` that was used to produce the results to help identify

            
            when this happens.
	---
	title: PerformanceMonitoring
	---

	The Go project monitors the performance characteristics of the Go implementation
	as well as that of subrepositories like golang.org/x/tools.

	## Benchmarks

	`golang.org/x/benchmarks/cmd/bench` is the entrypoint for our performance tests.
	For Go implementations, this runs both the
	[Sweet](https://golang.org/x/benchmarks/sweet) (end-to-end benchmarks)
	and [bent](https://golang.org/x/benchmarks/cmd/bent) (microbenchmarks)
	benchmarking suites.

	For the `golang.org/x/tools` project, it runs the repository's benchmarks.

	These benchmarks can all be invoked manually, as can `cmd/bench`, but using both
	Sweet and bent directly will likely offer a better user experience.
	See their documentation for more details.

	## Performance testing principles

	### Change with the times

	Our set of benchmarks is curated.
	It is allowed to change over time.
	Sticking to a single benchmark set over a long period of time can easily land
	us in a situation where we're optimizing for the wrong thing.

	### Always perform a comparison

	We never report performance numbers in isolation, and only relative to some
	baseline.
	This strategy comes from the fact that comparing performance data taken far
	apart in time, even on the same hardware, can result in a lot of noise that
	goes unaccounted for.
	The state of a machine or VM on one day is likely to be very different than
	the state of a machine or VM on the next day.

	We refer to the tested version of source code as the "experiment" and the
	baseline version of source code as the "baseline."

	## Presubmit

	Do you have a Gerrit change that you want to run against our benchmarks?

	Select a builder containing the word `perf` in the "Choose Tryjobs" dialog that
	appears when selecting a [SlowBot](/wiki/SlowBots).

	There are two kinds of presubmit builders for performance testing:

	- `perf_vs_parent`, which measures the performance delta of a change in isolation.
	- `perf_vs_tip`, which measures the performance delta versus the current
	tip-of-tree for whichever repository the change is for.
	(Remember to rebase your change(s) before using this one!)

	There's a third special presubmit builder for the tools repository as well which
	contains the string `perf_vs_gopls_0_11`.
	This measures the performance delta versus the `release-branch-gopls.0.11` branch
	of the tools repository.

	## Postsubmit

	The [performance dashboard](http://perf.golang.org/dashboard) provides
	continuous monitoring of benchmark performance for every commit that is made to
	the main Go repository and other subrepositories.
	The dashboard, more specifically, displays graphs showing the change in certain
	performance metrics (also called "units") over time for different benchmarks.
	Use the navigation interface at the top of the page to explore further.

	The [regressions page](https://perf.golang.org/dashboard/?benchmark=regressions)
	displays all benchmarks in order of biggest regression to biggest improvement,
	followed by all benchmarks for which there is no statistically clear answer.

	On the graphs, red means regression, blue means improvement.

	### Baselines

	In post-submit, the baseline version for Go repository performance tests is
	automatically determined.
	For performance tests against changes on release branches, the baseline is always
	the latest release for that branch (for example, the latest minor release for
	Go 1.21 on `release-branch.go1.21`).
	For performance tests against tip-of-tree, the baseline is always the latest
	overall release of Go.
	This is indicated by the name of the builder that produces these benchmark
	esults, which contains the string `perf_vs_release`.
	What this means is that on every minor release of Go, the baseline shifts.
	These baseline shifts can be observed in the [per-metric view](#per-metric-view).

	Performance tests on subrepositories typically operate against some known
	long-term fixed baseline.
	For the tools repository, it's the tip of the `release-branch-gopls.0.11`
	branch.

	### Per-metric view

	Click on any graph's performance metric name to view a more detailed timeline
	of the performance deltas for that metric.

	![Image displaying which link to click to reach the per-metric
	page.](images/performance-monitoring-per-metric-link.png)

	This view is particularly useful for identifying the culprit behind a regression
	and for pinpointing the source of an improvement.

	Sometimes, a performance change happens because a benchmark has changed or
	because the baseline version being used as changed.
	This view also displays information about the baseline versions and the version
	of `golang.org/x/benchmarks` that was used to produce the results to help identify
	when this happens.