cmd/benchstat: new version of benchstat

This is a complete rewrite of benchstat. Basic usage remains the same,
as does the core idea of showing statistical benchmark summaries and
A/B comparisons in a table, but there are several major improvements.

The statistics is now more robust. Previously, benchstat used
IQR-based outlier rejection, showed the mean of the reduced sample,
its range, and did a non-parametric difference-of-distribution test on
reduced samples. Any form of outlier rejection must start with
distributional assumptions, in this case assuming normality, which is
generally not sound for benchmark data. Hence, now benchstat does not
do any outlier rejection. As a result, it must use robust summary
statistics as well, so benchstat now uses the median and confidence
interval of the median as summary statistics. Benchstat continues to
use the same Mann-Whitney U-test for the delta, but now does it on the
full samples since the U-test is already non-parametric, hence
increasing the power of this test.

As part of these statistical improvements, benchstat now detects and
warns about several common mistakes, such as having too few samples
for meaningful statistical results, or having incomparable geomeans.

The output format is more consistent. Previously, benchstat
transformed units like "ns/op" into a metric like "time/op", which it
used as a column header; and a numerator like "sec", which it used to
label each measurement. This was easy enough for the standard units
used by the testing framework, but was basically impossible to
generalize to custom units. Now, benchstat does unit scaling, but
otherwise leaves units alone. The full (scaled) unit is used as a
column header and each measurement is simply a scaled value shown with
an SI prefix. This also means that the text and CSV formats can be
much more similar while still allowing the CSV format to be usefully

Benchstat will also now do A/B comparisons even if there are more than
two inputs. It shows a comparison to the base in the second and all
subsequent columns. This approach is consistent for any number of

Benchstat now supports the full Go benchmark format, including
sophisticated control over exactly how it structures the results into
rows, columns, and tables. This makes it easy to do meaningful
comparisons across benchmark data that isn't simply structured into
two input files, and gives significantly more control over how results
are sorted. The default behavior is still to turn each input file into
a column and each benchmark into a row.

Fixes golang/go#19565 by showing all results, even if the benchmark
sets don't match across columns, and warning when geomean sets are

Fixes golang/go#19634 by no longer doing outlier rejection and clearly
reporting when there are not enough samples to do a meaningful
difference test.

Updates golang/go#23471 by providing more through command
documentation. I'm not sure it quite fixes this issue, but it's much
better than it was.

Fixes golang/go#30368 because benchstat now supports filter
expressions, which can also filter down units.

Fixes golang/go#33169 because benchstat now always shows file
configuration labels.

Updates golang/go#43744 by integrating unit metadata to control
statistical assumptions into the main tool that implements those

Fixes golang/go#48380 by introducing a way to override labels from the
command line rather than always using file names.

Change-Id: Ie2c5a12024e84b4918e483df2223eb1f10413a4f
Run-TryBot: Austin Clements <>
Reviewed-by: Michael Pratt <>
TryBot-Result: Gopher Robot <>
64 files changed
tree: 8fcddf83ee4ea65eb5fe9a753ab52f293cccbcf4
  1. analysis/
  2. benchfmt/
  3. benchmath/
  4. benchproc/
  5. benchseries/
  6. benchstat/
  7. benchunit/
  8. cmd/
  9. internal/
  10. storage/
  12. go.mod
  13. go.sum

Go benchmark analysis tools

Go Reference

This subrepository holds tools and packages for analyzing Go benchmark results, such as the output of testing package benchmarks.


This subrepository contains command-line tools for analyzing benchmark result data.

cmd/benchstat computes statistical summaries and A/B comparisons of Go benchmarks.

cmd/benchsave publishes benchmark results to

To install all of these commands, run go install You can also git clone and run go install ./cmd/... in the checkout.


Underlying the above tools are several packages for working with benchmark data. These are designed to work together, but can also be used independently.

benchfmt reads and writes the Go benchmark format.

benchunit manipulates benchmark units and formats numbers in those units.

benchproc provides tools for filtering, grouping, and sorting benchmark results.

benchmath provides tools for computing statistics over distributions of benchmark measurements.

Deprecated packages

The following packages are deprecated and no longer supported:

storage contains a deprecated version of the benchmark result storage system. These packages have moved to

analysis contains a deprecated version of the benchmark result analysis system. These packages have moved to

Report Issues / Send Patches

This repository uses Gerrit for code changes. To learn how to submit changes to this repository, see

The main issue tracker for the perf repository is located at Prefix your issue with “x/perf:” in the subject line, so it is easy to find.