# Bug-resistant build constraints — Draft Design

Russ Cox\
June 30, 2020

This is a **Draft Design**, not a formal Go proposal,
because it describes
a potential [large change](https://research.swtch.com/proposals-large).
The goal of circulating this draft design is to collect feedback
to shape an intended eventual proposal.

We are using this change to experiment with new ways to
[scale discussions](https://research.swtch.com/proposals-discuss)
about large changes.

For this change, we will use
[a Go Reddit thread](https://golang.org/s/go-build-reddit)
to manage Q&A, since Reddit's threading support
can easily match questions with answers
and keep separate lines of discussion separate.

There is also a [video presentation](https://golang.org/s/go-build-video) of this draft design.

The [prototype code](https://golang.org/s/go-build-code) is also available for trying out.

## Abstract

We present a possible plan to
transition from the current `// +build` lines for build tag selection
to new `//go:build` lines that use standard boolean expression syntax.
The plan includes relaxing the possible placement of `//go:build` lines
compared to `// +build` lines, as well as rejecting misplaced `//go:build` lines.
These changes should make build constraints easier to use
and reduce time spent debugging mistakes.
The plan also includes a graceful transition from `// +build` to `//go:build`
syntax that avoids breaking the Go ecosystem.

This design draft is based on a preliminary discussion on
[golang.org/issue/25348](https://golang.org/issue/25348),
but discussion of this design draft should happen on
[the Go Reddit thread](https://golang.org/s/go-build-reddit).


## Background

It can be necessary to write different Go code for different compilation contexts.
Go’s solution to this problem is conditional compilation at the file level:
each file is either in the build or not.
(Compared with the C preprocessor’s conditional selection of individual lines,
selection of individual files is easier to understand and requires no special support
in tools that parse single source files.)

Go refers to the current operating system as `$GOOS`
and the current architecture as `$GOARCH`.
This section uses the generic names GOOS and GOARCH
to stand in for any of the specific names (windows, linux, 386, amd64, and so on).

When the `go/build` package was written in August 2011,
it added explicit support for the convention that files
named `*_GOOS.*`, `*_GOARCH.*`, or `*_GOOS_GOARCH.*`
only compiled on those specific systems.
Until then, the convention was only a manual one,
maintained by hand in the package Makefiles.

For more complex situations, such as files that applied
to multiple operating systems (but not all),
build constraints were introduced in September 2011.

### Syntax

Originally, the arguments to a build constraint
were a list of alternatives, each of which took
one of three possible forms:
an operating system name (GOOS),
an architecture (GOARCH),
or both separated by a slash (GOOS/GOARCH).

[CL 5018044](https://golang.org/cl/5018044)
used a `//build` prefix,
but a followup discussion on
[CL 5011046](https://codereview.appspot.com/5011046)
while updating the tree to use the comments
led to the syntax changing to `// +build`
in [CL 5015051](https://golang.org/cl/5015051).

For example, this line indicated that the file should build
on Linux for any architecture, or on Windows only for 386:

	// +build linux windows/386

That is, each line listed a set of OR’ed conditions.
Because each line applied independently, multiple lines
were in effect AND’ed together.

For example,

	// +build linux windows
	// +build amd64

and

	// +build linux/amd64 windows/amd64

were equivalent.

In December 2011, [CL 5489100](https://golang.org/cl/5489100)
added the `cgo` and `nocgo` build tags.
It also generalized slash syntax to mean AND of arbitrary tags,
not just GOOS and GOARCH, as in `// +build nocgo/linux`.

In January 2012, [CL 5554079](https://golang.org/cl/5554079)
added support for custom build tags (such as `appengine`),
changed slash to comma, and introduced `!` for negation.

In March 2013, [CL 7794043](https://golang.org/cl/7794043) added the `go1.1` build tag,
enabling release-specific file selection.
The syntax changed to allow dots in tag names.

Although each of these steps makes sense in isolation,
we have arrived at a non-standard boolean expression syntax
capable of expressing ANDs of ORs of ANDs of potential NOTs of tags.
It is difficult for developers to remember the syntax.
For example, two of these three mean the same thing, but which two?

	// +build linux,386

	// +build linux 386

	// +build linux
	// +build 386

The simple form worked well in the original context,
but it has not evolved gracefully.
The current richness of expression would be better served
by a more familiar syntax.

Surveying the public Go ecosystem for `// +build` lines in March 2020
turned up a few illustrative apparent bugs that had so far eluded
detection. These bugs might have been avoided entirely
if developers been working with more familiar syntax.

 - [github.com/streamsets/datacollector-edge](https://github.com/streamsets/datacollector-edge/issues/8)

       // +build 386 windows,amd64 windows

   Confused AND and OR: simplifies to “`386` OR `windows`”.\
   Apparently intended `// +build 386,windows amd64,windows`.

 - [github.com/zchee/xxhash3](https://github.com/zchee/xxhash3/issues/1)

       // +build 386 !gccgo,amd64 !gccgo,amd64p32 !gccgo

   Confused AND and OR: simplifies to “`386` OR NOT `gccgo`”.\
   Apparently intended:

       // +build 386 amd64 amd64p32
       // +build !gccgo


 - [github.com/gopherjs/vecty](https://github.com/gopherjs/vecty/issues/261)

       // +build go1.12,wasm,js js

   Intended meaning unclear; simplifies to just “`js`”.

 - [gitlab.com/aquachain/aquachain](https://gitlab.com/aquachain/aquachain/-/issues/2)

       // +build windows,solaris,nacl nacl solaris windows

   Intended (but at least equivalent to) `// +build nacl solaris windows`.

 - [github.com/katzenpost/core](https://github.com/katzenpost/core/issues/97)

       // +build linux,!amd64
       // +build linux,amd64,noasm
       // +build !go1.9

   Unsatisfiable (`!amd64` and `amd64` can’t both be true).\
   Apparently intended `// +build linux,!amd64 linux,amd64,noasm !go1.9`.

Later, in June 2020, Alan Donovan wrote some 64-bit specific code that he annotated with:

	//+build linux darwin
	//+build amd64 arm64 mips64x ppc64x

For the file implementing the generic fallback, he needed the negation of that condition and wrote:

	//+build !linux,!darwin
	//+build !amd64,!arm64,!mips64x,!ppc64x

This is subtly wrong. He correctly negated each line, but repeated lines apply constraints independently,
meaning they are ANDed together. To negate the overall meaning, the two lines in the fallback need
to be ORed together, meaning they need to be a single line:

	//+build !linux,!darwin !amd64,!arm64,!mips64x,!ppc64x

Alan has written a very good book about Go—he is certainly an experienced developer—and [still got this wrong](https://github.com/google/starlark-go/pull/280).
It's all clearly too subtle.
Getting ahead of ourselves just a little, if we used a standard boolean syntax,
then the first file would have used:

	(linux || darwin) && (amd64 || arm64 || mips64x || ppc64x)

and the negation in the fallback would have been easy:

	!((linux || darwin) && (amd64 || arm64 || mips64x || ppc64x))

### Placement

In addition to confusion about syntax, there is also confusion
about placement. The documentation (in `go doc go/build`) explains:

> “Constraints may appear in any kind of source file (not just Go),
> but they must appear near the top of the file, preceded only by blank lines
> and other line comments. These rules mean that in Go files a build
> constraint must appear before the package clause.
>
> To distinguish build constraints from package documentation, a series of
> build constraints must be followed by a blank line.”

Because the search for build constraints stops
at the first non-`//`, non-blank line (usually the Go `package` statement),
this is an ignored build constraint:

	package main

	// +build linux

The syntax even excludes
C-style `/* */` comments, so this is an ignored build constraint:

	/*
	Copyright ...
	*/

	// +build linux

	package main

Furthermore, to avoid confusion with doc comments,
the search stops stops at the last blank line before the
non-`//`, non-blank line, so this is an ignored build constraint
(and a doc comment):

	// +build linux
	package main

This is also an ignored build constraint, in an assembly file:

	// +build 386 amd64
	#include "textflag.h"

Surveying the public Go ecosystem for `// +build` lines in March 2020
turned up

 - 98 ignored build constraints after `/* */` comments,
   usually copyright notices;
 - 50 ignored build constraints in doc comments;
 - and 11 ignored build constraints after the `package` declaration.

These are small numbers compared to the 110,000 unique files found
that contained build constraints,
but these are only the ones that slipped through, unnoticed,
into the latest public commits.
We should expect that there are many more such mistakes
corrected in earlier commits
or that lead to head-scratching debugging sessions
but avoid being committed.

## Design

The core idea of the design is
to replace the current `// +build` lines for build tag selection
with new `//go:build` lines that use more familiar boolean expressions.
For example, the old syntax

	// +build linux
	// +build 386

would be replaced by the new syntax

	//go:build linux && 386

The design also admits `//go:build` lines in more locations
and rejects misplaced `//go:build` lines.

The key to the design is a smooth transition that avoids
breaking Go code.

The next three sections explain these three parts of the design in detail.

### Syntax

The new syntax is given by this grammar, using the [notation of the Go spec](https://golang.org/ref/spec#Notation):

	BuildLine      = "//go:build" Expr
	Expr           = OrExpr
	OrExpr         = AndExpr   { "||" AndExpr }
	AndExpr        = UnaryExpr { "&&" UnaryExpr }
	UnaryExpr      = "!" UnaryExpr | "(" Expr ")" | tag
	tag            = tag_letter { tag_letter }
	tag_letter     = unicode_letter | unicode_digit | "_" | "."

That is, the syntax of build tags is unchanged from its current form,
but the combination of build tags is now done with
Go’s `||`, `&&`, and `!` operators and parentheses.
(Note that build tags are not always valid Go expressions,
even though they share the operators,
because the tags are not always valid identifiers.
For example: “`go1.1`”.)

It is an error for a file to have more than one `//go:build` line,
to eliminate confusion about whether multiple lines are
implicitly ANDed or ORed together.

### Placement

The current search for build constraints can be explained concisely,
but it has unexpected behaviors that are difficult to understand,
as discussed in the Background section.

It remains useful for both people and programs like the `go` command
not to need to read the entire file to find any build constraints,
so this design still ends the search for build constraints
at the first non-comment text in the file.
However, this design allows placing `//go:build` constraints
after `/* */` comments or in doc comments.
([Proposal issue 37974](https://golang.org/issue/37974),
which will ship in Go 1.15,
strips those `//go:` lines out of the doc comments.)

The new rule would be:

> “Constraints may appear in any kind of source file (not just Go),
> but they must appear near the top of the file, preceded only by blank lines
> and other `//` and `/* */` comments. These rules mean that in Go files a build
> constraint must appear before the package clause.”

In addition to this more relaxed rule,
the design would change `gofmt` to move misplaced
build constraints
to valid locations,
and it would change the Go compiler and assembler
reject misplaced build constraints.
This will correct most misplaced constraints automatically
and report the others; no misplaced constraint should go unnoticed.
(The next section describes the tool changes in more detail.)

### Transition

A smooth transition is critical for a successful rollout.
By the [Go release policy](https://golang.org/doc/devel/release.html#policy),
the release of Go 1.N
ends support for Go 1.(N-2), but most users will still want to be
able to write code that works with both Go 1.(N−1) and Go 1.N.
If Go 1.N introduces support for `//go:build` lines,
code that needs to build with the past two releases can’t fully adopt
the new syntax until Go 1.(N+1) is released.
Publishers of popular dependencies may not realize this,
which may lead to breakage in the Go ecosystem.
We must make accidental breakage unlikely.

To help with the transition, we envision a plan
carried out over three Go releases.
For concreteness, we call them Go 1.(N−1), Go 1.N, and Go 1.(N+1).
The bulk of the work happens in Go 1.N,
with minor preparation in Go 1.(N−1) and minor cleanup in Go 1.(N+1).

**Go 1.(N−1)** would prepare for the transition with minimal changes.
In Go 1.(N−1):

 - Builds will fail when a Go source file
   contains `//go:build` lines without `// +build` lines.
 - Builds will _not_ otherwise look at `//go:build` lines
   for file selection.
 - Users will not be encouraged to use `//go:build` lines yet.

At this point:

 - Packages will build in Go 1.(N−1) using the same files as in Go 1.(N-2), always.
 - Go 1.(N−1) release notes will not mention `//go:build` at all.

**Go 1.N** would start the transition. In Go 1.N:

 - Builds will start preferring `//go:build` lines for file selection.
   If there is no `//go:build` in a file, then any `// +build` lines
   still apply.
 - Builds will no longer fail if a Go file contains `//go:build`
   without `// +build`.
 - Builds will fail if a Go or assembly file contains `//go:build` too late in the file.
 - `Gofmt` will move misplaced `//go:build` and `// +build`
   lines to their proper location in the file.
 - `Gofmt` will format the expressions in `//go:build` lines
   using the same rules as for other Go boolean expressions
   (spaces around all `&&` and `||` operators).
 - If a file contains only `// +build` lines,
   `gofmt` will add an equivalent `//go:build` line above them.
 - If a file contains both `//go:build` and `// +build` lines,
   `gofmt` will consider the `//go:build` the source of truth
   and update the `// +build` lines to match,
   preserving compatibility with earlier versions of Go.
   `Gofmt` will also reject `//go:build` lines that are deemed
   too complex to convert into `// +build` format,
   although this situation will be rare.
   (Note the “If” at the start of this bullet.
   `Gofmt` will _not_ add `// +build` lines to a file
   that only has `//go:build`.)
 - The `buildtags` check in `go vet` will add support for `//go:build` constraints.
   It will fail when a Go source file contains
   `//go:build` and `// +build` lines with different meanings.
   If the check fails, one can run `gofmt` `-w`.
 - The `buildtags` check will also fail when a Go source file contains
   `//go:build` without `// +build` and its containing module
   has a `go` line listing a version before Go 1.N.
   If the check fails, one can add any `// +build` line and then run `gofmt` `-w`,
   which will replace it with the correct ones.
   Or one can bump the `go.mod` go version to Go 1.N.
 - Release notes will explain `//go:build` and the transition.

At this point:

 - Go 1.(N-2) is now unsupported, per the [Go release policy](https://golang.org/doc/devel/release.html#policy).
 - Packages will build in Go 1.N using the same files as in Go 1.(N−1),
   provided they pass Go 1.N `go vet`.
 - Packages that contain conflicting `//go:build` and `// +build` lines
   will fail Go 1.N `go vet`.
 - Anyone using `gofmt` on save will not fail `go vet`.
 - Packages that contain only `//go:build` lines will work fine when
   using only Go 1.N.
   If such packages are built using Go 1.(N−1), the build will fail, loud and clear.

**Go 1.(N+1)** would complete the transition. In Go 1.(N+1):

 - A new fix in `go fix` will remove `// +build` stanzas,
   making sure to leave behind equivalent `//go:build` lines.
   The removal only happens when `go fix` is being run in a module
   with a `go 1.N` (or later) line, which is taken as an explicit signal
   that the developer no longer needs compatibility with Go 1.(N−1).
   The removal is never done in GOPATH mode, which lacks
   any such explicit signal.
 - Release notes will explain that the transition is complete.

At this point:

 - Go 1.(N−1) is now unsupported, per the Go release policy.
 - Packages will build in Go 1.(N+1) using the same file as in Go 1.N, always.
 - Running `go fix` will remove all `// +build` lines from the source tree,
   leaving behind equivalent, easier-to-read `//go:build` lines,
   but only on modules that have set a base requirement of Go 1.N.

## Rationale

The motivation is laid out in the Background section above.

The rationale for using Go syntax is that Go developers already
understand one boolean expression syntax.
It makes more sense to reuse that one
than to maintain a second one.
No other syntaxes were considered: any other syntax would be
a second (well, a third) syntax to learn.

As Michael Munday noted in 2018, these lines are difficult to test,
so we would be well served to make them as straightforward as possible
to understand.
This observation is reinforced by the examples in the introduction.

Tooling will need to keep supporting `// +build` lines indefinitely,
in order to continue to build old code.
Similarly, the `go vet` check that `//go:build` and `// +build` lines
match when both are present will be kept indefinitely.
But the `go fix` should at least help us remove `// +build` lines
from new versions of code, so that most developers stop seeing them
or needing to edit them.

The rationale for disallowing multiple `//go:build` lines
is that the entire goal of this design is to replace
implicit AND and OR with explicit `&&` and `||` operators.
Allowing multiple lines reintroduces an implicit operator.

The rationale for the transition is laid out in that section.
We don’t want to break Go developers unnecessarily,
nor to make it easy for dependency authors to break
their dependents.

The rationale for using `//go:build` as the new prefix is that
it matches our now-established convention of using `//go:`
prefixes for directives to the build system or compilers
(see also `//go:generate`, `//go:noinline`, and so on).

The rationale for introducing a new prefix (instead of reusing `// +build`)
is that the new prefix makes it possible to write files that
contain both syntaxes, enabling a smooth transition.

The main alternative to this design is to do nothing at all
and simply stick with the current syntax.
That is an attractive option, since it avoids going through
a transition.
On the other hand, the benefit of the clearer syntax will only grow as we get
more and more Go developers and more and more Go code,
and the transition should be fairly smooth and low-cost.

## Compatibility

Go code that builds today will keep building indefinitely,
without any changes.

The transition aims to make it difficult
to cause new incompatibilities accidentally,
with `gofmt` and `go vet` working to keep old and
new syntaxes in sync during the transition.

Compatibility is also the reason for not providing the `go` `fix`
that removes `// +build` lines until Go 1.(N+1) is released:
that way, the automated tool that breaks Go 1.(N−1) users
is not even available until Go 1.(N−1) is no longer supported.