Russ Cox
June 30, 2020
This is a Draft Design, not a formal Go proposal, because it describes a potential large change. The goal of circulating this draft design is to collect feedback to shape an intended eventual proposal.
We are using this change to experiment with new ways to scale discussions about large changes.
For this change, we will use a Go Reddit thread to manage Q&A, since Reddit's threading support can easily match questions with answers and keep separate lines of discussion separate.
There is also a video presentation of this draft design.
The prototype code is also available for trying out.
We present a possible plan to transition from the current // +build lines for build tag selection to new //go:build lines that use standard boolean expression syntax. The plan includes relaxing the possible placement of //go:build lines compared to // +build lines, as well as rejecting misplaced //go:build lines. These changes should make build constraints easier to use and reduce time spent debugging mistakes. The plan also includes a graceful transition from // +build to //go:build syntax that avoids breaking the Go ecosystem.
This design draft is based on a preliminary discussion on golang.org/issue/25348, but discussion of this design draft should happen on the Go Reddit thread.
It can be necessary to write different Go code for different compilation contexts. Go’s solution to this problem is conditional compilation at the file level: each file is either in the build or not. (Compared with the C preprocessor’s conditional selection of individual lines, selection of individual files is easier to understand and requires no special support in tools that parse single source files.)
Go refers to the current operating system as $GOOS and the current architecture as $GOARCH. This section uses the generic names GOOS and GOARCH to stand in for any of the specific names (windows, linux, 386, amd64, and so on).
When the go/build package was written in August 2011, it added explicit support for the convention that files named *_GOOS.*, *_GOARCH.*, or *_GOOS_GOARCH.* only compiled on those specific systems. Until then, the convention was only a manual one, maintained by hand in the package Makefiles.
For more complex situations, such as files that applied to multiple operating systems (but not all), build constraints were introduced in September 2011.
Originally, the arguments to a build constraint were a list of alternatives, each of which took one of three possible forms: an operating system name (GOOS), an architecture (GOARCH), or both separated by a slash (GOOS/GOARCH).
CL 5018044 used a //build prefix, but a followup discussion on CL 5011046 while updating the tree to use the comments led to the syntax changing to // +build in CL 5015051.
For example, this line indicated that the file should build on Linux for any architecture, or on Windows only for 386:
// +build linux windows/386
That is, each line listed a set of OR’ed conditions. Because each line applied independently, multiple lines were in effect AND’ed together.
For example,
// +build linux windows // +build amd64
and
// +build linux/amd64 windows/amd64
were equivalent.
In December 2011, CL 5489100 added the cgo and nocgo build tags. It also generalized slash syntax to mean AND of arbitrary tags, not just GOOS and GOARCH, as in // +build nocgo/linux.
In January 2012, CL 5554079 added support for custom build tags (such as appengine), changed slash to comma, and introduced ! for negation.
In March 2013, CL 7794043 added the go1.1 build tag, enabling release-specific file selection. The syntax changed to allow dots in tag names.
Although each of these steps makes sense in isolation, we have arrived at a non-standard boolean expression syntax capable of expressing ANDs of ORs of ANDs of potential NOTs of tags. It is difficult for developers to remember the syntax. For example, two of these three mean the same thing, but which two?
// +build linux,386 // +build linux 386 // +build linux // +build 386
The simple form worked well in the original context, but it has not evolved gracefully. The current richness of expression would be better served by a more familiar syntax.
Surveying the public Go ecosystem for // +build lines in March 2020 turned up a few illustrative apparent bugs that had so far eluded detection. These bugs might have been avoided entirely if developers been working with more familiar syntax.
github.com/streamsets/datacollector-edge
// +build 386 windows,amd64 windows
Confused AND and OR: simplifies to “386 OR windows”.
Apparently intended // +build 386,windows amd64,windows.
// +build 386 !gccgo,amd64 !gccgo,amd64p32 !gccgo
Confused AND and OR: simplifies to “386 OR NOT gccgo”.
Apparently intended:
// +build 386 amd64 amd64p32 // +build !gccgo
// +build go1.12,wasm,js js
Intended meaning unclear; simplifies to just “js”.
gitlab.com/aquachain/aquachain
// +build windows,solaris,nacl nacl solaris windows
Intended (but at least equivalent to) // +build nacl solaris windows.
// +build linux,!amd64 // +build linux,amd64,noasm // +build !go1.9
Unsatisfiable (!amd64 and amd64 can’t both be true).
Apparently intended // +build linux,!amd64 linux,amd64,noasm !go1.9.
Later, in June 2020, Alan Donovan wrote some 64-bit specific code that he annotated with:
//+build linux darwin //+build amd64 arm64 mips64x ppc64x
For the file implementing the generic fallback, he needed the negation of that condition and wrote:
//+build !linux,!darwin //+build !amd64,!arm64,!mips64x,!ppc64x
This is subtly wrong. He correctly negated each line, but repeated lines apply constraints independently, meaning they are ANDed together. To negate the overall meaning, the two lines in the fallback need to be ORed together, meaning they need to be a single line:
//+build !linux,!darwin !amd64,!arm64,!mips64x,!ppc64x
Alan has written a very good book about Go—he is certainly an experienced developer—and still got this wrong. It's all clearly too subtle. Getting ahead of ourselves just a little, if we used a standard boolean syntax, then the first file would have used:
(linux || darwin) && (amd64 || arm64 || mips64x || ppc64x)
and the negation in the fallback would have been easy:
!((linux || darwin) && (amd64 || arm64 || mips64x || ppc64x))
In addition to confusion about syntax, there is also confusion about placement. The documentation (in go doc go/build) explains:
“Constraints may appear in any kind of source file (not just Go), but they must appear near the top of the file, preceded only by blank lines and other line comments. These rules mean that in Go files a build constraint must appear before the package clause.
To distinguish build constraints from package documentation, a series of build constraints must be followed by a blank line.”
Because the search for build constraints stops at the first non-//, non-blank line (usually the Go package statement), this is an ignored build constraint:
package main // +build linux
The syntax even excludes C-style /* */ comments, so this is an ignored build constraint:
/* Copyright ... */ // +build linux package main
Furthermore, to avoid confusion with doc comments, the search stops stops at the last blank line before the non-//, non-blank line, so this is an ignored build constraint (and a doc comment):
// +build linux package main
This is also an ignored build constraint, in an assembly file:
// +build 386 amd64 #include "textflag.h"
Surveying the public Go ecosystem for // +build lines in March 2020 turned up
/* */ comments, usually copyright notices;package declaration.These are small numbers compared to the 110,000 unique files found that contained build constraints, but these are only the ones that slipped through, unnoticed, into the latest public commits. We should expect that there are many more such mistakes corrected in earlier commits or that lead to head-scratching debugging sessions but avoid being committed.
The core idea of the design is to replace the current // +build lines for build tag selection with new //go:build lines that use more familiar boolean expressions. For example, the old syntax
// +build linux // +build 386
would be replaced by the new syntax
//go:build linux && 386
The design also admits //go:build lines in more locations and rejects misplaced //go:build lines.
The key to the design is a smooth transition that avoids breaking Go code.
The next three sections explain these three parts of the design in detail.
The new syntax is given by this grammar, using the notation of the Go spec:
BuildLine = "//go:build" Expr
Expr = OrExpr
OrExpr = AndExpr { "||" AndExpr }
AndExpr = UnaryExpr { "&&" UnaryExpr }
UnaryExpr = "!" UnaryExpr | "(" Expr ")" | tag
tag = tag_letter { tag_letter }
tag_letter = unicode_letter | unicode_digit | "_" | "."
That is, the syntax of build tags is unchanged from its current form, but the combination of build tags is now done with Go’s ||, &&, and ! operators and parentheses. (Note that build tags are not always valid Go expressions, even though they share the operators, because the tags are not always valid identifiers. For example: “go1.1”.)
It is an error for a file to have more than one //go:build line, to eliminate confusion about whether multiple lines are implicitly ANDed or ORed together.
The current search for build constraints can be explained concisely, but it has unexpected behaviors that are difficult to understand, as discussed in the Background section.
It remains useful for both people and programs like the go command not to need to read the entire file to find any build constraints, so this design still ends the search for build constraints at the first non-comment text in the file. However, this design allows placing //go:build constraints after /* */ comments or in doc comments. (Proposal issue 37974, which will ship in Go 1.15, strips those //go: lines out of the doc comments.)
The new rule would be:
“Constraints may appear in any kind of source file (not just Go), but they must appear near the top of the file, preceded only by blank lines and other
//and/* */comments. These rules mean that in Go files a build constraint must appear before the package clause.”
In addition to this more relaxed rule, the design would change gofmt to move misplaced build constraints to valid locations, and it would change the Go compiler and assembler reject misplaced build constraints. This will correct most misplaced constraints automatically and report the others; no misplaced constraint should go unnoticed. (The next section describes the tool changes in more detail.)
A smooth transition is critical for a successful rollout. By the Go release policy, the release of Go 1.N ends support for Go 1.(N-2), but most users will still want to be able to write code that works with both Go 1.(N−1) and Go 1.N. If Go 1.N introduces support for //go:build lines, code that needs to build with the past two releases can’t fully adopt the new syntax until Go 1.(N+1) is released. Publishers of popular dependencies may not realize this, which may lead to breakage in the Go ecosystem. We must make accidental breakage unlikely.
To help with the transition, we envision a plan carried out over three Go releases. For concreteness, we call them Go 1.(N−1), Go 1.N, and Go 1.(N+1). The bulk of the work happens in Go 1.N, with minor preparation in Go 1.(N−1) and minor cleanup in Go 1.(N+1).
Go 1.(N−1) would prepare for the transition with minimal changes. In Go 1.(N−1):
//go:build lines without // +build lines.//go:build lines for file selection.//go:build lines yet.At this point:
//go:build at all.Go 1.N would start the transition. In Go 1.N:
//go:build lines for file selection. If there is no //go:build in a file, then any // +build lines still apply.//go:build without // +build.//go:build too late in the file.Gofmt will move misplaced //go:build and // +build lines to their proper location in the file.Gofmt will format the expressions in //go:build lines using the same rules as for other Go boolean expressions (spaces around all && and || operators).// +build lines, gofmt will add an equivalent //go:build line above them.//go:build and // +build lines, gofmt will consider the //go:build the source of truth and update the // +build lines to match, preserving compatibility with earlier versions of Go. Gofmt will also reject //go:build lines that are deemed too complex to convert into // +build format, although this situation will be rare. (Note the “If” at the start of this bullet. Gofmt will not add // +build lines to a file that only has //go:build.)buildtags check in go vet will add support for //go:build constraints. It will fail when a Go source file contains //go:build and // +build lines with different meanings. If the check fails, one can run gofmt -w.buildtags check will also fail when a Go source file contains //go:build without // +build and its containing module has a go line listing a version before Go 1.N. If the check fails, one can add any // +build line and then run gofmt -w, which will replace it with the correct ones. Or one can bump the go.mod go version to Go 1.N.//go:build and the transition.At this point:
go vet.//go:build and // +build lines will fail Go 1.N go vet.gofmt on save will not fail go vet.//go:build lines will work fine when using only Go 1.N. If such packages are built using Go 1.(N−1), the build will fail, loud and clear.Go 1.(N+1) would complete the transition. In Go 1.(N+1):
go fix will remove // +build stanzas, making sure to leave behind equivalent //go:build lines. The removal only happens when go fix is being run in a module with a go 1.N (or later) line, which is taken as an explicit signal that the developer no longer needs compatibility with Go 1.(N−1). The removal is never done in GOPATH mode, which lacks any such explicit signal.At this point:
go fix will remove all // +build lines from the source tree, leaving behind equivalent, easier-to-read //go:build lines, but only on modules that have set a base requirement of Go 1.N.The motivation is laid out in the Background section above.
The rationale for using Go syntax is that Go developers already understand one boolean expression syntax. It makes more sense to reuse that one than to maintain a second one. No other syntaxes were considered: any other syntax would be a second (well, a third) syntax to learn.
As Michael Munday noted in 2018, these lines are difficult to test, so we would be well served to make them as straightforward as possible to understand. This observation is reinforced by the examples in the introduction.
Tooling will need to keep supporting // +build lines indefinitely, in order to continue to build old code. Similarly, the go vet check that //go:build and // +build lines match when both are present will be kept indefinitely. But the go fix should at least help us remove // +build lines from new versions of code, so that most developers stop seeing them or needing to edit them.
The rationale for disallowing multiple //go:build lines is that the entire goal of this design is to replace implicit AND and OR with explicit && and || operators. Allowing multiple lines reintroduces an implicit operator.
The rationale for the transition is laid out in that section. We don’t want to break Go developers unnecessarily, nor to make it easy for dependency authors to break their dependents.
The rationale for using //go:build as the new prefix is that it matches our now-established convention of using //go: prefixes for directives to the build system or compilers (see also //go:generate, //go:noinline, and so on).
The rationale for introducing a new prefix (instead of reusing // +build) is that the new prefix makes it possible to write files that contain both syntaxes, enabling a smooth transition.
The main alternative to this design is to do nothing at all and simply stick with the current syntax. That is an attractive option, since it avoids going through a transition. On the other hand, the benefit of the clearer syntax will only grow as we get more and more Go developers and more and more Go code, and the transition should be fairly smooth and low-cost.
Go code that builds today will keep building indefinitely, without any changes.
The transition aims to make it difficult to cause new incompatibilities accidentally, with gofmt and go vet working to keep old and new syntaxes in sync during the transition.
Compatibility is also the reason for not providing the go fix that removes // +build lines until Go 1.(N+1) is released: that way, the automated tool that breaks Go 1.(N−1) users is not even available until Go 1.(N−1) is no longer supported.