Authors: Jay Conrod, Daniel Martí
Last Updated: 2020-09-29
Discussion at https://golang.org/issue/40276.
Authors of executables need a simple, reliable, consistent way for users to build and install exectuables in module mode without updating module requirements in the current module's
go get is used to download and install executables, but it‘s also responsible for managing dependencies in
go.mod files. This causes confusion and unintended side effects: for example, the command
go get golang.org/x/tools/gopls builds and installs
gopls. If there’s a
go.mod file in the current directory or any parent, this command also adds a requirement on the module
golang.org/x/tools/gopls, which is usually not intended. When
GO111MODULE is not set,
go get will also run in GOPATH mode when invoked outside a module.
These problems lead authors to write complex installation commands such as:
(cd $(mktemp -d); GO111MODULE=on go get golang.org/x/tools/gopls)
We propose augmenting the
go install command to build and install packages at specific versions, regardless of the current module context.
go install email@example.com
To eliminate redundancy and confusion, we also propose deprecating and removing
go get functionality for building and installing packages.
go install behavior will be enabled when an argument has a version suffix like
go install does not allow version suffixes. When a version suffix is used:
go installruns in module mode, regardless of whether a
go.modfile is present. If
go installreports an error, similar to what
go mod downloadand other module commands do.
go installacts as if no
go.modfile is present in the current directory or parent directory.
cmd) or local directories (
...), it will only match main packages.
go.modfile (if it has one).
go.modfile, it must not contain directives that would cause it to be interpreted differently if the module were the main module. In particular, it must not contain
go install has arguments without version suffixes, its behavior will not change. It will operate in the context of the main module. If run in module mode outside of a module,
go install will report an error.
With these restrictions, users can install executables using consistent commands. Authors can provide simple installation instructions without worrying about the user's working directory.
With this change,
go install would overlap with
go get even more, so we also propose deprecating and removing the ability for
go get to install packages.
go getis invoked outside a module or when
go getis invoked without the
-dflag with arguments matching one or more main packages,
go getwould print a deprecation warning recommending an equivalent
go getwould no longer build or install packages. The
-dflag would be enabled by default. Setting
-d=falsewould be an error. If
go getis invoked outside a module, it would print an error recommending an equivalent
# Install a single executable at the latest version $ go install example.com/cmd/tool@latest # Install multiple executables at the latest version $ go install example.com/cmd/...@latest # Install at a specific version $ go install firstname.lastname@example.org
go install is used for building and installing packages within the context of the main module.
go install reports an error when invoked outside of a module or when given arguments with version queries like
go get is used both for updating module dependencies in
go.mod and for building and installing executables.
go get also works differently depending on whether it's invoked inside or outside of a module.
These overlapping responsibilities lead to confusion. Ideally, we would have one command (
go install) for installing executables and one command (
go get) for changing dependencies.
go get is invoked outside a module in module mode (with
GO111MODULE=on), its primary purpose is to build and install executables. In this configuration, there is no main module, even if only one module provides packages named on the command line. The build list (the set of module versions used in the build) is calculated from requirements in
go.mod files of modules providing packages named on the command line.
exclude directives from all modules are ignored. Vendor directories are also ignored.
go get is invoked inside a module, its primary purpose is to update requirements in
-d flag is often used, which instructs
go get not to build or install packages. Explicit
go build or
go install commands are often better for installing tools when dependency versions are specified in
go.mod and no update is desired. Like other build commands,
go get loads the build list from the main module‘s
go.mod file, applying any
exclude directives it finds there.
exclude directives in other modules’
go.mod files are never applied. Vendor directories in the main module and in other modules are ignored; the
-mod=vendor flag is not allowed.
The motivation for the current
go get behavior was to make usage in module mode similar to usage in GOPATH mode. In GOPATH mode,
go get would download repositories for any missing packages into
$GOPATH/src, then build and install those packages into
go get -u would update repositories to their latest versions.
go get -d would download repositories without building packages. In module mode,
go get works with requirements in
go.mod instead of repositories in
In module mode, the
go command typically fetches dependencies from a proxy. Modules are distributed as zip files that contain sources for specific module versions. Even when
go connects directly to a repository instead of a proxy, it still generates zip files so that builds work consistently no matter how modules are fetched. Those zip files don't contain nested modules or vendor directories.
go get cloned repositories, it would work very differently from other build commands. That causes several problems:
gocommand to support a new build mode.
Vendor directories are not included in module zip files. Since they‘re not present when a module is downloaded, there’s no way to build with them.
We don't plan to include vendor directories in zip files in the future either. Changing the set of files included in module zip files would break
replace example.com/sibling => ../sibling
replace directives with a directory path on the right side can‘t be used because the directory must be outside the module. These directories can’t be present when the module is downloaded, so there's no way to build with them.
replace example.com/mod v1.0.0 => example.com/fork v1.0.1-bugfix
It is technically possible to apply these directives. If we did this, we would still want some restrictions. First, an error would be reported if more than one module provided packages named on the command line: we must be able to identify a main module. Second, an error would be reported if any directory
replace directives were present: we don't want to introduce a new configuration where some
replace directives are applied but others are silently ignored.
However, there are two reasons to avoid applying
replace directives at all.
replace directives would create inconsistency for users inside and outside a module. When a package is built within a module with
go build or
go install, only
replace directives from the main module are applied, not the module providing the package. When a package is built outside a module with
go get, no
replace directives are applied. If
go install applied
replace directives from the module providing the package, it would not be consistent with the current behavior of any other build command. To eliminate confusion about whether
replace directives are applied, we propose that
go install reports errors when encountering them.
go install applied
replace directives, it would take power away from developers that depend on modules that provide tools. For example, suppose the author of a popular code generation tool
gogen forks a dependency
genutil to add a feature. They add a
replace directive pointing to their fork of
genutil while waiting for a PR to merge. A user of
gogen wants to track the version they use in their
go.mod file to ensure everyone on their team uses a consistent version. Unfortunately, they can no longer build
go install because the
replace is ignored. The author of
gogen might instruct their users to build with
go install, but then users can‘t track the dependency in their
go.mod file, and they can’t apply their own
replace directives to upgrade or fix other transitive dependencies. The author of
gogen could also instruct their users to copy the
replace directive, but this may conflict with other
replace directives, and it may cause similar problems for users further downstream.
go install ignored
replace directives, it would be consistent with the current behavior of
go get when invoked outside a module. However, in #30515 and related discussions, we found that many developers are surprised by that behavior.
It seems better to be explicit that
replace directives are only applied locally within a module during development and not when users build packages from outside the module. We‘d like to encourage module authors to release versions of their modules that don’t rely on
replace directives so that users in other modules may depend on them easily.
If this behavior turns out not to be suitable (for example, authors prefer to keep
replace directives in
go.mod at release versions and understand that they won't affect users), then we could start ignoring
replace directives in the future, matching current
go get behavior.
Because there is no main module,
go install will not use a
go.sum file to authenticate any downloaded module or
go.mod file. The
go command will still use the checksum database (sum.golang.org) to authenticate downloads, subject to privacy settings. This is consistent with the current behavior of
go get: when invoked outside a module, no
go.sum file is used.
go install command requires that only one module may provide packages named on the command line, so it may be logical to use that module‘s
go.sum file to verify downloads. This avoids a problem in #28802, a related proposal to verify downloads against all
go.sum files in dependencies: the build can’t be broken by one bad
go.sum file in a dependency.
However, using the
go.sum from the module named on the command line only provides a marginal security benefit: it lets us authenticate private module dependencies (those not available to the checksum database) when the module on the command line is public. If the module named on the command line is private or if the checksum database isn‘t used, then we can’t authenticate the download of its content (including the
go.sum file), and we must trust the proxy. If all dependencies are public, we can authenticate all downloads without
If no version suffix were required when
go install is invoked outside a module, then the meaning of the command would depend on whether the user's working directory is inside a module. For example:
go install golang.org/x/tools/gopls
When invoked outside of a module, this command would run in
GOPATH mode, unless
GO111MODULE=on is set. In module mode, it would install the latest version of the executable.
When invoked inside a module, this command would use the main module's
go.mod file to determine the versions of the modules needed to build the package.
We currently have a similar problem with
go get. Requiring the version suffix makes the meaning of a
go install command unambiguous.
To install the latest version of an executable, the two commands below would be equivalent:
go install -g golang.org/x/tools/gopls go install golang.org/x/tools/gopls@latest
-g flag has the advantage of being shorter for a common use case. However, it would only be useful when installing the latest version of a package, since
-g would be implied by any version suffix.
@latest suffix is clearer, and it implies that the command is time-dependent and not reproducible. We prefer it for those reasons.
go install part of this proposal only applies to commands with version suffixes on each argument.
go install reports an error for these, and this proposal does not recommend changing other functionality of
go install, so that part of the proposal is backward compatible.
go get part of this proposal recommends deprecating and removing functionality, so it's certainly not backward compatible.
go get -d commands will continue to work without modification though, and eventually, the
-d flag can be dropped.
Parts of this proposal are more strict than is technically necessary (for example, requiring one module, forbidding
replace directives). We could relax these restrictions without breaking compatibility in the future if it seems expedient. It would be much harder to add restrictions later.
An initial implementation of this feature was merged in CL 254365. Please try it out!
The behavior with respect to
replace directives was discussed extensively before this proposal was written. There are three potential behaviors:
replacedirectives in all modules. This would be consistent with other module-aware commands, which only apply
replacedirectives from the main module (defined in the current directory or a parent directory).
go install pkg@versionignores the current directory and any
go.modfile that might be present, so there is no main module.
replacedirectives from it. Report errors for directory
replacedirectives. This is feasible, but it may have wider ecosystem effects; see “Why can't module
replacedirectives be used?” above.
replacedirectives it contains. This is the behavior currently proposed.
Most people involved in this discussion have advocated for either (1) or (2). The behavior in (3) is a compromise. If we find that the behavior in (1) is strictly better than (2) or vice versa, we can switch to that behavior from (3) without an incompatible change. Additionally, (3) eliminates ambiguity about whether
replace directives are applied for users and module authors.
Note that applying directory
replace directives is not considered here for the reasons in “Why can't directory
replace directives be used?”.
replace directives from different modules would conflict, and that would make dependency management harder for most users.
For example, consider a case where two dependencies replace the same module with different forks.
// in example.com/mod/a replace example.com/mod/c => example.com/fork-a/c v1.0.0 // in example.com/mod/b replace example.com/mod/c => example.com/fork-b/c v1.0.0
Another conflict would occur where two dependencies pin different versions of the same module.
// in example.com/mod/a replace example.com/mod/c => example.com/mod/c v1.1.0 // in example.com/mod/b replace example.com/mod/c => example.com/mod/c v1.2.0
To avoid the possibility of conflict, the
go command ignores
replace directives in modules other than the main module.
Modules are intended to scale to a large ecosystem, and in order for upgrades to be safe, fast, and predictable, some rules must be followed, like semantic versioning and import compatibility. Not relying on
replace is one of these rules.
replace is useful in several situations for local or short-term development, for example:
github.com/golang/lint. Many of these problems should be fixed by lazy module loading (#36460).
replace is safe to use in a module that is not depended on by other modules. It‘s also safe to use in revisions that aren’t depended on by other modules.
replacedirective is just meant for temporary local development by one person, avoid checking it in. The
-modfileflag may be used to build with an alternative
go.modfile. See also #26640 a feature request for a
go.mod.localfile containing replacements and other local modifications.
replacedirective must be checked in to fix a short-term problem, ensure at least one release or pre-release version is tagged before checking it in. Don‘t tag a new release version with
replacechecked in (pre-release versions may be okay, depending on how they’re used). When the
gocommand looks for a new version of a module (for example, when running
go getwith no version specified), it will prefer release versions. Tagging versions lets you continue development on the main branch without worrying about users fetching arbitrary commits.
replacedirective must be checked in to solve a long-term problem, consider solutions that won't cause issues for dependent modules. If possible, tag versions on a release branch with
go install command will build an executable with the same set of module versions on every invocation if both the following conditions are true:
go install email@example.com.
go.modfile of the module providing the executable. If the executable only imports standard library packages or packages from its own module, no
go.modfile is necessary.
An executable may not be bit-for-bit reproducible for other reasons. Debugging information will include system paths (unless
-trimpath is used). A package may import different packages on different platforms (or may not build at all). The installed Go version and the C toolchain may also affect binary reproducibility.
go install will report an error, as
go get already does.
This sometimes happens when two modules depend on each other, and releases are not tagged on the main branch. A command like
go get example.com/m@master will resolve
@master to a pseudo-version lower than any release version. The
go.mod file at that pseudo-version may transitively depend on a newer release version.
go get reports an error in this situation. In general,
go get reports an error when command line arguments different versions of the same module, directly or indirectly.
go install doesn't support this yet, but this should be one of the conditions checked when running with version suffix arguments.
In this proposal,
go install would report errors for
replace directives in the module providing packages named on the command line.
go get ignores these, but the behavior may still surprise module authors and users. I've tried to estimate the impact on the existing set of open source modules.
mainpackages that Russ Cox built during an earlier study.
go getwould fetch.
go.modfile. 4,519 were left.
go.modfiles using module
replaceonly, I tried to classify why
replacewas used. A module may have multiple
replacedirectives and multiple classifications, so the percentages below don't add to 100%.
replaceas a soft fork, for example, to point to a bug fix PR instead of the original module.
replaceto pin a specific version of a dependency (the module path is the same on both sides).
replaceto rename a dependency that was imported with another name, for example, replacing
github.com/golang/lintwith the correct path,
golang.org/xrepos with their
replaceto bypass semantic import versioning.
k8s.iomodules. Kubernetes has used
replaceto bypass MVS, and dependent modules have been forced to do the same.
replacedirectives I couldn't automatically classify. The ones I looked at seemed to mostly be forks or pins.
The modules I‘m most concerned about are those that use
replace as a soft fork while submitting a bug fix to an upstream module; other problems have other solutions that I don’t think we need to design for here. Modules using soft fork replacements are about 4% of the the modules with
go.mod files I sampled (165 / 4519). This is a small enough set that I think we should move forward with the proposal above.