Go 1.11 includes preliminary support for versioned modules as proposed here.
Go modules are an experimental opt-in feature in Go 1.11, with the hope of incorporating feedback and finalizing the feature for Go 1.12. Even though the details may change, future releases will support modules defined using Go 1.11 or vgo.
vgo tool, but on July 12, 2018 support for versioned modules landed in the main Go repository (announcement thread).vgo.go mod commands. See FAQ below for an overview of these changes.NOTE: Some issues you might experience:
The remaining content on this page is organized as follows:
To use modules, you currently have three install options:
master branch.vgo binary from the vgo subrepository (and replace go with vgo in the commands below).Once installed, you can then activate module support in one of three ways:
go command in a directory outside of the $GOPATH/src tree, with a valid go.mod file in the current directory or any parent of it and the environment variable GO111MODULE unset (or explicitly set to auto).go command with GO111MODULE=on in the command environment.vgo (if you have installed vgo from the subrepository).In addition to trying modules-specific workflows:
These sections provide a high-level introduction to the main new concepts. For more details and rationale, please see the official proposal document, this 40 minute introductory video by Russ Cox describing the philosophy behind the design, or the more detailed initial vgo blog series.
A module is a collection of related Go packages that are versioned together as a single unit. Most often, a single version-control repository corresponds exactly to a single module, but alternatively, a single version-control repository can hold multiple modules.
Modules record precise dependency requirements and create reproducible builds.
Modules must be semantically versioned in the form v(major).(minor).(patch), such as v0.1.0, v1.2.3, or v3.0.1. The leading v is required. If using Git, tag released commits with their versions. Public and private module repositories and proxies are becoming available (see for example FAQ below).
A module is defined by a tree of Go source files with a go.mod file in the tree's root directory. Module source code may be located outside of GOPATH.
All of the packages in a module share a common prefix -- the module path. The go.mod file defines the module path via the module directive. For example, if you are defining a module for two packages example.com/my/thing/foo and example.com/my/thing/bar, the first line in your go.mod file typically would be module example.com/my/thing.
Module files may include comments and will look familiar to a Go programmer. Here is an example go.mod file:
module github.com/my/module/v3
require (
github.com/some/dependency v1.2.3
github.com/another/dependency v0.1.0
github.com/additional/dependency/v4 v4.0.0
)
There are four directives: module, require, exclude, replace.
exclude and replace directives only operate on the current (“main”) module. exclude and replace directives in modules other than the main module are ignored when building the main module. The replace and exclude statements therefore allow the main module complete control over its own build, without also being subject to complete control by dependencies. (TODO: show example exclude and replace directives and/or FAQ on when to use them).
In Go source code, packages are imported using the full path including the module, for example:
import "example.com/my/module/v2/pkg/foo" to import foo from the v2 version of module example.com/my/module.If you add a new import to your source code that is not yet covered by a require in go.mod, any go command run (e.g., ‘go build’) will automatically look up the proper module and add the highest version of that new direct dependency to your module‘s go.mod as a require directive. For example, if your new import corresponds to dependency M whose latest tagged release version is v1.2.3, your module’s go.mod will end up with require M v1.2.3, which indicates module M is a dependency with allowed version >= v1.2.3 (and < v2, given v2 is considered incompatible with v1).
The minimal version selection algorithm is used to select the versions of all modules used in a build. For each module in a build, the version selected by minimal version selection is always the semantically highest of the versions explicitly listed by a require directive in the main module or one of its dependencies.
As an example, if your module depends on module A which has a require D v1.0.0, and your module also depends on module B which has a require D v1.1.1, then minimal version selection would choose v1.1.1 of D to include in the build (given it is the highest listed require version). This selection of v1.1.1 remains consistent even if some time later a v1.2.0 of D becomes available. This is an example of how the modules system provides 100% reproducible builds. When ready, the module author or user might choose to upgrade to the latest available version of D or choose an explicit version for D.
For a brief rationale and overview of the minimal version selection algorithm, see the “High Fidelity Builds” section of the official proposal, or see the more detailed vgo blog series.
To see a list of the selected module versions (including indirect dependencies), use go list -m all.
See also the “How to Upgrade and Downgrade Dependencies” section below and the “How are versions marked as incompatible?” FAQ below.
For many years, the official Go FAQ has included this advice on package versioning:
“Packages intended for public use should try to maintain backwards compatibility as they evolve. The Go 1 compatibility guidelines are a good reference here: don't remove exported names, encourage tagged composite literals, and so on. If different functionality is required, add a new name instead of changing an old one. If a complete break is required, create a new package with a new import path.”
The last sentence is especially important. With Go 1.11 modules, that advice is formalized into the import compatibility rule:
“If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.”
Recall semantic versioning requires a major version change when a v1 or higher package makes a backwards incompatible change. The result of following both the import compatibility rule and semantic versioning is called semantic import versioning.
For code opting in to Go modules, the net result is the following rules:
v1.2.3).go.mod file (e.g., module example.com/my/mod/v2) and the package import path (e.g., import "example.com/my/mod/v2/foo").In general, packages with different import paths (e.g., due to different major versions) are different packages. Thus example.com/my/mod/foo is a different package than example.com/my/mod/v2/foo, and both may be imported in a single build, which among other benefits helps with diamond dependency problems and also allows a v1 module to be implemented in terms of its v2 replacement or vice versa.
See the “Module compatibility and semantic versioning” section of the tip documentation for more details.
This section so far has been focused on code that opts in to modules. However, putting major versions in import paths for v2+ modules could create incompatibilities with older versions of Go, or with code that has not yet opted in to modules. To help with this, Go versions 1.9.7+ and 1.10.3+ have been updated so that code built with those releases can properly consume v2+ modules without requiring modification of pre-existing code.
There are two ways to release a v2 or higher module version. Using the example of creating a v2.0.0 release, the two options are:
go.mod file to include a /v2 at the end of the module path. Tag the release with v2.0.0.v2.*.* commits on a separate v2 branch.v2.0.0 on master, that is a viable option, but consider creating a v1 branch for any future v1 bug fixes.v2 subdirectory (e.g., my/module/v2) and place a new go.mod file in that subdirectory. The module path must end with /v2. Copy or move the code into the v2 subdirectory. Tag the release with v2.0.0.The behavior of how modules interact with existing pre-module packages with v2+ tags has evolved over the prototype and beta processes; an important related recent change was issue 26238, which substantially improved the go command behavior for existing packages with v2+ tags.
To create a go.mod for an existing project:
Navigate to the root of the module's source tree and activate module mode in the go command:
$ cd $GOPATH/src/<project path> $ export GO111MODULE=on
or:
$ cd <project path outside $GOPATH/src>
Create the initial module definition and write it to the go.mod file:
# if using Go 1.11, 'master', or the latest 'vgo', use the newer form: $ go mod init # in go1.11beta2 and earlier, the older form was: $ go mod -init
This step converts from any existing dep Gopkg.lock file or from any of the other nine total supported dependency formats, adding require statements to match the existing configuration.
If go mod cannot automatically determine an appropriate module path (e.g., if running outside of VCS), or if you need to otherwise override that path, you can supply the module path as follows:
# if using Go 1.11, 'master', or the latest 'vgo', use the newer form: $ go mod init example.com/my/module/v2 # in go1.11beta2 and earlier, the older form was: $ go mod -init -module example.com/my/module/v2
Build the module. This will automatically add missing or unconverted dependencies as needed to satisfy imports for this particular build invocation:
$ go build ./...
Test the module as configured to ensure that it works with the selected versions:
$ go test ./...
(Optional) Run the tests for your module plus the tests for all direct and indirect dependencies to check for incompatibilities:
$ go test all
Prior to tagging a release, see the “How to Prepare for a Release” section below.
For more information on all of these topics, the primary entry point to the official modules documentation is available on tip.golang.org.
Day-to-day adding, removing, upgrading, and downgrading of dependencies should be done using ‘go get’, which will automatically update the go.mod file.
In addition, go commands like ‘go build’, ‘go test’, or even ‘go list’ will automatically add new dependencies as needed to satisfy imports (updating go.mod and downloading the new dependencies).
To upgrade to the latest version for all transitive dependencies of the current module:
go get -u to use the latest minor or patch releasesgo get -u=patch to use the latest patch releasesTo upgrade or downgrade to a more specific version, ‘go get’ allows version selection to be overridden by adding an @version suffix or “module query” to the package argument, such as go get github.com/gorilla/mux@v1.6.2, go get github.com/gorilla/mux@e3702bed2, or go get github.com/gorilla/mux@'<v1.6.2'.
go get github.com/gorilla/mux obtains the latest version with a semver tag. Using a branch name such as go get github.com/gorilla/mux@master is one way to obtain the latest commit regardless of whether or not it has a semver tag.
In general, module queries that do not resolve to a semver tag will be recorded as pseudo-versions in the go.mod file.
Modules are capable of consuming packages that have not yet opted into modules. Modules can also consume packages that do not yet have any proper semver tags (in which case they will be recorded using pseudo-versions in go.mod).
See the “Module-aware go get” and “Module queries” sections of the tip documentation for more information on the topics here.
After upgrading or downgrading any dependencies, you may then want to run the tests again for all packages in your build (including direct and indirect dependencies) to check for incompatibilities:
$ go test all
Best practices for creating a release of a module are expected to emerge as part of the initial modules experiment. Many of these might end up being automated by a future ‘go release’ tool.
Some current suggested best practices to consider prior to tagging a release:
Run go mod tidy (or if running go1.11beta2 or earlier: go mod -sync) to possibly prune any extraneous requirements (as described here) and also ensure your current go.mod reflects all possible build tags/OS/architecture combinations (as described here).
go build and go test will not remove dependencies from go.mod that are no longer required and only update go.mod based on the current build invocation's tags/OS/architecture.Run go test all to test your module (including running the tests for your direct and indirect dependencies) as a way of validating that the currently selected packages versions are compatible.
go test all has been re-defined to be more useful to include all the packages in the current module, plus all the packages they depend on through a sequence of one or more imports, while excluding packages that don't matter in the current module.Typically the go.sum file should be checked in along with the go.mod file.
go.sum contains the expected cryptographic checksums of the content of specific module versions.go.sum.go mod verify checks that the on-disk cached copies of module downloads still match the entries in go.sum.go.sum is not a lock file as used in some alternative dependency management systems. (go.mod provides enough information for reproducible builds).If you are releasing a v2 or higher module, please also see the related considerations in the “Semantic Import Versioning” section above.
go help modules for more about modules. (This is the main entry point for modules topics via go help)go help mod for more about the go mod command.go help module-get for more about the behavior of go get when in module-aware mode.go help goproxy for more about the module proxy, including a pure file-based option via a file:/// URL.vgo by Russ Cox (first posted February 20, 2018)vgo blog series, along with some of the history and process behind the proposalAs part of the proposal, prototype, and beta processes, there have been over 400 issues created by the overall community. Please continue to supply feedback.
Here is a partial list of some of the larger changes and improvements, almost all of which were primarily based on feedback from the community:
go get -u=patch to update all transitive dependencies to the latest available patch-level versions on the same minor version (discussion, documentation)The require directive allows any module to declare that it should be built with version >= x.y.z of a dependency D (which may be specified due to incompatibilities with version < x.y.z of module D). Empirical data suggests this is the dominant form of constraints used in dep and cargo. In addition, the top-level module in the build can exclude specific versions of dependencies or replace other modules with different code. See the full proposal for more details and rationale.
One of the key goals of the versioned modules proposal is to add a common vocabulary and semantics around versions of Go code for both tools and developers. This lays a foundation for future capabilities to declare additional forms of incompatibilities, such as possibly:
vgo blog seriesIn general, modules are opt-in for Go 1.11, so by design old behavior is preserved by default.
Summarizing when you get the old 1.10 status quo behavior vs. the new opt-in modules-based behavior:
go.mod — defaults to modules behaviorauto — default behavior aboveon — force module support on regardless of directory locationoff — force module support off regardless of directory locationgo get example.com/cmd fail with error cannot find main module when run with GO111MODULE=on?In general, when running a command like go get in module-aware mode, the resulting behavior depends on which module you are currently “in” as the current module, which means it depends on where you are in your filesystem hierarchy when you execute the go get command.
One aspect that might catch people by surprise is if the GO111MODULE environment variable is explicitly set to GO111MODULE=on, then go get example.com/cmd requires a go.mod file and hence will fail if run outside of a module (that is, if run outside of a file tree rooted by a go.mod). The current error message is:
go: cannot find main module; see 'go help modules'
One solution is to disable modules temporarily and enable Go 1.10 behavior, such as:
$ GO111MODULE=off go get example.com/cmd
Setting GO111MODULE=auto (or leaving GO111MODULE unset) will also avoid this particular error:
$ GO111MODULE=auto go get example.com/cmd
However, note that using path@version syntax such as go get example.com/cmd@v1.2.3 is not supported when running outside of a module, where the current error is: go: cannot use path@version syntax in GOPATH mode
The primary issue is that modules are currently opt-in, and a full solution will likely wait until GO111MODULE=on becomes the default behavior. See #24250 for more discussion, including this comment from there:
This isn‘t blocking Go 1.11. The main reason being that if GO111MODULE=auto (the default) then we can’t reasonably do anything with modules when outside a go.mod tree. That's the opt-in. And inside a go.mod tree this already works.
If you want to use a go.mod to track globally installed tools, one perhaps less common approach could be to create a module whose entire purpose is just to track installed tools as described in more detail here.
If instead you want to track the tools required by a specific module, see the next FAQ.
If you:
go.mod filethen one currently recommended approach is to add a tools.go file to your module with a // +build tools build constraint as shown in this comment in #25922.
The brief rationale (also from #25922):
I think the tools.go file is in fact the best practice for tool dependencies, certainly for Go 1.11.
I like it because it does not introduce new mechanisms.
It simply reuses existing ones.
go mod commands changed recently in go1.11beta3?As of go1.11beta3, there has been a significant change for the go mod commands. See https://tip.golang.org/cmd/go/#hdr-Module_maintenance as well as two snippets from the CL briefly covering the rationale and the list of new vs. old commands:
The current "go mod" command does too many things. It looks like "everything you might want to do with modules" which causes people to think all module operations go through "go mod", which is the opposite of the seamless integration we're going for. In particular too many people think "go mod -require" and "go get" are the same.
and:
split "go mod" into multiple subcommands: go mod edit # old go mod -require ... go mod fix # old go mod -fix go mod graph # old go mod -graph go mod init # old go mod -init go mod tidy # old go mod -sync go mod vendor # old go mod -vendor go mod verify # old go mod -verify Splitting out the individual commands makes both the docs and the implementations dramatically easier to read. It simplifies the command lines (go mod -init -module m is now 'go mod init m') and allows command-specific flags.
In response to a question “k8s does minor releases but changes the Go API in each minor release. How would people handle importing k8s as a vendored project via vgo?”, Russ Cox made the following comment:
I don't fully understand the k8s dev cycle etc, but I think generally the k8s team needs to decide/confirm what they intend to guarantee to users about stability and then apply version numbers accordingly to express that.
- To make a promise about API compatibility (which seems like the best user experience!) then start doing that and use 1.X.Y.
- To have the flexibility to make backwards-incompatible changes in every release but allow different parts of a large program to upgrade their code on different schedules, meaning different parts can use different major versions of the API in one program, then use X.Y.0, along with import paths like k8s.io/client/vX/foo.
- To make no promises about API compatible and also require every build to have only one copy of the k8s libraries no matter what, with the implied forcing of all parts of a build to use the same version even if not all of them are ready for it, then use 0.X.Y.
On a related note, Kubernetes has some atypical build approaches (currently including custom wrapper scripts on top of godep), and hence Kubernetes is an imperfect example for many other projects, but it will likely be an interesting example as Kubernetes moves towards adopting Go 1.11 modules.
Publicly hosted “always on” immutable module repositories and optional privately hosted proxies and repositories are becoming available.
For example:
Note that you are not required to run a proxy. Rather, the go tooling in 1.11 has added optional proxy support via GOPROXY to enable more enterprise use cases (such as greater control), and also to better handle situations such as “GitHub is down” or people deleting GitHub repositories.