Using Go Modules
19 Mar 2019
Tags: tools, versioning

Tyler Bui-Palsulich

Eno Compton

* Introduction

Go 1.11 and 1.12 include preliminary
[[https://golang.org/doc/go1.11#modules][support for modules]],
Go’s
[[https://blog.golang.org/versioning-proposal][new dependency management system]]
that makes dependency version information explicit
and easier to manage.
This blog post is an introduction to the basic operations needed
to get started using modules.
A followup post will cover releasing modules for others to use.

A module is a collection of
[[https://golang.org/ref/spec#Packages][Go packages]]
stored in a file tree with a `go.mod` file at its root.
The `go.mod` file defines the module’s _module_path_,
which is also the import path used for the root directory,
and its _dependency_requirements_,
which are the other modules needed for a successful build.
Each dependency requirement is
written as a module path and a specific
[[http://semver.org/][semantic version]].

As of Go 1.11, the go command enables the use of modules
when the current directory or any parent directory has a `go.mod`,
provided the directory is _outside_ `$GOPATH/src`.
(Inside `$GOPATH/src`, for compatibility, the go command
still runs in the old GOPATH mode, even if a `go.mod` is found.
See the
[[https://golang.org/cmd/go/#hdr-Preliminary_module_support][go command documentation]]
for details.)
Starting in Go 1.13, module mode will be the default for all development.

This post walks through a sequence of common operations
that arise when developing Go code with modules:


- Creating a new module.
- Adding a dependency.
- Upgrading dependencies.
- Adding a dependency on a new major version.
- Upgrading a dependency to a new major version.
- Removing unused dependencies.

* Creating a new module

Let's create a new module.

Create a new, empty directory somewhere outside `$GOPATH/src`,
`cd` into that directory, and then create a new source file, `hello.go`:

	package hello

	func Hello() string {
		return "Hello, world."
	}

Let's write a test, too, in `hello_test.go`:

	package hello

	import "testing"

	func TestHello(t *testing.T) {
		want := "Hello, world."
		if got := Hello(); got != want {
			t.Errorf("Hello() = %q, want %q", got, want)
		}
	}

At this point, the directory contains a package, but not a module,
because there is no `go.mod` file.
If we were working in `/home/gopher/hello` and ran `go`test` now,
we'd see:

	$ go test
	PASS
	ok  	_/home/gopher/hello	0.020s
	$

The last line summarizes the overall package test.
Because we are working outside `$GOPATH`
and also outside any module,
the `go` command knows no import path for
the current directory and makes up a fake one based
on the directory name: `_/home/gopher/hello`.

Let's make the current directory the root of a module
by using `go`mod`init` and then try `go`test` again:

	$ go mod init example.com/hello
	go: creating new go.mod: module example.com/hello
	$ go test
	PASS
	ok  	example.com/hello	0.020s
	$

Congratulations! You’ve written and tested your first module.

The `go`mod`init` command wrote a `go.mod` file:

	$ cat go.mod
	module example.com/hello

	go 1.12
	$

The `go.mod` file only appears in the root of the module.
Packages in subdirectories have import paths consisting of
the module path plus the path to the subdirectory.
For example, if we created a subdirectory `world`,
we would not need to (nor want to) run `go`mod`init` there.
The package would automatically be recognized as part of the
`example.com/hello` module, with import path
`example.com/hello/world`.

* Adding a dependency

The primary motivation for Go modules was to improve the
experience of using (that is, adding a dependency on)
code written by other developers.

Let's update our `hello.go` to import `rsc.io/quote`
and use it to implement `Hello`:

	package hello

	import "rsc.io/quote"

	func Hello() string {
		return quote.Hello()
	}

Now let’s run the test again:

	$ go test
	go: finding rsc.io/quote v1.5.2
	go: downloading rsc.io/quote v1.5.2
	go: extracting rsc.io/quote v1.5.2
	go: finding rsc.io/sampler v1.3.0
	go: finding golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
	go: downloading rsc.io/sampler v1.3.0
	go: extracting rsc.io/sampler v1.3.0
	go: downloading golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
	go: extracting golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
	PASS
	ok  	example.com/hello	0.023s
	$


The `go` command resolves imports by using the specific
dependency module versions listed in `go.mod`.
When it encounters an `import` of a package not provided
by any module in `go.mod`, the `go` command automatically
looks up the module containing that package and adds it to
`go.mod`, using the latest version.
(“Latest” is defined as the
latest tagged stable (non-[[https://semver.org/#spec-item-9][prerelease]]) version,
or else the latest tagged prerelease version,
or else the latest untagged version.)
In our example, `go`test` resolved the new import `rsc.io/quote`
to the module `rsc.io/quote`v1.5.2`.
It also downloaded two dependencies used by `rsc.io/quote`,
namely `rsc.io/sampler` and `golang.org/x/text`.
Only direct dependencies are recorded in the `go.mod` file:

	$ cat go.mod
	module example.com/hello

	go 1.12

	require rsc.io/quote v1.5.2
	$

A second `go`test` command will not repeat this work,
since the `go.mod` is now up-to-date and the downloaded
modules are cached locally (in `$GOPATH/pkg/mod`):

	$ go test
	PASS
	ok  	example.com/hello	0.020s
	$


Note that while the `go` command makes adding a new dependency
quick and easy, it is not without cost.
Your module now literally _depends_ on the new dependency
in critical areas such as correctness, security, and proper licensing,
just to name a few.
For more considerations, see Russ Cox's blog post,
“[[https://research.swtch.com/deps][Our Software Dependency Problem]].”

As we saw above, adding one direct dependency often
brings in other indirect dependencies too.
The command `go`list`-m`all` lists the current module
and all its dependencies:

	$ go list -m all
	example.com/hello
	golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c
	rsc.io/quote v1.5.2
	rsc.io/sampler v1.3.0
	$

In the `go`list` output, the current module,
also known as the _main_module_,
is always the first line,
followed by dependencies sorted by module path.

The `golang.org/x/text` version `v0.0.0-20170915032832-14c0d48ead0c`
is an example of a
[[https://golang.org/cmd/go/#hdr-Pseudo_versions][pseudo-version]],
which is the `go` command's version syntax
for a specific untagged commit.

In addition to `go.mod`, the `go` command
maintains a file named `go.sum` containing
the expected [[https://golang.org/cmd/go/#hdr-Module_downloading_and_verification][cryptographic hashes]] of the content of specific module versions:

	$ cat go.sum
	golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZO...
	golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:Nq...
	rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3...
	rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPX...
	rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/Q...
	rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9...
	$

The `go` command uses the `go.sum` file to ensure that
future downloads of these modules retrieve the same bits
as the first download,
to ensure the modules your project depends on
do not change unexpectedly,
whether for malicious, accidental, or other reasons.
Both `go.mod` and `go.sum` should be checked into version control.

* Upgrading dependencies

With Go modules, versions are referenced with semantic version tags.
A semantic version has three parts: major, minor, and patch.
For example, for `v0.1.2`, the major version is 0, the minor version is 1,
and the patch version is 2.
Let's walk through a couple minor version upgrades.
In the next section, we’ll consider a major version upgrade.

From the output of `go`list`-m`all`,
we can see we're using an untagged version of `golang.org/x/text`.
Let's upgrade to the latest tagged version and test that everything still works:

	$ go get golang.org/x/text
	go: finding golang.org/x/text v0.3.0
	go: downloading golang.org/x/text v0.3.0
	go: extracting golang.org/x/text v0.3.0
	$ go test
	PASS
	ok  	example.com/hello	0.013s
	$

Woohoo! Everything passes.
Let's take another look at `go`list`-m`all` and the `go.mod` file:

	$ go list -m all
	example.com/hello
	golang.org/x/text v0.3.0
	rsc.io/quote v1.5.2
	rsc.io/sampler v1.3.0
	$ cat go.mod
	module example.com/hello

	go 1.12

	require (
		golang.org/x/text v0.3.0 // indirect
		rsc.io/quote v1.5.2
	)
	$

The `golang.org/x/text` package has been upgraded to the latest tagged version (`v0.3.0`).
The `go.mod` file has been updated to specify `v0.3.0` too.
The `indirect` comment indicates a dependency is not used directly
by this module, only indirectly by other module dependencies.
See `go`help`modules` for details.

Now let's try upgrading the `rsc.io/sampler` minor version.
Start the same way, by running `go`get` and running tests:

	$ go get rsc.io/sampler
	go: finding rsc.io/sampler v1.99.99
	go: downloading rsc.io/sampler v1.99.99
	go: extracting rsc.io/sampler v1.99.99
	$ go test
	--- FAIL: TestHello (0.00s)
	    hello_test.go:8: Hello() = "99 bottles of beer on the wall, 99 bottles of beer, ...", want "Hello, world."
	FAIL
	exit status 1
	FAIL	example.com/hello	0.014s
	$

Uh, oh! The test failure shows that the
latest version of `rsc.io/sampler` is incompatible with our usage.
Let's list the available tagged versions of that module:

	$ go list -m -versions rsc.io/sampler
	rsc.io/sampler v1.0.0 v1.2.0 v1.2.1 v1.3.0 v1.3.1 v1.99.99
	$

We had been using v1.3.0; v1.99.99 is clearly no good.
Maybe we can try using v1.3.1 instead:

	$ go get rsc.io/sampler@v1.3.1
	go: finding rsc.io/sampler v1.3.1
	go: downloading rsc.io/sampler v1.3.1
	go: extracting rsc.io/sampler v1.3.1
	$ go test
	PASS
	ok  	example.com/hello	0.022s
	$

Note the explicit `@v1.3.1` in the `go`get` argument.
In general each argument passed to `go`get` can take
an explicit version; the default is `@latest`,
which resolves to the latest version as defined earlier.

* Adding a dependency on a new major version

Let's add a new function to our package:
`func`Proverb` returns a Go concurrency proverb,
by calling `quote.Concurrency`, which is provided by
the module `rsc.io/quote/v3`.
First we update `hello.go` to add the new function:

	package hello

	import (
		"rsc.io/quote"
		quoteV3 "rsc.io/quote/v3"
	)

	func Hello() string {
		return quote.Hello()
	}

	func Proverb() string {
		return quoteV3.Concurrency()
	}

Then we add a test to `hello_test.go`:

	func TestProverb(t *testing.T) {
		want := "Concurrency is not parallelism."
		if got := Proverb(); got != want {
			t.Errorf("Proverb() = %q, want %q", got, want)
		}
	}

Then we can test our code:

	$ go test
	go: finding rsc.io/quote/v3 v3.1.0
	go: downloading rsc.io/quote/v3 v3.1.0
	go: extracting rsc.io/quote/v3 v3.1.0
	PASS
	ok  	example.com/hello	0.024s
	$


Note that our module now depends on both `rsc.io/quote` and `rsc.io/quote/v3`:

	$ go list -m rsc.io/q...
	rsc.io/quote v1.5.2
	rsc.io/quote/v3 v3.1.0
	$

Each different major version (`v1`, `v2`, and so on) of a Go module
uses a different module path: starting at `v2`, the path must end in the major version.
In the example, `v3` of `rsc.io/quote` is no longer `rsc.io/quote`: instead,
it is identified by the module path `rsc.io/quote/v3`.
This convention is called
[[https://research.swtch.com/vgo-import][semantic import versioning]],
and it gives incompatible packages (those with different major versions)
different names.
In contrast, `v1.6.0` of `rsc.io/quote` should be backwards-compatible
with `v1.5.2`, so it reuses the name `rsc.io/quote`.
(In the previous section, `rsc.io/sampler` `v1.99.99`
_should_ have been backwards-compatible
with `rsc.io/sampler` `v1.3.0`, but bugs or incorrect client assumptions about
module behavior can both happen.)

The `go` command allows a build to include at most one version of
any particular module path, meaning at most one of each major
version: one `rsc.io/quote`, one `rsc.io/quote/v2`, one `rsc.io/quote/v3`,
and so on.
This gives module authors a clear rule about possible duplication
of a single module path: it is impossible for a program to build with both
`rsc.io/quote`v1.5.2` and `rsc.io/quote`v1.6.0`.
At the same time, allowing different major versions of a module
(because they have different paths)
gives module consumers the ability to
upgrade to a new major version incrementally.
In this example, we wanted to use `quote.Concurrency` from `rsc/quote/v3`v3.1.0`
but are not yet ready to migrate our uses of `rsc.io/quote`v1.5.2`.
The ability to migrate incrementally
is especially important in a large program or codebase.

* Upgrading a dependency to a new major version

Let's complete our conversion from using `rsc.io/quote` to using only `rsc.io/quote/v3`.
Because of the major version change, we should expect that some APIs may have
been removed, renamed, or otherwise changed in incompatible ways.
Reading the docs, we can see that `Hello` has become `HelloV3`:

	$ go doc rsc.io/quote/v3
	package quote // import "rsc.io/quote"

	Package quote collects pithy sayings.

	func Concurrency() string
	func GlassV3() string
	func GoV3() string
	func HelloV3() string
	func OptV3() string
	$

(There is also a
[[https://golang.org/issue/30778][known bug]] in the output;
the displayed import path has incorrectly dropped the `/v3`.)

We can update our use of `quote.Hello()` in `hello.go` to use `quoteV3.HelloV3()`:

	package hello

	import quoteV3 "rsc.io/quote/v3"

	func Hello() string {
		return quoteV3.HelloV3()
	}

	func Proverb() string {
		return quoteV3.Concurrency()
	}

And then at this point, there's no need for the renamed import anymore,
so we can undo that:

	package hello

	import "rsc.io/quote/v3"

	func Hello() string {
		return quote.HelloV3()
	}

	func Proverb() string {
		return quote.Concurrency()
	}


Let's re-run the tests to make sure everything is working:

	$ go test
	PASS
	ok      example.com/hello       0.014s

* Removing unused dependencies

We've removed all our uses of `rsc.io/quote`,
but it still shows up in `go`list`-m`all` and in our `go.mod` file:

	$ go list -m all
	example.com/hello
	golang.org/x/text v0.3.0
	rsc.io/quote v1.5.2
	rsc.io/quote/v3 v3.1.0
	rsc.io/sampler v1.3.1
	$ cat go.mod
	module example.com/hello

	go 1.12

	require (
		golang.org/x/text v0.3.0 // indirect
		rsc.io/quote v1.5.2
		rsc.io/quote/v3 v3.0.0
		rsc.io/sampler v1.3.1 // indirect
	)
	$

Why? Because building a single package, like with `go`build` or `go`test`,
can easily tell when something is missing and needs to be added,
but not when something can safely be removed.
Removing a dependency can only be done after
checking all packages in a module,
and all possible build tag combinations for those packages.
An ordinary build command does not load this information,
and so it cannot safely remove dependencies.

The `go`mod`tidy` command cleans up these unused dependencies:

	$ go mod tidy
	$ go list -m all
	example.com/hello
	golang.org/x/text v0.3.0
	rsc.io/quote/v3 v3.1.0
	rsc.io/sampler v1.3.1
	$ cat go.mod
	module example.com/hello

	go 1.12

	require (
		golang.org/x/text v0.3.0 // indirect
		rsc.io/quote/v3 v3.1.0
		rsc.io/sampler v1.3.1 // indirect
	)

	$ go test
	PASS
	ok  	example.com/hello	0.020s
	$

* Conclusion

Go modules are the future of dependency management in Go.
Module functionality is now available in all supported Go versions
(that is, in Go 1.11 and Go 1.12).

This post introduced these workflows using Go modules:

- `go`mod`init` creates a new module, initializing the `go.mod` file that describes it.
- `go`build`, `go`test`, and other package-building commands add new dependencies to `go.mod` as needed.
- `go`list`-m`all` prints the current module’s dependencies.
- `go`get` changes the required version of a dependency (or adds a new dependency).
- `go`mod`tidy` removes unused dependencies.

We encourage you to start using modules in your local development
and to add `go.mod` and `go.sum` files to your projects.
To provide feedback and help shape the future of dependency management in Go,
please send us
[[https://golang.org/issue/new][bug reports]] or [[https://golang.org/wiki/ExperienceReports][experience reports]].

Thanks for all your feedback and help improving modules.