Migrating to Go Modules
21 Aug 2019
Tags: tools, versioning, modules

Jean de Klerk

* Introduction
This post is part 2 in a series. See part 1 — [[https://blog.golang.org/using-go-modules][Using Go Modules]].

Go projects use a wide variety of dependency management strategies. [[https://golang.org/cmd/go/#hdr-Vendor_Directories][Vendoring]] tools such as [[https://github.com/golang/dep][dep]] and [[https://github.com/Masterminds/glide][glide]] are popular, but they have wide differences in behavior and don't always work well together. Some projects store their entire GOPATH directory in a single Git repository. Others simply rely on `go`get` and expect fairly recent versions of dependencies to be installed in GOPATH.

Go's module system, introduced in Go 1.11, provides an official dependency management solution built into the `go` command. This article describes tools and techniques for converting a project to modules.

Please note: if your project is already tagged at v2.0.0 or higher, you will need to update your module path when you add a `go.mod` file. We'll explain how to do that without breaking your users in a future article focused on v2 and beyond.

* Migrating to Go modules in your project

A project might be in one of three states when beginning the transition to Go modules:

- A brand new Go project.
- An established Go project with a non-modules dependency manager.
- An established Go project without any dependency manager.

The first case is covered in [[https://blog.golang.org/using-go-modules][Using Go Modules]]; we'll address the latter two in this post.

* With a dependency manager

To convert a project that already uses a dependency management tool, run the following commands:

    $ git clone https://github.com/my/project
    [...]
    $ cd project
    $ cat Godeps/Godeps.json
    {
        "ImportPath": "github.com/my/project",
        "GoVersion": "go1.12",
        "GodepVersion": "v80",
        "Deps": [
            {
                "ImportPath": "rsc.io/binaryregexp",
                "Comment": "v0.2.0-1-g545cabd",
                "Rev": "545cabda89ca36b48b8e681a30d9d769a30b3074"
            },
            {
                "ImportPath": "rsc.io/binaryregexp/syntax",
                "Comment": "v0.2.0-1-g545cabd",
                "Rev": "545cabda89ca36b48b8e681a30d9d769a30b3074"
            }
        ]
    }
    $ go mod init github.com/my/project
    go: creating new go.mod: module github.com/my/project
    go: copying requirements from Godeps/Godeps.json
    $ cat go.mod
    module github.com/my/project

    go 1.12

    require rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca
    $

`go`mod`init` creates a new go.mod file and automatically imports dependencies from `Godeps.json`, `Gopkg.lock`, or a number of [[https://go.googlesource.com/go/+/362625209b6cd2bc059b6b0a67712ddebab312d9/src/cmd/go/internal/modconv/modconv.go#9][other supported formats]]. The argument to `go`mod`init` is the module path, the location where the module may be found.

This is a good time to pause and run `go`build`./...` and `go`test`./...` before continuing. Later steps may modify your `go.mod` file, so if you prefer to take an iterative approach, this is the closest your `go.mod` file will be to your pre-modules dependency specification.

    $ go mod tidy
    go: downloading rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca
    go: extracting rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca
    $ cat go.sum
    rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca h1:FKXXXJ6G2bFoVe7hX3kEX6Izxw5ZKRH57DFBJmHCbkU=
    rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
    $

`go`mod`tidy` finds all the packages transitively imported by packages in your module. It adds new module requirements for packages not provided by any known module, and it removes requirements on modules that don't provide any imported packages. If a module provides packages that are only imported by projects that haven't migrated to modules yet, the module requirement will be marked with an `//`indirect` comment. It is always good practice to run `go`mod`tidy` before committing a `go.mod` file to version control.

Let's finish by making sure the code builds and tests pass:

    $ go build ./...
    $ go test ./...
    [...]
    $

Note that other dependency managers may specify dependencies at the level of individual packages or entire repositories (not modules), and generally do not recognize the requirements specified in the `go.mod` files of dependencies. Consequently, you may not get exactly the same version of every package as before, and there's some risk of upgrading past breaking changes. Therefore, it's important to follow the above commands with an audit of the resulting dependencies. To do so, run

    $ go list -m all
    go: finding rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca
    github.com/my/project
    rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca
    $

and compare the resulting versions with your old dependency management file to ensure that the selected versions are appropriate. If you find a version that wasn't what you wanted, you can find out why using `go`mod`why`-m` and/or `go`mod`graph`, and upgrade or downgrade to the correct version using `go`get`. (If the version you request is older than the version that was previously selected, `go`get` will downgrade other dependencies as needed to maintain compatibility.) For example,

    $ go mod why -m rsc.io/binaryregexp
    [...]
    $ go mod graph | grep rsc.io/binaryregexp
    [...]
    $ go get rsc.io/binaryregexp@v0.2.0
    $

* Without a dependency manager

For a Go project without a dependency management system, start by creating a `go.mod` file:

    $ git clone https://go.googlesource.com/blog
    [...]
    $ cd blog
    $ go mod init golang.org/x/blog
    go: creating new go.mod: module golang.org/x/blog
    $ cat go.mod
    module golang.org/x/blog

    go 1.12
    $

Without a configuration file from a previous dependency manager, `go`mod`init` will create a `go.mod` file with only the `module` and `go` directives. In this example, we set the module path to `golang.org/x/blog` because that is its [[https://golang.org/cmd/go/#hdr-Remote_import_paths][custom import path]]. Users may import packages with this path, and we must be careful not to change it.

The `module` directive declares the module path, and the `go` directive declares the expected version of the Go language used to compile the code within the module.

Next, run `go`mod`tidy` to add the module's dependencies:

    $ go mod tidy
    go: finding golang.org/x/website latest
    go: finding gopkg.in/tomb.v2 latest
    go: finding golang.org/x/net latest
    go: finding golang.org/x/tools latest
    go: downloading github.com/gorilla/context v1.1.1
    go: downloading golang.org/x/tools v0.0.0-20190813214729-9dba7caff850
    go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
    go: extracting github.com/gorilla/context v1.1.1
    go: extracting golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
    go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
    go: extracting gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
    go: extracting golang.org/x/tools v0.0.0-20190813214729-9dba7caff850
    go: downloading golang.org/x/website v0.0.0-20190809153340-86a7442ada7c
    go: extracting golang.org/x/website v0.0.0-20190809153340-86a7442ada7c
    $ cat go.mod
    module golang.org/x/blog

    go 1.12

    require (
        github.com/gorilla/context v1.1.1
        golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
        golang.org/x/text v0.3.2
        golang.org/x/tools v0.0.0-20190813214729-9dba7caff850
        golang.org/x/website v0.0.0-20190809153340-86a7442ada7c
        gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637
    )
    $ cat go.sum
    cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
    cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
    git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
    git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg=
    github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
    [...]
    $

`go`mod`tidy` added module requirements for all the packages transitively imported by packages in your module and built a `go.sum` with checksums for each library at a specific version. Let's finish by making sure the code still builds and tests still pass:

    $ go build ./...
    $ go test ./...
    ok  	golang.org/x/blog	0.335s
    ?   	golang.org/x/blog/content/appengine	[no test files]
    ok  	golang.org/x/blog/content/cover	0.040s
    ?   	golang.org/x/blog/content/h2push/server	[no test files]
    ?   	golang.org/x/blog/content/survey2016	[no test files]
    ?   	golang.org/x/blog/content/survey2017	[no test files]
    ?   	golang.org/x/blog/support/racy	[no test files]
    $

Note that when `go`mod`tidy` adds a requirement, it adds the latest version of the module. If your `GOPATH` included an older version of a dependency that subsequently published a breaking change, you may see errors in `go`mod`tidy`, `go`build`, or `go`test`. If this happens, try downgrading to an older version with `go`get` (for example, `go`get`github.com/broken/module@v1.1.0`), or take the time to make your module compatible with the latest version of each dependency.

** Tests in module mode

Some tests may need tweaks after migrating to Go modules.

If a test needs to write files in the package directory, it may fail when the package directory is in the module cache, which is read-only. In particular, this may cause `go`test`all` to fail. The test should copy files it needs to write to a temporary directory instead.

If a test relies on relative paths (`../package-in-another-module`) to locate and read files in another package, it will fail if the package is in another module, which will be located in a versioned subdirectory of the module cache or a path specified in a `replace` directive. If this is the case, you may need to copy the test inputs into your module, or convert the test inputs from raw files to data embedded in `.go` source files.

If a test expects `go` commands within the test to run in GOPATH mode, it may fail. If this is the case, you may need to add a `go.mod` file to the source tree to be tested, or set `GO111MODULE=off` explicitly.

* Publishing a release

Finally, you should tag and publish a release version for your new module. This is optional if you haven't released any versions yet, but without an official release, downstream users will depend on specific commits using [[https://golang.org/cmd/go/#hdr-Pseudo_versions][pseudo-versions]], which may be more difficult to support.

    $ git tag v1.2.0
    $ git push origin v1.2.0

Your new `go.mod` file defines a canonical import path for your module and adds new minimum version requirements. If your users are already using the correct import path, and your dependencies haven't made breaking changes, then adding the `go.mod` file is backwards-compatible — but it's a significant change, and may expose existing problems. If you have existing version tags, you should increment the [[https://semver.org/#spec-item-7][minor version]].

* Imports and canonical module paths

Each module declares its module path in its `go.mod` file. Each `import` statement that refers to a package within the module must have the module path as a prefix of the package path. However, the `go` command may encounter a repository containing the module through many different [[https://golang.org/cmd/go/#hdr-Remote_import_paths][remote import paths]]. For example, both `golang.org/x/lint` and `github.com/golang/lint` resolve to repositories containing the code hosted at [[https://go.googlesource.com/lint][go.googlesource.com/lint]]. The [[https://go.googlesource.com/lint/+/refs/heads/master/go.mod][`go.mod` file]] contained in that repository declares its path to be `golang.org/x/lint`, so only that path corresponds to a valid module.

Go 1.4 provided a mechanism for declaring canonical import paths using [[https://golang.org/cmd/go/#hdr-Import_path_checking][`//`import` comments]], but package authors did not always provide them. As a result, code written prior to modules may have used a non-canonical import path for a module without surfacing an error for the mismatch. When using modules, the import path must match the canonical module path, so you may need to update `import` statements: for example, you may need to change `import`"github.com/golang/lint"` to `import`"golang.org/x/lint"`.

Another scenario in which a module's canonical path may differ from its repository path occurs for Go modules at major version 2 or higher. A Go module with a major version above 1 must include a major-version suffix in its module path: for example, version `v2.0.0` must have the suffix `/v2`. However, `import` statements may have referred to the packages within the module _without_ that suffix. For example, non-module users of `github.com/russross/blackfriday/v2` at `v2.0.1` may have imported it as `github.com/russross/blackfriday` instead, and will need to update the import path to include the `/v2` suffix.

* Conclusion

Converting to Go modules should be a straightforward process for most users. Occasional issues may arise due to non-canonical import paths or breaking changes within dependency. Future posts will explore publishing new versions, v2 and beyond, and ways to debug strange situations.

To provide feedback and help shape the future of dependency management in Go, please send us [[https://golang.org/issue/new][bug reports]] or [[https://golang.org/wiki/ExperienceReports][experience reports]].

Thanks for all your feedback and help improving modules.