| Migrating to Go Modules |
| 21 Aug 2019 |
| Tags: tools, versioning, modules |
| |
| Jean de Klerk |
| |
| * Introduction |
| |
| This post is part 2 in a series. |
| |
| - Part 1 — [[/using-go-modules][Using Go Modules]] |
| - *Part*2*—*Migrating*To*Go*Modules* (this post) |
| - Part 3 — [[/publishing-go-modules][Publishing Go Modules]] |
| - Part 4 — [[/v2-go-modules][Go Modules: v2 and Beyond]] |
| |
| Go projects use a wide variety of dependency management strategies. |
| [[https://golang.org/cmd/go/#hdr-Vendor_Directories][Vendoring]] tools such |
| as [[https://github.com/golang/dep][dep]] and [[https://github.com/Masterminds/glide][glide]] are popular, |
| but they have wide differences in behavior and don't always work well together. |
| Some projects store their entire GOPATH directory in a single Git repository. |
| Others simply rely on `go`get` and expect fairly recent versions of dependencies |
| to be installed in GOPATH. |
| |
| Go's module system, introduced in Go 1.11, |
| provides an official dependency management solution built into the `go` command. |
| This article describes tools and techniques for converting a project to modules. |
| |
| Please note: if your project is already tagged at v2.0.0 or higher, |
| you will need to update your module path when you add a `go.mod` file. |
| We'll explain how to do that without breaking your users in a future article |
| focused on v2 and beyond. |
| |
| * Migrating to Go modules in your project |
| |
| A project might be in one of three states when beginning the transition to Go modules: |
| |
| - A brand new Go project. |
| - An established Go project with a non-modules dependency manager. |
| - An established Go project without any dependency manager. |
| |
| The first case is covered in [[https://blog.golang.org/using-go-modules][Using Go Modules]]; |
| we'll address the latter two in this post. |
| |
| * With a dependency manager |
| |
| To convert a project that already uses a dependency management tool, run the following commands: |
| |
| $ git clone https://github.com/my/project |
| [...] |
| $ cd project |
| $ cat Godeps/Godeps.json |
| { |
| "ImportPath": "github.com/my/project", |
| "GoVersion": "go1.12", |
| "GodepVersion": "v80", |
| "Deps": [ |
| { |
| "ImportPath": "rsc.io/binaryregexp", |
| "Comment": "v0.2.0-1-g545cabd", |
| "Rev": "545cabda89ca36b48b8e681a30d9d769a30b3074" |
| }, |
| { |
| "ImportPath": "rsc.io/binaryregexp/syntax", |
| "Comment": "v0.2.0-1-g545cabd", |
| "Rev": "545cabda89ca36b48b8e681a30d9d769a30b3074" |
| } |
| ] |
| } |
| $ go mod init github.com/my/project |
| go: creating new go.mod: module github.com/my/project |
| go: copying requirements from Godeps/Godeps.json |
| $ cat go.mod |
| module github.com/my/project |
| |
| go 1.12 |
| |
| require rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca |
| $ |
| |
| `go`mod`init` creates a new go.mod file and automatically imports dependencies from `Godeps.json`, |
| `Gopkg.lock`, or a number of [[https://go.googlesource.com/go/+/362625209b6cd2bc059b6b0a67712ddebab312d9/src/cmd/go/internal/modconv/modconv.go#9][other supported formats]]. |
| The argument to `go`mod`init` is the module path, |
| the location where the module may be found. |
| |
| This is a good time to pause and run `go`build`./...` and `go`test`./...` before continuing. |
| Later steps may modify your `go.mod` file, |
| so if you prefer to take an iterative approach, |
| this is the closest your `go.mod` file will be to your pre-modules dependency specification. |
| |
| $ go mod tidy |
| go: downloading rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca |
| go: extracting rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca |
| $ cat go.sum |
| rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca h1:FKXXXJ6G2bFoVe7hX3kEX6Izxw5ZKRH57DFBJmHCbkU= |
| rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= |
| $ |
| |
| `go`mod`tidy` finds all the packages transitively imported by packages in your module. |
| It adds new module requirements for packages not provided by any known module, |
| and it removes requirements on modules that don't provide any imported packages. |
| If a module provides packages that are only imported by projects that haven't |
| migrated to modules yet, |
| the module requirement will be marked with an `//`indirect` comment. |
| It is always good practice to run `go`mod`tidy` before committing a `go.mod` |
| file to version control. |
| |
| Let's finish by making sure the code builds and tests pass: |
| |
| $ go build ./... |
| $ go test ./... |
| [...] |
| $ |
| |
| Note that other dependency managers may specify dependencies at the level |
| of individual packages or entire repositories (not modules), |
| and generally do not recognize the requirements specified in the `go.mod` |
| files of dependencies. |
| Consequently, you may not get exactly the same version of every package as before, |
| and there's some risk of upgrading past breaking changes. |
| Therefore, it's important to follow the above commands with an audit of |
| the resulting dependencies. To do so, run |
| |
| $ go list -m all |
| go: finding rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca |
| github.com/my/project |
| rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca |
| $ |
| |
| and compare the resulting versions with your old dependency management file |
| to ensure that the selected versions are appropriate. |
| If you find a version that wasn't what you wanted, |
| you can find out why using `go`mod`why`-m` and/or `go`mod`graph`, |
| and upgrade or downgrade to the correct version using `go`get`. |
| (If the version you request is older than the version that was previously selected, |
| `go`get` will downgrade other dependencies as needed to maintain compatibility.) For example, |
| |
| $ go mod why -m rsc.io/binaryregexp |
| [...] |
| $ go mod graph | grep rsc.io/binaryregexp |
| [...] |
| $ go get rsc.io/binaryregexp@v0.2.0 |
| $ |
| |
| * Without a dependency manager |
| |
| For a Go project without a dependency management system, start by creating a `go.mod` file: |
| |
| $ git clone https://go.googlesource.com/blog |
| [...] |
| $ cd blog |
| $ go mod init golang.org/x/blog |
| go: creating new go.mod: module golang.org/x/blog |
| $ cat go.mod |
| module golang.org/x/blog |
| |
| go 1.12 |
| $ |
| |
| Without a configuration file from a previous dependency manager, |
| `go`mod`init` will create a `go.mod` file with only the `module` and `go` directives. |
| In this example, we set the module path to `golang.org/x/blog` because that |
| is its [[https://golang.org/cmd/go/#hdr-Remote_import_paths][custom import path]]. |
| Users may import packages with this path, |
| and we must be careful not to change it. |
| |
| The `module` directive declares the module path, |
| and the `go` directive declares the expected version of the Go language |
| used to compile the code within the module. |
| |
| Next, run `go`mod`tidy` to add the module's dependencies: |
| |
| $ go mod tidy |
| go: finding golang.org/x/website latest |
| go: finding gopkg.in/tomb.v2 latest |
| go: finding golang.org/x/net latest |
| go: finding golang.org/x/tools latest |
| go: downloading github.com/gorilla/context v1.1.1 |
| go: downloading golang.org/x/tools v0.0.0-20190813214729-9dba7caff850 |
| go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 |
| go: extracting github.com/gorilla/context v1.1.1 |
| go: extracting golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 |
| go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 |
| go: extracting gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 |
| go: extracting golang.org/x/tools v0.0.0-20190813214729-9dba7caff850 |
| go: downloading golang.org/x/website v0.0.0-20190809153340-86a7442ada7c |
| go: extracting golang.org/x/website v0.0.0-20190809153340-86a7442ada7c |
| $ cat go.mod |
| module golang.org/x/blog |
| |
| go 1.12 |
| |
| require ( |
| github.com/gorilla/context v1.1.1 |
| golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 |
| golang.org/x/text v0.3.2 |
| golang.org/x/tools v0.0.0-20190813214729-9dba7caff850 |
| golang.org/x/website v0.0.0-20190809153340-86a7442ada7c |
| gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 |
| ) |
| $ cat go.sum |
| cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= |
| cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= |
| git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= |
| git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= |
| github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= |
| [...] |
| $ |
| |
| `go`mod`tidy` added module requirements for all the packages transitively |
| imported by packages in your module and built a `go.sum` with checksums |
| for each library at a specific version. |
| Let's finish by making sure the code still builds and tests still pass: |
| |
| $ go build ./... |
| $ go test ./... |
| ok golang.org/x/blog 0.335s |
| ? golang.org/x/blog/content/appengine [no test files] |
| ok golang.org/x/blog/content/cover 0.040s |
| ? golang.org/x/blog/content/h2push/server [no test files] |
| ? golang.org/x/blog/content/survey2016 [no test files] |
| ? golang.org/x/blog/content/survey2017 [no test files] |
| ? golang.org/x/blog/support/racy [no test files] |
| $ |
| |
| Note that when `go`mod`tidy` adds a requirement, |
| it adds the latest version of the module. |
| If your `GOPATH` included an older version of a dependency that subsequently |
| published a breaking change, |
| you may see errors in `go`mod`tidy`, `go`build`, or `go`test`. |
| If this happens, try downgrading to an older version with `go`get` (for example, |
| `go`get`github.com/broken/module@v1.1.0`), |
| or take the time to make your module compatible with the latest version of each dependency. |
| |
| ** Tests in module mode |
| |
| Some tests may need tweaks after migrating to Go modules. |
| |
| If a test needs to write files in the package directory, |
| it may fail when the package directory is in the module cache, which is read-only. |
| In particular, this may cause `go`test`all` to fail. |
| The test should copy files it needs to write to a temporary directory instead. |
| |
| If a test relies on relative paths (`../package-in-another-module`) to locate |
| and read files in another package, |
| it will fail if the package is in another module, |
| which will be located in a versioned subdirectory of the module cache or |
| a path specified in a `replace` directive. |
| If this is the case, you may need to copy the test inputs into your module, |
| or convert the test inputs from raw files to data embedded in `.go` source files. |
| |
| If a test expects `go` commands within the test to run in GOPATH mode, it may fail. |
| If this is the case, you may need to add a `go.mod` file to the source tree to be tested, |
| or set `GO111MODULE=off` explicitly. |
| |
| * Publishing a release |
| |
| Finally, you should tag and publish a release version for your new module. |
| This is optional if you haven't released any versions yet, |
| but without an official release, downstream users will depend on specific |
| commits using [[https://golang.org/cmd/go/#hdr-Pseudo_versions][pseudo-versions]], |
| which may be more difficult to support. |
| |
| $ git tag v1.2.0 |
| $ git push origin v1.2.0 |
| |
| Your new `go.mod` file defines a canonical import path for your module and adds |
| new minimum version requirements. If your users are already using the correct |
| import path, and your dependencies haven't made breaking changes, then adding |
| the `go.mod` file is backwards-compatible — but it's a significant change, and |
| may expose existing problems. If you have existing version tags, you should |
| increment the [[https://semver.org/#spec-item-7][minor version]]. See |
| [[/publishing-go-modules][Publishing Go Modules]] to learn how to increment and |
| publish versions. |
| |
| * Imports and canonical module paths |
| |
| Each module declares its module path in its `go.mod` file. |
| Each `import` statement that refers to a package within the module must |
| have the module path as a prefix of the package path. |
| However, the `go` command may encounter a repository containing the module |
| through many different [[https://golang.org/cmd/go/#hdr-Remote_import_paths][remote import paths]]. |
| For example, both `golang.org/x/lint` and `github.com/golang/lint` resolve |
| to repositories containing the code hosted at [[https://go.googlesource.com/lint][go.googlesource.com/lint]]. |
| The [[https://go.googlesource.com/lint/+/refs/heads/master/go.mod][`go.mod` file]] |
| contained in that repository declares its path to be `golang.org/x/lint`, |
| so only that path corresponds to a valid module. |
| |
| Go 1.4 provided a mechanism for declaring canonical import paths using [[https://golang.org/cmd/go/#hdr-Import_path_checking][`//`import` comments]], |
| but package authors did not always provide them. |
| As a result, code written prior to modules may have used a non-canonical |
| import path for a module without surfacing an error for the mismatch. |
| When using modules, the import path must match the canonical module path, |
| so you may need to update `import` statements: |
| for example, you may need to change `import`"github.com/golang/lint"` to |
| `import`"golang.org/x/lint"`. |
| |
| Another scenario in which a module's canonical path may differ from its |
| repository path occurs for Go modules at major version 2 or higher. |
| A Go module with a major version above 1 must include a major-version suffix in its module path: |
| for example, version `v2.0.0` must have the suffix `/v2`. |
| However, `import` statements may have referred to the packages within the |
| module _without_ that suffix. |
| For example, non-module users of `github.com/russross/blackfriday/v2` at |
| `v2.0.1` may have imported it as `github.com/russross/blackfriday` instead, |
| and will need to update the import path to include the `/v2` suffix. |
| |
| * Conclusion |
| |
| Converting to Go modules should be a straightforward process for most users. |
| Occasional issues may arise due to non-canonical import paths or breaking |
| changes within a dependency. |
| Future posts will explore [[/publishing-go-modules][publishing new versions]], |
| v2 and beyond, and ways to debug strange situations. |
| |
| To provide feedback and help shape the future of dependency management in Go, |
| please send us [[https://golang.org/issue/new][bug reports]] or [[https://golang.org/wiki/ExperienceReports][experience reports]]. |
| |
| Thanks for all your feedback and help improving modules. |