A Proposal for Package Versioning in Go
26 Mar 2018
Tags: tools, versioning

Russ Cox

* Introduction

Eight years ago, the Go team introduced `goinstall`
(which led to `go`get`)
and with it the decentralized, URL-like import paths
that Go developers are familiar with today.
After we released `goinstall`, one of the first questions people asked
was how to incorporate version information.
We admitted we didn’t know.
For a long time, we believed that the problem of package versioning
would be best solved by an add-on tool,
and we encouraged people to create one.
The Go community created many tools with different approaches.
Each one helped us all better understand the problem,
but by mid-2016 it was clear that there were now too many solutions.
We needed to adopt a single, official tool.

After a community discussion started at GopherCon in July 2016 and continuing into the fall,
we all believed the answer would be to follow the package versioning approach
exemplified by Rust’s Cargo, with tagged semantic versions,
a manifest, a lock file, and a
[[https://research.swtch.com/version-sat][SAT solver]] to decide which versions to use.
Sam Boyer led a team to create Dep, which followed this rough plan,
and which we intended to serve as the model for `go` command integration.
But as we learned more about the implications of the Cargo/Dep approach,
it became clear to me that Go would benefit from changing
some of the details, especially concerning backwards compatibility.

* The Impact of Compatibility

The most important new feature of
[[https://blog.golang.org/preview-of-go-version-1][Go 1]]
was not a language feature.
It was Go 1’s emphasis on backwards compatibility.
Until that point we’d issued stable release 
snapshots approximately monthly,
each with significant incompatible changes.
We observed significant acceleration in interest and adoption
immediately after the release of Go 1. 
We believe that the
[[https://golang.org/doc/go1compat.html][promise of compatibility]]
made developers feel much more comfortable relying on
Go for production use
and is a key reason that Go is popular today.
Since 2013 the
[[https://golang.org/doc/faq#get_version][Go FAQ]]
has encouraged package developers to provide their own
users with similar expectations of compatibility.
We call this the _import_compatibility_rule_:
“If an old package and a new package have the same import path,
the new package must be backwards compatible with the old package.”

Independently,
[[http://semver.org/][semantic versioning]]
has become the _de_facto_
standard for describing software versions in many language communities,
including the Go community.
Using semantic versioning, later versions are expected to be
backwards-compatible with earlier versions,
but only within a single major version:
v1.2.3 must be compatible with v1.2.1 and v1.1.5,
but v2.3.4 need not be compatible with any of those.

If we adopt semantic versioning for Go packages,
as most Go developers expect,
then the import compatibility rule requires that
different major versions must use different import paths.
This observation led us to _semantic_import_versioning_,
in which versions starting at v2.0.0 include the major
version in the import path: `my/thing/v2/sub/pkg`.

A year ago I strongly believed that whether to include
version numbers in import paths was largely a matter of taste,
and I was skeptical that having them was particularly elegant.
But the decision turns out to be a matter not of taste but of logic:
import compatibility and semantic versioning together require
semantic import versioning.
When I realized this, the logical necessity surprised me.

I was also surprised to realize that
there is a second, independent logical route to 
semantic import versioning:
[[https://talks.golang.org/2016/refactor.article][gradual code repair]]
or partial code upgrades.
In a large program, it’s unrealistic to expect all packages in the program
to update from v1 to v2 of a particular dependency at the same time.
Instead, it must be possible for some of the program to keep using v1
while other parts have upgraded to v2.
But then the program’s build, and the program’s final binary,
must include both v1 and v2 of the dependency.
Giving them the same import path would lead to confusion,
violating what we might call the _import_uniqueness_rule_:
different packages must have different import paths.
The only way to have
partial code upgrades, import uniqueness, _and_ semantic versioning
is to adopt
semantic import versioning as well.

It is of course possible to build systems that use semantic versioning
without semantic import versioning,
but only by giving up either partial code upgrades or import uniqueness.
Cargo allows partial code upgrades by
giving up import uniqueness:
a given import path can have different meanings
in different parts of a large build.
Dep ensures import uniqueness by
giving up partial code upgrades:
all packages involved in a large build must find
a single agreed-upon version of a given dependency,
raising the possibility that large programs will be unbuildable.
Cargo is right to insist on partial code upgrades,
which are critical to large-scale software development.
Dep is equally right to insist on import uniqueness.
Complex uses of Go’s current vendoring support can violate import uniqueness.
When they have, the resulting problems have been quite challenging
for both developers and tools to understand.
Deciding between partial code upgrades
and import uniqueness
requires predicting which will hurt more to give up.
Semantic import versioning lets us avoid the choice
and keep both instead.

I was also surprised to discover how much
import compatibility simplifies version selection,
which is the problem of deciding which package versions to use for a given build.
The constraints of Cargo and Dep make version selection
equivalent to 
[[https://research.swtch.com/version-sat][solving Boolean satisfiability]],
meaning it can be very expensive to determine whether
a valid version configuration even exists.
And then there may be many valid configurations,
with no clear criteria for choosing the “best” one.
Relying on import compatibility can instead let Go use
a trivial, linear-time algorithm
to find the single best configuration, which always exists.
This algorithm,
which I call 
[[https://research.swtch.com/vgo-mvs][_minimal_version_selection_]],
in turn eliminates the need for separate lock and manifest files.
It replaces them with a single, short configuration file,
edited directly by both developers and tools,
that still supports reproducible builds.

Our experience with Dep demonstrates the impact of compatibility.
Following the lead of Cargo and earlier systems,
we designed Dep to give up import compatibility
as part of adopting semantic versioning.
I don’t believe we decided this deliberately;
we just followed those other systems.
The first-hand experience of using Dep helped us
better understand exactly how much complexity
is created by permitting incompatible import paths.
Reviving the import compatibility rule
by introducing semantic import versioning
eliminates that complexity,
leading to a much simpler system.

* Progress, a Prototype, and a Proposal

Dep was released in January 2017.
Its basic model—code tagged with
semantic versions, along with a configuration file that
specified dependency requirements—was
a clear step forward from most of the Go vendoring tools,
and converging on Dep itself was also a clear step forward.
I wholeheartedly encouraged its adoption,
especially to help developers get used to thinking about Go package versions,
both for their own code and their dependencies.
While Dep was clearly moving us in the right direction, I had lingering concerns
about the complexity devil in the details.
I was particularly concerned about Dep
lacking support for gradual code upgrades in large programs.
Over the course of 2017, I talked to many people,
including Sam Boyer and the rest of the
package management working group,
but none of us could see any clear way to reduce the complexity.
(I did find many approaches that added to it.)
Approaching the end of the year,
it still seemed like SAT solvers and unsatisfiable builds
might be the best we could do.

In mid-November, trying once again to work through
how Dep could support gradual code upgrades,
I realized that our old advice about import compatibility
implied semantic import versioning.
That seemed like a real breakthrough.
I wrote a first draft of what became my
[[https://research.swtch.com/vgo-import][semantic import versioning]]
blog post,
concluding it by suggesting that Dep adopt the convention.
I sent the draft to the people I’d been talking to,
and it elicited very strong responses:
everyone loved it or hated it.
I realized that I needed to work out more of the
implications of semantic import versioning
before circulating the idea further,
and I set out to do that.

In mid-December, I discovered that import compatibility
and semantic import versioning together allowed
cutting version selection down to [[https://research.swtch.com/vgo-mvs][minimal version selection]].
I wrote a basic implementation to be sure I understood it,
I spent a while learning the theory behind why it was so simple,
and I wrote a draft of the post describing it.
Even so, I still wasn’t sure the approach would be practical
in a real tool like Dep.
It was clear that a prototype was needed.

In January, I started work on a simple `go` command wrapper
that implemented semantic import versioning
and minimal version selection.
Trivial tests worked well.
Approaching the end of the month,
my simple wrapper could build Dep,
a real program that made use of many versioned packages.
The wrapper still had no command-line interface—the fact that
it was building Dep was hard-coded in a few string constants—but
the approach was clearly viable.

I spent the first three weeks of February turning the
wrapper into a full versioned `go` command, `vgo`;
writing drafts of a
[[https://research.swtch.com/vgo][blog post series introducing `vgo`]];
and discussing them with 
Sam Boyer, the package management working group,
and the Go team.
And then I spent the last week of February finally
sharing `vgo` and the ideas behind it with the whole Go community.

In addition to the core ideas of import compatibility,
semantic import versioning, and minimal version selection,
the `vgo` prototype introduces a number of smaller
but significant changes motivated by eight years of
experience with `goinstall` and `go`get`:
the new concept of a [[https://research.swtch.com/vgo-module][Go module]],
which is a collection of packages versioned as a unit;
[[https://research.swtch.com/vgo-repro][verifiable and verified builds]];
and
[[https://research.swtch.com/vgo-cmd][version-awareness throughout the `go` command]],
enabling work outside `$GOPATH`
and the elimination of (most) `vendor` directories.

The result of all of this is the [[https://golang.org/design/24301-versioned-go][official Go proposal]],
which I filed last week.
Even though it might look like a complete implementation,
it’s still just a prototype,
one that we will all need to work together to complete.
You can download and try the `vgo` prototype from [[https://golang.org/x/vgo][golang.org/x/vgo]],
and you can read the
[[https://research.swtch.com/vgo-tour][Tour of Versioned Go]]
to get a sense of what using `vgo` is like.

* The Path Forward

The proposal I filed last week is exactly that: an initial proposal.
I know there are problems with it that the Go team and I can’t see,
because Go developers use Go in many clever ways that we don’t know about.
The goal of the proposal feedback process is for us all to work together
to identify and address the problems in the current proposal,
to make sure that the final implementation that ships in a future
Go release works well for as many developers as possible.
Please point out problems on the [[https://golang.org/issue/24301][proposal discussion issue]].
I will keep the
[[https://golang.org/issue/24301#issuecomment-371228742][discussion summary]]
and
[[https://golang.org/issue/24301#issuecomment-371228664][FAQ]]
updated as feedback arrives.

For this proposal to succeed, the Go ecosystem as a
whole—and in particular today’s major Go projects—will need to
adopt the import compatibility rule and semantic import versioning.
To make sure that can happen smoothly, 
we will also be conducting user feedback sessions
by video conference with projects that have questions about
how to incorporate the new versioning proposal into their code bases
or have feedback about their experiences.
If you are interested in participating in such a session,
please email Steve Francia at spf@golang.org.

We’re looking forward to (finally!) providing the Go community with a single, official answer
to the question of how to incorporate package versioning into `go`get`.
Thanks to everyone who helped us get this far, and to everyone who will help us going forward.
We hope that, with your help, we can ship something that Go developers will love.