content: add versioning-proposal.article

Change-Id: I1ea2c3d31245109f7ec68ac1a75c8e3d24005410
Reviewed-on: https://go-review.googlesource.com/102595
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Steve Francia <spf@golang.org>
diff --git a/content/versioning-proposal.article b/content/versioning-proposal.article
new file mode 100644
index 0000000..20c4426
--- /dev/null
+++ b/content/versioning-proposal.article
@@ -0,0 +1,289 @@
+A Proposal for Package Versioning in Go
+26 Mar 2018
+Tags: tools, versioning
+
+Russ Cox
+
+* Introduction
+
+Eight years ago, the Go team introduced `goinstall`
+(which led to `go`get`)
+and with it the decentralized, URL-like import paths
+that Go developers are familiar with today.
+After we released `goinstall`, one of the first questions people asked
+was how to incorporate version information.
+We admitted we didn’t know.
+For a long time, we believed that the problem of package versioning
+would be best solved by an add-on tool,
+and we encouraged people to create one.
+The Go community created many tools with different approaches.
+Each one helped us all better understand the problem,
+but by mid-2015 it was clear that there were now too many solutions.
+We needed to adopt a single, official tool.
+
+After a community discussion started at GopherCon in July 2015 and continuing into the fall,
+we all believed the answer would be to follow the package versioning approach
+exemplified by Rust’s Cargo, with tagged semantic versions,
+a manifest, a lock file, and a
+[[https://research.swtch.com/version-sat][SAT solver]] to decide which versions to use.
+Sam Boyer led a team to create Dep, which followed this rough plan,
+and which we intended to serve as the model for `go` command integration.
+But as we learned more about the implications of the Cargo/Dep approach,
+it became clear to me that Go would benefit from changing
+some of the details, especially concerning backwards compatibility.
+
+* The Impact of Compatibility
+
+The most important new feature of
+[[https://blog.golang.org/preview-of-go-version-1][Go 1]]
+was not a language feature.
+It was Go 1’s emphasis on backwards compatibility.
+Until that point we’d issued stable release 
+snapshots approximately monthly,
+each with significant incompatible changes.
+We observed significant acceleration in interest and adoption
+immediately after the release of Go 1. 
+We believe that the
+[[https://golang.org/doc/go1compat.html][promise of compatibility]]
+made developers feel much more comfortable relying on
+Go for production use
+and is a key reason that Go is popular today.
+Since 2013 the
+[[https://golang.org/doc/faq][Go FAQ]]
+has encouraged package developers to provide their own
+users with similar expectations of compatibility.
+We call this the _import_compatibility_rule_:
+“If an old package and a new package have the same import path,
+the new package must be backwards compatible with the old package.”
+
+Independently,
+[[http://semver.org/][semantic versioning]]
+has become the _de_facto_
+standard for describing software versions in many language communities,
+including the Go community.
+Using semantic versioning, later versions are expected to be
+backwards-compatible with earlier versions,
+but only within a single major version:
+v1.2.3 must be compatible with v1.2.1 and v1.1.5,
+but v2.3.4 need not be compatible with any of those.
+
+If we adopt semantic versioning for Go packages,
+as most Go developers expect,
+then the import compatibility rule requires that
+different major versions must use different import paths.
+This observation led us to _semantic_import_versioning_,
+in which versions starting at v2.0.0 include the major
+version in the import path: `my/thing/v2/sub/pkg`.
+
+A year ago I strongly believed that whether to include
+version numbers in import paths was largely a matter of taste,
+and I was skeptical that having them was particularly elegant.
+But the decision turns out to be a matter not of taste but of logic:
+import compatibility and semantic versioning together require
+semantic import versioning.
+When I realized this, the logical necessity surprised me.
+
+I was also surprised to realize that
+there is a second, independent logical route to 
+semantic import versioning:
+[[https://talks.golang.org/2016/refactor.article][gradual code repair]]
+or partial code upgrades.
+In a large program, it’s unrealistic to expect all packages in the program
+to update from v1 to v2 of a particular dependency at the same time.
+Instead, it must be possible for some of the program to keep using v1
+while other parts have upgraded to v2.
+But then the program’s build, and the program’s final binary,
+must include both v1 and v2 of the dependency.
+Giving them the same import path would lead to confusion,
+violating what we might call the _import_uniqueness_rule_:
+different packages must have different import paths.
+The only way to have
+partial code upgrades, import uniqueness, _and_ semantic versioning
+is to adopt
+semantic import versioning as well.
+
+It is of course possible to build systems that use semantic versioning
+without semantic import versioning,
+but only by giving up either partial code upgrades or import uniqueness.
+Cargo allows partial code upgrades by
+giving up import uniqueness:
+a given import path can have different meanings
+in different parts of a large build.
+Dep ensures import uniqueness by
+giving up partial code upgrades:
+all packages involved in a large build must find
+a single agreed-upon version of a given dependency,
+raising the possibility that large programs will be unbuildable.
+Cargo is right to insist on partial code upgrades,
+which are critical to large-scale software development.
+Dep is equally right to insist on import uniqueness.
+Complex uses of Go’s current vendoring support can violate import uniqueness.
+When they have, the resulting problems have been quite challenging
+for both developers and tools to understand.
+Deciding between partial code upgrades
+and import uniqueness
+requires predicting which will hurt more to give up.
+Semantic import versioning lets us avoid the choice
+and keep both instead.
+
+I was also surprised to discover how much
+import compatibility simplifies version selection,
+which is the problem of deciding which package versions to use for a given build.
+The constraints of Cargo and Dep make version selection
+equivalent to 
+[[https://research.swtch.com/version-sat][solving Boolean satisfiability]],
+meaning it can be very expensive to determine whether
+a valid version configuration even exists.
+And then there may be many valid configurations,
+with no clear criteria for choosing the “best” one.
+Relying on import compatibility can instead let Go use
+a trivial, linear-time algorithm
+to find the single best configuration, which always exists.
+This algorithm,
+which I call 
+[[https://research.swtch.com/vgo-mvs][_minimal_version_selection_]],
+in turn eliminates the need for separate lock and manifest files.
+It replaces them with a single, short configuration file,
+edited directly by both developers and tools,
+that still supports reproducible builds.
+
+Our experience with Dep demonstrates the impact of compatibility.
+Following the lead of Cargo and earlier systems,
+we designed Dep to give up import compatibility
+as part of adopting semantic versioning.
+I don’t believe we decided this deliberately;
+we just followed those other systems.
+The first-hand experience of using Dep helped us
+better understand exactly how much complexity
+is created by permitting incompatible import paths.
+Reviving the import compatibility rule
+by introducing semantic import versioning
+eliminates that complexity,
+leading to a much simpler system.
+
+* Progress, a Prototype, and a Proposal
+
+Dep was released in January 2017.
+Its basic model—code tagged with
+semantic versions, along with a configuration file that
+specified dependency requirements—was
+a clear step forward from most of the Go vendoring tools,
+and converging on Dep itself was also a clear step forward.
+I wholeheartedly encouraged its adoption,
+especially to help developers get used to thinking about Go package versions,
+both for their own code and their dependencies.
+While Dep was clearly moving us in the right direction, I had lingering concerns
+about the complexity devil in the details.
+I was particularly concerned about Dep
+lacking support for gradual code upgrades in large programs.
+Over the course of 2017, I talked to many people,
+including Sam Boyer and the rest of the
+package management working group,
+but none of us could see any clear way to reduce the complexity.
+(I did find many approaches that added to it.)
+Approaching the end of the year,
+it still seemed like SAT solvers and unsatisfiable builds
+might be the best we could do.
+
+In mid-November, trying once again to work through
+how Dep could support gradual code upgrades,
+I realized that our old advice about import compatibility
+implied semantic import versioning.
+That seemed like a real breakthrough.
+I wrote a first draft of what became my
+[[https://research.swtch.com/vgo-import][semantic import versioning]]
+blog post,
+concluding it by suggesting that Dep adopt the convention.
+I sent the draft to the people I’d been talking to,
+and it elicited very strong responses:
+everyone loved it or hated it.
+I realized that I needed to work out more of the
+implications of semantic import versioning
+before circulating the idea further,
+and I set out to do that.
+
+In mid-December, I discovered that import compatibility
+and semantic import versioning together allowed
+cutting version selection down to [[https://research.swtch.com/vgo-mvs][minimal version selection]].
+I wrote a basic implementation to be sure I understood it,
+I spent a while learning the theory behind why it was so simple,
+and I wrote a draft of the post describing it.
+Even so, I still wasn’t sure the approach would be practical
+in a real tool like Dep.
+It was clear that a prototype was needed.
+
+In January, I started work on a simple `go` command wrapper
+that implemented semantic import versioning
+and minimal version selection.
+Trivial tests worked well.
+Approaching the end of the month,
+my simple wrapper could build Dep,
+a real program that made use of many versioned packages.
+The wrapper still had no command-line interface—the fact that
+it was building Dep was hard-coded in a few string constants—but
+the approach was clearly viable.
+
+I spent the first three weeks of February turning the
+wrapper into a full versioned `go` command, `vgo`;
+writing drafts of a
+[[https://research.swtch.com/vgo][blog post series introducing `vgo`]];
+and discussing them with 
+Sam Boyer, the package management working group,
+and the Go team.
+And then I spent the last week of February finally
+sharing `vgo` and the ideas behind it with the whole Go community.
+
+In addition to the core ideas of import compatibility,
+semantic import versioning, and minimal version selection,
+the `vgo` prototype introduces a number of smaller
+but significant changes motivated by eight years of
+experience with `goinstall` and `go`get`:
+the new concept of a [[https://research.swtch.com/vgo-module][Go module]],
+which is a collection of packages versioned as a unit;
+[[https://research.swtch.com/vgo-repro][verifiable and verified builds]];
+and
+[[https://research.swtch.com/vgo-cmd][version-awareness throughout the `go` command]],
+enabling work outside `$GOPATH`
+and the elimination of (most) `vendor` directories.
+
+The result of all of this is the [[https://golang.org/design/24301-versioned-go][official Go proposal]],
+which I filed last week.
+Even though it might look like a complete implementation,
+it’s still just a prototype,
+one that we will all need to work together to complete.
+You can download and try the `vgo` prototype from [[golang.org/x/vgo][https://golang.org/x/vgo]],
+and you can read the
+[[https://research.swtch.com/vgo-tour][Tour of Versioned Go]]
+to get a sense of what using `vgo` is like.
+
+* The Path Forward
+
+The proposal I filed last week is exactly that: an initial proposal.
+I know there are problems with it that the Go team and I can’t see,
+because Go developers use Go in many clever ways that we don’t know about.
+The goal of the proposal feedback process is for us all to work together
+to identify and address the problems in the current proposal,
+to make sure that the final implementation that ships in a future
+Go release works well for as many developers as possible.
+Please point out problems on the [[https://golang.org/issue/24301][proposal discussion issue]].
+I will keep the
+[[https://golang.org/issue/24301#issuecomment-371228742][discussion summary]]
+and
+[[https://golang.org/issue/24301#issuecomment-371228664][FAQ]]
+updated as feedback arrives.
+
+For this proposal to succeed, the Go ecosystem as a
+whole—and in particular today’s major Go projects—will need to
+adopt the import compatibility rule and semantic import versioning.
+To make sure that can happen smoothly, 
+we will also be conducting user feedback sessions
+by video conference with projects that have questions about
+how to incorporate the new versioning proposal into their code bases
+or have feedback about their experiences.
+If you are interested in participating in such a session,
+please email Steve Francia at spf@golang.org.
+
+We’re looking forward to (finally!) providing the Go community with a single, official answer
+to the question of how to incorporate package versioning into `go`get`.
+Thanks to everyone who helped us get this far, and to everyone who will help us going forward.
+We hope that, with your help, we can ship something that Go developers will love.