blob: 1f37f68a666de0779c80102c75c3fcc6d447abfe [file] [log] [blame]
How Go was made
GopherCon Closing Keynote
9 Jul 2015
Andrew Gerrand
adg@golang.org
* Video
A video of this talk was recorded at GopherCon in Denver.
.link https://www.youtube.com/watch?v=0ht89TxZZnk Watch the talk on YouTube
* This talk
How was Go made?
What was its development process?
How has the process changed?
What is its future?
.image how-go-was-made/gopherswrench.jpg 300 _
* The early days
* Three friends
Robert Griesemer, Rob Pike, and Ken Thompson's thought experiment:
"What should a modern, practical programming language look like?"
* The early process
Design discussions by mail and in-person.
Consensus-driven:
A feature was accepted only when *all* the three authors agreed it necessary.
Most proposed changes were rejected.
* The initial spec
The first artifact of Go was the language specification.
The first commit (now git hash `18c5b48`) was a draft of that spec:
Author: Robert Griesemer <gri@golang.org>
Date: Sun Mar 2 20:47:34 2008 -0800
Go spec starting point.
Go's entire history is preserved in the core reposistory.
* The first Go program
The first Go program was a Prime Sieve, and was included in the spec.
func Filter(in *chan< int, out *chan> int, prime int) {
for {
i := <in; // Receive value of new variable 'i' from 'in'.
if i % prime != 0 {
>out = i // Send 'i' to channel 'out'.
}
}
}
func Sieve() {
ch := new(chan int); // Create a new channel.
go Generate(ch); // Start Generate() as a subprocess.
for {
prime := <ch;
printf("%d\n", prime);
ch1 := new(chan int);
go Filter(ch, ch1, prime);
ch = ch1
}
}
* Version control
Subversion was the project's first version control system. (no code review!)
The trio made 400 commits to Subversion.
The last Subversion commit (git hash `777ee7`, 21 July 2008) contained:
- the spec,
- a Go compiler (linux/darwin amd64),
- a few packages (fmt, rand, and math),
- some test programs.
`test/helloworld.go`:
package main
func main() {
print "hello, world\n";
}
* Getting serious
In July 2008 the project migrated from Subversion to Perforce,
to use Google's excellent code review system.
The team grew:
- Russ Cox joined in August 2008,
- Ian Lance Taylor joined in September,
- ~20 other Googlers got involved in their 20% time.
* Early changes
It was easy to make changes in those days.
Everyone knew each other.
There were ~0 users.
All the source code in one place.
Workflow informal, but design-oriented.
* Waiting for good design (1/2)
Many problems don't have obvious solutions.
The team were prepared to wait until they found the correct design.
For example:
The `fmt` package was written long before `reflect`.
Before then, `fmt` had an awkward chaining API:
fmt.New().s("i = ").d(i).putnl()
Now:
fmt.Println("i = ", i)
* Waiting for good design (2/2)
Another example:
Slices took more than a year to figure out.
Before then, there were "open arrays" but they were awkward.
`src/lib/container/vector.go` @ `f4dcf51`:
// BUG: workaround for non-constant allocation.
// i must be a power of 10.
func Alloc(i int) *[]Element {
switch i {
case 1:
return new([1]Element);
case 10:
return new([10]Element);
case 100:
return new([100]Element);
case 1000:
return new([1000]Element);
}
print "bad size ", i, "\n";
panic "not known size\n";
}
* An early language change
`chan` and `map` were originally spelled `*chan` and `*map`.
The team had a meeting to discuss removing the asterisks.
Russ implemented the compiler change (`dc7b2e9`),
and updated ~every Go file in existence (`08ca30b`).
.code how-go-was-made/mapchan.diff
This rapid approach works well at a small scale.
* Toward open source
In mid-2009 the team prepared for the open source release.
Moved to their third version control system:
Mercurial, with Rietveld for code review.
# (We would go on to use these tools for five years.)
The last Perforce commit (`9e96f25`, Oct 29) contained
- a mature spec and compiler,
- a broad standard library (82 packages),
- gofmt, godoc, cgo,
- the testing framework,
- the FAQ, Effective Go, and more.
* Open source
* The release
10 November 2009: Go was released. (`78c47c3`)
.image how-go-was-made/website.png 500 _
* The response
There was a huge response to the release. The team was overwhelmed.
People sent changes from day one. The first non-Googler change:
commit 022e3ae2659491e519d392e266acd86223a510f4
Author: Kevin Ballard <kevin@sb.org>
Date: Tue Nov 10 20:04:14 2009 -0800
Fix go-mode.el to work on empty buffers
Fixes #8.
R=agl, agl1, rsc
https://golang.org/cl/153056
In the first month, 32 people from outside Google contributed to Go.
The contribution process worked.
* Changes after the release
The team continued to make big changes after the release.
But the process was now different, as they now had a community.
* The first major public change (1/2)
On December 9th, Rob Pike sent the first public change proposal.
He proposed to remove semicolons at line endings.
[[https://golang.org/s/semicolon-proposal][golang.org/s/semicolon-proposal]]
The proposal was a "design doc" that included:
- background,
- rationale,
- a formal specification of the change,
- an implementation plan, and
- examples (including a copy of the Prime Sieve program without semicolons).
* The first major public change (2/2)
The design doc was shared with the community for feedback.
"Please read the proposal and think about its consequences.
We're pretty sure it makes the language nicer to use and sacrifices almost nothing in precision or safety."
The response was positive.
The proposal was implemented.
Most of the changes were made mechanically (thanks `gofmt`)).
This would be the shape of things to come.
* Curating change
* Curating change
The team was careful to curate the changes that made it into the language.
(Remember: "Wait for good design.")
There were many proposed changes and additions to Go.
As before, many more were declined than accepted.
Some fit with the project's goals, others did not.
An early accepted proposal:
- Make the upper and lower bounds of slice operations optional. (`x[lo:]`, `x[:hi]`)
And a declined proposal:
- Allow negative indices in slice operations. (`x[-n]` `==` `x[len(x)-n]`)
* Communicating goals
At first, we did a poor job explaining the project's goals and development process.
This caused frustration. ("Why don't they accept my suggestions?")
It took us a while to articulate it:
Rob Pike's [[http://talks.golang.org/2012/splash.article][talk]] in October 2012
"Go at Google: Language Design in the Service of Software Engineering"
was the first thorough explanation of Go's _raison_d'être_.
I wish we'd had this document written in 2009.
(But we couldn't have written it then.)
Read it, if you haven't already.
* A thought on openness
What if Go has been open source from day one?
It may have been easier for the public to understand the project.
But it's not that simple:
Ideas are fragile in their early stages; they need to be nurtured before exposed to the world.
* Managing change
* Weekly snapshots
Attempt to keep everyone in sync.
- Apply a Mercurial tag to a specific, stable revision.
- Announce to user mailing list with detailed changelog.
Great for early adopters and core developers.
* Problems with weeklies
Contributors work at tip; users sync to weeklies.
Burden on users:
- annoying to update weekly,
- painful to update less often.
Version skew results because users are at different weeklies.
Skew fragments the community and slows adoption.
* Formal release process
March 2011: introduced releases every 1-2 months.
- Pick the most stable of the past few snapshots and tag it.
- Announce with abridged "must read" release notes.
Keeps the community more in sync. Reduces churn.
Popular with users.
But skew still prevalent: adventurers and core devs still use weeklies (or tip!).
* Introducing Gofix
A tool to mechanically update code to accommodate language and library changes.
gofix prog.go
Announced in May 2011.
Eases the burden of staying current.
Release notes now mostly say "run gofix."
Not a sed script. Works on the AST.
* A gofix example
The `reflect` API was completely redesigned in 2011.
Gofix made most of the changes:
.image how-go-was-made/reflect1.png _ 1000
.image how-go-was-made/reflect3.png _ 1000
.image how-go-was-made/reflect2.png _ 1000
* Versioning issues persist
Gofix is no panacea.
As the root of the dependency graph, a programming language can suffer acutely from version skew.
The fundamental issue remains:
Code you write today may not compile tomorrow.
Some companies unwilling to bet on Go as they saw it as unstable.
* A need for stability
Gofix makes changes very easy, and also makes it easy to experiment.
But it can't do everything.
Priorities: If change is easy, what change is important?
Wanted to make major changes to the language and libraries,
but this requires planning.
Decision: design and implement a stable version of Go, its libraries, and its tools.
* Go 1
* What is Go 1?
A specification of the language and libraries that will be supported for years.
Available as downloadable binary packages.
An opportunity to:
- fix minor language irritations,
- fix inconsistencies in the standard library,
- focus on bug fixing and cleaning up TODOs,
- design and build a strong build tool set (get rid of make),
- bring Windows support up to par.
Polish and refine, not redesign.
* Planning Go 1
The team at Google prepared a detailed design document.
Implemented (but did not commit) many of the proposed changes.
Met for a week to discuss and refine the document (October 2011).
Presented the document to the community for discussion.
Community feedback essential in refining the document.
.link http://blog.golang.org/preview-of-go-version-1
* An example: errors (1/5)
Before Go 1, there was no error type.
Instead, the `os` package provided an `Error` type:
package os
type Error interface {
String() string
}
References to `os.Error` were ubiquitous.
package io
type Reader interface {
Read(p []byte) (n int, err os.Error)
}
type Closer interface {
Close() os.Error
}
* An example: errors (2/5)
Before the Go 1 meeting, Russ raised the issue in the design document:
.image how-go-was-made/errors-issue.png 500 _
* An example: errors (3/5)
At the meeting the team discussed the issue,
Russ presented data from his experiments with the change,
and we made a tentative decision:
.image how-go-was-made/errors-discussion.png 350 _
* An example: errors (4/5)
On the list, the community made some keen suggestions:
.image how-go-was-made/errors-rog.png 500 _
* An example: errors (5/5)
Many of those suggestions were rolled into the Go 1 design document:
.image how-go-was-made/errors-final.png 500 _
* Implementing Go 1
Create many new issues on the tracker.
Contributors nominate themselves to address specific issues.
Stop developing new features; prioritize stability.
* The success of Go 1
The Go 1 release in March 2012 heralded a new era for the project.
Users appreciated the stability. Huge uptick in community growth.
.image how-go-was-made/trends.png
Contributors focused on implementation, tools, and ecosystem.
* The release cycle
The next release was Go 1.1, more than a year after Go 1.
This is too long; switch to a 6-month release cycle.
Stuck to this plan for 1.2, 1.3, and 1.4, and we're (almost) on track for 1.5.
* The future of change
* A question
Go's development process emphasizes up-front design.
The project has ~450 contributors and many committers from outside Google.
But most change proposals
(despite being driven by community feedback)
were made by the Go team at Google.
Why is this?
* Contribution guidelines
The Go contribution guidelines are thorough on the code review process.
But on design, just this:
"Before undertaking to write something new for the Go project, send mail to the mailing list to discuss what you plan to do."
Does this decribe the project's approach to design? Only superficially.
Successful proposals include design docs
that discuss rationale, tradeoffs, and implementation.
There is a gap in our documented process.
* Go Change Proposal Process: a proposal
Brad Fitzpatrick and I recently proposed a formal Change Proposal Process.
Its goals:
- Make the design process clear to new contributors.
- Commit to timely evaluations of proposals.
- Build a recorded history of proposals.
* How does it work?
The new change process, in brief:
- File an issue.
- The contributors triage the issue.
- Write a design document (following a template).
- The contributors review the document.
- If the proposal is accepted, implement it.
[[https://golang.org/s/proposal-process][golang.org/s/proposal-process]]
* An experimental process
The new process is an experiment.
We're still discussing exactly how it should work.
I hope it will make Go's design process more accessible to the community.
* Conclusion
Go's design-driven process has served us well.
But as Go is more widely used,
the community needs a larger role in shaping its future.
With your help, we can make Go's next 5 years more spectacular than the last.
.image how-go-was-made/5years.png 350 _