| How Go was made |
| GopherCon Closing Keynote |
| 9 Jul 2015 |
| |
| Andrew Gerrand |
| adg@golang.org |
| |
| |
| * Video |
| |
| A video of this talk was recorded at GopherCon in Denver. |
| |
| .link https://www.youtube.com/watch?v=0ht89TxZZnk Watch the talk on YouTube |
| |
| |
| * This talk |
| |
| How was Go made? |
| |
| What was its development process? |
| |
| How has the process changed? |
| |
| What is its future? |
| |
| .image how-go-was-made/gopherswrench.jpg 300 _ |
| |
| |
| * The early days |
| |
| |
| * Three friends |
| |
| Robert Griesemer, Rob Pike, and Ken Thompson's thought experiment: |
| |
| "What should a modern, practical programming language look like?" |
| |
| |
| * The early process |
| |
| Design discussions by mail and in-person. |
| |
| Consensus-driven: |
| A feature was accepted only when *all* the three authors agreed it necessary. |
| |
| Most proposed changes were rejected. |
| |
| |
| * The initial spec |
| |
| The first artifact of Go was the language specification. |
| |
| The first commit (now git hash `18c5b48`) was a draft of that spec: |
| |
| Author: Robert Griesemer <gri@golang.org> |
| Date: Sun Mar 2 20:47:34 2008 -0800 |
| |
| Go spec starting point. |
| |
| Go's entire history is preserved in the core reposistory. |
| |
| |
| * The first Go program |
| |
| The first Go program was a Prime Sieve, and was included in the spec. |
| |
| func Filter(in *chan< int, out *chan> int, prime int) { |
| for { |
| i := <in; // Receive value of new variable 'i' from 'in'. |
| if i % prime != 0 { |
| >out = i // Send 'i' to channel 'out'. |
| } |
| } |
| } |
| |
| func Sieve() { |
| ch := new(chan int); // Create a new channel. |
| go Generate(ch); // Start Generate() as a subprocess. |
| for { |
| prime := <ch; |
| printf("%d\n", prime); |
| ch1 := new(chan int); |
| go Filter(ch, ch1, prime); |
| ch = ch1 |
| } |
| } |
| |
| |
| * Version control |
| |
| Subversion was the project's first version control system. (no code review!) |
| The trio made 400 commits to Subversion. |
| |
| The last Subversion commit (git hash `777ee7`, 21 July 2008) contained: |
| |
| - the spec, |
| - a Go compiler (linux/darwin amd64), |
| - a few packages (fmt, rand, and math), |
| - some test programs. |
| |
| `test/helloworld.go`: |
| |
| package main |
| |
| func main() { |
| print "hello, world\n"; |
| } |
| |
| |
| * Getting serious |
| |
| In July 2008 the project migrated from Subversion to Perforce, |
| to use Google's excellent code review system. |
| |
| The team grew: |
| |
| - Russ Cox joined in August 2008, |
| - Ian Lance Taylor joined in September, |
| - ~20 other Googlers got involved in their 20% time. |
| |
| |
| * Early changes |
| |
| It was easy to make changes in those days. |
| Everyone knew each other. |
| There were ~0 users. |
| All the source code in one place. |
| |
| Workflow informal, but design-oriented. |
| |
| |
| * Waiting for good design (1/2) |
| |
| Many problems don't have obvious solutions. |
| |
| The team were prepared to wait until they found the correct design. |
| |
| For example: |
| |
| The `fmt` package was written long before `reflect`. |
| Before then, `fmt` had an awkward chaining API: |
| |
| fmt.New().s("i = ").d(i).putnl() |
| |
| Now: |
| |
| fmt.Println("i = ", i) |
| |
| |
| * Waiting for good design (2/2) |
| |
| Another example: |
| |
| Slices took more than a year to figure out. |
| Before then, there were "open arrays" but they were awkward. |
| |
| `src/lib/container/vector.go` @ `f4dcf51`: |
| |
| // BUG: workaround for non-constant allocation. |
| // i must be a power of 10. |
| func Alloc(i int) *[]Element { |
| switch i { |
| case 1: |
| return new([1]Element); |
| case 10: |
| return new([10]Element); |
| case 100: |
| return new([100]Element); |
| case 1000: |
| return new([1000]Element); |
| } |
| print "bad size ", i, "\n"; |
| panic "not known size\n"; |
| } |
| |
| |
| * An early language change |
| |
| `chan` and `map` were originally spelled `*chan` and `*map`. |
| |
| The team had a meeting to discuss removing the asterisks. |
| |
| Russ implemented the compiler change (`dc7b2e9`), |
| and updated ~every Go file in existence (`08ca30b`). |
| |
| .code how-go-was-made/mapchan.diff |
| |
| This rapid approach works well at a small scale. |
| |
| |
| * Toward open source |
| |
| In mid-2009 the team prepared for the open source release. |
| |
| Moved to their third version control system: |
| Mercurial, with Rietveld for code review. |
| # (We would go on to use these tools for five years.) |
| |
| The last Perforce commit (`9e96f25`, Oct 29) contained |
| |
| - a mature spec and compiler, |
| - a broad standard library (82 packages), |
| - gofmt, godoc, cgo, |
| - the testing framework, |
| - the FAQ, Effective Go, and more. |
| |
| |
| * Open source |
| |
| |
| * The release |
| |
| 10 November 2009: Go was released. (`78c47c3`) |
| |
| .image how-go-was-made/website.png 500 _ |
| |
| |
| * The response |
| |
| There was a huge response to the release. The team was overwhelmed. |
| |
| People sent changes from day one. The first non-Googler change: |
| |
| commit 022e3ae2659491e519d392e266acd86223a510f4 |
| Author: Kevin Ballard <kevin@sb.org> |
| Date: Tue Nov 10 20:04:14 2009 -0800 |
| |
| Fix go-mode.el to work on empty buffers |
| |
| Fixes #8. |
| |
| R=agl, agl1, rsc |
| https://golang.org/cl/153056 |
| |
| In the first month, 32 people from outside Google contributed to Go. |
| |
| The contribution process worked. |
| |
| |
| * Changes after the release |
| |
| The team continued to make big changes after the release. |
| |
| But the process was now different, as they now had a community. |
| |
| |
| * The first major public change (1/2) |
| |
| On December 9th, Rob Pike sent the first public change proposal. |
| |
| He proposed to remove semicolons at line endings. |
| [[https://golang.org/s/semicolon-proposal][golang.org/s/semicolon-proposal]] |
| |
| The proposal was a "design doc" that included: |
| |
| - background, |
| - rationale, |
| - a formal specification of the change, |
| - an implementation plan, and |
| - examples (including a copy of the Prime Sieve program without semicolons). |
| |
| |
| * The first major public change (2/2) |
| |
| The design doc was shared with the community for feedback. |
| |
| "Please read the proposal and think about its consequences. |
| We're pretty sure it makes the language nicer to use and sacrifices almost nothing in precision or safety." |
| |
| The response was positive. |
| |
| The proposal was implemented. |
| |
| Most of the changes were made mechanically (thanks `gofmt`)). |
| |
| This would be the shape of things to come. |
| |
| |
| * Curating change |
| |
| * Curating change |
| |
| The team was careful to curate the changes that made it into the language. |
| (Remember: "Wait for good design.") |
| |
| There were many proposed changes and additions to Go. |
| |
| As before, many more were declined than accepted. |
| |
| Some fit with the project's goals, others did not. |
| |
| An early accepted proposal: |
| |
| - Make the upper and lower bounds of slice operations optional. (`x[lo:]`, `x[:hi]`) |
| |
| And a declined proposal: |
| |
| - Allow negative indices in slice operations. (`x[-n]` `==` `x[len(x)-n]`) |
| |
| |
| * Communicating goals |
| |
| At first, we did a poor job explaining the project's goals and development process. |
| |
| This caused frustration. ("Why don't they accept my suggestions?") |
| |
| It took us a while to articulate it: |
| Rob Pike's [[http://talks.golang.org/2012/splash.article][talk]] in October 2012 |
| "Go at Google: Language Design in the Service of Software Engineering" |
| was the first thorough explanation of Go's _raison_d'être_. |
| |
| I wish we'd had this document written in 2009. |
| (But we couldn't have written it then.) |
| |
| Read it, if you haven't already. |
| |
| |
| * A thought on openness |
| |
| What if Go has been open source from day one? |
| |
| It may have been easier for the public to understand the project. |
| |
| But it's not that simple: |
| |
| Ideas are fragile in their early stages; they need to be nurtured before exposed to the world. |
| |
| |
| * Managing change |
| |
| * Weekly snapshots |
| |
| Attempt to keep everyone in sync. |
| |
| - Apply a Mercurial tag to a specific, stable revision. |
| - Announce to user mailing list with detailed changelog. |
| |
| Great for early adopters and core developers. |
| |
| |
| * Problems with weeklies |
| |
| Contributors work at tip; users sync to weeklies. |
| |
| Burden on users: |
| |
| - annoying to update weekly, |
| - painful to update less often. |
| |
| Version skew results because users are at different weeklies. |
| |
| Skew fragments the community and slows adoption. |
| |
| |
| * Formal release process |
| |
| March 2011: introduced releases every 1-2 months. |
| |
| - Pick the most stable of the past few snapshots and tag it. |
| - Announce with abridged "must read" release notes. |
| |
| Keeps the community more in sync. Reduces churn. |
| |
| Popular with users. |
| |
| But skew still prevalent: adventurers and core devs still use weeklies (or tip!). |
| |
| |
| * Introducing Gofix |
| |
| A tool to mechanically update code to accommodate language and library changes. |
| |
| gofix prog.go |
| |
| Announced in May 2011. |
| |
| Eases the burden of staying current. |
| Release notes now mostly say "run gofix." |
| |
| Not a sed script. Works on the AST. |
| |
| |
| * A gofix example |
| |
| The `reflect` API was completely redesigned in 2011. |
| Gofix made most of the changes: |
| |
| .image how-go-was-made/reflect1.png _ 1000 |
| .image how-go-was-made/reflect3.png _ 1000 |
| .image how-go-was-made/reflect2.png _ 1000 |
| |
| |
| * Versioning issues persist |
| |
| Gofix is no panacea. |
| |
| As the root of the dependency graph, a programming language can suffer acutely from version skew. |
| |
| The fundamental issue remains: |
| Code you write today may not compile tomorrow. |
| |
| Some companies unwilling to bet on Go as they saw it as unstable. |
| |
| |
| * A need for stability |
| |
| Gofix makes changes very easy, and also makes it easy to experiment. |
| But it can't do everything. |
| |
| Priorities: If change is easy, what change is important? |
| |
| Wanted to make major changes to the language and libraries, |
| but this requires planning. |
| |
| Decision: design and implement a stable version of Go, its libraries, and its tools. |
| |
| |
| * Go 1 |
| |
| * What is Go 1? |
| |
| A specification of the language and libraries that will be supported for years. |
| |
| Available as downloadable binary packages. |
| |
| An opportunity to: |
| |
| - fix minor language irritations, |
| - fix inconsistencies in the standard library, |
| - focus on bug fixing and cleaning up TODOs, |
| - design and build a strong build tool set (get rid of make), |
| - bring Windows support up to par. |
| |
| Polish and refine, not redesign. |
| |
| |
| * Planning Go 1 |
| |
| The team at Google prepared a detailed design document. |
| |
| Implemented (but did not commit) many of the proposed changes. |
| |
| Met for a week to discuss and refine the document (October 2011). |
| |
| Presented the document to the community for discussion. |
| |
| Community feedback essential in refining the document. |
| |
| .link http://blog.golang.org/preview-of-go-version-1 |
| |
| |
| * An example: errors (1/5) |
| |
| Before Go 1, there was no error type. |
| Instead, the `os` package provided an `Error` type: |
| |
| package os |
| |
| type Error interface { |
| String() string |
| } |
| |
| References to `os.Error` were ubiquitous. |
| |
| package io |
| |
| type Reader interface { |
| Read(p []byte) (n int, err os.Error) |
| } |
| |
| type Closer interface { |
| Close() os.Error |
| } |
| |
| |
| |
| * An example: errors (2/5) |
| |
| Before the Go 1 meeting, Russ raised the issue in the design document: |
| |
| .image how-go-was-made/errors-issue.png 500 _ |
| |
| |
| * An example: errors (3/5) |
| |
| At the meeting the team discussed the issue, |
| Russ presented data from his experiments with the change, |
| and we made a tentative decision: |
| |
| .image how-go-was-made/errors-discussion.png 350 _ |
| |
| |
| * An example: errors (4/5) |
| |
| On the list, the community made some keen suggestions: |
| |
| .image how-go-was-made/errors-rog.png 500 _ |
| |
| |
| * An example: errors (5/5) |
| |
| Many of those suggestions were rolled into the Go 1 design document: |
| |
| .image how-go-was-made/errors-final.png 500 _ |
| |
| |
| * Implementing Go 1 |
| |
| Create many new issues on the tracker. |
| |
| Contributors nominate themselves to address specific issues. |
| |
| Stop developing new features; prioritize stability. |
| |
| |
| * The success of Go 1 |
| |
| The Go 1 release in March 2012 heralded a new era for the project. |
| |
| Users appreciated the stability. Huge uptick in community growth. |
| |
| .image how-go-was-made/trends.png |
| |
| Contributors focused on implementation, tools, and ecosystem. |
| |
| |
| * The release cycle |
| |
| The next release was Go 1.1, more than a year after Go 1. |
| |
| This is too long; switch to a 6-month release cycle. |
| |
| Stuck to this plan for 1.2, 1.3, and 1.4, and we're (almost) on track for 1.5. |
| |
| |
| * The future of change |
| |
| |
| * A question |
| |
| Go's development process emphasizes up-front design. |
| |
| The project has ~450 contributors and many committers from outside Google. |
| |
| But most change proposals |
| (despite being driven by community feedback) |
| were made by the Go team at Google. |
| |
| Why is this? |
| |
| |
| * Contribution guidelines |
| |
| The Go contribution guidelines are thorough on the code review process. |
| |
| But on design, just this: |
| |
| "Before undertaking to write something new for the Go project, send mail to the mailing list to discuss what you plan to do." |
| |
| Does this decribe the project's approach to design? Only superficially. |
| |
| Successful proposals include design docs |
| that discuss rationale, tradeoffs, and implementation. |
| |
| There is a gap in our documented process. |
| |
| |
| * Go Change Proposal Process: a proposal |
| |
| Brad Fitzpatrick and I recently proposed a formal Change Proposal Process. |
| |
| Its goals: |
| |
| - Make the design process clear to new contributors. |
| - Commit to timely evaluations of proposals. |
| - Build a recorded history of proposals. |
| |
| |
| * How does it work? |
| |
| The new change process, in brief: |
| |
| - File an issue. |
| - The contributors triage the issue. |
| - Write a design document (following a template). |
| - The contributors review the document. |
| - If the proposal is accepted, implement it. |
| |
| [[https://golang.org/s/proposal-process][golang.org/s/proposal-process]] |
| |
| |
| * An experimental process |
| |
| The new process is an experiment. |
| |
| We're still discussing exactly how it should work. |
| |
| I hope it will make Go's design process more accessible to the community. |
| |
| |
| * Conclusion |
| |
| Go's design-driven process has served us well. |
| |
| But as Go is more widely used, |
| the community needs a larger role in shaping its future. |
| |
| With your help, we can make Go's next 5 years more spectacular than the last. |
| |
| .image how-go-was-made/5years.png 350 _ |