The State of the Gopher (Oct)
10 Oct 2014
Tags: dotgo, gopher

Brad Fitzpatrick
Gopher, Google
@bradfitz
bradfitz@golang.org

* The State of the Gopher

- where we've been
- where we're going

.image state-of-the-gopher/gopher.jpg

* bradfitz

- Work on Go's standard library, builders, etc
- Started contributing to Go May 2010
- [[http://camlistore.org]], my big, old Go project: personal data storage & indexing for life

* The Past

* Userbase before November 2009

- 3-10 people inside Google?

* November 2009

- Go is [[http://google-opensource.blogspot.com/2009/11/hey-ho-lets-go.html][open sourced]]:

.image state-of-the-gopher/opensource.png

* Website in 2009

.image state-of-the-gopher/website1.png

* Website in 2009

.image state-of-the-gopher/website2.png

* Compilers in 2009

- already two: `gc` and `gccgo`

* Aside: GC vs gc

- `gc`: "Go compiler", based originally on Plan 9's C compiler. The main Go compiler.
- `GC`: "Garbage Collector"

* Ports in 2009

- linux-386
- linux-amd64
- linux-arm, just starting to work
- darwin-386
- darwin-amd64
- nacl-386 (first attempt)
With gccgo, more.

* Tools in 2009

- gofmt
- gofix
- godoc

* Release Highlights

* weekly releases

- "weekly.2009-12-09"
- break users weekly, not daily
- gofix appears

* monthly releases

- "r56" ... "r60"
- break users monthly, not weekly
- still painful to write and use packages (Makefiles)

* Go 1.0 (2012-03-28)

- the "error" type
- delete(m, k)

  // before Go 1:
  latLong[storeID] = LatLong{}, false
  name[userID] = "", false

  // new:
  delete(name, userID)

- [[/doc/go1compat][stability promise]] + API check tool:

  pkg net/http, const StatusOK = 200
  pkg net/http, const StatusOK ideal-int
  pkg net/http, type RoundTripper interface { RoundTrip }
  pkg net/http, type Server struct, TLSConfig *tls.Config
  pkg net/http, var DefaultServeMux *ServeMux
  pkg time, method (*Timer) Reset(Duration) bool
  pkg unicode/utf8, func ValidRune(int32) bool

- the `go` tool (get, build, test, ...)

* Go 1.1 (2013-05-03)

- method values

  type S struct { once sync.Once; ... }
  func (s *S) init() { ... }
  func (s *S) Foo() {
      s.once.Do(s.init)

- func "terminating statement"

  func abs(n int) int {
      switch {
      case n < 0: return -n
      default:    return n
      }
  }

- 64-bit "int", large heaps
- 30-40% speed improvement, stdlib++
- more precise heap GC, usually

* Aside: precise vs conservative GC

Conservative GC: if it "looks like" a pointer, treat it like one.

  [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ]
  [ ptr? ] [ byte slice (skipped in Go 1.0)                             ]
  [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ] [ ptr? ]

Terrible for 32-bit (e.g. ARM) computers: many integers look like pointers.

Precise GC:

  [ int ] [ ptr     ] [ string      ] [ byte slice          ]
  [ int ] [ t f t f ] [ ptr ] [ int ] [ float64     ] [ ptr ]

Requires knowing what every position in memory _actually_ is.

- no false retention, only look at some memory during GC

* Go 1.2 (2013-12-01)

- 6 month release cycle begins
- setting slice capacity with s[i:j:k]

  s := make([]byte, 10, 20)
  foo(s[10:12:15])

  func foo(s []byte) { fmt.Println(len(s), cap(s)) }

- preemption in scheduler
- go test -cover

* Go 1.3 (2014-06-18)

- no language changes
- GOOS=solaris,dragonfly,plan9,nacl
- precise GC of the stack (mostly)
- contiguous stacks (mostly)

* Aside: Segmented vs. Contiguous stacks

- C++/Java: one huge stack per thread
- Go <= 1.2: multiple discontiguous 4K stacks, bounce between
- Go 1.3: one little stack, grow & shrink as needed
.image state-of-the-gopher/contig-stack.png _ 550

* The present (relative to Nov 2009)

* More ports

- Windows
- FreeBSD, OpenBSD, NetBSD
- Solaris, Plan 9, DragonFly BSD
- NaCl (again)

* Better docs

- [[/blog][Blog]] articles,
- Community [[/wiki][Wiki]],
- [[/wiki/Books][Books]]

* Users: up and to the right

- Go [[https://code.google.com/p/go-wiki/wiki/GoUserGroups][user groups]], [[http://go-meetups.appspot.com/][meetups]]
- GitHub commits, stars, forks
- Since 2009: `golang-nuts@` (1450 → 14134), `-dev` (37 → 2152)
.link http://redmonk.com/dberkholz/2014/03/18/go-the-emerging-language-of-cloud-infrastructure/ Go: the emerging language of cloud infrastructure

.image state-of-the-gopher/oloh.png

* The Cloud

- Docker
- CoreOS, etcd, fleet
- Juju
- NATS, gnatsd, Packer, Heka
- dl.google.com, Vitess, Kubernetes
- App Engine

* Conferences

[[http://connpass.com/series/312/][GoCon Tokyo]]

* Conferences

.image state-of-the-gopher/gophercon.png

* Conferences

.image state-of-the-gopher/dotgo.png

* Conferences

.image state-of-the-gopher/india.png

* CoreOS

.image state-of-the-gopher/bus.jpg

- increasingly running [[http://build.golang.org]], each in a Docker container
- Go powering Go development!

* build.golang.org

.image state-of-the-gopher/build.png 400 _

* Better tools

- go/parser: godoc, [[https://pkg.go.dev]], gofmt, gofix →
- [[https://pkg.go.dev/code.google.com/p/go.tools/cmd/goimports][goimports]]: automatic import lines
- go/types, gofix+types,
- go/ast, go/ssa, [[https://pkg.go.dev/code.google.com/p/go.tools/cmd/oracle][oracle]],
- [[https://pkg.go.dev/code.google.com/p/go.tools/cmd/eg][eg]], example-based refactoring
- [[https://pkg.go.dev/code.google.com/p/go.tools/cmd/gorename][gorename]], safe global renaming tool

Tooling begets better tooling!

- goimports + go/types + go/parser == [[https://twitter.com/sqs/status/520203018957250560][goreturns]] (lazy "return err")

* The Future

* Go 1.4

* Go 1.4 (2014-12-01?)

- fully precise GC of the stack + heap
- segmented stacks & conservative GC deleted
- stack copier & shrinker requires precise stack types too

- `runtime` conversion from C to Go: maps, channels, interfaces, type checks, println, defer, panic, etc.

- Garbage collector and scheduler are still in C

* Go 1.4

- [[/s/go14android][GOOS=android]] (Elias Naur, David Crawshaw)
- GOOS=nacl GOARCH=arm
- [[/s/go14internal]["internal" packages]]
- [[/s/go1.4-generate]["go generate"]]
- [[https://docs.google.com/a/golang.org/document/d/1QXzI9I1pOfZPujQzxhyRy6EeHYTQitKKjHfpq0zpxZs][syscall package frozen, go.sys]]
- "go tool pprof" converted from Perl (!) to Go
- minor stdlib additions, improvements

* Go 1.5

* Go 1.5 (2014-06-01?)

- Concurrent GC is main theme
- Roadmap: [[/s/go14gc]] (GC in 1.4, 1.5, 1.6+)

* Go 1.5: ports

- GOOS=ios revival? (Minux, David Crawshaw)
- GOARCH=ppc64 (minux)
- GOARCH=arm64 (aram, dfc)

* Go 1.5: removing more C

Didn't make Go 1.4:

- `cmd/link`: linker rewrite in Go
- `cmd/asm`: assembler rewrite in Go
- "gc" compiler in Go

* "gc" Compiler in Go

- lots of C
- C → Go translator == kinda ugly Go
- refactor that Go with tools: `eg`, `gorename`, etc
- add packages, tests
- unify all `8g`, `5g`, `6g`, `9g`
- SSA form, new optimizations

See rsc's GopherCon [[/talks/2014/c2go.slide#1][slides]], [[http://gophercon.sourcegraph.com/post/83820197495/russ-cox-porting-the-go-compiler-from-c-to-go][notes]], [[https://www.youtube.com/watch?v=QIE5nV5fDwA][video]].

* Go 1.5 Tracing

- Dmitry's GOTRACE + Chrome trace viewer

.image state-of-the-gopher/trace.png

* Go 1.6

- More GC work
- Ongoing compiler-in-Go cleanup & optimizations
- PNaCl?
- [[https://docs.google.com/a/google.com/document/d/1d3iI2QWURgDIsSR6G2275vMeQ_X7w-qxM2Vp7iGwwuM/pub][NUMA-aware scheduler]]?
- unknown surprises
Who knows.
Find out next year at dotGo Paris 2015!

* Go 2.0

- maybe one day

* Outside of the core

* libraries & tools

- HTTP/2, to be merged into `net/http`
- go.tools: `goimports`, `eg`, `gorename`, `oracle`, ...
- go.text: Unicode collation, normalization, ...
- go.mobile: Android, iOS, gaming, 3D graphics, event handling
- go.crypto
- go.net
- go.tools
- go.syscall

* gccgo

- keeping up with spec
- better in some ways (codegen, OS, arch)
- worse in others (precise GC, escape analysis)
- work underway to improve

All checking & improving each other:

- gc, gccgo, go/types, language spec, unit tests

* Notable other compilers

- [[https://github.com/go-llvm/llgo][llgo]], Go → LLVM → PNaCL, etc
- [[https://github.com/tardisgo/tardisgo][TARDIS Go]], Go → Haxe → JS, anything
- [[https://github.com/gopherjs/gopherjs][GopherJS]], Go → JS

And all three written in Go, use `go/parser`, `go/types`, etc.
We can't wait to join them.

* In summary

* The growth is exciting

- things are getting cleaner,
- things are getting faster,
- things are getting more impressive,
- everything is growing on top of everything else,
- hard to keep up with golang-dev@, golang-nuts@, bugs, wiki, new github projects, code reviews,

* You all are awesome

- keep being awesome
- keep making awesome things