2015/tricks.slide

Stupid Gopher Tricks
GolangUK
21 Aug 2015

Andrew Gerrand
adg@golang.org


* Video

A video of this talk was recorded at GolangUK in London.

.link https://www.youtube.com/watch?v=UECh7X07m6E Watch the talk on YouTube


* This talk

This talk is about things you might not know about Go.

Some of this stuff you can add to your Go vocabulary.

Other things are probably best left for special occasions.


* Language


* Type literals (1/2)

Here's a familiar type declaration:

	type Foo struct {
		i int
		s string
	}

The latter part of a type declaration is the *type*literal*:

	struct {
		i int
		s string
	}

Other examples of type literals are `int` and `[]string`,
which can also be declared as named types:

	type Bar int
	type Qux []string


* Type literals (2/2)

While we commonly use `int` and `[]string` in variable declarations:

	var i int
	var s []string

It is less common (but equally valid) to do the same with structs:

	var t struct {
		i int
		s string
	}

An unnamed struct literal is often called an *anonymous*struct*.


* Anonymous structs: template data

A common use is providing data to templates:

.play tricks/template.go /BEGIN/,/END/

You could also use `map[string]interface{}`,
but then you sacrifice performance and type safety.


* Anonymous structs: JSON (1/2)

The same technique can be used for encoding JSON objects:

.play tricks/json-encode.go /Marshal/,/Print/

And also decoding:

.play tricks/json-decode.go /data/,/Print/


* Anonymous structs: JSON (2/2)

Structs can be nested to describe more complex JSON objects:

.play tricks/json-nest.go /data/,/Print/


* Repeated literals and struct names

In repeated literals like slices and maps, Go lets you omit the inner type name:

.code tricks/repeated.go /BEGIN/,/END/


* Repeated literals and anonymous structs

Combined with anonymous structs, this convenience shortens the code dramatically:

.code tricks/repeated2.go /BEGIN/,/END/


* Anonymous structs: test cases (1/2)

These properties enable a nice way to express test cases:

.code tricks/string_test.go /TestIndex/,/^}/


* Anonymous structs: test cases (2/2)

You can go a step further and put the composite literal in the range statement itself:

.code tricks/string_test2.go /TestIndex/,/^}/

But this is harder to read.


* Embedded fields

A struct field that has no name is an *embedded*field*.
The embedded type's methods (and fields, if it is a struct)
are accessible as if they are part of the embedding struct.

.play tricks/embed.go /BEGIN/,$


* Anonymous structs: embedded mutex

Of course, you can embed fields in an anonymous struct.

It's common to protect a global variable with a mutex variable:

	var (
		viewCount   int64
		viewCountMu sync.Mutex
	)

By embedding a mutex in an anonymous struct, we can group the related values:

	var viewCount struct {
		sync.Mutex
		n int64
	}

Users of `viewCount` access it like this:

	viewCount.Lock()
	viewCount.n++
	viewCount.Unlock()



* Anonymous structs: implementing interfaces

And you can embed interfaces, too.

Here's a real example from [[https://camlistore.org/][Camlistore]]:

The function is expected to return a `ReadSeekCloser`,
but the programmer only had a string.

Anonymous struct (and its standard library friends) to the rescue!

	return struct {
		io.ReadSeeker
		io.Closer
	}{
		io.NewSectionReader(strings.NewReader(s), 0, int64(len(s))),
		ioutil.NopCloser(nil),
	}


* Anonymous interfaces

Interfaces can be anonymous, the most common being `interface{}`.

But the interface needn't be empty:

.play tricks/anon-interface.go /var s/,/Println/

Useful for a sly type assertion (from `src/os/exec/exec_test.go`):

	// Check that we can access methods of the underlying os.File.
	if _, ok := stdin.(interface {
		Fd() uintptr
	}); !ok {
		t.Error("can't access methods of underlying *os.File")
	}


* Method values

A "method value" is what you get when you evaluate a method as an expression.
The result is a function value.

Evaluating a method from a _type_ yields a function:

.play tricks/method-values-1.go /var f/,/Stdout/

Evaluating a method from a _value_ creates a closure that holds that value:

.play tricks/method-values-2.go /var /,/Stdout/


* Method values: sync.Once

The `sync.Once` type is used to perform a task once with concurrency safety.

	// Once is an object that will perform exactly one action.
	type Once struct { /* Has unexported fields. */ }

	func (o *Once) Do(f func())
	
This `LazyPrimes` type computes a slice of prime numbers the first time it is used:

.code tricks/method-once.go /type/,$


* Method values: HTTP handlers

You can use method values to implement multiple HTTP handlers with one type:

.code tricks/method-http.go /type Server/,$


* Method values: another example

In package `os/exec`, the `Cmd` type implements methods to set up
standard input, output, and error:

	func (c *Cmd) stdin() (f *os.File, err error)
	func (c *Cmd) stdout() (f *os.File, err error)
	func (c *Cmd) stderr() (f *os.File, err error)

The caller handles each in the same way,
so it iterates over a slice of method values:

	type F func(*Cmd) (*os.File, error)
	for _, setupFd := range []F{(*Cmd).stdin, (*Cmd).stdout, (*Cmd).stderr} {
		fd, err := setupFd(c)
		if err != nil {
			c.closeDescriptors(c.closeAfterStart)
			c.closeDescriptors(c.closeAfterWait)
			return err
		}
		c.childFiles = append(c.childFiles, fd)
	}


* Comparable types

The Go spec defines a set of types as "comparable";
they may be compared with == and !=.

Bools, ints, floats, complex numbers, strings, pointers,
channels, *structs*, and *interfaces* are comparable.

.play tricks/compare.go /BEGIN/,/END/

A struct is comparable only if its fields are comparable:

.play tricks/compare2.go /BEGIN/,/END/


* Comparable types and map keys

Any comparable type may be used as a map key.

.code tricks/compare-map.go /BEGIN/,/END/


* Structs as map keys

An example from the Go continuous build infrastructure:

	type builderRev struct {
		builder, rev string
	}

	var br = builderRev{"linux-amd64", "0cd299"}

We track in-flight builds in a map.
The pre-Go 1 way was to flatten the data to a string first:

	inflight := map[string]bool{}

	inflight[br.builder + "-" + br.rev] = true

But with struct keys, you can avoid the allocation and have cleaner code:

	inflight := map[builderRev]bool{}

	inflight[br] = true


* Interfaces as map keys

An example of interface map keys from Docker's `broadcastwriter` package:

.code tricks/broadcastwriter/broadcastwriter.go /type/,/END/


* Structs and interfaces together as map keys

A (very) contrived example: (Don't do this! Ever!)

.play tricks/cons.go /type/,$


* Libraries

* sync/atomic

For a simple counter, you can use the `sync/atomic` package's
functions to make atomic updates without the lock.

	func AddInt64(addr *int64, delta int64) (new int64)
	func CompareAndSwapInt64(addr *int64, old, new int64) (swapped bool)
	func LoadInt64(addr *int64) (val int64)
	func StoreInt64(addr *int64, val int64)
	func SwapInt64(addr *int64, new int64) (old int64)

First, define the global variable (appropriately documented!):

	// viewCount must be updated atomically.
	var viewCount int64

Then increment it with `AddInt64`:

	count := atomic.AddInt64(&viewCount, 1)

The set are available for `Int32`, `Uint32`, `Int64`, `Uint64`, `Pointer`, and `Uintptr`.


* sync/atomic.Value

Another option for sharing state is `atomic.Value`.

For instance, to share configuration between many goroutines:

	type Config struct {
		Timeout time.Duration
	}

	var config atomic.Value

To set or update, use the `Store` method:
	
	config.Store(&Config{Timeout: 2*time.Second})

To read, each goroutine calls the `Load` method:

	cfg := config.Load().(*Config)

Note that storing different types in the same `Value` will cause a panic.


* sync/atomic.Value: how it works (1/5)

The `atomic.Value` primitive is the size of a single interface value:

	package atomic

	type Value struct {
		v interface{}
	}

An interface value is represented by the runtime as two pointers:
one for the type, and one for the value.

	// ifaceWords is interface{} internal representation.
	type ifaceWords struct {
		typ  unsafe.Pointer
		data unsafe.Pointer
	}

To load and store an interface value atomically, it operates on the parts of the interface value with `atomic.LoadPointer` and `atomic.StorePointer`.


* sync/atomic.Value: how it works (2/5)

The `Store` method first validates the input:

	func (v *Value) Store(x interface{}) {
		if x == nil {
			panic("sync/atomic: store of nil value into Value")
		}

Then uses `unsafe` to cast the current and new `interface{}` values to `ifaceWords`:

	// ...
		vp := (*ifaceWords)(unsafe.Pointer(v))
		xp := (*ifaceWords)(unsafe.Pointer(&x))

(This allows us to get at the internals of those interface values.)


* sync/atomic.Value: how it works (3/5)

Spin while loading the type field:

	// ...
		for {
			typ := LoadPointer(&vp.typ)

If it's `nil` the this is the first time the value has been stored.
Put a sentinel value (max uintptr) in the type field to "lock" it while we work with it:

	// ...
			if typ == nil {
				if !CompareAndSwapPointer(&vp.typ, nil, unsafe.Pointer(^uintptr(0))) {
					continue // Someone beat us to it. Wait.
				}

Store the data field, then the type field, and we're done!

	// ...
				StorePointer(&vp.data, xp.data)
				StorePointer(&vp.typ, xp.typ)
				return
			}

* sync/atomic.Value: how it works (4/5)

If this isn't the first store, check whether a store is already happening:

	// ...
			if uintptr(typ) == ^uintptr(0) {
				continue // First store in progress. Wait.
			}

Sanity check whether the type changed:

	// ...
			if typ != xp.typ {
				panic("sync/atomic: store of inconsistently typed value into Value")
			}

If the type field is what we expect, go ahead and atomically store the value:

	// ...
			StorePointer(&vp.data, xp.data)
			return
		}
	}

* sync/atomic.Value: how it works (5/5)

The `Load` method first loads the interface's type field:

	func (v *Value) Load() (x interface{}) {
		vp := (*ifaceWords)(unsafe.Pointer(v))
		typ := LoadPointer(&vp.typ)

Then, check whether a store has happened, or is happening:

	// ...
		if typ == nil || uintptr(typ) == ^uintptr(0) {
			return nil
		}

Otherwise, load the data field and return both type and data as a new interface value:

	// ...
		data := LoadPointer(&vp.data)
		xp := (*ifaceWords)(unsafe.Pointer(&x))
		xp.typ = typ
		xp.data = data
		return
	}


* A note on sync/atomic

Usually you don't want or need the stuff in this package.

Try channels or the `sync` package first.

You almost certainly shouldn't write code like the `atomic.Value` implementation.
(And I didn't show all of it; the real code has hooks into the runtime.)


* Tools


* Testing

* Subprocess tests

Sometimes you need to test the behavior of a process, not just a function.

.code tricks/subprocess/subprocess.go /func Crasher/,/^}/

To test this code, we invoke the test binary itself as a subprocess:

.code tricks/subprocess/subprocess_test.go /func TestCrasher/,/^}/


* Subprocess benchmarks (1/2)

Go's CPU and memory profilers report data for an entire process.
To profile just one side of a concurrent operation, you can use a sub-process.

This benchmark from the `net/http` package spawns a child process to make
requests to a server running in the main process.

	func BenchmarkServer(b *testing.B) {
		b.ReportAllocs()
		// Child process mode;
		if url := os.Getenv("TEST_BENCH_SERVER_URL"); url != "" {
			n, err := strconv.Atoi(os.Getenv("TEST_BENCH_CLIENT_N"))
			if err != nil {
				panic(err)
			}
			for i := 0; i < n; i++ {
				res, err := Get(url)
				// ...
			}
			os.Exit(0)
			return
		}
	// ...

* Subprocess benchmarks (2/2)

	// ...
		res := []byte("Hello world.\n")
		ts := httptest.NewServer(HandlerFunc(func(rw ResponseWriter, r *Request) {
			rw.Header().Set("Content-Type", "text/html; charset=utf-8")
			rw.Write(res)
		}))
		defer ts.Close()

		cmd := exec.Command(os.Args[0], "-test.run=XXXX", "-test.bench=BenchmarkServer$")
		cmd.Env = append([]string{
			fmt.Sprintf("TEST_BENCH_CLIENT_N=%d", b.N),
			fmt.Sprintf("TEST_BENCH_SERVER_URL=%s", ts.URL),
		}, os.Environ()...)
		if out, err := cmd.CombinedOutput(); err != nil {
			b.Errorf("Test failure: %v, with output: %s", err, out)
		}
	}

To run:

	$ go test -run=XX -bench=BenchmarkServer -benchtime=15s -cpuprofile=http.prof
	$ go tool pprof http.test http.prof


* go list

* go list

	$ go help list
	usage: go list [-e] [-f format] [-json] [build flags] [packages]

	List lists the packages named by the import paths, one per line.

Show the packages under a path:

	$ go list golang.org/x/oauth2/...
	golang.org/x/oauth2
	...
	golang.org/x/oauth2/vk

Show the standard library:

	$ go list std
	archive/tar
	...
	unsafe

Show all packages:

	$ go list all

* go list -json

The `-json` flag tells you everything the go tool knows about a package:

	$ go list -json bytes
	{
		"Dir": "/Users/adg/go/src/bytes",
		"ImportPath": "bytes",
		"Name": "bytes",
		"Doc": "Package bytes implements functions for the manipulation of byte slices.",
		"Target": "/Users/adg/go/pkg/darwin_amd64/bytes.a",
		"Goroot": true,
		"Standard": true,
		"Stale": true,
		"Root": "/Users/adg/go",
		"GoFiles": [
			"buffer.go",
			"bytes.go",
			"bytes_decl.go",
			"reader.go"
		],
		...
	}

It's easy to write programs that consume this data.

* go list's Package struct (1/3)

The go tool's documented `Package` struct describes all the possible fields:

	type Package struct {
		Dir           string // directory containing package sources
		ImportPath    string // import path of package in dir
		ImportComment string // path in import comment on package statement
		Name          string // package name
		Doc           string // package documentation string
		Target        string // install path
		Shlib         string // the shared library that contains this package (only set when -linkshared)
		Goroot        bool   // is this package in the Go root?
		Standard      bool   // is this package part of the standard Go library?
		Stale         bool   // would 'go install' do anything for this package?
		Root          string // Go root or Go path dir containing this package

		// (more fields on next slide)
	}

* go list's Package struct (2/3)

	type Package struct {
		// (more fields on previous slide)

		// Source files
		GoFiles        []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles)
		CgoFiles       []string // .go sources files that import "C"
		IgnoredGoFiles []string // .go sources ignored due to build constraints
		CFiles         []string // .c source files
		CXXFiles       []string // .cc, .cxx and .cpp source files
		MFiles         []string // .m source files
		HFiles         []string // .h, .hh, .hpp and .hxx source files
		SFiles         []string // .s source files
		SwigFiles      []string // .swig files
		SwigCXXFiles   []string // .swigcxx files
		SysoFiles      []string // .syso object files to add to archive

		// (more fields on next slide)
	}

* go list's Package struct (3/3)

	type Package struct {
		// (more fields on previous slide)

		// Cgo directives
		CgoCFLAGS    []string // cgo: flags for C compiler
		CgoCPPFLAGS  []string // cgo: flags for C preprocessor
		CgoCXXFLAGS  []string // cgo: flags for C++ compiler
		CgoLDFLAGS   []string // cgo: flags for linker
		CgoPkgConfig []string // cgo: pkg-config names

		// Dependency information
		Imports []string // import paths used by this package
		Deps    []string // all (recursively) imported dependencies

		// Error information
		Incomplete bool            // this package or a dependency has an error
		Error      *PackageError   // error loading package
		DepsErrors []*PackageError // errors loading dependencies

		TestGoFiles  []string // _test.go files in package
		TestImports  []string // imports from TestGoFiles
		XTestGoFiles []string // _test.go files outside package
		XTestImports []string // imports from XTestGoFiles
	}

* go list -f (1/2)

That's a ton of information, so what can we do with it?

The `-f` flag lets you use Go's `text/template` package to format the ouput.

Show package doc strings:

	$ go list -f '{{.Doc}}' golang.org/x/oauth2/...
	Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
	Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
	...

Show the Go files in a package:

	$ go list -f '{{.GoFiles}}' bytes
	[buffer.go bytes.go bytes_decl.go reader.go]

Make the output cleaner with a `join`:

	$ go list -f '{{join .GoFiles " "}}' bytes
	buffer.go bytes.go bytes_decl.go reader.go 

* go list -f (2/2)

With template logic we can test packages for certain conditions.

Find standard libraries that lack documentation:

	$ go list -f '{{if not .Doc}}{{.ImportPath}}{{end}}' std
	internal/format
	internal/trace

Find packages that depend (directly or indirectly) on a given package:

	$ go list -f '{{range .Deps}}{{if eq . "golang.org/x/oauth2"}}{{$.ImportPath}}{{end}}{{end}}' all
	golang.org/x/build/auth
	golang.org/x/build/buildlet
	golang.org/x/build/cmd/buildlet
	...

Find packages that are broken somehow (note `-e`):

	$ go list -e -f '{{with .Error}}{{.}}{{end}}' all
	package github.com/golang/oauth2: code in directory /Users/adg/src/github.com/golang/oauth2
		expects import "golang.org/x/oauth2"
	...

* go list and the shell

Things get interesting once we start using the shell.

For instance, we can use `go` `list` output as input to the go tool.

Test all packages except vendored packages:

	$ go test $(go list ./... | grep -v '/vendor/')

Test the dependencies of a specific package:

	$ go test $(go list -f '{{join .Deps " "}}' golang.org/x/oauth2)

The same, but don't test the standard library:

	$ go test $(
		go list -f '{{if (not .Goroot)}}{{.ImportPath}}{{end}}' $(
			go list -f '{{join .Deps " "}}' golang.org/x/oauth2
		)
	)

* go list printing line counts

	for pkg in $(go list golang.org/x/oauth2/...); do
		wc -l $(go list -f '{{range .GoFiles}}{{$.Dir}}/{{.}} {{end}}' $pkg) | \
			tail -1 | awk '{ print $1 " '$pkg'" }'
	done | sort -nr

The output:

	617 golang.org/x/oauth2/google
	600 golang.org/x/oauth2
	357 golang.org/x/oauth2/internal
	160 golang.org/x/oauth2/jws
	147 golang.org/x/oauth2/jwt
	112 golang.org/x/oauth2/clientcredentials
	22 golang.org/x/oauth2/paypal
	16 golang.org/x/oauth2/vk
	16 golang.org/x/oauth2/odnoklassniki
	16 golang.org/x/oauth2/linkedin
	16 golang.org/x/oauth2/github
	16 golang.org/x/oauth2/facebook

* go list generating dependency graphs

	(   echo "digraph G {"
		go list -f '{{range .Imports}}{{printf "\t%q -> %q;\n" $.ImportPath .}}{{end}}' \
			$(go list -f '{{join .Deps " "}}' time) time
		echo "}"
	) | dot -Tsvg -o time-deps.svg

.image tricks/time-deps.png 400 _


* And there's more!

There are many more fun corners of Go.
Can you find them all? :-)

Read the docs, explore, and have fun!