blob: 1746d77bbc40e75efcabe3bbf28fc6b14c55b2a0 [file] [log] [blame]
Stupid Gopher Tricks
GolangUK
21 Aug 2015
Andrew Gerrand
adg@golang.org
* Video
A video of this talk was recorded at GolangUK in London.
.link https://www.youtube.com/watch?v=UECh7X07m6E Watch the talk on YouTube
* This talk
This talk is about things you might not know about Go.
Some of this stuff you can add to your Go vocabulary.
Other things are probably best left for special occasions.
* Language
* Type literals (1/2)
Here's a familiar type declaration:
type Foo struct {
i int
s string
}
The latter part of a type declaration is the *type*literal*:
struct {
i int
s string
}
Other examples of type literals are `int` and `[]string`,
which can also be declared as named types:
type Bar int
type Qux []string
* Type literals (2/2)
While we commonly use `int` and `[]string` in variable declarations:
var i int
var s []string
It is less common (but equally valid) to do the same with structs:
var t struct {
i int
s string
}
An unnamed struct literal is often called an *anonymous*struct*.
* Anonymous structs: template data
A common use is providing data to templates:
.play tricks/template.go /BEGIN/,/END/
You could also use `map[string]interface{}`,
but then you sacrifice performance and type safety.
* Anonymous structs: JSON (1/2)
The same technique can be used for encoding JSON objects:
.play tricks/json-encode.go /Marshal/,/Print/
And also decoding:
.play tricks/json-decode.go /data/,/Print/
* Anonymous structs: JSON (2/2)
Structs can be nested to describe more complex JSON objects:
.play tricks/json-nest.go /data/,/Print/
* Repeated literals and struct names
In repeated literals like slices and maps, Go lets you omit the inner type name:
.code tricks/repeated.go /BEGIN/,/END/
* Repeated literals and anonymous structs
Combined with anonymous structs, this convenience shortens the code dramatically:
.code tricks/repeated2.go /BEGIN/,/END/
* Anonymous structs: test cases (1/2)
These properties enable a nice way to express test cases:
.code tricks/string_test.go /TestIndex/,/^}/
* Anonymous structs: test cases (2/2)
You can go a step further and put the composite literal in the range statement itself:
.code tricks/string_test2.go /TestIndex/,/^}/
But this is harder to read.
* Embedded fields
A struct field that has no name is an *embedded*field*.
The embedded type's methods (and fields, if it is a struct)
are accessible as if they are part of the embedding struct.
.play tricks/embed.go /BEGIN/,$
* Anonymous structs: embedded mutex
Of course, you can embed fields in an anonymous struct.
It's common to protect a global variable with a mutex variable:
var (
viewCount int64
viewCountMu sync.Mutex
)
By embedding a mutex in an anonymous struct, we can group the related values:
var viewCount struct {
sync.Mutex
n int64
}
Users of `viewCount` access it like this:
viewCount.Lock()
viewCount.n++
viewCount.Unlock()
* Anonymous structs: implementing interfaces
And you can embed interfaces, too.
Here's a real example from [[https://camlistore.org/][Camlistore]]:
The function is expected to return a `ReadSeekCloser`,
but the programmer only had a string.
Anonymous struct (and its standard library friends) to the rescue!
return struct {
io.ReadSeeker
io.Closer
}{
io.NewSectionReader(strings.NewReader(s), 0, int64(len(s))),
ioutil.NopCloser(nil),
}
* Anonymous interfaces
Interfaces can be anonymous, the most common being `interface{}`.
But the interface needn't be empty:
.play tricks/anon-interface.go /var s/,/Println/
Useful for a sly type assertion (from `src/os/exec/exec_test.go`):
// Check that we can access methods of the underlying os.File.
if _, ok := stdin.(interface {
Fd() uintptr
}); !ok {
t.Error("can't access methods of underlying *os.File")
}
* Method values
A "method value" is what you get when you evaluate a method as an expression.
The result is a function value.
Evaluating a method from a _type_ yields a function:
.play tricks/method-values-1.go /var f/,/Stdout/
Evaluating a method from a _value_ creates a closure that holds that value:
.play tricks/method-values-2.go /var /,/Stdout/
* Method values: sync.Once
The `sync.Once` type is used to perform a task once with concurrency safety.
// Once is an object that will perform exactly one action.
type Once struct { /* Has unexported fields. */ }
func (o *Once) Do(f func())
This `LazyPrimes` type computes a slice of prime numbers the first time it is used:
.code tricks/method-once.go /type/,$
* Method values: HTTP handlers
You can use method values to implement multiple HTTP handlers with one type:
.code tricks/method-http.go /type Server/,$
* Method values: another example
In package `os/exec`, the `Cmd` type implements methods to set up
standard input, output, and error:
func (c *Cmd) stdin() (f *os.File, err error)
func (c *Cmd) stdout() (f *os.File, err error)
func (c *Cmd) stderr() (f *os.File, err error)
The caller handles each in the same way,
so it iterates over a slice of method values:
type F func(*Cmd) (*os.File, error)
for _, setupFd := range []F{(*Cmd).stdin, (*Cmd).stdout, (*Cmd).stderr} {
fd, err := setupFd(c)
if err != nil {
c.closeDescriptors(c.closeAfterStart)
c.closeDescriptors(c.closeAfterWait)
return err
}
c.childFiles = append(c.childFiles, fd)
}
* Comparable types
The Go spec defines a set of types as "comparable";
they may be compared with == and !=.
Bools, ints, floats, complex numbers, strings, pointers,
channels, *structs*, and *interfaces* are comparable.
.play tricks/compare.go /BEGIN/,/END/
A struct is comparable only if its fields are comparable:
.play tricks/compare2.go /BEGIN/,/END/
* Comparable types and map keys
Any comparable type may be used as a map key.
.code tricks/compare-map.go /BEGIN/,/END/
* Structs as map keys
An example from the Go continuous build infrastructure:
type builderRev struct {
builder, rev string
}
var br = builderRev{"linux-amd64", "0cd299"}
We track in-flight builds in a map.
The pre-Go 1 way was to flatten the data to a string first:
inflight := map[string]bool{}
inflight[br.builder + "-" + br.rev] = true
But with struct keys, you can avoid the allocation and have cleaner code:
inflight := map[builderRev]bool{}
inflight[br] = true
* Interfaces as map keys
An example of interface map keys from Docker's `broadcastwriter` package:
.code tricks/broadcastwriter/broadcastwriter.go /type/,/END/
* Structs and interfaces together as map keys
A (very) contrived example: (Don't do this! Ever!)
.play tricks/cons.go /type/,$
* Libraries
* sync/atomic
For a simple counter, you can use the `sync/atomic` package's
functions to make atomic updates without the lock.
func AddInt64(addr *int64, delta int64) (new int64)
func CompareAndSwapInt64(addr *int64, old, new int64) (swapped bool)
func LoadInt64(addr *int64) (val int64)
func StoreInt64(addr *int64, val int64)
func SwapInt64(addr *int64, new int64) (old int64)
First, define the global variable (appropriately documented!):
// viewCount must be updated atomically.
var viewCount int64
Then increment it with `AddInt64`:
count := atomic.AddInt64(&viewCount, 1)
The set are available for `Int32`, `Uint32`, `Int64`, `Uint64`, `Pointer`, and `Uintptr`.
* sync/atomic.Value
Another option for sharing state is `atomic.Value`.
For instance, to share configuration between many goroutines:
type Config struct {
Timeout time.Duration
}
var config atomic.Value
To set or update, use the `Store` method:
config.Store(&Config{Timeout: 2*time.Second})
To read, each goroutine calls the `Load` method:
cfg := config.Load().(*Config)
Note that storing different types in the same `Value` will cause a panic.
* sync/atomic.Value: how it works (1/5)
The `atomic.Value` primitive is the size of a single interface value:
package atomic
type Value struct {
v interface{}
}
An interface value is represented by the runtime as two pointers:
one for the type, and one for the value.
// ifaceWords is interface{} internal representation.
type ifaceWords struct {
typ unsafe.Pointer
data unsafe.Pointer
}
To load and store an interface value atomically, it operates on the parts of the interface value with `atomic.LoadPointer` and `atomic.StorePointer`.
* sync/atomic.Value: how it works (2/5)
The `Store` method first validates the input:
func (v *Value) Store(x interface{}) {
if x == nil {
panic("sync/atomic: store of nil value into Value")
}
Then uses `unsafe` to cast the current and new `interface{}` values to `ifaceWords`:
// ...
vp := (*ifaceWords)(unsafe.Pointer(v))
xp := (*ifaceWords)(unsafe.Pointer(&x))
(This allows us to get at the internals of those interface values.)
* sync/atomic.Value: how it works (3/5)
Spin while loading the type field:
// ...
for {
typ := LoadPointer(&vp.typ)
If it's `nil` the this is the first time the value has been stored.
Put a sentinel value (max uintptr) in the type field to "lock" it while we work with it:
// ...
if typ == nil {
if !CompareAndSwapPointer(&vp.typ, nil, unsafe.Pointer(^uintptr(0))) {
continue // Someone beat us to it. Wait.
}
Store the data field, then the type field, and we're done!
// ...
StorePointer(&vp.data, xp.data)
StorePointer(&vp.typ, xp.typ)
return
}
* sync/atomic.Value: how it works (4/5)
If this isn't the first store, check whether a store is already happening:
// ...
if uintptr(typ) == ^uintptr(0) {
continue // First store in progress. Wait.
}
Sanity check whether the type changed:
// ...
if typ != xp.typ {
panic("sync/atomic: store of inconsistently typed value into Value")
}
If the type field is what we expect, go ahead and atomically store the value:
// ...
StorePointer(&vp.data, xp.data)
return
}
}
* sync/atomic.Value: how it works (5/5)
The `Load` method first loads the interface's type field:
func (v *Value) Load() (x interface{}) {
vp := (*ifaceWords)(unsafe.Pointer(v))
typ := LoadPointer(&vp.typ)
Then, check whether a store has happened, or is happening:
// ...
if typ == nil || uintptr(typ) == ^uintptr(0) {
return nil
}
Otherwise, load the data field and return both type and data as a new interface value:
// ...
data := LoadPointer(&vp.data)
xp := (*ifaceWords)(unsafe.Pointer(&x))
xp.typ = typ
xp.data = data
return
}
* A note on sync/atomic
Usually you don't want or need the stuff in this package.
Try channels or the `sync` package first.
You almost certainly shouldn't write code like the `atomic.Value` implementation.
(And I didn't show all of it; the real code has hooks into the runtime.)
* Tools
* Testing
* Subprocess tests
Sometimes you need to test the behavior of a process, not just a function.
.code tricks/subprocess/subprocess.go /func Crasher/,/^}/
To test this code, we invoke the test binary itself as a subprocess:
.code tricks/subprocess/subprocess_test.go /func TestCrasher/,/^}/
* Subprocess benchmarks (1/2)
Go's CPU and memory profilers report data for an entire process.
To profile just one side of a concurrent operation, you can use a sub-process.
This benchmark from the `net/http` package spawns a child process to make
requests to a server running in the main process.
func BenchmarkServer(b *testing.B) {
b.ReportAllocs()
// Child process mode;
if url := os.Getenv("TEST_BENCH_SERVER_URL"); url != "" {
n, err := strconv.Atoi(os.Getenv("TEST_BENCH_CLIENT_N"))
if err != nil {
panic(err)
}
for i := 0; i < n; i++ {
res, err := Get(url)
// ...
}
os.Exit(0)
return
}
// ...
* Subprocess benchmarks (2/2)
// ...
res := []byte("Hello world.\n")
ts := httptest.NewServer(HandlerFunc(func(rw ResponseWriter, r *Request) {
rw.Header().Set("Content-Type", "text/html; charset=utf-8")
rw.Write(res)
}))
defer ts.Close()
cmd := exec.Command(os.Args[0], "-test.run=XXXX", "-test.bench=BenchmarkServer$")
cmd.Env = append([]string{
fmt.Sprintf("TEST_BENCH_CLIENT_N=%d", b.N),
fmt.Sprintf("TEST_BENCH_SERVER_URL=%s", ts.URL),
}, os.Environ()...)
if out, err := cmd.CombinedOutput(); err != nil {
b.Errorf("Test failure: %v, with output: %s", err, out)
}
}
To run:
$ go test -run=XX -bench=BenchmarkServer -benchtime=15s -cpuprofile=http.prof
$ go tool pprof http.test http.prof
* go list
* go list
$ go help list
usage: go list [-e] [-f format] [-json] [build flags] [packages]
List lists the packages named by the import paths, one per line.
Show the packages under a path:
$ go list golang.org/x/oauth2/...
golang.org/x/oauth2
...
golang.org/x/oauth2/vk
Show the standard library:
$ go list std
archive/tar
...
unsafe
Show all packages:
$ go list all
* go list -json
The `-json` flag tells you everything the go tool knows about a package:
$ go list -json bytes
{
"Dir": "/Users/adg/go/src/bytes",
"ImportPath": "bytes",
"Name": "bytes",
"Doc": "Package bytes implements functions for the manipulation of byte slices.",
"Target": "/Users/adg/go/pkg/darwin_amd64/bytes.a",
"Goroot": true,
"Standard": true,
"Stale": true,
"Root": "/Users/adg/go",
"GoFiles": [
"buffer.go",
"bytes.go",
"bytes_decl.go",
"reader.go"
],
...
}
It's easy to write programs that consume this data.
* go list's Package struct (1/3)
The go tool's documented `Package` struct describes all the possible fields:
type Package struct {
Dir string // directory containing package sources
ImportPath string // import path of package in dir
ImportComment string // path in import comment on package statement
Name string // package name
Doc string // package documentation string
Target string // install path
Shlib string // the shared library that contains this package (only set when -linkshared)
Goroot bool // is this package in the Go root?
Standard bool // is this package part of the standard Go library?
Stale bool // would 'go install' do anything for this package?
Root string // Go root or Go path dir containing this package
// (more fields on next slide)
}
* go list's Package struct (2/3)
type Package struct {
// (more fields on previous slide)
// Source files
GoFiles []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles)
CgoFiles []string // .go sources files that import "C"
IgnoredGoFiles []string // .go sources ignored due to build constraints
CFiles []string // .c source files
CXXFiles []string // .cc, .cxx and .cpp source files
MFiles []string // .m source files
HFiles []string // .h, .hh, .hpp and .hxx source files
SFiles []string // .s source files
SwigFiles []string // .swig files
SwigCXXFiles []string // .swigcxx files
SysoFiles []string // .syso object files to add to archive
// (more fields on next slide)
}
* go list's Package struct (3/3)
type Package struct {
// (more fields on previous slide)
// Cgo directives
CgoCFLAGS []string // cgo: flags for C compiler
CgoCPPFLAGS []string // cgo: flags for C preprocessor
CgoCXXFLAGS []string // cgo: flags for C++ compiler
CgoLDFLAGS []string // cgo: flags for linker
CgoPkgConfig []string // cgo: pkg-config names
// Dependency information
Imports []string // import paths used by this package
Deps []string // all (recursively) imported dependencies
// Error information
Incomplete bool // this package or a dependency has an error
Error *PackageError // error loading package
DepsErrors []*PackageError // errors loading dependencies
TestGoFiles []string // _test.go files in package
TestImports []string // imports from TestGoFiles
XTestGoFiles []string // _test.go files outside package
XTestImports []string // imports from XTestGoFiles
}
* go list -f (1/2)
That's a ton of information, so what can we do with it?
The `-f` flag lets you use Go's `text/template` package to format the ouput.
Show package doc strings:
$ go list -f '{{.Doc}}' golang.org/x/oauth2/...
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
...
Show the Go files in a package:
$ go list -f '{{.GoFiles}}' bytes
[buffer.go bytes.go bytes_decl.go reader.go]
Make the output cleaner with a `join`:
$ go list -f '{{join .GoFiles " "}}' bytes
buffer.go bytes.go bytes_decl.go reader.go
* go list -f (2/2)
With template logic we can test packages for certain conditions.
Find standard libraries that lack documentation:
$ go list -f '{{if not .Doc}}{{.ImportPath}}{{end}}' std
internal/format
internal/trace
Find packages that depend (directly or indirectly) on a given package:
$ go list -f '{{range .Deps}}{{if eq . "golang.org/x/oauth2"}}{{$.ImportPath}}{{end}}{{end}}' all
golang.org/x/build/auth
golang.org/x/build/buildlet
golang.org/x/build/cmd/buildlet
...
Find packages that are broken somehow (note `-e`):
$ go list -e -f '{{with .Error}}{{.}}{{end}}' all
package github.com/golang/oauth2: code in directory /Users/adg/src/github.com/golang/oauth2
expects import "golang.org/x/oauth2"
...
* go list and the shell
Things get interesting once we start using the shell.
For instance, we can use `go` `list` output as input to the go tool.
Test all packages except vendored packages:
$ go test $(go list ./... | grep -v '/vendor/')
Test the dependencies of a specific package:
$ go test $(go list -f '{{join .Deps " "}}' golang.org/x/oauth2)
The same, but don't test the standard library:
$ go test $(
go list -f '{{if (not .Goroot)}}{{.ImportPath}}{{end}}' $(
go list -f '{{join .Deps " "}}' golang.org/x/oauth2
)
)
* go list printing line counts
for pkg in $(go list golang.org/x/oauth2/...); do
wc -l $(go list -f '{{range .GoFiles}}{{$.Dir}}/{{.}} {{end}}' $pkg) | \
tail -1 | awk '{ print $1 " '$pkg'" }'
done | sort -nr
The output:
617 golang.org/x/oauth2/google
600 golang.org/x/oauth2
357 golang.org/x/oauth2/internal
160 golang.org/x/oauth2/jws
147 golang.org/x/oauth2/jwt
112 golang.org/x/oauth2/clientcredentials
22 golang.org/x/oauth2/paypal
16 golang.org/x/oauth2/vk
16 golang.org/x/oauth2/odnoklassniki
16 golang.org/x/oauth2/linkedin
16 golang.org/x/oauth2/github
16 golang.org/x/oauth2/facebook
* go list generating dependency graphs
( echo "digraph G {"
go list -f '{{range .Imports}}{{printf "\t%q -> %q;\n" $.ImportPath .}}{{end}}' \
$(go list -f '{{join .Deps " "}}' time) time
echo "}"
) | dot -Tsvg -o time-deps.svg
.image tricks/time-deps.png 400 _
* And there's more!
There are many more fun corners of Go.
Can you find them all? :-)
Read the docs, explore, and have fun!