blob: 4a53e23d04e4f9009a1951a8a40818dae911a591 [file] [log] [blame]
Static analysis tools
for Go code comprehension and refactoring
golang nyc meetup
13 Nov 2014
Alan Donovan
Google, New York
adonovan@google.com
* Video
This talk was presented at the GothamGo Kickoff Meetup in New York City, November 2014.
.link http://vimeo.com/114736889 Watch the talk on Vimeo
* Introduction
Programmers say "writing code", but most of that time is spent _reading_
Actually: reading code, and making logical deductions
Machines are good at reading and logic
*Machines* *should* *be* *helping* *us* *read* *code.*
And write it! Refactoring can be tedious and error-prone.
* Outline
- Code comprehension tools: `oracle`, `godoc`
- Analysis technology
- Refactoring tools: `gorename`, `eg`
* Demo: the Go oracle
Supports interactive, editor-integrated queries:
- where is this thing defined?
- what are the methods of this type?
- what are the free variables of this selection?
- what might this expression point to?
- what dynamic types might this interface or `reflect.Value` contain?
* Demo: godoc analysis features
.link /lib/godoc/analysis/help.html godoc -analysis=type,pointer
Package view
- method set and _implements_ relation for every type
- internal call graph of every package
Source code view
- kind, type, definition of every identifier
- method set and _implements_ relation for every type
- peers of every channel send/receive
- callers of every function
- callees of every call site (even dynamic)
* How it works
* Libraries and tools
.image static-analysis/tools.svg
Mostly in `golang.org/x/tools` repo
* go/types: the Go type checker
Computes mappings from:
- identifiers to definitions
- constant expressions to values
- expressions to types
Many subtle corners:
- method set computation
- recursive interfaces
- shifts
Making it correct, fast, and clean was a substantial project
.link https://pkg.go.dev/golang.org/x/tools/go/types golang.org/x/tools/go/types
Author: Robert Griesemer
* go/ssa: intermediate representation (IR)
Typed abstract syntax trees (ASTs) are too varied and subtle to analyze directly
Programs are lowered into Static Single-Assignment form (SSA):
- simplifies dataflow analyses since _reaching_ _definitions_ are implicit
- invented 1991, now mainstream (gcc, llvm)
All Go programs can be expressed using only ~30 basic instructions
Simple, explicit, high-level, high source fidelity
.link https://pkg.go.dev/golang.org/x/tools/go/ssa golang.org/x/tools/go/ssa
The llgo project is using go/ssa as a front-end for LLVM
* Demo: ssadump
# Simple fibonacci function
# % ssadump -build=F fib.go basic
# % ssadump -build=FD fib.go debug info
# % ssadump -test -run unicode toy interpreter
* go/pointer: pointer analysis
Pointers complicate reasoning about program behaviour
- functions may be called dynamically
- a variable may be updated and accessed via different names ("aliases")
- runtime types are values too: `interface`, `reflect.Type`
We use *pointer* *analysis* to answer the question:
which variables might this pointer point to?
.link https://pkg.go.dev/golang.org/x/tools/go/pointer golang.org/x/tools/go/pointer
# comment on go's appropriateness for this analysis:
# (closed program---no dlopen, classloading, no generics, typesafe)
* Pointer analysis
Let `pts(p)` be the _points-to_ _set_ of pointer p
Its elements are abstract variables: globals, locals, unnamed (`new(T)`)
Overview:
- analyze each SSA statement in the whole program
- generate appropriate constraints on the points-to set of each variable
Statement Constraint
y = &x pts(y) {x}
y = x pts(y) pts(x) "inclusion-based"
*y = x o pts(y). pts(o) pts(x)
y = *x o pts(x). pts(y) pts(o)
y = &x->f \
y = x.(T) } not shown
reflection /
- solve the constraint system
* Pointer analysis: constraint generation
All Go operations can be expressed in these constraints
Function, map, slice and channel ops all simplify to struct ops
Treatment of `unsafe.Pointer` conversion is unsound
But we don't care! This isn't a compiler
The constraint system is:
- *field-sensitive*: struct fields x.f and x.g have distinct solutions
- *flow-insensitive*: the order of statements is ignored
- *context-insensitive*: each function is analyzed only once
- *whole-program*: Go source is required for all dependencies
- *inclusion-based*: i.e. Andersen's analysis
Optimization: don't make constraints for "uninteresting" types
such as types not related to a one-shot Oracle query
* Pointer analysis: pre-solver optimizations
Constraint system for 124KLoC program (oracle) has:
254,000 variables
184,000 equations
Solving phase is in O(|v|³), so pre-solver optimisation is crucial
We can bring this down to:
88,000 variables
65,000 equations
* Pointer Equivalence by Hash-Value Numbering (Hardekopf & Lin '07)
p = &x a = &x if ... {
q = p b = a p, q = &x, &y
r = q c = b } else {
s = r if ... { a = c } p, q = &y, &x
}
.image static-analysis/hvn.svg
Hash-Value Numbering
* Pointer analysis: solving
It's transitive closure of a graph, essentially
Lots of optimizations, for example:
_sparse_bit_vectors_, a very compact representation for points-to sets
.link https://pkg.go.dev/golang.org/x/tools/container/ints golang.org/x/tools/container/ints
Solver log is >1GB. Debugging is fun.
#* Sparse bit vector
#`golang.org/x/tools/container/ints`
#
#.image sparsebitvector.svg
#
#Very compact representation of sparse `set<int>`
#Doubly-linked list of (offset int, data [256]bit) blocks.
* Call graph
The *call* *graph* can be read directly from the solution
The `golang.org/x/tools/cmd/callgraph` tool prints raw call graphs
Demo (time permitting)
* Refactoring tools
* gorename: precise, type-safe identifier renaming
A refactoring tool, usable from...
- the command line
% gorename -from bytes.Buffer.Len -to Size
Renamed 105 occurrences in 42 files in 33 packages.
- many editors ([[https://code.google.com/p/go/source/browse/refactor/rename/rename.el?repo=tools&r=511801758bb9a0b83f9bf931342889fbedbc9b48][Emacs]], [[http://quick.as/dgof2p7][Vim]], [[https://github.com/smartystreets/sublime-gorename][Sublime]], [[https://github.com/davidrjenni/agorn][Acme]])
All renamings are reversible
Sound: either a conflict is reported, or the refactoring preserves behaviour*
*except reflection
.link https://pkg.go.dev/golang.org/x/tools/cmd/gorename golang.org/x/tools/cmd/gorename
* Demo: gorename
* Example-based refactoring with eg
A Go port of the Java _Refaster_ tool
A template specifies the pattern and replacement as Go expressions:
package P
import ( "errors"; "fmt" )
func before(s string) error { return fmt.Errorf("%s", s) }
func after(s string) error { return errors.New(s) }
Parameters are _wildcards_
% eg -t template.go <package> ...
.link https://pkg.go.dev/golang.org/x/tools/cmd/eg golang.org/x/tools/cmd/eg
* Demo: eg
* Conclusion
From the outset, Go has been renowned for its tools: `go`, `gofmt`
We have many building blocks for sophisticated source analysis tools
What should we build next?