passes: add a statepoint insertion pass

This pass performs liveness analysis and attaches the liveness
information to statepoints, to support precise stack scan for
the Go GC. Statepoints are function calls. Currently a
non-moving GC is implemented.

This pass is based on LLVM's RewriteStatepointsForGC pass, with
the following modifications:

- We implement a non-moving GC, so gc.relocate is not necessary.
  Related code are removed.

- The original code only tracks live in-register values. For Go,
  the GC roots also include stack allocated objects. So the
  liveness analysis is extended with stack slot (alloca)
  liveness tracking.

- We encode the stack maps in the exception table. To do that,
  we rewrite all the statepoint calls to may-throw invoke, and
  attach the stack map to the landing pad as the type info. For
  each statepoint we attach a symbol go..stackmap.ID, and later
  the stack map generation code will fill in the content of the
  symbol with the actual stack map.

TODO: have some way to unit test

Change-Id: I166135eb9ac753686e6b47d9ae84f7a746fb933f
Reviewed-on: https://go-review.googlesource.com/c/137761
Reviewed-by: Than McIntosh <thanm@google.com>
5 files changed
tree: 509265cf4aa0e6e8c30c61392062587833c828f3
  1. .gitignore
  2. AUTHORS
  3. CMakeLists.txt
  4. CONTRIBUTORS
  5. LICENSE
  6. PATENTS
  7. README.md
  8. bridge/
  9. cmake/
  10. driver-main/
  11. driver/
  12. gotools/
  13. libgo/
  14. passes/
  15. tools/
  16. unittests/
README.md

Gollvm

Gollvm is an LLVM-based Go compiler. It incorporates “gofrontend” (a Go language front end written in C++ and shared with GCCGO), a bridge component (which translates from gofrontend IR to LLVM IR), and a driver that sends the resulting IR through the LLVM back end.

Gollvm is set up to be a subproject within the LLVM tools directory, similar to how things work for “clang” or “compiler-rt”: you check out a copy of the LLVM source tree, then within the LLVM tree you check out additional git repos.

Table of contents

FAQ

Building gollvm

Gollvm is currently in development -- releases are not yet available for download. Instructions for building gollvm follow.

Setting up a gollvm work area

To set up a work area for Gollvm, check out a copy of LLVM, the overlay the gollvm repo (and other associated dependencies) within the LLVM tools subdir, as follows:

// Here 'workarea' will contain a copy of the LLVM source tree and one or more build areas
% mkdir workarea
% cd workarea

// Sources
% git clone http://llvm.org/git/llvm.git
...
% cd llvm/tools
% git clone https://go.googlesource.com/gollvm
...
% cd gollvm
% git clone https://go.googlesource.com/gofrontend
...
% cd libgo
% git clone https://github.com/libffi/libffi.git
...
% git clone https://github.com/ianlancetaylor/libbacktrace.git
...
%

Building gollvm with cmake and ninja

You'll need to have an up-to-date copy of cmake on your system (3.6 or later vintage) to build Gollvm.

Create a build directory (separate from the source tree) and run ‘cmake’ within the build area to set up for the build. Assuming that ‘workarea’ is the directory created as above:

% cd workarea
% mkdir build-debug
% cd build-debug
% cmake -DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_LINKER=gold -G Ninja ../llvm
...
% ninja gollvm
...
%

This will build the various tools and libraries needed for Gollvm.

Installing gollvm

A gollvm installation will contain ‘llvm-goc’ (the compiler driver), the libgo standard Go libraries, and the standard Go tools (“go”, “vet”, “cgo”, etc).

The installation directory for gollvm needs to be specified when invoking cmake prior to the build:

% mkdir build.rel
% cd build.rel
% cmake -DCMAKE_INSTALL_PREFIX=/my/install/dir -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_LINKER=gold -G Ninja ../llvm

// Build all of gollvm
% ninja gollvm
...

// Install gollvm to "/my/install/dir"
% ninja install-gollvm

Using an installed copy of gollvm

Programs build with the Gollvm Go compiler default to shared linkage, meaning that they need to pick up the Go runtime library via LD_LIBRARY_PATH:

// Root of Gollvm install is /tmp/gollvm-install
% export LD_LIBRARY_PATH=/tmp/gollvm-install/lib64
% export PATH=/tmp/gollvm-install/bin:$PATH
% go run himom.go
hi mom!
%

Information for gollvm developers

Source code structure

Within <workarea>/llvm/tools/gollvm, the following directories are of interest:

.../llvm/tools/gollvm:

  • contains rules to build third party libraries needed for gollvm, along with common definitions for subdirs.

.../llvm/tools/gollvm/driver, .../llvm/tools/gollvm/driver-main:

  • contains build rules and source code for llvm-goc

.../llvm/tools/gollvm/gofrontend:

  • source code for gofrontend and libgo (note: no cmake files here)

.../llvm/tools/gollvm/bridge:

  • contains build rules for the libLLVMCppGoFrontEnd.a, a library that contains both the gofrontend code and the LLVM-specific middle layer (for example, the definition of the class Llvm_backend, which inherits from Backend).

.../llvm/tools/gollvm/libgo:

  • build rules and supporting infrastructure to build Gollvm's copy of the Go runtime and standard packages.

.../llvm/tools/gollvm/unittests:

  • source code for the unit tests

The llvm-goc program

The executable llvm-goc is the main compiler driver for gollvm; it functions as a compiler (consuming source for a Go package and producing an object file), an assembler, and/or a linker. While it is possible to build and run llvm-goc directly from the command line, in practice there is little point in doing this (better to build using “go build”, which will invoke llvm-goc on your behalf.

// From within <workarea>/build.opt:

% ninja llvm-goc
...
% cat micro.go
package foo
func Bar() int {
	return 1
}
% ./bin/llvm-goc -fgo-pkgpath=foo -O3 -S -o micro.s micro.go
%

Building and running the unit tests

Here are instructions on building and running the unit tests for the middle layer:

// From within <workarea>/build.opt:

// Build unit test
% ninja GoBackendCoreTests

// Run a unit test
% ./tools/gollvm/unittests/BackendCore/GoBackendCoreTests
[==========] Running 10 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from BackendCoreTests
[ RUN      ] BackendCoreTests.MakeBackend
[       OK ] BackendCoreTests.MakeBackend (1 ms)
[ RUN      ] BackendCoreTests.ScalarTypes
[       OK ] BackendCoreTests.ScalarTypes (0 ms)
[ RUN      ] BackendCoreTests.StructTypes
[       OK ] BackendCoreTests.StructTypes (1 ms)
[ RUN      ] BackendCoreTests.ComplexTypes
[       OK ] BackendCoreTests.ComplexTypes (0 ms)
[ RUN      ] BackendCoreTests.FunctionTypes
[       OK ] BackendCoreTests.FunctionTypes (0 ms)
[ RUN      ] BackendCoreTests.PlaceholderTypes
[       OK ] BackendCoreTests.PlaceholderTypes (0 ms)
[ RUN      ] BackendCoreTests.ArrayTypes
[       OK ] BackendCoreTests.ArrayTypes (0 ms)
[ RUN      ] BackendCoreTests.NamedTypes
[       OK ] BackendCoreTests.NamedTypes (0 ms)
[ RUN      ] BackendCoreTests.TypeUtils

...

[  PASSED  ] 10 tests.

The unit tests currently work by instantiating an LLVM Backend instance and making backend method calls (to mimic what the frontend would do), then inspects the results to make sure they are as expected. Here is an example:

TEST(BackendCoreTests, ComplexTypes) {
  LLVMContext C;

  Type *ft = Type::getFloatTy(C);
  Type *dt = Type::getDoubleTy(C);

  std::unique_ptr<Backend> be(go_get_backend(C));
  Btype *c32 = be->complex_type(64);
  ASSERT_TRUE(c32 != NULL);
  ASSERT_EQ(c32->type(), mkTwoFieldLLvmStruct(C, ft, ft));
  Btype *c64 = be->complex_type(128);
  ASSERT_TRUE(c64 != NULL);
  ASSERT_EQ(c64->type(), mkTwoFieldLLvmStruct(C, dt, dt));
}

The test above makes sure that the LLVM type we get as a result of calling Backend::complex_type() is kosher and matches up to expectations.

Building libgo (Go runtime and standard libraries)

To build the Go runtime and standard libraries, use the following:

// From within <workarea>/build.opt:

// Build Go runtime and standard libraries
% ninja libgo_all

This will compile static (*.a) and dynamic (*.so) versions of the library.

FAQ

Where should I post questions about gollvm?

Please send questions about gollvm to the golang-nuts mailing list. Posting questions to the issue tracker is generally not the right way to start discussions or get information.

Where should I file gollvm bugs?

Please file an issue on the golang issue tracker; please be sure to use “gollvm” somewhere in the headline.

How can I go about contributing to gollvm?

Please see the Go project guidelines at https://golang.org/doc/contribute.html. Changes to https://go.googlesource.com/gollvm can be made by any Go contributor; for changes to gofrontend see the gccgo guidelines.

Is gollvm a replacement for the main Go compiler? (gc)

Gollvm is not intended as a replacement for the main Go compiler -- the expectation is that the bulk of users will want to continue to use the main Go compiler due to its superior compilation speed, ease of use, broader functionality, and higher-performance runtime. Gollvm is intended to provide a Go compiler with a more powerful back end, enabling such benefits as better inlining, vectorization, register allocation, etc.

Which architectures and operating systems are supported for gollvm?

Gollvm is currently supported only for x86_64 Linux.

How does the gollvm runtime differ from the main Go runtime?

The main Go runtime supports generation of accurate stack maps, which allows the garbage collector to do precise stack scanning; gollvm does not yet support stack map generation (note that we're actively working on fixing this), hence for gollvm the garbage collector has to scan stacks conservatively (which can lead to longer scan times and increased memory usage). The main Go runtime compiles to a different calling convention, whereas Gollvm uses the standard C/C++ calling convention. There are many other smaller differences as well.

Shared linkage is the default for gollvm. How do I build non-shared?

Linking with “-static-libgo” will yield a binary that incorporates a full copy of the Go runtime. Example:

 % go build -gccgoflags -static-libgo myprogram.go

Note that this will increase binary size.

What command line options are supported for gollvm?

You can run ‘llvm-goc -help’ to see a full set of supported options. These can be passed to the compiler via ‘-gccgoflags’ option. Example:

% go build -gccgoflags -fno-inline mumble.go

How do I see the LLVM IR generated by gollvm?

The ‘llvm-goc’ command supports the -emit-llvm flag, however passing this option to a “go build” command is not practical, since the “go build” won't be expecting the compiler to emit LLVM bitcode or assembly.

A better recipe is to run “go build” with “-x -work” to capture the commands being executed, then rerun the llvm-goc command shown adding “-S -emit-llvm”. The resulting output will be an LLVM IR dump. Example:

% go build -work -x mypackage.go 1> transcript.txt 2>&1
% egrep '(WORK=|llvm-goc -c)' transcript.txt
WORK=/tmp/go-build887931787
/t/bin/llvm-goc -c -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build \
  -gno-record-gcc-switches -fgo-pkgpath=command-line-arguments \
  -fgo-relative-import-path=/mygopath/src/tmp -o $WORK/b001/_go_.o \
  -I $WORK/b001/_importcfgroot_ ./mypackage.go
% /t/bin/llvm-goc -c -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build \
  -gno-record-gcc-switches -fgo-pkgpath=command-line-arguments \
  -fgo-relative-import-path=/mygopath/src/tmp \
  -I $WORK/b001/_importcfgroot_ -o mypackage.ll -S -emit-llvm \
  ./mypackage.go
% ls -l mypackage.ll
...
%

What is the relationship between gollvm and gccgo?

Gollvm and gccgo share a common front end (gofrontend) and associated runtime (libgo), however each uses a separate back end. When using “go build”, the Go command currently treats gollvm as an instance of gccgo (hence the need to pass compile flags via “-gccgoflags”). This is expected to be temporary.

Can I use FDO or ThinLTO with gollvm?

There are plans to support FDO, AutoFDO, and ThinLTO for gollvm, however these features have not yet been implemented.

Can I use the race detector?

Gollvm does not support the Go race detector; please use the main Go compiler for this purpose.