gollvm: sync with trunk

Sync with trunk at e53d4869a0d, part 1.

This update required quite a few changes. Highlights:

- the LLVM LIRBuilder class now has even more paths that assume that
  you are inserting instructions into an existing basic block parented
  by an existing function, as opposed to generating them without a
  parent. This required converting over all of the builder uses in the
  bridge to use the BlockLIRBuilder, which works around these issues.
  After this change the BinstructionsLIRBuilder helper is no longer
  referenced, so it has been removed.

- convert to using llvm::MaybeAlign as opposed to raw unsigned values
  in places where alignment is set on variables, etc.

- adapt to changes in the LLVM value type class hierarchy (no more
  SequentialType, new sibling types for fixed and variable length
  vector types)

- adapt to some signature changes in LLVM instruction creation
  functions (typically to be more explicit about types)

This CL takes care of llvm-goc, gotools, and libgo, however a lot
of the unit tests need to be remastered (this will be in a follow
on CL).

Updates golang/go#38728.

Change-Id: I491fe30edb4fb3ee9d6e509ebd8b2777e0373b8f
Reviewed-on: https://go-review.googlesource.com/c/gollvm/+/232497
Reviewed-by: eric fang <eric.fang@arm.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
11 files changed
tree: 2ab4a587fd457e2fa1c690623b5546bedd108896
  1. bridge/
  2. cmake/
  3. driver/
  4. driver-main/
  5. gotools/
  6. libgo/
  7. passes/
  8. tools/
  9. unittests/
  10. .gitignore
  11. AUTHORS
  12. CMakeLists.txt
  13. CONTRIBUTORS
  14. LICENSE
  15. PATENTS
  16. README.md
README.md

Gollvm

Gollvm is an LLVM-based Go compiler. It incorporates “gofrontend” (a Go language front end written in C++ and shared with GCCGO), a bridge component (which translates from gofrontend IR to LLVM IR), and a driver that sends the resulting IR through the LLVM back end.

Gollvm is set up to be a subproject within the LLVM tools directory, similar to how things work for “clang” or “compiler-rt”: you check out a copy of the LLVM source tree, then within the LLVM tree you check out additional git repos.

Table of contents

FAQ

Building gollvm

Gollvm is currently in development -- releases are not yet available for download. Instructions for building gollvm follow.

Setting up a gollvm work area

To set up a work area for Gollvm, check out a copy of LLVM, the overlay the gollvm repo (and other associated dependencies) within the LLVM tools subdir, as follows:

// Here 'workarea' will contain a copy of the LLVM source tree and one or more build areas
% mkdir workarea
% cd workarea

// Sources
% git clone https://github.com/llvm/llvm-project.git
...
% cd llvm-project/llvm/tools
% git clone https://go.googlesource.com/gollvm
...
% cd gollvm
% git clone https://go.googlesource.com/gofrontend
...
% cd libgo
% git clone https://github.com/libffi/libffi.git
...
% git clone https://github.com/ianlancetaylor/libbacktrace.git
...
%

Building gollvm with cmake and ninja

You'll need to have an up-to-date copy of cmake on your system (3.6 or later vintage) to build Gollvm, as well as a C/C++ compiler (V5.0 or later for Clang, or V6.0 or later of GCC).

Create a build directory (separate from the source tree) and run ‘cmake’ within the build area to set up for the build. Assuming that ‘workarea’ is the directory created as above:

% cd workarea
% mkdir build-debug
% cd build-debug
% cmake -DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_LINKER=gold -G Ninja ../llvm-project/llvm
...
% ninja gollvm
...
%

This will build the various tools and libraries needed for Gollvm. To select a specific C/C++ compiler for the build, you can use the “-DCMAKE_C_COMPILER” and “-DCMAKE_CXX_COMPILER” options to select your desired C/C++ compiler when invoking cmake (details here).

Installing gollvm

A gollvm installation will contain ‘llvm-goc’ (the compiler driver), the libgo standard Go libraries, and the standard Go tools (“go”, “vet”, “cgo”, etc).

The installation directory for gollvm needs to be specified when invoking cmake prior to the build:

% mkdir build.rel
% cd build.rel
% cmake -DCMAKE_INSTALL_PREFIX=/my/install/dir -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_LINKER=gold -G Ninja ../llvm-project/llvm

// Build all of gollvm
% ninja gollvm
...

// Install gollvm to "/my/install/dir"
% ninja install-gollvm

Using an installed copy of gollvm

Programs build with the Gollvm Go compiler default to shared linkage, meaning that they need to pick up the Go runtime library via LD_LIBRARY_PATH:

// Root of Gollvm install is /tmp/gollvm-install
% export LD_LIBRARY_PATH=/tmp/gollvm-install/lib64
% export PATH=/tmp/gollvm-install/bin:$PATH
% go run himom.go
hi mom!
%

Information for gollvm developers

Source code structure

Within <workarea>/llvm/tools/gollvm, the following directories are of interest:

.../llvm/tools/gollvm:

  • contains rules to build third party libraries needed for gollvm, along with common definitions for subdirs.

.../llvm/tools/gollvm/driver, .../llvm/tools/gollvm/driver-main:

  • contains build rules and source code for llvm-goc

.../llvm/tools/gollvm/gofrontend:

  • source code for gofrontend and libgo (note: no cmake files here)

.../llvm/tools/gollvm/bridge:

  • contains build rules for the libLLVMCppGoFrontEnd.a, a library that contains both the gofrontend code and the LLVM-specific middle layer (for example, the definition of the class Llvm_backend, which inherits from Backend).

.../llvm/tools/gollvm/libgo:

  • build rules and supporting infrastructure to build Gollvm's copy of the Go runtime and standard packages.

.../llvm/tools/gollvm/unittests:

  • source code for the unit tests

The llvm-goc program

The executable llvm-goc is the main compiler driver for gollvm; it functions as a compiler (consuming source for a Go package and producing an object file), an assembler, and/or a linker. While it is possible to build and run llvm-goc directly from the command line, in practice there is little point in doing this (better to build using “go build”, which will invoke llvm-goc on your behalf.

// From within <workarea>/build.opt:

% ninja llvm-goc
...
% cat micro.go
package foo
func Bar() int {
	return 1
}
% ./bin/llvm-goc -fgo-pkgpath=foo -O3 -S -o micro.s micro.go
%

Building and running the unit tests

Here are instructions on building and running the unit tests for the middle layer:

// From within <workarea>/build.opt:

// Build unit test
% ninja GoBackendCoreTests

// Run a unit test
% ./tools/gollvm/unittests/BackendCore/GoBackendCoreTests
[==========] Running 10 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 9 tests from BackendCoreTests
[ RUN      ] BackendCoreTests.MakeBackend
[       OK ] BackendCoreTests.MakeBackend (1 ms)
[ RUN      ] BackendCoreTests.ScalarTypes
[       OK ] BackendCoreTests.ScalarTypes (0 ms)
[ RUN      ] BackendCoreTests.StructTypes
[       OK ] BackendCoreTests.StructTypes (1 ms)
[ RUN      ] BackendCoreTests.ComplexTypes
[       OK ] BackendCoreTests.ComplexTypes (0 ms)
[ RUN      ] BackendCoreTests.FunctionTypes
[       OK ] BackendCoreTests.FunctionTypes (0 ms)
[ RUN      ] BackendCoreTests.PlaceholderTypes
[       OK ] BackendCoreTests.PlaceholderTypes (0 ms)
[ RUN      ] BackendCoreTests.ArrayTypes
[       OK ] BackendCoreTests.ArrayTypes (0 ms)
[ RUN      ] BackendCoreTests.NamedTypes
[       OK ] BackendCoreTests.NamedTypes (0 ms)
[ RUN      ] BackendCoreTests.TypeUtils

...

[  PASSED  ] 10 tests.

The unit tests currently work by instantiating an LLVM Backend instance and making backend method calls (to mimic what the frontend would do), then inspects the results to make sure they are as expected. Here is an example:

TEST(BackendCoreTests, ComplexTypes) {
  LLVMContext C;

  Type *ft = Type::getFloatTy(C);
  Type *dt = Type::getDoubleTy(C);

  std::unique_ptr<Backend> be(go_get_backend(C, llvm::CallingConv::X86_64_SysV));
  Btype *c32 = be->complex_type(64);
  ASSERT_TRUE(c32 != NULL);
  ASSERT_EQ(c32->type(), mkTwoFieldLLvmStruct(C, ft, ft));
  Btype *c64 = be->complex_type(128);
  ASSERT_TRUE(c64 != NULL);
  ASSERT_EQ(c64->type(), mkTwoFieldLLvmStruct(C, dt, dt));
}

The test above makes sure that the LLVM type we get as a result of calling Backend::complex_type() is kosher and matches up to expectations.

Building libgo (Go runtime and standard libraries)

To build the Go runtime and standard libraries, use the following:

// From within <workarea>/build.opt:

// Build Go runtime and standard libraries
% ninja libgo_all

This will compile static (*.a) and dynamic (*.so) versions of the library.

FAQ

Where should I post questions about gollvm?

Please send questions about gollvm to the golang-nuts mailing list. Posting questions to the issue tracker is generally not the right way to start discussions or get information.

Where should I file gollvm bugs?

Please file an issue on the golang issue tracker; please be sure to use “gollvm” somewhere in the headline.

How can I go about contributing to gollvm?

Please see the Go project guidelines at https://golang.org/doc/contribute.html. Changes to https://go.googlesource.com/gollvm can be made by any Go contributor; for changes to gofrontend see the gccgo guidelines.

Is gollvm a replacement for the main Go compiler? (gc)

Gollvm is not intended as a replacement for the main Go compiler -- the expectation is that the bulk of users will want to continue to use the main Go compiler due to its superior compilation speed, ease of use, broader functionality, and higher-performance runtime. Gollvm is intended to provide a Go compiler with a more powerful back end, enabling such benefits as better inlining, vectorization, register allocation, etc.

Which architectures and operating systems are supported for gollvm?

Gollvm is currently supported only for x86_64 and aarch64 Linux.

How does the gollvm runtime differ from the main Go runtime?

The main Go runtime supports generation of accurate stack maps, which allows the garbage collector to do precise stack scanning; gollvm does not yet support stack map generation (note that we're actively working on fixing this), hence for gollvm the garbage collector has to scan stacks conservatively (which can lead to longer scan times and increased memory usage). The main Go runtime compiles to a different calling convention, whereas Gollvm uses the standard C/C++ calling convention. There are many other smaller differences as well.

Shared linkage is the default for gollvm. How do I build non-shared?

Linking with “-static-libgo” will yield a binary that incorporates a full copy of the Go runtime. Example:

 % go build -gccgoflags -static-libgo myprogram.go

Note that this will increase binary size.

What command line options are supported for gollvm?

You can run ‘llvm-goc -help’ to see a full set of supported options. These can be passed to the compiler via ‘-gccgoflags’ option. Example:

% go build -gccgoflags -fno-inline mumble.go

How do I see the LLVM IR generated by gollvm?

The ‘llvm-goc’ command supports the -emit-llvm flag, however passing this option to a “go build” command is not practical, since the “go build” won't be expecting the compiler to emit LLVM bitcode or assembly.

A better recipe is to run “go build” with “-x -work” to capture the commands being executed, then rerun the llvm-goc command shown adding “-S -emit-llvm”. The resulting output will be an LLVM IR dump. Example:

% go build -work -x mypackage.go 1> transcript.txt 2>&1
% egrep '(WORK=|llvm-goc -c)' transcript.txt
WORK=/tmp/go-build887931787
/t/bin/llvm-goc -c -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build \
  -gno-record-gcc-switches -fgo-pkgpath=command-line-arguments \
  -fgo-relative-import-path=/mygopath/src/tmp -o $WORK/b001/_go_.o \
  -I $WORK/b001/_importcfgroot_ ./mypackage.go
% /t/bin/llvm-goc -c -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build \
  -gno-record-gcc-switches -fgo-pkgpath=command-line-arguments \
  -fgo-relative-import-path=/mygopath/src/tmp \
  -I $WORK/b001/_importcfgroot_ -o mypackage.ll -S -emit-llvm \
  ./mypackage.go
% ls -l mypackage.ll
...
%

What is the relationship between gollvm and gccgo?

Gollvm and gccgo share a common front end (gofrontend) and associated runtime (libgo), however each uses a separate back end. When using “go build”, the Go command currently treats gollvm as an instance of gccgo (hence the need to pass compile flags via “-gccgoflags”). This is expected to be temporary.

Can I use FDO or ThinLTO with gollvm?

There are plans to support FDO, AutoFDO, and ThinLTO for gollvm, however these features have not yet been implemented.

Can I use the race detector?

Gollvm does not support the Go race detector; please use the main Go compiler for this purpose.

I am seeing “undefined symbol: __get_cpuid_count” from my gollvm install

The Gollvm build procedure requires an up-to-date C/C++ compiler; there is code in the gollvm runtime (libgo) that refers to functions defined in <cpuid.h>, however some older versions of clang (prior to 5.0) don't provide definitions for all the needed functions. If you encounter this problem, rerun cmake to configure your build to use a more recent version of Clang (or use GCC), as described above.