Build configs (at the top) and host configs (bottom) are listed here:
https://farmer.golang.org/builders
A builder runs on a certain host type. (e.g. linux-386-387
is a build type. It runs on host-linux-kubestd
, a Kubernetes-based linux/amd64 host)
They come from the file https://github.com/golang/build/blob/master/dashboard/builders.go
For design details about the coordinator, see https://go.dev/s/builderplan
Information about builder machines, how many are running and their status can be found at https://farmer.golang.org/
golang.org/x/build/cmd/genbuilderkey
tool), and put that in ~/.gobuildkey
or ~/.gobuildkey-host-foo-bar
or the file pointed to by env var $GO_BUILD_KEY_PATH
.go get -u golang.org/x/build/cmd/buildlet
buildlet -coordinator=farmer.golang.org -reverse-type=host-foo-bar -reboot=false
For WIP ports, the steps above can be done out of order as needed. But as a port matures, be sure each step above is done. In particular, make sure that you're not just running a fixed copy of the buildlet binary in a loop forever. We need to be able to update it over time without your involvement. You should be running the stage0 binary (or equivalent shell script or similar for your platform) in a loop instead.
GOGC
setting to avoid thrashing.)Generally, community-run builders only run code that's already been reviewed & submitted. We only enable pre-submit testing for builders run by the Go team that have a lot of hardware available. However, the Gomote tool is available for a number of people on the Go team and in the Go community that lets them have arbitrary access to the builders for development & debugging.
For paranoia reasons, you might want to run your builder in an isolated network that can't access any of your internal resources.
The Go team is migrating the testing pipeline from a custom solution, the coordinator, to LUCI. LUCI is an open source continuous integration system created by the Chrome open source team at Google. The Go team has adopted the use of LUCI in order to leverage a continuous integration solution which is used and supported by a larger group of developers. This should enable the team to provide a more featureful solution to the community.
The LUCI system requires builders to run two applications which authenticate to LUCI and receive and process builds. LUCI token daemon generates a token needed to authenticate. The swarming bot uses the token to connect to LUCI and process builds.
PATH
.swarming
user (without root rights).SWARMING_NEVER_REBOOT
.Create an issue on the Go Issue tracker requesting the addition of a new builder and assign it yourself.
x/build: add LUCI <os-arch> builder
.@gopherbot, please add label new-builder.
in the issue to have gopherbot add it for you.)Use golang.org/x/build/cmd/genbotcert
to generate both a certificate signing request (hostname.csr) and a TLS private key (hostname.key) using the hostname (chosen beforehand) as input. Add a .txt file extension to the certificate signing request (hostname.csr.txt) and attach it to the GitHub issue. A team member will attach the resulting certificate (hostname.cert) to the GitHub issue.
genbotcert -bot-hostname <hostname>
A Go team member will define your new builder in LUCI. A comment will be added to the issue when this is completed.
The Machine Token Daemon communicates with the Token Server to generate and renew a LUCI machine token. Install go.chromium.org/luci/tokenserver/cmd/luci_machine_tokend
and configure it to run every 10 minutes via cron. The private key shouldn't be readable by the swarming
user, so the cron job should run as a separate user.
luci_machine_tokend -backend luci-token-server.appspot.com -cert-pem <path-to-the-certificate> -pkey-pem <path-to-the-private-key> -token-file=/var/lib/luci_machine_tokend/token.json
-token-file-path
to bootstrapswarm
below to match.Install golang.org/x/build/cmd/bootstrapswarm
and configure it to run in a loop under your operating system's process supervisor (systemd, etc) as the swarming
user. Bootstrapswarm
downloads the initial version of the swarming bot and ensures that it is always running.
bootstrapswarm -hostname <hostname>
Verify the bot starts up without any errors in the logs.