| commit | 76065a2ab1b195a790c803d11e713491cfcb3ba5 | [log] [tgz] |
|---|---|---|
| author | Russ Cox <rsc@golang.org> | Thu Jul 28 20:55:40 2022 -0400 |
| committer | Russ Cox <rsc@golang.org> | Thu Aug 04 13:33:01 2022 +0000 |
| tree | 539af6ee53266a24c3af0b90b081092e3a266e93 | |
| parent | baeff8aaedd776462c474f0b1303eb4fd6ab4f53 [diff] |
dashboard: update builders to Go 1.17.13 for bootstrap Compute GoBootstrapURL for all hosts. This removes another obviously computable field, assuming the bootstrap toolchains have all been uploaded with consistent names (as genbootstrap -upload now does). It also makes all builders work the same: they all start by downloading the bootstrap toolchain, instead of having some download and others use a toolchain installed into the boot image. The builders with pre-installed toolchains will continue to have those toolchains, but now nothing will use them, because the coordinator or gomote will see the GoBootstrapURL result and push an updated toolchain to $workdir/go1.4. As part of all this simplification, switch to Go 1.17.13 as the bootstrap toolchain, for golang/go#44505. (And also because we have no Go 1.4 for some of these platforms.) Updating the bootstrap toolchain again in a couple years will now be easy: run genbootstrap -upload '*' and then change GoBootstrap here. For golang/go#44505. (Won't be fixed until coordinator is redeployed.) Succeeded running all.bash/bat/rc using updated debugnewvm: android-386-emu android-amd64-emu dragonfly-amd64-622 freebsd-386-12_3 freebsd-386-13_0 freebsd-amd64-13_0 freebsd-amd64-race js-wasm linux-386 linux-386-buster linux-386-clang linux-386-longtest linux-386-sid linux-386-softfloat linux-386-stretch linux-amd64 linux-amd64-alpine linux-amd64-boringcrypto linux-amd64-bullseye linux-amd64-buster linux-amd64-clang linux-amd64-fedora linux-amd64-goamd64v3 linux-amd64-longtest linux-amd64-nocgo linux-amd64-noopt linux-amd64-nounified linux-amd64-perf linux-amd64-race linux-amd64-racecompile linux-amd64-sid linux-amd64-ssacheck linux-amd64-staticlockranking linux-amd64-stretch linux-amd64-unified linux-amd64-vmx netbsd-386-9_0 netbsd-amd64-9_0 openbsd-386-68 openbsd-386-70 openbsd-amd64-68 openbsd-amd64-70 windows-386-2008 windows-386-2012 windows-amd64-2008 windows-amd64-2012 windows-amd64-2016 windows-amd64-longtest windows-amd64-newcc-race windows-amd64-race Succeeded running make.bash using updated debugnewvm: linux-s390x-crosscompile Succeeded running make.bash using updated gomote (real time for put14 shown): aix-ppc64 (5.2s) darwin-amd64-10_14 (3.4s) darwin-amd64-10_15 (3.5s) darwin-amd64-11_0 (4.5s) darwin-amd64-12_0 (4.5s) darwin-amd64-nocgo (4.4s) darwin-amd64-race (4.2s) darwin-arm64-11 (1.7s) darwin-arm64-12 (1.8s) freebsd-arm64-dmgk (4.0s) freebsd-arm-paulzhol (35.1s) illumos-amd64 (5.9s) linux-amd64-wsl (126.7s) linux-arm64-aws (2.8s) linux-arm-aws (3.8s) linux-mips64le-rtrk (20.4s) linux-mips64-rtrk (16.0s) linux-mipsle-rtrk (16.3s) linux-mips-rtrk (16.3s) linux-ppc64-buildlet (4.7s) linux-ppc64le-buildlet (3.6s) linux-ppc64le-power9osu (5.6s) linux-riscv64-unmatched (38.2s) linux-s390x-ibm (4.4s) netbsd-arm64-bsiegert (35.0s) openbsd-arm64-jsing (64.9s) openbsd-mips64-jsing (154.2s) plan9-386-0intro (8.7s) plan9-arm (18.2s) windows-arm64-10 (7.1s) windows-arm64-11 (4.1s) Put14 failed using gomote: android-arm64-corellium (403 Forbidden in fetch) android-arm-corellium (403 Forbidden in fetch) ios-arm64-corellium (403 Forbidden in fetch) - fails build, kept old GOROOT_BOOTSTRAP linux-loong64-3a5000 (proxyconnect tcp: dial tcp 117.50.175.110:7890: i/o timeout) - falls back to preinstalled Go 1.18, kept old GOROOT_BOOTSTRAP Builds fine, failing tests but not on dashboard anyway: freebsd-386-11_4 freebsd-amd64-11_4 linux-amd64-androidemu plan9-386 Failed in same way as current builders: openbsd-386-70-n2d windows-386-2008-newcc windows-386-2012-newcc windows-amd64-2008-newcc windows-amd64-2012-newcc windows-amd64-2016-newcc windows-amd64-longtest-newcc Can't connect with gomote, so untested: linux-riscv64-jsing (but linux-riscv64-unmatched works) netbsd-arm-bsiegert openbsd-arm-jsing plan9-amd64-0intro solaris-amd64-oraclerel (but illumos-amd64 works) Change-Id: If4354a8681dbac28e1d34897b6356b881592c241 Reviewed-on: https://go-review.googlesource.com/c/build/+/419993 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Jenny Rakoczy <jenny@golang.org> Run-TryBot: Russ Cox <rsc@golang.org>
This repository holds the source for various packages and tools that support Go's build system and the development of the Go programming language.
Warning: Packages here are internal to Go's build system and its needs. Some may one day be promoted to another golang.org/x repository, or they may be modified arbitrarily or even disappear altogether. In short, code in this repository is not subject to the Go 1 compatibility promise nor the Release Policy.
This repository uses Gerrit for code changes. To contribute, see https://golang.org/doc/contribute.html.
The main issue tracker for the blog is located at https://github.com/golang/go/issues. Prefix your issue with “x/build/DIR: ” in the subject line.
The main components of the Go build system are:
The coordinator, in cmd/coordinator/, serves https://farmer.golang.org/ and https://build.golang.org/. It runs on GKE and coordinates the whole build system. It finds work to do (both pre-submit “TryBot” work, and post-submit work) and executes builds, allocating machines to run the builds. It is the owner of all machines. It holds the state for which builds passed or failed, and the build logs.
The Go package in buildenv/ contains constants for where the dashboard and coordinator run, for prod, staging, and local development.
The buildlet, in cmd/buildlet/, is the HTTP server that runs on each worker machine to execute builds on the coordinator's behalf. This runs on every possible GOOS/GOARCH value. The buildlet binaries are stored on Google Cloud Storage and fetched per-build, so we can update the buildlet binary independently of the underlying machine images. The buildlet is the most insecure server possible: it has HTTP handlers to read & write arbitrary content to disk, and to execute any file on disk. It also has an SSH tunnel handler. The buildlet must never be exposed to the Internet. The coordinator provisions buildlets in one of three ways:
by creating VMs on Google Compute Engine (GCE) with custom images configured to fetch & run the buildlet on boot, listening on port 80 in a private network.
by running Linux containers (on either Google Kubernetes Engine or GCE with the Container-Optimized OS image), with the container images configured to fetch & run the buildlet on start, also listening on port 80 in a private network.
by taking buildlets out of a pool of connected, dedicated machines. The buildlet can run in either listen mode (as on GCE and GKE) or in reverse mode. In reverse mode, the buildlet connects out to https://farmer.golang.org/ and registers itself with the coordinator. The TCP connection is then logically reversed (using revdial and when the coordinator needs to do a build, it makes HTTP requests to the coordinator over the already-open TCP connection.
These three pools can be viewed at the coordinator's https://farmer.golang.org/#pools.
The env/ directory describes build environments. It contains scripts to create VM images, Dockerfiles to create Kubernetes containers, and instructions and tools for dedicated machines.
maintner in maintner/ is a library for slurping all of Go's GitHub and Gerrit state into memory. The daemon maintnerd in maintner/maintnerd/ runs on GKE and serves https://maintner.golang.org/. The daemon watches GitHub and Gerrit and appends to a mutation log whenever it sees new activity. The logs are stored on GCS and served to clients.
The godata package in maintner/godata/ provides a trivial API to let anybody write programs against Go's maintner corpus (all of our GitHub and Gerrit history), live up to the second. It takes a few seconds to load into memory and a few hundred MB of RAM after it downloads the mutation log from the network.
pubsubhelper in cmd/pubsubhelper/ is a dependency of maintnerd. It runs on GKE, is available at https://pubsubhelper.golang.org/, and runs an HTTP server to receive Webhook updates from GitHub on new activity and an SMTP server to receive new activity emails from Gerrit. It then is a pubsub system for maintnerd to subscribe to.
The gitmirror server in cmd/gitmirror/ mirrors Gerrit to GitHub, and also serves a mirror of the Gerrit code to the coordinator for builds, so we don't overwhelm Gerrit and blow our quota.
The Go gopherbot bot logic runs on GKE. The code is in cmd/gopherbot. It depends on maintner via the godata package.
The developer dashboard at https://dev.golang.org/ runs on GKE. Its code is in devapp/. It also depends on maintner via the godata package.
cmd/retrybuilds: a Go client program to delete build results from the dashboard
The perfdata server, in perfdata/appengine serves https://perfdata.golang.org/. It runs on App Engine and serves the benchmark result storage system.
The perf server, in perf/appengine serves https://perf.golang.org/. It runs on App Engine and serves the benchmark result analysis system. See its README for how to start a local testing instance.
If you wish to run a Go builder, please email golang-dev@googlegroups.com first. There is documentation at https://golang.org/wiki/DashboardBuilders, but depending on the type of builder, we may want to run it ourselves, after you prepare an environment description (resulting in a VM image) of it. See the env directory.