commit d4ea014cebed1bb4a68b8b4dbeabea4f7ea0c78b
author Brad Fitzpatrick <bradfitz@golang.org> Fri Jun 12 10:31:58 2015 -0700
committer Brad Fitzpatrick <bradfitz@golang.org> Sat Jun 13 01:50:33 2015 +0000
tree fb16d016f55ae765081ebdf6cdc9bb09dd207931
parent 5e8cfd4b8cff2e2ffc9d4619744426ec1323a65c

cmd/coordinator, buildlet: reliability fixes

Add background heartbeats to detect dead GCE VMs (the OpenBSD
buildlets seem to hang a lot),

Add timeouts to test executions.

Take helper buildlets out of service once they're marked bad.

Keep the in-order buildlet running forever when sharding tests, in
case all the helpers die. (observed once)

Keep a cache of recently deleted VMs and don't try to delete VMs again
if we've recently deleted them. (they're slow to delete)

More reverse buildlets more paranoid in their health checking and closing
of the connection.

Make status page link to /try set URLs.

Also, better logging (more sometimes, less others0, and misc bug fixes.

Change-Id: I57a5e8e39381234006cac4dd799b655d64be71bb
Reviewed-on: https://go-review.googlesource.com/10981
Reviewed-by: Andrew Gerrand <adg@golang.org>