tree f3455b0b677e253765f162c9912bd40554f97b08
parent 38a4707ee56cd0fa7936058aff694585f13c02a8
author Heschi Kreinick <heschi@google.com> 1659130030 -0400
committer Heschi Kreinick <heschi@google.com> 1659626688 +0000

internal/workflow: set a watchdog on tasks by default

We have an ongoing problem with gomotes that freeze up during the
release process. I've seen hangs during tests and MSI fetching. Maybe
we're missing a keepalive somewhere, maybe there's a bug in the
coordinator or the buildlet, who knows.

Rather than try to set a deadline everywhere, which seems error-prone
and annoying, add a watchdog to each task that is reset when they log or
explicitly heartbeat.

The predecessors to this CL should make sure that every task logs at a
reasonable frequency. The long pole is long test builders; the cmd/go
tests can take up to 4 minutes, during which there might not be any
output. Set the watchdog to a very generous 10 minutes, and have
AwaitCondition heartbeat at half that interval.

Fixes golang/go#54134.

Change-Id: I921ff9d9b695efac45015846ebd57dbb398aab89
Reviewed-on: https://go-review.googlesource.com/c/build/+/420221
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Jenny Rakoczy <jenny@golang.org>