cmd/coordinator: fix watcher restart on coordinator restart, reuse git dirs

The Docker watcher process wasn't being restarted when the coordinator
restarted, leading to confusion when trying to deploy new versions,
and spammy error messages.

Also, make the watcher start up faster by reusing its previous
instance's git mirrors if they appear to be not corrupt. This reduces
the start from 45+ seconds to 11 seconds. (All the repos are fetched in
parallel and most 1-2seconds, but the "go" repo is 11 seconds.)

It's no longer on tmpfs, but it's still on SSD and will be in the page
cache, and there's plenty of disk:

farmer watcher-git # pwd
/var/cache/watcher-git

farmer watcher-git # du -h -c | tail -1
412M    total

farmer watcher-git # df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda9        47G  5.7G   39G  13% /

Fixes golang/go#12673

Change-Id: I1ad58ab94b02dbe79340e2226023a3e52cf3be49
Reviewed-on: https://go-review.googlesource.com/21625
Reviewed-by: Evan Brown <evanbrown@google.com>
Reviewed-by: Andrew Gerrand <adg@golang.org>
2 files changed