godoc: rate limit file parsing on startup, improve diagnostics

When building the corpus of local packages, a "too many open files"
error would cause some directories inside GOPATH to be skipped.
Further, the error would not be reported because it was masked by a
"file not found" error from the GOROOT VFS layer.

This change adds a rate limit around parsing files when buildling
the directory tree, error reporting when godoc is run with -v, and
fixes the masked error issue in the vfs package.

It's possible that the rate limiting could be put into the
godoc/vfs/gatefs package, but I tried making the gate account for
open files (not just individual open/close/read/write operations)
but then godoc just hard locks (it wasn't designed to only open 20
files at once).

Change-Id: I925d120b53d9a86430b6977cb90eb143785ecc48
Reviewed-on: https://go-review.googlesource.com/24060
Reviewed-by: Dave Day <djd@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2 files changed