design/25530-sumdb: various minor design updates
- Record blocks are tiled under /tile now.
- The database key is in $GOSUMDB.
- $GONOVERIFY is now $GONOSUMDB.
- Lookup results are cached now (not just tiles).
- Latest is only needed for auditors.
Run-TryBot: Russ Cox <firstname.lastname@example.org>
Reviewed-by: Russ Cox <email@example.com>
diff --git a/design/25530-sumdb.md b/design/25530-sumdb.md
index ca897aa..2ec929d 100644
@@ -3,7 +3,7 @@
-Last updated: March 4, 2019.
+Last updated: April 24, 2019.
@@ -189,16 +189,17 @@
authenticating it against the signed tree hash
and authenticating the signed tree hash against the client's
timeline of signed tree hashes.
- - `/records/N?limit=C` will serve the data (that is, the `go.sum` lines) for up to
- C records starting at record number N.
- `/tile/H/L/K[.p/W]` will serve a [log tile](https://research.swtch.com/tlog#serving_tiles).
The optional `.p/W` suffix indicates a partial log tile with only `W` hashes.
Clients must fall back to fetching the full tile if a partial tile is not found.
+ The record data for the leaf hashes in `/tile/H/0/K[.p/W]` are served as `/tile/H/data/K[.p/W]`
+ (with a literal `data` path element).
-Clients are expected to use `/lookup` and `/tile` during normal operations,
-while auditors will want to use `/latest` and `/records`.
+Clients are expected to use `/lookup` and `/tile/H/L/...` during normal operations,
+while auditors will want to use `/latest` and `/tile/L/data/...`.
+A special `go` command may also fetch `/latest` to force incorporation
+of that signed tree head into the local timeline.
### Proxying a Checksum Database
@@ -244,16 +245,19 @@
ensuring that every record it reads is actually in the log
and that no observed log ever drops a record from an earlier observed log.
-The `go` command will store the database’s public key in
-That file will also contain the default starting signed tree size and tree hash,
-updated with each major release.
+The `go` command will refer to `$GOSUMDB` to find the name and public key
+of the Go checksum database.
+That variable will default to the `sum.golang.org` server.
-The `go` command will then cache the latest signed tree size and tree hash
+The `go` command will cache the latest signed tree size and tree hash
-It will cache tiles in `$GOPATH/pkg/mod/download/cache/sumdb/sum.golang.org/tile/H/L/K[.W]`.
-These two different locations let `go clean -modcache` delete any cached tiles as well,
-but no `go` command (only a manual `rm -rf $GOPATH/pkg`)
+It will cache lookup results and tiles in
+This way, `go clean -modcache` deletes cached lookup results and tiles
+but not the latest signed tree hash, which should be preserved for
+detection of timeline inconsistency.
+No `go` command (only a manual `rm -rf $GOPATH/pkg`)
will wipe out the memory of the latest observed tree size and hash.
If the `go` command ever does observe a pair of inconsistent signed tree sizes and hashes,
it will complain loudly on standard error and fail the build.
@@ -264,7 +268,8 @@
especially since that would transmit potentially private import paths
over the network to the database `/lookup` endpoint.
A few new environment variables control this configuration.
-(See the [`go env -w` proposal](https://golang.org/design/30411-env)
+(See the [`go env -w` proposal](https://golang.org/design/30411-env),
+now available in the Go 1.13 development branch,
for a way to manage these variables more easily.)
- `GOPROXY=https://proxy.site/path` sets the Go module proxy to use, as before.
@@ -278,25 +283,27 @@
will bypass the proxy for the modules foo.corp.google.com, foo.corp.google.com/bar, rsc.io/private, and rsc.io/private/bar,
though not rsc.io/privateer (the patterns are path prefixes, not string prefixes).
-- `GONOVERIFY=prefix1,prefix2,prefix3` sets a list of module path prefixes,
+- `GOSUMDB=<database-key>` sets the Go checksum database to use.
+- `GONOSUMDB=prefix1,prefix2,prefix3` sets a list of module path prefixes,
again possibly containing globs, that should not be looked up using the database.
We expect that corporate environments may fetch all modules, public and private,
through an internal proxy;
- `GONOVERIFY` allows them to disable checksum database lookups for
+ `GONOSUMDB` allows them to disable checksum database lookups for
internal modules while still verifying public modules.
- Therefore, `GONOVERIFY` must not imply `GONOPROXY`.
+ Therefore, `GONOSUMDB` must not imply `GONOPROXY`.
We also expect that other users may prefer to connect directly to source origins
but still want verification of open source modules or proxying of the database itself;
- `GONOPROXY` allows them to arrange that and therefore must not imply `GONOVERIFY`.
+ `GONOPROXY` allows them to arrange that and therefore must not imply `GONOSUMDB`.
The database not being able to report `go.sum` lines for a module version
is a hard failure:
-any private modules must be explicitly listed in `$GONOVERIFY`.
+any private modules must be explicitly listed in `$GONOSUMDB`.
(Otherwise an attacker could block traffic to the database
and make all module versions appear to be genuine.)
-The database can be disabled entirely with `GONOVERIFY=*`.
+The database can be disabled entirely with `GONOSUMDB=*`.
The command `go get -insecure` will report but not stop after database lookup
failures or database mismatches.
@@ -445,7 +452,7 @@
the download fails loudly and the `go` command stops.
This ensures both that all public modules are in fact
authenticated and also that any misconfiguration
-must be corrected (by setting `$GONOVERIFY` to avoid
+must be corrected (by setting `$GONOSUMDB` to avoid
the database for those private modules)
in order to achieve a successful build.
This way, the frequency of misconfiguration-induced
@@ -454,10 +461,10 @@
One possibility to further reduce exposure of private module path text
is to provide additional ways to
-set `$GONOVERIFY`, although it is not clear what those
+set `$GONOSUMDB`, although it is not clear what those
A top-level module's source code repository is an attractive place to
-want to store configuration such as `$GONOVERIFY`
+want to store configuration such as `$GONOSUMDB`
and `$GOPROXY`, but then that configuration changes
depending on which version of the repo is checked out,
which would cause interesting behavior when testing old
@@ -525,14 +532,14 @@
it only requests information about dependencies with
-One avenue worth exploring would be to cache database lookup results
+The `go` command will also cache database lookup results
(reauthenticating them against cached tiles at each use),
so that using a single computer to
upgrade the version of a particular dependency used by N different modules
-would result in only one database lookup, not N.
-That would further reduce the strength of any usage signal.
+will result in only one database lookup, not N.
+That further reduces the strength of any usage signal.
-One possible way to further reduce the usage signal
+One possible way to even further reduce the usage signal
observable by the database might be to use a truncated hash
for K-anonymity, as described in the previous section,
but the efficiency problems described earlier still apply.