internal/worker: use HEAD requests to pkgsite

When we test for the existence of a module, we just need the status
code; there's no need to read the entire page.

Add a test against the real pkgsite to confirm.

The test revealed that although it seems we were able to use the
internal pkgsite staging environment, in fact we were always getting
200s because we were fetching a login page. We may add auth in the
future, but for now drop the ability to access anything other than the
public pkgsite.

Change-Id: I8934e17c76aad6aa36bd974a813f300b02549491
Reviewed-on: https://go-review.googlesource.com/c/vuln/+/368856
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Julie Qiu <julie@golang.org>
3 files changed
tree: 26e775442116eaa897c295196b3dc9a13f155958
  1. client/
  2. cmd/
  3. devtools/
  4. internal/
  5. osv/
  6. vlint/
  7. all.bash
  8. AUTHORS
  9. CONTRIBUTING.md
  10. CONTRIBUTORS
  11. go.mod
  12. go.sum
  13. LICENSE
  14. PATENTS
  15. README.md
  16. triaged-cve-list
README.md

Go Vulnerability Database

Go Reference

This repository contains code for hosting the Go Vulnerability Database. The actual reports can be found at x/vulndb.

Neither the code, nor the data, nor the existence of this repository is to be considered stable. See the Draft Design for details on this project.

Accessing the database

The Go vulnerability database is rooted at https://storage.googleapis.com/go-vulndb and provides data as JSON. We recommend using client.Client to read data from the Go vulnerability database.

Do not rely on the contents of the x/vulndb repository. The YAML files in that repository are maintained using an internal format that is subject to change without warning.

The endpoints the table below are supported. For each path:

  • $base is the path portion of a Go vulnerability database URL (https://storage.googleapis.com/go-vulndb).
  • $module is a module path
  • $vuln is a Go vulnerabilitiy ID (for example, GO-2021-1234)
PathDescription
$base/index.jsonList of module paths in the database mapped to its last modified timestamp (link).
$base/$module.jsonList of vulnerability entries for that module (example).
$base/ID/index.jsonList of all the vulnerability entries in the database
$base/ID/$vuln.jsonAn individual Go vulnerability report

Note that these paths and format are provisional and likely to change until an approved proposal.

Packages

Some of these packages can probably be coalesced, but for now are easier to work on in a more segmented fashion.

  • osv provides a package for generating OSV-style JSON vulnerability entries from a report.Report
  • client contains a client for accessing HTTP/fs based vulnerability databases, as well as a minimal caching implementation
  • cmd/dbdiff provides a tool for comparing two different versions of the vulnerability database
  • cmd/gendb provides a tool for converting YAML reports into JSON database
  • cmd/linter provides a tool for linting individual reports
  • cmd/report2cve provides a tool for converting YAML reports into JSON CVEs

License

Unless otherwise noted, the Go source files are distributed under the BSD-style license found in the LICENSE file.

Database entries available at https://storage.googleapis.com/go-vulndb/ are distributed under the terms of the CC-BY 4.0 license.