internal/vulndbreqs: replace manual log counting with big query

Before CL 854266, vulndbreqs needed to inspect each line of the logs
because it had to extract the IP address of each request. So it
downloaded the log files and read them line by line.

After CL 854266, the only job of vulndbreqs is to count the number of
requests. There is no need to inspect the content of the files in this
process. The counting can be done by BigQuery directly over the files
in the GCS bucket, using a temporary external table:

https://cloud.google.com/bigquery/docs/external-data-sources

This avoids downloading entirely: instead of "moving the data to the
compute", we "move the compute to the data". Computing one day used to
take over an hour for large days (exceeding the Cloud Run request
timeout); it now takes ~10 seconds.

The counts produced by the BigQuery approach were verified to match the
existing rows in vulndb.requests exactly for 10 historical dates.

Change-Id: I9c89ab8436a7a1655db6f27fe109d268cefbcfb7
Reviewed-on: https://go-review.googlesource.com/c/pkgsite-metrics/+/854685
Auto-Submit: Hongxiang Jiang <hxjiang@golang.org>
LUCI-TryBot-Result: golang-scoped@luci-project-accounts.iam.gserviceaccount.com <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Ethan Lee <ethanalee@google.com>
3 files changed
tree: 00c2de288c81a7794f50b0f0a417dc040b2d58a2
  1. cmd/
  2. deploy/
  3. devtools/
  4. internal/
  5. terraform/
  6. .dockerignore
  7. .gitignore
  8. all_test.go
  9. checks.bash
  10. codereview.cfg
  11. config.json.commented
  12. CONTRIBUTING.md
  13. go.mod
  14. go.sum
  15. LICENSE
  16. Makefile
  17. PATENTS
  18. README.md
  19. tools.go
README.md

pkgsite-metrics

This repository contains code that enables collecting and evaluating metrics for the Go ecosystem.

Report Issues / Send Patches

This repository uses Gerrit for code changes. To learn how to submit changes to this repository, see https://golang.org/doc/contribute.html.

The main issue tracker for the time repository is located at https://github.com/golang/go/issues. Prefix your issue with “x/pkgsite-metrics:” in the subject line, so it is easy to find.