internal/vulndbreqs: replace manual log counting with big query Before CL 854266, vulndbreqs needed to inspect each line of the logs because it had to extract the IP address of each request. So it downloaded the log files and read them line by line. After CL 854266, the only job of vulndbreqs is to count the number of requests. There is no need to inspect the content of the files in this process. The counting can be done by BigQuery directly over the files in the GCS bucket, using a temporary external table: https://cloud.google.com/bigquery/docs/external-data-sources This avoids downloading entirely: instead of "moving the data to the compute", we "move the compute to the data". Computing one day used to take over an hour for large days (exceeding the Cloud Run request timeout); it now takes ~10 seconds. The counts produced by the BigQuery approach were verified to match the existing rows in vulndb.requests exactly for 10 historical dates. Change-Id: I9c89ab8436a7a1655db6f27fe109d268cefbcfb7 Reviewed-on: https://go-review.googlesource.com/c/pkgsite-metrics/+/854685 Auto-Submit: Hongxiang Jiang <hxjiang@golang.org> LUCI-TryBot-Result: golang-scoped@luci-project-accounts.iam.gserviceaccount.com <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ethan Lee <ethanalee@google.com>
This repository contains code that enables collecting and evaluating metrics for the Go ecosystem.
This repository uses Gerrit for code changes. To learn how to submit changes to this repository, see https://golang.org/doc/contribute.html.
The main issue tracker for the time repository is located at https://github.com/golang/go/issues. Prefix your issue with “x/pkgsite-metrics:” in the subject line, so it is easy to find.