all: migrate go/pkgsite-metrics from gsutil to gcloud storage This CL is part of the on going effort to migrate from the legacy "gsutil" tool to the new and improved "gcloud storage" command-line interface. "gcloud storage" is the recommended and modern tool for interacting with Google Cloud Storage, offering better performance, unified authentication, and a more consistent command structure with other "gcloud" components. 🚀 ⚠️ Action Required: Please Review and Test Carefully: While we have based the automation on the migration guide, every use case is unique. "IT IS CRUCIAL that you THOROUGHLY TEST these changes in environments appropriate to your use-case BEFORE MERGING." Be aware of potential differences between "gsutil" and `gcloud storage` that could impact your workflows. For instance, the structure of command output may have changed, requiring updates to any scripts that parse it. Similarly, command behavior can differ subtly; the "gcloud storage rsync" command has a different file deletion logic than "gsutil rsync", which could lead to unintended file deletions. Our migration guides can help guide you through a list of mappings and some notable differences between the two tools. Standard presubmit tests are run as part of this CL's workflow. "If you need to target an ADDITIONAL TEST WORKFLOW or REQUIRE ASSISTANCE WITH TESTING, please let us know." Please verify that all your Cloud Storage operations continue to work as expected to avoid any potential disruptions in production. Automation Details: This change was GENERATED AUTOMATICALLY by an agent that targets users of "gsutil". The transformations applied are based on the gsutil to gcloud storage migration guide - http://go/gsutil-gcloud-storage-migration-guide, external guide - https://docs.cloud.google.com/storage/docs/gsutil-transition-to-gcloud. Support and Collaboration: The "GCS CLI" team is here to help! If you encounter any issues, have a complex use case that this automated change doesn't cover, or face any other blockers, please don't hesitate to reach out. We are happy to work with you to test and adjust these changes as needed. Contact: gcs-cli-hyd@google.com We appreciate your partnership in this important migration effort! BUG: b:486536908 Change-Id: Id444cd329b7c8446e741625ef4cdcc0e408613a2 Reviewed-on: https://go-review.googlesource.com/c/pkgsite-metrics/+/845725 Reviewed-by: Jonathan Amsterdam <jba@google.com> Reviewed-by: Ethan Lee <ethanalee@google.com> Auto-Submit: Ethan Lee <ethanalee@google.com> LUCI-TryBot-Result: golang-scoped@luci-project-accounts.iam.gserviceaccount.com <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
diff --git a/Makefile b/Makefile index b808169..a931d7c 100644 --- a/Makefile +++ b/Makefile
@@ -11,17 +11,17 @@ # This is the image that is used by the sandbox. # The file must live in the repo directory so that cmd/worker/Dockerfile # can access it. -# We assume that the file exists in the bucket. +# We assume that the file exists in the bucket. # See the rule below for how to create the image. go-image.tar.gz: - gsutil cp gs://go-ecosystem/$@ $@ + gcloud storage cp gs://go-ecosystem/$@ $@ # Use this rule to build a go image for a specific Go version. # E.g. # make go-image-1.19.4.tar.gz # To change the sandbox image permanently, copy it to GCP: # -# gsutil cp go-image.1.19.4.tar.gz gs://go-ecosystem/go-image.tar.gz +# gcloud storage cp go-image.1.19.4.tar.gz gs://go-ecosystem/go-image.tar.gz # Then delete the local copy. go-image-%.tar.gz: docker export $(shell docker create golang:$*) | gzip > go-image-$*.tar.gz @@ -31,15 +31,14 @@ # # Get the last-modified time of the index.json file, which is reported as the # last-modified time of the DB, and save it to a local file. (The last-modified -# time of the local index.json is not what we want: it is the time that gsutil cp +# time of the local index.json is not what we want: it is the time that gcloud storage cp # wrote the file.) # # This directory must live in the repo directory so that cmd/worker/Dockerfile # can access it. go-vulndb: - gsutil -m -q cp -r gs://go-vulndb . - gsutil stat gs://go-vulndb/index.json | \ - awk '$$1 == "Update" { for (i = 4; i <= NF; i++) printf("%s ", $$i); printf("\n"); }' \ + gcloud storage cp --recursive --quiet gs://go-vulndb . + gcloud storage objects describe gs://go-vulndb/index.json --format="value(update_time)" \ > go-vulndb/LAST_MODIFIED # Remove comments from a json file.
diff --git a/deploy/worker.yaml b/deploy/worker.yaml index ee69942..8acb7c8 100644 --- a/deploy/worker.yaml +++ b/deploy/worker.yaml
@@ -64,12 +64,11 @@ # Download the vuln DB from its bucket to a local directory, and remember # its last-modified time in a file. - gsutil -m -q cp -r gs://go-vulndb /workspace - gsutil stat gs://go-vulndb/index.json | \ - awk '$$1 == "Update" { for (i = 4; i <= NF; i++) printf("%s ", $$i); printf("\n"); }' \ + gcloud storage cp --recursive --quiet gs://go-vulndb /workspace + gcloud storage objects describe gs://go-vulndb/index.json --format="value(update_time)" \ > /workspace/go-vulndb/LAST_MODIFIED # Download a tarball of a docker Go image. - gsutil cp gs://go-ecosystem/go-image.tar.gz /workspace + gcloud storage cp gs://go-ecosystem/go-image.tar.gz /workspace - id: Build # Build the docker image.