Google Git
Sign in
go / telemetry / 9621737653dc21d5c57c48b2281cc21fd4ea7428 / . / godev / cmd / worker
tree: 71f371bb414a00e589a859b786ae7fb8f0e6534f [path history] [tgz]
  1. cloudbuild.yaml
  2. Dockerfile
  3. main.go
  4. main_test.go
  5. README.md
godev/cmd/worker/README.md

worker

Endpoints

/merge/?date=<YYYY-MM-DD>

The merge endpoint reads the set of reports from the upload bucket prefixed with the value of the data param and encodes each report as newline separated JSON in a merged report. It returns the number of reports merged and the location of the merged report.

/chart

The /chart endpoint reads the file named ‘YYYY-MM-DD.json’ containing reports from the given date from the merge bucket and generates chart data based on these reports. It returns the number of reports used to generate the chart and the location where the chart data is stored.

/chart/?date=<YYYY-MM-DD>

Use this endpoint to generate charts from a report on a specific date. The worker will retrieve the report for the provided date from the merge bucket.

/chart/?start=<YYYY-MM-DD>&end=<YYYY-MM-DD>

Use this endpoint to generate an aggregate chart file containing data from the provided date range (inclusive) from the merge bucket.

/copy (dev env only)

This endpoint facilitates the copying of uploaded reports from the prod environment‘s “uploaded” bucket to the dev environment’s “uploaded” bucket.

Since we don't have clients regularly uploading data to dev, copying seeds the dev environment with data.

Similar to the /chart endpoint, /copy also supports the following query parameters:

  • `/?date=``: Copies reports for a specific date.
  • `/?start=&end=``: Copies reports within a specified date range.

/queue-tasks

The queue-tasks endpoint is responsible for task distribution. When invoked, it triggers the following actions:

  • call merge endpoint to merge uploaded reports for the past 7 days.
  • call chart endpoint to generate daily charts for the 7 days preceding today.
  • call chart endpoint to generate weekly charts for the past 8 days.

Local Development

For local development, simply build and run. It serves on localhost:8082.

go run ./godev/cmd/worker

By default, the server will use the filesystem for storage object I/O. Use the -gcs flag to use the Cloud Storage API.

go run ./godev/cmd/worker --gcs

Optionally, use the localstorage devtool the emulate the GCS server on your machine.

./godev/devtools/localstorage.sh
STORAGE_EMULATOR_HOST=localhost:8081 go run ./godev/cmd/worker --gcs

Environment Variables

NameDefaultDescription
GO_TELEMETRY_PROJECT_IDgo-telemetryGCP project ID
GO_TELEMETRY_LOCAL_STORAGE.localstorageDirectory for storage emulator I/O or file system storage
GO_TELEMETRY_UPLOAD_CONFIG../config/config.jsonLocation of the upload config used for report validation
GO_TELEMETRY_MAX_REQUEST_BYTES102400Maximum request body size the server allows
GO_TELEMETRY_ENVlocalDeployment environment (e.g. prod, dev, local, ... )
GO_TELEMETRY_LOCATION_IDGCP location of the service (e.g, us-east1)
GO_TELEMETRY_SERVICE_ACCOUNTGCP service account used for queueing work tasks
GO_TELEMETRY_CLIENT_IDGCP OAuth client used in authentication for queue tasks
GO_TELEMETRY_WORKER_URLhttp://localhost:8082

Testing

The worker servie has a suite of regression tests that can be run with:

go test golang.org/x/telemetry/...

Deploying

Each time a CL is reviewed and submitted, the site is automatically deployed to Cloud Run. If it passes its serving-readiness checks, it will be automatically promoted to handle traffic.

If the automatic deployment is not working, or to check on the status of a pending deployment, see the “worker” trigger in the Cloud Build history.

Test Instance

To deploy a test instance of this service, push to a branch and manually trigger the deploy job from the Cloud Build console with the desired values for branch and service name.