Worker

The worker populates the database with information about new modules.

For additional information on functionality of the worker, see the design document.

Development

The main program lives in cmd/worker.

Experiments

Set environment variable GO_DISCOVERY_CONFIG_DYNAMIC to the filename of a file containing experiments in YAML format. The file can be empty, but it must exist. Example:

experiments:
  - name: sidenav
    rollout: 100
    description: Display documentation index on the left sidenav.

Running

You can run the worker locally like so:

go run ./cmd/worker

Populating data locally using the worker

When run locally, the worker uses an in-memory queue. This implementation has bounded parallelism (configurable via the -workers flag) but does not automatically retry failures.

In order to populate local versions, you can either fetch the version explicitly (via http://localhost:8000/fetch/path/to/package/@v/v1.2.3), or you can visit the Worker dashboard, and click ‘Enqueue from module index’. This will enqueue the next N versions from the index for processing.

Bypassing license checks

By default, the worker does not insert readme contents or documentation into the database if we determine that the module or package is not redistributable, based on the licenses it finds in the module zip. To bypass the license check, pass the flag -bypass_license_check.