The worker populates the database with information about new modules.
For additional information on functionality of the worker, see the design document.
The main program lives in
You can run the worker locally like so:
go run ./cmd/worker
See experiment.md for instructions how to enable experiments.
When run locally, the worker uses an in-memory queue. This implementation has bounded parallelism (configurable via the
-workers flag) but does not automatically retry failures.
In order to populate local versions, you can either fetch the version explicitly (via
http://localhost:8000/fetch/path/to/package/@v/v1.2.3), or you can visit the Worker dashboard, and click ‘Enqueue from module index’. This will enqueue the next N versions from the index for processing.
By default, the worker does not insert readme contents or documentation into the database if we determine that the module or package is not redistributable, based on the licenses it finds in the module zip. To bypass the license check, pass the flag