| # Worker |
| |
| The _worker_ populates the database with information about new modules. |
| |
| For additional information on functionality of the worker, see the |
| [design document](design.md). |
| |
| ## Development |
| |
| The main program lives in `cmd/worker`. |
| |
| ### Running |
| |
| You can run the worker locally like so: |
| |
| go run ./cmd/worker |
| |
| See [experiment.md](experiment.md) for instructions how to enable experiments. |
| |
| ### Populating data locally using the worker |
| |
| When run locally, the worker uses an in-memory queue. This implementation has |
| bounded parallelism (configurable via the `-workers` flag) but does not |
| automatically retry failures. |
| |
| In order to populate local versions, you can either fetch the version explicitly |
| (via `http://localhost:8000/fetch/path/to/package/@v/v1.2.3`), or you can visit the |
| Worker dashboard, and click 'Enqueue from module index'. This will enqueue the |
| next N versions from the index for processing. |
| |
| ## Bypassing license checks |
| |
| By default, the worker does not insert readme contents or documentation into the |
| database if we determine that the module or package is not redistributable, |
| based on the licenses it finds in the module zip. To bypass the license check, |
| pass the flag `-bypass_license_check`. |