# Worker

The _worker_ populates the database with information about new modules.

For additional information on functionality of the worker, see the
[design document](design.md).

## Development

The main program lives in `cmd/worker`.

### Running

You can run the worker locally like so:

    go run ./cmd/worker

See [experiment.md](experiment.md) for instructions how to enable experiments.

### Populating data locally using the worker

When run locally, the worker uses an in-memory queue. This implementation has
bounded parallelism (configurable via the `-workers` flag) but does not
automatically retry failures.

In order to populate local versions, you can either fetch the version explicitly
(via `http://localhost:8000/fetch/path/to/package/@v/v1.2.3`), or you can visit the
Worker dashboard, and click 'Enqueue from module index'. This will enqueue the
next N versions from the index for processing.

## Bypassing license checks

By default, the worker does not insert readme contents or documentation into the
database if we determine that the module or package is not redistributable,
based on the licenses it finds in the module zip. To bypass the license check,
pass the flag `-bypass_license_check`.