tests/README.md

# Tests

The tests directory contains integration tests that require a pre-seeded
database.

Each tests/<group> contains a seed.txt file at tests/<group>/seed.txt, which is
used by the docker seeddb container to setup the test database.

The tests can be run using tests/<group>/run.sh.

## Search Tests

The tests/search/scripts directory contains test scripts \*.txt that are run
using tests/search/run.sh.

These scripts follow the format:

```
Test name A
search query A
Expected result A1
Expected result A2

Test name B
search query B
Expected result B1
Expected result B2

...
```

Each test set is separated by a newline. Comments (lines that being with a #),
are ignored.

The imported by counts for a dataset can also be updated in
search/importedby.txt. The file has the format:

```
<package-path>, <imported-by-count>
```

It is expected that the modules for these packages are in
tests/search/seed.txt.

## Symbol History API Tests

The tests/api/scripts directory contains tests that are run
using tests/api/run.sh.

The tests compare API data for a module `tests/api/testdata` with the API
history on the frontend versions page. The frontend that is used for comparison
must have `GO_DISCOVERY_SERVE_STATS` set so that the
`/<path>?tab=versions&content=json` endpoint is available.

### Adding a package

Data in `tests/api/testdata` is generated by running:

```
go run tests/api/main.go generate [module path]:[package path suffix]
```

For example, `go run tests/api/main.go generate github.com/julieqiu/api-demo:pkg` for package
`github.com/julieqiu/api-demo/pkg`.

This data can now be used to check the API history on a frontend by running:

```
go run tests/api/main.go compare [module path]:[package path suffix]
```

## Screentest

The screentest/ directory contains visual diff tests for pages on pkg.go.dev.
Tests are written for the screentest tool from golang.org/x/website.
See the [command](https://pkg.go.dev/golang.org/x/website/cmd/screentest) docs for
more information.

### Running the tests

In order to run the tests, run this command from the root of the repository:

```
./tests/screentest/run.sh <env>
```

`./tests/screentest/run.sh ci` sets up a series of docker containers that run a
postgres database, frontend, and headless chrome, and runs the screentest using
headless chrome.

Tests in `./tests/screentest/testcases.ci.txt` run only for CI and local runs of the
screentest. These tests may rely on mocked data, like the vuln page tests.
Tests in `./tests/screentest/testcases.txt` run for all environments and
are suitable for tests with data that is consistent between environments or
static pages.

### Updating the tests

Update screen shot diffs using the following:

1. Run `./tests/screentest/run.sh -update ci` to collect the updated screenshots.
2. Optionally use `./tests/screentest/run.sh -update -run <test> ci` to update specific tests.
3. Copy the new screenshots from `./tests/screentest/output` into `./tests/screentest/testdata/testcases`

### Understanding Test Failures

If the tests fail with diffs, a file with the diffs will be created in the
testdata directory that show the cause of the failure.

Use `./devtools/docker/compose.sh logs` to get detailed logs of the docker containers.

### Troubleshooting

If docker fails with "connect: cannot assign requested address" during `run.sh`,
try running it again. It will eventually succeed. Do not pass the '-rm' flag so that
prior state is preserved.
You can also try reducing the value of cmdgonetlimit in devtools/docker/compose.yaml.