tree 230b553713bffd364321db456594aa70026c8dca
parent 6e28a50e59803c0b1f05ca9e18a38d33b9e2868b
author Jonathan Amsterdam <jba@google.com> 1630495366 -0400
committer Jonathan Amsterdam <jba@google.com> 1630505801 +0000

internal/{worker/fetch}: include DB activity in load-shedding

Move the load-shedding logic to the worker and have it span both the
fetch and processing of the module (as previously) as well as
inserting it into the database.

This is a more accurate estimation of load, since running a lot of
concurrent queries definitely slows down processing.

Most of the time this won't make much difference, but under high load,
such as when processing multiple large modules, it will reduce DB
contention and should result in greater throughput.

For golang/go#48010

Change-Id: I7d0922e02d00182e867fd3b29fc284c32ecab5ee
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/346749
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>