tree 570f90bd90906f4be7094cfb62530de4ee1d9bea
parent 2cfdcb5d9ede961efdbbbc790ee3c7f2722ef1fa
author Jonathan Amsterdam <jba@google.com> 1623939447 -0400
committer Jonathan Amsterdam <jba@google.com> 1624038062 +0000

internal/database: add ability to run queries incrementally

Postgres lets you define a cursor and fetch rows from it as top-level
statements, without needing to write any PS-SQL. See
https://www.postgresql.org/docs/11/sql-declare.html and
https://www.postgresql.org/docs/11/sql-fetch.html for details.

Use this feature to define RunQueryIncrementally, which repeatedly
fetches query rows in batches until it runs out or the passed function
says it's done.

If grouping by module paths is enabled, use RunQueryIncrementally
with a very large limit to read rows until we've seen a page's worth
of module paths.

This CL doesn't handle pagination correctly. Any page after the first
is going to be wrong.

Change-Id: Idf8233160b0cf74412a688e1a6b95f4f2b720008
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/329469
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
TryBot-Result: kokoro <noreply+kokoro@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>