internal/postgres: get next modules to fetch in one query

Construct a single query that returns all the modules we want to
reprocess, in the order we want.

The main ideas are:

- Use a WITH statement (CTE) to construct a table of (module path,
  latest version).

- Add a computed "latest" column to every row by testing membership in
  that table.

- Use a CASE in the ORDER BY clause to express the ordering we want.

Some other changes:
- We process status 0 first (the original motivation for this CL).

- As a consequence of using one query, we'll always return a number of
  results. up to the limit.  To handle the lower limit for large
  modules, we add code to truncate the results.

- Before, all 5xx that were not one of the four special codes were
  processed after everything else. Now it's relatively easy to process
  other 5xx codes at the end of each category (latest versions,
  non-large modules).

Change-Id: I4dacb093cfb93299ccfc8d29c1fd0b1cb49bf56f
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/239479
Reviewed-by: Julie Qiu <julie@golang.org>
2 files changed
tree: ea04145d44ac8395bb529aa47b4db7a5d2d31c0a
  1. cmd/
  2. content/
  3. devtools/
  4. doc/
  5. internal/
  6. migrations/
  7. third_party/
  8. .gitignore
  9. .prettierignore
  10. .prettierrc.yaml
  11. all.bash
  12. cloudbuild.yaml
  13. CONTRIBUTING.md
  14. go.mod
  15. go.sum
  16. LICENSE
  17. PATENTS
  18. README.md
README.md

Pkg.go.dev

A site for discovering Go packages

Pkg.go.dev is a website for discovering and evaluating Go packages and modules.

You can check it out at https://pkg.go.dev.

Roadmap

Pkg.go.dev launched in November 2019, and is currently under active development by the Go team.

Here's what we are currently working on:

  • Design updates: We have some design changes planned for pkg.go.dev, to address UX feedback that we have received. You can expect a more cohesive search and navigation experience coming soon. We plan to share these designs for feedback once they are ready.

  • Godoc.org redirect: Longer term, we are working towards redirecting godoc.org traffic to pkg.go.dev. We know that there are features available on godoc.org that users want to see on pkg.go.dev, and we want to ensure that we address these. We’ve been keeping track of issues related to redirecting godoc.org traffic on Go issue #39144. These issues will be prioritized in the next few months. We also plan to continue improving our license detection algorithm.

  • Search improvements: We’ll be improving our search experience based on feedback in Go issue #37810, to make it easier for users to find the dependencies they are looking for and make better decisions around which ones to import.

We encourage everyone to begin using pkg.go.dev today for all of their needs and to file feedback! You can redirect all of your requests from godoc.org to pkg.go.dev, by clicking Always use pkg.go.dev at the top of any page on godoc.org.

Issues

If you want to report a bug or have a feature suggestion, please first check the known issues to see if your issue is already being discussed. If an issue does not already exist, feel free to file an issue.

For answers to frequently asked questions, see go.dev.

You can also chat with us on the #tools slack channel on the Gophers slack.

Contributing

We would love your help!

Our canonical Git repository is located at go.googlesource.com/pkgsite. There is a mirror of the repository at github.com/golang/pkgsite.

To contribute, please read our contributing guide.

License

Unless otherwise noted, the Go source files are distributed under the BSD-style license found in the LICENSE file.