commit | df5132719eb9a0bca73a2ed9de696398037e1d40 | [log] [tgz] |
---|---|---|
author | Jonathan Amsterdam <jba@google.com> | Mon Sep 28 08:37:27 2020 -0400 |
committer | Jonathan Amsterdam <jba@google.com> | Tue Sep 29 15:35:01 2020 +0000 |
tree | 59ce5b2cc4fa09bfd2410472579c22c2b524fb6b | |
parent | bbe36a9256ee194e5c401a3322e7af97aafe1520 [diff] |
internal/postgres: enqueue modules in a random order For requeuing, we were sorting modules first by importance of reprocessing, then by module path. That clumped all versions of the same module together (at least all those that needed reprocessing). The main problem with reprocessing multiple versions of the same module was that if the load shedder was off about the size, that error would persist while all versions of the module were being processed. For example, github.com/elastic/beats/v7 has an 86Mi zip file, but only a few Ki of actual Go files. So a node with a 100Mi threshold would shed one version if it was processing another. This frequent shedding slowed the task queue rate to a crawl. This CL effectively randomizes the modules that are enqueued, by hashing the name and version. This small change has had a dramatic effect on processing. Workers are now living for many minutes, even hours, before OOMing; before, even with a load-shed threshold of 100Mi they would still OOM in a half hour or so. Before, processing was erratic: bursts of fast progress interspersed with intervals where processing slowed, as many versions of the same module were processed together, where the module was large or had other problems, like the the elastic/beats module discussed above. Now, processing is uniform and fast, with 8 workers processing about 5 packages/sec and very few sheds. Change-Id: Id4f2010f7ab3131d4b4e37721cab6d5ff1680a54 Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/258018 Reviewed-by: Julie Qiu <julie@golang.org> Trust: Jonathan Amsterdam <jba@google.com>
Pkg.go.dev is a website for discovering and evaluating Go packages and modules.
You can check it out at https://pkg.go.dev.
Pkg.go.dev launched in November 2019, and is currently under active development by the Go team.
Here's what we are currently working on:
Design updates: We have some design changes planned for pkg.go.dev, to address UX feedback that we have received. You can expect a more cohesive search and navigation experience coming soon. We plan to share these designs for feedback once they are ready.
Godoc.org redirect: Longer term, we are working towards redirecting godoc.org traffic to pkg.go.dev. We know that there are features available on godoc.org that users want to see on pkg.go.dev, and we want to ensure that we address these. We’ve been keeping track of issues related to redirecting godoc.org traffic on Go issue #39144. These issues will be prioritized in the next few months. We also plan to continue improving our license detection algorithm.
Search improvements: We’ll be improving our search experience based on feedback in Go issue #37810, to make it easier for users to find the dependencies they are looking for and make better decisions around which ones to import.
We encourage everyone to begin using pkg.go.dev today for all of their needs and to file feedback! You can redirect all of your requests from godoc.org to pkg.go.dev, by clicking Always use pkg.go.dev
at the top of any page on godoc.org.
If you want to report a bug or have a feature suggestion, please first check the known issues to see if your issue is already being discussed. If an issue does not already exist, feel free to file an issue.
For answers to frequently asked questions, see go.dev/about.
You can also chat with us on the #tools slack channel on the Gophers slack.
We would love your help!
Our canonical Git repository is located at go.googlesource.com/pkgsite. There is a mirror of the repository at github.com/golang/pkgsite.
To contribute, please read our contributing guide.
Unless otherwise noted, the Go source files are distributed under the BSD-style license found in the LICENSE file.