internal/frontend: limit the offset given to search
Searching with large offsets can use a lot of DB CPU, since the
database must process that many results before it even arrives
at the results it's going to return.
So limit the maximum offset. In the normal case, where a person is
manually advancing through search results, they would have to visit a
thousand 10-result pages before hitting the limit, so it's unlikely
that will happen. This should mostly stop scrapers.
Change-Id: I9e2740d020bc3f117c4f98b5cdd7eef03f71583e
Reviewed-on: https://go-review.googlesource.com/c/pkgsite/+/258198
Trust: Jonathan Amsterdam <jba@google.com>
Run-TryBot: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
diff --git a/internal/frontend/search.go b/internal/frontend/search.go
index 15b68a0..08eeb3e 100644
--- a/internal/frontend/search.go
+++ b/internal/frontend/search.go
@@ -107,6 +107,10 @@
// maxSearchQueryLength.
const maxSearchQueryLength = 500
+// maxSearchOffset is the maximum allowed. offset into the search results.
+// This prevents some very CPU-intensive queries from running.
+const maxSearchOffset = 100
+
// serveSearch applies database data to the search template. Handles endpoint
// /search?q=<query>. If <query> is an exact match for a package path, the user
// will be redirected to the details page.
@@ -139,7 +143,18 @@
http.Redirect(w, r, path, http.StatusFound)
return nil
}
- page, err := fetchSearchPage(ctx, db, query, newPaginationParams(r, defaultSearchLimit))
+ pageParams := newPaginationParams(r, defaultSearchLimit)
+ if pageParams.offset() > maxSearchOffset {
+ return &serverError{
+ status: http.StatusBadRequest,
+ epage: &errorPage{
+ messageTemplate: template.MakeTrustedTemplate(
+ `<h3 class="Error-message">Search offset too large.</h3>`),
+ },
+ }
+ }
+
+ page, err := fetchSearchPage(ctx, db, query, pageParams)
if err != nil {
return fmt.Errorf("fetchSearchPage(ctx, db, %q): %v", query, err)
}
diff --git a/internal/frontend/server_test.go b/internal/frontend/server_test.go
index a046cbc..df2ed7c 100644
--- a/internal/frontend/server_test.go
+++ b/internal/frontend/server_test.go
@@ -395,6 +395,11 @@
text(sample.ModulePath+"/foo")))),
},
{
+ name: "search large offset",
+ urlPath: fmt.Sprintf("/search?q=github.com&page=1002"),
+ wantStatusCode: http.StatusBadRequest,
+ },
+ {
name: "package default",
urlPath: fmt.Sprintf("/%s", sample.PackagePath),
wantStatusCode: http.StatusOK,
