tree 30b6a17890780741b8d23dcbbfeaaa41d3bcf6c5
parent 2b3b7078c6cbbea6fc9e20341b5261a6a0b363fd
author Rob Findley <rfindley@google.com> 1591753282 -0400
committer Robert Findley <rfindley@google.com> 1591927795 +0000

internal/postgres: fail hedged search on the first failed searcher

Waiting for subsequent searchers to complete after the first one fails
behaves poorly in overload scenarios.

For example, one of the ways that overload can manifest itself is by
connection exhaustion or Postgres actively refusing connections. Let's
assume this is happening, and Postgres randomly refuses half the
connections.

Now let's assume that we have a particular query for a rare term that is
much faster and cheaper when using the reverse index. To be concrete,
let's say that 'deep' search (using the index) costs 0.1, and 'popular'
search (doing a linear scan) costs 10 -- the units don't matter.
Normally, the expected query cost of this search is approximately 0.2:
0.1 for the deep search, and around 0.1 for the popular search that was
simultaneously occurring until deep completed.

If queries are randomly refused with a 50% probability, 25% the time
this query is going to cost 0.1, because popular search was refused and
deep succeeds, 25% of the time it's going to cost 10 (!), because deep
search was refused and we continued to wait for popular search to
complete, 25% of the time it costs ~0 because both searches fail, and
25% of the time it costs 0.2 as usual, because they both succeed.

That means that in this particular scenario, our query went from an
expected cost of 0.2, to an expected cost of ~2.6, an order of magnitude
more! Not a good thing to happen while the database is already
overloaded.

Instead, let's error out the search on the first error from any
searcher.

A test is added to exercise the behavior of hedgedSearch in the presence
of errors.

Also fix some miscellaneous stale comments along the way.

Change-Id: Id0496b3d402a403a17160c63edaacad537302284
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/766809
CI-Result: Cloud Build <devtools-proctor-result-processor@system.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>