[release-branch.go1.8] http2: fix lock contention slowdown due to gracefulShutdownCh
gracefulShutdownCh is shared by all connections in a server. When a
server accumulates many connections (e.g., 5000 in the kubemark-5000
benchmark), we have 5000 serverConn.serve goroutines selecting on this
channel. This means 5000 goroutines hammer the channel's lock, which
causes severe lock contention.
The fix in this CL is to make a local proxy for gracefulShutdownCh in
each connection so that each connection selects on gracefulShutdownCh
at most once per connection rather than once per serverConn.serve loop
This fix is intended to be backported quickly into Go 1.8.2. The
downside of this fix is 2KB extra stack usage per connection. A better
fix will be implemented in Go 1.9.
Unfortunately, I have been unable to reproduce this problem locally.
This fix was verified by the kubernetes team. See:
Run-TryBot: Tom Bergan <firstname.lastname@example.org>
TryBot-Result: Gobot Gobot <email@example.com>
Reviewed-by: Brad Fitzpatrick <firstname.lastname@example.org>
(cherry picked from commit d3ede01617fc9ddf0827da5a431705cf91cf563d)
Run-TryBot: Brad Fitzpatrick <email@example.com>
Reviewed-by: Chris Broadfoot <firstname.lastname@example.org>
1 file changed