regexp/syntax: reject large repetitions created by nesting small ones

Fixes #7609.

LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/150270043
2 files changed