tree bd21f64d98108b94df2ea3a8e2bed615d1593937
parent 87ad79f47ac96edbf2568691d123d113fd7ac535
author Marcel van Lohuizen <mpvl@golang.org> 1449651197 +0100
committer Marcel van Lohuizen <mpvl@golang.org> 1450089912 +0000

encoding/unicode: add WhatWG-compliant UTF8 Decoder

For background see CL https://go-review.googlesource.com/#/c/11494.
I think this implemenation is better as it:
1) is faster, and
2) doesn't expose the alternative handling of invalid UTF-8 bytes to
outside this transform, even if only as an internal function.
This Transform is likely the only place one should ever handle the
invalid bytes differently from standard Go.

The algorithm to handle UTF-8 is similar to the one in unicode/utf8.

The table is put in a separate package as it can be used to optimize
other parts, such as the trie, which doesn't use the utf8 package,
and possibly other transforms.
The utf8internal package can also be a placeholder for other
often-needed, but low-level utf8 stuff (hangul conversion, surrugate
info, etc.).

Change-Id: Ib211033b79b438c6a016c1fb56f54e2be049ef1e
Reviewed-on: https://go-review.googlesource.com/17650
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>