go /
text /
a0f606852dec5915cd2e76d0f125eb0feb7cec79 encoding/unicode: add WhatWG-compliant UTF8 Decoder
For background see CL https://go-review.googlesource.com/#/c/11494.
I think this implemenation is better as it:
1) is faster, and
2) doesn't expose the alternative handling of invalid UTF-8 bytes to
outside this transform, even if only as an internal function.
This Transform is likely the only place one should ever handle the
invalid bytes differently from standard Go.
The algorithm to handle UTF-8 is similar to the one in unicode/utf8.
The table is put in a separate package as it can be used to optimize
other parts, such as the trie, which doesn't use the utf8 package,
and possibly other transforms.
The utf8internal package can also be a placeholder for other
often-needed, but low-level utf8 stuff (hangul conversion, surrugate
info, etc.).
Change-Id: Ib211033b79b438c6a016c1fb56f54e2be049ef1e
Reviewed-on: https://go-review.googlesource.com/17650
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
4 files changed
