unicode/norm: simplify short source detection in Form.transform

Form.transform returns ErrShortSrc when not all source bytes
have been processed:

	if err == nil && n < rb.nsrc && !atEOF {
		err = transform.ErrShortSrc
	}

Simplify this check to check whether all bytes in src have been
processed (in which case the Transformer interface says we need
to return ErrShortSrc):

	if err == nil && nSrc < rb.nsrc {
		err = transform.ErrShortSrc
	}

This change has no actual practical effect.

Changing n (number of bytes handled in the current iteration)
to nSrc (total number of bytes handled) looks like a bugfix,
since we don't want to return ErrShortSrc just because
we ran for more than one iteration.

However, when atEOF==false, Form.transform never processes
its complete input (Because quickSpan won't process *its*
complete input except when at EOF, I believe because it's
waiting to see if any future characters combine with the
current input suffix.) It always returns ErrShortSrc
with or without this change.

Dropping !atEOF looks like it would cause us to return
ErrShortSrc when at EOF and not processing all source
characters (which would be correct behavior), but
Form.transform always processes the complete source when at EOF.

To summarize: This change is a complete no-op, but the
code is simpler, more obviously correct, and more robust
to future changes elsewhere. (For example, if something
causes Form.transform to fail to process the complete source
when at EOF, it will now correctly return ErrShortSrc.)

Change-Id: Id76f3dea328ac8407936adfcfb14b1066a6a6964
Reviewed-on: https://go-review.googlesource.com/c/text/+/794102
Reviewed-by: Neal Patel <nealpatel@google.com>
Reviewed-by: Neal Patel <neal@golang.org>
LUCI-TryBot-Result: golang-scoped@luci-project-accounts.iam.gserviceaccount.com <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Damien Neil <dneil@google.com>
1 file changed
tree: 20cb9a417d496ad1ce4c7b5f5d846b75a44815b0
  1. cases/
  2. cmd/
  3. collate/
  4. currency/
  5. date/
  6. encoding/
  7. feature/
  8. internal/
  9. language/
  10. message/
  11. number/
  12. runes/
  13. search/
  14. secure/
  15. transform/
  16. unicode/
  17. width/
  18. .gitattributes
  19. .gitignore
  20. codereview.cfg
  21. CONTRIBUTING.md
  22. doc.go
  23. gen.go
  24. go.mod
  25. go.sum
  26. LICENSE
  27. PATENTS
  28. README.md
README.md

Go Text

Go Reference

This repository holds supplementary Go packages for text processing, many involving Unicode.

CLDR Versioning

It is important that the Unicode version used in x/text matches the one used by your Go compiler. The x/text repository supports multiple versions of Unicode and will match the version of Unicode to that of the Go compiler. At the moment this is supported for Go compilers from version 1.7.

Contribute

To submit changes to this repository, see http://go.dev/doc/contribute.

The git repository is https://go.googlesource.com/text.

To generate the tables in this repository (except for the encoding tables), run go generate from this directory. By default tables are generated for the Unicode version in core and the CLDR version defined in golang.org/x/text/unicode/cldr.

Running go generate will as a side effect create a DATA subdirectory in this directory, which holds all files that are used as a source for generating the tables. This directory will also serve as a cache.

Testing

Run

go test ./...

from this directory to run all tests. Add the “-tags icu” flag to also run ICU conformance tests (if available). This requires that you have the correct ICU version installed on your system.

TODO:

  • updating unversioned source files.

Generating Tables

To generate the tables in this repository (except for the encoding tables), run go generate from this directory. By default tables are generated for the Unicode version in core and the CLDR version defined in golang.org/x/text/unicode/cldr.

Running go generate will as a side effect create a DATA subdirectory in this directory which holds all files that are used as a source for generating the tables. This directory will also serve as a cache.

Versions

To update a Unicode version run

UNICODE_VERSION=x.x.x go generate

where x.x.x must correspond to a directory in https://www.unicode.org/Public/. If this version is newer than the version in core it will also update the relevant packages there. The idna package in x/net will always be updated.

To update a CLDR version run

CLDR_VERSION=version go generate

where version must correspond to a directory in https://www.unicode.org/Public/cldr/.

Note that the code gets adapted over time to changes in the data and that backwards compatibility is not maintained. So updating to a different version may not work.

The files in DATA/{iana|icu|w3|whatwg} are currently not versioned.

Report Issues

The main issue tracker for the text repository is located at https://go.dev/issues. Prefix your issue with “x/text:” in the subject line, so it is easy to find.