commit 5890e25b7ccb2d2249b2f8a02ef5dbc36047868b
author Russ Cox <rsc@golang.org> Thu Jul 12 12:36:34 2018 -0400
committer Russ Cox <rsc@golang.org> Tue Jul 17 14:17:23 2018 +0000
tree 804a77bce36b49f333f7823dbbea65ce9cd4c4fa
parent b59b42cee8c1ca209dafd952da872fd493b77405

cmd/go: add new test script facility

The original cmd/go tests were tiny shell scripts
written against a library of shell functions.
They were okay to write but difficult to run:
you couldn't select individual tests (with -run)
they didn't run on Windows, they were slow, and so on.

CL 10464 introduced go_test.go's testgo framework
and later CLs translated the test shell script over to
individual go tests. This let us run tests selectively,
run tests on Windows, run tests in parallel, isolate
different tests, and so on. It was a big advance.

The tests had always been awkward to write.
Here was the first test in test.bash:

	TEST 'file:line in error messages'
	# Test that error messages have file:line information at beginning of
	# the line. Also test issue 4917: that the error is on stderr.
	d=$(TMPDIR=/var/tmp mktemp -d -t testgoXXX)
	fn=$d/err.go
	echo "package main" > $fn
	echo 'import "bar"' >> $fn
	./testgo run $fn 2>$d/err.out || true
	if ! grep -q "^$fn:" $d/err.out; then
		echo "missing file:line in error message"
		cat $d/err.out
		ok=false
	fi
	rm -r $d

The final Go version of this test was:

	func TestFileLineInErrorMessages(t *testing.T) {
		tg := testgo(t)
		defer tg.cleanup()
		tg.parallel()
		tg.tempFile("err.go", `package main; import "bar"`)
		path := tg.path("err.go")
		tg.runFail("run", path)
		shortPath := path
		if rel, err := filepath.Rel(tg.pwd(), path); err == nil && len(rel) < len(path) {
			shortPath = rel
		}
		tg.grepStderr("^"+regexp.QuoteMeta(shortPath)+":", "missing file:line in error message")
	}

It's better but still quite difficult to skim.

This CL introduces a new facility meant as a successor to the testgo
approach that brings back the style of writing tests as little scripts,
but they are now scripts in a built-for-purpose shell-like language,
not bash itself. In this new form, the test above is a single file,
testdata/script/fileline.txt:

	# look for short, relative file:line in error message
	! go run ../../gopath/x/y/z/err.go
	stderr ^..[\\/]x[\\/]y[\\/]z[\\/]err.go:

	-- ../x/y/z/err.go --
	package main; import "bar"

The file is a txtar text archive (see CL 123359) in which the leading comment
is the test script and the files are the initial state of the temporary file
system where the script runs.

Each script runs as a subtest, so that they can still be selected individually.

The scripts are kept isolated from each other by default,
so all script subtests are treated as parallel tests, for the
testing package to run in parallel. Even for the 15 tests in
this CL, that cuts the time for TestScript from 5.5s to 2.5s.

The scripts do not have access to the cmd/go source directory,
nor to cmd/go/testdata, so they are prevented from creating temporary
files in those places or modifying existing ones. (Many existing tests
scribble in testdata, unfortunately, especially testdata/pkg when
they run builds with GOPATH=testdata.)

This CL introduces the script facility and converts 15 tests.
The txtar archive form will allow us to delete the large trees of trivial
files in testdata; a few are deleted in this CL.

See testdata/script/README for details and a larger conversion example.

As part of converting testdata/script/test_badtest.txt,
I discovered that 'go test' was incorrectly printing a FAIL line
to stderr (not stdout) in one corner case. This CL fixes that
to keep the test passing.

Future CLs will convert more tests.

Change-Id: I11aa9e18dd2d4c7dcd8e310dbdc6a1ea5f7e54c1
Reviewed-on: https://go-review.googlesource.com/123577
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>