design/4899-testing-helper.md - proposal - Git at Google


            
            # Proposal: testing: better support test helper functions with TB.Helper


            Author: Caleb Spare, based on previous work by Josh Bleecher Snyder


            Last updated: 2016-12-27


            Discussion at https://golang.org/issue/4899.


            ## Abstract


            This proposal is about fixing the long-standing issue

            
            [#4899](https://golang.org/issue/4899).


            When a test calls a helper function that invokes, for instance,

            
            `"*testing.T".Error`, the line number that is printed for the test failure

            
            indicates the `Error` call site as inside the helper method.

            
            This is almost always unhelpful for pinpointing the actual failure.


            We propose to add a new `testing.TB` method, `Helper`,

            
            which marks the calling function as a test helper.

            
            When logging test messages, package testing ignores frames

            
            that are inside marked helper functions.

            
            It prints the first stack position inside a non-helper function.


            ## Background


            In Go tests, it is common to use a helper function to perform some repeated

            
            non-trivial check.

            
            These are often of the form


                func helper(t *testing.T, other, args)


            though other variants exist.

            
            Such helper functions may be local to the test package or may come from external packages.

            
            There are many examples of such helper functions in the standard library tests.

            
            Some are listed below.


            When a helper function calls `t.Error`, `t.Fatal`, or a related method,

            
            the error message includes file:lineno output that indicates the location of the failure.

            
            The failure location is currently considered inside the helper method, which is unhelpful.


            The misplaced failure location also inhibits useful IDE features like

            
            automatically jumping to the failure position.


            There are a variety of workarounds to which people have resorted.


            ### 1. Ignore the problem, making it harder to debug test failures


            This is a common approach.


            If the helper is only called once from the `Test*` function,

            
            then the problem is less severe:

            
            the test failure prints the name of the `Test*` function that failed,

            
            and by locating the only call to the helper within that function,

            
            the user knows the failure site.

            
            This is just an annoyance.


            When the helper is called more than once, it can be impossible to locate the

            
            source of the failure without further debugging.


            A few examples of this pattern in the standard library:


            - cmd/cover: `func run(c *exec.Cmd, t *testing.T)`

            
            - cmd/go: the methods of `testgo`

            
            - compress/flate: `writeToType(t *testing.T, ttype string, bw *huffmanBitWriter, tok []token, input []byte)`

            
            - crypto/aes: `func mustPanic(t *testing.T, msg string, f func())`

            
            - database/sql: `func numPrepares(t *testing.T, db *DB) int`

            
            - encoding/json: `func diff(t *testing.T, a, b []byte)`

            
            - fmt: `func presentInMap(s string, a []string, t *testing.T)`, `func check(t *testing.T, got, want string)`

            
            - html/template: the methods of `testCase`

            
            - image/gif: `func try(t *testing.T, b []byte, want string)`

            
            - net/http: `func checker(t *testing.T) func(string, error)`

            
            - os: `func touch(t *testing.T, name string)`

            
            - reflect: `func assert(t *testing.T, s, want string)`

            
            - sync/atomic: `func shouldPanic(t *testing.T, name string, f func())`

            
            - text/scanner: `func checkPos(t *testing.T, got, want Position)` and some of its callers


            ### 2. Pass around more context to be printed as part of the error message


            This approach adds enough information to the failure message to pinpoint the

            
            source of failure, at the cost of greater burden on the test writer.

            
            The result still isn't entirely satisfactory for the test invoker:

            
            if the user only looks at the file:lineno in the failure message,

            
            they are still led astray until they examine the full message.


            Some standard library examples:


            - bytes: `func check(t *testing.T, testname string, buf *Buffer, s string)`

            
            - context: `func testDeadline(c Context, name string, failAfter time.Duration, t testingT)`

            
            - debug/gosym: `func testDeadline(c Context, name string, failAfter time.Duration, t testingT)`

            
            - mime/multipart: `func expectEq(t *testing.T, expected, actual, what string)`

            
            - strings: `func equal(m string, s1, s2 string, t *testing.T) bool`

            
            - text/scanner: `func checkTok(t *testing.T, s *Scanner, line int, got, want rune, text string)`


            ### 3. Use the \r workaround


            This technique is used by test helper packages in the wild.

            
            The idea is to print a carriage return from inside the test helper

            
            in order to hide the file:lineno printed by the testing package.

            
            Then the helper can print its own file:lineno and message.


            One example is

            
            [github.com/stretchr/testify](https://github.com/stretchr/testify/blob/2402e8e7a02fc811447d11f881aa9746cdc57983/assert/assertions.go#L226).


            ## Proposal


            We propose to add two methods in package testing:


                // Helper marks the current function as a test helper function.

            
                // When printing file and line information

            
                // from methods such as t.Error, t.Fatal, and t.Log,

            
                // the current function will be skipped.

            
                func (t *T) Helper()


                // same doc comment

            
                func (b *B) Helper()


            When package testing prints file:lineno, it walks up the stack,

            
            skipping helper functions, and chooses the first entry in a non-helper function.


            We also propose to add `Helper()` to the `testing.TB` interface.


            ## Rationale


            ### Alternative 1: allow the user to specify how many stack frames to skip


            An alternative fix is to give the user control over the number of stack frames to skip.

            
            This is similar to what package log already provides:


                func Output(calldepth int, s string) error

            
                func (l *Logger) Output(calldepth int, s string) error


            For instance, in https://golang.org/cl/12405043 @robpike writes


            >     // Up returns a *T object whose error reports identify the line n callers

            
            >     // up the frame.

            
            >     func (t *T) Up(n) *t { .... }

            
            >

            
            > Then you could write

            
            >

            
            >     t.Up(1).Error("this would be tagged with the caller's line number")


            @bradfitz mentions similar APIs in [#4899](https://golang.org/issue/4899) and

            
            [#14128](https://golang.org/issue/14128).


            `Helper` is easier to use, because the user doesn't have to think about stack frames,

            
            and it does not break when refactoring.


            Also, it is not always easy to decide how many frames to skip.

            
            A helper may be called through multiple paths, so it may be a variable depth

            
            from the desired logging site.

            
            For example, in the cmd/go tests, the `"*testgoData".must` helper is called directly

            
            by some tests, but is also called by other helpers such as `"*testgoData".cd`.

            
            Manual stack control would require the user to pass state into this method

            
            in order to know whether to skip one or two frames.

            
            Using the `Helper` API, the user would simply mark both `must` and `cd` as helpers.


            ### Alternative 2: use a special Logf/Errorf/Fatalf sentinel


            Another approach given by @bradfitz in

            
            [#14128](https://github.com/golang/go/issues/14128#issuecomment-176254702)

            
            is to provide a magic format value:


                 t.Logf("some value = %v", val, testing.NoDecorate)


            This seems roughly equivalent in power to our proposal, but it has downsides:


            * It breaks usual `printf` conventions (@adg [points out that vet would have to

            
              be aware of it](https://github.com/golang/go/issues/14128#issuecomment-176456878)).

            
            * The mechanism is unusual -- it lacks precedent in the standard library.

            
            * `NoDecorate` is less obvious in godoc than a TB method.

            
            * Every `testing.T` method in a helper method must be decorated.

            
            * It is not clear how to handle nested helpers other than by manually specifying

            
              the number of stack frames to skip, inheriting the problems of alternative 1.


            ## Compatibility


            Adding a method to `*testing.T` and `*testing.B` raises no compatibility issues.


            We will also add the method to the `testing.TB` interface.

            
            Normally changing interface method sets is verboten,

            
            but in this case it is be fine because `TB` has a private method

            
            specifically to prevent other implementations:


                // A private method to prevent users implementing the

            
                // interface and so future additions to it will not

            
                // violate Go 1 compatibility.

            
                private()


            ## Implementation


            @cespare will send a CL implementing `"testing.TB".Helper` based on

            
            @josharian's previous work in https://golang.org/cl/79890043.


            The CL will be sent before April 30, 2017 in order to make the 1.9 release cycle.


            ## Open issues


            This change directly solves [#4899](https://golang.org/issue/4899).
	# Proposal: testing: better support test helper functions with TB.Helper

	Author: Caleb Spare, based on previous work by Josh Bleecher Snyder

	Last updated: 2016-12-27

	Discussion at https://golang.org/issue/4899.

	## Abstract

	This proposal is about fixing the long-standing issue
	[#4899](https://golang.org/issue/4899).

	When a test calls a helper function that invokes, for instance,
	`"*testing.T".Error`, the line number that is printed for the test failure
	indicates the `Error` call site as inside the helper method.
	This is almost always unhelpful for pinpointing the actual failure.

	We propose to add a new `testing.TB` method, `Helper`,
	which marks the calling function as a test helper.
	When logging test messages, package testing ignores frames
	that are inside marked helper functions.
	It prints the first stack position inside a non-helper function.

	## Background

	In Go tests, it is common to use a helper function to perform some repeated
	non-trivial check.
	These are often of the form

	func helper(t *testing.T, other, args)

	though other variants exist.
	Such helper functions may be local to the test package or may come from external packages.
	There are many examples of such helper functions in the standard library tests.
	Some are listed below.

	When a helper function calls `t.Error`, `t.Fatal`, or a related method,
	the error message includes file:lineno output that indicates the location of the failure.
	The failure location is currently considered inside the helper method, which is unhelpful.

	The misplaced failure location also inhibits useful IDE features like
	automatically jumping to the failure position.

	There are a variety of workarounds to which people have resorted.

	### 1. Ignore the problem, making it harder to debug test failures

	This is a common approach.

	If the helper is only called once from the `Test*` function,
	then the problem is less severe:
	the test failure prints the name of the `Test*` function that failed,
	and by locating the only call to the helper within that function,
	the user knows the failure site.
	This is just an annoyance.

	When the helper is called more than once, it can be impossible to locate the
	source of the failure without further debugging.

	A few examples of this pattern in the standard library:

	- cmd/cover: `func run(c exec.Cmd, t testing.T)`
	- cmd/go: the methods of `testgo`
	- compress/flate: `writeToType(t testing.T, ttype string, bw huffmanBitWriter, tok []token, input []byte)`
	- crypto/aes: `func mustPanic(t *testing.T, msg string, f func())`
	- database/sql: `func numPrepares(t testing.T, db DB) int`
	- encoding/json: `func diff(t *testing.T, a, b []byte)`
	- fmt: `func presentInMap(s string, a []string, t testing.T)`, `func check(t testing.T, got, want string)`
	- html/template: the methods of `testCase`
	- image/gif: `func try(t *testing.T, b []byte, want string)`
	- net/http: `func checker(t *testing.T) func(string, error)`
	- os: `func touch(t *testing.T, name string)`
	- reflect: `func assert(t *testing.T, s, want string)`
	- sync/atomic: `func shouldPanic(t *testing.T, name string, f func())`
	- text/scanner: `func checkPos(t *testing.T, got, want Position)` and some of its callers

	### 2. Pass around more context to be printed as part of the error message

	This approach adds enough information to the failure message to pinpoint the
	source of failure, at the cost of greater burden on the test writer.
	The result still isn't entirely satisfactory for the test invoker:
	if the user only looks at the file:lineno in the failure message,
	they are still led astray until they examine the full message.

	Some standard library examples:

	- bytes: `func check(t testing.T, testname string, buf Buffer, s string)`
	- context: `func testDeadline(c Context, name string, failAfter time.Duration, t testingT)`
	- debug/gosym: `func testDeadline(c Context, name string, failAfter time.Duration, t testingT)`
	- mime/multipart: `func expectEq(t *testing.T, expected, actual, what string)`
	- strings: `func equal(m string, s1, s2 string, t *testing.T) bool`
	- text/scanner: `func checkTok(t testing.T, s Scanner, line int, got, want rune, text string)`

	### 3. Use the \r workaround

	This technique is used by test helper packages in the wild.
	The idea is to print a carriage return from inside the test helper
	in order to hide the file:lineno printed by the testing package.
	Then the helper can print its own file:lineno and message.

	One example is
	[github.com/stretchr/testify](https://github.com/stretchr/testify/blob/2402e8e7a02fc811447d11f881aa9746cdc57983/assert/assertions.go#L226).

	## Proposal

	We propose to add two methods in package testing:

	// Helper marks the current function as a test helper function.
	// When printing file and line information
	// from methods such as t.Error, t.Fatal, and t.Log,
	// the current function will be skipped.
	func (t *T) Helper()

	// same doc comment
	func (b *B) Helper()

	When package testing prints file:lineno, it walks up the stack,
	skipping helper functions, and chooses the first entry in a non-helper function.

	We also propose to add `Helper()` to the `testing.TB` interface.

	## Rationale

	### Alternative 1: allow the user to specify how many stack frames to skip

	An alternative fix is to give the user control over the number of stack frames to skip.
	This is similar to what package log already provides:

	func Output(calldepth int, s string) error
	func (l *Logger) Output(calldepth int, s string) error

	For instance, in https://golang.org/cl/12405043 @robpike writes

	> // Up returns a *T object whose error reports identify the line n callers
	> // up the frame.
	> func (t T) Up(n) t { .... }
	>
	> Then you could write
	>
	> t.Up(1).Error("this would be tagged with the caller's line number")

	@bradfitz mentions similar APIs in [#4899](https://golang.org/issue/4899) and
	[#14128](https://golang.org/issue/14128).

	`Helper` is easier to use, because the user doesn't have to think about stack frames,
	and it does not break when refactoring.

	Also, it is not always easy to decide how many frames to skip.
	A helper may be called through multiple paths, so it may be a variable depth
	from the desired logging site.
	For example, in the cmd/go tests, the `"*testgoData".must` helper is called directly
	by some tests, but is also called by other helpers such as `"*testgoData".cd`.
	Manual stack control would require the user to pass state into this method
	in order to know whether to skip one or two frames.
	Using the `Helper` API, the user would simply mark both `must` and `cd` as helpers.

	### Alternative 2: use a special Logf/Errorf/Fatalf sentinel

	Another approach given by @bradfitz in
	[#14128](https://github.com/golang/go/issues/14128#issuecomment-176254702)
	is to provide a magic format value:

	t.Logf("some value = %v", val, testing.NoDecorate)

	This seems roughly equivalent in power to our proposal, but it has downsides:

	* It breaks usual `printf` conventions (@adg [points out that vet would have to
	be aware of it](https://github.com/golang/go/issues/14128#issuecomment-176456878)).
	* The mechanism is unusual -- it lacks precedent in the standard library.
	* `NoDecorate` is less obvious in godoc than a TB method.
	* Every `testing.T` method in a helper method must be decorated.
	* It is not clear how to handle nested helpers other than by manually specifying
	the number of stack frames to skip, inheriting the problems of alternative 1.

	## Compatibility

	Adding a method to `testing.T` and `testing.B` raises no compatibility issues.

	We will also add the method to the `testing.TB` interface.
	Normally changing interface method sets is verboten,
	but in this case it is be fine because `TB` has a private method
	specifically to prevent other implementations:

	// A private method to prevent users implementing the
	// interface and so future additions to it will not
	// violate Go 1 compatibility.
	private()

	## Implementation

	@cespare will send a CL implementing `"testing.TB".Helper` based on
	@josharian's previous work in https://golang.org/cl/79890043.

	The CL will be sent before April 30, 2017 in order to make the 1.9 release cycle.

	## Open issues

	This change directly solves [#4899](https://golang.org/issue/4899).