design/2775-binary-only-packages.md - proposal - Git at Google


            
            # Proposal: Binary-Only Packages


            Author: Russ Cox


            Last updated: April 24, 2016


            Discussion at [golang.org/issue/2775](https://golang.org/issue/2775).


            ## Abstract


            We propose a way to incorporate binary-only packages (without complete source code) into a cmd/go workspace.


            ## Background


            It is common in C for a code author to provide a C header file and the compiled form of a library

            
            but not the complete source code.

            
            The go command has never supported this officially.


            In very early versions of Go, it was possible to arrange for a binary-only package simply by

            
            removing the source code after compiling.

            
            But that state looks the same as when the package source code has been deleted

            
            because the package itself is no longer available, in which case the compiled form

            
            should not continue to be used.

            
            For the past many years the Go command has assumed the latter.


            Then it was possible to arrange for a binary-only package by replacing the source code

            
            after compiling, while keeping the modification time of the source code older than

            
            the modification time of the compiled form.

            
            But in normal usage, removing an individual source file is cause for recompilation

            
            even though that cannot be seen in the modification times.

            
            To detect that situation,

            
            Go 1.5 started using the full set of source file names that went into

            
            a package as one input to a hash that produced the package's ``build ID''.

            
            If the go command's expected build ID does not match the compiled package's

            
            build ID, the compiled package is out of date, even if the modification times suggest

            
            otherwise (see [golang.org/cl/9154](https://golang.org/cl/9154)).


            From Go 1.5 then, to arrange for a binary-only package,

            
            it has been necessary to replace the source code after compiling

            
            but keep the same set of file names and also keep the source

            
            modification times older than the compiled package's.


            In the future we may experiment with including the source code itself

            
            in the hash that produces the build ID, which would completely

            
            defeat any attempt at binary-only packages.


            Fundamentally, as time goes on the go command gets better and better at detecting

            
            mismatches between the source code and the compiled form,

            
            yet in some cases it is explicitly desired that the source code not match

            
            the compiled form (specifically, that the source code not be included at all).

            
            If this usage is to keep working, it must be explicitly supported.


            ## Proposal


            We propose to add official support for binary-only packages to the cmd/go toolchain,

            
            by introduction of a new `//go:binary-only-package` comment.


            The go/build package's type Package will contain a new field `IncompleteSources bool`  indicating

            
            whether the `//go:binary-only-package` comment is present.


            The go command will refuse to recompile a package containing the comment.

            
            If a suitable binary form of the package is already installed, the go command will use it.

            
            Otherwise the go command will report that the binary form is missing and cannot be built.


            Users must install the package binary into the correct location in the $GOPATH/pkg tree

            
            themselves. Distributors of binary-only packages might distribute

            
            them as .zip files to be unpacked in the root of a $GOPATH, including files in both the src/ and pkg/

            
            tree.


            The “go get” command will still require complete source code and will not

            
            recognize or otherwise enable the distribution of binary-only packages.


            ## Rationale


            Various users have reported working with companies that want to provide them with

            
            binary but not source forms of purchased packages.

            
            We want to define an explicit way to do that instead of fielding bug reports

            
            each time the go command gets smarter about detecting source-vs-binary mismatches.


            The package source code itself must be present in some form,

            
            or else we can't tell if the package was deleted entirely (see background above).

            
            The implication is that it will simply not be the actual source code for the package.

            
            A special comment is a natural way to signal this situation,

            
            especially since the go command is already reading the source code

            
            for package name, import information, and build tag comments.

            
            Having a “fake” version of the source code also provides a way to supply

            
            documentation compatible with “go doc” and “godoc”

            
            even though the complete source code is missing.


            The compiled form of the package does contain information about the source code,

            
            for example source file names, type definitions for data structures used in the

            
            public API, and inlined function bodies. It is assumed that the distributors of

            
            binary-only packages understand that they include this information.


            ## Compatibility


            There are no problems raised by the

            
            [compatibility guidelines](https://golang.org/doc/go1compat).

            
            If anything, the explicit support will help keep such binary-only packages

            
            working better than they have in the past.


            To the extent that tools process source code and not compiled packages,

            
            those tools will not work with binary-only packages.

            
            The compiler and linker will continue to enforce that all packages be compiled

            
            with the same version of the toolchain: a binary-only package built with Go 1.4

            
            will not work with Go 1.5.

            
            Authors and users of binary-only packages must live with these implications.


            ## Implementation


            The implementation is essentially as described in the proposal section above.


            One additional detail is that the go command must load the build ID for the package

            
            in question from the compiled binary form directly, instead of deriving it from the

            
            source files.


            I will implement this change for Go 1.7.
	# Proposal: Binary-Only Packages

	Author: Russ Cox

	Last updated: April 24, 2016

	Discussion at [golang.org/issue/2775](https://golang.org/issue/2775).

	## Abstract

	We propose a way to incorporate binary-only packages (without complete source code) into a cmd/go workspace.

	## Background

	It is common in C for a code author to provide a C header file and the compiled form of a library
	but not the complete source code.
	The go command has never supported this officially.


	In very early versions of Go, it was possible to arrange for a binary-only package simply by
	removing the source code after compiling.
	But that state looks the same as when the package source code has been deleted
	because the package itself is no longer available, in which case the compiled form
	should not continue to be used.
	For the past many years the Go command has assumed the latter.

	Then it was possible to arrange for a binary-only package by replacing the source code
	after compiling, while keeping the modification time of the source code older than
	the modification time of the compiled form.
	But in normal usage, removing an individual source file is cause for recompilation
	even though that cannot be seen in the modification times.
	To detect that situation,
	Go 1.5 started using the full set of source file names that went into
	a package as one input to a hash that produced the package's ``build ID''.
	If the go command's expected build ID does not match the compiled package's
	build ID, the compiled package is out of date, even if the modification times suggest
	otherwise (see [golang.org/cl/9154](https://golang.org/cl/9154)).

	From Go 1.5 then, to arrange for a binary-only package,
	it has been necessary to replace the source code after compiling
	but keep the same set of file names and also keep the source
	modification times older than the compiled package's.

	In the future we may experiment with including the source code itself
	in the hash that produces the build ID, which would completely
	defeat any attempt at binary-only packages.

	Fundamentally, as time goes on the go command gets better and better at detecting
	mismatches between the source code and the compiled form,
	yet in some cases it is explicitly desired that the source code not match
	the compiled form (specifically, that the source code not be included at all).
	If this usage is to keep working, it must be explicitly supported.

	## Proposal

	We propose to add official support for binary-only packages to the cmd/go toolchain,
	by introduction of a new `//go:binary-only-package` comment.

	The go/build package's type Package will contain a new field `IncompleteSources bool` indicating
	whether the `//go:binary-only-package` comment is present.

	The go command will refuse to recompile a package containing the comment.
	If a suitable binary form of the package is already installed, the go command will use it.
	Otherwise the go command will report that the binary form is missing and cannot be built.

	Users must install the package binary into the correct location in the $GOPATH/pkg tree
	themselves. Distributors of binary-only packages might distribute
	them as .zip files to be unpacked in the root of a $GOPATH, including files in both the src/ and pkg/
	tree.

	The “go get” command will still require complete source code and will not
	recognize or otherwise enable the distribution of binary-only packages.

	## Rationale

	Various users have reported working with companies that want to provide them with
	binary but not source forms of purchased packages.
	We want to define an explicit way to do that instead of fielding bug reports
	each time the go command gets smarter about detecting source-vs-binary mismatches.

	The package source code itself must be present in some form,
	or else we can't tell if the package was deleted entirely (see background above).
	The implication is that it will simply not be the actual source code for the package.
	A special comment is a natural way to signal this situation,
	especially since the go command is already reading the source code
	for package name, import information, and build tag comments.
	Having a “fake” version of the source code also provides a way to supply
	documentation compatible with “go doc” and “godoc”
	even though the complete source code is missing.

	The compiled form of the package does contain information about the source code,
	for example source file names, type definitions for data structures used in the
	public API, and inlined function bodies. It is assumed that the distributors of
	binary-only packages understand that they include this information.

	## Compatibility

	There are no problems raised by the
	[compatibility guidelines](https://golang.org/doc/go1compat).
	If anything, the explicit support will help keep such binary-only packages
	working better than they have in the past.

	To the extent that tools process source code and not compiled packages,
	those tools will not work with binary-only packages.
	The compiler and linker will continue to enforce that all packages be compiled
	with the same version of the toolchain: a binary-only package built with Go 1.4
	will not work with Go 1.5.
	Authors and users of binary-only packages must live with these implications.

	## Implementation

	The implementation is essentially as described in the proposal section above.

	One additional detail is that the go command must load the build ID for the package
	in question from the compiled binary form directly, instead of deriving it from the
	source files.

	I will implement this change for Go 1.7.