doc/articles/go_command.html - go - Git at Google

 <!--{
 	"title": "About the go command"
 }-->

 <p>The Go distribution includes a command, named
 "<code><a href="/cmd/go/">go</a></code>", that
 automates the downloading, building, installation, and testing of Go packages
 and commands.  This document talks about why we wrote a new command, what it
 is, what it's not, and how to use it.</p>

 <h2>Motivation</h2>

 <p>You might have seen early Go talks in which Rob Pike jokes that the idea
 for Go arose while waiting for a large Google server to compile.  That
 really was the motivation for Go: to build a language that worked well
 for building the large software that Google writes and runs. It was
 clear from the start that such a language must provide a way to
 express dependencies between code libraries clearly, hence the package
 grouping and the explicit import blocks.  It was also clear from the
 start that you might want arbitrary syntax for describing the code
 being imported; this is why import paths are string literals.</p>

 <p>An explicit goal for Go from the beginning was to be able to build Go
 code using only the information found in the source itself, not
 needing to write a makefile or one of the many modern replacements for
 makefiles.  If Go needed a configuration file to explain how to build
 your program, then Go would have failed.</p>

 <p>At first, there was no Go compiler, and the initial development
 focused on building one and then building libraries for it. For
 expedience, we postponed the automation of building Go code by using
 make and writing makefiles.  When compiling a single package involved
 multiple invocations of the Go compiler, we even used a program to
 write the makefiles for us.  You can find it if you dig through the
 repository history.</p>

 <p>The purpose of the new go command is our return to this ideal, that Go
 programs should compile without configuration or additional effort on
 the part of the developer beyond writing the necessary import
 statements.</p>

 <h2>Configuration versus convention</h2>

 <p>The way to achieve the simplicity of a configuration-free system is to
 establish conventions. The system works only to the extent that those conventions
 are followed. When we first launched Go, many people published packages that
 had to be installed in certain places, under certain names, using certain build
 tools, in order to be used. That's understandable: that's the way it works in
 most other languages. Over the last few years we consistently reminded people
 about the <code>goinstall</code> command
 (now replaced by <a href="/cmd/go/#Download_and_install_packages_and_dependencies"><code>go get</code></a>)
 and its conventions: first, that the import path is derived in a known way from
 the URL of the source code; second, that that the place to store the sources in
 the local file system is derived in a known way from the import path; third,
 that each directory in a source tree corresponds to a single package; and
 fourth, that the package is built using only information in the source code.
 Today, the vast majority of packages follow these conventions.
 The Go ecosystem is simpler and more powerful as a result.</p>

 <p>We received many requests to allow a makefile in a package directory to
 provide just a little extra configuration beyond what's in the source code.
 But that would have introduced new rules. Because we did not accede to such
 requests, we were able to write the go command and eliminate our use of make
 or any other build system.</p>

 <p>It is important to understand that the go command is not a general
 build tool. It cannot be configured and it does not attempt to build
 anything but Go packages.  These are important simplifying
 assumptions: they simplify not only the implementation but also, more
 important, the use of the tool itself.</p>

 <h2>Go's conventions</h2>

 <p>The <code>go</code> command requires that code adheres to a few key,
 well-established conventions.</p>

 <p>First, the import path is derived in an known way from the URL of the
 source code.  For Bitbucket, GitHub, Google Code, and Launchpad, the
 root directory of the repository is identified by the repository's
 main URL, without the <code>http://</code> prefix.  Subdirectories are named by
 adding to that path.  For example, the supplemental networking
 libraries for Go are obtained by running</p>

 <pre>
 hg clone http://code.google.com/p/go.net
 </pre>

 <p>and thus the import path for the root directory of that repository is
 "<code>code.google.com/p/go.net</code>".  The websocket package is stored in a
 subdirectory, so its import path is
 "<code>code.google.com/p/go.net/websocket</code>".</p>

 <p>These paths are on the long side, but in exchange we get an
 automatically managed name space for import paths and the ability for
 a tool like the go command to look at an unfamiliar import path and
 deduce where to obtain the source code.</p>

 <p>Second, the place to store sources in the local file system is derived
 in a known way from the import path.  Specifically, the first choice
 is <code>$GOPATH/src/&lt;import-path&gt;</code>.  If <code>$GOPATH</code> is
 unset, the go command will fall back to storing source code alongside the
 standard Go packages, in <code>$GOROOT/src/pkg/&lt;import-path&gt;</code>.
 If <code>$GOPATH</code> is set to a list of paths, the go command tries
 <code>&lt;dir&gt;/src/&lt;import-path&gt;</code> for each of the directories in
 that list.</p>

 <p>Each of those trees contains, by convention, a top-level directory named
 "<code>bin</code>", for holding compiled executables, and a top-level directory
 named "<code>pkg</code>", for holding compiled packages that can be imported,
 and the "<code>src</code>" directory, for holding package source files.
 Imposing this structure lets us keep each of these directory trees
 self-contained: the compiled form and the sources are always near each
 other.</p>

 <p>These naming conventions also let us work in the reverse direction,
 from a directory name to its import path. This mapping is important
 for many of the go command's subcommands, as we'll see below.</p>

 <p>Third, each directory in a source tree corresponds to a single
 package. By restricting a directory to a single package, we don't have
 to create hybrid import paths that specify first the directory and
 then the package within that directory.  Also, most file management
 tools and UIs work on  directories as fundamental units.  Tying the
 fundamental Go unit&mdash;the package&mdash;to file system structure means
 that file system tools become Go package tools.  Copying, moving, or
 deleting a package corresponds to copying, moving, or deleting a
 directory.</p>

 <p>Fourth, each package is built using only the information present in
 the source files.  This makes it much more likely that the tool will
 be able to adapt to changing build environments and conditions. For
 example, if we allowed extra configuration such as compiler flags or
 command line recipes, then that configuration would need to be updated
 each time the build tools changed; it would also be inherently tied
 to the use of a specific tool chain.</p>

 <h2>Getting started with the go command</h2>

 <p>Finally, a quick tour of how to use the go command, to supplement
 the information in <a href="/doc/code.html">How to Write Go Code</a>,
 which you might want to read first.  Assuming you want
 to keep your source code separate from the Go distribution source
 tree, the first step is to set <code>$GOPATH</code>, the one piece of global
 configuration that the go command needs.  The <code>$GOPATH</code> can be a
 list of directories, but by far the most common usage should be to set it to a
 single directory.  In particular, you do not need a separate entry in
 <code>$GOPATH</code> for each of your projects.  One <code>$GOPATH</code> can
 support many projects.</p>

 <p>Here’s an example.  Let’s say we decide to keep our Go code in the directory
 <code>$HOME/mygo</code>.  We need to create that directory and set
 <code>$GOPATH</code> accordingly.</p>

 <pre>
 $ mkdir $HOME/mygo
 $ export GOPATH=$HOME/mygo
 $
 </pre>

 <p>Into this directory, we now add some source code.  Suppose we want to use
 the indexing library from the codesearch project along with a left-leaning
 red-black tree.  We can install both with the "<code>go get</code>"
 subcommand:</p>

 <pre>
 $ go get code.google.com/p/codesearch/index
 $ go get github.com/petar/GoLLRB/llrb
 $
 </pre>

 <p>Both of these projects are now downloaded and installed into our
 <code>$GOPATH</code> directory. The one tree now contains the two directories
 <code>src/code.google.com/p/codesearch/index/</code> and
 <code>src/github.com/petar/GoLLRB/llrb/</code>, along with the compiled
 packages (in <code>pkg/</code>) for those libraries and their dependencies.</p>

 <p>Because we used version control systems (Mercurial and Git) to check
 out the sources, the source tree also contains the other files in the
 corresponding repositories, such as related packages. The "<code>go list</code>"
 subcommand lists the import paths corresponding to its arguments, and
 the pattern "<code>./...</code>" means start in the current directory
 ("<code>./</code>") and find all packages below that directory
 ("<code>...</code>"):</p>

 <pre>
 $ go list ./...
 code.google.com/p/codesearch/cmd/cgrep
 code.google.com/p/codesearch/cmd/cindex
 code.google.com/p/codesearch/cmd/csearch
 code.google.com/p/codesearch/index
 code.google.com/p/codesearch/regexp
 code.google.com/p/codesearch/sparse
 github.com/petar/GoLLRB/example
 github.com/petar/GoLLRB/llrb
 $
 </pre>

 <p>We can also test those packages:</p>

 <pre>
 $ go test ./...
 ?       code.google.com/p/codesearch/cmd/cgrep   [no test files]
 ?       code.google.com/p/codesearch/cmd/cindex  [no test files]
 ?       code.google.com/p/codesearch/cmd/csearch [no test files]
 ok      code.google.com/p/codesearch/index       0.239s
 ok      code.google.com/p/codesearch/regexp      0.021s
 ?       code.google.com/p/codesearch/sparse      [no test files]
 ?       github.com/petar/GoLLRB/example          [no test files]
 ok      github.com/petar/GoLLRB/llrb             0.231s
 $
 </pre>

 <p>If a go subcommand is invoked with no paths listed, it operates on the
 current directory:</p>

 <pre>
 $ cd $GOPATH/src/code.google.com/p/codesearch/regexp
 $ go list
 code.google.com/p/codesearch/regexp
 $ go test -v
 === RUN TestNstateEnc
 --- PASS: TestNstateEnc (0.00 seconds)
 === RUN TestMatch
 --- PASS: TestMatch (0.01 seconds)
 === RUN TestGrep
 --- PASS: TestGrep (0.00 seconds)
 PASS
 ok      code.google.com/p/codesearch/regexp     0.021s
 $ go install
 $
 </pre>

 <p>That "<code>go install</code>" subcommand installs the latest copy of the
 package into the pkg directory. Because the go command can analyze the
 dependency graph, "<code>go install</code>" also installs any packages that
 this package imports but that are out of date, recursively.</p>

 <p>Notice that "<code>go install</code>" was able to determine the name of the
 import path for the package in the current directory, because of the convention
 for directory naming.  It would be a little more convenient if we could pick
 the name of the directory where we kept source code, and we probably wouldn't
 pick such a long name, but that ability would require additional configuration
 and complexity in the tool. Typing an extra directory name or two is a small
 price to pay for the increased simplicity and power.</p>

 <p>As the example shows, it’s fine to work with packages from many different
 projects at once within a single <code>$GOPATH</code> root directory.</p>

 <h2>Limitations</h2>

 <p>As mentioned above, the go command is not a general-purpose build
 tool. In particular, it does not have any facility for generating Go
 source files during a build.  Instead, if you want to use a tool like
 yacc or the protocol buffer compiler, you will need to write a
 makefile (or a configuration file for the build tool of your choice)
 to generate the Go files and then check those generated source files
 into your repository. This is more work for you, the package author,
 but it is significantly less work for your users, who can use
 "<code>go get</code>" without needing to obtain and build
 any additional tools.</p>

 <h2>More information</h2>

 <p>For more information, read <a href="/doc/code.html">How to Write Go Code</a>
 and see the <a href="/cmd/go/">go command documentation</a>.</p>
	<!--{
	"title": "About the go command"
	}-->

	<p>The Go distribution includes a command, named
	"<code><a href="/cmd/go/">go</a></code>", that
	automates the downloading, building, installation, and testing of Go packages
	and commands. This document talks about why we wrote a new command, what it
	is, what it's not, and how to use it.</p>

	<h2>Motivation</h2>

	<p>You might have seen early Go talks in which Rob Pike jokes that the idea
	for Go arose while waiting for a large Google server to compile. That
	really was the motivation for Go: to build a language that worked well
	for building the large software that Google writes and runs. It was
	clear from the start that such a language must provide a way to
	express dependencies between code libraries clearly, hence the package
	grouping and the explicit import blocks. It was also clear from the
	start that you might want arbitrary syntax for describing the code
	being imported; this is why import paths are string literals.</p>

	<p>An explicit goal for Go from the beginning was to be able to build Go
	code using only the information found in the source itself, not
	needing to write a makefile or one of the many modern replacements for
	makefiles. If Go needed a configuration file to explain how to build
	your program, then Go would have failed.</p>

	<p>At first, there was no Go compiler, and the initial development
	focused on building one and then building libraries for it. For
	expedience, we postponed the automation of building Go code by using
	make and writing makefiles. When compiling a single package involved
	multiple invocations of the Go compiler, we even used a program to
	write the makefiles for us. You can find it if you dig through the
	repository history.</p>

	<p>The purpose of the new go command is our return to this ideal, that Go
	programs should compile without configuration or additional effort on
	the part of the developer beyond writing the necessary import
	statements.</p>

	<h2>Configuration versus convention</h2>

	<p>The way to achieve the simplicity of a configuration-free system is to
	establish conventions. The system works only to the extent that those conventions
	are followed. When we first launched Go, many people published packages that
	had to be installed in certain places, under certain names, using certain build
	tools, in order to be used. That's understandable: that's the way it works in
	most other languages. Over the last few years we consistently reminded people
	about the <code>goinstall</code> command
	(now replaced by <a href="/cmd/go/#Download_and_install_packages_and_dependencies"><code>go get</code></a>)
	and its conventions: first, that the import path is derived in a known way from
	the URL of the source code; second, that that the place to store the sources in
	the local file system is derived in a known way from the import path; third,
	that each directory in a source tree corresponds to a single package; and
	fourth, that the package is built using only information in the source code.
	Today, the vast majority of packages follow these conventions.
	The Go ecosystem is simpler and more powerful as a result.</p>

	<p>We received many requests to allow a makefile in a package directory to
	provide just a little extra configuration beyond what's in the source code.
	But that would have introduced new rules. Because we did not accede to such
	requests, we were able to write the go command and eliminate our use of make
	or any other build system.</p>

	<p>It is important to understand that the go command is not a general
	build tool. It cannot be configured and it does not attempt to build
	anything but Go packages. These are important simplifying
	assumptions: they simplify not only the implementation but also, more
	important, the use of the tool itself.</p>

	<h2>Go's conventions</h2>

	<p>The <code>go</code> command requires that code adheres to a few key,
	well-established conventions.</p>

	<p>First, the import path is derived in an known way from the URL of the
	source code. For Bitbucket, GitHub, Google Code, and Launchpad, the
	root directory of the repository is identified by the repository's
	main URL, without the <code>http://</code> prefix. Subdirectories are named by
	adding to that path. For example, the supplemental networking
	libraries for Go are obtained by running</p>

	<pre>
	hg clone http://code.google.com/p/go.net
	</pre>

	<p>and thus the import path for the root directory of that repository is
	"<code>code.google.com/p/go.net</code>". The websocket package is stored in a
	subdirectory, so its import path is
	"<code>code.google.com/p/go.net/websocket</code>".</p>

	<p>These paths are on the long side, but in exchange we get an
	automatically managed name space for import paths and the ability for
	a tool like the go command to look at an unfamiliar import path and
	deduce where to obtain the source code.</p>

	<p>Second, the place to store sources in the local file system is derived
	in a known way from the import path. Specifically, the first choice
	is <code>$GOPATH/src/<import-path></code>. If <code>$GOPATH</code> is
	unset, the go command will fall back to storing source code alongside the
	standard Go packages, in <code>$GOROOT/src/pkg/<import-path></code>.
	If <code>$GOPATH</code> is set to a list of paths, the go command tries
	<code><dir>/src/<import-path></code> for each of the directories in
	that list.</p>

	<p>Each of those trees contains, by convention, a top-level directory named
	"<code>bin</code>", for holding compiled executables, and a top-level directory
	named "<code>pkg</code>", for holding compiled packages that can be imported,
	and the "<code>src</code>" directory, for holding package source files.
	Imposing this structure lets us keep each of these directory trees
	self-contained: the compiled form and the sources are always near each
	other.</p>

	<p>These naming conventions also let us work in the reverse direction,
	from a directory name to its import path. This mapping is important
	for many of the go command's subcommands, as we'll see below.</p>

	<p>Third, each directory in a source tree corresponds to a single
	package. By restricting a directory to a single package, we don't have
	to create hybrid import paths that specify first the directory and
	then the package within that directory. Also, most file management
	tools and UIs work on directories as fundamental units. Tying the
	fundamental Go unit—the package—to file system structure means
	that file system tools become Go package tools. Copying, moving, or
	deleting a package corresponds to copying, moving, or deleting a
	directory.</p>

	<p>Fourth, each package is built using only the information present in
	the source files. This makes it much more likely that the tool will
	be able to adapt to changing build environments and conditions. For
	example, if we allowed extra configuration such as compiler flags or
	command line recipes, then that configuration would need to be updated
	each time the build tools changed; it would also be inherently tied
	to the use of a specific tool chain.</p>

	<h2>Getting started with the go command</h2>

	<p>Finally, a quick tour of how to use the go command, to supplement
	the information in <a href="/doc/code.html">How to Write Go Code</a>,
	which you might want to read first. Assuming you want
	to keep your source code separate from the Go distribution source
	tree, the first step is to set <code>$GOPATH</code>, the one piece of global
	configuration that the go command needs. The <code>$GOPATH</code> can be a
	list of directories, but by far the most common usage should be to set it to a
	single directory. In particular, you do not need a separate entry in
	<code>$GOPATH</code> for each of your projects. One <code>$GOPATH</code> can
	support many projects.</p>

	<p>Here’s an example. Let’s say we decide to keep our Go code in the directory
	<code>$HOME/mygo</code>. We need to create that directory and set
	<code>$GOPATH</code> accordingly.</p>

	<pre>
	$ mkdir $HOME/mygo
	$ export GOPATH=$HOME/mygo
	$
	</pre>

	<p>Into this directory, we now add some source code. Suppose we want to use
	the indexing library from the codesearch project along with a left-leaning
	red-black tree. We can install both with the "<code>go get</code>"
	subcommand:</p>

	<pre>
	$ go get code.google.com/p/codesearch/index
	$ go get github.com/petar/GoLLRB/llrb
	$
	</pre>

	<p>Both of these projects are now downloaded and installed into our
	<code>$GOPATH</code> directory. The one tree now contains the two directories
	<code>src/code.google.com/p/codesearch/index/</code> and
	<code>src/github.com/petar/GoLLRB/llrb/</code>, along with the compiled
	packages (in <code>pkg/</code>) for those libraries and their dependencies.</p>

	<p>Because we used version control systems (Mercurial and Git) to check
	out the sources, the source tree also contains the other files in the
	corresponding repositories, such as related packages. The "<code>go list</code>"
	subcommand lists the import paths corresponding to its arguments, and
	the pattern "<code>./...</code>" means start in the current directory
	("<code>./</code>") and find all packages below that directory
	("<code>...</code>"):</p>

	<pre>
	$ go list ./...
	code.google.com/p/codesearch/cmd/cgrep
	code.google.com/p/codesearch/cmd/cindex
	code.google.com/p/codesearch/cmd/csearch
	code.google.com/p/codesearch/index
	code.google.com/p/codesearch/regexp
	code.google.com/p/codesearch/sparse
	github.com/petar/GoLLRB/example
	github.com/petar/GoLLRB/llrb
	$
	</pre>

	<p>We can also test those packages:</p>

	<pre>
	$ go test ./...
	? code.google.com/p/codesearch/cmd/cgrep [no test files]
	? code.google.com/p/codesearch/cmd/cindex [no test files]
	? code.google.com/p/codesearch/cmd/csearch [no test files]
	ok code.google.com/p/codesearch/index 0.239s
	ok code.google.com/p/codesearch/regexp 0.021s
	? code.google.com/p/codesearch/sparse [no test files]
	? github.com/petar/GoLLRB/example [no test files]
	ok github.com/petar/GoLLRB/llrb 0.231s
	$
	</pre>

	<p>If a go subcommand is invoked with no paths listed, it operates on the
	current directory:</p>

	<pre>
	$ cd $GOPATH/src/code.google.com/p/codesearch/regexp
	$ go list
	code.google.com/p/codesearch/regexp
	$ go test -v
	=== RUN TestNstateEnc
	--- PASS: TestNstateEnc (0.00 seconds)
	=== RUN TestMatch
	--- PASS: TestMatch (0.01 seconds)
	=== RUN TestGrep
	--- PASS: TestGrep (0.00 seconds)
	PASS
	ok code.google.com/p/codesearch/regexp 0.021s
	$ go install
	$
	</pre>

	<p>That "<code>go install</code>" subcommand installs the latest copy of the
	package into the pkg directory. Because the go command can analyze the
	dependency graph, "<code>go install</code>" also installs any packages that
	this package imports but that are out of date, recursively.</p>

	<p>Notice that "<code>go install</code>" was able to determine the name of the
	import path for the package in the current directory, because of the convention
	for directory naming. It would be a little more convenient if we could pick
	the name of the directory where we kept source code, and we probably wouldn't
	pick such a long name, but that ability would require additional configuration
	and complexity in the tool. Typing an extra directory name or two is a small
	price to pay for the increased simplicity and power.</p>

	<p>As the example shows, it’s fine to work with packages from many different
	projects at once within a single <code>$GOPATH</code> root directory.</p>

	<h2>Limitations</h2>

	<p>As mentioned above, the go command is not a general-purpose build
	tool. In particular, it does not have any facility for generating Go
	source files during a build. Instead, if you want to use a tool like
	yacc or the protocol buffer compiler, you will need to write a
	makefile (or a configuration file for the build tool of your choice)
	to generate the Go files and then check those generated source files
	into your repository. This is more work for you, the package author,
	but it is significantly less work for your users, who can use
	"<code>go get</code>" without needing to obtain and build
	any additional tools.</p>

	<h2>More information</h2>

	<p>For more information, read <a href="/doc/code.html">How to Write Go Code</a>
	and see the <a href="/cmd/go/">go command documentation</a>.</p>