The App Engine SDK and workspaces (GOPATH)
9 Jan 2013
Tags: appengine, tools, gopath

Andrew Gerrand

* Introduction

When we released Go 1 we introduced the [[http://golang.org/cmd/go/][go tool]] and, with it, the concept of workspaces. Workspaces (specified by the GOPATH environment variable) are a convention for organizing code that simplifies fetching, building, and installing Go packages. If you're not familiar with workspaces, please read [[http://golang.org/doc/code.html][this article]] or watch [[http://www.youtube.com/watch?v=XCsL89YtqCs][this screencast]] before reading on.

Until recently, the tools in the App Engine SDK were not aware of workspaces. Without workspaces the "[[http://golang.org/cmd/go/#hdr-Download_and_install_packages_and_dependencies][go get]]" command cannot function, and so app authors had to install and update their app dependencies manually. It was a pain.

This has all changed with version 1.7.4 of the App Engine SDK. The [[https://developers.google.com/appengine/docs/go/tools/devserver][dev_appserver]] and [[https://developers.google.com/appengine/docs/go/tools/uploadinganapp][appcfg]] tools are now workspace-aware. When running locally or uploading an app, these tools now search for dependencies in the workspaces specified by the GOPATH environment variable. This means you can now use "go get" while building App Engine apps, and switch between normal Go programs and App Engine apps without changing your environment or habits.

For example, let's say you want to build an app that uses OAuth 2.0 to authenticate with a remote service. A popular OAuth 2.0 library for Go is the [[http://code.google.com/p/goauth2][goauth2]] package, which you can install to your workspace with this command:

	     go get code.google.com/p/goauth2/oauth

When writing your App Engine app, import the oauth package just as you would in a regular Go program:

	     import "code.google.com/p/goauth2/oauth"

Now, whether running your app with the dev_appserver or deploying it with appcfg, the tools will find the oauth package in your workspace. It just works.

* Hybrid stand-alone/App Engine apps

The Go App Engine SDK builds on Go's standard [[http://golang.org/pkg/net/http/][net/http]] package to serve web requests and, as a result, many Go web servers can be run on App Engine with only a few changes. For example, [[http://golang.org/cmd/godoc/][godoc]] is included in the Go distribution as a stand-alone program, but it can also run as an App Engine app (godoc serves [[http://golang.org/][golang.org]] from App Engine).

But wouldn't it be nice if you could write a program that is both a stand-alone web server and an App Engine app? By using [[http://golang.org/pkg/go/build/#hdr-Build_Constraints][build constraints]], you can.

Build constraints are line comments that determine whether a file should be included in a package. They are most often used in code that handles a variety of operating systems or processor architectures. For instance, the [[http://golang.org/pkg/path/filepath/][path/filepath]] package includes the file [[http://golang.org/src/pkg/path/filepath/symlink.go][symlink.go]], which specifies a build constraint to ensure that it is not built on Windows systems (which do not have symbolic links):

	    // +build !windows

The App Engine SDK introduces a new build constraint term: "appengine". Files that specify

	    // +build appengine

will be built by the App Engine SDK and ignored by the go tool. Conversely, files that specify

	    // +build !appengine

are ignored by the App Engine SDK, while the go tool will happily build them.

The [[http://code.google.com/p/goprotobuf/][goprotobuf]] library uses this mechanism to provide two implementations of a key part of its encode/decode machinery: [[http://code.google.com/p/goprotobuf/source/browse/proto/pointer_unsafe.go][pointer_unsafe.go]] is the faster version that cannot be used on App Engine because it uses the [[http://golang.org/pkg/unsafe/][unsafe package]], while [[http://code.google.com/p/goprotobuf/source/browse/proto/pointer_reflect.go][pointer_reflect.go]] is a slower version that avoids unsafe by using the [[http://golang.org/pkg/reflect/][reflect package]] instead.

Let's take a simple Go web server and turn it into a hybrid app. This is main.go:

	    package main
	    
	    import (
	        "fmt"
	        "net/http"
	    )
	    
	    func main() {
	        http.HandleFunc("/", handler)
	        http.ListenAndServe("localhost:8080", nil)
	    }
	    
	    func handler(w http.ResponseWriter, r *http.Request) {
	        fmt.Fprint(w, "Hello!")
	    }

Build this with the go tool and you'll get a stand-alone web server executable.

The App Engine infrastructure provides its own main function that runs its equivalent to ListenAndServe. To convert main.go to an App Engine app, drop the call to ListenAndServe and register the handler in an init function (which runs before main). This is app.go:

	    package main
	    
	    import (
	        "fmt"
	        "net/http"
	    )
	    
	    func init() {
	        http.HandleFunc("/", handler)
	    }
	    
	    func handler(w http.ResponseWriter, r *http.Request) {
	        fmt.Fprint(w, "Hello!")
	    }

To make this a hybrid app, we need to split it into an App Engine-specific part, an stand-alone binary-specific part, and the parts common to both versions. In this case, there is no App Engine-specific part, so we split it into just two files:

app.go specifies and registers the handler function. It is identical to the code listing above, and requires no build constraints as it should be included in all versions of the program.

main.go runs the web server. It includes the "!appengine" build constraint, as it must only included when building the stand-alone binary.

	    // +build !appengine
	    
	    package main
	    
	    import "net/http"
	    
	    func main() {
	        http.ListenAndServe("localhost:8080", nil)        
	    }

To see a more complex hybrid app, take a look at the [[http://code.google.com/p/go/source/browse/?repo=talks#hg%2Fpresent][present tool]].

* Conclusions

We hope these changes will make it easier to work on apps with external dependencies, and to maintain code bases that contain both stand-alone programs and App Engine apps.