doc/codelab/wiki/wiki.html

<div class="content">

<h1>Writing Web Applications</h1>

<h2>Introduction</h2>

<p>
Covered in this codelab:
</p>
<ul>
<li>Creating a data structure with load and save methods</li>
<li>Using the <code>http</code> package to build web applications
<li>Using the <code>template</code> package to process HTML templates</li>
<li>Using the <code>regexp</code> package to validate user input</li>
<li>Using closures</li>
</ul>

<p>
Assumed knowledge:
</p>
<ul>
<li>Programming experience</li>
<li>Understanding of basic web technologies (HTTP, HTML)</li>
<li>Some UNIX command-line knowledge</li>
</ul>

<h2>Getting Started</h2>

<p>
At present, you need to have a Linux, OS X, or FreeBSD machine to run Go. If
you don't have access to one, you could set up a Linux Virtual Machine (using 
<a href="http://www.virtualbox.org/">VirtualBox</a> or similar) or a
<a href="http://www.google.com/search?q=virtual+private+server">Virtual 
Private Server</a>.
</p>

<p>
Install Go (see the <a href="http://golang.org/doc/install.html">Installation Instructions</a>).
</p>

<p>
Make a new directory for this codelab and cd to it:
</p>

<pre>
$ mkdir ~/gowiki
$ cd ~/gowiki
</pre>

<p>
Create a file named <code>wiki.go</code>, open it in your favorite editor, and 
add the following lines:
</p>

<pre>
package main

import (
	"fmt"
	"io/ioutil"
	"os"
)
</pre>

<p>
We import the <code>fmt</code>, <code>ioutil</code> and <code>os</code>
packages from the Go standard library. Later, as we implement additional
functionality, we will add more packages to this <code>import</code>
declaration.
</p>

<h2>Data Structures</h2>

<p>
Let's start by defining the data structures. A wiki consists of a series of
interconnected pages, each of which has a title and a body (the page content).
Here, we define <code>page</code> as a struct with two fields representing
the title and body.
</p>

<pre>
!./srcextract.bin -src=part1.go -name=page
</pre>

<p>
The type <code>[]byte</code> means "a <code>byte</code> slice". 
(See <a href="http://golang.org/doc/effective_go.html#slices">Effective Go</a> 
for more on slices.)  
The <code>body</code> element is a <code>[]byte</code> rather than
<code>string</code> because that is the type expected by the <code>io</code>
libraries we will use, as you'll see below.
</p>

<p>
The <code>page</code> struct describes how page data will be stored in memory. 
But what about persistent storage? We can address that by creating a 
<code>save</code> method on <code>page</code>:
</p>

<pre>
!./srcextract.bin -src=part1.go -name=save
</pre>

<p>
This method's signature reads: "This is a method named <code>save</code> that
takes as its receiver <code>p</code>, a pointer to <code>page</code> . It takes
no parameters, and returns a value of type <code>os.Error</code>." 
</p>

<p>
This method will save the <code>page</code>'s <code>body</code> to a text 
file. For simplicity, we will use the <code>title</code> as the file name.
</p>

<p>
The <code>save</code> method returns an <code>os.Error</code> value because
that is the return type of <code>WriteFile</code> (a standard library function
that writes a byte slice to a file).  The <code>save</code> method returns the
error value, to let the application handle it should anything go wrong while
writing the file.  If all goes well, <code>page.save()</code> will return
<code>nil</code> (the zero-value for pointers, interfaces, and some other 
types).
</p>

<p>
The octal integer constant <code>0600</code>, passed as the third parameter to
<code>WriteFile</code>, indicates that the file should be created with
read-write permissions for the current user only. (See the Unix man page
<code>open(2)</code> for details.)
</p>

<p>
We will want to load pages, too:
</p>

<pre>
!./srcextract.bin -src=part1-noerror.go -name=loadPage
</pre>

<p>
The function <code>loadPage</code> constructs the file name from
<code>title</code>, reads the file's contents into a new
<code>page</code>, and returns a pointer to that new <code>page</code>.
</p>

<p>
Functions can return multiple values. The standard library function 
<code>io.ReadFile</code> returns <code>[]byte</code> and <code>os.Error</code>. 
In <code>loadPage</code>, error isn't being handled yet; the "blank identifier"
represented by the underscore (<code>_</code>) symbol is used to throw away the
error return value (in essence, assigning the value to nothing). 
</p>

<p>
But what happens if <code>ReadFile</code> encounters an error?  For example,
the file might not exist. We should not ignore such errors.  Let's modify the
function to return <code>*page</code> and <code>os.Error</code>.
</p>

<pre>
!./srcextract.bin -src=part1.go -name=loadPage
</pre>

<p>
Callers of this function can now check the second parameter; if it is
<code>nil</code> then it has successfully loaded a page. If not, it will be an
<code>os.Error</code> that can be handled by the caller (see the <a
href="http://golang.org/pkg/os/#Error">os package documentation</a> for 
details).
</p>

<p>
At this point we have a simple data structure and the ability to save to and
load from a file. Let's write a <code>main</code> function to test what we've
written:
</p>

<pre>
!./srcextract.bin -src=part1.go -name=main
</pre>

<p>
After compiling and executing this code, a file named <code>TestPage.txt</code>
would be created, containing the contents of <code>p1</code>. The file would
then be read into the struct <code>p2</code>, and its <code>body</code> element
printed to the screen.
</p>

<p>
You can compile and run the program like this: 
</p>

<pre>
$ 8g wiki.go
$ 8l wiki.8
$ ./8.out
This is a sample page.
</pre>

<p>
(The <code>8g</code> and <code>8l</code> commands are applicable to
<code>GOARCH=386</code>. If you're on an <code>amd64</code> system,
substitute 6's for the 8's.)
</p>

<p>
<a href="part1.go">Click here to view the code we've written so far.</a>
</p>

<h2>Introducing the <code>http</code> package (an interlude)</h2>

<p>
Here's a full working example of a simple web server:
</p>

<pre>
!./htmlify.bin < http-sample.go
</pre>

<p>
The <code>main</code> function begins with a call to 
<code>http.HandleFunc</code>, which tells the <code>http</code> package to 
handle all requests to the web root (<code>"/"</code>) with 
<code>handler</code>. 
</p>

<p>
It then calls <code>http.ListenAndServe</code>, specifying that it should
listen on port 8080 on any interface (<code>":8080"</code>). (Don't
worry about its second parameter, <code>nil</code>, for now.)
This function will block until the program is terminated.
</p>

<p>
The function <code>handler</code> is of the type <code>http.HandlerFunc</code>.
It takes an <code>http.Conn</code> and <code>http.Request</code> as its 
arguments.
</p>

<p>
An <code>http.Conn</code> is the server end of an HTTP connection; by writing 
to it, we send data to the HTTP client.
</p>

<p>
An <code>http.Request</code> is a data structure that represents the client
HTTP request.  The string <code>r.URL.Path</code> is the path component
of the request URL.  The trailing <code>[1:]</code> means
"create a sub-slice of <code>Path</code> from the 1st character to the end." 
This drops the leading "/" from the path name.
</p>

<p>
If you run this program and access the URL: 
</p>
<pre>http://localhost:8080/monkeys</pre>
<p>
the program would present a page containing:
</p>
<pre>Hi there, I love monkeys!</pre>

<h2>Using <code>http</code> to serve wiki pages</h2>

<p>
To use the <code>http</code> package, it must be imported:
</p>

<pre>
import (
	"fmt"
	<b>"http"</b>
	"io/ioutil"
	"os"
)
</pre>

<p>
Let's create a handler to view a wiki page: 
</p>

<pre>
!./srcextract.bin -src=part2.go -name=lenPath

!./srcextract.bin -src=part2.go -name=viewHandler
</pre>

<p>
First, this function extracts the page title from <code>r.URL.Path</code>,
the path component of the request URL. The global constant 
<code>lenPath</code> is the length of the leading <code>"/view/"</code>
component of the request path.
The <code>Path</code> is re-sliced with <code>[lenPath:]</code> to drop the 
first 6 characters of the string. This is because the path will invariably 
begin with <code>"/view/"</code>, which is not part of the page title.
</p>

<p>
The function then loads the page data, formats the page with a string of simple 
HTML, and writes it to <code>c</code>, the <code>http.Conn</code>. 
</p>

<p>
Again, note the use of <code>_</code> to ignore the <code>os.Error</code> 
return value from <code>loadPage</code>. This is done here for simplicity
and generally considered bad practice. We will attend to this later.
</p>

<p>
To use this handler, we create a <code>main</code> function that
initializes <code>http</code> using the <code>viewHandler</code> to handle
any requests under the path <code>/view/</code>.
</p>

<pre>
!./srcextract.bin -src=part2.go -name=main
</pre>

<p>
<a href="part2.go">Click here to view the code we've written so far.</a>
</p>

<p>
Let's create some page data (as <code>test.txt</code>), compile our code, and
try serving a wiki page:
</p>

<pre>
$ echo "Hello world" &gt; test.txt
$ 8g wiki.go
$ 8l wiki.8
$ ./8.out
</pre>

<p>
With this web server running, a visit to <code><a
href="http://localhost:8080/view/test">http://localhost:8080/view/test</a></code>
should show a page titled "test" containing the words "Hello world".
</p>

<h2>Editing pages</h2>

<p>
A wiki is not a wiki without the ability to edit pages. Let's create two new
handlers: one named <code>editHandler</code> to display an 'edit page' form,
and the other named <code>saveHandler</code> to save the data entered via the
form.
</p>

<p>
First, we add them to <code>main()</code>: 
</p>

<pre>
!./srcextract.bin -src=final-noclosure.go -name=main
</pre>

<p>
The function <code>editHandler</code> loads the page 
(or, if it doesn't exist, create an empty <code>page</code> struct), 
and displays an HTML form.
</p>

<pre>
!./srcextract.bin -src=notemplate.go -name=editHandler
</pre>

<p>
This function will work fine, but all that hard-coded HTML is ugly.
Of course, there is a better way.
</p>
 
<h2>The <code>template</code> package</h2>

<p>
The <code>template</code> package is part of the Go standard library.  We can
use <code>template</code> to keep the HTML in a separate file, allowing
us to change the layout of our edit page without modifying the underlying Go
code.
</p>

<p>
First, we must add <code>template</code> to the list of imports:
</p>

<pre>
import (
	"http"
	"io/ioutil"
	"os"
	<b>"template"</b>
)
</pre>

<p>
Let's create a template file containing the HTML form. 
Open a new file named <code>edit.html</code>, and add the following lines:
</p>

<pre>
!./htmlify.bin < edit.html
</pre>

<p>
Modify <code>editHandler</code> to use the template, instead of the hard-coded
HTML:
</p>

<pre>
!./srcextract.bin -src=final-noerror.go -name=editHandler
</pre>

<p>
The function <code>template.ParseFile</code> will read the contents of 
<code>edit.html</code> and return a <code>*template.Template</code>. 
</p>

<p>
The method <code>t.Execute</code> replaces all occurrences of 
<code>{title}</code> and <code>{body}</code> with the values of 
<code>p.title</code> and <code>p.body</code>, and writes the resultant
HTML to the <code>http.Conn</code>.
</p>

<p>
Note that we've used <code>{body|html}</code> in the above template.  
The <code>|html</code> part asks the template engine to pass the value
<code>body</code> through the <code>html</code> formatter before outputting it,
which escapes HTML characters (such as replacing <code>&gt;</code> with 
<code>&amp;gt;</code>). 
This will prevent user data from corrupting the form HTML. 
</p>

<p>
Now that we've removed the <code>fmt.Sprintf</code> statement, we can remove
<code>"fmt"</code> from the <code>import</code> list.
</p>

<p>
While we're working with templates, let's create a template for our
<code>viewHandler</code> called <code>view.html</code>:
</p>

<pre>
!./htmlify.bin < view.html
</pre>

<p>
Modify <code>viewHandler</code> accordingly:
</p>

<pre>
!./srcextract.bin -src=final-noerror.go -name=viewHandler
</pre>

<p>
Notice that we've used almost exactly the same templating code in both
handlers. Let's remove this duplication by moving the templating code
to its own function:
</p>

<pre>
!./srcextract.bin -src=final-template.go -name=viewHandler

!./srcextract.bin -src=final-template.go -name=editHandler

!./srcextract.bin -src=final-template.go -name=renderTemplate
</pre>

<p>
The handlers are now shorter and simpler. 
</p>

<h2>Handling non-existent pages</h2>

<p>
What if you visit <code>/view/APageThatDoesntExist</code>? The program will 
crash. This is because it ignores the error return value from
<code>loadPage</code>. Instead, if the requested page doesn't exist, it should 
redirect the client to the edit page so the content may be created:
</p>

<pre>
!./srcextract.bin -src=final.go -name=viewHandler
</pre>

<p>
The <code>http.Redirect</code> function adds an HTTP status code of 
<code>http.StatusFound</code> (302) and a <code>Location</code>
header to the HTTP response.
</p>

<h2>Saving pages</h2>

<p>
The function <code>saveHandler</code> will handle the form submission. 
</p>

<pre>
!./srcextract.bin -src=final-template.go -name=saveHandler
</pre>

<p>
The page title (provided in the URL) and the form's only field, 
<code>body</code>, are stored in a new <code>page</code>. 
The <code>save()</code> method is then called to write the data to a file,
and the client is redirected to the <code>/view/</code> page.
</p>

<p>
The value returned by <code>FormValue</code> is of type <code>string</code>.
We must convert that value to <code>[]byte</code> before it will fit into 
the <code>page</code> struct.  We use <code>[]byte(body)</code> to perform
the conversion.
</p>

<h2>Error handling</h2>

<p>
There are several places in our program where errors are being ignored.  This
is bad practice, not least because when an error does occur the program will
crash.  A better solution is to handle the errors and return an error message
to the user. That way if something does go wrong, the server will continue to
function and the user will be notified.
</p>

<p>
First, let's handle the errors in <code>renderTemplate</code>:
</p>

<pre>
!./srcextract.bin -src=final-parsetemplate.go -name=renderTemplate
</pre>

<p>
The <code>http.Error</code> function sends a specified HTTP response code 
(in this case "Internal Server Error") and error message.
Already the decision to put this in a separate function is paying off.
</p>

<p>
Now let's fix up <code>saveHandler</code>:
</p>

<pre>
!./srcextract.bin -src=final.go -name=saveHandler
</pre>

<p>
Any errors that occur during <code>p.save()</code> will be reported 
to the user.
</p>

<h2>Template caching</h2>

<p>
There is an inefficiency in this code: <code>renderTemplate</code> calls 
<code>ParseFile</code> every time a page is rendered. 
A better approach would be to call <code>ParseFile</code> once for each 
template at program initialization, and store the resultant 
<code>*Template</code> values in a data structure for later use.
</p>

<p>
First we create a global map named <code>templates</code> in which to store 
our <code>*Template</code> values, keyed by <code>string</code> 
(the template name):
</p>

<pre>
!./srcextract.bin -src=final.go -name=templates
</pre>

<p>
Then we create an <code>init</code> function, which will be called before
<code>main</code> at program initialization. The function
<code>template.MustParseFile</code> is a convenience wrapper around
<code>ParseFile</code> that does not return an error code; instead, it panics
if an error is encountered. A panic is appropriate here; if the templates can't
be loaded the only sensible thing to do is exit the program.
</p

<pre>
!./srcextract.bin -src=final.go -name=init
</pre>

<p>
A <code>for</code> loop is used with a <code>range</code> statement to iterate 
over an array constant containing the names of the templates we want parsed.
If we were to add more templates to our program, we would add their names to 
that array.
</p>

<p>
We then modify our <code>renderTemplate</code> function to call 
the <code>Execute</code> method on the appropriate <code>Template</code> from 
<code>templates</code>:

<pre>
!./srcextract.bin -src=final.go -name=renderTemplate
</pre>

<h2>Validation</h2>

<p>
As you may have observed, this program has a serious security flaw: a user
can supply an arbitrary path to be read/written on the server. To mitigate
this, we can write a function to validate the title with a regular expression.
</p>

<p>
First, add <code>"regexp"</code> to the <code>import</code> list.
Then we can create a global variable to store our validation regexp:
</p>

<pre>
!./srcextract.bin -src=final-noclosure.go -name=titleValidator
</pre>

<p>
The function <code>regexp.MustCompile</code> will parse and compile the 
regular expression, and return a <code>regexp.Regexp</code>. 
<code>MustCompile</code>, like <code>template.MustParseFile</code>,
is distinct from <code>Compile</code> in that it will panic if 
the expression compilation fails, while <code>Compile</code> returns an 
<code>os.Error</code> as a second parameter. 
</p>

<p>
Now, let's write a function that extracts the title string from the request 
URL, and tests it against our <code>titleValidator</code> expression:
</p>

<pre>
!./srcextract.bin -src=final-noclosure.go -name=getTitle
</pre>

<p>
If the title is valid, it will be returned along with a <code>nil</code>
error value.  If the title is invalid, the function will write a 
"404 Not Found" error to the HTTP connection, and return an error to the 
handler. 
</p>

<p>
Let's put a call to <code>getTitle</code> in each of the handlers:
</p>

<pre>
!./srcextract.bin -src=final-noclosure.go -name=viewHandler

!./srcextract.bin -src=final-noclosure.go -name=editHandler

!./srcextract.bin -src=final-noclosure.go -name=saveHandler
</pre>

<h2>Introducing Function Literals and Closures</h2>

<p>
Catching the error condition in each handler introduces a lot of repeated code.
What if we could wrap each of the handlers in a function that does this 
validation and error checking? Go's 
<a href="http://golang.org/doc/go_spec.html#Function_declarations">function 
literals</a> provide a powerful means of abstracting functionality 
that can help us here.
</p>

<p>
First, we re-write the function definition of each of the handlers to accept
a title string:
</p>

<pre>
func viewHandler(c *http.Conn, r *http.Request, title string)
func editHandler(c *http.Conn, r *http.Request, title string)
func saveHandler(c *http.Conn, r *http.Request, title string)
</pre>

<p>
Now let's define a wrapper function that <i>takes a function of the above
type</i>, and returns a function of type <code>http.HandlerFunc</code>
(suitable to be passed to the function <code>http.HandleFunc</code>):
</p>

<pre>
func makeHandler(fn func (*http.Conn, *http.Request, string)) http.HandlerFunc {
	return func(c *http.Conn, r *http.Request) {
		// Here we will extract the page title from the Request,
		// and call the provided handler 'fn'
	}
}
</pre>

<p>
The returned function is called a closure because it encloses values defined
outside of it. In this case, the variable <code>fn</code> (the single argument
to <code>makeHandler</code>) is enclosed by the closure. The variable
<code>fn</code> will be one of our save, edit, or view handlers.
</p>

<p>
Now we can take the code from <code>getTitle</code> and use it here
(with some minor modifications):
</p>

<pre>
!./srcextract.bin -src=final.go -name=makeHandler
</pre>

<p>
The closure returned by <code>makeHandler</code> is a function that takes
an <code>http.Conn</code> and <code>http.Request</code> (in other words,
an <code>http.HandlerFunc</code>). 
The closure extracts the <code>title</code> from the request path, and
validates it with the <code>titleValidator</code> regexp. If the
<code>title</code> is invalid, an error will be written to the
<code>Conn</code> using the <code>http.NotFound</code> function. 
If the <code>title</code> is valid, the enclosed handler function
<code>fn</code> will be called with the <code>Conn</code>,
<code>Request</code>, and <code>title</code> as arguments.
</p>

<p>
Now we can wrap the handler functions with <code>makeHandler</code> in 
<code>main</code>, before they are registered with the <code>http</code> 
package:
</p>

<pre>
!./srcextract.bin -src=final.go -name=main
</pre>

<p>
Finally we remove the calls to <code>getTitle</code> from the handler functions,
making them much simpler:
</p>

<pre>
!./srcextract.bin -src=final.go -name=viewHandler

!./srcextract.bin -src=final.go -name=editHandler

!./srcextract.bin -src=final.go -name=saveHandler
</pre>

<h2>Try it out!</h2>

<p>
<a href="final.go">Click here to view the final code listing.</a>
</p>

<p>
Recompile the code, and run the app:
</p>

<pre>
$ 8g wiki.go
$ 8l wiki.8
$ ./8.out
</pre>

<p>
Visiting <a href="http://localhost:8080/view/ANewPage">http://localhost:8080/view/ANewPage</a>
should present you with the page edit form. You should then be able to 
enter some text, click 'Save', and be redirected to the newly created page.
</p>

<h2>Other tasks</h2>

<p>
Here are some simple tasks you might want to tackle on your own:
</p>

<ul>
<li>Store templates in <code>tmpl/</code> and page data in <code>data/</code>.
<li>Add a handler to make the web root redirect to 
	<code>/view/FrontPage</code>.</li>
<li>Spruce up the page templates by making them valid HTML and adding some
	CSS rules.</li>
<li>Implement inter-page linking by converting instances of 
	<code>[PageName]</code> to <br>
	<code>&lt;a href="/view/PageName"&gt;PageName&lt;/a&gt;</code>.
	(hint: you could use <code>regexp.ReplaceAllFunc</code> to do this)
	</li>
</ul>

</div>