doc/articles/gobs_of_data.html

<!--{
"Title": "Gobs of data",
"Template": true
}-->

<p>
To transmit a data structure across a network or to store it in a file, it must
be encoded and then decoded again. There are many encodings available, of
course: <a href="http://www.json.org/">JSON</a>,
<a href="http://www.w3.org/XML/">XML</a>, Google's
<a href="http://code.google.com/p/protobuf">protocol buffers</a>, and more.
And now there's another, provided by Go's <a href="/pkg/encoding/gob/">gob</a>
package.
</p>

<p>
Why define a new encoding? It's a lot of work and redundant at that. Why not
just use one of the existing formats? Well, for one thing, we do! Go has
<a href="/pkg/">packages</a> supporting all the encodings just mentioned (the
<a href="http://code.google.com/p/goprotobuf">protocol buffer package</a> is in
a separate repository but it's one of the most frequently downloaded). And for
many purposes, including communicating with tools and systems written in other
languages, they're the right choice.
</p>

<p>
But for a Go-specific environment, such as communicating between two servers
written in Go, there's an opportunity to build something much easier to use and
possibly more efficient.
</p>

<p>
Gobs work with the language in a way that an externally-defined,
language-independent encoding cannot. At the same time, there are lessons to be
learned from the existing systems.
</p>

<p>
<b>Goals</b>
</p>

<p>
The gob package was designed with a number of goals in mind.
</p>

<p>
First, and most obvious, it had to be very easy to use. First, because Go has
reflection, there is no need for a separate interface definition language or
"protocol compiler". The data structure itself is all the package should need
to figure out how to encode and decode it. On the other hand, this approach
means that gobs will never work as well with other languages, but that's OK:
gobs are unashamedly Go-centric.
</p>

<p>
Efficiency is also important. Textual representations, exemplified by XML and
JSON, are too slow to put at the center of an efficient communications network.
A binary encoding is necessary.
</p>

<p>
Gob streams must be self-describing. Each gob stream, read from the beginning,
contains sufficient information that the entire stream can be parsed by an
agent that knows nothing a priori about its contents. This property means that
you will always be able to decode a gob stream stored in a file, even long
after you've forgotten what data it represents.
</p>

<p>
There were also some things to learn from our experiences with Google protocol
buffers.
</p>

<p>
<b>Protocol buffer misfeatures</b>
</p>

<p>
Protocol buffers had a major effect on the design of gobs, but have three
features that were deliberately avoided. (Leaving aside the property that
protocol buffers aren't self-describing: if you don't know the data definition
used to encode a protocol buffer, you might not be able to parse it.)
</p>

<p>
First, protocol buffers only work on the data type we call a struct in Go. You
can't encode an integer or array at the top level, only a struct with fields
inside it. That seems a pointless restriction, at least in Go. If all you want
to send is an array of integers, why should you have to put it into a
struct first?
</p>

<p>
Next, a protocol buffer definition may specify that fields <code>T.x</code> and
<code>T.y</code> are required to be present whenever a value of type
<code>T</code> is encoded or decoded.  Although such required fields may seem
like a good idea, they are costly to implement because the codec must maintain a
separate data structure while encoding and decoding, to be able to report when
required fields are missing.  They're also a maintenance problem. Over time, one
may want to modify the data definition to remove a required field, but that may
cause existing clients of the data to crash. It's better not to have them in the
encoding at all.  (Protocol buffers also have optional fields. But if we don't
have required fields, all fields are optional and that's that. There will be
more to say about optional fields a little later.)
</p>

<p>
The third protocol buffer misfeature is default values. If a protocol buffer
omits the value for a "defaulted" field, then the decoded structure behaves as
if the field were set to that value. This idea works nicely when you have
getter and setter methods to control access to the field, but is harder to
handle cleanly when the container is just a plain idiomatic struct. Required
fields are also tricky to implement: where does one define the default values,
what types do they have (is text UTF-8? uninterpreted bytes? how many bits in a
float?) and despite the apparent simplicity, there were a number of
complications in their design and implementation for protocol buffers. We
decided to leave them out of gobs and fall back to Go's trivial but effective
defaulting rule: unless you set something otherwise, it has the "zero value"
for that type - and it doesn't need to be transmitted.
</p>

<p>
So gobs end up looking like a sort of generalized, simplified protocol buffer.
How do they work?
</p>

<p>
<b>Values</b>
</p>

<p>
The encoded gob data isn't about <code>int8</code>s and <code>uint16</code>s.
Instead, somewhat analogous to constants in Go, its integer values are abstract,
sizeless numbers, either signed or unsigned. When you encode an
<code>int8</code>, its value is transmitted as an unsized, variable-length
integer. When you encode an <code>int64</code>, its value is also transmitted as
an unsized, variable-length integer. (Signed and unsigned are treated
distinctly, but the same unsized-ness applies to unsigned values too.) If both
have the value 7, the bits sent on the wire will be identical. When the receiver
decodes that value, it puts it into the receiver's variable, which may be of
arbitrary integer type. Thus an encoder may send a 7 that came from an
<code>int8</code>, but the receiver may store it in an <code>int64</code>. This
is fine: the value is an integer and as a long as it fits, everything works. (If
it doesn't fit, an error results.) This decoupling from the size of the variable
gives some flexibility to the encoding: we can expand the type of the integer
variable as the software evolves, but still be able to decode old data.
</p>

<p>
This flexibility also applies to pointers. Before transmission, all pointers are
flattened. Values of type <code>int8</code>, <code>*int8</code>,
<code>**int8</code>, <code>****int8</code>, etc. are all transmitted as an
integer value, which may then be stored in <code>int</code> of any size, or
<code>*int</code>, or <code>******int</code>, etc. Again, this allows for
flexibility.
</p>

<p>
Flexibility also happens because, when decoding a struct, only those fields
that are sent by the encoder are stored in the destination. Given the value
</p>

{{code "/doc/progs/gobs1.go" `/type T/` `/STOP/`}}

<p>
the encoding of <code>t</code> sends only the 7 and 8. Because it's zero, the
value of <code>Y</code> isn't even sent; there's no need to send a zero value.
</p>

<p>
The receiver could instead decode the value into this structure:
</p>

{{code "/doc/progs/gobs1.go" `/type U/` `/STOP/`}}

<p>
and acquire a value of <code>u</code> with only <code>X</code> set (to the
address of an <code>int8</code> variable set to 7); the <code>Z</code> field is
ignored - where would you put it? When decoding structs, fields are matched by
name and compatible type, and only fields that exist in both are affected. This
simple approach finesses the "optional field" problem: as the type
<code>T</code> evolves by adding fields, out of date receivers will still
function with the part of the type they recognize. Thus gobs provide the
important result of optional fields - extensibility - without any additional
mechanism or notation.
</p>

<p>
From integers we can build all the other types: bytes, strings, arrays, slices,
maps, even floats. Floating-point values are represented by their IEEE 754
floating-point bit pattern, stored as an integer, which works fine as long as
you know their type, which we always do. By the way, that integer is sent in
byte-reversed order because common values of floating-point numbers, such as
small integers, have a lot of zeros at the low end that we can avoid
transmitting.
</p>

<p>
One nice feature of gobs that Go makes possible is that they allow you to define
your own encoding by having your type satisfy the
<a href="/pkg/encoding/gob/#GobEncoder">GobEncoder</a> and
<a href="/pkg/encoding/gob/#GobDecoder">GobDecoder</a> interfaces, in a manner
analogous to the <a href="/pkg/encoding/json/">JSON</a> package's
<a href="/pkg/encoding/json/#Marshaler">Marshaler</a> and
<a href="/pkg/encoding/json/#Unmarshaler">Unmarshaler</a> and also to the
<a href="/pkg/fmt/#Stringer">Stringer</a> interface from
<a href="/pkg/fmt/">package fmt</a>. This facility makes it possible to
represent special features, enforce constraints, or hide secrets when you
transmit data. See the <a href="/pkg/encoding/gob/">documentation</a> for
details.
</p>

<p>
<b>Types on the wire</b>
</p>

<p>
The first time you send a given type, the gob package includes in the data
stream a description of that type. In fact, what happens is that the encoder is
used to encode, in the standard gob encoding format, an internal struct that
describes the type and gives it a unique number. (Basic types, plus the layout
of the type description structure, are predefined by the software for
bootstrapping.) After the type is described, it can be referenced by its type
number.
</p>

<p>
Thus when we send our first type <code>T</code>, the gob encoder sends a
description of <code>T</code> and tags it with a type number, say 127. All
values, including the first, are then prefixed by that number, so a stream of
<code>T</code> values looks like:
</p>

<pre>
("define type id" 127, definition of type T)(127, T value)(127, T value), ...
</pre>

<p>
These type numbers make it possible to describe recursive types and send values
of those types. Thus gobs can encode types such as trees:
</p>

{{code "/doc/progs/gobs1.go" `/type Node/` `/STOP/`}}

<p>
(It's an exercise for the reader to discover how the zero-defaulting rule makes
this work, even though gobs don't represent pointers.)
</p>

<p>
With the type information, a gob stream is fully self-describing except for the
set of bootstrap types, which is a well-defined starting point.
</p>

<p>
<b>Compiling a machine</b>
</p>

<p>
The first time you encode a value of a given type, the gob package builds a
little interpreted machine specific to that data type. It uses reflection on
the type to construct that machine, but once the machine is built it does not
depend on reflection. The machine uses package unsafe and some trickery to
convert the data into the encoded bytes at high speed. It could use reflection
and avoid unsafe, but would be significantly slower. (A similar high-speed
approach is taken by the protocol buffer support for Go, whose design was
influenced by the implementation of gobs.) Subsequent values of the same type
use the already-compiled machine, so they can be encoded right away.
</p>

<p>
Decoding is similar but harder. When you decode a value, the gob package holds
a byte slice representing a value of a given encoder-defined type to decode,
plus a Go value into which to decode it. The gob package builds a machine for
that pair: the gob type sent on the wire crossed with the Go type provided for
decoding. Once that decoding machine is built, though, it's again a
reflectionless engine that uses unsafe methods to get maximum speed.
</p>

<p>
<b>Use</b>
</p>

<p>
There's a lot going on under the hood, but the result is an efficient,
easy-to-use encoding system for transmitting data. Here's a complete example
showing differing encoded and decoded types. Note how easy it is to send and
receive values; all you need to do is present values and variables to the
<a href="/pkg/encoding/gob/">gob package</a> and it does all the work.
</p>

{{code "/doc/progs/gobs2.go" `/package main/` `$`}}

<p>
You can compile and run this example code in the
<a href="http://play.golang.org/p/_-OJV-rwMq">Go Playground</a>.
</p>

<p>
The <a href="/pkg/net/rpc/">rpc package</a> builds on gobs to turn this
encode/decode automation into transport for method calls across the network.
That's a subject for another article.
</p>

<p>
<b>Details</b>
</p>

<p>
The <a href="/pkg/encoding/gob/">gob package documentation</a>, especially the
file <a href="/src/pkg/encoding/gob/doc.go">doc.go</a>, expands on many of the
details described here and includes a full worked example showing how the
encoding represents data. If you are interested in the innards of the gob
implementation, that's a good place to start.
</p>