blob: 95ab3770055afffbb598270ba97ea9f324536ec0 [file] [log] [blame]
JSON and Go
25 Jan 2011
Tags: json, technical
Andrew Gerrand
* Introduction
JSON (JavaScript Object Notation) is a simple data interchange format. Syntactically it resembles the objects and lists of JavaScript. It is most commonly used for communication between web back-ends and JavaScript programs running in the browser, but it is used in many other places, too. Its home page, [[http://json.org][json.org]], provides a wonderfully clear and concise definition of the standard.
With the [[https://golang.org/pkg/encoding/json/][json package]] it's a snap to read and write JSON data from your Go programs.
* Encoding
To encode JSON data we use the [[https://golang.org/pkg/encoding/json/#Marshal][`Marshal`]] function.
func Marshal(v interface{}) ([]byte, error)
Given the Go data structure, `Message`,
type Message struct {
Name string
Body string
Time int64
}
and an instance of `Message`
m := Message{"Alice", "Hello", 1294706395881547000}
we can marshal a JSON-encoded version of m using `json.Marshal`:
b, err := json.Marshal(m)
If all is well, `err` will be `nil` and `b` will be a `[]byte` containing this JSON data:
b == []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`)
Only data structures that can be represented as valid JSON will be encoded:
- JSON objects only support strings as keys; to encode a Go map type it must be of the form `map[string]T` (where `T` is any Go type supported by the json package).
- Channel, complex, and function types cannot be encoded.
- Cyclic data structures are not supported; they will cause `Marshal` to go into an infinite loop.
- Pointers will be encoded as the values they point to (or 'null' if the pointer is `nil`).
The json package only accesses the exported fields of struct types (those that begin with an uppercase letter). Therefore only the the exported fields of a struct will be present in the JSON output.
* Decoding
To decode JSON data we use the [[https://golang.org/pkg/encoding/json/#Unmarshal][`Unmarshal`]] function.
func Unmarshal(data []byte, v interface{}) error
We must first create a place where the decoded data will be stored
var m Message
and call `json.Unmarshal`, passing it a `[]byte` of JSON data and a pointer to `m`
err := json.Unmarshal(b, &m)
If `b` contains valid JSON that fits in `m`, after the call `err` will be `nil` and the data from `b` will have been stored in the struct `m`, as if by an assignment like:
m = Message{
Name: "Alice",
Body: "Hello",
Time: 1294706395881547000,
}
How does `Unmarshal` identify the fields in which to store the decoded data? For a given JSON key `"Foo"`, `Unmarshal` will look through the destination struct's fields to find (in order of preference):
- An exported field with a tag of `"Foo"` (see the [[https://golang.org/ref/spec#Struct_types][Go spec]] for more on struct tags),
- An exported field named `"Foo"`, or
- An exported field named `"FOO"` or `"FoO"` or some other case-insensitive match of `"Foo"`.
What happens when the structure of the JSON data doesn't exactly match the Go type?
b := []byte(`{"Name":"Bob","Food":"Pickle"}`)
var m Message
err := json.Unmarshal(b, &m)
`Unmarshal` will decode only the fields that it can find in the destination type. In this case, only the Name field of m will be populated, and the Food field will be ignored. This behavior is particularly useful when you wish to pick only a few specific fields out of a large JSON blob. It also means that any unexported fields in the destination struct will be unaffected by `Unmarshal`.
But what if you don't know the structure of your JSON data beforehand?
* Generic JSON with interface{}
The `interface{}` (empty interface) type describes an interface with zero methods. Every Go type implements at least zero methods and therefore satisfies the empty interface.
The empty interface serves as a general container type:
var i interface{}
i = "a string"
i = 2011
i = 2.777
A type assertion accesses the underlying concrete type:
r := i.(float64)
fmt.Println("the circle's area", math.Pi*r*r)
Or, if the underlying type is unknown, a type switch determines the type:
switch v := i.(type) {
case int:
fmt.Println("twice i is", v*2)
case float64:
fmt.Println("the reciprocal of i is", 1/v)
case string:
h := len(v) / 2
fmt.Println("i swapped by halves is", v[h:]+v[:h])
default:
// i isn't one of the types above
}
The json package uses `map[string]interface{}` and
`[]interface{}` values to store arbitrary JSON objects and arrays;
it will happily unmarshal any valid JSON blob into a plain
`interface{}` value. The default concrete Go types are:
- `bool` for JSON booleans,
- `float64` for JSON numbers,
- `string` for JSON strings, and
- `nil` for JSON null.
* Decoding arbitrary data
Consider this JSON data, stored in the variable `b`:
b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"]}`)
Without knowing this data's structure, we can decode it into an `interface{}` value with `Unmarshal`:
var f interface{}
err := json.Unmarshal(b, &f)
At this point the Go value in `f` would be a map whose keys are strings and whose values are themselves stored as empty interface values:
f = map[string]interface{}{
"Name": "Wednesday",
"Age": 6,
"Parents": []interface{}{
"Gomez",
"Morticia",
},
}
To access this data we can use a type assertion to access `f`'s underlying `map[string]interface{}`:
m := f.(map[string]interface{})
We can then iterate through the map with a range statement and use a type switch to access its values as their concrete types:
for k, v := range m {
switch vv := v.(type) {
case string:
fmt.Println(k, "is string", vv)
case float64:
fmt.Println(k, "is float64", vv)
case []interface{}:
fmt.Println(k, "is an array:")
for i, u := range vv {
fmt.Println(i, u)
}
default:
fmt.Println(k, "is of a type I don't know how to handle")
}
}
In this way you can work with unknown JSON data while still enjoying the benefits of type safety.
* Reference Types
Let's define a Go type to contain the data from the previous example:
type FamilyMember struct {
Name string
Age int
Parents []string
}
var m FamilyMember
err := json.Unmarshal(b, &m)
Unmarshaling that data into a `FamilyMember` value works as expected, but if we look closely we can see a remarkable thing has happened. With the var statement we allocated a `FamilyMember` struct, and then provided a pointer to that value to `Unmarshal`, but at that time the `Parents` field was a `nil` slice value. To populate the `Parents` field, `Unmarshal` allocated a new slice behind the scenes. This is typical of how `Unmarshal` works with the supported reference types (pointers, slices, and maps).
Consider unmarshaling into this data structure:
type Foo struct {
Bar *Bar
}
If there were a `Bar` field in the JSON object, `Unmarshal` would allocate a new `Bar` and populate it. If not, `Bar` would be left as a `nil` pointer.
From this a useful pattern arises: if you have an application that receives a few distinct message types, you might define "receiver" structure like
type IncomingMessage struct {
Cmd *Command
Msg *Message
}
and the sending party can populate the `Cmd` field and/or the `Msg` field of the top-level JSON object, depending on the type of message they want to communicate. `Unmarshal`, when decoding the JSON into an `IncomingMessage` struct, will only allocate the data structures present in the JSON data. To know which messages to process, the programmer need simply test that either `Cmd` or `Msg` is not `nil`.
* Streaming Encoders and Decoders
The json package provides `Decoder` and `Encoder` types to support the common operation of reading and writing streams of JSON data. The `NewDecoder` and `NewEncoder` functions wrap the [[https://golang.org/pkg/io/#Reader][`io.Reader`]] and [[https://golang.org/pkg/io/#Writer][`io.Writer`]] interface types.
func NewDecoder(r io.Reader) *Decoder
func NewEncoder(w io.Writer) *Encoder
Here's an example program that reads a series of JSON objects from standard input, removes all but the `Name` field from each object, and then writes the objects to standard output:
package main
import (
"encoding/json"
"log"
"os"
)
func main() {
dec := json.NewDecoder(os.Stdin)
enc := json.NewEncoder(os.Stdout)
for {
var v map[string]interface{}
if err := dec.Decode(&v); err != nil {
log.Println(err)
return
}
for k := range v {
if k != "Name" {
delete(v, k)
}
}
if err := enc.Encode(&v); err != nil {
log.Println(err)
}
}
}
Due to the ubiquity of Readers and Writers, these `Encoder` and `Decoder` types can be used in a broad range of scenarios, such as reading and writing to HTTP connections, WebSockets, or files.
* References
For more information see the [[https://golang.org/pkg/encoding/json/][json package documentation]]. For an example usage of json see the source files of the [[https://golang.org/pkg/net/rpc/jsonrpc/][jsonrpc package]].