| Playground v3 |
| 2019-10-08 |
| |
| Brad Fitzpatrick |
| @bradfitz |
| bradfitz@golang.org |
| |
| * Overview |
| |
| - This is talk about play.golang.org |
| - What it does |
| - History of implementations |
| - Latest incarnation |
| |
| * What does it does |
| |
| - Lets users run untrusted code in browser |
| - Used by play.golang.org, tour.golang.org, godoc example snippets, etc. |
| |
| .image hello.png |
| |
| * Time |
| |
| - time is faked, cacheable |
| - starts in Nov 2009, when Go was announced publicly |
| |
| .image time.png |
| |
| * Sleep replays |
| |
| - sleeps are emulated server-side (special runtime support) |
| - stdout/stderr writes prefixed with binary header containing virtual time |
| - sandbox parses output, simulates passage of time in browser |
| - doesn't tie up server resources |
| |
| * Fun: clear screen with \f |
| |
| .link https://play.golang.org/p/lrSX-aXEOMe animation example |
| |
| func main() { |
| const x = `.oOo` |
| for i := 0; i < 100; i++ { |
| fmt.Printf("\f%c", x[i%len(x)]) |
| time.Sleep(200 * time.Millisecond) |
| } |
| } |
| |
| * Fun: images |
| |
| .link https://play.golang.org/p/t9ECgoq0XSD image example |
| |
| .image pic.png _ 600 |
| |
| * History |
| |
| Caveat: fake, retroactive version numbers follow for ease of discussion. |
| |
| In reality, v0.0.0-20141209125517-dba9ba1953c6 |
| |
| * Native Client (NaCl) |
| |
| - July 2009, four months before Go |
| - GOOS=nacl, GOARCH={386,amd64p32,arm} |
| - restricted, verifiable subset of {386,amd64,arm} |
| - some had ambitions to add it to Chrome, web |
| - most people opposed |
| - "Portable NativeClient" (PNaCl) later proposed, opposed |
| - GOARCH=amd64p32: amd64, but 32-bit pointers |
| |
| * Playground ~v1 |
| |
| - App Engine frontend, in Python, made RPC calls to pass source to: |
| - C++ Borg backend, compiled Go to NaCl, ran under ptrace syscall filter |
| - standard two layers of paranoia; either sufficient in theory alone, ptrace layer logs/alerts hard on any violation |
| |
| * Playground ~v1.1 |
| |
| - GAE frontend rewritten from Python to Go |
| - backend still C++ running NaCl on Borg |
| |
| * Playground ~v2 |
| |
| - move to App Engine Flex, where we can do more stuff |
| - unify frontend + backend |
| - run Native Client sel_ldr (NaCl runner binary) ourselves |
| - no more C++ |
| - no more Borg |
| |
| * 2017 |
| |
| - May 30, 2017: Portable Native Client (PNaCl) deprecated |
| - Go never even implemented PNaCl (fortunately, in retrospect) |
| - Go still implements the doubly deprecated arch-specific NaCl variants |
| |
| * 2018, 2019... |
| |
| - WebAssembly rise continues |
| - NaCl looking increasingly odd, high maintenance cost relative to its worth |
| - maintenance: builder resources |
| - maintenance: hoping runtime faketime patches still apply at every release |
| - maintenance: amd64p32 |
| - but play.golang.org still depends on it |
| |
| * How to sandbox play.golang.org without NaCl? |
| |
| * Plan: Playground v3 |
| |
| - delete nacl/* |
| - move Nacl's funky faketime support to linux/amd64 at least... |
| - +build faketime (in master for Go 1.14; thanks, Austin!) |
| - run otherwise regular linux/amd64 binaries in some sandbox |
| |
| * gVisor |
| |
| - gVisor is a mostly complete re-implementation of the Linux system call interface, in Go |
| - runs under various modes ("platforms"): KVM, ptrace |
| - everything is hookable: ~FUSE on steroids |
| - performance pretty good, especially with KVM, not that we really need performance |
| - was Google-internal, open source May 2018 |
| |
| * Who uses gVisor? |
| |
| - App Engine 2nd gen runtimes (all new ones) |
| - Cloud Run |
| - Cloud Functions |
| - GKE Sandbox (opt-in hardened containers) |
| - ... |
| |
| * How to use gVisor? |
| |
| - tricky, ~unsupported to use directly |
| - comes with a "runsc" wrapper command that team recommends |
| |
| * runsc woes |
| |
| - runsc ("run secure container") is an OCI "runc"-compatible frontend to gVisor |
| - runc interface itself is tricky |
| - kinda gross to use directly |
| - syzkaller fuzzer does, though, so Go driver code exists |
| - but ... |
| |
| * root |
| |
| - runsc requires root |
| - without networking and ptrace instead of kvm, it _almost_ doesn't require root, but gvisor team isn't quite done with rootless |
| - App Engine, etc doesn't give us real root (uid=0, but fake, not enough caps, under gVisor itself) |
| - gvisor can't run nested under gvisor (can't run on Cloud Run, for example) |
| |
| * Docker |
| |
| - But Docker supports runsc easily: |
| |
| docker run --runtime=runsc ... |
| |
| - So we can just run a dynamically scaled pool of VMs running Docker + a backend HTTP server driving Docker, with a load balancer in front of the pool of VMs. |
| |
| * v3 architecture |
| |
| - existing HTTP frontend takes source, compiles binary (for now) |
| - HTTP PUTs compiled binary to backend HTTP server |
| |
| * v3 architecture |
| |
| .image arch.png _ 700 |
| |
| * On each VM |
| |
| - one privileged container that speaks HTTP |
| - has bind-mounted /var/run/docker.dock, to control its parent docker |
| - creates sibling-but-gvisor'ed containers for each play |
| - HTTP server reads PUT'ed binary, pipes into secure container, reads stdout/stderr response & exit status |
| - replies over HTTP to frontend |
| |
| * VM view |
| |
| .image cos.png _ 900 |
| |
| * Transition plan |
| |
| * Today |
| |
| - Frontend compiles to nacl/amd64p32, runs binary |
| |
| * Today+1 |
| |
| - Frontend compiles to nacl/amd64p32, runs binary |
| - opt-in support for using new backend (in either linux/amd64 or nacl/amd64p32 mode) |
| - temporary support in new backend to understand how to wrap nacl with sel_ldr binary |
| |
| * Today+2: |
| |
| - Frontend compiles to nacl/amd64p2, PUTs all requests to new backend |
| - gets us operational experience with backend before Go 1.14 release day chaos |
| |
| * Go 1.14 release day |
| |
| - flip a bit in frontend that forces linux/amd64 |
| - NaCl no longer allowed |
| - can delete all its playground code (minimal) |
| |
| * Future possibilities |
| |
| - stop shipping compiled binaries around the network? |
| - ... minimal win, might not be worth it |
| - unify compilation & running in the backend? |
| - 0, 1, or 2 sandbox services to update per Go release? |
| - 2 is too many |
| - 1 is today, acceptable |
| - 0 might be too risky |
| |
| * Alternatives |
| |
| * Cloud Run? |
| |
| - Almost perfect |
| - but doesn't let you disable network (yet? Cloud Functions has VPC connectors, can route traffic to Linux VM that drops packets) |
| - if root-less runsc happens, can run in ptrace mode under Cloud Run |
| |
| * GKE Sandbox? |
| |
| - not obviously any simpler, faster than this setup |
| - but might be alternative later |
| - worry about node pool elasticity latency? COS boots quickly. |
| |
| * WebAssembly? |
| |
| - large binaries for users |
| - still experimental support, but improving |
| - use WebAssembly for x/tools/cmd/present? |
| - ... because this Docker runsc contraption is too complicated for casual users giving talks on laptops |
| - or just run unsandboxes in cmd/present |
