proposal: add heap dump viewer proposal
I'm starting by writing down a few ideas. The doc is very
rough for now. I'll add more details as we figure them out.
Change-Id: I6e80895847985c78dc7423d3b3ace8f1491c19fb
Reviewed-on: https://go-review.googlesource.com/25083
Reviewed-by: Alan Donovan <adonovan@google.com>
diff --git a/design/16410-heap-viewer.md b/design/16410-heap-viewer.md
new file mode 100644
index 0000000..52623ab
--- /dev/null
+++ b/design/16410-heap-viewer.md
@@ -0,0 +1,129 @@
+# Proposal: Go Heap Dump Viewer
+
+Author(s): Michael Matloob
+
+Last updated: 20 July 2016
+
+Discussion at https://golang.org/issue/16410
+
+## Abstract
+
+This proposal is for a heap dump viewer for Go programs. This proposal will provide a
+web-based, graphical viewer as well as packages for analyzing and understanding heap
+dumps.
+
+## Background
+
+Sometimes Go programs use too much memory and the programmer wants to know why. Profiling
+gives the programmer statistical information about rates of allocation, but doesn't gives
+a specific concrete snapshot that can explain why a variable is live or how many
+instances of a given type are live.
+
+There currently exists a tool written by Keith Randall
+that takes heap dumps produced by `runtime/debug.WriteHeapDump` and converts them into
+the hprof format which can be understood by those Java heap analysis tools, but there
+are some issues with the tool in its current state. First, the tool is
+out of sync with the heaps dumped by Go. In addition, that tool got its type information from
+data structures maintained by the GC algorithm, but as the GC has advanced, it has been
+storing less and less type information over time. Because of those issues, we'll have to
+make major changes to the tool or perhaps rewrite the whole thing.
+
+Also, the process of getting a heap analysis on the screen from a running Go program involves
+multiple tools and dependencies, and is more complicated than it needs to be. There should
+be a simple and fast "one-click" solution to make it as easy as possible to understand
+what's happening in a program's heap.
+
+## Proposal
+
+TODO(matloob): Some of the details are still fuzzy, but here's the general outline of a solution:
+
+We'll use ELF core dumps as the source format for our heap analysis tools. We would build packages that would use the
+debug information in the DWARF section of the dump to find the roots and reconstruct type
+information for as much of the program as it can. Implementing this will likely involve improving
+the DWARF data produced by the compiler.
+
+Windows doesn't traditionally use core files, and darwin uses mach-o as its core dump format,
+so we'll have to provide a mechanism for users on those platforms to extract ELF core dumps
+from their programs.
+
+We'd use those packages to build a graphical web-based tool for viewing and analyzing heap dumps.
+The program would be pointed to a core dump and would serve a graphical web app that could be used
+to analyze the heap.
+
+Ideally, there will be a 'one-click' solution to get from running program to dump. One possible way
+to do this would be to add a library to expose a special HTTP handler. Requesting the page would that
+would trigger a core dump to a user-specified location on disk while the program's running, and start
+the heap dump viewer program.
+
+## Rationale
+
+TODO(matloob): More through discussion.
+
+The primary rationale for this feature is that users want to understand the memory usage of their programs
+and we don't currently provide convenient ways of doing that. Adding a heap dump viewer will allow us to
+do that.
+
+### Heap dump format
+
+There are three candidates for the format our tools will consume: the current format output by
+the Go heap dumper, the hprof format, and the ELF format proposed here.
+
+The advantage of using the current format is that we already have tools that produce it and consume it. But the format
+is non-standard and requires a strong dependence between the heap viewer and the runtime. That's been one
+of the problems with the current viewer. And the format produced by the runtime has changed slightly in each
+of the last few Go releases because it's tightly coupled with the Go runtime.
+
+The advantage of the hprof format is that there already exist many tools for analyzing hprof dumps.
+It will be a good idea to consider this format more throughly before making a decision. On the
+other hand many of those tools are neither polished nor easy to use. We can probably build
+better tools tailored for Go without great effort.
+
+The advantage of understanding ELF is that we can use the same tools to look at cores produced when a program
+OOMs (at least on Linux) as we do to examine heap dumps. Another benefit is that some cluster
+environments already collect and store core files when programs fail in production. Reusing this
+machinery would help Go programmers in those environments. And there already exist tools that grab core dumps
+so we might be able to reduce the amount of code in the runtime for producing dumps.
+
+## Compatibility
+
+As long as the compiler can output all necessary data needed to reconstruct type information for the heap
+in the DWARF data, we won't need to have a strong dependency on the Go distribution. The code can live in a subrepo
+not subject to the Go compatibility guarantee.
+
+## Implementation
+
+The implementation will broadly consist of three parts: First, support in the compiler and runtime for dumping
+all the data needed by the viewer; second, 'backend' tools that understand the format; and third, a 'frontend'
+viewer for those tools.
+
+### Compiler and Runtime Work
+
+TODO(matloob): more details
+
+The compiler work will mostly be a consist of filling any holes in the DWARF data that we need to recover type
+information of data in the heap.
+
+If we decide to use ELF cores, we may need runtime support for dumping cores, especially on platforms that
+don't dump cores in ELF format.
+
+### Heap libraries and viewer
+
+We will provide a reusable library that decodes a core file as a Go object graph with partial type information.
+Users can build their own tools based on this low-level library, but we also provide a web-based graphical tool for
+viewing and querying heap graphs.
+
+These are some of the types of queries we aim to answer with the heap viewer:
+* Show a histogram of live variables grouped by typed
+* Which variables account for the most memory?
+* What is a path from a GC root to this variable?
+* How much memory would become garbage if this variable were to become unreachable
+ or this pointer to become nil?
+* What are the inbound/outbound pointer edges to this node (variable)?
+* How much memory is used by a variable, considering padding, alignment, and span size?
+
+## Open issues (if applicable)
+
+Most of this proposal is open at this point, including:
+* the heap dump format
+* the design and implementation of the backend packages
+* the tools we use to build the frontend client.
\ No newline at end of file
