| # pprof | |
| pprof is a tool for visualization and analysis of profiling data. | |
| pprof reads a collection of profiling samples in profile.proto format and | |
| generates reports to visualize and help analyze the data. It can generate both | |
| text and graphical reports (through the use of the dot visualization package). | |
| profile.proto is a protocol buffer that describes a set of callstacks | |
| and symbolization information. A common usage is to represent a set of | |
| sampled callstacks from statistical profiling. The format is | |
| described on the src/proto/profile.proto file. For details on protocol | |
| buffers, see https://developers.google.com/protocol-buffers | |
| Profiles can be read from a local file, or over http. Multiple | |
| profiles of the same type can be aggregated or compared. | |
| If the profile samples contain machine addresses, pprof can symbolize | |
| them through the use of the native binutils tools (addr2line and nm). | |
| # pprof profiles | |
| pprof operates on data in the profile.proto format. Each profile is a collection | |
| of samples, where each sample is associated to a point in a location hierarchy, | |
| one or more numeric values, and a set of labels. Often these profiles represents | |
| data collected through statistical sampling of a program, so each sample | |
| describes a program call stack and a number or weight of samples collected at a | |
| location. pprof is agnostic to the profile semantics, so other uses are | |
| possible. The interpretation of the reports generated by pprof depends on the | |
| semantics defined by the source of the profile. | |
| # Usage Modes | |
| There are few different ways of using `pprof`. | |
| ## Report generation | |
| If a report format is requested on the command line: | |
| pprof <format> [options] source | |
| pprof will generate a report in the specified format and exit. | |
| Formats can be either text, or graphical. See below for details about | |
| supported formats, options, and sources. | |
| ## Interactive terminal use | |
| Without a format specifier: | |
| pprof [options] source | |
| pprof will start an interactive shell in which the user can type | |
| commands. Type `help` to get online help. | |
| ## Web interface | |
| If a host:port is specified on the command line: | |
| pprof -http=[host]:[port] [options] source | |
| pprof will start serving HTTP requests on the specified port. Visit | |
| the HTTP url corresponding to the port (typically `http://<host>:<port>/`) | |
| in a browser to see the interface. | |
| # Details | |
| The objective of pprof is to generate a report for a profile. The report is | |
| generated from a location hierarchy, which is reconstructed from the profile | |
| samples. Each location contains two values: *flat* is the value of the location | |
| itself, while *cum* is the value of the location plus all its | |
| descendants. Samples that include a location multiple times (eg for recursive | |
| functions) are counted only once per location. | |
| ## Options | |
| *options* configure the contents of a report. Each option has a value, | |
| which can be boolean, numeric, or strings. While only one format can | |
| be specified, most options can be selected independently of each | |
| other. | |
| Some common pprof options are: | |
| * **-flat [default]:** Sort entries based on their flat weight, on text reports. | |
| * **-cum:** Sort entries based on cumulative weight, on text reports. | |
| * **-functions [default]:** Accumulate samples at the function level; profile | |
| locations that describe the same function will be merged into a report entry. | |
| * **-lines:** Accumulate samples at the source line level; profile locations that | |
| describe the same function will be merged into a report entry. | |
| * **-addresses:** Accumulate samples at the instruction address; profile locations | |
| that describe the same function address will be merged into a report entry. | |
| * **-nodecount= _int_:** Maximum number of entries in the report. pprof will only print | |
| this many entries and will use heuristics to select which entries to trim. | |
| * **-focus= _regex_:** Only include samples that include a report entry matching | |
| *regex*. | |
| * **-ignore= _regex_:** Do not include samples that include a report entry matching | |
| *regex*. | |
| * **-show= _regex_:** Only show entries that match *regex*. | |
| * **-hide= _regex_:** Do not show entries that match *regex*. | |
| Each sample in a profile may include multiple values, representing different | |
| entities associated to the sample. pprof reports include a single sample value, | |
| which by convention is the last one specified in the report. The `sample_index=` | |
| option selects which value to use, and can be set to a number (from 0 to the | |
| number of values - 1) or the name of the sample value. | |
| Sample values are numeric values associated to a unit. If pprof can recognize | |
| these units, it will attempt to scale the values to a suitable unit for | |
| visualization. The `unit=` option will force the use of a specific unit. For | |
| example, `unit=sec` will force any time values to be reported in | |
| seconds. pprof recognizes most common time and memory size units. | |
| ## Tag filtering | |
| Samples in a profile may have tags. These tags have a name and a value; this | |
| value can be either numeric or a string. pprof can select samples from a | |
| profile based on these tags using the `-tagfocus` and `-tagignore` options. | |
| Generally, these options work as follows: | |
| * **-tagfocus=_regex_** or **-tagfocus=_range_:** Restrict to samples with tags | |
| matched by regexp or in range. | |
| * **-tagignore=_regex_** or **-tagignore=_range_:** Discard samples with tags | |
| matched by regexp or in range. | |
| When using `-tagfocus=regex` and `-tagignore=regex`, the regex will be compared | |
| to each value associated with each tag. If one specifies a value | |
| like `regex1,regex2`, then only samples with a tag value matching `regex1` | |
| and a tag value matching `regex2` will be kept. | |
| In addition to being able to filter on tag values, one can specify the name of | |
| the tag which a certain value must be associated with using the notation | |
| `-tagfocus=tagName=value`. Here, the `tagName` must match the tag's name | |
| exactly, and the value can be either a regex or a range. If one specifies | |
| a value like `regex1,regex2`, then samples with a tag value (paired with the | |
| specified tag name) matching either `regex1` or matching `regex2` will match. | |
| Here are examples explaining how `tagfocus` can be used: | |
| * `-tagfocus 128kb:512kb` accepts a sample iff it has any numeric tag with | |
| memory value in the specified range. | |
| * `-tagfocus mytag=128kb:512kb` accepts a sample iff it has a numeric tag | |
| `mytag` with memory value in the specified range. There isn't a way to say | |
| `-tagfocus mytag=128kb:512kb,16kb:32kb` | |
| or `-tagfocus mytag=128kb:512kb,mytag2=128kb:512kb`. Just single value or | |
| range for numeric tags. | |
| * `-tagfocus someregex` accepts a sample iff it has any string tag with | |
| `tagName:tagValue` string matching specified regexp. In the future, this | |
| will change to accept sample iff it has any string tag with `tagValue` string | |
| matching specified regexp. | |
| * `-tagfocus mytag=myvalue1,myvalue2` matches if either of the two tag values | |
| are present. | |
| `-tagignore` works similarly, except that it discards matching samples, instead | |
| of keeping them. | |
| If both the `-tagignore` and `-tagfocus` expressions (either a regexp or a | |
| range) match a given sample, then the sample will be discarded. | |
| ## Text reports | |
| pprof text reports show the location hierarchy in text format. | |
| * **-text:** Prints the location entries, one per line, including the flat and cum | |
| values. | |
| * **-tree:** Prints each location entry with its predecessors and successors. | |
| * **-peek= _regex_:** Print the location entry with all its predecessors and | |
| successors, without trimming any entries. | |
| * **-traces:** Prints each sample with a location per line. | |
| ## Graphical reports | |
| pprof can generate graphical reports on the DOT format, and convert them to | |
| multiple formats using the graphviz package. | |
| These reports represent the location hierarchy as a graph, with a report entry | |
| represented as a node. Solid edges represent a direct connection between | |
| entries, while dotted edges represent a connection where some intermediate nodes | |
| have been removed. Nodes are removed using heuristics to limit the size of | |
| the graph, controlled by the *nodecount* option. | |
| The size of each node represents the flat weight of the node, and the width of | |
| each edge represents the cumulative weight of all samples going through | |
| it. Nodes are colored according to their cumulative weight, highlighting the | |
| paths with the highest cum weight. | |
| * **-dot:** Generates a report in .dot format. All other formats are generated from | |
| this one. | |
| * **-svg:** Generates a report in SVG format. | |
| * **-web:** Generates a report in SVG format on a temp file, and starts a web | |
| browser to view it. | |
| * **-png, -jpg, -gif, -pdf:** Generates a report in these formats, | |
| ## Annotated code | |
| pprof can also generate reports of annotated source with samples associated to | |
| them. For these, the source or binaries must be locally available, and the | |
| profile must contain data with the appropriate level of detail. | |
| pprof will look for source files on its current working directory and all its | |
| ancestors. pprof will look for binaries on the directories specified in the | |
| `$PPROF_BINARY_PATH` environment variable, by default `$HOME/pprof/binaries` | |
| (`%USERPROFILE%\pprof\binaries` on Windows). It will look binaries up by name, | |
| and if the profile includes linker build ids, it will also search for them in | |
| a directory named as the build id. | |
| pprof uses the binutils tools to examine and disassemble the binaries. By | |
| default it will search for those tools in the current path, but it can also | |
| search for them in a directory pointed to by the environment variable | |
| `$PPROF_TOOLS`. | |
| * **-disasm= _regex_:** Generates an annotated source listing for functions matching | |
| regex, with flat/cum weights for each source line. | |
| * **-list= _regex_:** Generates an annotated disassembly listing for functions | |
| matching *regex*. | |
| * **-weblist= _regex_:** Generates a source/assembly combined annotated listing for | |
| functions matching *regex*, and starts a web browser to display it. | |
| # Fetching profiles | |
| pprof can read profiles from a file or directly from a URL over http. Its native | |
| format is a gzipped profile.proto file, but it can also accept some legacy | |
| formats generated by [gperftools](https://github.com/gperftools/gperftools). | |
| When fetching from a URL handler, pprof accepts options to indicate how much to | |
| wait for the profile. | |
| * **-seconds= _int_:** Makes pprof request for a profile with the specified duration | |
| in seconds. Only makes sense for profiles based on elapsed time, such as CPU | |
| profiles. | |
| * **-timeout= _int_:** Makes pprof wait for the specified timeout when retrieving a | |
| profile over http. If not specified, pprof will use heuristics to determine a | |
| reasonable timeout. | |
| If multiple profiles are specified, pprof will fetch them all and merge | |
| them. This is useful to combine profiles from multiple processes of a | |
| distributed job. The profiles may be from different programs but must be | |
| compatible (for example, CPU profiles cannot be combined with heap profiles). | |
| pprof can subtract a profile from another in order to compare them. For that, | |
| use the **-base= _profile_** option, where *profile* is the filename or URL for the | |
| profile to be subtracted. This may result on some report entries having negative | |
| values. | |
| ## Symbolization | |
| pprof can add symbol information to a profile that was collected only with | |
| address information. This is useful for profiles for compiled languages, where | |
| it may not be easy or even possible for the profile source to include function | |
| names or source coordinates. | |
| pprof can extract the symbol information locally by examining the binaries using | |
| the binutils tools, or it can ask running jobs that provide a symbolization | |
| interface. | |
| pprof will attempt symbolizing profiles by default, and its `-symbolize` option | |
| provides some control over symbolization: | |
| * **-symbolize=none:** Disables any symbolization from pprof. | |
| * **-symbolize=local:** Only attempts symbolizing the profile from local | |
| binaries using the binutils tools. | |
| * **-symbolize=remote:** Only attempts to symbolize running jobs by contacting | |
| their symbolization handler. | |
| For local symbolization, pprof will look for the binaries on the paths specified | |
| by the profile, and then it will search for them on the path specified by the | |
| environment variable `$PPROF_BINARY_PATH`. Also, the name of the main binary can | |
| be passed directly to pprof as its first parameter, to override the name or | |
| location of the main binary of the profile, like this: | |
| pprof /path/to/binary profile.pb.gz | |
| By default pprof will attempt to demangle and simplify C++ names, to provide | |
| readable names for C++ symbols. It will aggressively discard template and | |
| function parameters. This can be controlled with the `-symbolize=demangle` | |
| option. Note that for remote symbolization mangled names may not be provided by | |
| the symbolization handler. | |
| * **--symbolize=demangle=none:** Do not perform any demangling. Show mangled | |
| names if available. | |
| * **-symbolize=demangle=full:** Demangle, but do not perform any | |
| simplification. Show full demangled names if available. | |
| * **-symbolize=demangle=templates:** Demangle, and trim function parameters, but | |
| not template parameters. | |