Catapult

Timeline-Based Measurement v2 is a system for computing metrics from traces.

A TBM2 metric is a Javascript function that takes a trace Model and produces Histograms.

Coding Practices

Please follow the Catapult Javascript style guide so that the TBM2 maintainers can refactor your metric when we need to update the TBM2 API.

Please write a unit test for your metric.

If your metric computes information from the trace that may be of general use to other metrics or the trace viewer UI, then the TBM2 maintainers may ask for your help to generalize your innovation into a part of the Trace Model such as the UserModel or ModelHelpers.

Use the dev server to develop and debug your metric.

  • Run ./bin/run_dev_server
  • Navigate to http://localhost:8003/tracing_examples/trace_viewer.html.
  • Open a trace that your metric can be computed from.
  • Open the Metrics side panel on the right.
  • Select your metric from the drop-down.
  • Inspect the results and change your metric if necessary.
  • Open different traces to explore corner cases in your metric.

Trace Model

Trace logs are JSON files produced by tracing systems in Chrome, Android, linux perf, BattOr, etc. The trace model is an object-level representation of events parsed from a trace log. The trace model contains Javascript objects representing

Histograms

A Histogram is basically a common histogram, but with a few extra bells and whistles that are particularly useful for TBM2 metrics.

  • Specify units of samples and improvement direction with Unit
  • JSON serialization with asDict()/fromDict()
  • Build custom bin boundaries with HistogramBinBoundaries
  • Compute statistics such as average, stddev, sum, and percentiles
  • Customize which statistics are serialized with customizeSummaryOptions()
  • Count non-numeric samples
  • Store a random subset of sample values
  • getDifferenceSignificance() computes whether two histograms are significantly different with a Mann-Whitney U hypothesis test
  • addHistogram() merges two Histograms with the same units and bin boundaries

But the most complex special feature of Histograms is their Diagnostics.

Diagnostics

When a metric significantly regresses, you then need to diagnose why it regressed. Diagnostics are pieces of information that metrics attach to Histograms in order help you diagnose regressions. Diagnostics may be associated either with the entire Histogram directly, or with a particular sample.

Attach a Diagnostic to a Histogram:

histogram.diagnostics.set('name', diagnostic)
// or
values.addHistogram(histogram, {name: diagnostic})

Attach a Diagnostic to a sample:

histogram.addSample(number, {name: diagnostic})

The types of Diagnostics are

  • Generic: This can contain any data that can be serialized and deserialized using JSON.stringify() and JSON.parse(), including numbers, strings, Arrays, and dictionaries (simple Objects). It will be visualized using generic-object-view, which is quite smart about displaying tabular data using tables, URLs using HTML anchor tags, pretty-printing, recursive data structures, and more.
  • RelatedEventSet: This is a Set of references to Events in the trace model. Visually, they are displayed as HTML links which, when clicked in the metrics-side-panel, select the referenced Events in the trace viewer's timeline view. When clicked in results2.html, they currently do nothing, but should eventually open the trace that contains the events and select them.
  • Breakdown: Structurally, these are Maps from strings to numbers. Conceptually, they describe what fraction of a whole (either a Histogram or a sample) is due to some sort of category - either a category of event, CPU sample, memory consumer, whathaveyou. Visually, they are a stacked bar chart with a single bar, which is spiritually a pie chart, but less misleading.
  • RelatedValueSet: These are Sets of references to other Histograms. Visually, they are a set of HTML links which, when clicked, select the contained Histograms. The text content of the HTML link is the name of the referenced Histogram.
  • RelatedValueMap: These are Maps from strings to references to other Histograms. Visually, they are a set of HTML links similar to RelatedValueSet, but the text content of the link is the Map‘s string key instead of the Histogram’s name. One example application is when a Histogram was produced not directly by a metric, but rather by merging together other Histograms, then it will have a RelatedValueMap named ‘merged from’ that refers to the Histograms that were merged by their grouping key, e.g. the telemetry story name.
  • RelatedHistogramBreakdown: Structurally, this is a RelatedValueMap, but conceptually and visually, this is a Breakdown. Whereas Breakdown‘s stacked bar chart derives its data from the numbers contained explicitly in the Breakdown, a RelatedHistogramBreakdown’s stacked bar chart derives its data from the referenced Histograms' sums.
  • IterationInfo: This is automatically attached to every Histogram produced by telemetry. Structurally, it's a class with explicit named fields. Conceptually, it contains information about the origins of the trace that was consumed by the metric that produced the Histogram, such as the benchmark name, story name, benchmark start timestamp, OS version, Chrome version, etc. Visually, IterationInfos are displayed as a table.

Consumers of Histograms

Histograms are consumed by

  • value-set-table in both results2.html and the Metrics side panel in trace viewer,
  • the dashboard indirectly via their statistics.

Currently, telemetry discards Histograms and Diagnostics, and only passes their statistics scalars to the dashboard. Histograms and their Diagnostics will be passed directly to the dashboard early 2017.