Fruit benchmarks

Fruit contains some code to benchmark various metrics (e.g. performance, compile time, executable size) in an automated way.

Benchmark suites

The suites folder contains suites of Fruit (or Fruit-related) benchmarks that can be run using run_benchmarks.py. For example:

$ ~/projects/fruit/extras/benchmark/run_benchmarks.py \
    --continue-benchmark=true \
    --benchmark-definition ~/projects/fruit/extras/benchmark/suites/fruit_full.yml
    --output-file ~/fruit_bench_results.txt \
    --fruit-sources-dir ~/projects/fruit \
    --fruit-benchmark-sources-dir ~/projects/fruit \
    --boost-di-sources-dir ~/projects/boost-di

Once the benchmark run completes, you can format the results using some pre-defined tables, see the section below.

The following benchmark suites are defined:

• fruit_full.yml: full set of Fruit benchmarks (using the Fruit 3.x API).
• fruit_mostly_full.yml: a subset of the tests in fruit_full.yml.
• fruit_quick.yml: this is an even smaller subset, and the number of runs is capped at 10 so the confidence intervals might be wider. It's useful as a quicker (around 10-15min) way to get a rough idea of the performance (e.g. to evaluate the performance impact of a commit, during development).
• fruit_single.yml: runs the Fruit runtime benchs under a single compiler and with just 1 combination of flags. This also caps the number of runs at 8, so the resulting confidence intervals might be wider than they would be with fruit_full.yml. This is a quick benchmark that can used during development of performance optimizations.
• fruit_debug.yml: a suite used to debug Fruit's benchmarking code. This is very quick, but the actual results are not meaningful. Run this after changing any benchmarking code, to check that it still works.
• boost_di: unlike the others, this benchmark suite exercises the Boost.DI library (still in boost-experimental at the time of writing) instead of Fruit.

Benchmark tables

The tables folder contains some table definitions that can be used to get a human-readable representations of benchmark results generated using run_benchmarks.py.

Note that there isn't a 1:1 mapping between benchmark suites and benchmark tables; the same table definition can be used with multiple benchmark suites (for example, a full suite and a quick variant that only has a subset of the dimensions) and multiple table definitions might be appropriate to display the results of a single suite (for example, there could be a table definition that displays only metrics meaningful to Fruit users and one that also displays more fine-grained metrics that are only meaningful to Fruit developers).

Example usage of format_bench_results.py with one of these table definitions:

$ ~/projects/fruit/extras/benchmark/format_bench_results.py \
    --benchmark-results ~/fruit_bench_results.txt \
    --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml

It's also possible to compare two benchmark results (for example, when running the same benchmarks before and after a Fruit commit):

$ ~/projects/fruit/extras/benchmark/format_bench_results.py \
    --benchmark-results ~/fruit_bench_results_after.txt \
    --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml \
    --baseline-benchmark-results ~/fruit_bench_results_before.txt

The following tables are defined:

• fruit_wiki.yml: the “main” table definition, with the tables that are in Fruit's wiki.
• fruit_internal.yml: a more detailed version of fruit_wiki.yml, also displaying metrics that are only meaningful to Fruit developers (e.g. splitting the setup time into component creation time and normalization time).

Manual benchmarks

In some cases, you might want to run the benchmarks manually (e.g. if you want to use callgrind to profile the benchmark run). This is how you can do that:

$ cd ~/projects/fruit
$ mkdir build
$ cd build
$ CXX=g++-6 cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo
$ make -j 10
$ cd ..
$ mkdir generated-benchs
$ extras/benchmark/generate_benchmark.py \
    --compiler g++-6 \
    --fruit-sources-dir ~/projects/fruit \
    --fruit-build-dir ~/projects/fruit/build \
    --num-components-with-no-deps 10 \
    --num-components-with-deps 90 \
    --num-deps 10 \
    --output-dir generated-benchs \
    --generate-debuginfo=true
$ cd generated-benchs
$ make -j 10
$ valgrind \
    --tool=callgrind \
    --simulate-cache=yes \
    --dump-instr=yes \
    ./main 10000