Comparing with baselines

Usually, two consecutive benchmark runs let Iai-Callgrind compare these two runs. It's sometimes desirable to compare the current benchmark run against a static reference, instead. For example, if you're working longer on the implementation of a feature, you may wish to compare against a baseline from another branch or the commit from which you started off hacking on your new feature to make sure you haven't introduced performance regressions. Iai-Callgrind offers such custom baselines. If you are familiar with criterion.rs, the following command line arguments should also be very familiar to you:

• --save-baseline=NAME (env: IAI_CALLGRIND_SAVE_BASELINE): Compare against the NAME baseline if present and then overwrite it.
• --baseline=NAME (env: IAI_CALLGRIND_BASELINE): Compare against the NAME baseline without overwriting it
• --load-baseline=NAME (env: IAI_CALLGRIND_LOAD_BASELINE): Load the NAME baseline as the new data set instead of creating a new one. This option needs also --baseline=NAME to be present.

If NAME is not present, NAME defaults to default.

For example to create a static reference from the main branch and compare it:

git checkout main
cargo bench --bench <benchmark> -- --save-baseline=main
git checkout feature
# ... HACK ... HACK
cargo bench --bench <benchmark> -- --baseline main

Sticking to the above execution sequence,

cargo bench --bench my_benchmark -- --save-baseline=main

prints something like that with an additional line Baselines in the output.

my_benchmark::my_group::bench_library
  Baselines:                   main|main
  Instructions:                 280|N/A             (*********)
  L1 Hits:                      374|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       6|N/A             (*********)
  Total read+write:             381|N/A             (*********)
  Estimated Cycles:             589|N/A             (*********)

After you've made some changes to your code, running

cargo bench --bench my_benchmark -- --baseline=main`

prints something like the following:

my_benchmark::my_group::bench_library
  Baselines:                       |main
  Instructions:                 214|280             (-23.5714%) [-1.30841x]
  L1 Hits:                      287|374             (-23.2620%) [-1.30314x]
  L2 Hits:                        1|1               (No change)
  RAM Hits:                       6|6               (No change)
  Total read+write:             294|381             (-22.8346%) [-1.29592x]
  Estimated Cycles:             502|589             (-14.7708%) [-1.17331x]