Performance Regressions

With Iai-Callgrind you can define limits for each event kinds over which a performance regression can be assumed. Per default, Iai-Callgrind does not perform default regression checks, and you have to opt-in with a RegressionConfig at benchmark level with a LibraryBenchmarkConfig or BinaryBenchmarkConfig or at a global level with Command-line arguments or Environment variables.

Define a performance regression

A performance regression check consists of an EventKind and a percentage. If the percentage is negative, then a regression is assumed to be below this limit.

The default EventKind is EventKind::Ir with a value of +10%.

For example, in a Library Benchmark, define a limit of +5% for the total instructions executed (the Ir event kind) in all benchmarks of this file :

extern crate iai_callgrind;
mod my_lib { pub fn bubble_sort(_: Vec<i32>) -> Vec<i32> { vec![] } }
use iai_callgrind::{
    library_benchmark, library_benchmark_group, main, LibraryBenchmarkConfig,
    RegressionConfig, EventKind
};
use std::hint::black_box;

#[library_benchmark]
fn bench_library() -> Vec<i32> {
    black_box(my_lib::bubble_sort(vec![3, 2, 1]))
}

library_benchmark_group!(name = my_group; benchmarks = bench_library);

fn main() {
main!(
    config = LibraryBenchmarkConfig::default()
        .regression(
            RegressionConfig::default()
                .limits([(EventKind::Ir, 5.0)])
        );
    library_benchmark_groups = my_group
);
}

Now, if the comparison of the Ir events of the current bench_library benchmark run with the previous run results in an increase of over 5%, the benchmark fails. Please, also have a look at the api docs for further configuration options.

Running the benchmark from above the first time results in the following output:

my_benchmark::my_group::bench_library
  Instructions:                 215|N/A             (*********)
  L1 Hits:                      288|N/A             (*********)
  L2 Hits:                        0|N/A             (*********)
  RAM Hits:                       7|N/A             (*********)
  Total read+write:             295|N/A             (*********)
  Estimated Cycles:             533|N/A             (*********)

Let's assume there's a change in my_lib::bubble_sort which has increased the instruction counts, then running the benchmark again results in an output something similar to this:

my_benchmark::my_group::bench_library
  Instructions:                 281|215             (+30.6977%) [+1.30698x]
  L1 Hits:                      374|288             (+29.8611%) [+1.29861x]
  L2 Hits:                        0|0               (No change)
  RAM Hits:                       8|7               (+14.2857%) [+1.14286x]
  Total read+write:             382|295             (+29.4915%) [+1.29492x]
  Estimated Cycles:             654|533             (+22.7017%) [+1.22702x]
Performance has regressed: Instructions (281 > 215) regressed by +30.6977% (>+5.00000)
iai_callgrind_runner: Error: Performance has regressed.
error: bench failed, to rerun pass `-p the-crate --bench my_benchmark`

Caused by:
  process didn't exit successfully: `/path/to/your/project/target/release/deps/my_benchmark-a9b36fec444944bd --bench` (exit status: 1)
error: Recipe `bench-test` failed on line 175 with exit code 1

Which event to choose to measure performance regressions?

If in doubt, the definite answer is Ir (instructions executed). If Ir event counts decrease noticeable the function (binary) runs faster. The inverse statement is also true: If the Ir counts increase noticeable, there's a slowdown of the function (binary).

These statements are not so easy to transfer to Estimated Cycles and the other event counts. But, depending on the scenario and the function (binary) under test, it can be reasonable to define more regression checks.

Who actually uses instructions to measure performance?

The ones known to the author of this humble guide are

• SQLite: They use mainly cpu instructions to measure performance improvements (and regressions).
• Also in benchmarks of the rustc compiler, instruction counts play a great role. But, they also use cache metrics and cycles.

If you know of others, please feel free to add them to this list.