Comparing benchmark functions

Comparing benchmark functions is supported via the optional library_benchmark_group! argument compare_by_id (The default value for compare_by_id is false). Only benches with the same id are compared, which allows to single out cases which don't need to be compared. In the following example, the case_3 and multiple bench are compared with each other in addition to the usual comparison with the previous run:

extern crate iai_callgrind;
mod my_lib { pub fn bubble_sort(_: Vec<i32>) -> Vec<i32> { vec![] } }
use iai_callgrind::{library_benchmark, library_benchmark_group, main};
use std::hint::black_box;

#[library_benchmark]
#[bench::case_3(vec![1, 2, 3])]
#[benches::multiple(args = [vec![1, 2], vec![1, 2, 3, 4]])]
fn bench_bubble_sort_best_case(input: Vec<i32>) -> Vec<i32> {
    black_box(my_lib::bubble_sort(input))
}

#[library_benchmark]
#[bench::case_3(vec![3, 2, 1])]
#[benches::multiple(args = [vec![2, 1], vec![4, 3, 2, 1]])]
fn bench_bubble_sort_worst_case(input: Vec<i32>) -> Vec<i32> {
    black_box(my_lib::bubble_sort(input))
}

library_benchmark_group!(
    name = bench_bubble_sort;
    compare_by_id = true;
    benchmarks = bench_bubble_sort_best_case, bench_bubble_sort_worst_case
);

fn main() {
main!(library_benchmark_groups = bench_bubble_sort);
}

Note if compare_by_id is true, all benchmark functions are compared with each other, so you are not limited to two benchmark functions per comparison group.

Here's the benchmark output of the above example to see what is happening:

my_benchmark::bubble_sort_group::bubble_sort_best_case case_2:vec! [1, 2]
  Instructions:                  63|N/A             (*********)
  L1 Hits:                       86|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       4|N/A             (*********)
  Total read+write:              91|N/A             (*********)
  Estimated Cycles:             231|N/A             (*********)
my_benchmark::bubble_sort_group::bubble_sort_best_case multiple_0:vec! [1, 2, 3]
  Instructions:                  94|N/A             (*********)
  L1 Hits:                      123|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       4|N/A             (*********)
  Total read+write:             128|N/A             (*********)
  Estimated Cycles:             268|N/A             (*********)
my_benchmark::bubble_sort_group::bubble_sort_best_case multiple_1:vec! [1, 2, 3, 4]
  Instructions:                 136|N/A             (*********)
  L1 Hits:                      174|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       4|N/A             (*********)
  Total read+write:             179|N/A             (*********)
  Estimated Cycles:             319|N/A             (*********)
my_benchmark::bubble_sort_group::bubble_sort_worst_case case_2:vec! [2, 1]
  Instructions:                  66|N/A             (*********)
  L1 Hits:                       91|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       4|N/A             (*********)
  Total read+write:              96|N/A             (*********)
  Estimated Cycles:             236|N/A             (*********)
  Comparison with bubble_sort_best_case case_2:vec! [1, 2]
  Instructions:                  63|66              (-4.54545%) [-1.04762x]
  L1 Hits:                       86|91              (-5.49451%) [-1.05814x]
  L2 Hits:                        1|1               (No change)
  RAM Hits:                       4|4               (No change)
  Total read+write:              91|96              (-5.20833%) [-1.05495x]
  Estimated Cycles:             231|236             (-2.11864%) [-1.02165x]
my_benchmark::bubble_sort_group::bubble_sort_worst_case multiple_0:vec! [3, 2, 1]
  Instructions:                 103|N/A             (*********)
  L1 Hits:                      138|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       4|N/A             (*********)
  Total read+write:             143|N/A             (*********)
  Estimated Cycles:             283|N/A             (*********)
  Comparison with bubble_sort_best_case multiple_0:vec! [1, 2, 3]
  Instructions:                  94|103             (-8.73786%) [-1.09574x]
  L1 Hits:                      123|138             (-10.8696%) [-1.12195x]
  L2 Hits:                        1|1               (No change)
  RAM Hits:                       4|4               (No change)
  Total read+write:             128|143             (-10.4895%) [-1.11719x]
  Estimated Cycles:             268|283             (-5.30035%) [-1.05597x]
my_benchmark::bubble_sort_group::bubble_sort_worst_case multiple_1:vec! [4, 3, 2, 1]
  Instructions:                 154|N/A             (*********)
  L1 Hits:                      204|N/A             (*********)
  L2 Hits:                        1|N/A             (*********)
  RAM Hits:                       4|N/A             (*********)
  Total read+write:             209|N/A             (*********)
  Estimated Cycles:             349|N/A             (*********)
  Comparison with bubble_sort_best_case multiple_1:vec! [1, 2, 3, 4]
  Instructions:                 136|154             (-11.6883%) [-1.13235x]
  L1 Hits:                      174|204             (-14.7059%) [-1.17241x]
  L2 Hits:                        1|1               (No change)
  RAM Hits:                       4|4               (No change)
  Total read+write:             179|209             (-14.3541%) [-1.16760x]
  Estimated Cycles:             319|349             (-8.59599%) [-1.09404x]

The procedure of the comparison algorithm:

1. Run all benches in the first benchmark function
2. Run the first bench in the second benchmark function and if there is a bench in the first benchmark function with the same id compare them
3. Run the second bench in the second benchmark function ...
4. ...
5. Run the first bench in the third benchmark function and if there is a bench in the first benchmark function with the same id compare them. If there is a bench with the same id in the second benchmark function compare them.
6. Run the second bench in the third benchmark function ...
7. and so on ... until all benches are compared with each other

Neither the order nor the amount of benches within the benchmark functions matters, so it is not strictly necessary to mirror the bench ids of the first benchmark function in the second, third, etc. benchmark function.