Class: Minitest::Benchmark

Relationships & Source Files
Extension / Inclusion / Inheritance Descendants
Subclasses: BenchSpec
Super Chains via Extension / Inclusion / Inheritance
Class Chain: self, Test, Guard, Runnable
Instance Chain: self, Test, Guard, Test::LifecycleHooks, Reportable, Assertions, Runnable
Inherits:	Minitest::Test ::Object Minitest::Runnable Minitest::Test Minitest::Benchmark
Defined in:	lib/minitest/benchmark.rb

Overview

Subclass Benchmark to create your own benchmark runs. Methods starting with “bench_” get executed on a per-class.

See Assertions

Class Method Summary

.bench_exp(min, max, base = 10)

Returns a set of ranges stepped exponentially from min to max by powers of base.
.bench_linear(min, max, step = 10)

Returns a set of ranges stepped linearly from min to max by step.
.bench_range

Specifies the ranges used for benchmarking for that class.

Test - Inherited

.i_suck_and_my_tests_are_order_dependent!	Call this at the top of your tests when you absolutely positively need to have ordered tests.
.make_my_diffs_pretty!	Make diffs for this Test use `#pretty_inspect` so that diff in assert_equal can have more details.
.parallelize_me!	Call this at the top of your tests when you want to run your tests in parallel.
.runnable_methods	Returns all instance methods starting with “test_”.
.test_order	Defines the order to run tests (:random by default).

Guard - Extended

jruby?	Is this running on jruby?
maglev?	Is this running on maglev?
mri?	Is this running on mri?
osx?	Is this running on macOS?
rubinius?	Is this running on rubinius?
windows?	Is this running on windows?

Runnable - Inherited

.methods_matching	Returns all instance methods matching the pattern `re`.
.run	Responsible for running all runnable methods in a given class, each in its own instance.
.run_one_method	Runs a single method and has the reporter record the result.
.runnable_methods	Each subclass of Runnable is responsible for overriding this method to return all runnable methods.
.runnables	Returns all subclasses of Runnable.

Instance Attribute Summary

Reportable - Included

#error?	Did this run error?
#passed?	Did this run pass?
#skipped?	Was this run skipped?

Assertions - Included

#skipped?

Was this testcase skipped? Meant for #teardown.

Runnable - Inherited

#assertions	Number of assertions executed in this run.
#failures	An assertion raised during the run, if any.
#name	Name of the run.
#name=	Set the name of the run.
#time	The time it took to run.
#passed?	Did this run pass?
#skipped?	Was this run skipped? See `#passed?` for more information.

Instance Method Summary

#assert_performance(validation, &work)

Runs the given work, gathering the times of each run.
#assert_performance_constant(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a constant rate (eg, linear slope == 0) within a given threshold.
#assert_performance_exponential(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a exponential curve within a given error threshold.
#assert_performance_linear(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a straight line within a given error threshold.
#assert_performance_logarithmic(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a logarithmic curve within a given error threshold.
#assert_performance_power(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered curve fit to match a power curve within a given error threshold.
#fit_error(xys)

Takes an array of x/y pairs and calculates the general R^2 value.
#fit_exponential(xs, ys)

To fit a functional form: y = ae^(bx).
#fit_linear(xs, ys)

Fits the functional form: a + bx.
#fit_logarithmic(xs, ys)

To fit a functional form: y = a + b*ln(x).
#fit_power(xs, ys)

To fit a functional form: y = ax^b.
#sigma(enum, &block)

Enumerates over enum mapping block if given, returning the sum of the result.
#validation_for_fit(msg, threshold)

Returns a proc that calls the specified fit method and asserts that the error is within a tolerable threshold.

Test - Inherited

#run	Runs a single test with setup/teardown hooks.

Guard - Included

#jruby?	Is this running on jruby?
#maglev?	Is this running on maglev?
#mri?	Is this running on mri?
#osx?	Is this running on macOS?
#rubinius?	Is this running on rubinius?
#windows?	Is this running on windows?

Test::LifecycleHooks - Included

#after_setup	Runs before every test, after setup.
#after_teardown	Runs after every test, after teardown.
#before_setup	Runs before every test, before setup.
#before_teardown	Runs after every test, before teardown.
#setup	Runs before every test.
#teardown	Runs after every test.

Reportable - Included

#location	The location identifier of this test.
#result_code	Returns “.”, “F”, or “E” based on the result of the run.

Assertions - Included

#assert	Fails unless `test` is truthy.
#assert_empty	Fails unless `obj` is empty.
#assert_equal	Fails unless `exp == act` printing the difference between the two, if possible.
#assert_in_delta	For comparing Floats.
#assert_in_epsilon	For comparing Floats.
#assert_includes	Fails unless `collection` includes `obj`.
#assert_instance_of	Fails unless `obj` is an instance of `cls`.
#assert_kind_of	Fails unless `obj` is a kind of `cls`.
#assert_match	Fails unless `matcher` `=~` `obj`.
#assert_mock	Assert that the mock verifies correctly.
#assert_nil	Fails unless `obj` is nil.
#assert_operator	For testing with binary operators.
#assert_output	Fails if stdout or stderr do not output the expected results.
#assert_path_exists	Fails unless `path` exists.
#assert_predicate	For testing with predicates.
#assert_raises	Fails unless the block raises one of `exp`.
#assert_respond_to	Fails unless `obj` responds to `meth`.
#assert_same	Fails unless `exp` and `act` are `#equal?`
#assert_send	`send_ary` is a receiver, message and arguments.
#assert_silent	Fails if the block outputs anything to stderr or stdout.
#assert_throws	Fails unless the block throws `sym`
#capture_io	Captures $stdout and $stderr into strings:
#capture_subprocess_io	Captures $stdout and $stderr into strings, using Tempfile to ensure that subprocess IO is captured as well.
#diff	Returns a diff between `exp` and `act`.
#exception_details	Returns details for exception `e`
#fail_after	Fails after a given date (in the local time zone).
#flunk	Fails with `msg`.
#message	Returns a proc that will output `msg` along with the default message.
#mu_pp	This returns a human-readable version of `obj`.
#mu_pp_for_diff	This returns a diff-able more human-readable version of `obj`.
#pass	used for counting assertions.
#refute	Fails if `test` is truthy.
#refute_empty	Fails if `obj` is empty.
#refute_equal	Fails if `exp == act`.
#refute_in_delta	For comparing Floats.
#refute_in_epsilon	For comparing Floats.
#refute_includes	Fails if `collection` includes `obj`.
#refute_instance_of	Fails if `obj` is an instance of `cls`.
#refute_kind_of	Fails if `obj` is a kind of `cls`.
#refute_match	Fails if `matcher` `=~` `obj`.
#refute_nil	Fails if `obj` is nil.
#refute_operator	Fails if `o1` is not `op` `o2`.
#refute_path_exists	Fails if `path` exists.
#refute_predicate	For testing with predicates.
#refute_respond_to	Fails if `obj` responds to the message `meth`.
#refute_same	Fails if `exp` is the same (by object identity) as `act`.
#skip	Skips the current run.
#skip_until	Skips the current run until a given date (in the local time zone).
#things_to_diff	Returns things to diff [expect, butwas], or [nil, nil] if nothing to diff.

Runnable - Inherited

#result_code	Returns a single character string to print based on the result of the run.
#run	Runs a single method.

Class Method Details

.bench_exp(min, max, base = 10)

Returns a set of ranges stepped exponentially from min to max by powers of base. Eg:

bench_exp(2, 16, 2) # => [2, 4, 8, 16]

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 35


def self.bench_exp min, max, base = 10
  min = (Math.log10(min) / Math.log10(base)).to_i
  max = (Math.log10(max) / Math.log10(base)).to_i

  (min..max).map { |m| base ** m }.to_a
end

.bench_linear(min, max, step = 10)

Returns a set of ranges stepped linearly from min to max by step. Eg:

bench_linear(20, 40, 10) # => [20, 30, 40]

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 48


def self.bench_linear min, max, step = 10
  (min..max).step(step).to_a
rescue LocalJumpError # 1.8.6
  r = []; (min..max).step(step) { |n| r << n }; r
end

.bench_range

Specifies the ranges used for benchmarking for that class. Defaults to exponential growth from 1 to 10k by powers of 10. Override if you need different ranges for your benchmarks.

Instance Method Details

#assert_performance(validation, &work)

Runs the given work, gathering the times of each run. Range and times are then passed to a given validation proc. Outputs the benchmark name and times in tab-separated format, making it easy to paste into a spreadsheet for graphing or further analysis.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  validation = proc { |x, y| ... }
  assert_performance validation do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 83


def assert_performance validation, &work
  range = self.class.bench_range

  io.print "#{self.name}"

  times = []

  range.each do |x|
    GC.start
    t0 = Minitest.clock_time
    instance_exec(x, &work)
    t = Minitest.clock_time - t0

    io.print "\t%9.6f" % t
    times << t
  end
  io.puts

  validation[range, times]
end

#assert_performance_constant(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a constant rate (eg, linear slope == 0) within a given threshold. Note: because we're testing for a slope of 0, R^2 is not a good determining factor for the fit, so the threshold is applied against the slope itself. As such, you probably want to tighten it from the default.

See www.graphpad.com/guides/prism/8/curve-fitting/reg_intepretingnonlinr2.htm for more details.

Fit is calculated by #fit_linear.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_constant 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 127


def assert_performance_constant threshold = 0.99, &work
  validation = proc do |range, times|
    a, b, rr = fit_linear range, times
    assert_in_delta 0, b, 1 - threshold
    [a, b, rr]
  end

  assert_performance validation, &work
end

#assert_performance_exponential(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a exponential curve within a given error threshold.

Fit is calculated by #fit_exponential.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_exponential 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 153


def assert_performance_exponential threshold = 0.99, &work
  assert_performance validation_for_fit(:exponential, threshold), &work
end

#assert_performance_linear(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a straight line within a given error threshold.

Fit is calculated by #fit_linear.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_linear 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 193


def assert_performance_linear threshold = 0.99, &work
  assert_performance validation_for_fit(:linear, threshold), &work
end

#assert_performance_logarithmic(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a logarithmic curve within a given error threshold.

Fit is calculated by #fit_logarithmic.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_logarithmic 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 173


def assert_performance_logarithmic threshold = 0.99, &work
  assert_performance validation_for_fit(:logarithmic, threshold), &work
end

#assert_performance_power(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered curve fit to match a power curve within a given error threshold.

Fit is calculated by #fit_power.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_power 0.9999 do |x|
    @obj.algorithm
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 213


def assert_performance_power threshold = 0.99, &work
  assert_performance validation_for_fit(:power, threshold), &work
end

#fit_error(xys)

Takes an array of x/y pairs and calculates the general R^2 value.

See: en.wikipedia.org/wiki/Coefficient_of_determination

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 222


def fit_error xys
  y_bar  = sigma(xys) { |_, y| y } / xys.size.to_f
  ss_tot = sigma(xys) { |_, y| (y    - y_bar) ** 2 }
  ss_err = sigma(xys) { |x, y| (yield(x) - y) ** 2 }

  1 - (ss_err / ss_tot)
end

#fit_exponential(xs, ys)

To fit a functional form: y = ae^(bx).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingExponential.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 237


def fit_exponential xs, ys
  n     = xs.size
  xys   = xs.zip(ys)
  sxlny = sigma(xys) { |x, y| x * Math.log(y) }
  slny  = sigma(xys) { |_, y| Math.log(y)     }
  sx2   = sigma(xys) { |x, _| x * x           }
  sx    = sigma xs

  c = n * sx2 - sx ** 2
  a = (slny * sx2 - sx * sxlny) / c
  b = ( n * sxlny - sx * slny ) / c

  return Math.exp(a), b, fit_error(xys) { |x| Math.exp(a + b * x) }
end

#fit_linear(xs, ys)

Fits the functional form: a + bx.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFitting.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 281


def fit_linear xs, ys
  n   = xs.size
  xys = xs.zip(ys)
  sx  = sigma xs
  sy  = sigma ys
  sx2 = sigma(xs)  { |x|   x ** 2 }
  sxy = sigma(xys) { |x, y| x * y  }

  c = n * sx2 - sx**2
  a = (sy * sx2 - sx * sxy) / c
  b = ( n * sxy - sx * sy ) / c

  return a, b, fit_error(xys) { |x| a + b * x }
end

#fit_logarithmic(xs, ys)

To fit a functional form: y = a + b*ln(x).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingLogarithmic.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 259


def fit_logarithmic xs, ys
  n     = xs.size
  xys   = xs.zip(ys)
  slnx2 = sigma(xys) { |x, _| Math.log(x) ** 2 }
  slnx  = sigma(xys) { |x, _| Math.log(x)      }
  sylnx = sigma(xys) { |x, y| y * Math.log(x)  }
  sy    = sigma(xys) { |_, y| y                }

  c = n * slnx2 - slnx ** 2
  b = ( n * sylnx - sy * slnx ) / c
  a = (sy - b * slnx) / n

  return a, b, fit_error(xys) { |x| a + b * Math.log(x) }
end

#fit_power(xs, ys)

To fit a functional form: y = ax^b.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingPowerLaw.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 303


def fit_power xs, ys
  n       = xs.size
  xys     = xs.zip(ys)
  slnxlny = sigma(xys) { |x, y| Math.log(x) * Math.log(y) }
  slnx    = sigma(xs)  { |x   | Math.log(x)               }
  slny    = sigma(ys)  { |   y| Math.log(y)               }
  slnx2   = sigma(xs)  { |x   | Math.log(x) ** 2          }

  b = (n * slnxlny - slnx * slny) / (n * slnx2 - slnx ** 2)
  a = (slny - b * slnx) / n

  return Math.exp(a), b, fit_error(xys) { |x| (Math.exp(a) * (x ** b)) }
end

#sigma(enum, &block)

Enumerates over enum mapping block if given, returning the sum of the result. Eg:

sigma([1, 2, 3])                # => 1 + 2 + 3 => 6
sigma([1, 2, 3]) { |n| n ** 2 } # => 1 + 4 + 9 => 14

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 324


def sigma enum, &block
  enum = enum.map(&block) if block
  enum.inject { |sum, n| sum + n }
end

#validation_for_fit(msg, threshold)

Returns a proc that calls the specified fit method and asserts that the error is within a tolerable threshold.

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 333


def validation_for_fit msg, threshold
  proc do |range, times|
    a, b, rr = send "fit_#{msg}", range, times
    assert_operator rr, :>=, threshold
    [a, b, rr]
  end
end