Class: Minitest::Benchmark

Relationships & Source Files
Extension / Inclusion / Inheritance Descendants
Subclasses: BenchSpec
Super Chains via Extension / Inclusion / Inheritance
Class Chain: self, `Test`, `Guard`, `Runnable`
Instance Chain: self, `Test`, `Guard`, `Test::LifecycleHooks`, `Assertions`, `Reportable`, `Runnable`
Inherits:	Minitest::Test Object Minitest::Runnable Minitest::Test Minitest::Benchmark
Defined in:	lib/minitest/benchmark.rb

Overview

Subclass Benchmark to create your own benchmark runs. Methods starting with “bench_” get executed on a per-class.

See Assertions

Constant Summary

`Runnable` - Inherited

SIGNALS

`Test` - Inherited

PASSTHROUGH_EXCEPTIONS, SETUP_METHODS, TEARDOWN_METHODS

Class Attribute Summary

`Test` - Inherited

.io_lock

Class Method Summary

.bench_exp(min, max, base = 10)

Returns a set of ranges stepped exponentially from min to max by powers of base.
.bench_linear(min, max, step = 10)

Returns a set of ranges stepped linearly from min to max by step.
.bench_range

Specifies the ranges used for benchmarking for that class.
.io Internal use only
.run(reporter, options = {}) Internal use only
.runnable_methods Internal use only

`Test` - Inherited

.i_suck_and_my_tests_are_order_dependent!	Call this at the top of your tests when you absolutely positively need to have ordered tests.
.make_my_diffs_pretty!	Make diffs for this `Test` use `#pretty_inspect` so that diff in assert_equal can have more details.
.parallelize_me!	Call this at the top of your tests (inside the `Test` subclass or `describe` block) when you want to run your tests in parallel.
.runnable_methods	Returns all instance methods starting with “test_”.

`Guard` - Extended

jruby?	Is this running on jruby?
mri?	Is this running on mri?
osx?	Is this running on macOS?
windows?	Is this running on windows?

`Runnable` - Inherited

.filter_runnable_methods	Returns an array of filtered .runnable_methods.
.methods_matching	Returns all instance methods matching the pattern `re`.
.run	Runs a single method and has the reporter record the result.
.run_order	Defines the order to run tests (:random by default).
.run_suite	Responsible for running all runnable methods in a given class, each in its own instance.
.runnable_methods	Each subclass of `Runnable` is responsible for overriding this method to return all runnable methods.
.runnables	Returns all subclasses of `Runnable`.
.inherited, .new, .on_signal, .reset, .with_info_handler

Instance Attribute Summary

`Assertions` - Included

#skipped?

Was this testcase skipped? Meant for #teardown.

`Reportable` - Included

#error?	Did this run error?
#passed?	Did this run pass?
#skipped?	Was this run skipped?

`Runnable` - Inherited

#assertions	Number of assertions executed in this run.
#failures	An assertion raised during the run, if any.
#metadata	Metadata you attach to the test results that get sent to the reporter.
#metadata=	Sets metadata, mainly used for Result.from.
#metadata?	Returns true if metadata exists.
#name	Name of the run.
#name=	Set the name of the run.
#passed?	Did this run pass?
#skipped?	Was this run skipped? See `#passed?` for more information.
#time	The time it took to run.

Instance Method Summary

#assert_performance(validation, &work)

Runs the given work, gathering the times of each run.
#assert_performance_constant(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a constant rate (eg, linear slope == 0) within a given threshold.
#assert_performance_exponential(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a exponential curve within a given error threshold.
#assert_performance_linear(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a straight line within a given error threshold.
#assert_performance_logarithmic(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a logarithmic curve within a given error threshold.
#assert_performance_power(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered curve fit to match a power curve within a given error threshold.
#fit_error(xys)

Takes an array of x/y pairs and calculates the general R^2 value.
#fit_exponential(xs, ys)

To fit a functional form: y = ae^(bx).
#fit_linear(xs, ys)

Fits the functional form: a + bx.
#fit_logarithmic(xs, ys)

To fit a functional form: y = a + b*ln(x).
#fit_power(xs, ys)

To fit a functional form: y = ax^b.
#sigma(enum, &block)

Enumerates over enum mapping block if given, returning the sum of the result.
#validation_for_fit(msg, threshold)

Returns a proc that calls the specified fit method and asserts that the error is within a tolerable threshold.
#io Internal use only

`Test` - Inherited

#run	Runs a single test with setup/teardown hooks.
#capture_exceptions	LifecycleHooks.
#neuter_exception, #new_exception, #sanitize_exception

`Guard` - Included

#jruby?	Is this running on jruby?
#mri?	Is this running on mri?
#osx?	Is this running on macOS?
#windows?	Is this running on windows?

`Test::LifecycleHooks` - Included

#after_setup	Runs before every test, after setup.
#after_teardown	Runs after every test, after teardown.
#before_setup	Runs before every test, before setup.
#before_teardown	Runs after every test, before teardown.
#setup	Runs before every test.
#teardown	Runs after every test.

`Assertions` - Included

#assert	Fails unless `test` is truthy.
#assert_empty	Fails unless `obj` is empty.
#assert_equal	Fails unless `exp == act` printing the difference between the two, if possible.
#assert_in_delta	For comparing Floats.
#assert_in_epsilon	For comparing Floats.
#assert_includes	Fails unless `collection` includes `obj`.
#assert_instance_of	Fails unless `obj` is an instance of `cls`.
#assert_kind_of	Fails unless `obj` is a kind of `cls`.
#assert_match	Fails unless `matcher` `=~` `obj`.
#assert_nil	Fails unless `obj` is nil.
#assert_operator	For testing with binary operators.
#assert_output	Fails if stdout or stderr do not output the expected results.
#assert_path_exists	Fails unless `path` exists.
#assert_pattern	For testing with pattern matching (only supported with Ruby 3.0 and later).
#assert_predicate	For testing with predicates.
#assert_raises	Fails unless the block raises one of `exp`.
#assert_respond_to	Fails unless `obj` responds to `meth`.
#assert_same	Fails unless `exp` and `act` are `#equal?`
#assert_silent	Fails if the block outputs anything to stderr or stdout.
#assert_throws	Fails unless the block throws `sym`
#capture_io	Captures $stdout and $stderr into strings:
#capture_subprocess_io	Captures $stdout and $stderr into strings, using Tempfile to ensure that subprocess IO is captured as well.
#diff	Returns a diff between `exp` and `act`.
#exception_details	Returns details for exception `e`.
#fail_after	Fails after a given date (in the local time zone).
#flunk	Fails with `msg`.
#message	Returns a proc that delays generation of an output message.
#mu_pp	This returns a human-readable version of `obj`.
#mu_pp_for_diff	This returns a diff-able more human-readable version of `obj`.
#pass	used for counting assertions.
#refute	Fails if `test` is truthy.
#refute_empty	Fails if `obj` is empty.
#refute_equal	Fails if `exp == act`.
#refute_in_delta	For comparing Floats.
#refute_in_epsilon	For comparing Floats.
#refute_includes	Fails if `obj` includes `sub`.
#refute_instance_of	Fails if `obj` is an instance of `cls`.
#refute_kind_of	Fails if `obj` is a kind of `cls`.
#refute_match	Fails if `matcher` `=~` `obj`.
#refute_nil	Fails if `obj` is nil.
#refute_operator	Fails if `o1` is not `op` `o2`.
#refute_path_exists	Fails if `path` exists.
#refute_pattern	For testing with pattern matching (only supported with Ruby 3.0 and later).
#refute_predicate	For testing with predicates.
#refute_respond_to	Fails if `obj` responds to the message `meth`.
#refute_same	Fails if `exp` is the same (by object identity) as `act`.
#skip	Skips the current run.
#skip_until	Skips the current run until a given date (in the local time zone).
#things_to_diff	Returns things to diff [expect, butwas], or [nil, nil] if nothing to diff.
#_synchronize, #_where

`Reportable` - Included

#location	The location identifier of this test.
#result_code	Returns “.”, “F”, or “E” based on the result of the run.
#class_name

`Runnable` - Inherited

#result_code	Returns a single character string to print based on the result of the run.
#run	Runs a single method.
#failure, #time_it

Constructor Details

This class inherits a constructor from Minitest::Runnable

Class Method Details

.bench_exp(min, max, base = 10)

Returns a set of ranges stepped exponentially from min to max by powers of base. Eg:

bench_exp(2, 16, 2) # => [2, 4, 8, 16]

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 35


def self.bench_exp min, max, base = 10
  min = (Math.log10(min) / Math.log10(base)).to_i
  max = (Math.log10(max) / Math.log10(base)).to_i

  (min..max).map { |m| base ** m }.to_a
end

.bench_linear(min, max, step = 10)

Returns a set of ranges stepped linearly from min to max by step. Eg:

bench_linear(20, 40, 10) # => [20, 30, 40]

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 48


def self.bench_linear min, max, step = 10
  (min..max).step(step).to_a
end

.bench_range

Specifies the ranges used for benchmarking for that class. Defaults to exponential growth from 1 to 10k by powers of 10. Override if you need different ranges for your benchmarks.

.io

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 12


def self.io # :nodoc:
  @io
end

.run(reporter, options = {})

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 20


def self.run reporter, options = {} # :nodoc:
  @io = reporter.io
  super
end

.runnable_methods

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 25


def self.runnable_methods # :nodoc:
  methods_matching(/^bench_/)
end

Instance Method Details

#assert_performance(validation, &work)

Runs the given work, gathering the times of each run. Range and times are then passed to a given validation proc. Outputs the benchmark name and times in tab-separated format, making it easy to paste into a spreadsheet for graphing or further analysis.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  validation = proc { |x, y| ... }
  assert_performance validation do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 81


def assert_performance validation, &work
  range = self.class.bench_range

  io.print self.name

  times = []

  range.each do |x|
    GC.start
    t0 = Minitest.clock_time
    instance_exec(x, &work)
    t = Minitest.clock_time - t0

    io.print "\t%9.6f" % t
    times << t
  end
  io.puts

  validation[range, times]
end

#assert_performance_constant(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a constant rate (eg, linear slope == 0) within a given threshold. Note: because we’re testing for a slope of 0, R^2 is not a good determining factor for the fit, so the threshold is applied against the slope itself. As such, you probably want to tighten it from the default.

See www.graphpad.com/guides/prism/8/curve-fitting/reg_intepretingnonlinr2.htm for more details.

Fit is calculated by #fit_linear.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_constant 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 125


def assert_performance_constant threshold = 0.99, &work
  validation = proc do |range, times|
    a, b, rr = fit_linear range, times
    assert_in_delta 0, b, 1 - threshold
    [a, b, rr]
  end

  assert_performance validation, &work
end

#assert_performance_exponential(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a exponential curve within a given error threshold.

Fit is calculated by #fit_exponential.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_exponential 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 151


def assert_performance_exponential threshold = 0.99, &work
  assert_performance validation_for_fit(:exponential, threshold), &work
end

#assert_performance_linear(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a straight line within a given error threshold.

Fit is calculated by #fit_linear.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_linear 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 191


def assert_performance_linear threshold = 0.99, &work
  assert_performance validation_for_fit(:linear, threshold), &work
end

#assert_performance_logarithmic(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a logarithmic curve within a given error threshold.

Fit is calculated by #fit_logarithmic.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_logarithmic 0.9999 do |n|
    @obj.algorithm(n)
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 171


def assert_performance_logarithmic threshold = 0.99, &work
  assert_performance validation_for_fit(:logarithmic, threshold), &work
end

#assert_performance_power(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered curve fit to match a power curve within a given error threshold.

Fit is calculated by #fit_power.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_power 0.9999 do |x|
    @obj.algorithm
  end
end

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 211


def assert_performance_power threshold = 0.99, &work
  assert_performance validation_for_fit(:power, threshold), &work
end

#fit_error(xys)

Takes an array of x/y pairs and calculates the general R^2 value.

See: en.wikipedia.org/wiki/Coefficient_of_determination

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 220


def fit_error xys
  y_bar  = sigma(xys) { |_, y| y } / xys.size.to_f
  ss_tot = sigma(xys) { |_, y| (y    - y_bar) ** 2 }
  ss_err = sigma(xys) { |x, y| (yield(x) - y) ** 2 }

  1 - (ss_err / ss_tot)
end

#fit_exponential(xs, ys)

To fit a functional form: y = ae^(bx).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingExponential.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 235


def fit_exponential xs, ys
  n     = xs.size
  xys   = xs.zip ys
  sxlny = sigma(xys) { |x, y| x * Math.log(y) }
  slny  = sigma(xys) { |_, y| Math.log(y)     }
  sx2   = sigma(xys) { |x, _| x * x           }
  sx    = sigma xs

  c = n * sx2 - sx ** 2
  a = (slny * sx2 - sx * sxlny) / c
  b = ( n * sxlny - sx * slny ) / c

  return Math.exp(a), b, fit_error(xys) { |x| Math.exp(a + b * x) }
end

#fit_linear(xs, ys)

Fits the functional form: a + bx.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFitting.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 279


def fit_linear xs, ys
  n   = xs.size
  xys = xs.zip ys
  sx  = sigma xs
  sy  = sigma ys
  sx2 = sigma(xs)  { |x|   x ** 2 }
  sxy = sigma(xys) { |x, y| x * y  }

  c = n * sx2 - sx**2
  a = (sy * sx2 - sx * sxy) / c
  b = ( n * sxy - sx * sy ) / c

  return a, b, fit_error(xys) { |x| a + b * x }
end

#fit_logarithmic(xs, ys)

To fit a functional form: y = a + b*ln(x).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingLogarithmic.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 257


def fit_logarithmic xs, ys
  n     = xs.size
  xys   = xs.zip ys
  slnx2 = sigma(xys) { |x, _| Math.log(x) ** 2 }
  slnx  = sigma(xys) { |x, _| Math.log(x)      }
  sylnx = sigma(xys) { |x, y| y * Math.log(x)  }
  sy    = sigma(xys) { |_, y| y                }

  c = n * slnx2 - slnx ** 2
  b = ( n * sylnx - sy * slnx ) / c
  a = (sy - b * slnx) / n

  return a, b, fit_error(xys) { |x| a + b * Math.log(x) }
end

#fit_power(xs, ys)

To fit a functional form: y = ax^b.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingPowerLaw.html

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 301


def fit_power xs, ys
  n       = xs.size
  xys     = xs.zip ys
  slnxlny = sigma(xys) { |x, y| Math.log(x) * Math.log(y) }
  slnx    = sigma(xs)  { |x   | Math.log(x)               }
  slny    = sigma(ys)  { |   y| Math.log(y)               }
  slnx2   = sigma(xs)  { |x   | Math.log(x) ** 2          }

  b = (n * slnxlny - slnx * slny) / (n * slnx2 - slnx ** 2)
  a = (slny - b * slnx) / n

  return Math.exp(a), b, fit_error(xys) { |x| (Math.exp(a) * (x ** b)) }
end

#io

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 16


def io # :nodoc:
  self.class.io
end

#sigma(enum, &block)

Enumerates over enum mapping block if given, returning the sum of the result. Eg:

sigma([1, 2, 3])                # => 1 + 2 + 3 => 6
sigma([1, 2, 3]) { |n| n ** 2 } # => 1 + 4 + 9 => 14

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 322


def sigma enum, &block
  enum = enum.map(&block) if block
  enum.sum
end

#validation_for_fit(msg, threshold)

Returns a proc that calls the specified fit method and asserts that the error is within a tolerable threshold.

[ GitHub ]

# File 'lib/minitest/benchmark.rb', line 331


def validation_for_fit msg, threshold
  proc do |range, times|
    a, b, rr = send "fit_#{msg}", range, times
    assert_operator rr, :>=, threshold
    [a, b, rr]
  end
end

Class: Minitest::Benchmark

Overview

Constant Summary

Runnable - Inherited

Reportable - Included

Assertions - Included

Test - Inherited

Class Attribute Summary

Test - Inherited

Class Method Summary

Test - Inherited

Guard - Extended

Runnable - Inherited

Instance Attribute Summary

Assertions - Included

Reportable - Included

Runnable - Inherited

Instance Method Summary

Test - Inherited

Guard - Included

Test::LifecycleHooks - Included

Assertions - Included

Reportable - Included

Runnable - Inherited

Constructor Details

Class Method Details

.bench_exp(min, max, base = 10)

.bench_linear(min, max, step = 10)

.bench_range

.io

.run(reporter, options = {})

.runnable_methods

Instance Method Details

#assert_performance(validation, &work)

#assert_performance_constant(threshold = 0.99, &work)

#assert_performance_exponential(threshold = 0.99, &work)

#assert_performance_linear(threshold = 0.99, &work)

#assert_performance_logarithmic(threshold = 0.99, &work)

#assert_performance_power(threshold = 0.99, &work)

#fit_error(xys)

#fit_exponential(xs, ys)

#fit_linear(xs, ys)

#fit_logarithmic(xs, ys)

#fit_power(xs, ys)

#io

#sigma(enum, &block)

#validation_for_fit(msg, threshold)

`Runnable` - Inherited

`Reportable` - Included

`Assertions` - Included

`Test` - Inherited

`Test` - Inherited

`Test` - Inherited

`Guard` - Extended

`Runnable` - Inherited

`Assertions` - Included

`Reportable` - Included

`Runnable` - Inherited

`Test` - Inherited

`Guard` - Included

`Test::LifecycleHooks` - Included

`Assertions` - Included

`Reportable` - Included

`Runnable` - Inherited