123456789_123456789_123456789_123456789_123456789_

Class: Minitest::Benchmark

Relationships & Source Files
Extension / Inclusion / Inheritance Descendants
Subclasses:
Super Chains via Extension / Inclusion / Inheritance
Class Chain:
self, Test, Guard, Runnable
Instance Chain:
Inherits: Minitest::Test
Defined in: lib/minitest/benchmark.rb

Overview

Subclass Benchmark to create your own benchmark runs. Methods starting with “bench_” get executed on a per-class.

See Assertions

Constant Summary

Runnable - Inherited

SIGNALS

Assertions - Included

E, UNDEFINED

Test - Inherited

PASSTHROUGH_EXCEPTIONS, TEARDOWN_METHODS

Class Attribute Summary

Test - Inherited

Class Method Summary

Test - Inherited

.i_suck_and_my_tests_are_order_dependent!

Call this at the top of your tests when you absolutely positively need to have ordered tests.

.make_my_diffs_pretty!

Make diffs for this Test use #pretty_inspect so that diff in assert_equal can have more details.

.parallelize_me!

Call this at the top of your tests when you want to run your tests in parallel.

.runnable_methods

Returns all instance methods starting with “test_”.

.test_order

Defines the order to run tests (:random by default).

Guard - Extended

jruby?

Is this running on jruby?

maglev?

Is this running on maglev?

mri?

Is this running on mri?

osx?

Is this running on macOS?

rubinius?

Is this running on rubinius?

windows?

Is this running on windows?

Runnable - Inherited

.methods_matching

Returns all instance methods matching the pattern re.

.run

Responsible for running all runnable methods in a given class, each in its own instance.

.run_one_method

Runs a single method and has the reporter record the result.

.runnable_methods

Each subclass of Runnable is responsible for overriding this method to return all runnable methods.

.runnables

Returns all subclasses of Runnable.

.inherited, .new, .on_signal, .reset, .with_info_handler

Instance Attribute Summary

Reportable - Included

#error?

Did this run error?

#passed?

Did this run pass?

#skipped?

Was this run skipped?

Assertions - Included

#skipped?

Was this testcase skipped? Meant for #teardown.

Runnable - Inherited

#assertions

Number of assertions executed in this run.

#failures

An assertion raised during the run, if any.

#name

Name of the run.

#name=

Set the name of the run.

#passed?

Did this run pass?

#skipped?

Was this run skipped? See #passed? for more information.

#time

The time it took to run.

Instance Method Summary

Test - Inherited

#neuter_exception, #new_exception,
#run

Runs a single test with setup/teardown hooks.

#capture_exceptions

LifecycleHooks.

#class_name, #sanitize_exception, #with_info_handler

Guard - Included

#jruby?

Is this running on jruby?

#maglev?

Is this running on maglev?

#mri?

Is this running on mri?

#osx?

Is this running on macOS?

#rubinius?

Is this running on rubinius?

#windows?

Is this running on windows?

Test::LifecycleHooks - Included

#after_setup

Runs before every test, after setup.

#after_teardown

Runs after every test, after teardown.

#before_setup

Runs before every test, before setup.

#before_teardown

Runs after every test, before teardown.

#setup

Runs before every test.

#teardown

Runs after every test.

Reportable - Included

#location

The location identifier of this test.

#result_code

Returns “.”, “F”, or “E” based on the result of the run.

#class_name

Assertions - Included

#assert

Fails unless test is truthy.

#assert_empty

Fails unless obj is empty.

#assert_equal

Fails unless exp == act printing the difference between the two, if possible.

#assert_in_delta

For comparing Floats.

#assert_in_epsilon

For comparing Floats.

#assert_includes

Fails unless collection includes obj.

#assert_instance_of

Fails unless obj is an instance of cls.

#assert_kind_of

Fails unless obj is a kind of cls.

#assert_match

Fails unless matcher =~ obj.

#assert_mock

Assert that the mock verifies correctly.

#assert_nil

Fails unless obj is nil.

#assert_operator

For testing with binary operators.

#assert_output

Fails if stdout or stderr do not output the expected results.

#assert_path_exists

Fails unless path exists.

#assert_predicate

For testing with predicates.

#assert_raises

Fails unless the block raises one of exp.

#assert_respond_to

Fails unless obj responds to meth.

#assert_same

Fails unless exp and act are #equal?

#assert_send

send_ary is a receiver, message and arguments.

#assert_silent

Fails if the block outputs anything to stderr or stdout.

#assert_throws

Fails unless the block throws sym

#capture_io

Captures $stdout and $stderr into strings:

#capture_subprocess_io

Captures $stdout and $stderr into strings, using Tempfile to ensure that subprocess IO is captured as well.

#diff

Returns a diff between exp and act.

#exception_details

Returns details for exception e.

#fail_after

Fails after a given date (in the local time zone).

#flunk

Fails with msg.

#message

Returns a proc that will output msg along with the default message.

#mu_pp

This returns a human-readable version of obj.

#mu_pp_for_diff

This returns a diff-able more human-readable version of obj.

#pass

used for counting assertions.

#refute

Fails if test is truthy.

#refute_empty

Fails if obj is empty.

#refute_equal

Fails if exp == act.

#refute_in_delta

For comparing Floats.

#refute_in_epsilon

For comparing Floats.

#refute_includes

Fails if collection includes obj.

#refute_instance_of

Fails if obj is an instance of cls.

#refute_kind_of

Fails if obj is a kind of cls.

#refute_match

Fails if matcher =~ obj.

#refute_nil

Fails if obj is nil.

#refute_operator

Fails if o1 is not op o2.

#refute_path_exists

Fails if path exists.

#refute_predicate

For testing with predicates.

#refute_respond_to

Fails if obj responds to the message meth.

#refute_same

Fails if exp is the same (by object identity) as act.

#skip

Skips the current run.

#skip_until

Skips the current run until a given date (in the local time zone).

#things_to_diff

Returns things to diff [expect, butwas], or [nil, nil] if nothing to diff.

#_synchronize

Runnable - Inherited

#result_code

Returns a single character string to print based on the result of the run.

#run

Runs a single method.

#failure, #marshal_dump, #marshal_load, #time_it

Constructor Details

This class inherits a constructor from Minitest::Runnable

Class Method Details

.bench_exp(min, max, base = 10)

Returns a set of ranges stepped exponentially from min to max by powers of base. Eg:

bench_exp(2, 16, 2) # => [2, 4, 8, 16]
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 35

def self.bench_exp min, max, base = 10
  min = (Math.log10(min) / Math.log10(base)).to_i
  max = (Math.log10(max) / Math.log10(base)).to_i

  (min..max).map { |m| base ** m }.to_a
end

.bench_linear(min, max, step = 10)

Returns a set of ranges stepped linearly from min to max by step. Eg:

bench_linear(20, 40, 10) # => [20, 30, 40]
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 48

def self.bench_linear min, max, step = 10
  (min..max).step(step).to_a
rescue LocalJumpError # 1.8.6
  r = []; (min..max).step(step) { |n| r << n }; r
end

.bench_range

Specifies the ranges used for benchmarking for that class. Defaults to exponential growth from 1 to 10k by powers of 10. Override if you need different ranges for your benchmarks.

See also: .bench_exp and .bench_linear.

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 61

def self.bench_range
  bench_exp 1, 10_000
end

.io

This method is for internal use only.
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 12

def self.io # :nodoc:
  @io
end

.run(reporter, options = {})

This method is for internal use only.
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 20

def self.run reporter, options = {} # :nodoc:
  @io = reporter.io
  super
end

.runnable_methods

This method is for internal use only.
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 25

def self.runnable_methods # :nodoc:
  methods_matching(/^bench_/)
end

Instance Method Details

#assert_performance(validation, &work)

Runs the given work, gathering the times of each run. Range and times are then passed to a given validation proc. Outputs the benchmark name and times in tab-separated format, making it easy to paste into a spreadsheet for graphing or further analysis.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  validation = proc { |x, y| ... }
  assert_performance validation do |n|
    @obj.algorithm(n)
  end
end
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 83

def assert_performance validation, &work
  range = self.class.bench_range

  io.print "#{self.name}"

  times = []

  range.each do |x|
    GC.start
    t0 = Minitest.clock_time
    instance_exec(x, &work)
    t = Minitest.clock_time - t0

    io.print "\t%9.6f" % t
    times << t
  end
  io.puts

  validation[range, times]
end

#assert_performance_constant(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a constant rate (eg, linear slope == 0) within a given threshold. Note: because we’re testing for a slope of 0, R^2 is not a good determining factor for the fit, so the threshold is applied against the slope itself. As such, you probably want to tighten it from the default.

See www.graphpad.com/guides/prism/8/curve-fitting/reg_intepretingnonlinr2.htm for more details.

Fit is calculated by #fit_linear.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_constant 0.9999 do |n|
    @obj.algorithm(n)
  end
end
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 127

def assert_performance_constant threshold = 0.99, &work
  validation = proc do |range, times|
    a, b, rr = fit_linear range, times
    assert_in_delta 0, b, 1 - threshold
    [a, b, rr]
  end

  assert_performance validation, &work
end

#assert_performance_exponential(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a exponential curve within a given error threshold.

Fit is calculated by #fit_exponential.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_exponential 0.9999 do |n|
    @obj.algorithm(n)
  end
end
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 153

def assert_performance_exponential threshold = 0.99, &work
  assert_performance validation_for_fit(:exponential, threshold), &work
end

#assert_performance_linear(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a straight line within a given error threshold.

Fit is calculated by #fit_linear.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_linear 0.9999 do |n|
    @obj.algorithm(n)
  end
end
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 193

def assert_performance_linear threshold = 0.99, &work
  assert_performance validation_for_fit(:linear, threshold), &work
end

#assert_performance_logarithmic(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered fit to match a logarithmic curve within a given error threshold.

Fit is calculated by #fit_logarithmic.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_logarithmic 0.9999 do |n|
    @obj.algorithm(n)
  end
end
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 173

def assert_performance_logarithmic threshold = 0.99, &work
  assert_performance validation_for_fit(:logarithmic, threshold), &work
end

#assert_performance_power(threshold = 0.99, &work)

Runs the given work and asserts that the times gathered curve fit to match a power curve within a given error threshold.

Fit is calculated by #fit_power.

Ranges are specified by .bench_range.

Eg:

def bench_algorithm
  assert_performance_power 0.9999 do |x|
    @obj.algorithm
  end
end
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 213

def assert_performance_power threshold = 0.99, &work
  assert_performance validation_for_fit(:power, threshold), &work
end

#fit_error(xys)

Takes an array of x/y pairs and calculates the general R^2 value.

See: en.wikipedia.org/wiki/Coefficient_of_determination

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 222

def fit_error xys
  y_bar  = sigma(xys) { |_, y| y } / xys.size.to_f
  ss_tot = sigma(xys) { |_, y| (y    - y_bar) ** 2 }
  ss_err = sigma(xys) { |x, y| (yield(x) - y) ** 2 }

  1 - (ss_err / ss_tot)
end

#fit_exponential(xs, ys)

To fit a functional form: y = ae^(bx).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingExponential.html

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 237

def fit_exponential xs, ys
  n     = xs.size
  xys   = xs.zip(ys)
  sxlny = sigma(xys) { |x, y| x * Math.log(y) }
  slny  = sigma(xys) { |_, y| Math.log(y)     }
  sx2   = sigma(xys) { |x, _| x * x           }
  sx    = sigma xs

  c = n * sx2 - sx ** 2
  a = (slny * sx2 - sx * sxlny) / c
  b = ( n * sxlny - sx * slny ) / c

  return Math.exp(a), b, fit_error(xys) { |x| Math.exp(a + b * x) }
end

#fit_linear(xs, ys)

Fits the functional form: a + bx.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFitting.html

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 281

def fit_linear xs, ys
  n   = xs.size
  xys = xs.zip(ys)
  sx  = sigma xs
  sy  = sigma ys
  sx2 = sigma(xs)  { |x|   x ** 2 }
  sxy = sigma(xys) { |x, y| x * y  }

  c = n * sx2 - sx**2
  a = (sy * sx2 - sx * sxy) / c
  b = ( n * sxy - sx * sy ) / c

  return a, b, fit_error(xys) { |x| a + b * x }
end

#fit_logarithmic(xs, ys)

To fit a functional form: y = a + b*ln(x).

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingLogarithmic.html

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 259

def fit_logarithmic xs, ys
  n     = xs.size
  xys   = xs.zip(ys)
  slnx2 = sigma(xys) { |x, _| Math.log(x) ** 2 }
  slnx  = sigma(xys) { |x, _| Math.log(x)      }
  sylnx = sigma(xys) { |x, y| y * Math.log(x)  }
  sy    = sigma(xys) { |_, y| y                }

  c = n * slnx2 - slnx ** 2
  b = ( n * sylnx - sy * slnx ) / c
  a = (sy - b * slnx) / n

  return a, b, fit_error(xys) { |x| a + b * Math.log(x) }
end

#fit_power(xs, ys)

To fit a functional form: y = ax^b.

Takes x and y values and returns [a, b, r^2].

See: mathworld.wolfram.com/LeastSquaresFittingPowerLaw.html

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 303

def fit_power xs, ys
  n       = xs.size
  xys     = xs.zip(ys)
  slnxlny = sigma(xys) { |x, y| Math.log(x) * Math.log(y) }
  slnx    = sigma(xs)  { |x   | Math.log(x)               }
  slny    = sigma(ys)  { |   y| Math.log(y)               }
  slnx2   = sigma(xs)  { |x   | Math.log(x) ** 2          }

  b = (n * slnxlny - slnx * slny) / (n * slnx2 - slnx ** 2)
  a = (slny - b * slnx) / n

  return Math.exp(a), b, fit_error(xys) { |x| (Math.exp(a) * (x ** b)) }
end

#io

This method is for internal use only.
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 16

def io # :nodoc:
  self.class.io
end

#sigma(enum, &block)

Enumerates over enum mapping block if given, returning the sum of the result. Eg:

sigma([1, 2, 3])                # => 1 + 2 + 3 => 6
sigma([1, 2, 3]) { |n| n ** 2 } # => 1 + 4 + 9 => 14
[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 324

def sigma enum, &block
  enum = enum.map(&block) if block
  enum.inject { |sum, n| sum + n }
end

#validation_for_fit(msg, threshold)

Returns a proc that calls the specified fit method and asserts that the error is within a tolerable threshold.

[ GitHub ]

  
# File 'lib/minitest/benchmark.rb', line 333

def validation_for_fit msg, threshold
  proc do |range, times|
    a, b, rr = send "fit_#{msg}", range, times
    assert_operator rr, :>=, threshold
    [a, b, rr]
  end
end