Class: Minitest::Bisect

Relationships & Source Files
Namespace Children
Classes: `PathExpander`
Inherits:	Object
Defined in:	lib/minitest/bisect.rb

Overview

Bisect helps you isolate and debug random test failures.

Constant Summary

RUBY =

Borrowed from rake

# File 'lib/minitest/bisect.rb', line 54

ENV['RUBY'] ||
    File.join(RbConfig::CONFIG['bindir'],
              RbConfig::CONFIG['ruby_install_name'] +
RbConfig::CONFIG['EXEEXT']).sub(/.*\s.*/m, '"\&"')

SHH = Internal use only
# File 'lib/minitest/bisect.rb', line 47

case # :nodoc:
when mtbv == 1 then " > /dev/null"
when mtbv >= 2 then nil
else " > /dev/null 2>&1"
end

VERSION = Internal use only
# File 'lib/minitest/bisect.rb', line 13
```
"1.8.0"
```

Class Method Summary

.new ⇒ Bisect constructor

Instantiate a new Bisect.
.run(files)

Top-level runner.

Instance Attribute Summary

#culprits rw

An array of tests seen so far.
#failures rw

Failures seen in this run.
#tainted? rw

Alias for #tainted.
#seen_bad rw Internal use only

Instance Method Summary

#bisect_methods(files, rb_flags, mt_flags)

Normal: find “what is the minimal combination of tests to run to.
#reset

Reset per-bisect-run variables.
#run(args)

Instance-level runner.
#tainted (also: #tainted?) rw

True if this run has seen a failure.
#build_files_cmd(culprits, rb, mt) Internal use only
#build_methods_cmd(cmd, culprits = [], bad = nil) Internal use only
#build_re(bad) Internal use only
#map_failures Internal use only
#minitest_result(file, klass, method, fails, assertions, time) Internal use only
#minitest_start Internal use only

Server Methods:
#re_escape(str) Internal use only
#time_it(prompt, cmd) Internal use only

Constructor Details

.new ⇒ `Bisect`

Instantiate a new Bisect.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 93


def initialize
  self.culprits = []
  self.failures = Hash.new { |h, k| h[k] = Hash.new { |h2, k2| h2[k2] = [] } }
end

Class Method Details

.run(files)

Top-level runner. Instantiate and call run, handling exceptions.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 82


def self.run files
  new.run files
rescue => e
  warn e.message
  warn "Try running with MTB_VERBOSE=2 to verify."
  exit 1
end

Instance Attribute Details

#culprits (rw)

An array of tests seen so far. NOT cleared by #reset.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 75


attr_accessor :culprits

#failures (rw)

Failures seen in this run. Shape:

{"file.rb"=>{"Class"=>["test_method1", "test_method2"] ...} ...}

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 70


attr_accessor :failures

#seen_bad (rw)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 77


attr_accessor :seen_bad # :nodoc:

#tainted? (rw)

Alias for #tainted.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 63


alias :tainted? :tainted

Instance Method Details

#bisect_methods(files, rb_flags, mt_flags)

Normal: find “what is the minimal combination of tests to run to

make X fail?"

Run with: minitest_bisect … –seed=N

Verify the failure running normally with the seed.
1. If no failure, punt.
2. If no passing tests before failure, punt. (No culprits == no debug)
Verify the failure doesn’t fail in isolation.
1. If it still fails by itself, warn that it might not be an ordering issue.
Cull all tests after the failure, they’re not involved.
Bisect the culprits + bad until you find a minimal combo that fails.
Display minimal combo by running one last time.

Inverted: find “what is the minimal combination of tests to run to

make this test pass?"

Run with: minitest_bisect … –seed=N -n=“/failing_test_name_regexp/”

Verify the failure by running normally w/ the seed and -n=/…/
1. If no failure, punt.
Verify the passing case by running everything.
1. If failure, punt. This is not a false positive.
Cull all tests after the bad test from #1, they’re not involved.
Bisect the culprits + bad until you find a minimal combo that passes.
Display minimal combo by running one last time.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 163


def bisect_methods files, rb_flags, mt_flags
  bad_names, mt_flags = mt_flags.partition { |s| s =~ /^(?:-n|--name)/ }
  normal   = bad_names.empty?
  inverted = !normal

  if inverted then
    time_it "reproducing w/ scoped failure (inverted run!)...", build_methods_cmd(build_files_cmd(files, rb_flags, mt_flags + bad_names))
    raise "No failures. Probably not a false positive. Aborting." if failures.empty?
    bad = map_failures
  end

  cmd = build_files_cmd(files, rb_flags, mt_flags)

  msg = normal ? "reproducing..." : "reproducing false positive..."
  time_it msg, build_methods_cmd(cmd)

  if normal then
    raise "Reproduction run passed? Aborting." unless tainted?
    raise "Verification failed. No culprits? Aborting." if culprits.empty? && seen_bad
  else
    raise "Reproduction failed? Not false positive. Aborting." if tainted?
    raise "Verification failed. No culprits? Aborting." if culprits.empty? || seen_bad
  end

  if normal then
    bad = map_failures

    time_it "verifying...", build_methods_cmd(cmd, [], bad)

    new_bad = map_failures

    if bad == new_bad then
      warn "Tests fail by themselves. This may not be an ordering issue."
    end
  end

  idx = culprits.index bad.first
  self.culprits = culprits.take idx+1 if idx # cull tests after bad

  # culprits populated by initial reproduction via minitest/server
  found, count = culprits.find_minimal_combination_and_count do |test|
    prompt = "# of culprit methods: #{test.size}"

    time_it prompt, build_methods_cmd(cmd, test, bad)

    normal == tainted? # either normal and failed, or inverse and passed
  end

  puts
  puts "Minimal methods found in #{count} steps:"
  puts
  puts "Culprit methods: %p" % [found + bad]
  puts
  cmd = build_methods_cmd cmd, found, bad
  puts cmd.sub(/--server \d+/, "")
  puts
  cmd
end

#build_files_cmd(culprits, rb, mt)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 237


def build_files_cmd culprits, rb, mt # :nodoc:
  tests = culprits.flatten.compact.map { |f| %(require "./#{f}") }.join " ; "

  %(#{RUBY} #{rb.shelljoin} -e '#{tests}' -- #{mt.map(&:to_s).shelljoin})
end

#build_methods_cmd(cmd, culprits = [], bad = nil)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 243


def build_methods_cmd cmd, culprits = [], bad = nil # :nodoc:
  reset

  if bad then
    re = build_re culprits + bad

    cmd += " -n \"#{re}\"" if bad
  end

  if ENV["MTB_VERBOSE"].to_i >= 1 then
    puts
    puts cmd
    puts
  end

  cmd
end

#build_re(bad)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 261


def build_re bad # :nodoc:
  re = []

  # bad by class, you perv
  bbc = bad.map { |s| s.split(/#/, 2) }.group_by(&:first)

  bbc.each do |klass, methods|
    methods = methods.map(&:last).flatten.uniq.map { |method|
      re_escape method
    }

    methods = methods.join "|"
    re << /#{re_escape klass}#(?:#{methods})/.to_s[7..-2] # (?-mix:...)
  end

  re = re.join("|").to_s.gsub(/-mix/, "")

  "/^(?:#{re})$/"
end

#map_failures

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 229


def map_failures # :nodoc:
  # from: {"file.rb"=>{"Class"=>["test_method1", "test_method2"]}}
  #   to: ["Class#test_method1", "Class#test_method2"]
  failures.values.map { |h|
    h.map { |k,vs| vs.map { |v| "#{k}##{v}" } }
  }.flatten.sort
end

#minitest_result(file, klass, method, fails, assertions, time)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 292


def minitest_result file, klass, method, fails, assertions, time # :nodoc:
  fails.reject! { |fail| Minitest::Skip === fail }

  if fails.empty? then
    culprits << "#{klass}##{method}" unless seen_bad # UGH
  else
    self.seen_bad = true
  end

  return if fails.empty?

  self.tainted = true
  self.failures[file][klass] << method
end

#minitest_start

This method is for internal use only.

Server Methods:

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 288


def minitest_start # :nodoc:
  self.failures.clear
end

#re_escape(str)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 281


def re_escape str # :nodoc:
  str.gsub(/([`'"!?&\[\]\(\)\{\}\|\+])/, '\\\\\1')
end

#reset

Reset per-bisect-run variables.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 101


def reset
  self.seen_bad = false
  self.tainted  = false
  failures.clear
  # not clearing culprits on purpose
end

#run(args)

Instance-level runner. Handles Server, argument processing, and invoking #bisect_methods.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 112


def run args
  Minitest::Server.run self

  cmd = nil

  mt_flags = args.dup
  expander = Minitest::Bisect::PathExpander.new mt_flags

  files = expander.process.to_a
  rb_flags = expander.rb_flags
  mt_flags += ["--server", $$.to_s]

  cmd = bisect_methods files, rb_flags, mt_flags

  puts "Final reproduction:"
  puts

  system cmd.sub(/--server \d+/, "")
ensure
  Minitest::Server.stop
end

#tainted (rw) Also known as: #tainted?

True if this run has seen a failure.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 62


attr_accessor :tainted

#time_it(prompt, cmd)

This method is for internal use only.

[ GitHub ]

# File 'lib/minitest/bisect.rb', line 222


def time_it prompt, cmd # :nodoc:
  print prompt
  t0 = Time.now
  system "#{cmd} #{SHH}"
  puts " in %.2f sec" % (Time.now - t0)
end

Class: Minitest::Bisect

Overview

Constant Summary

Class Method Summary

Instance Attribute Summary

Instance Method Summary

Constructor Details

.new ⇒ Bisect

Class Method Details

.run(files)

Instance Attribute Details

#culprits (rw)

#failures (rw)

#seen_bad (rw)

#tainted? (rw)

Instance Method Details

#bisect_methods(files, rb_flags, mt_flags)

#build_files_cmd(culprits, rb, mt)

#build_methods_cmd(cmd, culprits = [], bad = nil)

#build_re(bad)

#map_failures

#minitest_result(file, klass, method, fails, assertions, time)

#minitest_start

#re_escape(str)

#reset

#run(args)

#tainted (rw) Also known as: #tainted?

#time_it(prompt, cmd)

.new ⇒ `Bisect`