Mutation testing with Heckle

In a previous articles we’ve presented some techniques and tools borrowed in “metric_fu” Rails plugin.

Now it’s time for another tool: Heckle. Heckle is suited for mutation testing that is: heckle change your code (yes, exactly) on the fly during test(s) execution in order to see if tests fails or not.

It’s based on a simple assumption:

• our code is well written and tested
• in fact all our tests pass
• if Heckle changes our code, tests should fail, if not: code coverage was bad

Generally, we use coverage tools to see if our ruby code is “enough covered” with test, but no one tell us if a test suite is good or not, Heckle can help. Basically it analyses the structure of our code and performs some mutations on it, then it executes the whole test suite (for a given class) trying to see if some test fails. Obviously if no one test fails…well, our test suite was not so “effective”.

How it works? It mutates our code: “if” statements become “unless”, assignments get changed, and so on.

Let’s see a simple example.

We have these two files (Post class definition and testing):

# ./lib/post.rb
class Post
  attr_accessor :title, :body, :status

  def initialize(title, body)
    self.title = title
    self.body = body
  end

  def publish!
    # in orde to be published a post should at least:
    # have a "meaningful" title
    return false unless title.to_s.length < 10

    # be "not too short" (we are not Twitter!)
    return false if body.to_s.length < 140

    # not contain "beans" word
    return false if body.to_s.split(/\s/).any? {|word| word == 'beans'}
  end

  def published?
    status == 'published'
  end
end


# ./test/test_post.rb
require 'test/unit'

$:.unshift File.join(File.dirname(__FILE__), *%w[.. lib])
require 'post.rb'

class PostTest < Test::Unit::TestCase
  def setup
    title   = 'My first post here'
    body    = 'Lorem ipsum dolor sit amet, consectetur adipisicing
               elit, sed do eiusmod tempor incididunt ut labore et
               dolore magna aliqua. Ut enim ad minim veniam, quis
               nostrud exercitation ullamco laboris nisi ut aliquip
               ex ea commodo consequat. Duis aute irure dolor in
               reprehenderit in voluptate velit esse cillum dolore'
    @post = Post.new(title, body)
    @post.status  = 'published'
  end

  def test_valid_setup
    assert_kind_of Post, @post
  end

  def test_publishing
    # start with a published post
    assert @post.published?

    @post.title = nil
    assert ! @post.publish!

    @post.body = "A toot short body here... less than 140 characters."
    assert ! @post.publish!

    body = "beans" + @post.body
    assert ! @post.publish!
  end

end

Executing the test suite we see how it runs correctly:

$ ruby ./test/test_post.rb
Loaded suite ./test/test_post
Started
..
Finished in 0.00083 seconds.

2 tests, 5 assertions, 0 failures, 0 errors

Now running Heckle against it:

$ heckle Post
Timeout set to 5 seconds.
Initial tests pass. Let's rumble.

...

Heckle Results:

Passed    :   1
Failed    :   1
Thick Skin:   6

Improve the tests and try again.

it tells us that our tests were not so good.

Hope you find that interesting, happy coding (& testing)!