David Chelimsky

random thoughtlessness

Story Runner in Plain English

Houston, we have Plain Text!

I just committed a first stab at a Plain Text Story Runner. It’s in RSpec’s trunk and will be (in some form) part of the next release.

Big thanks to Pat Maddox for the StoryPartFactory (which is now called StoryMediator) and to all on the rspec-users list who contributed their ideas and thoughts to the discussion about plain text stories.

Keep in mind that this is brand new and very experimental. I do not recommend that you start converting all your projects to using this.

That said …

A bit of background

[Update: modified to use And for multiple Givens, Whens or Thens]

The initial implementation of Story Runner supported syntax like this (slightly modified from Dan North’s article introducing rbehave):

Story "transfer to cash account",
%(As a savings account holder
I want to transfer money from my savings account
So that I can get cash easily from an ATM) do

Scenario "savings account is in credit" do
  Given "my savings account balance is", 100 do |balance|
    @savings_account = Account.new(balance)
  end
  And "my cash account balance is", 10 do |balance|
    @cash_account = Account.new(balance)
  end
  When "I transfer", 20 do |amount|
    @savings_account.transfer_to(@cash_account, amount)
  end
  Then "my savings account balance should be", 80 do |expected_amount|
    @savings_account.balance.should == expected_amount
  end
  And "my cash account balance should be", 30 do |expected_amount|
    @cash_account.balance.should == expected_amount
  end
end

Scenario "savings account is overdrawn" do
  Given "my savings account balance is", -20
  And "my cash account balance is", 10
  When "I transfer", 20
  Then "my savings account balance should be", -20
  And "my cash account balance should be", 10
end
end

While this is a really cool start, there are a couple of problems. One is that we’re constrained in the way we phrase things. Because the arguments become part of the phrase, we have to structure each phrase so that the argument comes at the end.

The other problem, for me, is that the differing levels of abstraction in the two scenarios make it difficult to read.

Enter Blockless Steps and Parameterized Steps

The first step in resolving this problem was to decouple the expression of the story from the steps, which is accomplished with the use of Parameterized Steps. Here’s how the story above might look:

[Update: using StepGroup/define instead of StepMatchers/add]

steps = StepGroup.new do |define|
define.given("my savings account balance is $balance") do |balance|
  @savings_account = Account.new(balance.to_f)
end

define.given("my cash account balance is $balance" do |balance|
  @cash_account = Account.new(balance.to_f)
end

define.when("I transfer $amount") do |amount|
  @savings_account.transfer_to(@cash_account, amount.to_f)
end

define.then("my savings account balance should be $expected_amount" do |expected_amount|
  @savings_account.balance.should == expected_amount.to_f
end

define.then("my cash account balance should be $expected_amount" do |expected_amount|
  @cash_account.balance.should == expected_amount.to_f
end
end

Story "transfer to cash account",
%(As a savings account holder
  I want to transfer money from my savings account
  So that I can get cash easily from an ATM),
  :steps => steps do

Scenario "savings account is in credit" do
  Given "my savings account balance is 100"
  And "my cash account balance is 10"
  When "I transfer 20"
  Then "my savings account balance should be 80"
  And "my cash account balance should be 30"
end

Scenario "savings account is overdrawn" do
  Given "my savings account balance is -20"
  And "my cash account balance is 10"
  When "I transfer 20"
  Then "my savings account balance should be -20"
  And "my cash account balance should be 10"
end
end

A bit nicer, yes? The steps coming first is a bit noisy, but that could be extracted to another file, or perhaps we can add a means of associating them with the Story after the Story has already been parsed so they can move below the Story.

That bit aside, look how much cleaner the Story reads now. And we can do a couple of additional things to make it even nicer. One thing you might notice is that the line about transfering (When “I transfer 20”) doesn’t specify which way the transfer goes. We can improve on that by enhancing that step:

steps = StepGroup.new do |define|
...

define.when("I transfer $amount from $source to $target") do |amount, source, target|
  if source == 'cash' and target == 'savings'
    @savings_account.transfer_to(@cash_account, amount.to_f)
  elsif source == 'savings' and target == 'cash'
    @cash_account.transfer_to(@savings_account, amount.to_f)
  else
    raise "I don't know how to transfer from #{source} to #{target}"
  end
end

...

That lets us write the step as

When “I transfer 20 from savings to cash”

As you can see, this is a big step towards making stories more clear and flexible.

More on Steps

Another thing you may have noticed is that the Steps are grouped together somewhat arbitrarily. Thanks to a couple of handy convenience methods, you can easily build up libraries of these steps and make them as broad or as granular as you like. Perhaps we want the account steps available to many stories, but the transfer step only to this one. Here’s how you can handle that:

class AccountSteps < Spec::Story::StepGroup
steps do |define|
  define.given("my savings account balance is $balance") do |balance|
    @savings_account = Account.new(balance.to_f)
  end

  define.given("my cash account balance is $balance" do |balance|
    @cash_account = Account.new(balance.to_f)
  end

  define.then("my savings account balance should be $expected_amount" do |expected_amount|
    @savings_account.balance.should == expected_amount.to_f
  end

  define.then("my cash account balance should be $expected_amount" do |expected_amount|
    @cash_account.balance.should == expected_amount.to_f
  end
end
end

steps = AccountSteps.new do |define|
define.when("I transfer $amount") do |amount|
  @savings_account.transfer_to(@cash_account, amount.to_f)
end
end

Here we’ve created a subclass of StepGroup, instantiated one and defined an additional ‘when’ that will only be available to this instance.

Goodbye quotes!

Once we were able to get rid of the blocks, the quotes made no sense. So we’ve added support for true plain text stories. So now our example can read like this:

Story: transfer to cash account
As a savings account holder
I want to transfer money from my savings account
So that I can get cash easily from an ATM

Scenario: savings account is in credit
  Given my savings account balance is 100
  And my cash account balance is 10
  When I transfer 20
  Then my savings account balance should be 80
  And my cash account balance should be 30

Scenario: savings account is overdrawn
  Given my savings account balance is -20
  And my cash account balance is 10
  When I transfer 20
  Then my savings account balance should be -20
  And my cash account balance should be 10

That gets stored in a plain text file and you can run it by running a ruby file that looks like this:

require 'spec'
require 'path/to/your/library/files'
require 'path/to/file/that/defines/account_steps.rb'

# assumes the other story file is named the same as this file minus ".rb"
runner = Spec::Story::Runner::PlainTextStoryRunner.new(File.expand_path(__FILE__).gsub(".rb",""))
runner.steps << AccountSteps.new
runner.run

And that’s it! It’s that simple. This is still in its very early phases and I’m certain there will be enhancements as people gain experience with it.

If you want to check it out yourself, grab the trunk and do the following:

cd trunk/rspec
ruby examples/stories/calculator.rb
ruby examples/stories/addition.rb

The first example uses Ruby with blockless steps. The second example uses a plain text story stored in examples/stories/addition.

Also, with a couple of small tweaks we’ll be able to consume plain text from any source (not just a local file) and feed it into the PlainTextStoryRunner. This means that we’ll be able to do things like email scenarios to an app that consumes email and runs the scenario against the system and emails you back a report! Crazy, huh?

Lastly, just a reminder, this is only in trunk right now (as of rev 2764), so if you want to explore it you’ll have to get the trunk.

Enjoy!!!!!

Simple Matchers made Simple

Although RSpec supports custom matchers, it has always been a bit more work than is ideal for simpler situations. This could be attributed to the desire for a system which would be flexible.

But now, with a bit of convention-over-configuration kool-aide, we offer you the SimpleMatcher.

The SimpleMatcher snuck its way into RSpec’s source when we merged in the Story Runner (formerly RBehave). Dan North had wanted a simpler way to create custom matchers, and so he created one and used it throughout the specs for the Story Runner.

And now we bring it to you (today if you use trunk, otherwise next release).

Here’s how you use it:

<code>def beat(hand)
return simple_matcher("hand that beats #{hand.to_s}") do |actual|
  actual.beats?(hand)
end
end

full_house.should beat(flush)
=> nil #passes

straight.should beat(flush)
=> Expected hand that beats Flush, got Straight
</code>

Admittedly, these are only useful for very simple cases. But what’s in a name?

159

Apparently I’ve reached some sense of balance. I’m ½ way there (somewhere). 1:59 since it all began, 1:59 left to go …

rSpec Basics Peep Code is Out

Topfunky has just released the first of a 3 part PeepCode series on rspec entitled rSpec Basics. I’ve gone through it myself and I am impressed.

I would recommend this for anybody who is just getting started with rspec and rails. As its title suggests, it covers the basics: a bit of BDD philosophy, getting things set up, writing simple examples, all the way through describing models.

As for advanced users, I’d recommend this to those of you who are interested in a few helpful tips and tricks. There is some material on the textmate bundles and integrating with growl. In fact, the approach to validating models is simple and pragmatic, and one that I suspect will become the standard in time.

The only constructive criticism I’d offer is that the discussion of the philosophy of the rspec team doesn’t recognize the roots of our philosophy in Test Driven Development. It seems to present writing focused examples, getting them to fail first, etc, as our idea. It is not. Just ask google.

Regardless, it is very exciting to see the beginnings of quality educational material on rspec and BDD emerging. Keep your eyes open in this space. I suspect there is quite a bit more on the horizon.

template.expects_render

One thing that’s been missing from rspec_on_rails for a while is a clean and consistent way to mock or stub calls to partials from inside partials. In fact, even mocking partials inside non-partial templates has been buggy. For example, let’s say you’re describing a template that should include a partial named ‘_thing’. You might want to do something like this:

<code>assigns[:thing] = thing = Object.new
template.should_receive(:render).with(
:partial => 'thing',
:object => thing
)
render 'things/index.html.erb'
</code>

Now if that is the only example in your entire suite that renders ‘things/index.html.erb’, no problem. In other words, in most cases, this is a problem.

The Problem

It turns out that Rails compiles ERB templates in memory the first time they are encountered and continues to use the compiled version throughout a given process. This is a GOOD thing for performance. It is, however, a challenge for testability. Why? Because when we stub methods using rspec, mocha or flexmock, we replace the real methods with implementations from the mocking framework. Those methods are part of what gets compiled. And that means WEIRD STUFF.

If you mock a method in the template and the template gets compiled, then every other access to that template accesses the mocked method (even accesses in other examples). Conversely, if you mock a method in a template that’s already been compiled, well, it just doesn’t get hooked up at all and the mock expectation fails.

The problem with mocking partials inside partials is that a response only has one instance of ActionView::Base, so if you mock one call to render on that instance, you mock them all. This means that you simply can not use a standard mocking framework to mock the call as they are simply not designed to pass some calls on to the real object and intercept others.

RSpec’s Solution

To solve this, I created a proxy that delegates to a mock object, but that mock behaviour is not added to ActionView::Base directly. When ActionView::Base receives #render, it asks the proxy if it is interested in the call based on the arguments that were passed in. If so, it passes it on to the mock proxy for later verification, and otherwise swallows the call, the way a mock normally would. If it is not, however, interested in the call, execution of the render method continues as normal.

I have to confess that this feels a bit dirty. I come from a land of fairly strict rules about what mocks should and should not do, but now live in a land in which a lot of rules I learned before are being challenged. This is the land of Ruby and Rails. And so I grit my teeth, and do what seems pragmatic.

View Examples

The result is that you’ll now be able to do this in View Examples:

<code>assigns[:thing] = thing = Object.new
template.expects_render(:partial => 'thing', :object => thing)
render 'things/index.html.erb'
</code>

You can even do this if the thing you’re rendering in the example IS a partial which contains a sub-partial:

<code>thing = Object.new
template.expects_render(:partial => 'thing', :object => thing)
render :partial => 'outer_partial', :locals => {:thing => thing}
</code>

Controller Examples too

I also added the same behaviour to controllers in Controller Examples. This essentially restores the old controller.should_render behaviour that we gave up in RSpec 0.9, but with the syntax similar to that in the view examples above:

<code>controller.expects_render(:action => 'login')
get 'index'
</code>

Thanks to Jake Scruggs for pairing on this with me. It might not have happened had he not offered to help.

When can I use it?

This is committed to RSpec’s trunk and will be released with RSpec-1.0.6, sometime very soon.

pending(“insert reason here”)

In RSpec-1.0, we introduced a Not Yet Implemented feature. When you say …

<code>it "should do something"</code>

… with no block, the summary report lists that example as not implemented.

<code>37 examples, 0 failures, 1 not implemented</code>

As I started to use this I found myself doing stuff like this:

<code>it "should do something"
# do
#   here.is(the).actual(implementation).but(commented).out
# end
</code>

This made me sad. I hate having things that are commented out like that, even if the summary report draws my attention to it.

Then came a conversation with Dan about rbehave. In his article introducing rbehave, Dan talks about identifying pending scenarios so instead of getting failures while he’s working on the objects that must implement the behaviour, he gets a nice list of scenarios that should pass pending the completion of those objects. We discussed the similarities and differences between the Not Yet Implemented feature in RSpec and the Pending feature in rbehave and agreed that RSpec should have the pending method.

And so it has come to pass.

RSpec (trunk, as of rev 2118 – will be included in 1.0.6) still supports calling #it with no block, but now also supports the #pending method, allowing you to say:

<code>describe "pending example (using pending method)" do
it %Q|should be reported as "PENDING: for some reason"| do
  pending("for some reason")
end
end

describe "pending example (with no block)" do
it %Q|should be reported as "PENDING: Not Yet Implemented"|
end
</code>

And hear:

<code>$ ruby bin/spec examples/pending_example.rb -fs

pending example (using pending method)
- should be reported as "PENDING: for some reason" (PENDING: for some reason)

pending example (with no block)
- should be reported as "PENDING: Not Yet Implemented" (PENDING: Not Yet Implemented)

Finished in 0.006639 seconds

2 examples, 0 failures, 2 pending
</code>

The #pending method raises a Spec::DSL::ExamplePendingError, which gets reported, in this case, as “PENDING: for some reason”. If you leave off the block the example will be reported as “PENDING: Not Yet Implemented”. Either way, the summary will combine these two types of pending examples as just “pending”.

Real Confusion over Mock Concepts

Various dictionaries define Mock (the noun) as an imitation, a counterfeit, a fake. The term “Mock Object” initially meant exactly that – an imitation object, which serves as a control (i.e. invariant) in a test, allowing you to limit the variables in your test to the object being tested.

Over time, new, more specific definitions have emerged and confusion has ensued. Now we are calling these things “Test Doubles”, of which there can be several types including what we now call Mocks and Stubs. And to make matters worse, if I use what we now call Mocks, I’m a Mockist and if I use what we now call Stubs I’m a Classicist. Why can’t we just use both of these tools when each is appropriate and dispense with the name calling?

But I digress.

The main problem that I see with all of this is that the thing that separates the different kinds of Test Doubles is really how they behave at the individual message/method level. We have frameworks called Mocking Frameworks that create objects you can train to supply pre-defined responses (in which case its acting like a Stub), record messages (in which case it’s a Recording Stub), and even verify that specific messages are received (in which case its a Mock). The same object can have all of these behaviours, so the struggle over what to call the object seems to miss the point.

And to make matters more confusing, mocking frameworks in dynamic languages like Ruby give you the ability to treat any real object in your system in the same way as you treat a generated Mock, Fake, Test Double, Test Spy, etc, etc, etc. In RSpec, for example, you can do this:

<code>real_collaborator = RealCollaborator.new
real_collaborator.should_receive(:some_message)
object_i_am_describing = InterestingClass.new(real_collaborator)
object_i_am_describing.do_something_that_should_send_some_message_to_collaborator
</code>

So what can we call this object? It’s real, but it behaves like a Mock because I tell it to. This has always been considered a mocking no-no for many reasons. For example, you risk replacing methods that other methods in the same class rely on, which can lead to some unexpected test failures or, worse, passing tests that should be failing. In practice, I find that I only do this when dealing with other frameworks that rely on class methods (like Ruby on Rails’ ActiveRecord).

But, again, I digress.

Fighting confusion with more confusion

Of late, I’ve gotten into the habit of talking about these things at the method level. You have an object (Test Double or Real Object) that can have Method Stubs and Message Expectations, either of which can return Stub Values. I’m hopeful that these are self-explanatory, but in the interest of minimizing confusion:

By “Test Double” I mean the Meszaros definition. Essentially, a test-specific substitute for a real object.

By “Method Stub” I mean a no-op implementation of a method that may or may not be called at any time during the test. A Method Stub returns a Stub Value unless it returns nil, None, void, nirvana, etc.

By “Message Expectation” I mean an expectation that a specific message will be received by the Test Double.

By “Stub Value” I mean a single, pre-defined value that will be returned by a Method Stub regardless of whether or not it is associated with a Message Expectation.

I recognize that I risk adding to the confusion instead of minimizing it, however I think this makes the whole thing easier to understand. What do you think?

RSpec 1.0 - belated thanks

The decision to release RSpec 1.0 happened quite spontaneously at RailsConf 2007 in Portland. I heard more than one person there say they like RSpec’s sensibilities and might use it if not for the changing API and upgrade problems, adding that they’d consider using it when it goes 1.0.

Aslak was in Portland as well, so he and I talked it over and decided that the time was right to put a stake in the ground and release RSpec-1.0. Brian Takita was there as well, and was happy to join in the effort. So the three of us sat down to close up some holes and ship it.

But the REALLY cool thing was that a few other people got wind of our plan and sat down with us to help make it all happen. Big thanks go out to Kurt Schrader, Chad Humphries, Ken Barker and Dustin Tinney for joining the three of us. It was a blast hanging out with all you guys and the effort is seriously appreciated.

oxymoron: testing behaviour of abstractions

The question of how test behaviour implemented in abstract classes came up on the RSpec list yesterday. This is something that comes up periodically, so I thought it worth posting about. Over the years I’ve seen and tried a few different approaches to dealing with this problem and I’ve come to an approach that feels right to me. It stems from two basic principles:

  • Abstract classes don’t have any behaviour

  • Only remove duplication that actually exists

Abstract classes don’t have any behaviour

Abstract classes are a structural tool that we can use to share implementation between concrete classes. Sometimes they actually represent abstract types, but all too often they’re misused as convenient placeholders for shared implementation. When that happens, their presence clouds meaning.

In statically typed languages like Java and C#, you can spot this misuse by looking at the types being referenced in method signatures. If no methods accept the abstract class, then it isn’t really functioning as an abstract type in your system.

This is a bit more subtle in Ruby because we don’t have an abstract class construct and we think in duck types instead of static or runtime types. Abstract classes are really more of a convention than a language feature.

Regardless, this rule of thumb steers me away from testing abstract classes directly.

Only remove duplication that actually exists

This is a general rule of thumb that I like to apply whether I’m dealing with examples or subject code. Only remove actual duplication. If you start with something general before you have anything specific, there is a tendency to make assumptions about what the duplication will be, and those assumptions are often misguided.

In terms of abstractions, this rule of thumb guides me to extract abstraction when I see duplication rather than imposing it up front.

Putting the two together

The question on the list was specifically about how to test methods that live in in Application in a Ruby on Rails app. For the uninitiated, Application is the base class for all controllers in an MVC framework. Ruby doesn’t support abstract classes, so you can actually initialize Application, but AFAIK the Rails framework never does.

Based on the two principles, and given that I write executable examples first, I start by describing the behaviour of a concrete controller, developing methods directly on that controller against those examples. When a new controller comes along that should have the same behaviour, I’ll often duplicate the examples and the behaviour first and then extract the duplication into shared examples and the Application controller. This way, if there are any differences at all in the expected behaviour (which there often are) it easier to figure out the exact bits that I want to extract.

Once I’ve got the abstraction in both the examples and code I can just plug them in to the third, fourth, etc examples.

Other schools of thought

There is a runtime cost to pay when you’re running the same examples for multiple subclasses of an abstract class. You could argue that this is wasteful because the same implementation is being tested multiple times, which flies in the face of the goal of fast running test suites. This argument might lead you to write tests directly for the abstract class.

I can’t really disagree with the performance cost, yet I still prefer to approach it as I do because I find it more clear and less brittle to be describing the behaviour of concrete types rather than the behaviour of abstract types (which doesn’t really exist).