A case against a case against mocking and stubbing

December 11th, 2008

In his blog entitled A case against Mocking and Stubbing, Brian Cardarella says

since I’ve been TATFT with TDD and some BDD I’ve come to believe that mocking/stubbing is a horrible idea and it can hurt the development process

Please take a minute to let that soak in. “a horrible idea” and “can hurt the development process”. In fact, please go read the post before you read on. I’d rather you read his words before you read my interpretation of them.

Back? Cool. Scary stuff, huh?

But never fear, because it’s not about you (unless you are Brian). What he is really saying is this:

since I’ve been TATFT with my own personal approach to testing Rails applications, which is a little bit different from what the TDD/BDD guys are doing and is largely based on Rails conventions which encourage you to couple layers together in your tests, I’ve come to believe that mocking and stubbing, two concepts that assist and encourage testing in isolation, which is the opposite of the kind of testing I like to do, is a horrible idea for me and can hurt my own personal development process

Before I defend my re-phrasing of Brian’s statement, let me say that he does have a couple of really good ideas in the post (specifically about the dilemma of databases), and I don’t intend to convince you that mocking and stubbing are inherently good ideas that will save the world, or that Brian’s process would be improved by adding mocks and stubs to it.

But Brian makes a broad generalization, attacking ideas that many view as inherently useful in the appropriate context, and I feel that the scope of his statement requires a bit of narrowing.

## Rails testing

First, the categories and hierarchy of tests that Brian describes suggest that he is writing Rails applications, and that he’s using Rails’ categorization of tests, but either replacing rails integration tests with a story/feature tool like cucumber or using such a tool to run in-browser tests, or some combination thereof.

Consider that, although Rails sports an MVC framework, there are really six different layers/compontents we’re dealing with: routing, views, controllers, models (model objects), persistence management (model classes), and persistent storage (the database).

Stories as a wrapper for rails integration tests exercise the full stack from the router down to the database and back, so it covers all 6 layers.

Rails functional tests typically start from the controller, down to the database, and back up past the controller to the view, so they incorporate 5 layers.

Rails unit tests start from the model classes and objects and go down to the database as well. Even if you don’t save the objects, just loading an ActiveRecord model does cause the model class to ask the database for information about its structure. So unless you’re explicitly blocking database access with a tool like NullDB or UnitRecord, these unit tests are really fine grained component integration tests (not to be confused with what rails calls integration tests).

So now we have three categories of tests that are all integration tests that operate on anywhere from 3 to 6 component layers per test. Brian states that “My functional tests should rely upon the models being written properly,” but the same is true of the stories. So now we have stories that depend on everything, and functional tests that depend on everything except the router.

Mocks and Stubs

So let’s talk about mocks and stubs for a sec.

Stubs are a form of Test Double that stand in for objects or methods that might require non-trivial setups, access expensive services, or even parts of your system that don’t yet exist. They exist to help you isolate the object under test, and assist in both fast running tests and quick fault isolation.

Mocks are generally useful in interaction testing. They help you to think about roles and more general types as opposed to specific objects. They are useful in the short term to help you stay focused on the object you’re working on now (mocking its collaborators) and discover new interfaces (again by mocking its collaborators) without having to go over and create those. And they are useful in the long term when you have polymorphic collaborators, so you can specify the interactions once, rather than doing state-based testing n times.

Integration vs Isolation

Brian’s approach to testing is all about integration. Mocking and stubbing is all about isolation. So naturally, these are not a good fit. But that doesn’t make mocking or stubbing a bad idea any more than it makes high levels of integration or isolation inherently bad ideas.

How I approach BDD with Rails

So let me tell you about my own approach to BDD in contrast, and perhaps this will provide some insight into why the rspec-rails gem/plugin is structured the way it is. Here’s how I generally like to develop rails apps (I say “like”, because this is my general goal, but there are sometimes good reasons to take other approaches):

  1. Write scenarios in plain text with cucumber (driven by user stories, organized in features).
  2. Write the code for a step (or part of a step), run the feature, and observe the failure.
  3. Optionally (yes, it depends – and why is the topic for another blog) drive out a view with a view spec. When I say “drive out,” I mean a very granular Red/Green/Refactor cycle that only involves this view, and only enough of this view to support satisfaction of the failing step in the cucumber feature.
  4. Drive out a controller action using the same, granular Red/Green/Refactor cycle. And it may not be the entire controller action I think I want, covering all the cases I think I want. Just enough to support satisfaction of the failing step.
  5. Drive out the parts of the model that I need to satisfy the failing step, using the same granular R/G/R process.
  6. Run the cucumber feature and assess where I am.

At this point Brian talks about going back and writing a new set of Stories. I, by contrast, go back and write the code for the next step.

Brian likes to paint with oil and work in broad strokes with a thick brush. I tend towards fine pencils with tiny strokes that only look like the thing they are part of when you stand back a few feet. Brian likes a lot of integration. I prefer working on small pieces in isolation, with just enough integration to give me confidence that they’ll work together correctly when they get wired up at run time.

Controllers and Views

Controller and view specs are run in isolation from each other so that changes to views will not cause my controller specs to fail, and vice versa. Though either might cause the cucumber scenarios to fail. That’s a good thing, and I believe that it is the same thing that Brian is looking for from both his stories and his functional tests.

Controller and view specs are also run in isolation from the models as much as possible. I actually don’t usually use tools like UnitRecord or NullDB, but that is because most of the apps I write tend to be pretty small (no more than 10 or 20 models). I think if I was on a bigger team writing bigger apps, I might cave in to the pain of slow test runs and introduce those tools.

But the thing that I try to keep the controller and view specs isolated from is not really the database. It’s the validation and other business rules expressed in the model. These are not concerns of the controllers.

Separation of Concerns

Typically, there are two paths through a controller action: success, or failure. And typically, success or failure is determined by the result of saving or updating the model. Why that save succeeds or fails, the controller doesn’t care about. And when model validation rules change, I really don’t want to have to go back to my controller specs and start modifying post params to get them to pass.

When model validation rules do change, those changes are first expressed in scenarios. So though application failures won’t bubble up from the model specs to the controller and view specs, they will bubble up to the cucumber scenarios.

This is clearly a VERY different style from Brian’s. Neither is inherently correct or superior. We both operate in our comfort zones. Clearly we both have the same high level goals of delivering quality, maintainable software that meets client need, and being able to do so with a consistent, smooth process that provides a lot of feedback and confidence every step of the way.

Mocks and stubs are great tools when they are used in the appropriate context

In the end, mocks and stubs are tools. Just like any other tool in our developer toolkits. They each have a narrow purpose. If you’re using stubs to isolate the code you’re testing, great! If you’re using mocks to discover new interfaces on collaborators, great!

If, however, you’re trying to stuff these isolation tools into a process that revolves around multiple levels of integration …. please …. stop. In that context, it’s really a horrible idea, and might even hurt your development process.

Posted by David Filed in Isolation Testing, RSpec, Test Doubles

12 Responses to “A case against a case against mocking and stubbing”

  1. Evan Light Says:
    December 11th, 2008 at 6:08 am

    Well said.

    <p>I&#8217;ve used a process right now similar to Brian&#8217;s sans stories.  Interestingly, it is for the very reason that you cite: the code was too tightly integrated to attempt to isolate individual units.  However, until now, I drew the conclusion unconsciously and hence gravitated toward a more integrated <span class="caps">BDD</span> approach to bolting on new features.</p>


<p>Handy to grasp it at a conscious level though.</p>
  2. Kurt Schrader Says:
    December 11th, 2008 at 6:08 am

    I’ve done Mock based testing in the way that Brian talks about and things started breaking without breaking the tests, leading to a similar backlash against mocks and stubs.

    <p>We were concentrating too much on the interactions and missing the integration points.  (To be fair, we started the project long before Cucumber or the Story Runner existed.)</p>


<p>Unfortunately, this backlash seems to lead to over testing to cover everything, and a test suite that ends up taking 10 times as long.</p>


<p>The <span class="caps">BDD</span> community needs to keep writing blog posts like this one to continue to clarify how they&#8217;re advocating doing things.  I&#8217;m now using a testing approach like the one described here and it&#8217;s striking a far better balance between test-suite run time, test-breakage when something goes wrong, and covering all of the integration points.</p>
  3. Steve Says:
    December 11th, 2008 at 6:08 am

    I think what’s also often missing in my experience is that mocks/stubs encourage a certain style of code design.

    <p>If your actions have loads of separate model calls then you&#8217;ll quickly get stuck if you start changing things further down the chain.  Therefore, m&#38;s point toward keeping controllers slim and creating nicely abstracted APIs in your models etc.</p>


<p>Every time I find myself or colleagues stuck with m&#38;s it&#8217;s because of this.</p>
  4. Geoffrey Grosenbach Says:
    December 11th, 2008 at 6:08 am

    There’s a seventh layer, too…remote services and APIs. My most useful mocks simulate replies from remote servers.

  5. Brian Cardarella Says:
    December 11th, 2008 at 6:08 am

    David,

    <p>It&#8217;s pretty clear that you are firmly an Rspec guy. (thus the book) Within the context of Rspec mocking/stubbing is essential. However, as I stated at the top of my blog post I left the Rspec world a while ago and have never looked back. Unfortunately I brought some of my Rspec-iness with me. I have found this to be true for others as well.</p>


<p>A major problem that I failed to point out in my post is that mocking and stubbing is just a huge pain in the ass. One ends up replicating behavior that ActiveRecord is giving us for free. I spent more time maintaing my mocked models and stubbed methods than I was doing anything useful. And I don&#8217;t feel that I am alone in this. Why else was Rspec recently removed from Cucumber? People are moving away from Rspec and I feel that the mocking and stubbing has a lot to do with this. I also feel that within the context of a Rails project that isolation testing does not lend itself to a proper Red/Green/Rafactor approach. (note I said within the context of a Rails project)</p>


<p>These are of course matters of opinion and I enjoy the spirited debate. Which is why I posted the article to begin with, I really wanted to hear a counter argument and you gave a great one. And I have to say that I agree with Geoffrey Grosenbach that mocking/stubbing should be used when making calls to resources that lie outside of your project. This is something that I practice. (again, something else I failed to mention in my blog post)</p>
  6. Brian Cardarella Says:
    December 11th, 2008 at 6:08 am

    Hmmm… not certain what happened with my comment post.

  7. Aslak Hellesøy Says:
    December 11th, 2008 at 6:08 am

    I think David’s fat brush (Cucumber) / thin pencil (RSpec) metaphor illustrates this topic rather well. Trivial applications or libraries can be painted with only a fat brush, and all you need is Cucumber (if that). More elaborate pieces of work have fine details and corners that can’t be reached with a fat brush like Cucumber or a similar end-to-end testing tool. It’s when I encounter these small spots that I like to drop down a level and use good old RSpec and M&S.

    <p>I use RSpec and M&#38;S when things get complicated. I use it as an aide to help me

    make complicated simple. Mocking helps me break down a complex problem into several simpler ones. When I write web apps in Rails I usually end up splitting up business logic among several classes (lib stuff) rather than lumping everything together inside AR models. Why? Because I like nicely decoupled code that follows the SRP, and mocking is my tool of choice to get there.

    <p>I think that a lot of the resistance against mocks is based on the overly simplistic design pattern

    a lot of people follow, which is Model + View + Controller – nothing else. Sure – if that’s how you write code, mocking – especially generated mocking – can be a PITA. If you OD on it, it can mean the drug is inherently bad, but it can also mean you used it wrong. I recommend the seminal OOPSLA mocking paper (http://www.jmock.org/oopsla2004.pdf) for a more in-depth analysis and introduction to mocking. Anyone bashing mocking should read it to avoid barking up the wrong tree.

    <p>Another reason why some people dislike (and misunderstand) mocks is because RSpec&#8217;s Rails generators

    generate them for you. Does anyone remember how Rails got slammed in the start for the generators? People got the wrong idea. Mocks are not supposed to be generated – they’re supposed to be consciously written by hand to discover new interfaces in your application. Maybe RSpec should never have added generators that generate mocks. Or maybe people should have been given more explicit warnings about the sideeffects (like Cucumber does with its feature generator – it says don’t fall in love with this it will bite).

    <p>As for &#8220;Why was RSpec removed from Cucumber?&#8221;. It wasn&#8217;t <strong>removed</strong>. It had some smaller couplings

    in the gemspec and a bias towards RSpec in the documentation, examples and Rails generators. It boils down to how you make assertions inside Cucumber step definitions. Since the decoupling you can do it with RSpec’s #should and #should_not, but you can also use #assert_ and most other testing/bdd flavours and tools out there.

    <p>I decided to decouple Cucumber from RSpec because I want everybody to have the freedom to choose what

    they want – not because I have any beefs against RSpec. Heck, I’m one of the main contributors to RSpec and it’s still my preferred low-level BDD/mocking framework.

    <p>Are people moving away from RSpec? Sure &#8211; all the time. Are people adopting it and getting enlightened?

    Sure – all the time.

  8. Moses Hohman Says:
    December 11th, 2008 at 6:08 am

    I appreciate your reminder that context matters when deciding whether one approach is better than another. No solution is perfect, in isolation or in general, and it usually comes down to deciding on the lesser of two evils, or to put it more positively, the solution that fits your particular situation better.

    <p>On our Rails app, we have 13,000 <span class="caps">LOC</span> and 200 models. Our business domain is rich and complex. Our code base has been in production for a couple years now. We tried the mocking approach without good black box integration testing, and did run into the problem of tests not breaking when the app was broken. However, once we got black box integration testing wrapped around things, that was much less of a problem (in fact it only really occurred once). The much bigger problem for us down the road was how mocks hindered/slowed down the refactoring process. They really became a pain in the butt (see http://www.moseshohman.com/blog/2008/07/06/too-many-mock-objects-ruby-refactoring-death/ for my frustrated blog post on the subject). As a result we usually translate any of the old mocking tests to use real objects whenever we come across a breakage.</p>


<p>We haven&#8217;t made all our decisions perfectly, but I do think this is a real problem that affects larger, more complex apps that use mocking. Unfortunately larger, more complex apps are also affected by the slow test suite problem. I hadn&#8217;t kept up to speed with the efforts to decouple from the database, looks like I&#8217;ll be learning more about those tools soon.</p>
  9. Moses Hohman Says:
    December 11th, 2008 at 6:08 am

    A few additional details:

    <p>First, go Textile: <a href="htttp://www.moseshohman.com/blog/2008/07/06/too-many-mock-objects-ruby-refactoring-death/">an actual link to my blog post on the topic of mocking and refactoring</a></p>


<p>Second: I would go farther and say that mocks don&#8217;t just slow down refactoring, they break the traditional (and sensible) role tests play in the refactoring process. Tests are supposed to pass before and after refactoring. When using mocks, because mocks/stubs, in order to function correctly, almost always depend on the details of a particular implementation, mock expectations/stub behavior breaks when you refactor the implementation on which they depend. You then have to fix your tests, making sure, by visual inspection, that they are still correct/testing the behavior you want.</p>


<p>To mitigate the problem of controller tests depending on knowledge of domain logic, we have used a combination of fixtures (for canned data that we won&#8217;t need to change much, because changing fixtures can cause nasty test breakage) and ObjectMother (to create slightly modified versions of fixtures or default objects). This has generally worked pretty well to decouple our controller tests from domain logic, although I admit mocking/stubbing does take this goal farther. That said, practically speaking, we haven&#8217;t suffered.</p>


<p>Another piece of context: I don&#8217;t believe in exhaustive black-box integration tests, i.e. integration tests that explore every aspect of your domain logic. I believe that&#8217;s the role of lower-level testing. This opinion is probably influenced by the size of our development team (&lt;=5 people) and the complexity of our app. If we did have exhausting integration testing, then we could theoretically depend on our integration tests during refactoring, but the problem there is they would definitely take too long to run for that to be practical.</p>


<p>Finally, I have written my share of view specs, and I never found that they broke for good reasons (i.e. because something was actually wrong). For me there are two things that are important to test about views: that they don&#8217;t blow up when sent realistic data (best sent in an integration test by the controller), and that they look nice (best checked by visual inspection).</p>
  10. Micah Geisel Says:
    December 11th, 2008 at 6:08 am

    Tests are supposed to pass before and after refactoring. When using mocks, because mocks/stubs, in order to function correctly, almost always depend on the details of a particular implementation, mock expectations/stub behavior breaks when you refactor the implementation on which they depend.

    <p>Yes! I would love to see a thoughtful pro-m&#38;s response to this statement. This issue alone has prevented me from successfully utilizing m&#38;s, and I would like to be shown that this problem is simply a result of my m&#38;s test-fu being clumsy.</p>


<p>Excellent discussion.</p>
  11. David Chelimsky Says:
    December 11th, 2008 at 6:08 am

    Tests are supposed to pass before and after refactoring

    <p>Sure, but who says the refactoring can never include the code examples?</p>


<p>If I do a Move Method refactoring to move the #foo method from a This to a That, all my This examples that call #foo are supposed to pass? Absolutely not! There are some refactorings that are simply going to require the examples follow the changes.</p>


<p>That said, there are certainly occasions where the use of mocks require you to make changes to the examples that are closer to implementation, but for me this is no reason not to use mocks/stubs, as long as you&#8217;re getting some other benefit out of using them.</p>
  12. Moses Hohman Says:
    December 11th, 2008 at 6:08 am

    True, but I think it’s a question of degree: mocks/stubs make tests break more often for bad reasons, leading to a mock maintenance overhead that can get unacceptably high.

    <p>It&#8217;s true that refactoring leads to some level of test breakage regardless of your approach. Ideally, however, it wouldn&#8217;t. If your tests really just tested the behavior of your code, then they wouldn&#8217;t break (taking as the definition of refactoring: design/structural change that does not affect behavior). However, tests also depend on your code&#8217;s interface and, if you use mocks/stubs, on its implementation. As a result, they can break when you refactor.</p>


<p>What I think might be true is that for more complex projects, the balance between the benefits (faster tests, design pressure toward simpler interfaces and loose coupling) and costs (slowed refactoring, etc.) shifts. At least I certainly feel it did on our project.</p>


<p>I took a very brief survey of our code base, at it looks like 30% of our model specs (*_spec.rb files) and half of our controller specs (again, files) use mocks at least once. So, we maintain our use in some areas, but it has decreased over time. I tend not to use mocks anymore unless there&#8217;s a pronounced benefit.</p>


<p>I wonder if you would feel the same way after working on a more complex, longer-term project (for some suitable definition of more complex/longer-term). Not knowing you, I&#8217;m going to guess probably not. That points to this also being a question of style, of what each developer personally holds to be more valuable (or more annoying :).</p>