Test Sclerosis – The Trouble with Unit Tests

Your typical system, written in a nice object-oriented way performs its task using a number of collaborating objects that send methods to each other. Ignoring some of the complexities like objects actually creating new objects in response to message, we can draw a diagram like the one below:

Each arrow represents a method call. Now for mocking and unit testing, usually you want to test one particular part of the system. In a lot of cases I have seen so far this would be a single object. Somewhat like this:

As we are all aware of dependencies by now and know how we inject them, the constructor for this object probably takes its two collaborators as parameters. So we can easily pass in mock objects, that are used to mimic the actual collaborators behaviour and/ or verify that the object under test actually does trigger certain actions in its collaborators:

So far so good. What is the problem with this approach?
Another thing, that the well-adjusted programmer does quite often, is refactoring. While a lot of people think of rename class/ method/ variable, when refactoring is mentioned, these are obviously the trivial cases. Let’s consider a serious change to a subsystem of our hypothetical application:

We might actually introduce some new objects to get a better factored system (contrived example). So we end up with the following:

If we actually went down the hard-core mocking road with that system the place we start from actually looks like this:

The red lines indicate interactions that are now asserted and mocked out throughout the tests. To do a refactoring like the one proposed above you have to change a whole lot of test cases. Even worse I don’t have the confidence, that the behaviour is still the same, because I had to fiddle with the tests during the refactoring, even though the “outer” interface remained the same. I call this situation test sclerosis as the tissue of your application is hardened by all the tests and mocks and thereby loses a lot of flexibility.

How to avoid this situation?

One important question that helps avoiding getting into the above mentioned situation is asking about the value of writing a particular test. Generally we write tests for the following reasons (not a complete list):

  • Ensure the code actually provides the required functionality.
  • Avoid regression (safety net).
  • Help with design.
  • Make debugging easier.
  • (Make you feel confident.)

It seems to me that naive massively stubbed, or mocked unit tests usually don’t help much with these goals.

The general advice seems to be to choose your units carefully. Single objects are probably too fine grain. Look for natural subsystems. You might for example stub out the file system or an external system that has a very clearly defined interface, e.g a mail server. Keeping the number of mocks and stubs needed per test-case low (i.e. zero or one) is a good tactic. Try to test things that are interesting and non-trivial.

This entry was posted in Software Development. Bookmark the permalink.

3 Responses to Test Sclerosis – The Trouble with Unit Tests

  1. Perryn Fowler says:

    Hi Felix,

    I’m afraid I’m going to have to disagree with you here. I would argue quite strongly that single classes are the right level to be unit tested.

    What you call “hard code mocking” is exactly what I would recommend exactly because it stops you code becoming hardened.

    The point I would make is that the change you are describing as a ‘refactoring’ is *not* a refactoring at the level you are talking about. You are changing the behaviour of those classes, hence their test cases should change. It follows that if you are TDDing then you should change the unit tests *first* and then make the changes to the code. Driving the coding of these classes this way ensures that these classes stay loosely coupled. In my experience they help primarily with

    # Help with design.
    # Make debugging easier.
    # (Make you feel confident.)

    The level at which the change *is* a refactoring is the level that you have circled in green. It follows that you would have an *integration* test at this level that should still pass. This helps primarily with

    # Ensure the code actually provides the required functionality.
    # Avoid regression (safety net).
    # (Make you feel confident.)

    So, what about how hard it is to change all those test cases? My response is that instead of not writing them, make them easier to change.

    As you said “Keeping the number of mocks and stubs needed per test-case low (i.e. zero or one)” is a good tactic. It helps you to make sure each of your (class level) unit tests is simple to work with and *also* makes sure that your class does not have too many dependencies.

    If you are finding that tests seem to be making change painful I would not take this as a sign you have too many tests, but as a sign that your tests are telling you your design may need some work

  2. Alex Chaffee says:

    The best solution to this problem is to inject nothing (i.e. remove as many dependencies as possible). Objects that are leaf nodes are easiest to test; therefore, make your object trees as shallow as possible. Unfortunately, some people, when confronted with a difficult-to-test design, conclude that it’s the testing strategy, not the design, that must change.

    Agreed that “object” and “unit” are not always synonymous, but they overlap often enough that the default unit should be an object, and if you want to break that rule you’d better have a clear reason. Far be it to me to invoke a slippery slope argument, but your concluding advice that “Single objects are probably too fine grain” is dangerous because it gives people an excuse not to test certain things that they really should. (Or, it’s helpful because it provokes a useful discussion of where the line is.)

    My favorite objects to write are also the easiest and most fun to test. I call them “jewels” or “gems” because their interfaces and behavior are so clear and they’re so self-contained, with all their facets facing out. If only every class could be a Set… by which I mean, with a little creativity, you may be able to transform a rat’s nest of interdependent objects into one or more perfect little gems.

    Alex Chaffee – alex@stinky.com
    Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch | http://alexch.tumblr.com

  3. Good post. Maintenance of unit tests can be costly on large projects.
    While Perryn’s arguments above are valid in some context, things change with the scale of the project, the number of unit tests, the granularity of your classes, and your object dependencies.
    State verification, as opposed to behavior verification, may be of value here. Refer to the discussion included in http://martinfowler.com/articles/mocksArentStubs.html.

Leave a Reply

Your email address will not be published. Required fields are marked *