JUnit-Shortcoming II: Fail on first failed Assertion

In an earlier post I did describe a possibly better way of handling fixture tear down in JUnit.

In this post I look at the way assertions work. JUnit is actually widely used as a driver for all kinds of tests, not only unit tests. I think this is perfectly reasonable as it leverages all the support that JUnit has in IDEs and build tools.

However one problem is that a test case will be aborted on the first failed assertion. It would be nice, if you could specify, whether the failure should abort the test or just be recorded as a failure, thereby enabling better feedback on potentially longer running tests.

While this is probably generally agreed for integration/ acceptance tests I recently found that it would also be nice to have “non failing asserts” for unit tests. Let’s consider the following two unit tests for a copy constructor:

  @Test
    public void copyConstructorPopulatesCenter(){
        Circle circle = new Circle(new Point2D.Double(10,10), 2);
        Circle clone = new Circle(circle);
        assertEquals(new Point2D.Double(10,10), clone.getCenter());
    }
 
 
    @Test
    public void copyConstructorPopulatesRadius(){
        Circle circle = new Circle(new Point2D.Double(10,10), 2);
        Circle clone = new Circle(circle);
        assertEquals(2, clone.getRadius(), 0.001);
    }

While we are testing two separate things, all the setup code actually hides what the copy constructor is about, so I’d prefer to write something along these lines:

    @Test
    public void clonePopulatesFields(){
        Circle circle = new Circle(new Point2D.Double(10,10), 2);
        Circle clone = new Circle(circle);
        assertEquals(new Point2D.Double(10,10), clone.getCenter());
        assertEquals(2, clone.getRadius(), 0.001);
    }

This is much terser. The zealots will cry that there shall be only one assert per test. My thinking is, that that’s all true, but then this method contains two tests. The only (albeit not big) problem is, that when the center is not populated properly it’s preventing the second test from being executed. Another interesting observation is that having multiple “tests” in one method saves us from naming them, which in itself is a bit of an issue.

I am not sure if such a feature can be implemented by just writing a custom JUnit runner. The beautiful simplicity of the JUnit approach stems from the fact, that it uses exceptions to report failures. Thereby allowing the test code not to carry around some kind of test context. However once you have got a test context other things become possible, which I hope to elaborate on in a later post.

This entry was posted in Software Development. Bookmark the permalink.

5 Responses to JUnit-Shortcoming II: Fail on first failed Assertion

  1. Li Xiao says:

    Maybe the hard of writing one assertion for your case is because your model is hard for asserting :). Shouldn’t it be easier that two Circle instances are equals? Or, you really want just to test your design(what the copy constructor is about,) instead of behavior.

  2. felix says:

    Hi Li,

    good observation especially given that I like to bash people about not implementing equals on value objects. So the example might be ill-chosen.

    However this would mean that you needed similar test cases for the equals method, where I would also feel like putting 5 or so assertions (i.e. tests) into a single method.

    Another example would be a bunch of tests for a regular expression, where I would be very wary of littering my tests with method declarations.

  3. Jeff Santini says:

    Felix,

    First I acknowledge the example was probably trivial, but I have to say I agree with Li.

    Having an equals on the object solves the specific problem, and in the context of testing an equals method, individual tests are more appropriate as each test would test a specific component of equals. Not sure how regular expressions would be different, there would actually be a component of the expression being tested in each test, as least as I envisualize these hypothetical tests.

    Or maybe I am not as brave as you. xUnit ideas are part of the foundation of my software world, so I get scared at the idea of changing them 🙂

    Jeff

  4. felix says:

    Jeff,

    there are actually two points in my post. First I would like to be able to use the same tool for integration/ acceptance tests, that I use for unit tests, because in my experience the border between the two is quite blurred. Forthermore a general purpose programming language is imho better suited to the task than let’s say FITnesse. Do you think this is a valid point?

    Secondly I tried to come up with a reasonable example for a unit test where not failing on the first assertion might be beneficial. I don’t say you should have more than one assert per test, but there might be more than one test per method. Another example:

    public void testIsEmptyString ()  throws Exception {		       
        assertTrue(StringUtils.isNullOrEmptyString(null));
        assertTrue(StringUtils.isNullOrEmptyString(""));
        assertFalse(StringUtils.isNullOrEmptyString("A non-empty string"));
    }
    

    So whenever I test something that is functional in nature I like to use this pattern. Also due to the lack of literals for lists and maps it is often desireable (even though varargs actually mitigated that to a large extend) to split a simple assertion into multiple assert statements (yes, I know hamcrest could help me). My point however is to challenge the assumption that a test is bound to a method. In haskell a test is just a predicate. Your test suite is hence defined as a list of predicates.
    The fact that methods are used for tests, in the old smalltalk was probably for convenience reasons to abuse the class-browser to browse them. In java it is definitely a case of poor man’s closures. Does that make any sense to you? Would you call the above test method a severe profanity ;-)?

    Cheers,
    felix

  5. szczepiq says:

    Guys, I think the most useful would be to build a prototype and start using it 🙂 This way we know if it makes sense or not. How can we create a runner that allows assertEquals() to continue after failure? I think the easiest is to hack on jUnit sources and introduce an extension point for runners to change the behavior of assertions 😉

    Maybe we should have two different methods: assertEquals() and assumeEquals() or something?

    Recently I’m helping some ‘agile’ teams. I wish they had problems like this: ‘jUnit shortcomings’. They hardly write any tests… 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.