Archive for the ‘Java’ Category.

Exception Handling – The Catch Block should go Last

I have just stumbled across a piece of code like this:

    Object getSomeValue() {
        Object value = null;
        try {
           value = errorProneOperation();
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }
        return value;
    }

Now I find this really awkward. Why initialise something to null to keep the compiler happy?
Surely you want to do this:

    Object getSomeValue() {
        try {
           Object value = errorProneOperation();
           return value;
        } catch (IOException e) {
            throw new IllegalStateException(e);
        }
    }

No useless initialisation here. Also the scope of the local variable is smaller, which is good. I think there is a general rule here, which is: “There should be nothing after your catch-handler apart from a potential finally”. As with all rules it might be broken, but only with good reason. I don’t want to see that crappy initialisation to null again.

Java Lib to Launch External Processes

I recently redesigned some of the code I tend to use to spawn external processes (pdflatex anyone?) in java. The implementation is still a bit buggy, but I am more interested in people’s opinions about the API (non-blocking killable invocations are not yet supported). The project on google code is called jproc. Here is the cookbook so far:

To launch an external program we’ll use a ProcBuilder. The run method
builds and spawns the actual process and blocks until the process exits.
The process takes care of writing the output to a stream (as opposed to the standard
facilities in the JDK that expect the client to actively consume the
output from an input stream:

ByteArrayOutputStream output = new ByteArrayOutputStream();
 
new ProcBuilder("echo")
        .withArg("Hello World!")
        .withOutputStream(output)
        .run();
 
assertEquals("Hello World!\n", output.toString());

The input can be read from an arbitrary input stream, like this:

ByteArrayInputStream input = new ByteArrayInputStream("Hello cruel World".getBytes());
 
ProcResult result = new ProcBuilder("wc")
        .withArgs("-w")
        .withInputStream(input).run();
 
assertEquals("3", result.getOutputString().trim());

If all you want to get is the string that gets returned and if there
is not a lot of data, using a streams is quite cumbersome. So for convenience
if no stream is provdied the output is captured by default and can be
obtained from the result.

ProcResult result = new ProcBuilder("echo")
                            .withArg("Hello World!")
                            .run();
 
assertEquals("Hello World!\n", result.getOutputString());
assertEquals(0, result.getExitValue());
assertEquals("echo \"Hello World!\"", result.getProcString());

For providing input there is a convenience method too:

ProcResult result = new ProcBuilder("cat")
   .withInput("This is a string").run();
 
assertEquals("This is a string", result.getOutputString());

Some external programs are using environment variables. These can also
be set using the withVar method

ProcResult result = new ProcBuilder("bash")
                            .withArgs("-c", "echo $MYVAR")
                            .withVar("MYVAR","my value").run();
 
assertEquals("my value\n", result.getOutputString());
assertEquals("bash -c \"echo $MYVAR\"", result.getProcString());

A common usecase for external programs is batch processing of data.
These programs might always run into difficulties. Therefore a timeout can be
specified. There is a default timeout of 5000ms. If the program does not terminate within the timeout
interval it will be terminated and the failure is indicated through
an exception:

ProcBuilder builder = new ProcBuilder("sleep")
        .withArg("2")
        .withTimeoutMillis(1000);
try {
    builder.run();
    fail("Should time out");
}
catch (TimeoutException ex){
    assertEquals("Process 'sleep 2' timed out after 1000ms.", ex.getMessage());
}

Even if the process does not timeout, we might be interested in the
execution time. It is also available through the result:

ProcResult result = new ProcBuilder("sleep")
        .withArg("0.5")
        .withTimeoutMillis(1000)
        .run();
 
assertTrue(result.getExecutionTime() > 500 && result.getExecutionTime() < 1000);

By default the new program is spawned in the working directory of
the parent process. This can be overidden:

ProcResult result = new ProcBuilder("pwd")
        .withWorkingDirectory(new File("/"))
        .run();
 
assertEquals("/\n", result.getOutputString());

It is a time honoured tradition that programs signal a failure
by returning a non-zero exit value. However in java failure is
signalled through exceptions. Non-Zero exit values therefore
get translated into an exception, that also grants access to
the output on standard error.

ProcBuilder builder = new ProcBuilder("ls")
                            .withArg("xyz");
try {
    builder.run();
    fail("Should throw exception");
} catch (ExternalProcessFailureException ex){
    assertEquals("ls: xyz: No such file or directory\n", ex.getStderr());
    assertEquals(1, ex.getExitValue());
    assertEquals("ls xyz", ex.getCommand());
    assertEquals("ls: xyz: No such file or directory\n", ex.getStderr());
    assertTrue(ex.getTime() > 0);
 
}

Input and output can also be provided as byte[].
ProcBuilder also copes with large amounts of
data.

int MEGA = 1024 * 1024;
byte[] data = new byte[4 * MEGA];
for (int i = 0; i < data.length; i++) {
    data[i] = (byte) Math.round(Math.random() * 255 - 128);
}
 
ProcResult result = new ProcBuilder("gzip")
   .withInput(data)
   .run();
 
assertTrue(result.getOutputBytes().length > 2 * MEGA);

The builder allows to build and spawn several processes from
the same builder instance:

ProcBuilder builder = new ProcBuilder("uuidgen");
String uuid1 = builder.run().getOutputString();
String uuid2 = builder.run().getOutputString();
 
assertNotNull(uuid1);
assertNotNull(uuid2);
assertTrue(!uuid1.equals(uuid2));

For convenience there is also a static method that just runs a
program and captures the ouput:

String output = ProcBuilder.run("echo", "Hello World!");
 
assertEquals("Hello World!\n", output);

Also there is a static method that filters a given string through
a program:

String output = ProcBuilder.filter("x y z","sed" ,"s/y/a/");
 
assertEquals("x a z\n", output);

Thoughts on handling Translations and Views on Source Code

One of my recent java projects was to be used by users with three different languages. We went with the standard java approach of using properties files for messages. In intellij there is decent tool support for that.
However it seems a bit odd to have a strongly typed language and then rely on string keys for text lookup. At some point we introduced enums representing the keys, but as they were not automatically generated they involved a lot of repetition. Also you are never quite sure how many arguments a message needs.

So I had this idea of using interfaces to represent resource bundles. Each message could be represented as a method, with parameters representing the arguments to the placeholders. It would look somewhat like this:

public interface ExampleMessagePool {
    String sayHello(String name);
    String bye(String name);
    String warning();    
}

From the clients point of view it would works as follows:

var factory = new MessagePoolFactory<ExampleMessagePool>(ExampleMessagePool.class);
 
 
ExampleMessagePool english = factory.getLanguageSource("en");
 
assertEquals("Hello Matthieu!", english.sayHello("Matthieu"));
assertEquals("Good bye Felix!", english.bye("Felix"));
assertEquals("Attention!", english.warning());
 
 
ExampleMessagePool french = factory.getLanguageSource("fr");
 
assertEquals("Bonjour Matthieu!", french.sayHello("Matthieu"));
assertEquals("Au revoir Felix!", french.bye("Felix"));
assertEquals("Attention!", french.warning());
 
 
ExampleMessagePool german = factory.getLanguageSource("de");
 
assertEquals("Guten Tag Matthieu!", german.sayHello("Matthieu"));
assertEquals("Auf Wiedersehen Felix!", german.bye("Felix"));
assertEquals("Achtung!", german.warning());

The interesting point here is that I get completion and also a hint as of which arguments a particular message takes. The MessagePoolFactory takes care of creating instances for the respective languages. The next question is obviously, where these messages do come from. One way would be to use annotations:

@MessagePool(languages={"en", "de", "fr"})
public interface ExampleMessagePool {
    @Translations(
            entries = {
                @Entry(key = "en", value = "Hello {0}!"),
                @Entry(key = "de", value = "Guten Tag {0}!"),
                @Entry(key = "fr", value = "Bonjour {0}!")
            }
    )
    String sayHello(String name);
 
    @Translations(
	        entries = {
	            @Entry(key = "en", value = "Good bye {0}!"),
	            @Entry(key = "de", value = "Auf Wiedersehen {0}!"),
	            @Entry(key = "fr", value = "Au revoir {0}!")
	        }
	)
	String bye(String name);
 
    @Translations(
	        entries = {
	            @Entry(key = "en", value = "Attention!"),
	            @Entry(key = "de", value = "Achtung!"),
	            @Entry(key = "fr", value = "Attention!")
	        }
	)
	String warning();
}

This approach is very sexy in so far as it allows to refactor method and parameter names. Also the implementation is trivial not least because java deals with unicode source files. On the other hand making your translators edit the java sources is probably a bit of a challenge. However you could provide a narrow view on that source code. The eclipse platform lends itself to that kind of experiment, so I actually did a bit of spike. Which I am demoing here:

Whilst this is very cool, it’s an entirely static approach. Especially with translations I find it beneficial if they are stored in a database so that they can be edited at run-time. Ideally some key users can then maintain translations, which are part of their domain anyway. Then the interface could be annotated with some sort of GUID, that could later be used at runtime to identify entries in the database:

public interface ExampleMessagePool {
    @Id("21850286-A914-4C13-88BA-75ACA2248B71")
    String sayHello(String name);
 
    @Id("DA3CD6BE-64E8-4501-A674-0789043CBD6F")
    String bye(String name);
 
    @Id("38211D99-BA2A-4572-9C23-976AC64181D8")
    String warning();    
}

This way the refactorability would be preservered. Also tools support could be provided. Annotating elements with GUUIDs seems to be an interesting concept. Just imagine your database mapping being immune to name changes.

There is definitely some more thinking required here, but it strikes me that the default mechanisms in java are very rudimentary. Much less than a talented developer can dream up in a rainy afternoon.
Further developments could include static or, if you go for the runtime approach, dynamic tools to analyse, whether translations are present, or whether there are duplicates in message pools.

Getting Real with Enumerable.java

Two weeks back I posted a video explaining how to get started with Enumerable.java. While this might have been enough information to start playing, there is a lot more to consider when going with Enumerable.java for production use.

Over the past few weeks I went through that experience on my project and here are some of the practices that we found useful. Håkan also released version 0.3.0 that incorporates some of our learnings.

When working in the IDE we used the javaagent to weave the lambdas. However there were a few caveats. In version 0.2.4 you could only black list packages to prevent them from being weaved. In most cases it is easier to specify explicitly for which package to enable weaving. Also when using contemporary java goodness there is other parties doing byte code magic. We experienced these problems with spring’s scoped proxies, therefore we introduced another parameter to filter out classes by a regular expression applied to the full classname. Proxy classes usually come with a lot of dollar signs in their names.
So the vmargs we are using in development are:

-javaagent:lib/deploy/enumerable-agent-0.x.x.jar
-Dlambda.weaving.included.packages=felix.application
-Dlambda.weaving.exclude.pattern=\$\$

If you are using eclipse you can specify these parameters as default vm-parameters in your JRE definition:

While using an agent in the ide is acceptable it gets messy, when deploying the application. Essentially all the JVMs on the way to production including your application server would need the agent. What we did instead is, we instrumented our class files in our build using the AOT lambda compiler. So this is a single step after the compilation. If you are using ant it could look somewhat like this:

<target name="lambda" depends="compile">
        <java fork="true"
              classname="lambda.weaving.LambdaCompiler"
              failonerror="true">
 
              <sysproperty key="lambda.weaving.debug" value="true"></sysproperty>
                  <classpath>
                      <path refid="prod"/>
                  </classpath>
              <arg value="prod.jar"></arg>
        </java>
    </target>

Important: The lambda weaver needs debug information on local variables and also you need to encode your source files as utf-8 to get the fancy lambda letter, so perhaps you have to change your compile target like this:

<javac srcdir="src/java" 
            destdir="${basedir}/tempCompile" 
            classpathref="prod"
            encoding="utf8"
            debug="on"
            debuglevel="lines,vars,source"
/>

From a technical point of view this is all you need to do. However there is also the question of how to use the new feature “responsibly”. In our code base we used the generic CollectionUtils to filter and transform Collections. While they are elegant from a theoretical point of view, they are quite an insult to the eye, so our aim was to replace all these with Enumerable.java. Functional operations on collections are very well understood (the method names used in Enumerable.java go at least back to Smalltalk-80, that’s thirty years).

So the advice introduce your team the following methods and aim to restrict the use of lambda expressions to those:

  • select
  • collect
  • find
  • inject
  • groupBy
  • While there is remarkable support for arrays and primitives, try to stick to
    Collections and proper Objects.

I think these alone justify the investment.
Things you should avoid or do only in after proper consideration:

  • Do not declare parameters of function types
  • Do not declare local variables of function types
  • Do not declare fields of function types
  • Do not modify enclosed state from within the closure, i.e. local variables or fields
  • Generally avoid side effects

The typical transformation we did, looked something like this:

    private Set<Order.Status> retrieveOrderStatuses() {
        Set<Order.Status> statuses = new HashSet<Order.Status>();
        if (orders == null) return statuses;
 
        for (Order  order : orders) {
            statuses.add(order.getStatus());
        }
        return statuses;
    }

becomes:

    @LambdaParameter private static Order o;
    private Set<Order.Status> retrieveOrderStatuses() {
        if (orders == null) return emptySet();
        return  collect(orders, λ(o, o.getStatus())).toSet();
    }

Because getting the lambda character (you might also want to use the alias fn) is a bit tricky you might want to use this java editor template for eclipse that also sorts out the static import:

λ(${impst:importStatic('lambda.Lambda.*')}${cursor})

Sticking to these simple rules actually helped us getting away from the anonymous inner classes and CollectionUtils and that was an easy sell to the mostly mathematically inclined team. In my opinion the fact that lambdas are restricted to expressions is actually a good thing.

Get Closures for Java Now!

The general state of affairs at Sun/ Oracle is very sad. If you are like me, you cannot take this much longer. Fortunately enough my esteemed colleague Håkan Råberg has invented what is at least a molotov cocktail perhaps even a guillotine to the revolution of java software development. Yes, I am talking about the modestly named Enumerable.java, which brings the Ruby enumerable module and closures to java. Without a special compiler, just by pushing the boundaries and enhancing byte code.

I created this little screencast, that shows how to get up and running with Enumerable.java:

There is also a hi-res version of the videos on google docs (approx. 20M download size).

For more examples and explanation watch Håkan’s blog and read the source.

Now, go, download and, enjoy!

What’s missing in Java 7

Assuming Sun manages to get “closures” into Java7 there is still a few things that are lacking. Today I once again ran into the restrictions around annotations. The .Net solution is simply more elegant. The fact that I can’t use arbitrary classes for annotation parameters is arbitrary and promotes widespread abuse of enums. Also that these guys aren’t regular classes with methods and all is quite annoying.

Using arbitrary classes as annotations would actually make the language simpler and is therefore good…

JUnit-Shortcoming II: Fail on first failed Assertion

In an earlier post I did describe a possibly better way of handling fixture tear down in JUnit.

In this post I look at the way assertions work. JUnit is actually widely used as a driver for all kinds of tests, not only unit tests. I think this is perfectly reasonable as it leverages all the support that JUnit has in IDEs and build tools.

However one problem is that a test case will be aborted on the first failed assertion. It would be nice, if you could specify, whether the failure should abort the test or just be recorded as a failure, thereby enabling better feedback on potentially longer running tests.

While this is probably generally agreed for integration/ acceptance tests I recently found that it would also be nice to have “non failing asserts” for unit tests. Let’s consider the following two unit tests for a copy constructor:

  @Test
    public void copyConstructorPopulatesCenter(){
        Circle circle = new Circle(new Point2D.Double(10,10), 2);
        Circle clone = new Circle(circle);
        assertEquals(new Point2D.Double(10,10), clone.getCenter());
    }
 
 
    @Test
    public void copyConstructorPopulatesRadius(){
        Circle circle = new Circle(new Point2D.Double(10,10), 2);
        Circle clone = new Circle(circle);
        assertEquals(2, clone.getRadius(), 0.001);
    }

While we are testing two separate things, all the setup code actually hides what the copy constructor is about, so I’d prefer to write something along these lines:

    @Test
    public void clonePopulatesFields(){
        Circle circle = new Circle(new Point2D.Double(10,10), 2);
        Circle clone = new Circle(circle);
        assertEquals(new Point2D.Double(10,10), clone.getCenter());
        assertEquals(2, clone.getRadius(), 0.001);
    }

This is much terser. The zealots will cry that there shall be only one assert per test. My thinking is, that that’s all true, but then this method contains two tests. The only (albeit not big) problem is, that when the center is not populated properly it’s preventing the second test from being executed. Another interesting observation is that having multiple “tests” in one method saves us from naming them, which in itself is a bit of an issue.

I am not sure if such a feature can be implemented by just writing a custom JUnit runner. The beautiful simplicity of the JUnit approach stems from the fact, that it uses exceptions to report failures. Thereby allowing the test code not to carry around some kind of test context. However once you have got a test context other things become possible, which I hope to elaborate on in a later post.

Mockito to the Rescue

After all the abuse in my previous post, I feel a bit guilty. Luckily Mockito provides a way out of that problem by allowing hamcrest matchers being used to verify arguments. Here we go:

 verify(listener).propertyChange(
                (PropertyChangeEvent)argThat(
                        new HasEqualState(
                                new PropertyChangeEvent(holder, "value", "Old", "New"))));

It is not nice, but it works. Of course I also had to implement the HasEqualState matcher along these lines:

  public boolean matches(Object actual) {
            if (actual == null)
                return false;
            if (actual.getClass() != reference.getClass()) // this might be naïve
                return false;
 
            Class clazz = reference.getClass();
            for (Field field : clazz.getDeclaredFields()) {
                field.setAccessible(true);
 
                Object refValue = field.get(reference);
                Object actValue = field.get(actual);
 
                if (refValue != actValue) {
                    if (refValue == null || !refValue.equals(actValue)) {
                        return false;
                    }
                }
            }       
        return true;
    }

Hallmark of the Stupid: PropertyChangeEvent

What troubles me with java is, that the implementors of its standard library utterly failed at setting an example. Java developers don’t know how to do proper christian oo, because every time they peek into the library sources they see C-ish hacks.

Today I realised that PropertyChangeEvent is not implementing equals based on its value.
It is almost immutable, apart from a stupid field that has been added for “future use”. And there is a lot
to be said in favour of immutable objects with value semantics.

This is, what I was trying to do:

ValueHolder holder = new ValueHolder();
holder.setValue("Old");
 
final PropertyChangeListener listener = mock(PropertyChangeListener.class);
holder.addPropertyChangeListener(listener);
 
holder.setValue("New");
 
verify(listener).propertyChange(new PropertyChangeEvent(holder, "value", "Old", "New"));

It is going to be messy…

Spreadsheet Driven

I had a quite a few chats these days with people fro a more QAish background. Originally I intended just to look at selenium and webdriver. I showed some of the stuff, which I wrote in Java, to my QA colleagues. I used a demo webapp that implements a phonebook. My testcase just added a new contact and verified, that the user got a success message.

After they saw a bit of it, they asked me how they could separate out the testdata. I was a bit stunned, because, well it was all there – at least to me as a java developer. I had separated out everything into a method somewhat like this:

 createNewContact("Felix", "Leipold");

So I could test for other inputs as well:

 createNewContact("", "Invalid");
 createNewContact("Invalid", "");
 createNewContact("Holden", "Caulfield");

Obviously the expectation of QAs was to have an external file, e.g. a spreadsheet that feeds the data into the test code. I am very critical of this approach, but I felt like playing with junit 4 a bit. Junit 4 allows you to specify a runner four your test and indeed they have got one prepackaged that does parameterized tests. So I thought how about having a runner reads a spreadsheet and feeds the data into the test. As an example I use a very simplistic function:

    double add(double a, double b) {
        return a + b;
    }

So to test this I created this table and saved it as mathTest.xls:

Then I went on to create a test case, like this:

@RunWith(SpreadsheetDriven.class)
@Spreadsheet(name = "mathTest.xls")
public class MathTest {
    double a;
    double b;
    double expectedSum;
 
    public MathTest(Double a, Double b, Double expectedSum) {
        this.a = a;
        this.b = b;
        this.expectedSum = expectedSum;
    }
 
    @Test
    public void testAdd(){
        assertEquals(expectedSum, add(a,b));
 
    }
 
    double add(double a, double b) {
        return a + b;
 
    }
}

Essentially this thing parses the file specified in the annotation and passes it into the constructor. It’s not doing any fancy type conversions. All numerics are passed in as double, booleans as booleans, and everything else should be a string. Implementation and tests are to be found here and depend on apache poi being on the classpath for reading the excel file.

I am also quite proud of the reporting I get:

I am wondering, if all this is a good idea. But if it helps keeping people from using fit, it’s a start. What I realised by now is that actually table or data driven testing is much more appropriate for unit tests. Funnily enough that people actually do it the other way round. For unit tests I would actually recommend not to use a spreadsheet, but the original JUnit Parameterized runner, which lets you define your data like this:

    @Parameterized.Parameters
      public static List getParameters(){
        return Arrays.asList(new Object[][]
                {
                        {2,2,4},
                        {3,4,7},
                        {4,2,6},
                        {2,4,6}}
        );
    }