Archive for the ‘Visionäre Projekte’ Category.

Getting Real with Enumerable.java

Two weeks back I posted a video explaining how to get started with Enumerable.java. While this might have been enough information to start playing, there is a lot more to consider when going with Enumerable.java for production use.

Over the past few weeks I went through that experience on my project and here are some of the practices that we found useful. Håkan also released version 0.3.0 that incorporates some of our learnings.

When working in the IDE we used the javaagent to weave the lambdas. However there were a few caveats. In version 0.2.4 you could only black list packages to prevent them from being weaved. In most cases it is easier to specify explicitly for which package to enable weaving. Also when using contemporary java goodness there is other parties doing byte code magic. We experienced these problems with spring’s scoped proxies, therefore we introduced another parameter to filter out classes by a regular expression applied to the full classname. Proxy classes usually come with a lot of dollar signs in their names.
So the vmargs we are using in development are:

-javaagent:lib/deploy/enumerable-agent-0.x.x.jar
-Dlambda.weaving.included.packages=felix.application
-Dlambda.weaving.exclude.pattern=\$\$

If you are using eclipse you can specify these parameters as default vm-parameters in your JRE definition:

While using an agent in the ide is acceptable it gets messy, when deploying the application. Essentially all the JVMs on the way to production including your application server would need the agent. What we did instead is, we instrumented our class files in our build using the AOT lambda compiler. So this is a single step after the compilation. If you are using ant it could look somewhat like this:

<target name="lambda" depends="compile">
        <java fork="true"
              classname="lambda.weaving.LambdaCompiler"
              failonerror="true">
 
              <sysproperty key="lambda.weaving.debug" value="true"></sysproperty>
                  <classpath>
                      <path refid="prod"/>
                  </classpath>
              <arg value="prod.jar"></arg>
        </java>
    </target>

Important: The lambda weaver needs debug information on local variables and also you need to encode your source files as utf-8 to get the fancy lambda letter, so perhaps you have to change your compile target like this:

<javac srcdir="src/java" 
            destdir="${basedir}/tempCompile" 
            classpathref="prod"
            encoding="utf8"
            debug="on"
            debuglevel="lines,vars,source"
/>

From a technical point of view this is all you need to do. However there is also the question of how to use the new feature “responsibly”. In our code base we used the generic CollectionUtils to filter and transform Collections. While they are elegant from a theoretical point of view, they are quite an insult to the eye, so our aim was to replace all these with Enumerable.java. Functional operations on collections are very well understood (the method names used in Enumerable.java go at least back to Smalltalk-80, that’s thirty years).

So the advice introduce your team the following methods and aim to restrict the use of lambda expressions to those:

  • select
  • collect
  • find
  • inject
  • groupBy
  • While there is remarkable support for arrays and primitives, try to stick to
    Collections and proper Objects.

I think these alone justify the investment.
Things you should avoid or do only in after proper consideration:

  • Do not declare parameters of function types
  • Do not declare local variables of function types
  • Do not declare fields of function types
  • Do not modify enclosed state from within the closure, i.e. local variables or fields
  • Generally avoid side effects

The typical transformation we did, looked something like this:

    private Set<Order.Status> retrieveOrderStatuses() {
        Set<Order.Status> statuses = new HashSet<Order.Status>();
        if (orders == null) return statuses;
 
        for (Order  order : orders) {
            statuses.add(order.getStatus());
        }
        return statuses;
    }

becomes:

    @LambdaParameter private static Order o;
    private Set<Order.Status> retrieveOrderStatuses() {
        if (orders == null) return emptySet();
        return  collect(orders, λ(o, o.getStatus())).toSet();
    }

Because getting the lambda character (you might also want to use the alias fn) is a bit tricky you might want to use this java editor template for eclipse that also sorts out the static import:

λ(${impst:importStatic('lambda.Lambda.*')}${cursor})

Sticking to these simple rules actually helped us getting away from the anonymous inner classes and CollectionUtils and that was an easy sell to the mostly mathematically inclined team. In my opinion the fact that lambdas are restricted to expressions is actually a good thing.

Get Closures for Java Now!

The general state of affairs at Sun/ Oracle is very sad. If you are like me, you cannot take this much longer. Fortunately enough my esteemed colleague Håkan Råberg has invented what is at least a molotov cocktail perhaps even a guillotine to the revolution of java software development. Yes, I am talking about the modestly named Enumerable.java, which brings the Ruby enumerable module and closures to java. Without a special compiler, just by pushing the boundaries and enhancing byte code.

I created this little screencast, that shows how to get up and running with Enumerable.java:

There is also a hi-res version of the videos on google docs (approx. 20M download size).

For more examples and explanation watch Håkan’s blog and read the source.

Now, go, download and, enjoy!

Visualising log files with gnuplot

I recently had the pleasure of supporting a new system throughout its first month of production. This was a good opportunity to refresh my command line skills. As it happened I spent a lot of time looking at log files trying to figure out what happened to the productions system. I figured, that a graphical representation of the events would be nice and started using gnuplot.

First I started out with a bunch of bash scripts, using what your usual unix installation provides, but then I actually came up with some groovy scripts to provide better abstractions. A log file generally looks somewhat like this:

04/01/1970 07:55:13 garbage
04/01/1970 09:27:48 Event 2
04/01/1970 10:01:28 garbage
04/01/1970 10:38:30 garbage
04/01/1970 10:48:36 garbage
04/01/1970 10:51:58 Event 2
04/01/1970 11:03:45 garbage
04/01/1970 11:34:03 Event 1
04/01/1970 12:24:33 garbage
05/01/1970 04:35:50 ERROR

There is a lot of garbage plus some events we might be interested in. It allows to specify events, e.g. by providing a regexp:

Event EVENT1 = new RegExEvent("Event 1", ~/Event 1/)
Event EVENT2 = new RegExEvent("Event 2", ~/Event 2/)
Event ERROR = new RegExEvent("Error", ~/ERROR/)

The next step is newing up a TimeLineVisualizer on these events and passing in a stream with the actual log:

def logFile = new File("test.log");
 
logFile.withInputStream {InputStream stream ->
  def visualizer = new TimeLineVisualizer([
          EVENT1,
          EVENT2,
          ERROR
  ]);
  visualizer.visualize(stream)
}

If you have the gnuplot binary on your path this will yield something like this:

timeline

Also in some cases you would like to know which time of day events are most likely to happen. For producing histograms I created another visualizer (which currently takes only one event).

logFile.withInputStream {InputStream stream ->
  def visualizer = new HistogramVisualizer(EVENT2, HistogramVisualizer.HOUR_OF_DAY_BINS)
  visualizer.visualize(stream)
}

For the example log file, which unfortunately has an even distribution of events, we get this:

histogram

The cool thing about gnuplot is, that you can actually run these things in a cron job to produce daily reports (and mail them to the appropriate people) or on a continuous integration server to visualise how the system is being exercised by the test suite.

Geld sparen und die Welt verbessern

Im Grunde meines Herzens finde ich ja die folkloristischen Umtriebe der katholischen Kirche ganz lustig. Wenn das ganze aber in Richtung strafrechtliche Relevanz geht, hört der Spaß wohl doch auf.

Ein kurzer Besuch beim Bund der Steuerzahler brachte folgende Erkenntnis: Wer in Hessen wohnt und 30.000,- EUR im Jahr verdient, zahlt im Jahr gut 400 EUR Kirchensteuer, bei 35.000,- sind es schon gut 550 EUR und bei 45.000,- EUR stolze 850 EUR. Das Geld kann man sicher besser anlegen. Der Mitgliedsbeitrag der Humanistischen Union beträgt beispielsweise nur 90 EUR. Die HU setzt sich neben Bürgerrechten im allgemeinen auch für eine klare Trennung von Staat und Kirche ein.

Update: Die HU hat heute eine Pressemitteilung zum Thema veröffentlicht.

Gegen das Ignorabimus

Musste bei Anekas Kommentar an Hilbergs Epitaph denken:

Wir müssen wissen, und wir werden wissen.

Schöner Spruch und die Tatsache, dass das Hilbertprogramm ja eigentlich gescheitert ist, tut dem keinen Abbruch, sondern macht es umso wichtiger weiterzusuchen.

swivel.com

Today I stumbled accross swivel.com. It’s supposed to be a YouTube for Data. Quite interesting, though I had problems figuring out how it actually works.

DabbleDB – Excel Killer

Today I finally signed up for a free trial of dabbledb. It’s a web-wased Database App. It supports multiple users and has sophisticated data import and normalization options. It is aimed at all the non-numbercrunching stuff that is done in excel-sheets, that are mailed around to organize or track things. I tried to model a small business and I was pretty successful. It supports various export media and formats, such as public web-sites, rss, or CSV.

I currently see two problems that might prevent dabble from catching on. The pricing scheme is a bit weird – just number of users and apps is considered, while the volume is irrelevant. As dabble targets collaborative activities the number of users is crucial – even if the volume/ number of transactions is very small. Other thing is the missing localization. Mircrosoft has spoilt us by providing germanized Apps.

Verdict: Dabble is very well suited for stuff we currently do using excel or custom build web apps. Make sure you check out the 7min screencast!

Interessanter Vorschlag

Thüringens Ministerpräsident Althaus schlägt eine negative Einkommenssteuer zur Rettung des Sozialstaats vor. Das ist eine gute Idee, weiter so! Dass es sich um neue Wege handeln würde, ist jetzt etwas übertrieben. Bekannt ist das schon seit 40 Jahren. Dringend nachzulesen in Capitalism and Freedom! Dass die CDU und die SPD sich sowas nicht trauen, liegt leider auf der Hand…

Tuesday Night Skating

War gestern mit Arne beim TNS. Waren leider etwas zu spät am Treffpunkt, aber haben mithilfe aufmerksam beobachtender Radfahrer den Pulk doch noch erreicht. Im Moment wird das TNS Sommerspecial angeboten. Das heißt zuerst 25km gemütliches Tempo und danach nochmal gut 10km in sportlichem Schritt. Die sportliche Runde ging die Darmstäder Landstraße raus nach Neu-Isenburg und dann durch die Isenburger Schneise zurück. Hat wie immer großen Spaß gemacht.

Mir Hawwe Gewonne

Beim Datterich Ultra 2006 haben die “moralischen Sieger” einen respektablen 36. Platz erreicht. Es war wie immer großartig sich in Darmstadt zu treffen und gemeinsam Sport zu treiben!