Hadoop 1.0

There is finally a 1.0 version of hadoop. One of my biggest complaints using hadoop since version 0.10 is that something breaks with every release and the usual retort is that compatibility will come with 1.0, and it seemed like 1.0 was just around the corner for years. I haven't been using hadoop as much for the last six months, so I didn't notice that there was finally a 1.0 release until today even though it was announced towards the end of last year. As a user this news inspires some hope that there might be a hadoop upgrade that just works. We'll have to see as new versions come out.

That said, it is a little dissapointing to see some of the cruft that is still around in the APIs. In particular org.apache.hadoop.mapred
and org.apache.hadoop.mapreduce packages are both still around. So if I want to write a map reduce job which API should I use? Even worse the javadocs still don't provide much clarity on which of these APIs should be preferred and neither appears to be deprecated. Maybe they have good reasons, but this adds a lot of confusion for users and, in my selfish opinion, should have been cleaned up before a 1.0 release.

The other big problem I've had with hadoop is trying to keep a stack of hadoop, pig, oozie, hbase, etc all working together flawlessly and being able to update individual components without too much worry on whether the rest of the stack will play nice. This is much easier to do if hadoop provides clean, simple, and well documented APIs that these other tools can build on. At first glance, the 1.0 apidocs look like they just slapped a 1.0 label on the current pile of crud and did not remove any of the old garbage and deprecated APIs.

If they actually maintain compatibility between 1.x releases it will still be a win for users, but hopefully for 2.0 they focus on a clean simple API for users and get rid of a bunch of old cruft. It would also be nice if we don't have to wait 6 years for 2.0.

Damn Java Socket Exception Messages

When creating an error message you should think about what information would be useful for understanding what went wrong. This should especially be true if you are creating a library that is likely to be used by many other systems. Providing good error messages up front means that even if the programmers using the library do not check and customize the messages, the user will still get a reasonable result. For some use cases, such as scripting, it can also be useful because it may be a quick one-off program where the goal is to quickly automate a repetitive task or perform some analysis. In this case, having useful default error messages can speed up the initial development so you get your answer faster.

In my case, I was checking logs for a system that crawls pages and hence attempts to resolve and connect to thousands of hosts. This system logs the exceptions, but unfortunately did not provide a customized message when doing so. Analyzing the logs showed that the two most common error messages were: 1) a failure to resolve the hostname and 2) failing to connect to an HTTP server on the host. An example error message for the first case is java.net.UnknownHostException: some-host-that-does-not-exist. This message is quite useful as the exception name explains the problem and the message tells me the name of the host that could not be resolved. An example message for the second case is java.net.SocketTimeoutException: connect timed out. This message is explains the problem, but doesn't given me the crucial information of what it was trying to connect to.

Though this can easily be fixed in the application code, it is disappointing that the default message is so bad. I have noticed that my opinion of a programming language or technology seems to go down steadily the longer I am forced to use it at work. Is the grass greener on one of the other sides? How do other languages, or rather the networking libraries they provide, fair for this use case? I looked at 13 options to see how many would give a decent error message for both use cases. The results were not very encouraging. Only one option, Go, had reasonable messages for both. For the host not found case 4 options included the hostname. Only two options provided the host and port in the failed to connect case. The results are summarized in the table below the fold with links to the source code and raw error messages.

InputStreamReader and invalid UTF-8

While writing some unit tests for some code to read in UTF-8 encoded data, I was surprised to find that the java InputStreamReader class did not throw an exception with bad data. Consider the following example program:

import java.io.*;

public class IConv {
    public static void main(String[] args) throws IOException {
        if (args.length != 2) {
            System.err.println("Usage: iconv <from> <to>");
            System.exit(1);
        }

        Reader input = new InputStreamReader(System.in, args[0]);
        Writer output = new OutputStreamWriter(System.out, args[1]);
        char[] buffer = new char[4096];
        int length;

        while ((length = input.read(buffer)) != -1) {
            output.write(buffer, 0, length);
        }

        output.flush();
    }
}

For correct UTF-8 input it does what you would expect:

$ printf "\xE2\x82\xAC\n" | iconv -f utf-8 -t utf-8 | od -t x1
0000000 e2 82 ac 0a
0000004
$ printf "\xE2\x82\xAC\n" | java IConv utf-8 utf-8 | od -t x1
0000000 e2 82 ac 0a
0000004

However, if you feed it bad data:

$ printf "\xE2\x82\xFC\n" | iconv -f utf-8 -t utf-8 | od -t x1
iconv: illegal input sequence at position 0
0000000

$ printf "\xE2\x82\xFC\n" | java IConv utf-8 utf-8 | od -t x1
0000000 ef bf bd ef bf bd
0000006

The unix iconv tool gives an error as expected. The java version is printing out the unicode replacement character twice. According to the javadocs for CharsetDecoder, the default behavior is supposed to be to report the error:

How a decoding error is handled depends upon the action requested for that type of error, which is described by an instance of the CodingErrorAction class. The possible error actions are to ignore the erroneous input, report the error to the invoker via the returned CoderResult object, or replace the erroneous input with the current value of the replacement string. The replacement has the initial value "\uFFFD"; its value may be changed via the replaceWith method.

The default action for malformed-input and unmappable-character errors is to report them. The malformed-input error action may be changed via the onMalformedInput method; the unmappable-character action may be changed via the onUnmappableCharacter method.

Looking at the javadocs for InputStreamReader, I didn't see any indication that it deviates from the the default behavior. But sure enough, look at the InputStreamReader source code and it creates a decoder using CodingErrorAction.REPLACE. One caveat, I used Sun/Oracle's java 6 VM for testing, but the source code I looked at was for the Apache Harmony VM because it conveniently came up in the search results. So now that we know why it is happening, we can fix it to throw an exception by explicitly specifying a CharsetDecoder configured to use CodingErrorAction.REPORT:

import java.io.*;
import java.nio.charset.*;

public class IConv {

    private static CharsetDecoder decoder(String encoding) {
        return Charset.forName(encoding).newDecoder()
            .onMalformedInput(CodingErrorAction.REPORT)
            .onUnmappableCharacter(CodingErrorAction.REPORT);
    }

    private static CharsetEncoder encoder(String encoding) {
        return Charset.forName(encoding).newEncoder()
            .onMalformedInput(CodingErrorAction.REPORT)
            .onUnmappableCharacter(CodingErrorAction.REPORT);
    }

    public static void main(String[] args) throws IOException {
        if (args.length != 2) {
            System.err.println("Usage: iconv <from> <to>");
            System.exit(1);
        }

        Reader input = new InputStreamReader(System.in, decoder(args[0]));
        Writer output = new OutputStreamWriter(System.out, encoder(args[1]));
        char[] buffer = new char[4096];
        int length;

        while ((length = input.read(buffer)) != -1) {
            output.write(buffer, 0, length);
        }

        output.flush();
    }
}

Now, if the conversion is not possible it will fail loudly with an exception:

$ printf "\xE2\x82\xFC\n" | java IConv utf-8 utf-8 | od -t x1
Exception in thread "main" java.nio.charset.MalformedInputException: Input length = 2
        at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:319)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
        at java.io.InputStreamReader.read(InputStreamReader.java:167)
        at java.io.Reader.read(Reader.java:123)
        at IConv.main(IConv.java:29)
0000000

$ printf "\xE2\x82\xAC\n" | java IConv utf-8 iso-8859-1 | od -t x1
Exception in thread "main" java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:261)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:266)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
        at IConv.main(IConv.java:30)
0000000

Damn Log4j

While debugging a problem, I was annoyed to find that the logs just contained the name of the exception and not the stack trace. The code in question was essentially:

try {
    doSomething();
} catch (Throwable t) {
    _logger.warn(t);
}

The problem is that the Logger.warn(Object) method of log4j accepts an object and not a string so the code compiles without issue. The correct method for exceptions is Logger.warn(Object,Throwable). Having a method in java that accepts a parameter of type Object should be done with caution as anything (even primitive types due to autoboxing) can be passed in with no compile time validation. Curious, I decided to look at what other logging libraries chose to do.

• java.util.logging.Logger: uses a string for the message.
• org.apache.commons.logging.Log: uses an object for the message.
• org.slf4j.Logger: uses a string for the message.

But, Eclipse makes it easy...

Charles Petzold has nice article titled Does Visual Studio Rot the Mind?. I was thinking about this article when doing a code review and frequently getting the excuse that the awful code was ok because "Eclipse makes it easy." The following snippet shows some of the problems:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

package example; import java.util.List; import scala.actors.threadpool.Arrays; /** * Immutable class for ... */ public class Example { private List<String> mData; private int mCount; public Example(String[] data, int count) { mData = Arrays.asList(data); mCount = count; } public String[] getmData() { return mData.toArray(new String[mData.size()]); } public void setmData(String[] mData) { this.mData = Arrays.asList(mData); } public int getmCount() { return mCount; } public void setmCount(int mCount) { this.mCount = mCount; } }

Here is a brief look at some of the questions and responses.

• Why do the accessor/mutator method names have the "m" prefix (by convention our code uses a prefix of "m" on member variable names, but it shouldn't be reflected on the accessor methods)?

  Eclipse generated the setters/getters. It makes it easy to generate them this way.

• The comments say the class is immutable, why are there mutator methods?

  Eclipse generated the setters/getters.

• The class is not immutable.

  Then why would Eclipse generate the setters?

• Why are you using scala.actors.threadpool.Arrays instead of java.util.Arrays?

  Eclipse generated the imports.

It was clear from his answers that he barely looked at the actual code. Eclipse enabled him to cobble together some crap that compiled without really knowing what he was doing. This is not really the problem with the IDE, idiots can write code without the aid of an IDE. What really surprised me is another senior developer on the team supposedly reviewed the code and found nothing wrong. When I asked what he checked the reply was "I brought it up in Eclipse and didn't see any warnings or errors flagged on the code."

Maven Review

At work, I recently started working on a Java project and had to use Maven as it is the standard build tool for Java projects in the company. I have mostly worked on C/C++ projects for several years and recall that Ant was the de-facto standard Java build system. This post is a quick review of my experience trying Maven.