Ever since we switched out build process from Java 1.4 to Java 5 (1.5.0_09) we have seen FindBugs crash with an OutOfMemoryError when started from ant. The whole thing is running under RedHat Enterprise Linux 4. The output is always the same:
[findbugs] Running FindBugs... [findbugs] Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested 134217736 bytes for Chunk::new. Out of swap space?
Our first attempts to increase the heap size with the -Xmx VM parameter did not help. We suspected the newer FindBugs release we had installed roughly at the same time, but this turned out to be wrong, because newer and older versions showed the same behaviour.
Armed with the fresh knowledge about the SAP Memory Analyzer just learned at jax.07 I tried to get a heap dump to try and find out what was causing the problem. Unfortunately the VM ignored my
-XX:+HeapDumpOnOutOfMemoryError option completely (on Sun's VM Options page there it says
-XX:-Heap... (with a minus), tried that, too).
Getting suspicious I realized that usually Java 5 VMs say something like
OutOfMemoryError: Java Heap Space which was missing in this case.
I had never before seen that strange "CompilerThread0" in an error message, same for the "requested xx bytes for Chunk::new. Out of swap space?" part. Looking around I stumbled across this bug report (and some more, all related to this one - just follow the related bugs list): Bug #6487381 "Additional path for 5.0 jvm crash on exhaustion of CodeBuffer".
Apparently something goes wrong when the HotSpot JIT tries to compile something to native code for faster execution. One of the other bugs you can find by following the link above mentions that there is problem with situations where there is not enough room for further compiles in what is called the CodeBuffer: instead of failing gently and just continuing to run the application without compiling or throwing some older compiled code away the VM would just crash. This should have been fixed with Java 5.0, but apparently there is another code path in the VM that will cause a hard failure. (I wonder what may be requesting another 128MB(!), but anyways...)
From what the bug says this issue will be fixed with 5.0u12, however 5.0u11 is the most recent release to download, so no help there.
Attempting to find whether it might be a single method that was causing the crash reproducibly I tried the
-XX:+PrintCompilation option. It produced lots and lots of output about compiled methods. A "guide" to its output can be found here: decyphering -XX:+PrintCompilation output.
As one might have expected, this did not reveal anything useful either. So the last resort was to switch to the Client VM which managed to complete the FindBugs run. In retrospect this explains why we were able to build without any problems on a Windows machine - the Client VM is default there unless you explicitly request the server.
For now we will leave it at that and try again as soon as 5.0u12 comes out.