Tuesday, November 07, 2006

Java 5 random VM crashes

We are currently evaluating the consequences of migrating our application from Java 1.4 to Java 5. While initial tests revealed only simple issues (like variables called enum etc.) we are now seeing a much more severe problem: Random VM crashes.

Currently we only see this on Linux (Kernel 2.4) only, however even there we cannot reliably reproduce the problem. On a single machine we have seen two crashes in a week. Notably the application was not being used, it was just started and waiting for user input. Some background threads are running in this situation, however they do not do any work, either. They just poll some database tables for external changes, but there were none.

All of a sudden a VM would crash, leaving a hs_err_pid1234.txt behind. This is what they look like (shortened):

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x402989b9, pid=8736, tid=1094691632
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_09-b03 mixed mode, sharing)
# Problematic frame:
# V  [libjvm.so+0x2689b9]
#

---------------  T H R E A D  ---------------

Current thread (0x08099230):  VMThread [id=8737]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000008

Registers:
EAX=0x00000000, EBX=0x403b7aec, ECX=0x00000008, EDX=0x6861ce38
ESP=0x413fa3d0, EBP=0x413fa3e0, ESI=0x403aa980, EDI=0x403c5ac8
EIP=0x402989b9, CR2=0x00000008, EFLAGS=0x00010206

Top of Stack: (sp=0x413fa3d0)
0x413fa3d0:   0806c558 0806c558 403b7aec 403af5c4
0x413fa3e0:   413fa400 40298ccc 403af5c4 413fa448
0x413fa3f0:   413fa410 40329d7c 413fa448 403b7aec
0x413fa400:   413fa430 40334754 403c5ac8 403af5c4
0x413fa410:   413fa4f0 403262a1 413fa448 413fa454
0x413fa420:   403630e3 403b7aec 00000002 0806c438
0x413fa430:   413fa470 40170ff0 403c5ac8 00000000
0x413fa440:   00000001 00000001 42133220 00000001 

Instructions: (pc=0x402989b9)
0x402989a9:   08 49 89 08 8b 40 08 8b 14 88 8b 42 04 8d 48 08
0x402989b9:   8b 40 08 52 51 ff 50 58 8b 06 83 c4 10 8b 08 85 

Stack: [0x4137a000,0x413fb000),  sp=0x413fa3d0,  free space=512k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x2689b9]
V  [libjvm.so+0x268ccc]
V  [libjvm.so+0x304754]
V  [libjvm.so+0x140ff0]
V  [libjvm.so+0x1438d8]
V  [libjvm.so+0x143122]
V  [libjvm.so+0x14c8ad]
V  [libjvm.so+0x2f0dce]
V  [libjvm.so+0x1409ff]
V  [libjvm.so+0x141c7f]
V  [libjvm.so+0x32df33]
V  [libjvm.so+0x32dad6]
V  [libjvm.so+0x32d0e7]
V  [libjvm.so+0x32d355]
V  [libjvm.so+0x32cec0]
V  [libjvm.so+0x28bbe8]
C  [libpthread.so.0+0x4484]

VM_Operation (0x4a74d258): full generation collection, mode: safepoint, requested by thread 0x0815d230
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGSEGV (0xb) at pc=0x40189396, pid=18790, tid=1094691632
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_09-b03 mixed mode, sharing)
# Problematic frame:
# V  [libjvm.so+0x159396]
#

---------------  T H R E A D  ---------------

Current thread (0x08099230):  VMThread [id=18791]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x59c83398

Registers:
EAX=0x59c83398, EBX=0x403b7aec, ECX=0x6eb6f518, EDX=0x5bc99a08
ESP=0x413fa0a8, EBP=0x413fa0c0, ESI=0x5bc99a04, EDI=0x6eb6f520
EIP=0x40189396, CR2=0x59c83398, EFLAGS=0x00010202

Top of Stack: (sp=0x413fa0a8)
0x413fa0a8:   6afc194c 5bc99a08 6eb6f524 403b7aec
0x413fa0b8:   403aa980 403c5ac8 413fa0e0 402989c1
0x413fa0c8:   6eb6f288 5bc999f8 0806c558 0806c558
0x413fa0d8:   403b7aec 403af5c4 413fa100 40298ccc
0x413fa0e8:   403af5c4 413fa148 413fa110 40329d7c
0x413fa0f8:   413fa148 403b7aec 413fa130 40334754
0x413fa108:   403c5ac8 403af5c4 413fa1f0 403262a1
0x413fa118:   413fa148 413fa154 403630e3 403b7aec 

Instructions: (pc=0x40189396)
0x40189386:   8d 14 86 89 55 ec 39 d6 73 18 8b 06 85 c0 74 0a
0x40189396:   8b 00 83 e0 03 83 f8 03 75 18 83 c6 04 3b 75 ec 

Stack: [0x4137a000,0x413fb000),  sp=0x413fa0a8,  free space=512k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x159396]
V  [libjvm.so+0x2689c1]
V  [libjvm.so+0x268ccc]
V  [libjvm.so+0x304754]
V  [libjvm.so+0x140ff0]
V  [libjvm.so+0x1438d8]
V  [libjvm.so+0x143122]
V  [libjvm.so+0x14c8ad]
V  [libjvm.so+0x2f0dce]
V  [libjvm.so+0x1409ff]
V  [libjvm.so+0x141c7f]
V  [libjvm.so+0x32df33]
V  [libjvm.so+0x32dad6]
V  [libjvm.so+0x32d0e7]
V  [libjvm.so+0x32d355]
V  [libjvm.so+0x32cec0]
V  [libjvm.so+0x28bbe8]
C  [libpthread.so.0+0x4484]

VM_Operation (0x48f8e4d8): full generation collection, mode: safepoint, requested by thread 0x08653a08

Looking through the Sun bug database I found several reports about similar crashes, however they were all closed as not reproducible. This is our problem, too. Right now the application has been running for 5 days without a problem. Nevertheless this is not too comforting, as we would have several thousand VMs in production use. Should we decide to migrate even a chance of 0.1% for this crash would leave us with several problem reports a day which we cannot accept.

Any comments and hints will be greatly appreciated.

2 comments:

markevans48@hotmail.com said...

hello there! seem to be having a similar problem on elderly laptop with new hard drive ... just randomly freezing ... most annoying!

enclosed is (part of)the text file generated
An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x3fbd2f1a, pid=2116, tid=3388
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_09-b03 mixed mode)
# Problematic frame:
# C 0x3fbd2f1a

Christof said...

Hey Daniel, alter B3-Veteran!

dieses Drecks-Java macht bei mir auch Probleme, allerdings mit 2.6er kernel unter ubuntu6.10. Desweiteren sind die crashes reproduzierbar, immer in der libglibjni-0.4. Ich bin hier noch am Durchdrehen, kauf mir bald nen Mac..

Lösung hab ich natürlich auch keine, haha :) Aber nen guten blog hast Du da!

Grüße auch an Udo!
Christof