JVM Crash/Core Dump Analysis 1

JVM CRASH

 An application typically crashes when it performs an operation which is not allowed by the operating system. The operating system then triggers an exception or signal in the application. Unix applications traditionally responded to the signal by dumping core.

JVM crash is happening from following components :

1. JVM internal code – example GC

The crash occurred during garbage collection, indicating a possible garbage collection fault.  The crash occurred after garbage collection, indicating a possible memory corruption

2. JVM JIT generated code, internal JIT code.

Failure in the native JIT code during compilation of java bytecodes. Valid bytecodes might compile into invalid native code – causing the Java program to fail.

3. Application JNI code

Fault in native code if the application has any JNI code or uses any third-party packages that use JNI code (for example, JDBC drivers, MQ libraries and HPROF Profiling plug-ins ..etc).

Generally JVM is crashing under the following scenarios

1. Segmentation Violation : When the instruction of an application accesses a bad/corrupt memory

address

2.             Native Stack Overflow  : Stack Pointer runs beyond the limits of the Thread stack
3.             Running out of native memory : Out of Memory (unable to allocate memory via malloc)

First thing to check on a JVM crash

In general, for the JVM crash, first check the application logs. If a JVM error message is part of the logs then a Java Virtual Machine error has occurred. After checking logs, determine whether or not the hardware or operating system is causing the problem. Look at the logs of the local machine and determine whether the machine has had a history of restarts, kernel panics, process segmentation faults and so forth. On Unix machines, system logs can be viewed using the dmesg command or by viewing logs in /var/logs. If nothing is there in the log check if any core dump file is generated. Then try to analyze the core dump using various tools like (gdb, dbx etc).

Locating the fatal error log

Locating the fatal error log is also a very important step for diagnosing a JVM crash. The fatal error log is a file named hs_err_pid<pid>.log where <pid> is the process id of the process. View this file and try to determine which part of the JVM caused the crash. This may provide clues as to how to resolve the situation.

Sample crashes

This section shows a number of examples and explains how the error logs are used to suggest the cause of a JVM crash.

A JVM crash in the hotspot compiler thread

If the “Current thread” is a JavaThread “CompilerThread0″, there might be possible a compiler bug that you have encountered. Here is an example of a such a crash:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  Internal Error (/home/chrisphi/east/ws/6415406/ws5u15/hotspot/src/share/vm/opto/loopopts.cpp, 850), pid=21426, tid=11
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_15ea_TEST_150_15ea+cr6415407_01_chrisphi_2007.10.12_11:18-debug mixed mode)
#
# Error: assert(dom_depth(n_ctrl) <= dom_depth(x_ctrl),”n is later than its clone”)
—————  T H R E A D  —————

Current thread (0x082088e0):  JavaThread “CompilerThread0” daemon [_thread_in_native, id=11]

It might be possible to temporarily workaround the issue by switching the compiler to run the HotSpot Client VM, or excluding the method that provokes the JVM crash from compilation if there is one.

If that is not working try with -Xint option to turn off JIT.

A JVM crash in native code

If the fatal error log indicates the JVM crash was in a native library (i.e. the problematic frame is marked with a “C”),, there might be a bug in native/JNI library code. For example, consider the following extract from the header of a fatal error log:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  SIGBUS (0×7) at pc=0xf73d7a62, pid=19500, tid=688157
#
# Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode)
# Problematic frame:
# C  [libzip.so+0xfa62]
#

—————  T H R E A D  —————

Current thread (0xcab30a28):  JavaThread “StageWorker/0″ [_thread_in_native, id=19627]
…
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libzip.so+0xfa62]
C  [libzip.so+0x10124]  ZIP_GetEntry+0xd8
C  [libzip.so+0x36b9]  Java_java_util_zip_ZipFile_getEntry+0x14b
…
c5ae4000-c5bae000 r-xs 00000000 00:15 65848888  /home/rhino/work/deployments/101/unit1143400190944/.nfs0000000003ecc63800000010
c5bae000-c5c00000 r-xs 00000000 00:15 65849319  /home/rhino/work/deployments/101/unit1143400190944/.nfs0000000003ecc7e700000011

In this case a SIGBUS arose with a thread executing in the library libzip.so.

In general, if you get a JVM crash in a native library, please collect the following information to analyse the root cause of the crash.

• Rerun with the -Xcheck:jni option added to the command line.
• Locate the core file if it is available. Attach the native debugger (dbx, dgb on Solaris) to the core file.
• Locate the fatal error log.

A JVM crash because of segmentation fault

#
# An unexpected error has been detected by Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d855b53, pid=9864, tid=10268
#
# Java VM: Java HotSpot(TM) Client VM (10.0-b23 mixed mode windows-x86)
# Problematic frame:
# V [jvm.dll+0x95b53]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp
#

In General this problem is occuring beause of memory is not sufficient.

• Tune your java application with JVM parameters. (-Xmx, -Xms, -Xmn, -Xss,  GC algorithms ,etc).
• Obeserve JVM through Jconsole.

Guideline to approch JVM crash

Some Quick Tips

• Update your java to latest stable version
• Check your JVM settings
• Check OS parameters (eg: ulimt)
• Try to disable JIT ( -Xint)
• Try to debug with (-Xcheck:jni) to check your JNI code

Note:    In IBM java enable Xdump agent to capture stack, system, heap, java and snap.

Eg:  -Xdump:java+heap+stack+system:events=abortuser

Useful Tools

• gdb
• dbx
• dgb
• jstack
• pstack

IBM Specific tools

• jextract
• IBM Support Assistant (ISA) with Dump analyzer tool

References:
 
http://publib.boulder.ibm.com/infocenter/javasdk/v1r4m2/index.jsp?topic=/com.ibm.java.doc.diagnostics.142/html/contents.html

http://www-01.ibm.com/support/docview.wss?uid=swg27015688&aid=1

http://java.sun.com/j2se/1.5/pdf/jdk50_ts_guide.pdf

data to collect on AIX when application crashes

http://www-01.ibm.com/support/docview.wss?uid=isg3T1011865#1