Java Core Dump/Thread Analysis (IBM JVM)
1. Java Core analysis is an incredibly useful tool to have in your toolbox. A customer running your Java application may encounter poor performance. Reproducing a performance issue, at times, can be very difficult due to the vast number of circumstances and environmental factors at play. Java core dump and thread analysis will prove useful when identifying bottlenecks, deadlocks, and resource contention.
1.1. Where do we start?
We start with a core dump. A core is typically generated by the following scenarios:
1. A JVM crash
JVM crashes will occur if an application instruction accessed bad/corrupted memory (Seg Fault), the stack pointer runs beyond the limits of the thread stack (Stack Overflow), or the JVM is unable to allocate more memory (OutOfMemory).
2. A core can also be provoked manually (assuming the JVM is configured to dump at user signal).
On Windows: Press CRTL+Break on the command window to generate the dumps.
On Linux: Press CTRL+\ on the shell window.
2. The Scenario
Let’s assume the following scenario:
A customer has filed a defect against one of our tools, a GUI application. The customer states that the application intermittently hangs for multiple seconds during various operations. The customer is unable to provide specific feedback about the hang, nor can they narrow the set of operations. It seems completely random and they cannot not identify a reproducible scenario.
For the support engineer or developer assigned to this task, the lack of detail makes it almost impossible to reproduce and attack.
In these situations, the support engineer or developer should ask the customer to obtain a core dump next time the hang occur. The core dump provides a great starting point toward debugging the issue.
2.1. Analyzing the Core
For this demonstration, we will use IBM’s Thread and Monitor Dump Analyzer. (download)
Note, although javacore is present in Oracle JVMs, much of the content in the javacore presented in the scenario is added by IBM and, therefore, is only present in IBM JVMs, thus this demonstration requires an IBM JVM. I will post a core analysis example using Oracle’s JVM soon.
Let’s debug this issue stated above.
First, launch the dump analysis tool
$JAVA_HOME/bin/java -jar jca447.jar
On the file menu, choose ‘Open Thread Dumps’
You will see the dumps listed in the table. The context menu will present a number of options. Let’s explorer ‘Thread Detail’ and ‘Monitor Detail’
2.2. Thread Detail
On the right side of the thread detail page, we see a summary of detected Deadlocked threads, threads waiting on conditions, etc.
On the left side, we see a listing of all active threads colored by thread status.
Based on the details below, we can quickly see that we have many threads waiting on condition and that the main thread is blocked. This is a GUI application.
By clicking on one of the threads, you will see a stack trace indicating what that thread was doing at the time the core was taken.
2.3. Monitor Detail
In the image below, we can see monitor details. In this particular case we see that main is blocked and its stack trace.
Using the information above, (and having a working knowledge of the application), a support engineer or developer can, at best, identify the cause of this thread hang, and at worst gain some idea of what the application was doing, so that a similar scenario can be replicated and debugged.
3. Next Steps
There are a variety of Java thread and dump analysis tools available for use. Explore them and find what works best for you.