How to Generate a JSTACK (and When to Run It)

When troubleshooting a “sick” JVM (high CPU, stuck threads, hangs), a jstack thread dump is one of the most useful artifacts you can capture. This article explains what jstack is, when to run it, and how to collect it safely in a RadiantOne/FID environment.

What jstack Does

• jstack attaches to a running Java process and prints stack traces for all Java threads in that JVM.
• The output shows, for each thread, which classes and methods are currently executing, including lock and wait states when used with options such as -l.
• It is a snapshot in time: it does not persist in memory after the process restarts, and it does not retroactively show what happened before the dump.
• Because of this, jstack is most valuable while the JVM is actually exhibiting the problem (for example, high CPU, long GC pauses, hangs), not after everything has already restarted and recovered.

When Is the Best Time to Run jstack?

• Run jstack before bringing the server or service down, otherwise you risk completely missing the problematic state.
• Once the JVM process has exited and been restarted, the previous state is gone; a new JVM PID means a new, clean runtime, with no direct memory or thread-state visibility into what just happened.
• A post-restart jstack can still be useful for baseline comparisons, but it will not capture the original error condition (for example, hung threads, deadlocks, runaway CPU) that occurred before the crash.
• In practical terms: if FID/VDS is “sick” but still up (memory high, CPU pegged, connections backing up), capture jstack immediately, then perform your controlled restart if needed.

How to Generate a jstack for FID/VDS on Linux

1. Ensure the Service Is Running

• You must have a live Java process to attach to.
• Confirm that the RadiantOne/FID service (VDS) is up and listening on its LDAP/LDAPS port.

2. Find the VDS JVM PID

Use one of these commands (replace <PORT> with your VDS port, for example, 2389):

lsof -i :<PORT>
Example:
lsof -i :2389

or

sudo ss -tulnp | grep :<PORT>
Example:
sudo ss -tulnp | grep :2389

From the output, note the PID associated with the VDS process.

Example lsof output:
COMMAND   PID   USER   FD   TYPE  DEVICE  SIZE/OFF  NODE  NAME
node      56789 user   3u   IPv4  12345   0t0       TCP   *:2389 (LISTEN)

Here, 56789 is the PID.

3. Run jstack Against That PID

From the bundled JDK under your VDS installation (path can vary, adjust as needed):

cd $RLI_HOME/vds/jdk/bin
./jstack <vds-pid> > jstack1.txt

Example:
./jstack 56789 > jstack1.txt

This writes a full thread dump to jstack1.txt in the current directory.

If you need multiple snapshots, you can repeat every few seconds with unique filenames, for example:

./jstack 56789 > jstack_1.txt
sleep 5
./jstack 56789 > jstack_2.txt
sleep 5
./jstack 56789 > jstack_3.txt

Shortly spaced dumps help you see whether threads are truly stuck or just transiently busy.

4. Collect Supporting Logs

Along with the jstack file, gather:

• Thread dump file: vds/jdk/bin/jstack1.txt (or your chosen path).
• VDS server access log: /vds_server/logs/vds_server_access.log (at least 2 hours around the event).
• VDS server log: $RLI_HOME/vds_server/logs/vds_server.log (enable Debug level if Support requests it).

These give context (load, errors, connection patterns) alongside the thread dump.

Monitoring and Early Detection

• To avoid missing important thread dumps, detect issues while the JVM is still running but already behaving abnormally.
• Rather than waiting for the process to terminate, ensure that monitoring and alerting are in place so you are notified when the node or service becomes degraded (for example, rising CPU, memory pressure, slow responses, or failed operations).
• With these alerts, you can capture one or more jstack thread dumps while the process is still alive and exhibiting the issue.

Automatic Thread Dumps with JStackMonitorTask (RadiantOne)

In RadiantOne deployments, VDS includes a built-in task that can automatically capture thread dumps when the service appears unresponsive.

• When VDS starts, a JStackMonitorTask is registered with the internal scheduler.
• The task runs periodically (every 5 seconds by default) and:
  • Attempts a TCP connection to the VDS LDAP/LDAPS port.
  • Performs a bind on that port.
• If the TCP connection succeeds but the bind fails, this indicates that the instance is reachable but not healthy. In that case, the task:
  • Executes the jstack command against the VDS JVM.
  • Writes a thread dump file to the instance logs directory.
• After creating a thread dump, the task waits for a configured period (5 minutes by default) before running again, to prevent excessive dump generation.
• Thread dump files use a timestamped naming pattern such as:
  RLI_HOME/INSTANCE_NAME/logs/thread_dumpYYYY-MM-DDTHH_MM_SS_SSS.log

You can tune this behavior using the scheduler properties file:

• Configuration file:
  $RLI_HOME/vds_server/conf/scheduler/JStackMonitorTask.properties
• Common properties:
  • interval=5 – frequency (in seconds) for the health check.
  • sleep.time.in.minutes=5 – cooldown period after a dump.
  • enabled=true – enables or disables the task.
• The properties file also maintains internal flags and runtime details such as the monitored PID, last start time, and next scheduled run.