Quantcast

JVM Exited with No details - Need Help

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

JVM Exited with No details - Need Help

Paul Manning

All,

 

We are currently running an application using Tanuki 3.2.3 with ServiceMix.

 

The application seems to randomly fail with the following error in the wrapper log:

 

ERROR  | wrapper  | 2017/01/24 13:00:06 | JVM exited unexpectedly.

DEBUG  | wrapper  | 2017/01/24 13:00:06 | Signal trapped.  Details:

DEBUG  | wrapper  | 2017/01/24 13:00:06 |   signal number=17 (SIGCHLD), source="unknown"

DEBUG  | wrapper  | 2017/01/24 13:00:06 | Received SIGCHLD, checking JVM process status.

STATUS | wrapper  | 2017/01/24 13:00:06 | JVM exited in response to signal SIGKILL (9).

DEBUG  | wrapper  | 2017/01/24 13:00:06 | JVM process exited with a code of 1, setting the wrapper exit code to 1.

 

We currently have Tanuki setup to auto restart, so the application comes back online immediately. However, it will eventually crash again. Sometimes this is within hours or sometimes within days.

 

Notes:

The application is running with Java 1.7.0_45 on Redhat Linux 64 bit.

There are no error messages or exceptions in our application logs (ServiceMix).

There are no JVM crash reports (hs_err_pid files).

There are no core dumps.

This is not being killed by the Linux OOM Killer or another user. The linux system logs are clean.

No other exceptions in the wrapper log.

Verbose GC Logging is enabled and the system memory is healthy. Memory is about 50% utilization when the crash recently occurred.

Thread dumps are being taken periodically and they look clean (no deadlock).

This is a commercial application that is running at many customer sites for many years and we have never seen a problem like this before.

I cannot reproduce this problem in house and I have limited access to the production system (thru a 3rd party).

 

Is there anything that we can do to get more intel on why this process is crashing?

 

Thanks,

 

Paul Manning

 

 

 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM Exited with No details - Need Help

Wendell MacKenzie-3
sounds like a system management process is determining that your JVM instance is consuming TOO much system resource
and it is being terminated with the kill -9.  try keep a running snapshot of system resources every 5 minutes or so to a log and 
see if there is a spike occuring at some point prior to the termination of the VM.

Wendell

On Wed, Jan 25, 2017 at 11:06 AM, Paul Manning <[hidden email]> wrote:

All,

 

We are currently running an application using Tanuki 3.2.3 with ServiceMix.

 

The application seems to randomly fail with the following error in the wrapper log:

 

ERROR  | wrapper  | 2017/01/24 13:00:06 | JVM exited unexpectedly.

DEBUG  | wrapper  | 2017/01/24 13:00:06 | Signal trapped.  Details:

DEBUG  | wrapper  | 2017/01/24 13:00:06 |   signal number=17 (SIGCHLD), source="unknown"

DEBUG  | wrapper  | 2017/01/24 13:00:06 | Received SIGCHLD, checking JVM process status.

STATUS | wrapper  | 2017/01/24 13:00:06 | JVM exited in response to signal SIGKILL (9).

DEBUG  | wrapper  | 2017/01/24 13:00:06 | JVM process exited with a code of 1, setting the wrapper exit code to 1.

 

We currently have Tanuki setup to auto restart, so the application comes back online immediately. However, it will eventually crash again. Sometimes this is within hours or sometimes within days.

 

Notes:

The application is running with Java 1.7.0_45 on Redhat Linux 64 bit.

There are no error messages or exceptions in our application logs (ServiceMix).

There are no JVM crash reports (hs_err_pid files).

There are no core dumps.

This is not being killed by the Linux OOM Killer or another user. The linux system logs are clean.

No other exceptions in the wrapper log.

Verbose GC Logging is enabled and the system memory is healthy. Memory is about 50% utilization when the crash recently occurred.

Thread dumps are being taken periodically and they look clean (no deadlock).

This is a commercial application that is running at many customer sites for many years and we have never seen a problem like this before.

I cannot reproduce this problem in house and I have limited access to the production system (thru a 3rd party).

 

Is there anything that we can do to get more intel on why this process is crashing?

 

Thanks,

 

Paul Manning

 

 

 


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--

D. Wendell MacKenzie
Software Developer

Email: [hidden email]
Phone: 902-626-8708 (cell)
            

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Loading...