Quantcast

Advice on stopping wrapper from triggering JVM restarts

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds. 

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually. 

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated. 

Thanks!

--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Leif Mortenson-3
Casey,
The configuration you specified will disable all pings and cpu warnings and should do what you want.
The Wrapper should never timeout  in this case.  There are other things such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not sure if that was a test on your part, but if the source of the signal has permission to kill the JVM's process then there is nothing we can do to prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]> wrote:
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds. 

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually. 

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated. 

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Hi Leif,

Thanks for the feedback. This is quite a mystery then because I am 100% sure that the process wasn't killed manually or by any other systems we have running. (This is just a standard CentOS 7 setup)

At about the same time that this log appeared I was running a very heavy build process in a separate JVM instance. 

It seems more than just coincidental that this happened at the same time. I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you think of any scenario where this makes sense? Perhaps I am missing some knowledge of java architecture here that is making this mysterious. 

Thanks


On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <[hidden email]> wrote:
Casey,
The configuration you specified will disable all pings and cpu warnings and should do what you want.
The Wrapper should never timeout  in this case.  There are other things such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not sure if that was a test on your part, but if the source of the signal has permission to kill the JVM's process then there is nothing we can do to prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]> wrote:
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds. 

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually. 

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated. 

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Mike Pilone
I’ve been commenting on a similar issue where we occasionally have a JVM killed and restarted by the wrapper because it doesn’t get a ping response. We still haven’t tracked down the root cause but we don’t get the CPU warning but rather “Timed out waiting for signal from JVM”. It always seems to happen around midnight which is when we have some background jobs kick off in various processes so we think it is related to that, but we don’t have any hard evidence. We’re running RedHat Linux guests in VMWare. Here’s our log output if it is any help:

STATUS | wrapper  | 2014/12/01 00:05:51 | JVM appears hung: Timed out waiting for signal from JVM.  Restarting JVM.
ERROR  | wrapper  | 2014/12/01 00:06:38 | Shutdown failed: Timed out waiting for the JVM to terminate.
ERROR  | wrapper  | 2014/12/01 00:06:39 | JVM did not exit on request, termination requested.
STATUS | wrapper  | 2014/12/01 00:06:39 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/01 00:06:39 | JVM process is gone.
STATUS | wrapper  | 2014/12/01 00:06:39 | JVM exited after being requested to terminate.

In most cases the wrapper attempts to restart the JVM 5 times, each of them timing out. It then finally just gives up and the process remains down.

-mike


NPR | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002




On Dec 2, 2014, at 10:39 AM, Casey Jordan <[hidden email]> wrote:

Hi Leif,

Thanks for the feedback. This is quite a mystery then because I am 100% sure that the process wasn't killed manually or by any other systems we have running. (This is just a standard CentOS 7 setup)

At about the same time that this log appeared I was running a very heavy build process in a separate JVM instance. 

It seems more than just coincidental that this happened at the same time. I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you think of any scenario where this makes sense? Perhaps I am missing some knowledge of java architecture here that is making this mysterious. 

Thanks


On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <[hidden email]> wrote:
Casey,
The configuration you specified will disable all pings and cpu warnings and should do what you want.
The Wrapper should never timeout  in this case.  There are other things such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not sure if that was a test on your part, but if the source of the signal has permission to kill the JVM's process then there is nothing we can do to prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]> wrote:
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds. 

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually. 

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated. 

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




-- 
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.
------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Leif Mortenson-3
In reply to this post by Casey Jordan
Casey,
If you are able to reproduce this, can you set wrapper.debug=true and repeat?  This log output will shed more light on exactly what is happening.  Depending on the platform, we will also be able to see the pid of the source of the kill signal.

Cheers,
Leif

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
Hi Leif,

Thanks for the feedback. This is quite a mystery then because I am 100% sure that the process wasn't killed manually or by any other systems we have running. (This is just a standard CentOS 7 setup)

At about the same time that this log appeared I was running a very heavy build process in a separate JVM instance. 

It seems more than just coincidental that this happened at the same time. I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you think of any scenario where this makes sense? Perhaps I am missing some knowledge of java architecture here that is making this mysterious. 

Thanks


On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <[hidden email]> wrote:
Casey,
The configuration you specified will disable all pings and cpu warnings and should do what you want.
The Wrapper should never timeout  in this case.  There are other things such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not sure if that was a test on your part, but if the source of the signal has permission to kill the JVM's process then there is nothing we can do to prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]> wrote:
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds. 

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually. 

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated. 

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Yeah I can try that, thanks!

On Tue, Dec 2, 2014 at 11:43 AM, Leif Mortenson <[hidden email]> wrote:
Casey,
If you are able to reproduce this, can you set wrapper.debug=true and repeat?  This log output will shed more light on exactly what is happening.  Depending on the platform, we will also be able to see the pid of the source of the kill signal.

Cheers,
Leif


On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
Hi Leif,

Thanks for the feedback. This is quite a mystery then because I am 100% sure that the process wasn't killed manually or by any other systems we have running. (This is just a standard CentOS 7 setup)

At about the same time that this log appeared I was running a very heavy build process in a separate JVM instance. 

It seems more than just coincidental that this happened at the same time. I assume that if the JVM crashed it wouldn't have received a SIG 9? Can you think of any scenario where this makes sense? Perhaps I am missing some knowledge of java architecture here that is making this mysterious. 

Thanks


On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson <[hidden email]> wrote:
Casey,
The configuration you specified will disable all pings and cpu warnings and should do what you want.
The Wrapper should never timeout  in this case.  There are other things such as a JVM crash would would still result in a restart after the fact.

The log that you send shows that the JVM will killed with a kill -9.  Not sure if that was a test on your part, but if the source of the signal has permission to kill the JVM's process then there is nothing we can do to prevent that.

Cheers,
Leif

On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]> wrote:
Hi all,

I am sure this is a common question, but I am having trouble extracting details I need from the documentation. From time to time we get a situation where our JVM gets killed and we see something like this in the logs:

INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not received any CPU time for 1 seconds.  Extending timeouts.
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL (9).
STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts disabled.  Shutting down.
STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped


This always happens in the case where some other process is eating up all the cpu for more than a few seconds. 

It's crucial for us that the wrapper never, ever restarts the JVM as this is something we always want to look into manually. 

So my question is, what is the best way to make sure this doesn't happen?

I am using community version 3.5.17, and have the following values in our wrapper.conf:

   #********************************************************************
   # Timeouts
   #********************************************************************
   wrapper.ping.timeout=0
   wrapper.ping.timeout.action=DEBUG
   wrapper.cpu.timeout=0
   wrapper.startup.timeout=300

Any help is much appreciated. 

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Roberto Espinoza-2
In reply to this post by Casey Jordan
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:

> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> (585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: +81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Tim Lammens
You are most probably experiencing a glibc bug as I mentioned before in another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <[hidden email]> wrote:
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211" target="_blank">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Hi Tim,

Thanks for the info, is this the thread you speak of: http://sourceforge.net/p/wrapper/mailman/message/33048371/

If not could you point me to the right one?

Thanks

On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <[hidden email]> wrote:
You are most probably experiencing a glibc bug as I mentioned before in another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <[hidden email]> wrote:
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211" target="_blank">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
By the way, this is my glibc version: glibc-2.17-55.el7_0.1.x86_64

On Wed, Dec 10, 2014 at 11:03 AM, Casey Jordan <[hidden email]> wrote:
Hi Tim,

Thanks for the info, is this the thread you speak of: http://sourceforge.net/p/wrapper/mailman/message/33048371/

If not could you point me to the right one?

Thanks

On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <[hidden email]> wrote:
You are most probably experiencing a glibc bug as I mentioned before in another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <[hidden email]> wrote:
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211" target="_blank">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Tim Lammens
In reply to this post by Casey Jordan
That is indeed the right thread.

Regards,
Tim

On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <[hidden email]> wrote:
Hi Tim,

Thanks for the info, is this the thread you speak of: http://sourceforge.net/p/wrapper/mailman/message/33048371/

If not could you point me to the right one?

Thanks

On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <[hidden email]> wrote:
You are most probably experiencing a glibc bug as I mentioned before in another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <[hidden email]> wrote:
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211" target="_blank">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Cool thanks, will I be able to see this memory usage increase by looking at the top command?

I would like to keep track of this a little more and see if I notice the wrapper memory growing.

Thanks

On Wed, Dec 10, 2014 at 11:06 AM, Tim Lammens <[hidden email]> wrote:
That is indeed the right thread.

Regards,
Tim

On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <[hidden email]> wrote:
Hi Tim,

Thanks for the info, is this the thread you speak of: http://sourceforge.net/p/wrapper/mailman/message/33048371/

If not could you point me to the right one?

Thanks

On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <[hidden email]> wrote:
You are most probably experiencing a glibc bug as I mentioned before in another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <[hidden email]> wrote:
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211" target="_blank">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Tim Lammens
Yes you can monitor this with top, but the speed depends on how much your application logs.

On Wed, Dec 10, 2014 at 5:46 PM, Casey Jordan <[hidden email]> wrote:
Cool thanks, will I be able to see this memory usage increase by looking at the top command?

I would like to keep track of this a little more and see if I notice the wrapper memory growing.

Thanks

On Wed, Dec 10, 2014 at 11:06 AM, Tim Lammens <[hidden email]> wrote:
That is indeed the right thread.

Regards,
Tim

On Wed, Dec 10, 2014 at 5:03 PM, Casey Jordan <[hidden email]> wrote:
Hi Tim,

Thanks for the info, is this the thread you speak of: http://sourceforge.net/p/wrapper/mailman/message/33048371/

If not could you point me to the right one?

Thanks

On Wed, Dec 10, 2014 at 10:58 AM, Tim Lammens <[hidden email]> wrote:
You are most probably experiencing a glibc bug as I mentioned before in another thread. The bug was fixed in recent glibc versions.
Try updating your glibc version for CentOS. If that is not possible you can apply the patches I provided in a different thread on this mailing list.

Regards,
Tim

On Wed, Dec 10, 2014 at 4:49 PM, Casey Jordan <[hidden email]> wrote:
Ok so this happened again and I was able to capture some additional log information (not from wrapper.debug though as I can't update this server just yet)

I got this in our logs:

INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Max Mem: 2223 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total Mem: 2204 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Used Mem: 1141 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Raw free-mem: 1082 MB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Total descriptors: 4096
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Open descriptors: 556
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Available descriptors: 3540
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>Free disk space: 18 GB
INFO | jvm 1 | 2014/12/09 23:15:50 | <br></br>
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 1 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | Pinging the JVM took 7 seconds to respond.
INFO | wrapper | 2014/12/09 23:17:09 | Wrapper Process has not received any CPU time for 5 seconds. Extending timeouts.
STATUS | wrapper | 2014/12/09 23:17:09 | JVM received a signal SIGKILL (9).
STATUS | wrapper | 2014/12/09 23:17:09 | JVM process is gone.
ERROR | wrapper | 2014/12/09 23:17:09 | JVM exited unexpectedly.
STATUS | wrapper | 2014/12/09 23:17:09 | Automatic JVM Restarts disabled. Shutting down.
STATUS | wrapper | 2014/12/09 23:17:09 | <-- Wrapper Stopped

and I found this in the /var/log/messages

Dec  9 23:17:09 jorsek-home2 kernel: Out of memory: Kill process 10262 (java) score 591 or sacrifice child
Dec  9 23:17:09 jorsek-home2 kernel: Killed process 10262 (java) total-vm:6478728kB, anon-rss:2600344kB, file-rss:0kB

So this does appear to be the kernel killing the process, but according to java I have plenty of memory available (>1GB). 

I know this is not really a wrapper issue any longer, but I would be very appreciative if anyone can give me some advice or point me to some resources that might help figure this out.

For reference this is a Centos7 box with 4GB of RAM and a 1GB swap space, and 40GB SSD drive.

Tonight I am going to add NewRelic monitoring to this server to see if I can get a better picture of what is going on.

Thanks! 

On Tue, Dec 2, 2014 at 7:30 PM, Roberto Espinoza <[hidden email]> wrote:
Casey,

Have you checked the kernel logs to see if it wasn't killed because
your system ran out of memory?

Certainly there is no way to know this inside the JVM because the
kernel out of memory killer will just score the current processes and
issue a kill.

On Wed, Dec 3, 2014 at 12:39 AM, Casey Jordan <[hidden email]> wrote:
> Hi Leif,
>
> Thanks for the feedback. This is quite a mystery then because I am 100% sure
> that the process wasn't killed manually or by any other systems we have
> running. (This is just a standard CentOS 7 setup)
>
> At about the same time that this log appeared I was running a very heavy
> build process in a separate JVM instance.
>
> It seems more than just coincidental that this happened at the same time. I
> assume that if the JVM crashed it wouldn't have received a SIG 9? Can you
> think of any scenario where this makes sense? Perhaps I am missing some
> knowledge of java architecture here that is making this mysterious.
>
> Thanks
>
>
> On Tue, Dec 2, 2014 at 4:27 AM, Leif Mortenson
> <[hidden email]> wrote:
>>
>> Casey,
>> The configuration you specified will disable all pings and cpu warnings
>> and should do what you want.
>> The Wrapper should never timeout  in this case.  There are other things
>> such as a JVM crash would would still result in a restart after the fact.
>>
>> The log that you send shows that the JVM will killed with a kill -9.  Not
>> sure if that was a test on your part, but if the source of the signal has
>> permission to kill the JVM's process then there is nothing we can do to
>> prevent that.
>>
>> Cheers,
>> Leif
>>
>> On Tue, Dec 2, 2014 at 12:16 PM, Casey Jordan <[hidden email]>
>> wrote:
>>>
>>> Hi all,
>>>
>>> I am sure this is a common question, but I am having trouble extracting
>>> details I need from the documentation. From time to time we get a situation
>>> where our JVM gets killed and we see something like this in the logs:
>>>
>>> INFO   | wrapper  | 2014/12/02 02:23:18 | Wrapper Process has not
>>> received any CPU time for 1 seconds.  Extending timeouts.
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM received a signal SIGKILL
>>> (9).
>>> STATUS | wrapper  | 2014/12/02 02:23:37 | JVM process is gone.
>>> ERROR  | wrapper  | 2014/12/02 02:23:37 | JVM exited unexpectedly.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | Automatic JVM Restarts
>>> disabled.  Shutting down.
>>> STATUS | wrapper  | 2014/12/02 02:23:38 | <-- Wrapper Stopped
>>>
>>>
>>> This always happens in the case where some other process is eating up all
>>> the cpu for more than a few seconds.
>>>
>>> It's crucial for us that the wrapper never, ever restarts the JVM as this
>>> is something we always want to look into manually.
>>>
>>> So my question is, what is the best way to make sure this doesn't happen?
>>>
>>> I am using community version 3.5.17, and have the following values in our
>>> wrapper.conf:
>>>
>>>    #********************************************************************
>>>    # Timeouts
>>>    #********************************************************************
>>>    wrapper.ping.timeout=0
>>>    wrapper.ping.timeout.action=DEBUG
>>>    wrapper.cpu.timeout=0
>>>    wrapper.startup.timeout=300
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
>> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
>> with Interactivity, Sharing, Native Excel Exports, App Integration & more
>> Get technology previously reserved for billion-dollar corporations, FREE
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>
>
> --
> --
> Casey Jordan
> easyDITA a product of Jorsek LLC
> "CaseyDJordan" on LinkedIn, Twitter & Facebook
> <a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
> easydita.com
>
>
> This message is intended only for the use of the Addressee(s) and may
> contain information that is privileged, confidential, and/or exempt from
> disclosure under applicable law.  If you are not the intended recipient,
> please be advised that any disclosure  copying, distribution, or use of
> the information contained herein is prohibited.  If you have received
> this communication in error, please destroy all copies of the message,
> whether in electronic or hard copy format, as well as attachments, and
> immediately contact the sender by replying to this e-mail or by phone.
> Thank you.
>
> ------------------------------------------------------------------------------
> Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
> from Actuate! Instantly Supercharge Your Business Reports and Dashboards
> with Interactivity, Sharing, Native Excel Exports, App Integration & more
> Get technology previously reserved for billion-dollar corporations, FREE
> http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>



--
Roberto Espinoza
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan
Tel/Fax: <a href="tel:%2B81-3-3878-3211" value="+81338783211" target="_blank">+81-3-3878-3211
http://www.tanukisoftware.com
[hidden email]

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Ok, so I have been monitoring for a bit now and I can definitely say that I have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my system, given that all the threads I read say that will cause major problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch directly in the wrapper to also fix this issue, does anyone know if this fix made it into a release that I could use?

Thanks!

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Hi Tim & Community,

Sorry I had to go back and review your previous comments before I realized you had posted those patches and also provided me with instructions.

I am currently running wrapper version 3.5.17. I also saw that 3.5.26 appears to have the memory leak fixes in it (Based on reading the release notes).

Which of the following would be the best option?

Patch v3.5.17
Upgrade to v3.5.25 and patch
Use v3.5.26 (Which I believe contains the fix I need)

Obviously using 3.5.26 seems the simplest approach here, but I just wanted to double check with the community. 

Thanks, your help is much appreciated.

On Sat, Dec 20, 2014 at 6:46 PM, Casey Jordan <[hidden email]> wrote:
Ok, so I have been monitoring for a bit now and I can definitely say that I have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my system, given that all the threads I read say that will cause major problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch directly in the wrapper to also fix this issue, does anyone know if this fix made it into a release that I could use?

Thanks!



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Leif Mortenson-3
Casey,
Rather than patching the Wrapper, I would suggest the released version with the fix.  There have been a number of other important improvements as well.
We usually wait a couple weeks after a release before deciding to raise a version to stable.  This is done to make sure that there are no unexpected errors reported from the user base.
So far 3.5.26 has been quite stable.

Cheers,
Leif


On Sun, Dec 21, 2014 at 9:52 AM, Casey Jordan <[hidden email]> wrote:
Hi Tim & Community,

Sorry I had to go back and review your previous comments before I realized you had posted those patches and also provided me with instructions.

I am currently running wrapper version 3.5.17. I also saw that 3.5.26 appears to have the memory leak fixes in it (Based on reading the release notes).

Which of the following would be the best option?

Patch v3.5.17
Upgrade to v3.5.25 and patch
Use v3.5.26 (Which I believe contains the fix I need)

Obviously using 3.5.26 seems the simplest approach here, but I just wanted to double check with the community. 

Thanks, your help is much appreciated.

On Sat, Dec 20, 2014 at 6:46 PM, Casey Jordan <[hidden email]> wrote:
Ok, so I have been monitoring for a bit now and I can definitely say that I have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my system, given that all the threads I read say that will cause major problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch directly in the wrapper to also fix this issue, does anyone know if this fix made it into a release that I could use?

Thanks!


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
Leif,

Thanks for replying so quickly! That's great to hear, I'm excited to try out the new version.

On Sunday, December 21, 2014, Leif Mortenson <[hidden email]> wrote:
Casey,
Rather than patching the Wrapper, I would suggest the released version with the fix.  There have been a number of other important improvements as well.
We usually wait a couple weeks after a release before deciding to raise a version to stable.  This is done to make sure that there are no unexpected errors reported from the user base.
So far 3.5.26 has been quite stable.

Cheers,
Leif


On Sun, Dec 21, 2014 at 9:52 AM, Casey Jordan <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;casey.jordan@jorsek.com&#39;);" target="_blank">casey.jordan@...> wrote:
Hi Tim & Community,

Sorry I had to go back and review your previous comments before I realized you had posted those patches and also provided me with instructions.

I am currently running wrapper version 3.5.17. I also saw that 3.5.26 appears to have the memory leak fixes in it (Based on reading the release notes).

Which of the following would be the best option?

Patch v3.5.17
Upgrade to v3.5.25 and patch
Use v3.5.26 (Which I believe contains the fix I need)

Obviously using 3.5.26 seems the simplest approach here, but I just wanted to double check with the community. 

Thanks, your help is much appreciated.

On Sat, Dec 20, 2014 at 6:46 PM, Casey Jordan <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;casey.jordan@jorsek.com&#39;);" target="_blank">casey.jordan@...> wrote:
Ok, so I have been monitoring for a bit now and I can definitely say that I have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my system, given that all the threads I read say that will cause major problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch directly in the wrapper to also fix this issue, does anyone know if this fix made it into a release that I could use?

Thanks!



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=164703151&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Casey Jordan
So I wanted to give an update on this. I created a simple stress testing script that triggered lots of logging to occur. I tested this on the 3.5.17 version and was easily able to see the memory leak as the wrapper process rose to about 3.5% (250MB) of memory usage over running this script for about 2 hours.

However, now I am running this same script on 2.5.26, it's been running for about 30 min now and is already up to 1% system memory usage. I am going to keep running for a few hours and see what results I get. However, I am concerned because on non CentOs systems we never see the wrapper process even reach .1% consumption even after running for 6 months. 

Perhaps what I am seeing is just some artifact of something else (cache/buffers?)? Maybe there is a better way to test this before I put it back into production? (I don't really want to get into doing a valgrind session :)

Thanks

On Sun, Dec 21, 2014 at 10:27 PM, Casey Jordan <[hidden email]> wrote:
Leif,

Thanks for replying so quickly! That's great to hear, I'm excited to try out the new version.


On Sunday, December 21, 2014, Leif Mortenson <[hidden email]> wrote:
Casey,
Rather than patching the Wrapper, I would suggest the released version with the fix.  There have been a number of other important improvements as well.
We usually wait a couple weeks after a release before deciding to raise a version to stable.  This is done to make sure that there are no unexpected errors reported from the user base.
So far 3.5.26 has been quite stable.

Cheers,
Leif


On Sun, Dec 21, 2014 at 9:52 AM, Casey Jordan <[hidden email]> wrote:
Hi Tim & Community,

Sorry I had to go back and review your previous comments before I realized you had posted those patches and also provided me with instructions.

I am currently running wrapper version 3.5.17. I also saw that 3.5.26 appears to have the memory leak fixes in it (Based on reading the release notes).

Which of the following would be the best option?

Patch v3.5.17
Upgrade to v3.5.25 and patch
Use v3.5.26 (Which I believe contains the fix I need)

Obviously using 3.5.26 seems the simplest approach here, but I just wanted to double check with the community. 

Thanks, your help is much appreciated.

On Sat, Dec 20, 2014 at 6:46 PM, Casey Jordan <[hidden email]> wrote:
Ok, so I have been monitoring for a bit now and I can definitely say that I have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my system, given that all the threads I read say that will cause major problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch directly in the wrapper to also fix this issue, does anyone know if this fix made it into a release that I could use?

Thanks!



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Advice on stopping wrapper from triggering JVM restarts

Tim Lammens
On CentOS there is still a memory leak in the glibc code, you can avoid this by applying one of the patches I provided on the wrapper code if you want to avoid updating glibc. It has to do with the append system calls.

Regards,
Tim


On 23-dec.-2014, at 16:45, Casey Jordan <[hidden email]> wrote:

So I wanted to give an update on this. I created a simple stress testing script that triggered lots of logging to occur. I tested this on the 3.5.17 version and was easily able to see the memory leak as the wrapper process rose to about 3.5% (250MB) of memory usage over running this script for about 2 hours.

However, now I am running this same script on 2.5.26, it's been running for about 30 min now and is already up to 1% system memory usage. I am going to keep running for a few hours and see what results I get. However, I am concerned because on non CentOs systems we never see the wrapper process even reach .1% consumption even after running for 6 months. 

Perhaps what I am seeing is just some artifact of something else (cache/buffers?)? Maybe there is a better way to test this before I put it back into production? (I don't really want to get into doing a valgrind session :)

Thanks

On Sun, Dec 21, 2014 at 10:27 PM, Casey Jordan <[hidden email]> wrote:
Leif,

Thanks for replying so quickly! That's great to hear, I'm excited to try out the new version.


On Sunday, December 21, 2014, Leif Mortenson <[hidden email]> wrote:
Casey,
Rather than patching the Wrapper, I would suggest the released version with the fix.  There have been a number of other important improvements as well.
We usually wait a couple weeks after a release before deciding to raise a version to stable.  This is done to make sure that there are no unexpected errors reported from the user base.
So far 3.5.26 has been quite stable.

Cheers,
Leif


On Sun, Dec 21, 2014 at 9:52 AM, Casey Jordan <[hidden email]> wrote:
Hi Tim & Community,

Sorry I had to go back and review your previous comments before I realized you had posted those patches and also provided me with instructions.

I am currently running wrapper version 3.5.17. I also saw that 3.5.26 appears to have the memory leak fixes in it (Based on reading the release notes).

Which of the following would be the best option?

Patch v3.5.17
Upgrade to v3.5.25 and patch
Use v3.5.26 (Which I believe contains the fix I need)

Obviously using 3.5.26 seems the simplest approach here, but I just wanted to double check with the community. 

Thanks, your help is much appreciated.

On Sat, Dec 20, 2014 at 6:46 PM, Casey Jordan <[hidden email]> wrote:
Ok, so I have been monitoring for a bit now and I can definitely say that I have this memory leak and it is causing the issues I am experiencing.

Now the issue remains that I do not believe that I can upgrade glibc on my system, given that all the threads I read say that will cause major problems. Also, I need to use CentOS 7 if at all possible.

I noticed in the previously referenced thread that someone created a patch directly in the wrapper to also fix this issue, does anyone know if this fix made it into a release that I could use?

Thanks!



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
<a href="tel:%28585%29%20348%207399" value="+15853487399" target="_blank">(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.




--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.
------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

------------------------------------------------------------------------------
Dive into the World of Parallel Programming! The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
12
Loading...