Quantcast

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Leif Mortenson-3
Tu,
Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

Cheers,
Leif

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Mike Pilone
We see this same problem with a Linux VM. We’re still investigating but we randomly have a process or two get killed by the wrapper when the guest VM load looks very low. As Leif pointed out, we think it is related to CPU or IO load on the physical host and VMWare doesn’t allocate CPU or IO time to one of the guests; however we only see one our of 10 wrapper processes killed which seams odd. We see the single process get killed for not responding after 30 seconds and then 5 attempts to restart it also timeout. The actual process that gets killed appears to be random and in most cases the process should be doing little work at the time it was killed which has made it tough for us to isolate the problem. 

For now we’re adding more monitoring to the host and guest and just manually restarting the processes when the issue occurs. I’d be happy to hear if anyone else has seen this issue in VMWare with Linux guests and if you have any suggestions.

-mike


--

 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002 | 202-513-2679 office

On Oct 1, 2014, at 9:32 AM, Leif Mortenson <[hidden email]> wrote:

Tu,
Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

Cheers,
Leif

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Leif Mortenson-3

Leif,

 

WoW that was fast, thank you so much for getting back to me.

 

Yes in all cases these are VM’s.

 

I had originally created my account and sent this request but got a bounce back “Due to a large volume of spam posts lately, I was unfortunately…”  This was because I had not confirmed my email.  When hitting resend, I guess the attachment got lost.  Here it is again.

 

We have suggested trying 3.5.25, but our customer doesn’t like the  “upgrade and see” response and wants to know specific fixes “Are you concerned about the major jump in revisions? I count there has being 37 increments since 3.2.5. I'm worried such a jump can induce other system bugs. Whats your confidence on this.”

 

Again, I really appreciate your response.  I will pursue trying to test the latest version, we cannot reproduce this issue outside of production.  Any additional help / insight you can provide would be much appreciated.

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

 

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

Port Eliz jvm restart on sos wrapper with debug.zip (94K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Mike Pilone

I actually had IT put in a CPU reservation on the VMware host, but this did not seem to help.  Also, VMware Performance counters show little usage on the host level during these outages as well.  But thanks for the feedback regarding Linux.  It was one of the things on our list to try.

 

Currently to work around this issue, we are running the Java program from a command line, without the wrapper.  Its been 13 days without a problem, whereas normally we would see this issue 2-3 times a week.  But running from the command line is not acceptable long term, should someone accidentally log off, or if the JVM dies there is no automatic restart, to name a couple possible issues.

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Mike Pilone [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:40 AM
To: [hidden email]
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

We see this same problem with a Linux VM. We’re still investigating but we randomly have a process or two get killed by the wrapper when the guest VM load looks very low. As Leif pointed out, we think it is related to CPU or IO load on the physical host and VMWare doesn’t allocate CPU or IO time to one of the guests; however we only see one our of 10 wrapper processes killed which seams odd. We see the single process get killed for not responding after 30 seconds and then 5 attempts to restart it also timeout. The actual process that gets killed appears to be random and in most cases the process should be doing little work at the time it was killed which has made it tough for us to isolate the problem. 

 

For now we’re adding more monitoring to the host and guest and just manually restarting the processes when the issue occurs. I’d be happy to hear if anyone else has seen this issue in VMWare with Linux guests and if you have any suggestions.

 

-mike



--


 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002 | 202-513-2679 office

 

On Oct 1, 2014, at 9:32 AM, Leif Mortenson <[hidden email]> wrote:



Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Leif Mortenson-3

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Alexandre Klein
Tu,

Since version 3.2.3, the Wrapper has evolved  a lot. 
Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

As you can see, the list is long.  Here are  some changes that might resolve your issue:
- we have fixed memory leaks
- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)
we have fixed heap corruption
- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

As Leif mentioned, we haven't received any log file.
Can you send it to: [hidden email]
We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

Regards,
Alexandre Klein


Alexandre Klein
Tanuki Software, Ltd.
6-16-7-1001 Nishi-Kasai, Edogawa-ku
Tokyo 134-0088 Japan

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Mike Pilone
In reply to this post by Tu T. Nguyen
I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 

I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.

-mike

--

 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.
 
Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?
 
 
From: Leif Mortenson [[hidden email]] 
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"
 
Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.
 
Cheers,
Leif
 
On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:
 
Hello,
 
I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 
 
We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!
 
Thank you,
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen

Thanks much for your feedback Mike.

 

From: Mike Pilone [mailto:[hidden email]]
Sent: Thursday, October 02, 2014 7:22 AM
To: [hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 

 

I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.

 

-mike

--


 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

 

On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:



What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [[hidden email]] 
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tim Lammens
In reply to this post by Mike Pilone
Mike, 

Are you sure the process is being killed by the wrapper and not by the linux kernel?
A memory leak in glibc was causing our wrapper to consume a lot of memory (bug in appending to a file) but the linux oom killer decided to kill to process which was protected by the wrapper.
Memory shortage than prevented the process being restarted by the wrapper.

Regards,
Tim

On Thu, Oct 2, 2014 at 4:22 PM, Mike Pilone <[hidden email]> wrote:
I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 

I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.

-mike

--

 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.
 
Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?
 
 
From: Leif Mortenson [[hidden email]] 
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"
 
Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.
 
Cheers,
Leif
 
On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:
 
Hello,
 
I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 
 
We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!
 
Thank you,
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user



------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Mike Pilone
It doesn’t look like a memory issue with the wrapper because we’ll see it at different times on different nodes with different services even though we tend to start all of our services at the same time on a given node. The logs also indicate that the wrapper detected a problem and is attempting to restart the service (not just getting killed). I attached the logs from our most recent occurrence where we saw the same service get killed on two different guest VMs which are on two different physical hosts. The only thing the two physical nodes have in common is storage and network IO.

-mike



--

 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002 | 202-513-2679 office |  703-969-7493 cell | [hidden email]


On Oct 2, 2014, at 11:06 AM, Tim Lammens <[hidden email]> wrote:

Mike, 

Are you sure the process is being killed by the wrapper and not by the linux kernel?
A memory leak in glibc was causing our wrapper to consume a lot of memory (bug in appending to a file) but the linux oom killer decided to kill to process which was protected by the wrapper.
Memory shortage than prevented the process being restarted by the wrapper.

Regards,
Tim

On Thu, Oct 2, 2014 at 4:22 PM, Mike Pilone <[hidden email]> wrote:
I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 

I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.

-mike

--

<unknown.png> | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.
 
Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?
 
 
From: Leif Mortenson [[hidden email]] 
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"
 
Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.
 
Cheers,
Leif
 
On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:
 
Hello,
 
I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 
 
We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!
 
Thank you,
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

node1-wrapper.log (4K) Download Attachment
node2-wrapper.log (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen

Mike,

 

Does your Java process have many threads and is very busy?  Our industrial application creates many threads to go out and reads tags from Programmable Logic Controllers (PLC).  I’m just collecting data points for how to possibly to and reproduce this issue in house.  For us it only seems to happen on Windows 2003.  I’m surprised to see that its happening to you on linux.  The other data point I am collecting is vmware hardware versions for the guest machines.

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Mike Pilone [mailto:[hidden email]]
Sent: Thursday, October 02, 2014 9:08 AM
To: [hidden email]
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

It doesn’t look like a memory issue with the wrapper because we’ll see it at different times on different nodes with different services even though we tend to start all of our services at the same time on a given node. The logs also indicate that the wrapper detected a problem and is attempting to restart the service (not just getting killed). I attached the logs from our most recent occurrence where we saw the same service get killed on two different guest VMs which are on two different physical hosts. The only thing the two physical nodes have in common is storage and network IO.

 

-mike

 



--



 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002 | 202-513-2679 office |  703-969-7493 cell | [hidden email]

 

 

On Oct 2, 2014, at 11:06 AM, Tim Lammens <[hidden email]> wrote:



Mike, 

 

Are you sure the process is being killed by the wrapper and not by the linux kernel?

A memory leak in glibc was causing our wrapper to consume a lot of memory (bug in appending to a file) but the linux oom killer decided to kill to process which was protected by the wrapper.

Memory shortage than prevented the process being restarted by the wrapper.

 

Regards,

Tim

 

On Thu, Oct 2, 2014 at 4:22 PM, Mike Pilone <[hidden email]> wrote:

I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 

 

I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.

 

-mike

--


<unknown.png>
 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

 

On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:

 

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [[hidden email]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Alexandre Klein

Sorry there must be a problem sending the file, here it is again

 

 

 

From: Alexandre Klein [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 10:54 PM
To: [hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

 

Since version 3.2.3, the Wrapper has evolved  a lot. 

Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

 

As you can see, the list is long.  Here are  some changes that might resolve your issue:

- we have fixed memory leaks

- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)

we have fixed heap corruption

- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

 

As Leif mentioned, we haven't received any log file.

Can you send it to: [hidden email]

We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

 

Regards,

Alexandre Klein

 


Alexandre Klein

Tanuki Software, Ltd.

6-16-7-1001 Nishi-Kasai, Edogawa-ku

Tokyo 134-0088 Japan

Tel: +81-3-3878-3211

Fax: +81-3-3878-0313

 

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

Port Eliz jvm restart on sos wrapper with debug.zip (94K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Alexandre Klein

Alexandre,

 

I have tried sending it again to both emails.  I did get this reply:

 

Your mail to 'Wrapper-user' with the subject

 

    RE: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Is being held until the list moderator can review it for approval.

 

The reason it is being held:

 

    Message body is too big: 121187 bytes with a limit of 100 KB

 

Either the message will get posted to the list, or you will receive notification of the moderator's decision.  If you would like to cancel this posting, please visit the following URL:

 

    https://lists.sourceforge.net/lists/confirm/wrapper-user/960a8d94be5aa9ed0b0ce620b5f76edc98d73fef

 

 

 

 

From: Alexandre Klein [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 10:54 PM
To: [hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

 

Since version 3.2.3, the Wrapper has evolved  a lot. 

Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

 

As you can see, the list is long.  Here are  some changes that might resolve your issue:

- we have fixed memory leaks

- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)

we have fixed heap corruption

- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

 

As Leif mentioned, we haven't received any log file.

Can you send it to: [hidden email]

We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

 

Regards,

Alexandre Klein

 


Alexandre Klein

Tanuki Software, Ltd.

6-16-7-1001 Nishi-Kasai, Edogawa-ku

Tokyo 134-0088 Japan

Tel: +81-3-3878-3211

Fax: +81-3-3878-0313

 

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Mike Pilone
In reply to this post by Tu T. Nguyen
I would say we have under 20 threads in the application and the vast majority should be idle. When we do have a service get killed this way it is almost always between 12AM and 3AM which is when we run a lot of nightly background jobs and batch processing. However the component that gets killed (usually 1 out of 12) varies and the load of that component also varies so we haven’t been able to tie the issue to load on any one specific component at any particular time. We can sometimes go weeks without the issue and then have different processes killed 3 nights in a row. That’s what makes debugging this so hard.

At one point I was thinking it might be a paging issue where some process was swapped out and couldn’t be swapped back in fast enough to answer the wrapper’s ping but I don’t have any hard evidence other than using swap on a VM guest is usually bad news. Hopefully with some more metrics I’ll be able to find some correlations to other activities on the guest or host boxes.

-mike



NPR | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002




On Oct 2, 2014, at 6:10 PM, Tu T. Nguyen <[hidden email]> wrote:

Mike,
 
Does your Java process have many threads and is very busy?  Our industrial application creates many threads to go out and reads tags from Programmable Logic Controllers (PLC).  I’m just collecting data points for how to possibly to and reproduce this issue in house.  For us it only seems to happen on Windows 2003.  I’m surprised to see that its happening to you on linux.  The other data point I am collecting is vmware hardware versions for the guest machines.
 
 
Regards,
 
Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)
 
 
 
From: Mike Pilone [[hidden email]] 
Sent: Thursday, October 02, 2014 9:08 AM
To: [hidden email]
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"
 
It doesn’t look like a memory issue with the wrapper because we’ll see it at different times on different nodes with different services even though we tend to start all of our services at the same time on a given node. The logs also indicate that the wrapper detected a problem and is attempting to restart the service (not just getting killed). I attached the logs from our most recent occurrence where we saw the same service get killed on two different guest VMs which are on two different physical hosts. The only thing the two physical nodes have in common is storage and network IO.
 
-mike
 


--


<image001.png>
 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002 | 202-513-2679 office |  703-969-7493 cell | [hidden email]
 
<image002.jpg>
<image003.png>
<image004.png>
<image005.png>
 
On Oct 2, 2014, at 11:06 AM, Tim Lammens <[hidden email]> wrote:


Mike, 
 
Are you sure the process is being killed by the wrapper and not by the linux kernel?
A memory leak in glibc was causing our wrapper to consume a lot of memory (bug in appending to a file) but the linux oom killer decided to kill to process which was protected by the wrapper.
Memory shortage than prevented the process being restarted by the wrapper.
 
Regards,
Tim
 
On Thu, Oct 2, 2014 at 4:22 PM, Mike Pilone <[hidden email]> wrote:
I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 
 
I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.
 
-mike

--

<unknown.png>
 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002
 
On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:
 
What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.
 
Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?
 
 
From: Leif Mortenson [[hidden email]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"
 
Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.
 
Cheers,
Leif
 
On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:
 
Hello,
 
I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 
 
We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!
 
Thank you,
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
 
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen

I have one site that gets this issue multiple times a day.  Each time it happens the JVM # gets bumped up in the wrapper log, and theirs goes up into the triple digits.  A second site has this issue about 2-3 times a week.  Running without the wrapper, just from the command line, they have gone 2 weeks without a problem now.

 

From: Mike Pilone [mailto:[hidden email]]
Sent: Friday, October 03, 2014 5:15 AM
To: [hidden email]
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

I would say we have under 20 threads in the application and the vast majority should be idle. When we do have a service get killed this way it is almost always between 12AM and 3AM which is when we run a lot of nightly background jobs and batch processing. However the component that gets killed (usually 1 out of 12) varies and the load of that component also varies so we haven’t been able to tie the issue to load on any one specific component at any particular time. We can sometimes go weeks without the issue and then have different processes killed 3 nights in a row. That’s what makes debugging this so hard.

 

At one point I was thinking it might be a paging issue where some process was swapped out and couldn’t be swapped back in fast enough to answer the wrapper’s ping but I don’t have any hard evidence other than using swap on a VM guest is usually bad news. Hopefully with some more metrics I’ll be able to find some correlations to other activities on the guest or host boxes.

 

-mike



NPR | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

 

 

 

 

On Oct 2, 2014, at 6:10 PM, Tu T. Nguyen <[hidden email]> wrote:



Mike,

 

Does your Java process have many threads and is very busy?  Our industrial application creates many threads to go out and reads tags from Programmable Logic Controllers (PLC).  I’m just collecting data points for how to possibly to and reproduce this issue in house.  For us it only seems to happen on Windows 2003.  I’m surprised to see that its happening to you on linux.  The other data point I am collecting is vmware hardware versions for the guest machines.

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Mike Pilone [[hidden email]] 
Sent: Thursday, October 02, 2014 9:08 AM
To: [hidden email]
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

It doesn’t look like a memory issue with the wrapper because we’ll see it at different times on different nodes with different services even though we tend to start all of our services at the same time on a given node. The logs also indicate that the wrapper detected a problem and is attempting to restart the service (not just getting killed). I attached the logs from our most recent occurrence where we saw the same service get killed on two different guest VMs which are on two different physical hosts. The only thing the two physical nodes have in common is storage and network IO.

 

-mike

 



--




<image001.png>

 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002 | 202-513-2679 office |  703-969-7493 cell | [hidden email]

 

<image002.jpg>

<image003.png>

<image004.png>

<image005.png>

 

On Oct 2, 2014, at 11:06 AM, Tim Lammens <[hidden email]> wrote:




Mike, 

 

Are you sure the process is being killed by the wrapper and not by the linux kernel?

A memory leak in glibc was causing our wrapper to consume a lot of memory (bug in appending to a file) but the linux oom killer decided to kill to process which was protected by the wrapper.

Memory shortage than prevented the process being restarted by the wrapper.

 

Regards,

Tim

 

On Thu, Oct 2, 2014 at 4:22 PM, Mike Pilone <[hidden email]> wrote:

I can tell you that I’m using 3.5.24 and I see the problem on a Linux guest VM as I described earlier. So while I always recommend updating to the latest version, I wouldn’t have much confidence that it is going to fix this specific issue. I’m wondering if the issue has something to do with and interaction between how the wrapper pings the JVM and VMWare guests. I think the ping is done by sending a packet over a socket from the wrapper process to the JVM and getting a simple packet reply. Maybe when the VM guest or host are under load (or something else) the ping packet gets buffered or something. So it might not be a CPU issue but some kind of IO or networking/socket issue. 

 

I’d like to have more time to investigate but when the issue only happens a couple of times a week at 2AM it is hard to get any good data.

 

-mike

--


<unknown.png>
 | Mike Pilone | Software Architect, Distribution | 1111 North Capital St., NE | Washington, DC 20002

 

On Oct 1, 2014, at 8:21 PM, Tu T. Nguyen <[hidden email]> wrote:

 

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [[hidden email]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 

------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Alexandre Klein

Hello All,

 

An update on my issue, I had two sites that has this CPU Time issue.  One I sent the wrapper debug log in and it did indeed end up being an application issue, which was fixed.  There were millions of files waiting to be picked up and it caused the OS/app to hang.

 

At the other site, we continue to have this issue even after upgrading from 3.2.3 to 3.5.25 and trying the PIPE logging option.

I am in the middle of working with them to get logs with wrapper debug on.

 

Finally is there a way to get priority support, pay support?

 

 

Thank you,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Alexandre Klein [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 10:54 PM
To: [hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

 

Since version 3.2.3, the Wrapper has evolved  a lot. 

Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

 

As you can see, the list is long.  Here are  some changes that might resolve your issue:

- we have fixed memory leaks

- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)

we have fixed heap corruption

- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

 

As Leif mentioned, we haven't received any log file.

Can you send it to: [hidden email]

We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

 

Regards,

Alexandre Klein

 


Alexandre Klein

Tanuki Software, Ltd.

6-16-7-1001 Nishi-Kasai, Edogawa-ku

Tokyo 134-0088 Japan

Tel: +81-3-3878-3211

Fax: +81-3-3878-0313

 

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------

_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Alexandre Klein

Hello Alexandre,

 

I think I am close to figuring out this issue.  If I could get a little advice from you.

 

I think this issue has to do with excessive logging from the application.

 

In 3 (out of 3 that I have looked into ) instances where we see this CPU time message,  the logges are filled with 800 or more lines of logging in the exact second the message happens.

 

To support this, there was a time when the logging was not configured and the app ran 50 days without an issue.

 

Do you think this amount of logging may cause the CPU time issue/message?

 

Also.  For the logging level, it was set to “WARNING”, when the actual level should be “WARN”.  I see debug and info message in the log that should not be there.  Can you confirm that “WARNING” is not a valid setting?

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Alexandre Klein [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 10:54 PM
To: [hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

 

Since version 3.2.3, the Wrapper has evolved  a lot. 

Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

 

As you can see, the list is long.  Here are  some changes that might resolve your issue:

- we have fixed memory leaks

- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)

we have fixed heap corruption

- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

 

As Leif mentioned, we haven't received any log file.

Can you send it to: [hidden email]

We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

 

Regards,

Alexandre Klein

 


Alexandre Klein

Tanuki Software, Ltd.

6-16-7-1001 Nishi-Kasai, Edogawa-ku

Tokyo 134-0088 Japan

Tel: +81-3-3878-3211

Fax: +81-3-3878-0313

 

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Alexandre Klein

Hello Alexandre,

 

Sorry I should have included some details with my last email.

 

Attached is a log segment of one of the single seconds I mentioned.  You can see that its 854 lines of logging.

 

An interesting note is that there are 2 separate TIME  (there is only 1 date shown) stamps.  The first one (left most) shows the same time down to the second throughout.  But if you look at the 2nd time that follows, it spans  11:51:05,269 to 11:57:24,863.  This corresponds with the message

 

INFO   | wrapper  | 2015/01/12 11:57:24 | Wrapper Process has not received any CPU time for 347 seconds.  Extending timeouts.

 

Which is the roughly the duration of this span of time.

 

So upon further investigation, the question is asked, was the wrapper hung or storing these messages between  11:51:05,269 to 11:57:24,863 and then finally wrote them out at 11:57:24 ?

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Tu T. Nguyen
Sent: Monday, January 26, 2015 11:29 AM
To: [hidden email]
Cc: Elaine M. Julius
Subject: RE: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Hello Alexandre,

 

I think I am close to figuring out this issue.  If I could get a little advice from you.

 

I think this issue has to do with excessive logging from the application.

 

In 3 (out of 3 that I have looked into ) instances where we see this CPU time message,  the logges are filled with 800 or more lines of logging in the exact second the message happens.

 

To support this, there was a time when the logging was not configured and the app ran 50 days without an issue.

 

Do you think this amount of logging may cause the CPU time issue/message?

 

Also.  For the logging level, it was set to “WARNING”, when the actual level should be “WARN”.  I see debug and info message in the log that should not be there.  Can you confirm that “WARNING” is not a valid setting?

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Alexandre Klein [[hidden email]]
Sent: Wednesday, October 01, 2014 10:54 PM
To:
[hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

 

Since version 3.2.3, the Wrapper has evolved  a lot. 

Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

 

As you can see, the list is long.  Here are  some changes that might resolve your issue:

- we have fixed memory leaks

- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)

we have fixed heap corruption

- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

 

As Leif mentioned, we haven't received any log file.

Can you send it to: [hidden email]

We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

 

Regards,

Alexandre Klein

 


Alexandre Klein

Tanuki Software, Ltd.

6-16-7-1001 Nishi-Kasai, Edogawa-ku

Tokyo 134-0088 Japan

Tel: +81-3-3878-3211

Fax: +81-3-3878-0313

 

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

11-15-24.txt (163K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

Tu T. Nguyen
In reply to this post by Alexandre Klein

There is a 100KB restriction on updates.  I’m sending the log in a zip.

 

 

Your mail to 'Wrapper-user' with the subject

 

    RE: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Is being held until the list moderator can review it for approval.

 

The reason it is being held:

 

    Message body is too big: 206993 bytes with a limit of 100 KB

 

Either the message will get posted to the list, or you will receive notification of the moderator's decision.  If you would like to cancel this posting, please visit the following URL:

 

    https://lists.sourceforge.net/lists/confirm/wrapper-user/2f3d726f66ced27237fcde0879a514eed9898257

 

 

 

From: Tu T. Nguyen
Sent: Monday, January 26, 2015 1:46 PM
To: '[hidden email]'
Cc: Elaine M. Julius
Subject: RE: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Hello Alexandre,

 

Sorry I should have included some details with my last email.

 

Attached is a log segment of one of the single seconds I mentioned.  You can see that its 854 lines of logging.

 

An interesting note is that there are 2 separate TIME  (there is only 1 date shown) stamps.  The first one (left most) shows the same time down to the second throughout.  But if you look at the 2nd time that follows, it spans  11:51:05,269 to 11:57:24,863.  This corresponds with the message

 

INFO   | wrapper  | 2015/01/12 11:57:24 | Wrapper Process has not received any CPU time for 347 seconds.  Extending timeouts.

 

Which is the roughly the duration of this span of time.

 

So upon further investigation, the question is asked, was the wrapper hung or storing these messages between  11:51:05,269 to 11:57:24,863 and then finally wrote them out at 11:57:24 ?

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Tu T. Nguyen
Sent: Monday, January 26, 2015 11:29 AM
To:
[hidden email]
Cc: Elaine M. Julius
Subject: RE: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Hello Alexandre,

 

I think I am close to figuring out this issue.  If I could get a little advice from you.

 

I think this issue has to do with excessive logging from the application.

 

In 3 (out of 3 that I have looked into ) instances where we see this CPU time message,  the logges are filled with 800 or more lines of logging in the exact second the message happens.

 

To support this, there was a time when the logging was not configured and the app ran 50 days without an issue.

 

Do you think this amount of logging may cause the CPU time issue/message?

 

Also.  For the logging level, it was set to “WARNING”, when the actual level should be “WARN”.  I see debug and info message in the log that should not be there.  Can you confirm that “WARNING” is not a valid setting?

 

 

Regards,

 

Tu Nguyen | FTPC Support Engineer | Office: 408.271.3464 | Mobile: 408.464.3252 | Rockwell Automation | Mission Viejo, CA (GMT -8)

 

 

 

From: Alexandre Klein [[hidden email]]
Sent: Wednesday, October 01, 2014 10:54 PM
To:
[hidden email]
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

 

Since version 3.2.3, the Wrapper has evolved  a lot. 

Please take a look at the release notes which contains all bug fixes and new features for the Java Service Wrapper.

 

As you can see, the list is long.  Here are  some changes that might resolve your issue:

- we have fixed memory leaks

- we have improved the way to write large amounts of output (otherwise the Wrapper thought the JVM was frozen)

we have fixed heap corruption

- we have improved the way the Java side of the Wrapper behaves when the Wrapper fails to ping the JVM for an extended period of time. (This was causing problems on systems which were heavily IO bound because the Wrapper could block for a while while trying to write to the log file and the JVM was exiting)

 

As Leif mentioned, we haven't received any log file.

Can you send it to: [hidden email]

We would be happy to have a look. If possible, please send us the configuration file (wrapper.conf) as well.

 

Regards,

Alexandre Klein

 


Alexandre Klein

Tanuki Software, Ltd.

6-16-7-1001 Nishi-Kasai, Edogawa-ku

Tokyo 134-0088 Japan

Tel: +81-3-3878-3211

Fax: +81-3-3878-0313

 

On Thu, Oct 2, 2014 at 9:21 AM, Tu T. Nguyen <[hidden email]> wrote:

What we are looking for is some justification that we can give our customer that upgrading even might resolve the issue they are seeing.

 

Based on what you see in the logs, is there anything that would point to an upgrade being likely to resolve the problem?

 

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: Wednesday, October 01, 2014 6:33 AM
To: Wrapper User List
Cc: Elaine M. Julius
Subject: Re: [Wrapper-user] JVM restarts "Wrapper Process has not received any CPU time for 69 seconds"

 

Tu,

Is the machine you are running on a physical or a virtual machine?  We have seen cases where a loaded host causes the VM to freeze up when the guest itself does not show any load.

3.2.3 is also a VERY old version of the Wrapper.  There have been a lot of improvements over the years in this area.  Please try 3.5.25 and see how that works for you.

You mentioned log files, but there was nothing attached to your mail.

 

Cheers,
Leif

 

On Wed, Oct 1, 2014 at 10:23 PM, Tu T. Nguyen <[hidden email]> wrote:

 

Hello,

 

I have an issue where the wrapper restarts the JVM multiple times a day.  We are using version 3.2.3 on Windows 2003 SP2.  The duration always varies.  We checked the CPU usage and its always very little when this happens .  We have about 6 services that use the wrapper on this machine and they restart at different times.  The suggestion is that if there was a CPU bottle neck they would all have this error at the same time.  We also have other sites which use the same configuration but do not have this problem. 

 

We have collected logs with wrapper debugging enabled.  Please have a look, any help would be much appreciated!

 

Thank you,


------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer
Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports
Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper
Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

 


------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

11-15-24.zip (13K) Download Attachment
Loading...