Quantcast

wrapper process is hung

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

wrapper process is hung

Cade Thacker
Hello All,
I'm not sure how to figure this out so figured I would turn to the experts.  

I documented everything here: 


But the simple summary is that when I issue a kill $pid (SIGTERM/15) on the wrapper process (the JVM is long dead via something else killing it with SIGKILL/9) nothing happens. (this is the same call from stopit() in the wrapper.sh distributed by you all. 

As documented in the other question, that causes other issues downstream because without the wrapper dying correctling, I can't restart them both. 

Any ideas on how to best trouble shoot this?  I'm not sure what would be "stuck" in the wrapper C code so that a SIGTERM would not be properly processed? SIGKILL/9 works as expected. 

If this was a Java process, I'd get a thread dump and a heap dump, but not sure how to attack this C code.  Maybe dig through something in /proc/$pid? 

Any help is incredibly appreciated!


--

--cade

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapper process is hung

David Brown
Howdy, ANSI C is notoriously prone to memory leaks because of the use of
functions: malloc() and free() to dynamically allocate memory for the
target program's runtime.

If you have considerable ANSI C background you will already know the
associated problems and possible diagnostics.

If not the C language is much more tied to the hardware platform the
compiler is running on as opposed to something like: Java.

In the case you are running on a Linux you are in luck. Without
investment you can start w/ mtrace and with a little more work to
profile you program you can use Valgrind.

In the event you are on Windows (wrapper of course) then you can use
Purify.

In any event you will have to profile the runtime and 1 solution is
heavy use of logging to help narrow down the culprit code.

I saw no responses on SO so it appears you will have to dig into the
code.

C language is because of age and distribution very well documented and
lots of references on Google as you probably already know.


On 2016-02-08 22:41, Cade Thacker wrote:

> Hello All,
> I'm not sure how to figure this out so figured I would turn to the
> experts.
>
> I documented everything here:
>
> http://stackoverflow.com/questions/35283549/java-service-wrapper-parent-process-is-hung
> [1]
>
> But the simple summary is that when I issue a kill $pid (SIGTERM/15)
> on the wrapper process (the JVM is long dead via something else
> killing it with SIGKILL/9) nothing happens. (this is the same call
> from stopit() in the wrapper.sh distributed by you all.
>
> As documented in the other question, that causes other issues
> downstream because without the wrapper dying correctling, I can't
> restart them both.
>
> Any ideas on how to best trouble shoot this?  I'm not sure what would
> be "stuck" in the wrapper C code so that a SIGTERM would not be
> properly processed? SIGKILL/9 works as expected.
>
> If this was a Java process, I'd get a thread dump and a heap dump, but
> not sure how to attack this C code.  Maybe dig through something in
> /proc/$pid?
>
> Any help is incredibly appreciated!
>
> --
>
> --cade
>
> Links:
> ------
> [1]
> http://stackoverflow.com/questions/35283549/java-service-wrapper-parent-process-is-hung
>
> ------------------------------------------------------------------------------
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapper process is hung

Leif Mortenson-3
In reply to this post by Cade Thacker
Cade,
I am sorry for the trouble.
I replied to you on Stack Overflow.
Please let me know the version of the Wrapper, OS, and content of your syslog when the JVM was killed.

Thanks,
Leif


On Tue, Feb 9, 2016 at 1:41 PM, Cade Thacker <[hidden email]> wrote:
Hello All,
I'm not sure how to figure this out so figured I would turn to the experts.  

I documented everything here: 


But the simple summary is that when I issue a kill $pid (SIGTERM/15) on the wrapper process (the JVM is long dead via something else killing it with SIGKILL/9) nothing happens. (this is the same call from stopit() in the wrapper.sh distributed by you all. 

As documented in the other question, that causes other issues downstream because without the wrapper dying correctling, I can't restart them both. 

Any ideas on how to best trouble shoot this?  I'm not sure what would be "stuck" in the wrapper C code so that a SIGTERM would not be properly processed? SIGKILL/9 works as expected. 

If this was a Java process, I'd get a thread dump and a heap dump, but not sure how to attack this C code.  Maybe dig through something in /proc/$pid? 

Any help is incredibly appreciated!


--

--cade


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: wrapper process is hung

Cade Thacker
Hello Leif,
First off thank you so much for the response.  

I updated the SO question with some additional details.  We are running a very old version at 3.3.6 (not sure how to exactly verify this, but that is my best guess).  It has been running years without issue, so unfortunately out of site, out of mind. 

I looked at the /var/log/messages at the time of the SIGKILL, but didn't see anything in there logged from the C code.

I'm open to any ideas on how to isolate both the SIGKILL and the hung wrapper.  

Thanks in advance. 

--cade

On Tue, Feb 9, 2016 at 10:05 PM, Leif Mortenson <[hidden email]> wrote:
Cade,
I am sorry for the trouble.
I replied to you on Stack Overflow.
Please let me know the version of the Wrapper, OS, and content of your syslog when the JVM was killed.

Thanks,
Leif


On Tue, Feb 9, 2016 at 1:41 PM, Cade Thacker <[hidden email]> wrote:
Hello All,
I'm not sure how to figure this out so figured I would turn to the experts.  

I documented everything here: 


But the simple summary is that when I issue a kill $pid (SIGTERM/15) on the wrapper process (the JVM is long dead via something else killing it with SIGKILL/9) nothing happens. (this is the same call from stopit() in the wrapper.sh distributed by you all. 

As documented in the other question, that causes other issues downstream because without the wrapper dying correctling, I can't restart them both. 

Any ideas on how to best trouble shoot this?  I'm not sure what would be "stuck" in the wrapper C code so that a SIGTERM would not be properly processed? SIGKILL/9 works as expected. 

If this was a Java process, I'd get a thread dump and a heap dump, but not sure how to attack this C code.  Maybe dig through something in /proc/$pid? 

Any help is incredibly appreciated!


--

--cade


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user




--

--cade

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Loading...