RESTART not wanting to work

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

RESTART not wanting to work

David McCullars
I've got the wrapper working well on a CentOS 3.6 system to wrap JBoss 4
(with Sun's Java 1.5.0_05-b05).  It starts and stops perfectly, but I
was really needing the restart filter to work on out of memory errors
(something we've had a problem with historically).  I added the trigger,
and it attempts to do a restart when I create an OOME, but the wrapper
crashes 9 times out of 10 when it checks to see if the JVM halted:

Shutdown complete
Halting VM
Critical error: wait for JVM process failed (No child processes)

I've tried all sorts of wrapper.conf settings to see if something works,
but it's only restarted twice, and it never did it more than once in a
row.  Here's the latest config file for what it's worth.  I'm really
hoping I'm just doing something stupid because JSW kicks ass!

set.JAVA_HOME=/usr/local/java
set.JBOSS_HOME=/usr/local/jboss
wrapper.java.command=%JAVA_HOME%/bin/java
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
wrapper.java.classpath.1=../lib/wrapper.jar
wrapper.java.classpath.2=%JBOSS_HOME%/bin/run.jar
wrapper.java.classpath.3=%JAVA_HOME%/lib/tools.jar
wrapper.java.library.path.1=../lib
wrapper.java.additional.1=-Dprogram.name=run.sh
wrapper.java.additional.2=-Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
wrapper.java.additional.3=-Djava.endorsed.dirs=%JBOSS_HOME%/lib/endorsed
wrapper.java.additional.4=-verbosegc
wrapper.java.initmemory=1024
wrapper.java.maxmemory=1024
wrapper.app.parameter.1=org.jboss.Main
wrapper.console.format=M
wrapper.console.loglevel=INFO
wrapper.logfile=../jboss.log
wrapper.logfile.format=LTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=10m
wrapper.logfile.maxfiles=4
wrapper.syslog.loglevel=NONE
wrapper.filter.trigger.1=java.lang.OutOfMemoryError
wrapper.filter.action.1=RESTART
wrapper.restart.delay=30

If anyone can throw some light my way, I'd be most indebted!

David


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|

Re: RESTART not wanting to work

Leif Mortenson-2
David,
    Reviewing the code of 3.1.2 and the latest trunk, I think that I may
have fixed this
while implementing another change.  Could you please add the following
properties:
wrapper.debug=true
wrapper.state_output=true
wrapper.loop_output=true

    This will kick out a lot of output, but it will tell me exactly what
is happening.
Once you have reproduced the problem, tar.gz the resulting wrapper.log
file and
then send it to me directly off list.

    I have also created a snapshot of the latest build undergoing
testing for the next
release.  I would appreciate any feedback about whether or not it is
fixed using this
version.  If not, then I would also like to see the same output as for
the 3.1.2
version above.
http://wrapper.tanukisoftware.org/tmp/3.2.0-c/wrapper-linux-x86-32-3.2.0-c.tar.gz
(This version will NOT be supported once the 3.2.0 release is out.)

Cheers,
Leif


David McCullars wrote:

> I've got the wrapper working well on a CentOS 3.6 system to wrap JBoss 4
> (with Sun's Java 1.5.0_05-b05).  It starts and stops perfectly, but I
> was really needing the restart filter to work on out of memory errors
> (something we've had a problem with historically).  I added the trigger,
> and it attempts to do a restart when I create an OOME, but the wrapper
> crashes 9 times out of 10 when it checks to see if the JVM halted:
>
> Shutdown complete
> Halting VM
> Critical error: wait for JVM process failed (No child processes)
>
> I've tried all sorts of wrapper.conf settings to see if something works,
> but it's only restarted twice, and it never did it more than once in a
> row.  Here's the latest config file for what it's worth.  I'm really
> hoping I'm just doing something stupid because JSW kicks ass!
>
> set.JAVA_HOME=/usr/local/java
> set.JBOSS_HOME=/usr/local/jboss
> wrapper.java.command=%JAVA_HOME%/bin/java
> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
> wrapper.java.classpath.1=../lib/wrapper.jar
> wrapper.java.classpath.2=%JBOSS_HOME%/bin/run.jar
> wrapper.java.classpath.3=%JAVA_HOME%/lib/tools.jar
> wrapper.java.library.path.1=../lib
> wrapper.java.additional.1=-Dprogram.name=run.sh
> wrapper.java.additional.2=-Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
>
> wrapper.java.additional.3=-Djava.endorsed.dirs=%JBOSS_HOME%/lib/endorsed
> wrapper.java.additional.4=-verbosegc
> wrapper.java.initmemory=1024
> wrapper.java.maxmemory=1024
> wrapper.app.parameter.1=org.jboss.Main
> wrapper.console.format=M
> wrapper.console.loglevel=INFO
> wrapper.logfile=../jboss.log
> wrapper.logfile.format=LTM
> wrapper.logfile.loglevel=INFO
> wrapper.logfile.maxsize=10m
> wrapper.logfile.maxfiles=4
> wrapper.syslog.loglevel=NONE
> wrapper.filter.trigger.1=java.lang.OutOfMemoryError
> wrapper.filter.action.1=RESTART
> wrapper.restart.delay=30
>
> If anyone can throw some light my way, I'd be most indebted!
>
> David



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|

Re: RESTART not wanting to work

Richard Emberson
If you are using JBoss and its logging to a file, then the wrapper will
not see
the OOME error log message. You MUST also log (at least at the error level)
to the console for the wrapper to see the OOME error message.
We just discovered this truth. Restart on OOME used to work. Then someone
change the log4j.xml file (no longer logging to console). Then the JBoss
system stopped restarting on OOME. It took a while to figure out that
the left hand did not know why the right hand had set up the log4j in a
certain way.
good luck.

RME


Leif Mortenson wrote:

> David,
>    Reviewing the code of 3.1.2 and the latest trunk, I think that I
> may have fixed this
> while implementing another change.  Could you please add the following
> properties:
> wrapper.debug=true
> wrapper.state_output=true
> wrapper.loop_output=true
>
>    This will kick out a lot of output, but it will tell me exactly
> what is happening.
> Once you have reproduced the problem, tar.gz the resulting wrapper.log
> file and
> then send it to me directly off list.
>
>    I have also created a snapshot of the latest build undergoing
> testing for the next
> release.  I would appreciate any feedback about whether or not it is
> fixed using this
> version.  If not, then I would also like to see the same output as for
> the 3.1.2
> version above.
> http://wrapper.tanukisoftware.org/tmp/3.2.0-c/wrapper-linux-x86-32-3.2.0-c.tar.gz 
>
> (This version will NOT be supported once the 3.2.0 release is out.)
>
> Cheers,
> Leif
>
>
> David McCullars wrote:
>
>> I've got the wrapper working well on a CentOS 3.6 system to wrap JBoss 4
>> (with Sun's Java 1.5.0_05-b05).  It starts and stops perfectly, but I
>> was really needing the restart filter to work on out of memory errors
>> (something we've had a problem with historically).  I added the trigger,
>> and it attempts to do a restart when I create an OOME, but the wrapper
>> crashes 9 times out of 10 when it checks to see if the JVM halted:
>>
>> Shutdown complete
>> Halting VM
>> Critical error: wait for JVM process failed (No child processes)
>>
>> I've tried all sorts of wrapper.conf settings to see if something works,
>> but it's only restarted twice, and it never did it more than once in a
>> row.  Here's the latest config file for what it's worth.  I'm really
>> hoping I'm just doing something stupid because JSW kicks ass!
>>
>> set.JAVA_HOME=/usr/local/java
>> set.JBOSS_HOME=/usr/local/jboss
>> wrapper.java.command=%JAVA_HOME%/bin/java
>> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
>> wrapper.java.classpath.1=../lib/wrapper.jar
>> wrapper.java.classpath.2=%JBOSS_HOME%/bin/run.jar
>> wrapper.java.classpath.3=%JAVA_HOME%/lib/tools.jar
>> wrapper.java.library.path.1=../lib
>> wrapper.java.additional.1=-Dprogram.name=run.sh
>> wrapper.java.additional.2=-Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
>>
>> wrapper.java.additional.3=-Djava.endorsed.dirs=%JBOSS_HOME%/lib/endorsed
>> wrapper.java.additional.4=-verbosegc
>> wrapper.java.initmemory=1024
>> wrapper.java.maxmemory=1024
>> wrapper.app.parameter.1=org.jboss.Main
>> wrapper.console.format=M
>> wrapper.console.loglevel=INFO
>> wrapper.logfile=../jboss.log
>> wrapper.logfile.format=LTM
>> wrapper.logfile.loglevel=INFO
>> wrapper.logfile.maxsize=10m
>> wrapper.logfile.maxfiles=4
>> wrapper.syslog.loglevel=NONE
>> wrapper.filter.trigger.1=java.lang.OutOfMemoryError
>> wrapper.filter.action.1=RESTART
>> wrapper.restart.delay=30
>>
>> If anyone can throw some light my way, I'd be most indebted!
>>
>> David
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through log
> files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
> _______________________________________________
> Wrapper-user mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>


--
This email message is for the sole use of the intended recipient(s) and
may contain confidential information.  Any unauthorized review, use,
disclosure or distribution is prohibited.  If you are not the intended
recipient, please contact the sender by reply email and destroy all
copies of the original message.



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|

Re: RESTART not wanting to work

David McCullars
Thanks for the feedback.  However, we actually already had JBoss
outputing to the console, so the wrapper was seeing the OOME -- it was
shuting down just fine upon detection of the OOME, just not coming back up.

The good news, though, is that Leif's latest changes mentioned below
fixed the issue.  So if anyone else is having this issue, you'll want to
upgrade to 3.2.0.

David

Richard Emberson wrote:

> If you are using JBoss and its logging to a file, then the wrapper
> will not see
> the OOME error log message. You MUST also log (at least at the error
> level)
> to the console for the wrapper to see the OOME error message.
> We just discovered this truth. Restart on OOME used to work. Then someone
> change the log4j.xml file (no longer logging to console). Then the JBoss
> system stopped restarting on OOME. It took a while to figure out that
> the left hand did not know why the right hand had set up the log4j in a
> certain way.
> good luck.
>
> RME
>
>
> Leif Mortenson wrote:
>
>> David,
>>    Reviewing the code of 3.1.2 and the latest trunk, I think that I
>> may have fixed this
>> while implementing another change.  Could you please add the
>> following properties:
>> wrapper.debug=true
>> wrapper.state_output=true
>> wrapper.loop_output=true
>>
>>    This will kick out a lot of output, but it will tell me exactly
>> what is happening.
>> Once you have reproduced the problem, tar.gz the resulting
>> wrapper.log file and
>> then send it to me directly off list.
>>
>>    I have also created a snapshot of the latest build undergoing
>> testing for the next
>> release.  I would appreciate any feedback about whether or not it is
>> fixed using this
>> version.  If not, then I would also like to see the same output as
>> for the 3.1.2
>> version above.
>> http://wrapper.tanukisoftware.org/tmp/3.2.0-c/wrapper-linux-x86-32-3.2.0-c.tar.gz
>>
>> (This version will NOT be supported once the 3.2.0 release is out.)
>>
>> Cheers,
>> Leif
>>
>>
>> David McCullars wrote:
>>
>>> I've got the wrapper working well on a CentOS 3.6 system to wrap
>>> JBoss 4
>>> (with Sun's Java 1.5.0_05-b05).  It starts and stops perfectly, but I
>>> was really needing the restart filter to work on out of memory errors
>>> (something we've had a problem with historically).  I added the
>>> trigger,
>>> and it attempts to do a restart when I create an OOME, but the wrapper
>>> crashes 9 times out of 10 when it checks to see if the JVM halted:
>>>
>>> Shutdown complete
>>> Halting VM
>>> Critical error: wait for JVM process failed (No child processes)
>>>
>>> I've tried all sorts of wrapper.conf settings to see if something
>>> works,
>>> but it's only restarted twice, and it never did it more than once in a
>>> row.  Here's the latest config file for what it's worth.  I'm really
>>> hoping I'm just doing something stupid because JSW kicks ass!
>>>
>>> set.JAVA_HOME=/usr/local/java
>>> set.JBOSS_HOME=/usr/local/jboss
>>> wrapper.java.command=%JAVA_HOME%/bin/java
>>> wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
>>> wrapper.java.classpath.1=../lib/wrapper.jar
>>> wrapper.java.classpath.2=%JBOSS_HOME%/bin/run.jar
>>> wrapper.java.classpath.3=%JAVA_HOME%/lib/tools.jar
>>> wrapper.java.library.path.1=../lib
>>> wrapper.java.additional.1=-Dprogram.name=run.sh
>>> wrapper.java.additional.2=-Djavax.xml.transform.TransformerFactory=org.apache.xalan.processor.TransformerFactoryImpl
>>>
>>> wrapper.java.additional.3=-Djava.endorsed.dirs=%JBOSS_HOME%/lib/endorsed
>>>
>>> wrapper.java.additional.4=-verbosegc
>>> wrapper.java.initmemory=1024
>>> wrapper.java.maxmemory=1024
>>> wrapper.app.parameter.1=org.jboss.Main
>>> wrapper.console.format=M
>>> wrapper.console.loglevel=INFO
>>> wrapper.logfile=../jboss.log
>>> wrapper.logfile.format=LTM
>>> wrapper.logfile.loglevel=INFO
>>> wrapper.logfile.maxsize=10m
>>> wrapper.logfile.maxfiles=4
>>> wrapper.syslog.loglevel=NONE
>>> wrapper.filter.trigger.1=java.lang.OutOfMemoryError
>>> wrapper.filter.action.1=RESTART
>>> wrapper.restart.delay=30
>>>
>>> If anyone can throw some light my way, I'd be most indebted!
>>>
>>> David
>>
>>
>>
>>
>>
>> -------------------------------------------------------
>> This SF.net email is sponsored by: Splunk Inc. Do you grep through
>> log files
>> for problems?  Stop!  Download the new AJAX search engine that makes
>> searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
>> _______________________________________________
>> Wrapper-user mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/wrapper-user
>>
>
>


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|

Re: RESTART not wanting to work

Leif Mortenson-2
In reply to this post by Leif Mortenson-2
David,
    Thanks for the log.  I'm pretty sure this problem is now fixed.  I
added a note
about this problem to the release notes as well.

    Glad you have found the wrapper useful. :-)

Cheers,
Leif


David McCullars wrote:

> Thanks for such a speedy reply!  I added the debugging as requested
> and attached the output of the error.  But good news, I tried the
> latest build as you suggested, and it has worked several times in a
> row.  I am assuming since it seems to be working you don't wish a log
> file for it, but I'll be happy to send you one if you desire.
>
> So I guess I'll just use this latest version until 3.2.0 becomes
> available.  (Makes me feel a little better to know the problem wasn't
> between the keyboard and chair.)  Anyway, absolutely love this
> application -- it kicks ass!
>
> David



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user