Quantcast

Is it possible to have a dynamically defined restart delay?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Is it possible to have a dynamically defined restart delay?

Tue S. Dissing

Hi,

 

We’re running with a lot of simultaneous jobs each processing its own thing.

Each run with its own wrapper configuration.

 

We have no control over how busy these processes are, but we would like to get tasks processed as soon as they tick in – i.e. we have no can control over the amount of work coming in.

 

So I’m looking for a way to dynamically defining the restart delay.

 

 

Let’s say our jobs run every 60 seconds, then I would like to increase the delay to say 5 minutes for individual jobs if their processing time is below a certain threshold; i.e. if it only run for 5 sec.

 

The reason for this is that we’re seeing a heavy load on the CPU when all these jobs starts and stops all the time (and fires up the JVM which then, in vain, looks for new records to process)– we do not process much data but the CPU load is close to 90% at all time.

 

Any other suggestion that might help us with this problem is more than welcome.

 

Best regards

 

Tue S. Dissing

Technical Lead

 

cid:image001.jpg@01C9EB3A.0E7503B0

 

Mobile: +45 29710400

[hidden email]

www.cdrator.com

 


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it possible to have a dynamically defined restart delay?

Leif Mortenson-3
Tue,
The restart delay itself can't be changed in that way because it sounds like you are actually not sure how long it will take.

Maybe a better solution is to put the Wrapper into a Paused state:
http://wrapper.tanukisoftware.com/doc/english/prop-pausable.html

There is then a number of ways to resume the Wrapper and thus start your application.
* Command File and the RESUME command:
http://wrapper.tanukisoftware.com/doc/english/prop-commandfile.html

* Resume the service using service manager or command line

If you use the Pause / Resume method, you will have to make sure that the Wrapper is reliably resumed in all cases.
If you use a timer to resume every minute for example, you could have your Java code immediately repause it and stop the JVM if some condition is not met.
You could also use the timer to run every 15 mintues or something in the same way as a last resort in case the normal resume fails to be fired.
Normally you could run your external job and then use "net resume myapp" at the end to resume the Wrapper.

Please let me know how this works for you.

Cheers,
Leif




On Thu, Jun 5, 2014 at 5:11 PM, Tue S. Dissing <[hidden email]> wrote:

Hi,

 

We’re running with a lot of simultaneous jobs each processing its own thing.

Each run with its own wrapper configuration.

 

We have no control over how busy these processes are, but we would like to get tasks processed as soon as they tick in – i.e. we have no can control over the amount of work coming in.

 

So I’m looking for a way to dynamically defining the restart delay.

 

 

Let’s say our jobs run every 60 seconds, then I would like to increase the delay to say 5 minutes for individual jobs if their processing time is below a certain threshold; i.e. if it only run for 5 sec.

 

The reason for this is that we’re seeing a heavy load on the CPU when all these jobs starts and stops all the time (and fires up the JVM which then, in vain, looks for new records to process)– we do not process much data but the CPU load is close to 90% at all time.

 

Any other suggestion that might help us with this problem is more than welcome.

 

Best regards

 

Tue S. Dissing

Technical Lead

 



------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it possible to have a dynamically defined restart delay?

Leif Mortenson-3
Tue,
Another option that I just thought of requires the Professional Edition but should work smoothly.  You can give it a try with a free Trial license.

Register a command to be executed in response to the "jvm_prelaunch" event and set it up to block until complete.
You can then run your external script and the Wrapper will wait for it to complete before launching the new JVM.
This would happen automatically on startup and whenever you restarted the JVM.

It would look something like this in your wrapper.conf:
wrapper.event.jvm_prelaunch.command.argv.1=MyJob.bat
wrapper.event.jvm_prelaunch.command.block=TRUE

You can also set it up to have a block timeout, control what to do if the that timeout takes place, and perform a different action if the batch script returns specific error codes.

Cheers,
Leif




On Thu, Jun 5, 2014 at 5:52 PM, Leif Mortenson <[hidden email]> wrote:
Tue,
The restart delay itself can't be changed in that way because it sounds like you are actually not sure how long it will take.

Maybe a better solution is to put the Wrapper into a Paused state:
http://wrapper.tanukisoftware.com/doc/english/prop-pausable.html

There is then a number of ways to resume the Wrapper and thus start your application.
* Command File and the RESUME command:
http://wrapper.tanukisoftware.com/doc/english/prop-commandfile.html

* Resume the service using service manager or command line

If you use the Pause / Resume method, you will have to make sure that the Wrapper is reliably resumed in all cases.
If you use a timer to resume every minute for example, you could have your Java code immediately repause it and stop the JVM if some condition is not met.
You could also use the timer to run every 15 mintues or something in the same way as a last resort in case the normal resume fails to be fired.
Normally you could run your external job and then use "net resume myapp" at the end to resume the Wrapper.

Please let me know how this works for you.

Cheers,
Leif




On Thu, Jun 5, 2014 at 5:11 PM, Tue S. Dissing <[hidden email]> wrote:

Hi,

 

We’re running with a lot of simultaneous jobs each processing its own thing.

Each run with its own wrapper configuration.

 

We have no control over how busy these processes are, but we would like to get tasks processed as soon as they tick in – i.e. we have no can control over the amount of work coming in.

 

So I’m looking for a way to dynamically defining the restart delay.

 

 

Let’s say our jobs run every 60 seconds, then I would like to increase the delay to say 5 minutes for individual jobs if their processing time is below a certain threshold; i.e. if it only run for 5 sec.

 

The reason for this is that we’re seeing a heavy load on the CPU when all these jobs starts and stops all the time (and fires up the JVM which then, in vain, looks for new records to process)– we do not process much data but the CPU load is close to 90% at all time.

 

Any other suggestion that might help us with this problem is more than welcome.

 

Best regards

 

Tue S. Dissing

Technical Lead




------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it possible to have a dynamically defined restart delay?

Tue S. Dissing

Thanks, but not 100% what I was looking for.

 

 

Gave me another idea though.

 

What if I set both failed_invocation_time and fail_invocations to 5.

Is there some way to pause the wrapper let’s say 10 minutes and reset the failed_invocations?

 

 

Best regards

 

Tue S. Dissing

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: 5. juni 2014 10:58
To: Wrapper User List
Subject: Re: [Wrapper-user] Is it possible to have a dynamically defined restart delay?

 

Tue,
Another option that I just thought of requires the Professional Edition but should work smoothly.  You can give it a try with a free Trial license.

Register a command to be executed in response to the "jvm_prelaunch" event and set it up to block until complete.
You can then run your external script and the Wrapper will wait for it to complete before launching the new JVM.
This would happen automatically on startup and whenever you restarted the JVM.

It would look something like this in your wrapper.conf:
wrapper.event.jvm_prelaunch.command.argv.1=MyJob.bat
wrapper.event.jvm_prelaunch.command.block=TRUE

You can also set it up to have a block timeout, control what to do if the that timeout takes place, and perform a different action if the batch script returns specific error codes.

Cheers,
Leif

 

 

On Thu, Jun 5, 2014 at 5:52 PM, Leif Mortenson <[hidden email]> wrote:

Tue,
The restart delay itself can't be changed in that way because it sounds like you are actually not sure how long it will take.

Maybe a better solution is to put the Wrapper into a Paused state:
http://wrapper.tanukisoftware.com/doc/english/prop-pausable.html

There is then a number of ways to resume the Wrapper and thus start your application.
* Command File and the RESUME command:
http://wrapper.tanukisoftware.com/doc/english/prop-commandfile.html


* Resume the service using service manager or command line

If you use the Pause / Resume method, you will have to make sure that the Wrapper is reliably resumed in all cases.

If you use a timer to resume every minute for example, you could have your Java code immediately repause it and stop the JVM if some condition is not met.

You could also use the timer to run every 15 mintues or something in the same way as a last resort in case the normal resume fails to be fired.

Normally you could run your external job and then use "net resume myapp" at the end to resume the Wrapper.

Please let me know how this works for you.

 

Cheers,
Leif

 

 

On Thu, Jun 5, 2014 at 5:11 PM, Tue S. Dissing <[hidden email]> wrote:

Hi,

 

We’re running with a lot of simultaneous jobs each processing its own thing.

Each run with its own wrapper configuration.

 

We have no control over how busy these processes are, but we would like to get tasks processed as soon as they tick in – i.e. we have no can control over the amount of work coming in.

 

So I’m looking for a way to dynamically defining the restart delay.

 

 

Let’s say our jobs run every 60 seconds, then I would like to increase the delay to say 5 minutes for individual jobs if their processing time is below a certain threshold; i.e. if it only run for 5 sec.

 

The reason for this is that we’re seeing a heavy load on the CPU when all these jobs starts and stops all the time (and fires up the JVM which then, in vain, looks for new records to process)– we do not process much data but the CPU load is close to 90% at all time.

 

Any other suggestion that might help us with this problem is more than welcome.

 

Best regards

 

Tue S. Dissing

Technical Lead

 

 


------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it possible to have a dynamically defined restart delay?

Leif Mortenson-3
Tue,
The wrapper.successful_invocation_time property lets you define how many seconds a JVM needs to be running before the failed invocation count is reset.
The wrapper.max_failed_invocations property then controls how many times the Wrapper will try to restart the JVM before giving up and shutting down.
I guess it would be possible to use these to restart the JVM several times waiting for your other process to complete.
The problem though is that JVMs are pretty resource intensive when they are starting up.

The idea using the jvm_prelaunch event would let you avoid the extra restarts by waiting for your process to complete.

Let us know what you end up doing.

Cheers,
Leif



On Thu, Jun 5, 2014 at 9:28 PM, Tue S. Dissing <[hidden email]> wrote:

Thanks, but not 100% what I was looking for.

 

 

Gave me another idea though.

 

What if I set both failed_invocation_time and fail_invocations to 5.

Is there some way to pause the wrapper let’s say 10 minutes and reset the failed_invocations?

 

 

Best regards

 

Tue S. Dissing

 

From: Leif Mortenson [mailto:[hidden email]]
Sent: 5. juni 2014 10:58
To: Wrapper User List
Subject: Re: [Wrapper-user] Is it possible to have a dynamically defined restart delay?

 

Tue,
Another option that I just thought of requires the Professional Edition but should work smoothly.  You can give it a try with a free Trial license.

Register a command to be executed in response to the "jvm_prelaunch" event and set it up to block until complete.
You can then run your external script and the Wrapper will wait for it to complete before launching the new JVM.
This would happen automatically on startup and whenever you restarted the JVM.

It would look something like this in your wrapper.conf:
wrapper.event.jvm_prelaunch.command.argv.1=MyJob.bat
wrapper.event.jvm_prelaunch.command.block=TRUE

You can also set it up to have a block timeout, control what to do if the that timeout takes place, and perform a different action if the batch script returns specific error codes.

Cheers,
Leif

 

 

On Thu, Jun 5, 2014 at 5:52 PM, Leif Mortenson <[hidden email]> wrote:

Tue,
The restart delay itself can't be changed in that way because it sounds like you are actually not sure how long it will take.

Maybe a better solution is to put the Wrapper into a Paused state:
http://wrapper.tanukisoftware.com/doc/english/prop-pausable.html

There is then a number of ways to resume the Wrapper and thus start your application.
* Command File and the RESUME command:
http://wrapper.tanukisoftware.com/doc/english/prop-commandfile.html


* Resume the service using service manager or command line

If you use the Pause / Resume method, you will have to make sure that the Wrapper is reliably resumed in all cases.

If you use a timer to resume every minute for example, you could have your Java code immediately repause it and stop the JVM if some condition is not met.

You could also use the timer to run every 15 mintues or something in the same way as a last resort in case the normal resume fails to be fired.

Normally you could run your external job and then use "net resume myapp" at the end to resume the Wrapper.

Please let me know how this works for you.

 

Cheers,
Leif

 

 

On Thu, Jun 5, 2014 at 5:11 PM, Tue S. Dissing <[hidden email]> wrote:

Hi,

 

We’re running with a lot of simultaneous jobs each processing its own thing.

Each run with its own wrapper configuration.

 

We have no control over how busy these processes are, but we would like to get tasks processed as soon as they tick in – i.e. we have no can control over the amount of work coming in.

 

So I’m looking for a way to dynamically defining the restart delay.

 

 

Let’s say our jobs run every 60 seconds, then I would like to increase the delay to say 5 minutes for individual jobs if their processing time is below a certain threshold; i.e. if it only run for 5 sec.

 

The reason for this is that we’re seeing a heavy load on the CPU when all these jobs starts and stops all the time (and fires up the JVM which then, in vain, looks for new records to process)– we do not process much data but the CPU load is close to 90% at all time.

 

Any other suggestion that might help us with this problem is more than welcome.

 

Best regards

 

Tue S. Dissing

Technical Lead

 



------------------------------------------------------------------------------
Learn Graph Databases - Download FREE O'Reilly Book
"Graph Databases" is the definitive new guide to graph databases and their
applications. Written by three acclaimed leaders in the field,
this first edition is now available. Download your free book today!
http://p.sf.net/sfu/NeoTech
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Loading...