wrapper 3.5.25 memory leak on CentOS 7

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

wrapper 3.5.25 memory leak on CentOS 7

Aleksey Chudov
Hi,

We are using wrapper 3.5.25 community edition on a bunch of high loaded
servers without any issues.

Currently I'm testing the same wrapper and java application on a new
CentOS 7 server and facing a huge memory leak by the wrapper process.

KiB Mem:  16423540 total, 16210652 used,   212888 free,      728 buffers
KiB Swap:  4194300 total,        0 used,  4194300 free.   262020 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
20381 root      20   0 7531520 6.979g    808 S   0.0 44.6   3:00.90 wrapper
20383 root      20   0 19.037g 6.959g   8860 S   1.3 44.4   1175:24 java


At the same time there where a lot of wrapper messages dropped by
systemd-journal because of rate limiting

Nov 15 13:28:09 srv94 journal: Suppressed 27674 messages from
/user.slice/user-1013.slice
Nov 15 13:28:39 srv94 journal: Suppressed 29040 messages from
/user.slice/user-1013.slice
Nov 15 13:29:09 srv94 journal: Suppressed 28509 messages from
/user.slice/user-1013.slice
Nov 15 13:29:39 srv94 journal: Suppressed 19124 messages from
/user.slice/user-1013.slice
Nov 15 13:30:09 srv94 journal: Suppressed 18889 messages from
/user.slice/user-1013.slice
Nov 15 13:30:40 srv94 journal: Suppressed 10076 messages from
/user.slice/user-1013.slice

Do not ask why so many messages :) It's normal behaviour for our
application.

Seem like wrapper leak memory on dropped syslog messages. May be some
internal buffers are not cleaned?

We are using the following log related settings

wrapper.console.format=PM
wrapper.console.loglevel=INFO

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

wrapper.syslog.facility=LOCAL4
wrapper.syslog.loglevel=INFO
wrapper.syslog.ident=myapp


I can reproduce the problem.


Regards,
Aleksey

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|

Re: wrapper 3.5.25 memory leak on CentOS 7

Aleksey Chudov
Finally I found that memory leak relates to wrapper.logfile options. As
soon as I comment the following options in configuration file the
problem goes away.

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

After enabling options one by one I found two problem options. As soon
as I uncomment the following options wrapper starts to eat memory.

wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

I can reproduce the problem.


Regards,
Aleksey


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user
Reply | Threaded
Open this post in threaded view
|

Re: wrapper 3.5.25 memory leak on CentOS 7

Tim Lammens
There are two bugs at play here. One is in glibc, the other in the wrapper itself.

- glibc leak:
It is fixed in newer releases of glibc, so if you can upgrade your glibc libraries please do so.

If you can not upgrade your glibc libraries, see patch 02.workaround-memory-leak-in-ftell.

- wrapper leak:
I've also attached a fix for the memory leak in logging in the wrapper itself in patch 01.fixes-memory-leak-in-error-logging.
It uses the same solution as the windows version to prevent using up all the memory.

Regards,
Tim


On Sun, Nov 16, 2014 at 11:51 AM, Aleksey Chudov <[hidden email]> wrote:
Finally I found that memory leak relates to wrapper.logfile options. As
soon as I comment the following options in configuration file the
problem goes away.

wrapper.logfile=/var/log/myapp.log
wrapper.logfile.format=LPTM
wrapper.logfile.loglevel=INFO
wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

After enabling options one by one I found two problem options. As soon
as I uncomment the following options wrapper starts to eat memory.

wrapper.logfile.maxsize=100m
wrapper.logfile.maxfiles=10

I can reproduce the problem.


Regards,
Aleksey


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://pubads.g.doubleclick.net/gampad/clk?id=154624111&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user

01.fixes-memory-leak-in-error-logging (1K) Download Attachment
02.workaround-memory-leak-in-ftell (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: wrapper 3.5.25 memory leak on CentOS 7

Aleksey Chudov
Thanks for the reply!

Unfortunately I can't update glibc unless it will be updated by upstream
(CentOS maintainers).

So I patched wrapper sources using previously attached patches and quick
test shows memory leak is fixed!

Do you have plans to release a new wrapper version includes the proposed
fixes? Don't want to maintain custom wrapper version :)

Regards,
Aleksey


On 11/17/2014 11:39 AM, Tim Lammens wrote:

> There are two bugs at play here. One is in glibc, the other in the
> wrapper itself.
>
> - glibc leak:
> https://sourceware.org/bugzilla/show_bug.cgi?id=17370
> It is fixed in newer releases of glibc, so if you can upgrade your
> glibc libraries please do so.
>
> If you can not upgrade your glibc libraries, see patch
> 02.workaround-memory-leak-in-ftell.
>
> - wrapper leak:
> I've also attached a fix for the memory leak in logging in the wrapper
> itself in patch 01.fixes-memory-leak-in-error-logging.
> It uses the same solution as the windows version to prevent using up
> all the memory.
>
> Regards,
> Tim
>


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Wrapper-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/wrapper-user