Features Download
From: Breno Silva <breno.silva <at> gmail.com>
Subject: Re: Failed to access DBM file (data/ip): Resource deadlock avoided
Newsgroups: gmane.comp.apache.mod-security.user
Date: Wednesday 7th September 2011 16:49:31 UTC (over 5 years ago)
Hi Kevin,

We are planning to implement  a DDoS learning algorithm for ModSecurity. I
know using data collection in heavy systems can cause performance issues.
Most of Apache modules which works for  bandwidth limitation also stores
data into temporary files, and it will also cause performance problems. So
we need some code to store network information only in memory, persist only
the learned data for recovery. It will also requires a time-to-time cleanup
of internal modsecurity data structs. So i think in the future we will have
a good solution.

Until we do not have a better solution, maybe you could try limit per ip
connections in a another network device (firewall, sw) ?



On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson

> Dear all,
> It's been a while since I posted this issue and I want to share with
> you some of my experiences as I'm seeing issues when the IP collection
> grows in size.
> My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
> After a lot of performance testing and configuring ModSecurity to do 1
> job (protect against a DoS against the web servers [which in turn
> would protect against the real assets that ModSecurity setup is
> proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
> between 200 and 300 hits per second against a static resource served
> from Apache.  Not great, but acceptable.  What isn't acceptable is the
> wildly fluctuating figures I get and load hitting 200+ when the IP
> data collection grows in size.
> My performance tests are replaying existing logs with real IP and URIs
> in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
> it as a new IP), it uses the hash of the UA to give a level of
> uniqueness to the IP as per the CRS configs and I rewrite all URLs to
> a static file (to ensure I'm not running stuff against live in any
> shape or form).
> All is good at the beginning of the run and my IP collection database
> is at 0Mb.  I get around 300 hits per second at a load avg of around
> 0.4.  Over a short period of time (at the rate mentioned above), this
> IP collection builds up and when it hits a certain size, its game
> over.  High load starts around 5-10 minutes.  Leaving this running for
> 20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
> hits a point and fails.  This seems to be file size as interrogating
> the SDBM file lists for the IP collections it is a small figure.  For
> example, I have a 700Mb SDBM file and inside are only 28 IP
> collections.
> I've played with TIMEOUT values to see if keeping less is helpful and
> won't affect things too much, but ideally for DoS protection I'd hope
> that keeping the IP in the database for as long as possible will be
> more beneficial.
> Any help is appreciated - as a potential mitigation step against a
> Layer 7 DDoS (network and other good stuff in place to protect against
> this aside) - what can I do from here?
> Cheers,
> Kev
> On 19 August 2011 13:37, Ryan Barnett  wrote:
> >
> > On 8/19/11 6:55 AM, "Kevin Jackson"  wrote:
> >
> >>Dear all,
> >>After a number of weeks fine tuning the CRS rules appropriate for a
> >>busy website, we did a full live test pushing traffic through Apache
> >>with ModSecurity 2.6.1 proxying requests to evaluate the performance
> >>of ModSecurity through the backend service.
> >>
> >>Traffic is around 1k/second and we had Apache running on 18 servers
> >>sitting behind a load balancer and Apache coping without issue.
> >>
> >>ModSecurity on the other hand didn't fare so well - it struggled to
> >>keep pace causing resource deadlock issues on the ip database.  The
> >>upshot was incredibly high load averages for obvious reasons and poor
> >>user experience.
> >>
> >>--ba66410d-H--
> >>Message: Failed to access DBM file "/usr/local/apache/data/ip":
> >>Resource deadlock avoided
> >>Apache-Handler: proxy-server
> >>Stopwatch: 1313660329409820 283469 (- - -)
> >>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
> >>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
> >>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
> >>core ruleset/2.2.1.
> >>Server: Apache
> >>
> >>--ba66410d-Z--
> >>
> >>My question is what to suggest here? People must be running
> >>ModSecurity on high volume websites? Have you encountered this issue?
> >>This traffic is a fraction of our peak traffic at a quiet time during
> >>the day and the environment only coped for about 10 minutes before
> >>keeling over.
> >
> > Hey Kevin,
> > Question for you - are you using the IP persistent collection to track
> > data about clients across requests?  If not, then you may want to
> > the initcol rules at the end of the modsecurity_crs_10_config.conf
> >
> > See if that helps.
> >
> > -Ryan
> >
> >>
> >>Cheers,
> >>
> >>Kev
> >>
> >>----
> >>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
> >>user administration capabilities and model configuration. Take
> >>the hassle out of deploying and managing Subversion and the
> >>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
> >>_______________________________________________
> >>mod-security-users mailing list
> >>[email protected]
> >>https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >>ModSecurity Services from Trustwave's SpiderLabs:
> >>https://www.trustwave.com/application-security.php
> >>
> >
> >
> > This transmission may contain information that is privileged,
> confidential, and/or exempt from disclosure under applicable law. If you
> not the intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> any reliance thereon) is STRICTLY PROHIBITED. If you received this
> transmission in error, please immediately contact the sender and destroy
> material in its entirety, whether in electronic or hard copy format.
> >
> >
> Using storage to extend the benefits of virtualization and iSCSI
> Virtualization increases hardware utilization and delivers a new level of
> agility. Learn what those decisions are and how to modernize your storage
> and backup environments for virtualization.
> http://www.accelacomm.com/jaw/sfnl/114/51434361/
> _______________________________________________
> mod-security-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/mod-security-users
> ModSecurity Services from Trustwave's SpiderLabs:
> https://www.trustwave.com/application-security.php
CD: 2ms