Breno Silva | 7 Sep 18:49 2011
Picon

Re: Failed to access DBM file (data/ip): Resource deadlock avoided

Hi Kevin,

We are planning to implement  a DDoS learning algorithm for ModSecurity. I know using data collection in heavy systems can cause performance issues. Most of Apache modules which works for  bandwidth limitation also stores data into temporary files, and it will also cause performance problems. So we need some code to store network information only in memory, persist only the learned data for recovery. It will also requires a time-to-time cleanup of internal modsecurity data structs. So i think in the future we will have a good solution.

Until we do not have a better solution, maybe you could try limit per ip connections in a another network device (firewall, sw) ?

Thanks

Breno

On Wed, Sep 7, 2011 at 10:42 AM, Kevin Jackson <kevin <at> linuxservices.co.uk> wrote:
Dear all,
It's been a while since I posted this issue and I want to share with
you some of my experiences as I'm seeing issues when the IP collection
grows in size.

My setup is Apache 2.2.20, ModSecurity 2.6.1 and CRS 2.2.1.
After a lot of performance testing and configuring ModSecurity to do 1
job (protect against a DoS against the web servers [which in turn
would protect against the real assets that ModSecurity setup is
proxying for]) on a "2CPU", 7Gb box (in EC2, m1.large) I get somewhere
between 200 and 300 hits per second against a static resource served
from Apache.  Not great, but acceptable.  What isn't acceptable is the
wildly fluctuating figures I get and load hitting 200+ when the IP
data collection grows in size.

My performance tests are replaying existing logs with real IP and URIs
in, but I use the IP to set an X-Forwarded-For header (so ModSec sees
it as a new IP), it uses the hash of the UA to give a level of
uniqueness to the IP as per the CRS configs and I rewrite all URLs to
a static file (to ensure I'm not running stuff against live in any
shape or form).

All is good at the beginning of the run and my IP collection database
is at 0Mb.  I get around 300 hits per second at a load avg of around
0.4.  Over a short period of time (at the rate mentioned above), this
IP collection builds up and when it hits a certain size, its game
over.  High load starts around 5-10 minutes.  Leaving this running for
20 mins sees load shoot up to 200+.  This isn't a gradual increase, it
hits a point and fails.  This seems to be file size as interrogating
the SDBM file lists for the IP collections it is a small figure.  For
example, I have a 700Mb SDBM file and inside are only 28 IP
collections.

I've played with TIMEOUT values to see if keeping less is helpful and
won't affect things too much, but ideally for DoS protection I'd hope
that keeping the IP in the database for as long as possible will be
more beneficial.

Any help is appreciated - as a potential mitigation step against a
Layer 7 DDoS (network and other good stuff in place to protect against
this aside) - what can I do from here?

Cheers,

Kev

On 19 August 2011 13:37, Ryan Barnett <RBarnett <at> trustwave.com> wrote:
>
> On 8/19/11 6:55 AM, "Kevin Jackson" <kevin <at> linuxservices.co.uk> wrote:
>
>>Dear all,
>>After a number of weeks fine tuning the CRS rules appropriate for a
>>busy website, we did a full live test pushing traffic through Apache
>>with ModSecurity 2.6.1 proxying requests to evaluate the performance
>>of ModSecurity through the backend service.
>>
>>Traffic is around 1k/second and we had Apache running on 18 servers
>>sitting behind a load balancer and Apache coping without issue.
>>
>>ModSecurity on the other hand didn't fare so well - it struggled to
>>keep pace causing resource deadlock issues on the ip database.  The
>>upshot was incredibly high load averages for obvious reasons and poor
>>user experience.
>>
>>--ba66410d-H--
>>Message: Failed to access DBM file "/usr/local/apache/data/ip":
>>Resource deadlock avoided
>>Apache-Handler: proxy-server
>>Stopwatch: 1313660329409820 283469 (- - -)
>>Stopwatch2: 1313660329409820 283469; combined=99526, p1=3446,
>>p2=12838, p3=3, p4=0, p5=41666, sr=3272, sw=41573, l=0, gc=0
>>Producer: ModSecurity for Apache/2.6.1 (http://www.modsecurity.org/);
>>core ruleset/2.2.1.
>>Server: Apache
>>
>>--ba66410d-Z--
>>
>>My question is what to suggest here? People must be running
>>ModSecurity on high volume websites? Have you encountered this issue?
>>This traffic is a fraction of our peak traffic at a quiet time during
>>the day and the environment only coped for about 10 minutes before
>>keeling over.
>
> Hey Kevin,
> Question for you - are you using the IP persistent collection to track
> data about clients across requests?  If not, then you may want to disable
> the initcol rules at the end of the modsecurity_crs_10_config.conf file.
>
> See if that helps.
>
> -Ryan
>
>>
>>Cheers,
>>
>>Kev
>>
>>--------------------------------------------------------------------------
>>----
>>Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
>>user administration capabilities and model configuration. Take
>>the hassle out of deploying and managing Subversion and the
>>tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
>>_______________________________________________
>>mod-security-users mailing list
>>mod-security-users <at> lists.sourceforge.net
>>https://lists.sourceforge.net/lists/listinfo/mod-security-users
>>ModSecurity Services from Trustwave's SpiderLabs:
>>https://www.trustwave.com/application-security.php
>>
>
>
> This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.
>
>

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

------------------------------------------------------------------------------
Using storage to extend the benefits of virtualization and iSCSI
Virtualization increases hardware utilization and delivers a new level of
agility. Learn what those decisions are and how to modernize your storage 
and backup environments for virtualization.
http://www.accelacomm.com/jaw/sfnl/114/51434361/
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users
ModSecurity Services from Trustwave's SpiderLabs:
https://www.trustwave.com/application-security.php

Gmane