Nicola Bianchi | 26 Jun 07:18

Re: Apache hang on https protocol violation

Brian,
have you tryed with httpS request? Without S I don't have hang problems...

My compiling configurations:

################################################################
tar xvfz httpd-${APACHE_VERSIONE}.tar.gz
cd httpd-${APACHE_VERSIONE}/
./configure \
--prefix=/opt/waf/bin/httpd-${APACHE_VERSIONE} \
--with-mpm=worker --enable-so \
--enable-unique-id \
--enable-proxy --enable-proxy-http --enable-proxy-balancer \
--enable-rewrite --enable-headers \
--enable-logio \
--enable-expires \
--enable-ssl \
--enable-deflate --enable-cache --enable-disk-cache --enable-mem-cache \
--disable-autoindex --disable-asis --disable-cgi --disable-cgid \
--disable-negotiation --disable-userdir \
--with-pcre=/opt/waf/bin/pcre-${PCRE_VERSIONE}
 ################################################################

################################################################
cd modsecurity-apache_${MODSEC_VERSIONE}/apache2/
./configure \
--prefix=/opt/waf/bin/modsecurity-apache_${MODSEC_VERSIONE} \
--with-apxs=/opt/waf/bin/httpd-${APACHE_VERSIONE}/bin/apxs \
--with-apr=/opt/waf/bin/httpd-${APACHE_VERSIONE}/bin \
--with-apu=/opt/waf/bin/httpd-${APACHE_VERSIONE}/bin \
--with-pcre=/opt/waf/bin/pcre-${PCRE_VERSIONE} \
--with-libxml=/opt/waf/bin/libxml2-${XML_VERSIONE} \
--with-lua=/opt/waf/bin/lua-${LUA_VERSIONE} \
--enable-strict-compile
################################################################


On Thu, Jun 26, 2008 at 1:38 AM, Brian Rectanus <Brian.Rectanus <at> breach.com> wrote:
Nick,

I was not able to duplicate this.  Below I have 2.2.9 apache running as
a reverse proxy with modsecurity 2.5.5 and core rules 1.6.1 and mlogc
running to a console.  Each request produced an alert about the IP in
the host header.  Additionally, I up'ed the ab test considerably.  I
also tried mis-configuring  mlogc in various ways, but these yielded
similar results.

There are some differences in our setups.  I have most modules as
modules vs compiled in as you have them.  I am also running 64bit.  But
I do not think these should make that much difference.

If you would send me the exact configure options you used with your
2.2.9 apache I will compile one here and test if you want.


$ httpd -V
Server version: Apache/2.2.9 (Unix)
Server built:   Jun 25 2008 16:25:03
Server's Module Magic Number: 20051115:15
Server loaded:  APR 1.3.0, APR-Util 1.3.0
Compiled using: APR 1.3.0, APR-Util 1.3.0
Architecture:   64-bit
Server MPM:     Worker
 threaded:     yes (fixed thread count)
   forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/apps/httpd-2.2.9"
 -D SUEXEC_BIN="/apps/httpd-2.2.9/bin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

$ httpd -lCompiled in modules:
 core.c
 worker.c
 http_core.c
 mod_so.c

$ ab -k -c 1000 -n 10000 http://127.0.1.1:8100/cgi-bin/dump
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.1.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        FooBar/1.2.3
Server Hostname:        127.0.1.1
Server Port:            8100

Document Path:          /cgi-bin/dump
Document Length:        226 bytes

Concurrency Level:      1000
Time taken for tests:   44.678 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Non-2xx responses:      10000
Keep-Alive requests:    0
Total transferred:      3980000 bytes
HTML transferred:       2260000 bytes
Requests per second:    223.82 [#/sec] (mean)
Time per request:       4467.792 [ms] (mean)
Time per request:       4.468 [ms] (mean, across all concurrent requests)
Transfer rate:          86.99 [Kbytes/sec] received

Connection Times (ms)
             min  mean[+/-sd] median   max
Connect:        0  469 1819.0      0   20999
Processing:     3 3814 4000.3   2614   27551
Waiting:        3 3258 3543.1   2191   26116
Total:          3 4283 4748.7   3025   36558

Percentage of the requests served within a certain time (ms)
 50%   3025
 66%   4818
 75%   6226
 80%   7324
 90%  10264
 95%  13155
 98%  18743
 99%  23293
 100%  36558 (longest request)



Nicola Bianchi wrote:
> Hi Brian,
> here the information that you require!
> If you need additional info just tell me...
>
> Thank you a lot for the help ;)
> Regards.
>   Nick
>
> ##### grep -v "^#" modsecurity_crs_10_config.conf | grep ..
> SecRuleEngine On
> SecRequestBodyAccess On
> SecResponseBodyAccess On
> SecResponseBodyMimeType (null) text/html text/plain text/xml
> SecResponseBodyLimit 524288
> SecServerSignature "Apache/2.2.0 (Fedora)"
> SecComponentSignature "core ruleset/1.6.1"
> SecUploadDir /tmp
> SecUploadKeepFiles Off
> SecAuditEngine RelevantOnly
> SecAuditLogRelevantStatus "^(?:5|4(?!04))"
> SecAuditLogType Serial
> SecAuditLog logs/modsec_audit.log
> SecAuditLogParts "ABIFHKZ"
> SecArgumentSeparator "&"
> SecCookieFormat 0
> SecRequestBodyInMemoryLimit 131072
> SecDebugLog             logs/modsec_debug.log
> SecDebugLogLevel        1
> SecDataDir /tmp
> SecTmpDir /tmp
>
>
> ##### grep -v "^#" modsecurity_crs_15_cb_config.conf | grep ..
> SecRuleEngine On
> SecRequestBodyAccess On
> SecResponseBodyAccess On
> SecResponseBodyMimeType (null) text/html text/plain text/xml
> SecDefaultAction
> "phase:2,log,auditlog,deny,status:403,t:lowercase,t:replaceNulls,t:compressWhitespace"
> SecServerSignature "Server X"
> SecUploadDir /opt/jail/tmp
> SecAuditLogType Concurrent
> SecAuditLog "|bin/mlogc /opt/waf/mod_security/prod/bin/mlogc.conf"
> SecAuditLogStorageDir logs/modsec_audit/
> SecDebugLogLevel        0
> SecDataDir /opt/jail/tmp
> SecTmpDir /opt/jail/tmp
>
>
> ##### /opt/waf/bin/apache_prod/bin/httpd -V
> Server version: Apache/2.2.9 (Unix)
> Server built:   Jun 18 2008 11:18:47
> Server's Module Magic Number: 20051115:15
> Server loaded:  APR 1.3.0, APR-Util 1.3.0
> Compiled using: APR 1.3.0, APR-Util 1.3.0
> Architecture:   32-bit
> Server MPM:     Worker
>   threaded:     yes (fixed thread count)
>     forked:     yes (variable process count)
> Server compiled with....
>  -D APACHE_MPM_DIR="server/mpm/worker"
>  -D APR_HAS_SENDFILE
>  -D APR_HAS_MMAP
>  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>  -D APR_USE_SYSVSEM_SERIALIZE
>  -D APR_USE_PTHREAD_SERIALIZE
>  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>  -D APR_HAS_OTHER_CHILD
>  -D AP_HAVE_RELIABLE_PIPED_LOGS
>  -D DYNAMIC_MODULE_LIMIT=128
>  -D HTTPD_ROOT="/opt/waf/bin/httpd-2.2.9"
>  -D SUEXEC_BIN="/opt/waf/bin/httpd-2.2.9/bin/suexec"
>  -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>  -D DEFAULT_ERRORLOG="logs/error_log"
>  -D AP_TYPES_CONFIG_FILE="conf/mime.types"
>  -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
>
>
> ##### /opt/waf/bin/apache_prod/bin/httpd -l
> Compiled in modules:
>   core.c
>   mod_authn_file.c
>   mod_authn_default.c
>   mod_authz_host.c
>   mod_authz_groupfile.c
>   mod_authz_user.c
>   mod_authz_default.c
>   mod_auth_basic.c
>   mod_cache.c
>   mod_disk_cache.c
>   mod_mem_cache.c
>   mod_include.c
>   mod_filter.c
>   mod_deflate.c
>   mod_log_config.c
>   mod_logio.c
>   mod_env.c
>   mod_expires.c
>   mod_headers.c
>   mod_unique_id.c
>   mod_setenvif.c
>   mod_proxy.c
>   mod_proxy_connect.c
>   mod_proxy_ftp.c
>   mod_proxy_http.c
>   mod_proxy_ajp.c
>   mod_proxy_balancer.c
>   mod_ssl.c
>   worker.c
>   http_core.c
>   mod_mime.c
>   mod_status.c
>   mod_dir.c
>   mod_actions.c
>   mod_alias.c
>   mod_rewrite.c
>   mod_so.c
>
>
> ##### grep -v "^#" httpd-mpm.conf | grep ..
> <IfModule !mpm_netware_module>
>     PidFile "logs/httpd.pid"
> </IfModule>
> <IfModule !mpm_winnt_module>
> <IfModule !mpm_netware_module>
> LockFile "logs/accept.lock"
> </IfModule>
> </IfModule>
> <IfModule mpm_worker_module>
>     StartServers          5
>     MaxClients          400
>     MinSpareThreads      25
>     MaxSpareThreads      75
>     ThreadsPerChild      25
>     MaxRequestsPerChild  1000
> </IfModule>
>
>
> #### grep KeepAlive httpd-default.conf | grep -v "^#"
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 5
>
>
> #### cat vhosts.d/www.mysite.com.conf
>
> <VirtualHost 192.168.168.100:80 <http://192.168.168.100:80>>
>    ServerName www.mysite.com <http://www.mysite.com>
>    ServerAlias mysite.com <http://mysite.com>
>
>    # Log files
>    # ErrorLog logs/www.mysite.com-error_log
>    # CustomLog logs/www.mysite.com-access_log combined
>
>    # Add ClientIP to the Request Headers
>    RewriteEngine On
>    RewriteCond %{REMOTE_ADDR} (.*)
>    RewriteRule .* - [E=R_A:%1]
>    RequestHeader add ClientIP %{R_A}e
>
>    # Send all pages except the manut one to the internal web server
>    ProxyPreserveHost On
>    ProxyPass         /manut.html !
>    ProxyPass         / http://www.mysite.com/
>    ProxyPassReverse  / http://www.mysite.com/
>
>    # ModSecurity specific rules (no additional rules enabled for the moment)
>    Include conf/rules.d/www.mysite.com.rules
> </VirtualHost>
>
> <VirtualHost 192.168.168.100:443 <http://192.168.168.100:443>>
>    ServerName www.mysite.com <http://www.mysite.com>
>    ServerAlias mysite.com <http://mysite.com>
>
>    # Log files
>    # ErrorLog logs/www.mysite.com-error_log
>    # CustomLog logs/www.mysite.com-access_log combined
>
>    # SSL config
>    SSLEngine on
>    SSLProtocol All -SSLv2
>    SSLCipherSuite ALL:!EXP:!NULL:!ADH:!LOW
>    SSLCertificateFile conf/cert/www.mysite.com.crt
>    SSLCertificateKeyFile conf/cert/www.mysite.com.key
>    SSLCertificateChainFile conf/cert/Verisign04.crt
>
>    # Add ClientIP to the Request Headers
>    RewriteEngine On
>    RewriteCond %{REMOTE_ADDR} (.*)
>    RewriteRule .* - [E=R_A:%1]
>    RequestHeader add ClientIP %{R_A}e
>
>    # Send all pages except the manut one to the internal web server
>    ProxyPreserveHost On
>    ProxyPass         /manut.html !
>    ProxyPass         / http://www.mysite.com/
>    ProxyPassReverse  / http://www.mysite.com/
>
>    # ModSecurity specific rules  (no additional rules enabled for the
> moment)
>    Include conf/rules.d/www.mysite.com.rules
>
> </VirtualHost>
>
>
> In attach the error_log of a test with:
> #### ./ab -k -c 200 -n 2000 https://192.168.168.100/
> Hang after 272 request... (restart of apache needed!)
>
>
> #### top -d 1 (snapshot in the half of test)
> Tasks: 240 total,   1 running, 237 sleeping,   0 stopped,   2 zombie
> Cpu(s):  9.5%us,  0.5%sy,  0.0%ni, 75.4%id, 14.4%wa,  0.0%hi,  0.2%si,
> 0.0%st
> Mem:   5185028k total,  1462924k used,  3722104k free,     2832k buffers
> Swap:  4194296k total,        0k used,  4194296k free,  1130024k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
> COMMAND
>
>  9302 wwwrun    18   0  233m  11m 2332 S    6  0.2   0:00.44
> httpd
>
>  9388 wwwrun    16   0  233m  10m 2232 S    5  0.2   0:00.27
> httpd
>
>  9332 wwwrun    16   0  234m  10m 2312 S    4  0.2   0:00.32
> httpd
>
>  9532 wwwrun    16   0  231m 9144 2240 S    4  0.2   0:00.11
> httpd
>
>  9392 wwwrun    16   0  234m  10m 2232 S    3  0.2   0:00.29
> httpd
>
>  9498 wwwrun    17   0  231m 9856 2296 S    3  0.2   0:00.13
> httpd
>
>  9499 wwwrun    17   0  230m 9100 2264 S    3  0.2   0:00.08
> httpd
>
>  9600 wwwrun    21   0  230m 9140 2272 S    3  0.2   0:00.08
> httpd
>
>  9386 wwwrun    15   0  232m  10m 2284 S    2  0.2   0:00.20
> httpd
>
>  9390 wwwrun    16   0  234m  10m 2220 S    2  0.2   0:00.23
> httpd
>
>  9530 wwwrun    16   0  230m 9056 2264 S    2  0.2   0:00.09
> httpd
>
>  1024 root      10  -5     0    0    0 S    1  0.0   0:02.81
> xfsdatad/0
>
>  9330 wwwrun    16   0  234m  10m 2288 S    1  0.2   0:00.30
> httpd
>
>  9505 wwwrun    16   0  230m 9124 2224 S    1  0.2   0:00.09
> httpd
>
>     1 root      16   0   732  284  244 S    0  0.0   0:02.00
> init
>
>     2 root      RT   0     0    0    0 S    0  0.0   0:00.74
> migration/0
>
>     3 root      34  19     0    0    0 S    0  0.0   0:00.05
> ksoftirqd/0
>
>
>
>
> On Tue, Jun 24, 2008 at 7:18 PM, Brian Rectanus
> <Brian.Rectanus <at> breach.com <mailto:Brian.Rectanus <at> breach.com>> wrote:
>
>     Nicola,
>
>     I need to be able to duplicate this problem.  Would you please send your
>     settings for Apache and modsecurity?
>
>     For ModSecurity, I need your config settings (usually in
>     modsecurity_crs_10_config.conf) and which other files you are including.
>
>     For Apache I at least need these:
>
>     1. Output from "httpd -V" and "httpd -l"
>
>     2. Values for the following directives:
>
>     ServerLimit
>     StartServers
>     MaxClients
>     MinSpareThreads
>     MaxSpareThreads
>     ThreadsPerChild
>     MaxRequestsPerChild
>     MaxRequestsPerThread
>     KeepAlive
>     KeepAliveTimeout
>
>     3. As well as your config for proxying (Balancer, ProxyPass, etc)?
>
>     4. Additionally, your entire error_log at at least level "info" (cleared
>     before the test), the server-status output during (or near) the hang and
>     CPU/Mem usage stats during the test would be nice as well.
>
>     thanks,
>     -B
>
>
>     Ivan Ristic wrote:
>     > Hi Nicola,
>     >
>     > We'll have to try to reproduce your problem somehow, as it doesn't
>     > happen in my tests. I've been using ab constantly over the years for
>     > testing, and I don't recall any problems either.
>     >
>     > Are you using mlogc or any other mechanism to transmit alerts
>     elsewhere?
>     >
>     >
>     > On Mon, Jun 23, 2008 at 2:51 PM, Nicola Bianchi
>     > <bianchi.nicola <at> gmail.com <mailto:bianchi.nicola <at> gmail.com>> wrote:
>     >> Hi people,
>     >> I'm a new modsecurity user and I've a problem which maybe some of
>     you can
>     >> resolve ;).
>     >>
>     >> My configuration is: reverse proxy (http/https) with apache 2.2.9 and
>     >> modsecurity 2.5.5 (core rules 2.5-1.6.1) on Linux SUSE SLES10.
>     >> Hardware: 2CPU dual core Intel(R) Xeon(R) <at> 2.33GHz, 4GB of RAM
>     >>
>     >> If I try this benchmark all work fine, without problem:
>     >>  ab -k -c 200 -n 8000 http://www.mysite.com/
>     >>  ab -k -c 200 -n 8000 https://www.mysite.com/
>     >>
>     >> ... no lost requests, no particular delay.
>     >>
>     >> The problem come out if I try to do a "DOS attack" pointing directly
>     > to the
>     >> ip address of mysite in https
>     >> After few request (~200) apache hang and stop responding ...
>     >>
>     >>  ab -k -c 200 -n 8000 https://192.168.168.100/).
>     >>
>     >
>     #############################################################################
>     >> # This is ApacheBench, Version 2.3 <$Revision: 655654 $>
>     >> # Copyright 1996 Adam Twiss, Zeus Technology Ltd,
>     http://www.zeustech.net/
>     >> # Licensed to The Apache Software Foundation, http://www.apache.org/
>     >> #
>     >> # Benchmarking 192.168.168.100 <http://192.168.168.100> (be patient)
>     >> # Completed 200 requests
>     >> # apr_poll: The timeout specified has expired (70007)
>     >> # Total of 272 requests completed
>     >>
>     >
>     #############################################################################
>     >>
>     >> Here an extract from the logs:
>     >>
>     >
>     #############################################################################
>     >> Jun 23 14:31:47 ulxbwaf httpd[8103]: [error] [client
>     192.168.168.168 <http://192.168.168.168>]
>     >> ModSecurity: Access denied with code 400 (phase 2). Pattern match
>     >> "^[\\d\\.]+$" at REQUEST_HEADERS:Host. [file
>     >>
>     >
>     "/opt/jail/opt/waf/mod_security/prod/conf/core_rules/modsecurity_crs_21_protocol_anomalies.conf"]
>     >> [line "60"] [id "960017"] [msg "Host header is a numeric IP address"]
>     >> [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/IP_HOST"] [hostname
>     >> "192.168.168.100 <http://192.168.168.100>"] [uri "/"] [unique_id
>     "SF <at> XssIL0NIAAB <at> ncMAAAACI"]
>     >>
>     >
>     #############################################################################
>     >>
>     >> If I turn off modsecurity (SecRuleEngine Off) and I repeat the test I
>     > don't
>     >> have problem!
>     >> If I disable the specific rule (SecRuleRemoveById "960017") all
>     work fine!
>     >>
>     >> So, have you some idea about this issue?
>     >> How can I prevent this kind of "DOS attack"?
>     >>
>     >> Thanks a lot! Regards
>     >>  Nick
>     >>
>     >> PS: sorry for my ridicolous english ;)
>     >>
>     >>
>     -------------------------------------------------------------------------
>     >> Check out the new SourceForge.net Marketplace.
>     >> It's the best place to buy or sell services for
>     >> just about anything Open Source.
>     >> http://sourceforge.net/services/buy/index.php
>     >> _______________________________________________
>     >> mod-security-users mailing list
>     >> mod-security-users <at> lists.sourceforge.net
>     <mailto:mod-security-users <at> lists.sourceforge.net>
>     >> https://lists.sourceforge.net/lists/listinfo/mod-security-users
>     >>
>     >>
>     >
>     >
>     >
>     > --
>     > Ivan Ristic
>     >
>     >
>     -------------------------------------------------------------------------
>     > Check out the new SourceForge.net Marketplace.
>     > It's the best place to buy or sell services for
>     > just about anything Open Source.
>     > http://sourceforge.net/services/buy/index.php
>     > _______________________________________________
>     > mod-security-users mailing list
>     > mod-security-users <at> lists.sourceforge.net
>     <mailto:mod-security-users <at> lists.sourceforge.net>
>     > https://lists.sourceforge.net/lists/listinfo/mod-security-users
>     >
>
>
>     --
>     Brian Rectanus
>     Breach Security
>
>


--
Brian Rectanus
Breach Security

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
mod-security-users mailing list
mod-security-users <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mod-security-users

Gmane