Re: Apache hang on https protocol violation
Subject: Re: Apache hang on https protocol violation
Newsgroups: gmane.comp.apache.mod-security.user
Date: 2008-06-26 05:18:59 GMT
Brian,
have you tryed with httpS request? Without S I don't have hang problems...
My compiling configurations:
################################################################
tar xvfz httpd-${APACHE_VERSIONE}.tar.gz
cd httpd-${APACHE_VERSIONE}/
./configure \
--prefix=/opt/waf/bin/httpd-${APACHE_VERSIONE} \
--with-mpm=worker --enable-so \
--enable-unique-id \
--enable-proxy --enable-proxy-http --enable-proxy-balancer \
--enable-rewrite --enable-headers \
--enable-logio \
--enable-expires \
--enable-ssl \
--enable-deflate --enable-cache --enable-disk-cache --enable-mem-cache \
--disable-autoindex --disable-asis --disable-cgi --disable-cgid \
--disable-negotiation --disable-userdir \
--with-pcre=/opt/waf/bin/pcre-${PCRE_VERSIONE}
################################################################
################################################################
cd modsecurity-apache_${MODSEC_VERSIONE}/apache2/
./configure \
--prefix=/opt/waf/bin/modsecurity-apache_${MODSEC_VERSIONE} \
--with-apxs=/opt/waf/bin/httpd-${APACHE_VERSIONE}/bin/apxs \
--with-apr=/opt/waf/bin/httpd-${APACHE_VERSIONE}/bin \
--with-apu=/opt/waf/bin/httpd-${APACHE_VERSIONE}/bin \
--with-pcre=/opt/waf/bin/pcre-${PCRE_VERSIONE} \
--with-libxml=/opt/waf/bin/libxml2-${XML_VERSIONE} \
--with-lua=/opt/waf/bin/lua-${LUA_VERSIONE} \
--enable-strict-compile
################################################################
Nick,
I was not able to duplicate this. Below I have 2.2.9 apache running as
a reverse proxy with modsecurity 2.5.5 and core rules 1.6.1 and mlogc
running to a console. Each request produced an alert about the IP in
the host header. Additionally, I up'ed the ab test considerably. I
also tried mis-configuring mlogc in various ways, but these yielded
similar results.
There are some differences in our setups. I have most modules as
modules vs compiled in as you have them. I am also running 64bit. But
I do not think these should make that much difference.
If you would send me the exact configure options you used with your
2.2.9 apache I will compile one here and test if you want.
$ httpd -VServer version: Apache/2.2.9 (Unix)Server built: Jun 25 2008 16:25:03
Server's Module Magic Number: 20051115:15Architecture: 64-bit
Server loaded: APR 1.3.0, APR-Util 1.3.0
Compiled using: APR 1.3.0, APR-Util 1.3.0
Server MPM: Worker-D HTTPD_ROOT="/apps/httpd-2.2.9"
threaded: yes (fixed thread count)
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D SUEXEC_BIN="/apps/httpd-2.2.9/bin/suexec"-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"$ httpd -lCompiled in modules:
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"
core.cworker.cmod_so.c
http_core.c
$ ab -k -c 1000 -n 10000 http://127.0.1.1:8100/cgi-bin/dumpThis is ApacheBench, Version 2.3 <$Revision: 655654 $>Benchmarking 127.0.1.1 (be patient)
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: FooBar/1.2.3
Server Hostname: 127.0.1.1
Server Port: 8100
Document Path: /cgi-bin/dump
Document Length: 226 bytes
Concurrency Level: 1000
Time taken for tests: 44.678 seconds
Complete requests: 10000Failed requests: 0Non-2xx responses: 10000
Write errors: 0
Keep-Alive requests: 0
Total transferred: 3980000 bytes
HTML transferred: 2260000 bytes
Requests per second: 223.82 [#/sec] (mean)
Time per request: 4467.792 [ms] (mean)
Time per request: 4.468 [ms] (mean, across all concurrent requests)
Transfer rate: 86.99 [Kbytes/sec] receivedConnect: 0 469 1819.0 0 20999
Connection Times (ms)
min mean[+/-sd] median max
Processing: 3 3814 4000.3 2614 27551
Waiting: 3 3258 3543.1 2191 26116
Total: 3 4283 4748.7 3025 3655850% 3025
Percentage of the requests served within a certain time (ms)
66% 4818
75% 6226
80% 7324
90% 10264
95% 13155
98% 18743
99% 23293
100% 36558 (longest request)> <VirtualHost 192.168.168.100:80 <http://192.168.168.100:80>>
Nicola Bianchi wrote:
> Hi Brian,
> here the information that you require!
> If you need additional info just tell me...
>
> Thank you a lot for the help ;)
> Regards.
> Nick
>
> ##### grep -v "^#" modsecurity_crs_10_config.conf | grep ..
> SecRuleEngine On
> SecRequestBodyAccess On
> SecResponseBodyAccess On
> SecResponseBodyMimeType (null) text/html text/plain text/xml
> SecResponseBodyLimit 524288
> SecServerSignature "Apache/2.2.0 (Fedora)"
> SecComponentSignature "core ruleset/1.6.1"
> SecUploadDir /tmp
> SecUploadKeepFiles Off
> SecAuditEngine RelevantOnly
> SecAuditLogRelevantStatus "^(?:5|4(?!04))"
> SecAuditLogType Serial
> SecAuditLog logs/modsec_audit.log
> SecAuditLogParts "ABIFHKZ"
> SecArgumentSeparator "&"
> SecCookieFormat 0
> SecRequestBodyInMemoryLimit 131072
> SecDebugLog logs/modsec_debug.log
> SecDebugLogLevel 1
> SecDataDir /tmp
> SecTmpDir /tmp
>
>
> ##### grep -v "^#" modsecurity_crs_15_cb_config.conf | grep ..
> SecRuleEngine On
> SecRequestBodyAccess On
> SecResponseBodyAccess On
> SecResponseBodyMimeType (null) text/html text/plain text/xml
> SecDefaultAction
> "phase:2,log,auditlog,deny,status:403,t:lowercase,t:replaceNulls,t:compressWhitespace"
> SecServerSignature "Server X"
> SecUploadDir /opt/jail/tmp
> SecAuditLogType Concurrent
> SecAuditLog "|bin/mlogc /opt/waf/mod_security/prod/bin/mlogc.conf"
> SecAuditLogStorageDir logs/modsec_audit/
> SecDebugLogLevel 0
> SecDataDir /opt/jail/tmp
> SecTmpDir /opt/jail/tmp
>
>
> ##### /opt/waf/bin/apache_prod/bin/httpd -V
> Server version: Apache/2.2.9 (Unix)
> Server built: Jun 18 2008 11:18:47
> Server's Module Magic Number: 20051115:15
> Server loaded: APR 1.3.0, APR-Util 1.3.0
> Compiled using: APR 1.3.0, APR-Util 1.3.0
> Architecture: 32-bit
> Server MPM: Worker
> threaded: yes (fixed thread count)
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/worker"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/opt/waf/bin/httpd-2.2.9"
> -D SUEXEC_BIN="/opt/waf/bin/httpd-2.2.9/bin/suexec"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"
>
>
>
> ##### /opt/waf/bin/apache_prod/bin/httpd -l
> Compiled in modules:
> core.c
> mod_authn_file.c
> mod_authn_default.c
> mod_authz_host.c
> mod_authz_groupfile.c
> mod_authz_user.c
> mod_authz_default.c
> mod_auth_basic.c
> mod_cache.c
> mod_disk_cache.c
> mod_mem_cache.c
> mod_include.c
> mod_filter.c
> mod_deflate.c
> mod_log_config.c
> mod_logio.c
> mod_env.c
> mod_expires.c
> mod_headers.c
> mod_unique_id.c
> mod_setenvif.c
> mod_proxy.c
> mod_proxy_connect.c
> mod_proxy_ftp.c
> mod_proxy_http.c
> mod_proxy_ajp.c
> mod_proxy_balancer.c
> mod_ssl.c
> worker.c
> http_core.c
> mod_mime.c
> mod_status.c
> mod_dir.c
> mod_actions.c
> mod_alias.c
> mod_rewrite.c
> mod_so.c
>
>
> ##### grep -v "^#" httpd-mpm.conf | grep ..
> <IfModule !mpm_netware_module>
> PidFile "logs/httpd.pid"
> </IfModule>
> <IfModule !mpm_winnt_module>
> <IfModule !mpm_netware_module>
> LockFile "logs/accept.lock"
> </IfModule>
> </IfModule>
> <IfModule mpm_worker_module>
> StartServers 5
> MaxClients 400
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadsPerChild 25
> MaxRequestsPerChild 1000
> </IfModule>
>
>
> #### grep KeepAlive httpd-default.conf | grep -v "^#"
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 5
>
>
> #### cat vhosts.d/www.mysite.com.conf
>
> ServerName www.mysite.com <http://www.mysite.com>
> ServerAlias mysite.com <http://mysite.com>>> <VirtualHost 192.168.168.100:443 <http://192.168.168.100:443>>
> # Log files
> # ErrorLog logs/www.mysite.com-error_log
> # CustomLog logs/www.mysite.com-access_log combined
>
> # Add ClientIP to the Request Headers
> RewriteEngine On
> RewriteCond %{REMOTE_ADDR} (.*)
> RewriteRule .* - [E=R_A:%1]
> RequestHeader add ClientIP %{R_A}e
>
> # Send all pages except the manut one to the internal web server
> ProxyPreserveHost On
> ProxyPass /manut.html !
> ProxyPass / http://www.mysite.com/
> ProxyPassReverse / http://www.mysite.com/
>
> # ModSecurity specific rules (no additional rules enabled for the moment)
> Include conf/rules.d/www.mysite.com.rules
> </VirtualHost>
>
> ServerName www.mysite.com <http://www.mysite.com>
> ServerAlias mysite.com <http://mysite.com>>
> # Log files
> # ErrorLog logs/www.mysite.com-error_log
> # CustomLog logs/www.mysite.com-access_log combined
>
> # SSL config
> SSLEngine on
> SSLProtocol All -SSLv2
> SSLCipherSuite ALL:!EXP:!NULL:!ADH:!LOW
> SSLCertificateFile conf/cert/www.mysite.com.crt
> SSLCertificateKeyFile conf/cert/www.mysite.com.key
> SSLCertificateChainFile conf/cert/Verisign04.crt
>
> # Add ClientIP to the Request Headers
> RewriteEngine On
> RewriteCond %{REMOTE_ADDR} (.*)
> RewriteRule .* - [E=R_A:%1]
> RequestHeader add ClientIP %{R_A}e
>
> # Send all pages except the manut one to the internal web server
> ProxyPreserveHost On
> ProxyPass /manut.html !
> ProxyPass / http://www.mysite.com/
> ProxyPassReverse / http://www.mysite.com/
>
> # ModSecurity specific rules (no additional rules enabled for the
> moment)
> Include conf/rules.d/www.mysite.com.rules
>
> </VirtualHost>
>
>
> In attach the error_log of a test with:
> #### ./ab -k -c 200 -n 2000 https://192.168.168.100/
> Hang after 272 request... (restart of apache needed!)
>
>
> #### top -d 1 (snapshot in the half of test)
> Tasks: 240 total, 1 running, 237 sleeping, 0 stopped, 2 zombie
> Cpu(s): 9.5%us, 0.5%sy, 0.0%ni, 75.4%id, 14.4%wa, 0.0%hi, 0.2%si,
> 0.0%st
> Mem: 5185028k total, 1462924k used, 3722104k free, 2832k buffers
> Swap: 4194296k total, 0k used, 4194296k free, 1130024k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
>
> 9302 wwwrun 18 0 233m 11m 2332 S 6 0.2 0:00.44
> httpd
>
> 9388 wwwrun 16 0 233m 10m 2232 S 5 0.2 0:00.27
> httpd
>
> 9332 wwwrun 16 0 234m 10m 2312 S 4 0.2 0:00.32
> httpd
>
> 9532 wwwrun 16 0 231m 9144 2240 S 4 0.2 0:00.11
> httpd
>
> 9392 wwwrun 16 0 234m 10m 2232 S 3 0.2 0:00.29
> httpd
>
> 9498 wwwrun 17 0 231m 9856 2296 S 3 0.2 0:00.13
> httpd
>
> 9499 wwwrun 17 0 230m 9100 2264 S 3 0.2 0:00.08
> httpd
>
> 9600 wwwrun 21 0 230m 9140 2272 S 3 0.2 0:00.08
> httpd
>
> 9386 wwwrun 15 0 232m 10m 2284 S 2 0.2 0:00.20
> httpd
>
> 9390 wwwrun 16 0 234m 10m 2220 S 2 0.2 0:00.23
> httpd
>
> 9530 wwwrun 16 0 230m 9056 2264 S 2 0.2 0:00.09
> httpd
>
> 1024 root 10 -5 0 0 0 S 1 0.0 0:02.81
> xfsdatad/0
>
> 9330 wwwrun 16 0 234m 10m 2288 S 1 0.2 0:00.30
> httpd
>
> 9505 wwwrun 16 0 230m 9124 2224 S 1 0.2 0:00.09
> httpd
>
> 1 root 16 0 732 284 244 S 0 0.0 0:02.00
> init
>
> 2 root RT 0 0 0 0 S 0 0.0 0:00.74
> migration/0
>
> 3 root 34 19 0 0 0 S 0 0.0 0:00.05
> ksoftirqd/0
>
>
>
>
> On Tue, Jun 24, 2008 at 7:18 PM, Brian Rectanus
> <Brian.Rectanus <at> breach.com <mailto:Brian.Rectanus <at> breach.com>> wrote:
>
> Nicola,
>
> I need to be able to duplicate this problem. Would you please send your
> settings for Apache and modsecurity?
>
> For ModSecurity, I need your config settings (usually in
> modsecurity_crs_10_config.conf) and which other files you are including.
>
> For Apache I at least need these:
>
> 1. Output from "httpd -V" and "httpd -l"
>
> 2. Values for the following directives:
>
> ServerLimit
> StartServers
> MaxClients
> MinSpareThreads
> MaxSpareThreads
> ThreadsPerChild
> MaxRequestsPerChild
> MaxRequestsPerThread
> KeepAlive
> KeepAliveTimeout
>
> 3. As well as your config for proxying (Balancer, ProxyPass, etc)?
>
> 4. Additionally, your entire error_log at at least level "info" (cleared
> before the test), the server-status output during (or near) the hang and
> CPU/Mem usage stats during the test would be nice as well.
>
> thanks,
> -B
>
>
> Ivan Ristic wrote:
> > Hi Nicola,
> >
> > We'll have to try to reproduce your problem somehow, as it doesn't
> > happen in my tests. I've been using ab constantly over the years for
> > testing, and I don't recall any problems either.
> >
> > Are you using mlogc or any other mechanism to transmit alerts
> elsewhere?
> >
> >
> > On Mon, Jun 23, 2008 at 2:51 PM, Nicola Bianchi
> > <bianchi.nicola <at> gmail.com <mailto:bianchi.nicola <at> gmail.com>> wrote:> >> # Benchmarking 192.168.168.100 <http://192.168.168.100> (be patient)
> >> Hi people,
> >> I'm a new modsecurity user and I've a problem which maybe some of
> you can
> >> resolve ;).
> >>
> >> My configuration is: reverse proxy (http/https) with apache 2.2.9 and
> >> modsecurity 2.5.5 (core rules 2.5-1.6.1) on Linux SUSE SLES10.
> >> Hardware: 2CPU dual core Intel(R) Xeon(R) <at> 2.33GHz, 4GB of RAM
> >>
> >> If I try this benchmark all work fine, without problem:
> >> ab -k -c 200 -n 8000 http://www.mysite.com/
> >> ab -k -c 200 -n 8000 https://www.mysite.com/
> >>
> >> ... no lost requests, no particular delay.
> >>
> >> The problem come out if I try to do a "DOS attack" pointing directly
> > to the
> >> ip address of mysite in https
> >> After few request (~200) apache hang and stop responding ...
> >>
> >> ab -k -c 200 -n 8000 https://192.168.168.100/).
> >>
> >
> #############################################################################
> >> # This is ApacheBench, Version 2.3 <$Revision: 655654 $>
> >> # Copyright 1996 Adam Twiss, Zeus Technology Ltd,
> http://www.zeustech.net/
> >> # Licensed to The Apache Software Foundation, http://www.apache.org/
> >> #
> >> # Completed 200 requests> 192.168.168.168 <http://192.168.168.168>]
> >> # apr_poll: The timeout specified has expired (70007)
> >> # Total of 272 requests completed
> >>
> >
> #############################################################################
> >>
> >> Here an extract from the logs:
> >>
> >
> #############################################################################
> >> Jun 23 14:31:47 ulxbwaf httpd[8103]: [error] [client
> >> ModSecurity: Access denied with code 400 (phase 2). Pattern match> >> "192.168.168.100 <http://192.168.168.100>"] [uri "/"] [unique_id
> >> "^[\\d\\.]+$" at REQUEST_HEADERS:Host. [file
> >>
> >
> "/opt/jail/opt/waf/mod_security/prod/conf/core_rules/modsecurity_crs_21_protocol_anomalies.conf"]
> >> [line "60"] [id "960017"] [msg "Host header is a numeric IP address"]
> >> [severity "CRITICAL"] [tag "PROTOCOL_VIOLATION/IP_HOST"] [hostname
> "SF <at> XssIL0NIAAB <at> ncMAAAACI"]> <mailto:mod-security-users <at> lists.sourceforge.net>
> >>
> >
> #############################################################################
> >>
> >> If I turn off modsecurity (SecRuleEngine Off) and I repeat the test I
> > don't
> >> have problem!
> >> If I disable the specific rule (SecRuleRemoveById "960017") all
> work fine!
> >>
> >> So, have you some idea about this issue?
> >> How can I prevent this kind of "DOS attack"?
> >>
> >> Thanks a lot! Regards
> >> Nick
> >>
> >> PS: sorry for my ridicolous english ;)
> >>
> >>
> -------------------------------------------------------------------------
> >> Check out the new SourceForge.net Marketplace.
> >> It's the best place to buy or sell services for
> >> just about anything Open Source.
> >> http://sourceforge.net/services/buy/index.php
> >> _______________________________________________
> >> mod-security-users mailing list
> >> mod-security-users <at> lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/mod-security-users> <mailto:mod-security-users <at> lists.sourceforge.net>
> >>
> >>
> >
> >
> >
> > --
> > Ivan Ristic
> >
> >
> -------------------------------------------------------------------------
> > Check out the new SourceForge.net Marketplace.
> > It's the best place to buy or sell services for
> > just about anything Open Source.
> > http://sourceforge.net/services/buy/index.php
> > _______________________________________________
> > mod-security-users mailing list
> > mod-security-users <at> lists.sourceforge.net
--> > https://lists.sourceforge.net/lists/listinfo/mod-security-users
> >
>
>
> --
> Brian Rectanus
> Breach Security
>
>
Brian Rectanus
Breach Security
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php
_______________________________________________ mod-security-users mailing list mod-security-users <at> lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mod-security-users
RSS Feed