Re: problem with my regex and single line HTMLcomment in RESPONSE_BODY

Stephen,
First off, I am sending you a "Virtual High-Five" as you are
demonstrating that you are using many of the various tools and
techniques that we have been discussing for trouble-shooting!  Using
RegEx Coach/Expresso and also leveraging the debug log is outstanding.

To your exact issues, I don't believe that Mod was "choking" on anything
but rather that your RegEx was not working correctly for the single line
comment.  Here are a few comments -

1) While RegEx tools (such as Expresso) are invaluable for helping to
create or trouble-shoot regex issues, beware that you can also run into
issues if the tool is not configured identically to how ModSecurity's
pcre/transformations will be used.  I just ran into this recently where
Expresso had the option checked to "Ignore Whitespace" and it screwed up
some of my tests.  In Expresso 3.0, click on the "Design Mode" tab and
check at the bottom to see how it is handling some of these issues.

2) In order to account for both single or multi-line comments, I had to
update your RegEx to add a few quantifiers to allow for the
presence/absence of CR, LF, etc...  I also made a few changes to catch
other "admin" or "password" variations.  One other item, in order to
handle mixed-case issues you can use "t:lowercase" however keep in mind
that transformation functions against the RESPONSE_BODY are rather
expensive so if you can specify all of the variations in the RegEx you
might want to for performance reasons.  You could also try and use the
regex embedded ignore case flag (?i:).  Here is an updated rule that
works for me -

SecRule RESPONSE_BODY "<!--[
\r\n\t]*?(.*)?(?i:adm(in)?|pwd|passw(or)?d)(.*)?[ \r\n\t]*?-->"
"phase:4,t:none,log,auditlog,deny,status:503,msg:'Authentication
Credentials in HTML Comment Tag.',tag:'LEAKAGE'"

With this rule, it was able to catch all of the examples you showed.

3) In the debug log, it wasn't that Mod was choking on the RESPONSE_BODY
data, but rather that it will only display up to a certain amount of
data within the log file itself.  It will still parse the entire payload
however.  It simply looks as though your example comment data was
further down in the payload and thus was not displayed in the log.

Hope this info helps.

-- 
Ryan C. Barnett
ModSecurity Community Manager
Breach Security: Director of Application Security
Web Application Security Consortium (WASC) Member
CIS Apache Benchmark Project Lead
SANS Instructor, GCIA, GCFA, GCIH, GSNA, GCUX, GSEC
Author: Preventing Web Attacks with Apache

> -----Original Message-----
> From: mod-security-users-bounces <at> lists.sourceforge.net [mailto:mod-
> security-users-bounces <at> lists.sourceforge.net] On Behalf Of Stephen
Craig
> Evans
> Sent: Friday, June 27, 2008 8:01 AM
> To: mod-security-users <at> lists.sourceforge.net
> Subject: [mod-security-users] problem with my regex and single line
> HTMLcomment in RESPONSE_BODY
> 
> Hi,
> 
> I'm having a problem with a regex.
> 
> I want to prevent content in an HTTP response being returned as in:
> <!--
> 	TODO admin:adminpw
>   -->
> 
> My regular expression:
> <!--[ \r\n\t]*(.?)*(Admin|admin|administrator|pwd|password)(.*)[
\r\n\t]*
> -->
> 
> works both in Regex Coach and Expresso 3.0 and on variations such as:
>     <!--
> 	Use Admin to regenerate database
>   -->
> 
> which is what I want.
> 
> To try to make a long story short - without a bunch of debug stuff -
> it works as designed on the 'bad' pages with multiline HTML comments
> like above but when it processes a page with any HTML comment on one
> line (e.g. <!-- Stop Instructions -->) it chokes and stops processing
> the page.
> 
> 
> My rule:
> SecRule RESPONSE_BODY "<!--[
> \r\n\t]*(.?)*(Admin|admin|administrator|pwd|password)(.*)[ \r\n\t]*
> -->"
> "phase:4,ctl:auditLogParts=+E,log,deny,status:501,auditlog,msg:'HTML
> comment source code
> leakage',id:'8',tag:'LEAKAGE/SOURCE_CODE',severity:'4'"
> 
> Adding 'capture' to the rule as in "phase:4,capture,ctl..." gets rid
> of the debug message "Ignoring regex captures since "capture" action
> is not enabled." but the result is the exact same.
> 
> I also use "SecCacheTransformations=Off" so that transformations
> aren't being cached.
> 
> The end of the debug log file shows (with some sanitation):
> 
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][4] Starting phase
> RESPONSE_BODY.
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][9] This phase consists of
> 5 rule(s).
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][4] Recipe: Invoking rule
> 80c3c28; [file "/etc/modsecurity/rulefile_fail-open-auth2.conf"] [line
> "268"] [id "8"].
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][5] Rule 80c3c28: SecRule
> "RESPONSE_BODY" "@rx <!--[
> \\r\\n\\t]*(.?)*(Admin|admin|administrator|pwd|password)(.*)[
> \\r\\n\\t]* -->"
>
"phase:4,capture,ctl:auditLogParts=+E,log,deny,status:501,auditlog,msg:'
HT
> ML
> comment source code leakage',id:8,tag:LEAKAGE/SOURCE_CODE,severity:4"
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][4] Transformation
> completed in 4 usec.
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][4] Executing operator
> "rx" with param "<!--[
> \\r\\n\\t]*(.?)*(Admin|admin|administrator|pwd|password)(.*)[
> \\r\\n\\t]* -->" against RESPONSE_BODY.
> [26/Jun/2008:19:36:29 +0800]
> [192.168.0.5/sid#82100e0][rid#8424118][/xxx][9] Target value:
> "\r\n\r\n\r\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0
> Transitional//EN"
>
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\r\n\r\n<html
> xmlns="http://www.w3.org/1999/xhtml">\r\n<head>\r\n<meta
> http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"
> />\r\n<title>Sample Page</title>\r\n<link rel="stylesheet"
> href="css/draw.css" type="text/css" />\r\n<link rel="stylesheet"
> href="css/page1.css" type="text/css" />\r\n<link rel="stylesheet"
> href="css/menu.css" type="text/css" />\r\n<link rel="stylesheet"
> href="css/layers.css" type="text/css" />\r\n<script
> language="JavaScript1.2" src="javascript/javascript.js"
> type="text/javascript"></script>\r\n<script language="JavaScript1.2"
> src="javascript/menu_system.js"
> type="text/javascript"></script>\r\n<script language="JavaScript1.2"
> src="javascript/pageNav.js"
> type="text/javascript"></script>\r\n<script language="JavaScript1.2"
> src="javascript/makeWindow.js"
> type="text/javascript"></script>\r\n<script language="Java
> 
> And it stops there. This page below the output contains an HTML
> comment: <!-- Stop Instructions -->
> 
> I even added 'Instructions' to the rule to see if a one line HTML
> comment with a match worked, but it doesn't. Same result.
> 
> Can anybody help? I'm baffled.
> 
> I'm using ModSecurity 2.5.1 with Apache 2.2 on Kubuntu 7.10.
> 
> Thanks in advance,
> Stephen
> 
>
------------------------------------------------------------------------
-
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> mod-security-users mailing list
> mod-security-users <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mod-security-users

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php