26 Apr 13:40 2007
Re: RBL queries: filtering possible?
Ivan Ristic <ivan.ristic <at> gmail.com>
2007-04-26 11:40:48 GMT
2007-04-26 11:40:48 GMT
Hi Michael, Thank you for the detailed explanation. Adding that functionality to ModSecurity shouldn't be much work. I was thinking implementing it along these lines: # Perform lookup. We are not blocking straight away but by specifying # the "capture" action we instruct the <at> rbl operator to extract the response # bits and place them into transaction variables (only when successful). SecRule REMOTE_ADDR " <at> rbl whatever.org" nolog,pass,capture # At this point the TX:0 variable contains the actual response returned. # Variables TX:1, TX:2, ..., TX:9 contain individual bits so you can use them for any purpose. For example: SecRule TX:1 log,deny SecRule TX:2 pass On 4/24/07, Michael Renzmann <mrenzmann <at> otaku42.de> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Ryan. > > > Interesting... I see the resource/time advantage of being able to just > > do one <at> rbl looking to multi.surbl.org and it will then query the other > > list. > > >From what I understand the dnsbl-server does not perform "local > sub-queries to other zones". Instead, the database used for that zone is > fed by a appropriate zonefile (which, in return, most probably will be > built by some script or whatever that combines the current contents of the > zones in question). > > > As for inspecting the bitmask portion, does it really matter > > which list it was on? [...] Is there something specific that you are > > looking to do depending on which list had the client IP address listed? > > Yes, but I have to explain a bit. > > A friend of mine just started a dnsbl that lists known spider IPs. This > dnsbl is fed with data from iplists.com, which in return allows pretty > good destinction between various "big" spiders such as Googlebot or the MS > live.com spider. > > One of the websites I'm responsible for makes use of Trac > (http://trac.edgewall.org). This has proven to be very useful for our > purposes, but it can be pretty demanding when processing some requests. > Unfortunately the webserver isn't very powerful (P3-850), and I'm happy > for any unnecessary expensive request I can prevent. One alternative would > be to upgrade to a more powerful server (but we don't have the money for > that at this time), another would be to implement caching for Trac (which > is actually on my to-do list, but will take some time). > > Analysing the access logs reveals that a good share of the server's load > seems to be caused by spiders hitting places that have been disallowed in > robots.txt. At first I thought that I'm just too dumb to write a proper > robots.txt, but after double-checking the file content and reading through > the online available information on that topic I came to believe that I'm > not at fault. > > Hence I'm currently considering to use a combination of mod-security rules > and queries to the mentioned spider dnsbl to implement some kind of > "robots.txt enforcement". GET-requests from IP-addresses that are listed > in the spider dnsbl to locations that are disallowed in robots.txt should > be answered with an error (403 or 410 or ...). > > You may wonder: where is the need for bitmasking the result? Well, some of > the locations in robots.txt are not disallowed for all spiders. One > example is the feed generator which is allowed to be accessed by Google's > Feedfetcher, but not by all the other "ordinary" spiders. Without > bitmasking this will be pretty hard/expensive to achieve. > > > I'm aware that this is a very special case, and most probably one that > looks pretty borked. > > But think of a case where a user is interested to check an IP against > three zones of a dnsbl service. If that service offers a multi zone that > combines more than these three zones it's still easier to perform one > lookup and mask only the interesting bits in the result, rather than > performing three consecutive lookups to get the same result. > > Bye, Mike > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.1 (GNU/Linux) > > iEYEARECAAYFAkYuBwIACgkQa3V7dXg8JKuKyQCeII4bUd46D/ekgmMKpBGgOJxN > rI8AoI6chV4t8NefGDYRpG5AC/vMKDcm > =WOQo > -----END PGP SIGNATURE----- > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > mod-security-users mailing list > mod-security-users <at> lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/mod-security-users > -- -- Ivan Ristic ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/