URLs from Apache log

Hi,

I  think that one of the obvious usage of tool like Grinder would be 
having a script reading data from a production apache (other httpd 
server) logs to create a same same set of request on test box(s). Maybe 
I'm a bit blind but I couldn't find such example in your script gallery.

Anyway it's not so hard to write such thing beside I have some 
additional issues. The file is rather big (apache log 14 GB and cut 
version - only URL - 1.2 GB), so it's not ideal to read such files in 
one move. So, I decided to make it incremental, but I wanted to make 
additional step and results of each chunk of data to be save as a 
separate test. I wrote the following script, which 'nearly works' and 
falls with following error message:

8/4/09 7:08:17 PM (thread 0 run 0 test 1): Aborted run due to Java 
exception calling TestRunner
Java exception calling TestRunner
        File "/home/grinder/projects/b_stress_test.py", line 22, in __call__
Caused by: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
[...]

The tail of out log file indicates that the first test has been finished:

[...]
  Tests resulting in error only contribute to the Errors column.         
  Statistics for individual tests can be found in the data file, including
  (possibly incomplete) statistics for erroneous tests. Composite tests  
  are marked with () and not included in the totals.                     
8/4/09 7:08:18 PM (process dc01gen-01.dqs.lhr1.booking.com-0): finished

What wrong with my script?

Regards,
Wawrzek

from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair

#
class TestRunner:
        def __call__(self):
                myurl='http://test-static.booking.com'
                request= HTTPRequest(url=myurl)
                chunk=0
                f = open('/home/grinder/sourcedata/bo_access.reqs', 'r')
                while 1:
                        imageslist=[w.rstrip('\n') for w in 
f.readlines(2**24)]
                        if not imageslist:
                                break
                        else:
                                chunk = chunk+1
                                print ("chunk=%05d  size=%d" 
%(chunk,len(imageslist)))
                                request= Test(chunk, 'Stress test %03d' 
% chunk).wrap(request)
                                for l in imageslist:
                                        response = request.GET(l)

                f.close()

        def __init__(self):
                connectionDefaults = 
HTTPPluginControl.getConnectionDefaults()
                httpUtilities = HTTPPluginControl.getHTTPUtilities()
                log = grinder.logger.output

                connectionDefaults.defaultHeaders = ( 
NVPair('Accept-Language', 'en-gb,en;q=0.5'), NVPair('Host', 
'static.booking.com'), )

--

-- 
Dr Wawrzyniec Niewodniczański  
Unix System Administrator at Booking.com,  Cambridge Office
tel: +44 02086 128 028 email: wawrzek.niewodniczanski <at> booking.com
PhD in Quantum Chemistry, MSc in Material (Molecular) Science

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
grinder-use mailing list
grinder-use <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/grinder-use