Re: Massive memory leak

On Tue, Mar 9, 2010 at 3:14 PM, Kapil Thangavelu <kapilt@...> wrote:
> i'd try something like meliae from pdb to dump your reference counts, that
> should give your some insight if your generating objects cycles that are the
> source of the leaks.
> http://jam-bazaar.blogspot.com/2009/11/memory-debugging-with-meliae.html
> https://launchpad.net/meliae
>
> cheers,
> kapil
> On Tue, Mar 9, 2010 at 12:48 PM, Dominic LoBue <dom.lobue@...> wrote:
>>
>> Hello,
>>
>> I am seeing a massive memory leak in xappy, but I'm having trouble
>> pinning it down exactly.
>>
>> Some background: I'm using xappy to index my email and store useful
>> headers for quick access.
>>
>> In the program I'm developing I will open a new search connection,
>> perform a query, copy all the header information into a custom
>> container class, and then close the search connection. I have found
>> that as I keep performing these operations my program continues to use
>> more and more ram, and never releases anything.
>>
>> Here's a really simple example that makes the problem obvious:
>> import pdb
>> import xappy
>> from overwatch import xapidx
>> from databasics import msg_factory
>> sconn = xappy.SearchConnection(xapidx)
>> r = sconn.search(sconn.query_all(), 0, 99999999, checkatleast= -1,
>> sortby= '-sent')
>> r = map(msg_factory, r)
>> del r
>> del sconn
>> pdb.set_trace()
>>
>>
>>
>>
>> msg_factory is just a factory function that returns a named tuple that
>> contains all the header information contained in the ProcessedDocument
>> it gets.
>>
>> Running that script on my machine and running `ps aux` when it starts
>> pdb I see that the script is using 128568k, or ~125megs of ram. Now,
>> correct me if I'm wrong, but since I've deleted all objects, shouldn't
>> the only things using up memory still be the python interpreter, and
>> everything I imported?
>>
>>  I'm using the latest xappy from trunk and xapian 1.0.17.
>>
>> Any idea how to fix this?
>>
>> Dominic
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "xappy-discuss" group.
>> To post to this group, send email to xappy-discuss@...
>> To unsubscribe from this group, send email to
>> xappy-discuss+unsubscribe@...
>> For more options, visit this group at
>> http://groups.google.com/group/xappy-discuss?hl=en.
>>
>

Kapil,

Funny you should mention meliae, because I've been using it and
objgraph in order to try and debug this problem.

What I found is that while `ps aux` reports python is using > 120 megs
of ram, meliae only reports python using 44 megs.

From my testing I was able to rule out everything except xappy/xapian.
I figure either something is not being destructed in xapian by xappy,
or there's a memory leak in xapian or the python swig xapian bindings.
Since the answer is out of my league, I thought I'd start with xappy
first and work my way down.

-- 
Dominic LoBue

--

-- 
You received this message because you are subscribed to the Google Groups "xappy-discuss" group.
To post to this group, send email to xappy-discuss@...
To unsubscribe from this group, send email to xappy-discuss+unsubscribe <at> googlegroups.com.
For more options, visit this group at http://groups.google.com/group/xappy-discuss?hl=en.