Re: Recommendations for copying/processing large byte arrays

On 8/3/11 4:03 PM, Gregor Thalhammer wrote:
> Am 4.8.2011 um 00:26 schrieb Robert Bradshaw:
>> Yeah, that's the tricky part. Just out of curiosity, is there a way to
>> register a callback that gets invoked when the NumPy array is freed?
>
> Travis Oliphant describes how to do this (in plain C):
> http://blog.enthought.com/python/numpy-arrays-with-pre-allocated-memory/
> I would like to see this translated to cython, hope somebody else did/will do this 

yep -- that would be nice -- and Travis proposed that there be a 
pre-written version in the numpy C api too, that might be the place to 
start.

thanks for the link.

I've actually been thinking about a numpy-compatible way to do nice 
nd-arrays in C++, and have been thinking that it would build on two core 
classes:

1) a "data block" class -- this would be a pretty simple wrapper around 
a pointer to a block of memory. Pretty much all it would have is the 
pointer and a reference count, and then ways to increment and decrement 
the ref count, and delete the memory when the count goes to zero. It 
would have no idea how the memory was formatted, or how it was used, 
etc. (could this just be a "smart pointer")

2) an array class that would use a data block for the data itself, and 
it would hold all the info about types, strides, etc. When one gets 
created from scratch, it would create a data block instance, if one was 
created by a view or slicing, etc operation, it would increment the data 
block ref count. (and, of course, decrement when arrays are deleted)

It seems this would allow me to create nifty numpy-like operations in 
C++, and have something easy to pass on to a numpy array object when I 
wanted to interact with Python. Travis' blog post shows me how to 
connect python/numpy ref counting with my "data block" class.

NOTE: this is all still just an idea -- who knows if I'll ever write the 
code....

Oh -- maybe a class with a scheme like this already exists (boost 
arrays?) I haven't looked hard enough yet, though want I've found 
doesn't support numpy's slices-as-views approach well -- and I really 
like that.

-Chris

--

-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker <at> noaa.gov