Eric Frederich | 22 Feb 2012 18:44
Picon

Re: Large C Arrays to Python lists

I have 0.15.1 and it doesn't seem to have these memoryviews.
What is the deal with them?  Are they coming in a future release?

On Wed, Feb 22, 2012 at 11:54 AM, mark florisson <markflorisson88 <at> gmail.com> wrote:
On 22 February 2012 16:47, Eric Frederich <eric.frederich <at> gmail.com> wrote:
> Henry,
>
> Great reply... all options seem feasible.
> My library that I'm creating bindings for allocates the array and expects me
> to free it.
> So what I have been doing in Cython is calling the C function, using list
> comprehension to convert it into a Python list and then free'ing the C
> array.
> With NumPy I would like to use either your option 1 or 3 and never
> explicitly free the returned array but let garbage collection do its thing.
>
> I'll have to experiment, but I may have to go with your 2nd option since
> this 3rd party library tells me to use their own MEM_free function.
> At least with NumPy instead of list comprehension I will 1) only be doing 0
> or 1 extra memory allocation, 2) not creating Python objects for every index
> of the array.
>
> I was already in the process of creating last example on github (almost
> verbatim).
> I got stuck because I missed the line...
>     np.PyArray_UpdateFlags(ndarray, ndarray.flags.num | np.NPY_OWNDATA)
>
> Not having that line had weird results.
> I could run from the command line...
>     python -c "from ericbindings import *; print gimmie_an_array(5)"
> ... and it seemed to work and printed my array.
> But then when I went to use it interactively I got a segfault.
>
> Adding that UpdateFlags call helped.
>
> I think I'm all set now.
>
> Thanks all,
> ~Eric
>

You could also go through the buffer interface using memoryviews:
https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/userguide/memoryviews.html#cython-array

> On Wed, Feb 22, 2012 at 11:06 AM, Henry Gomersall <heng <at> cantab.net> wrote:
>>
>> On Wed, 2012-02-22 at 10:47 -0500, Eric Frederich wrote:
>> > Do I need to install cython bindings for NumPy?
>> > How else would numpy be able to get a C pointer to my array?
>> > If it is simple could you provide an example?
>> > Most of the arrays returned from the library are unsigned int arrays.
>>
>> How is your array being filled? Do you invoke the data getter (written
>> in C) from within Numpy?
>>
>> The easiest way to do this is to call your C code with an array that is
>> already being managed by Numpy. So you create the Numpy array, then you
>> call your C code, passing it a pointer to the Numpy array, which it then
>> fills.
>>
>> Another option is to do a memcopy from the C array into a preallocated
>> Numpy array within Cython. This obviously has the overhead of a copy
>> (albeit a very fast, low level copy).
>>
>> Finally, if the memory is necessarily being allocated within the C code
>> and you don't want to do a copy, then you're outside my experience, but
>> I understand it's possible. https://gist.github.com/1253698 seems to be
>> a neat example (though I defer to someone more knowledgeable on this
>> one).
>>
>> If you want an example for either of the first two options, I can do
>> that.
>>
>> cheers,
>>
>> hen
>>
>


Gmane