22 Feb 2012 18:44
Re: Large C Arrays to Python lists
Eric Frederich <eric.frederich <at> gmail.com>
2012-02-22 17:44:08 GMT
2012-02-22 17:44:08 GMT
I have 0.15.1 and it doesn't seem to have these memoryviews.
What is the deal with them? Are they coming in a future release?
On Wed, Feb 22, 2012 at 11:54 AM, mark florisson <markflorisson88 <at> gmail.com> wrote:
You could also go through the buffer interface using memoryviews:On 22 February 2012 16:47, Eric Frederich <eric.frederich <at> gmail.com> wrote:
> Great reply... all options seem feasible.
> My library that I'm creating bindings for allocates the array and expects me
> to free it.
> So what I have been doing in Cython is calling the C function, using list
> comprehension to convert it into a Python list and then free'ing the C
> With NumPy I would like to use either your option 1 or 3 and never
> explicitly free the returned array but let garbage collection do its thing.
> I'll have to experiment, but I may have to go with your 2nd option since
> this 3rd party library tells me to use their own MEM_free function.
> At least with NumPy instead of list comprehension I will 1) only be doing 0
> or 1 extra memory allocation, 2) not creating Python objects for every index
> of the array.
> I was already in the process of creating last example on github (almost
> I got stuck because I missed the line...
> np.PyArray_UpdateFlags(ndarray, ndarray.flags.num | np.NPY_OWNDATA)
> Not having that line had weird results.
> I could run from the command line...
> python -c "from ericbindings import *; print gimmie_an_array(5)"
> ... and it seemed to work and printed my array.
> But then when I went to use it interactively I got a segfault.
> Adding that UpdateFlags call helped.
> I think I'm all set now.
> Thanks all,
> On Wed, Feb 22, 2012 at 11:06 AM, Henry Gomersall <heng <at> cantab.net> wrote:
>> On Wed, 2012-02-22 at 10:47 -0500, Eric Frederich wrote:
>> > Do I need to install cython bindings for NumPy?
>> > How else would numpy be able to get a C pointer to my array?
>> > If it is simple could you provide an example?
>> > Most of the arrays returned from the library are unsigned int arrays.
>> How is your array being filled? Do you invoke the data getter (written
>> in C) from within Numpy?
>> The easiest way to do this is to call your C code with an array that is
>> already being managed by Numpy. So you create the Numpy array, then you
>> call your C code, passing it a pointer to the Numpy array, which it then
>> Another option is to do a memcopy from the C array into a preallocated
>> Numpy array within Cython. This obviously has the overhead of a copy
>> (albeit a very fast, low level copy).
>> Finally, if the memory is necessarily being allocated within the C code
>> and you don't want to do a copy, then you're outside my experience, but
>> I understand it's possible. https://gist.github.com/1253698 seems to be
>> a neat example (though I defer to someone more knowledgeable on this
>> If you want an example for either of the first two options, I can do