Features Download
From: Wes McKinney <wesmckinn-Re5JQEeQqe8AvxtiuMwx3w <at> public.gmane.org>
Subject: Re: [Cython] Wacky idea: proper macros
Newsgroups: gmane.comp.python.cython.devel
Date: Monday 30th April 2012 16:30:32 UTC (over 6 years ago)
On Mon, Apr 30, 2012 at 11:19 AM, mark florisson
> On 30 April 2012 14:49, Wes McKinney  wrote:
>> On Sun, Apr 29, 2012 at 5:56 AM, mark florisson
>>  wrote:
>>> On 29 April 2012 08:42, Nathaniel Smith  wrote:
>>>> On Sat, Apr 28, 2012 at 10:25 PM, mark florisson
>>>>  wrote:
>>>>> On 28 April 2012 22:04, Nathaniel Smith  wrote:
>>>>>> Was chatting with Wes today about the usual problem many of us have
>>>>>> encountered with needing to use some sort of templating system to
>>>>>> generate code handling multiple types, operations, etc., and a wacky
>>>>>> idea occurred to me. So I thought I'd through it out here.
>>>>>> What if we added a simple macro facility to Cython, that worked at
>>>>>> AST level? (I.e. I'm talking lisp-style macros, *not* C-style
>>>>>> Basically some way to write arbitrary Python code into a .pyx file
>>>>>> that gets executed at compile time and can transform the AST, plus
>>>>>> some nice convenience APIs for simple transformations.
>>>>>> E.g., if we steal the illegal token sequence @@ as our marker, we
>>>>>> could have something like:
>>>>>> @@ # alone on a line, starts a block of Python code
>>>>>> from Cython.MacroUtil import replace_ctype
>>>>>> def expand_types(placeholder, typelist):
>>>>>>  def my_decorator(function_name, ast):
>>>>>>    functions = {}
>>>>>>    for typename in typelist:
>>>>>>      new_name = "%s_%s" % (function_name, typename)
>>>>>>      functions[name] = replace_ctype(ast, placeholder, typename)
>>>>>>    return functions
>>>>>>  return function_decorator
>>>>>> @@ # this token sequence cannot occur in Python, so it's a safe
>>>>>> # Compile-time function decorator
>>>>>> # Results in two cdef functions named sum_double and sum_int
>>>>>> @@expand_types("T", ["double", "int"])
>>>>>> cdef T sum(np.ndarray[T] arr):
>>>>>>  cdef T start = 0;
>>>>>>  for i in range(arr.size):
>>>>>>    start += arr[i]
>>>>>>  return start
>>>>>> I don't know if this is a good idea, but it seems like it'd be very
>>>>>> easy to do on the Cython side, fairly clean, and be dramatically
>>>>>> horrible than all the ad-hoc templating stuff people do now.
>>>>>> Presumably there'd be strict limits on how much backwards
>>>>>> compatibility we'd be willing to guarantee for code that went poking
>>>>>> around in the AST by hand, but a small handful of functions like my
>>>>>> notional "replace_ctype" would go a long way, and wouldn't impose
>>>>>> of a compatibility burden.
>>>>>> -- Nathaniel
>>>>>> _______________________________________________
>>>>>> cython-devel mailing list
>>>>>> [email protected]
>>>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>> Have you looked at http://wiki.cython.org/enhancements/metaprogramming
>>>>> In general I would like better meta-programming support, maybe even
>>>>> allow defining new operators (although I'm not sure any of it is very
>>>>> pythonic), but for templates I think fused types should be used, or
>>>>> improved when they fall short. Maybe a plugin system could also help
>>>>> people.
>>>> I hadn't seen that, no -- thanks for the link.
>>>> I have to say that the examples in that link, though, give me the
>>>> impression of a cool solution looking for a problem. I've never wished
>>>> I could symbolically differentiate Python expressions at compile time,
>>>> or create a mutant Python+SQL hybrid language. Actually I guess I've
>>>> only missed define-syntax once in maybe 10 years of hacking in
>>>> Python-the-language: it's neat how if you do 'plot(x, log(y))' in R it
>>>> will peek at the caller's syntax tree to automagically label the axes
>>>> as "x" and "log(y)", and that can't be done in Python. But that's not
>>>> exactly a convincing argument for a macro system.
>>>> But generating optimized code is Cython's whole selling point, and
>>>> people really are doing klugey tricks with string-based preprocessors
>>>> just to generate multiple copies of loops in Cython and C.
>>>> Also, fused types are great, but: (1) IIUC you can't actually do
>>>> ndarray[fused_type] yet, which speaks to the feature's complexity, and
>>> What? Yes you can do that.
>> I haven't been able to get ndarray[fused_t] to work as we've discussed
>> off-list. In your own words "Unfortunately, the automatic buffer
>> dispatch didn't make it into 0.16, so you need to manually
>> specialize". I'm a bit hamstrung by other users needing to be able to
>> compile pandas using the latest released Cython.
> Well, as I said, it does work, but you need to tell Cython which type
> you meant. If you don't want to do that, you have to use this branch:
> https://github.com/markflorisson88/cython/tree/_fused_dispatch_rebased
> . This never made it in since we had no consensus on whether to allow
> the compiler to bootstrap itself and because of possible immaturity of
> the branch.
> So what doesn't work is automatic dispatch for Python functions (def
> functions and the object version of a cpdef function). They don't
> automatically select the right specialization for buffer arguments.
> Anything else should work, otherwise it's a bug.
> Note also that figuring out which specialization to call dynamically
> (i.e. not from Cython space at compile time, but from Python space at
> runtime) has non-trivial overhead on top of just argument unpacking.
> But you can't say "doesn't work" without giving a concrete example of
> what doesn't work besides automatic dispatch, and how it fails.

Sorry, I meant automatic dispatch re "doesn't work" and want to
reiterate how much I appreciate the work you're doing. To give some
context, my code is riddled with stuff like this:


where the only difference between these functions is the type of the
buffer in the two arrays passed in. I have a template string for these
functions that looks like this:

inner_join_template = """@cython.wraparound(False)
def inner_join_indexer_%(name)s(ndarray[%(c_type)s] left,
                              ndarray[%(c_type)s] right):

I would _love_ to replace this with fused types.

In any case, lately I've been sort of yearning for the kinds of things
you can do with an APL-variant like J. Like here's a groupby in J:

1 1 2 2 2 3 1
3 4 5.5 6 7.5 _2 8.3
   labels >>> (2) to handle Wes's original example on his blog (duplicating a bunch
>>>> of code between a "sum" path and a "product" path), you'd actually
>>>> need something like "fused operators", which aren't even on the
>>>> horizon. So it seems unlikely that fused types will grow to cover all
>>>> these cases in the near future.
>>> Although it doesn't handle contiguity or dimensional differences,
>>> currently the efficient fused operator is a function pointer. Wouldn't
>>> passing in a float64_t (*reducer)(float64_t, float64_t) work in this
>>> case (in the face of multiple types, you can have fused parameters in
>>> the function pointer as well)?
>> I have to think that using function pointers everywhere is going to
>> lose out to "inlined" C. Maybe gcc is smart enough to optimize
>> observed code paths. In other words, you don't want
>> for (i = 0; i < nlabels; i++) {
>>    lab = labels[i];
>>    result[lab] = reducer(sumx[i], data[i]);
>> }
>> when you can have
>> for (i = 0; i < nlabels; i++) {
>>    lab = labels[i];
>>    result[lab] = sumx[i] + data[i];
>> }
>> I guess I should start writing some C code and actually measuring the
>> performance gap as I might completely be off-base here; what you want
>> eventually is to look at the array data graph and "rewrite" it to
>> better leverage data parallelism (parallelize pieces that you can) and
>> cache efficiency.
>> The bigger problem with all this is that I want to avoid an
>> accumulation of ad hoc solutions.
> It's probably a good idea to check the overhead first, but I agree it
> will likely be non-trivial. In that sense, I would like something
> similar to julia like
> def  func(runtime_arguments, etc, $compile_time_expr):
>    ...
>         use $compile_time_expr(sumx[i], data[i]) # or maybe
> cython.ceval(compile_time_expr, {'op1': ..., 'op2': ...})
> Maybe such functions should only be restricted to Cython space,
> though. Fused types weren't designed to handle this in any way, they
> are only there to support different types, not operators. If you would
> have objects you could give them all different methods, so this is
> really rather somewhat of a special case.
>>> I agree with Dag that Julia has nice metaprogramming support, maybe
>>> functions could take arbitrary compile time expressions as extra
>>> arguments.
>>>> Of course some experimentation would be needed to find the right
>>>> syntax and convenience functions for this feature too, so maybe I'm
>>>> just being over-optimistic and it would also turn out to be very
>>>> complicated :-). But it seems like some simple AST search/replace
>>>> functions would get you a long way.
>>>> - N
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> [email protected]
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>> _______________________________________________
>>> cython-devel mailing list
>>> [email protected]
>>> http://mail.python.org/mailman/listinfo/cython-devel
>> _______________________________________________
>> cython-devel mailing list
>> [email protected]
>> http://mail.python.org/mailman/listinfo/cython-devel
> _______________________________________________
> cython-devel mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/cython-devel
cython-devel mailing list
[email protected]
CD: 3ms