Dag Sverre Seljebotn | 8 Mar 08:29 2011
Picon
Picon

Re: correct way to compile cython project with icc?

On 03/08/2011 08:25 AM, Dag Sverre Seljebotn wrote:
> On 03/06/2011 05:20 AM, Zak Stone wrote:
>> Hello everyone,
>>
>> I would like to compile a Cython project with Intel's compiler icc to
>> benchmark the resulting binary against one produced by gcc. It appears
>> that calling "python setup.py build_ext" with CC="icc" is enough to
>> switch the Cython compilation step to icc, but it is difficult to
>> prevent distutils from passing the wrong flags to icc, and distutils
>> insists on performing the linking step with gcc.
>>
>> It is possible to follow NumPy's lead and patch distutils to support
>> icc -- that only requires a minor change to ccompiler.py and the
>> addition of a file like intelccompiler.py -- but I'd rather not
>> maintain a patch on my system-wide Python installation, and it appears
>> that virtualenv doesn't make it easy to patch Python itself in an
>> isolated local environment. It might be possible to use a modified
>> local copy of distutils only as NumPy seems to, but then I would also
>> have to modify Cython to interface with the modified copy of
>> distutils, and the resulting toolchain would be rather fragile.
>>
>> Of course, there could be deeper problems still, considering that the
>> system Python installation was compiled with gcc, as were NumPy and
>> SciPy; I don't know whether gcc and icc play well together.
>>
>> Is the "right solution" to use icc to compile a completely separate
>> installation of Python, NumPy, SciPy, and Cython and then to develop
>> my project in that environment? If so, could anyone recommend tools to
>> build and maintain such an environment? (pythonbrew appears to be a
>> step in the right direction.) Or is there a simpler route? For
>> example, the binary produced with CC="icc" in my current test case
>> appears to run properly despite the various gcc confounds, but I don't
>> know whether to trust it.
>
> If you want this kind of flexibility I'd switch to a real build tool, 
> such as 'waf'. Here's one example; it does more than you need since it 
> also builds and links with Fortran code:
>
> https://github.com/dagss/healpix4py/blob/9a730266a15ca8986165cfdc9077a8537ed595e9/wscript 
>
>
https://github.com/dagss/healpix4py/blob/9a730266a15ca8986165cfdc9077a8537ed595e9/healpix/wscript 
>

Just realized that waf may or may not work for your usecase, depending 
on how good the Python tool in waf is.

At least if you're on Linux, if you get a module to compile and load 
using 'icc' I think you're pretty much safe, even if 'gcc' is used to 
link etc. IIRC 'icc' is written to be a "drop-in replacement" for gcc, 
Intel very much understands that gcc is the de facto standard on Linux.

Dag Sverre

>
> The downside is you can't that easily distribute your package in 
> standard ways, but "bento" is here to help with that:
>
> http://cournape.wordpress.com/2011/03/06/adding-a-distutils-compatibility-layer-to-bento/ 
>
>
> (I didn't try this yet, as I don't need to easy_install my packages.) 
> So, it's outside of the beaten path, but in the end it may be better 
> than wasting your time with distutils.
>
> Dag Sverre


Gmane