NumPy/SciPy Application Note
Step 1 - Overview
This guide is intended to help current NumPy/SciPy users to take advantage of Intel® Math Kernel Library (Intel® MKL), Intel® Fortran and Intel® C++ Compilers on Microsoft Windows platform.
This application note was created to help NumPy/SciPy users to make use of the latest versions of Intel MKL, Intel Fortran and Intel C++ compilers on Windows platforms.
The instructions given in this articles apply to Intel MKL 11.0 and above and Intel Compiler 15.0 and above. The procedures described in this article have been tested for Python 2.7.3 and Python 3.4.2. These have been verified with Intel MKL 11.2, 11.3, Intel Compilers 15.0, 16.0, NumPy 1.8.2, 1.9.1, 1.10.x and SciPy 0.15.1.
Intel C++ compiler support on Windows has been integrated into Numpy Master and Maintenance-1.10.x branches.
Step 2 - Downloading NumPy and SciPy Source Code
The NumPy source code can be downloaded from:
Intel MKL can be obtained from the following options:
Download a FREE evaluation version of the Intel MKL product.
Download the FREE non-commercial* version of the Intel MKL product.
All of these can be obtained at: Intel® Math Kernel Library product web page.
Intel® MKL and Intel® Fortran are also bundled with Intel® Parallel Studio XE Professional, Composer or Cluster Editions.
Visual Studio 2008 or above.
Python 2.7.3 (compatible with Numpy 1.8.2) or Python 3.4.2 (compatible with Numpy 1.9.1 or above).
Note: You will need to rebuild Python from source files. This is due to the downloaded Python binary on Windows might be incompatible with the Visual Studio version you used. Otherwise you will encounter runtime crash when run numpy or script tests.
Cython 0.22. (Starting from Numpy 1.10.x, Cython is required to build Numpy. You can obtain it from http://cython.org/)
Step 3 - Configuration
Extract the NumPy zip file from the downloaded numpy-x.x.x.zip, it will create a directory named numpy-x.x.x.
Extract the SciPy zip file from the downloaded scipy-x.x.x.zip, it will create directory named scipy-x.x.x.
Set Intel(R) Fortran compiler environment on Windows. Go to All Programs -> Intel Parallel Studio XE 20XX -> Command Prompt, select IA32/Intel64 Visual Studio XX mode for IA32 or Intel64 build binaries specifically.
Step 4 - Building and Installing NumPy
Change directory to numpy-x.x.x
Create a site.cfg from the existing one
Edit site.cfg as follows:
Add the following lines to site.cfg in your top level NumPy directory to use Intel® MKL, if you are building on Intel 64 platform, assuming the default path for the Intel MKL installation from the Intel Parallel Studio XE or Intel Composer XE versions. You may adjust the exact location according to your MKL installation path. For example, if you are using Intel Composer XE 2015, Update 4, you will have:
[mkl] library_dirs = C:\Program Files (x86)\Intel\Composer XE 2015.4.221\mkl\lib\intel64;C:\Program Files (x86)\Intel\Composer XE 2015.4.221\compiler\lib\intel64 include_dirs = C:\Program Files (x86)\Intel\Composer XE 2015.4.221\mkl\include mkl_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md lapack_libs = mkl_lapack95_lp64,mkl_blas95_lp64,mkl_intel_lp64,mkl_intel_thread,mkl_core,libiomp5md
If you are building NumPy for 32 bit, you may add as the following:
[mkl] library_dirs = C:\Program Files (x86)\Intel\Composer XE 2015.4.221\mkl\lib\ia32;C:\Program Files (x86)\Intel\Composer XE 2015.4.221\compiler\lib\ia32 include_dirs = C:\Program Files (x86)\Intel\Composer XE 2015.4.221\mkl\include mkl_libs = mkl_lapack95,mkl_blas95,mkl_intel_c,mkl_intel_thread,mkl_core,libiomp5md lapack_libs = mkl_lapack95,mkl_blas95,mkl_intel_c,mkl_intel_thread,mkl_core,libiomp5md
For old version numpy, modify Intel Fortran compiler configuration in numpy-x.x.x/numpy/distutil/fcompiler/intel.py to use the following compiler options in class IntelVisualFCompiler(BaseIntelFCompiler):
def get_flags(self): opt = ['/nologo', '/MD', '/nbs', '/names:lowercase', '/assume:underscore']
Above change is unnecessary for Numpy 1.10.x or above.
For old version Numpy without Intel C++ Compiler for Windows* integration, compile and install NumPy using Intel Fortran and Microsoft Visual C++ compiler. For Intel64 and IA32 platforms run:
> python setup.py config --compiler=msvc build_clib --compiler=msvc build_ext --compiler=msvc install
For Numpy 1.10.x and above versions, firstly modify self.compile_options in numpy/distutils/intelccompiler.py according to your needs. For example:
self.compile_options = [ '/nologo', '/O3', '/MD', '/W3', '/Qstd=c99', '/QxHost', '/fp:strict', '/Qopenmp']
Here we use, /O3, optimizations for speed and enables more aggressive loop transformations such as Fusion, Block-Unroll-and-Jam, and collapsing IF statements, /Qopenmp for OpenMP threading and /Qxhost option tells the compiler to generate instructions for the highest instruction set available on the compilation host processor. If you are using the ILP64 interface, please add -DMKL_ILP64 compiler flag.
Run icl /help for more information on processor-specific options, and refer Intel Compiler documentation for more details on the various compiler flags.
Compile and install NumPy with the Intel Fortran and Intel C++ compilers: For Intel64 platform, run:
> python setup.py config --compiler=intelemw build_clib --compiler=intelemw build_ext --compiler=intelemw install
For IA32 build:
> python setup.py config --compiler=intelw build_clib --compiler=intelw build_ext --compiler=intelw install
Intel Compiler uses different icl.exe to generate code running on IA32 or Intel64 platforms, besides above configuration It depends on which Intel compiler environment you configured in Step 3 for different platforms. You may pass --prefix=<install dir>, in case, if you want to install in a directory of your choice.
Build and Install SciPy
For old version Numpy without Intel C++ Compiler for Windows* integration, compile and install SciPy using Intel Fortran and Microsoft Visual C++ compilers, for Intel64 build:
> python setup.py config --compiler=msvc --fcompiler=intelvem build_clib --compiler=msvc --fcompiler=intelvem build_ext --compiler=msvc --fcompiler=intelvem install
and for the IA32 builds:
python setup.py config --compiler=msvc --fcompiler=intelv build_clib --compiler=msvc --fcompiler=intelv build_ext --compiler=msvc --fcompiler=intelv install
For Numpy versions later than 1.10.x, compile and install SciPy with the Intel Fortran and Intel C++ compilers, for Intel64 build:
> python setup.py config --compiler=intelemw --fcompiler=intelvem build_clib --compiler=intelemw --fcompiler=intelvem build_ext --compiler=intelemw --fcompiler=intelvem install
and for the IA32 builds:
python setup.py config --compiler=intelw --fcompiler=intelv build_clib --compiler=intelw --fcompiler=intelv build_ext --compiler=intelw --fcompiler=intelv install
The same as mentioned above, Intel Compiler uses different icl.exe to generate code running on IA32 or Intel64 platforms. Besides above configuration It depends on which Intel compiler environment you configured in Step 3 for different platforms.
1. Unresolved external symbol errors.
If you received unresolved external symbol errors when build Scipy with Intel Fortran like followings:
zfft.obj : error LNK2019: unresolved external symbol zfftf_ referenced in function zfft zfft.obj : error LNK2019: unresolved external symbol zfftb_ referenced in function zfft zfft.obj : error LNK2019: unresolved external symbol zffti_ referenced in function get_cache_id_zfft
It probably due to using of removed Intel Fortran options: '/nbs','/Qlowercase'. The patch on fcompiler/intel.py in Step 4 will resolve this problem.
2. Runtime crash when run numpy or scipy tests on Windows.
The downloaded Python libraries were built by old Visual Studio versions(2010 or 2008). When build Numpy with newer version Visual Studio, like 2012, you will encounter Python runtime crash for numpy and scipy tests.
This can be resolved by downloading the source code of Python and rebuild it with the same Visual Studio version for Numpy build.
3. You normally will receive a "Too many open files" error when building Scipy for a while:
error: [Errno 24] Too many open files
This is a bug with openning more than 16K files simultaneously. Running the same python build command for the second time is a simple work around.
4. One known runtime issue for ia32 build scripy test: linalg.interpolative, TestInterpolativeDecomposition.test_id.
Since linalg.interpolative is a large module, we are going to open a Scipy issue for it.
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision #20110804