# Numpy/Scipy with Intel® MKL and Intel® Compilers

NumPy/SciPy Application Note

Please note: The application notes is outdated, but keep here for reference. Instead of build Numpy/Scipy with Intel® MKL manually as below, we strongly recommend developer to use  Intel®  Distribution  for  Python* , which has prebuild Numpy/Scipy based on Intel® Math Kernel Library (Intel® MKL) and more.

Please refer to  Intel®  Distribution  for  Python* mainpage

Installing Intel® Distribution for Python* and Intel® Performance Libraries with Anaconda* by

Step 1 - Overview

This guide is intended to help current NumPy/SciPy users to take advantage of Intel® Math Kernel Library (Intel® MKL). For a prebuilt ready solution, download the Intel® Distribution for Python*.

NumPy automatically maps operations on vectors and matrices to the BLAS and LAPACK functions wherever possible. Since Intel® MKL supports these de-facto interfaces, NumPy can benefit from Intel MKL optimizations through simple modifications to the NumPy scripts.

NumPy is the fundamental package required for scientific computing with Python. It consists of:

• a powerful N-dimensional array object
• sophisticated (broadcasting) functions
• tools for integrating C/C++ and Fortran code
• useful linear algebra, Fourier transform, and random number capabilities.

Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data.

SciPy include modules for statistics, optimization, integration, linear algebra, Fourier transforms, signal and image processing, ODE solvers, and more.  The SciPy library depends on NumPy, which provides convenient and fast N-dimensional array manipulation. The SciPy library is built to work with NumPy arrays, and provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization for python users. Please refer http://www.scipy.org for more details on SciPy.

Version Information

This application note was created to help NumPy/SciPy users to make use of the latest versions of Intel MKL on Linux platforms.

The procedures described in this article have been tested for both Python 2.7 and Python 3.6.  These have been verified with Intel® MKL 2018, Intel® Compilers 18.0 from Intel® Parallel Studio XE 2018, numpy 1.13.3 and scipy 1.0.0rc2.

The NumPy source code can be downloaded from:

Prerequisites

Intel® MKL is bundled with Intel® Parallel Studio XE.  If you are compiling with Intel C/C++ and Fortran Compilers, they are also included as part of any of the three (Composer, Professional and Cluster) Intel Parallel Studio XE editions, .

Step 3 - Configuration

Use the following commands to extract the NumPy tar files from the downloaded NumPy-x.x.x.tar.gz.

`\$gunzip numpy-x.x.x.tar.gz \$tar -xvf numpy-x.x.x.tar`

The above will create a directory named numpy-x.x.x

And to extract SciPy, use the below commands

```\$gunzip scipy-x.x.x.tar.gz \$tar -xvf scipy-x.x.x.tar.gz

```

The scipy-x.x.x directory will be created with extracted files.

You may also get the latest numpy and scipy source from their respective github repositories.

Make sure that C++ and FORTRAN compilers are installed and they are in PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), and MKL libraries.

Step 4 - Building and Installing NumPy

Change directory to numpy-x.x.x
Create a site.cfg from the existing one

Edit site.cfg as follows:

Add the following lines to site.cfg in your top level NumPy directory to use Intel® MKL, if you are building on Intel 64 platform, assuming the default path for the Intel MKL installation from the Intel Parallel Studio XE or Intel Composer XE versions:

```
[mkl]
library_dirs = /opt/intel/compilers_and_libraries_2018/linux/mkl/lib/intel64
include_dirs = /opt/intel/compilers_and_libraries_2018/linux/mkl/include
mkl_libs = mkl_rt
lapack_libs =

```

If you are building NumPy for 32 bit, please add as the following

```[mkl]
library_dirs = /opt/intel/compilers_and_libraries_2018/linux/mkl/lib/ia32
include_dirs = /opt/intel/compilers_and_libraries_2018/linux/mkl/include
mkl_libs = mkl_rt
lapack_libs =
```

Modify self.cc_exe line in numpy/distutils/intelccompiler.py

Modify this line depending on whether you are building 32 bit or 64 bit.  For e.g: If you are building 64 bit, please modify this line part of the IntelEM64TCCompiler Class and compiler_type is 'intelem'

```
mpopt = 'openmp' if v and v < '15' else 'qopenmp'
self.cc_exe = ('icc -O3 -g -fPIC -fp-model strict -fomit-frame-pointer -xhost -{}').format(mpopt)

```

Here we use, -O3, optimizations for speed and enables more aggressive loop transformations such as Fusion, Block-Unroll-and-Jam, and collapsing IF statements, -openmp for OpenMP threading and -xhost option tells the compiler to generate instructions for the highest SIMD instruction set available on the compilation host processor.  If you are using the ILP64 interface, please add -DMKL_ILP64 compiler flag.

Run icc --help for more information on processor-specific options, and refer Intel Compiler documentation for more details on the various compiler flags.

Modify the the Fortran compiler configuration in numpy-x.x.x/numpy/distutil/fcompiler/intel.py to use the following compiler options for the Intel Fortran Compiler:

For ia32 and Intel64

`mpopt = 'openmp' if v and v < '15' else 'qopenmp'`
```return ['-xhost -fp-model strict -fPIC -{}'.format(mpopt)]

```

If you are using the latest source, this is already modified in intel.py.   You may explore using other compiler optimization flags.

If you are using ILP64 interface of Intel MKL, please add -i8 flag above.  If you are using older versions of Numpy/SciPy, please refer the new intel.py for your reference from the latest version of NumPy, which can be replaced to use the above mentioned compiler options.

Compile and install NumPy with the Intel compiler: For Intel64 platforms run:

```
\$python setup.py config --compiler=intelem build_clib --compiler=intelem build_ext --compiler=intelem install

```

and for the ia32 builds:

```
\$python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install

```

The difference is, using "intel" for ia32 and intelem" for the intel64.

You may pass

`--prefix=<install_dir> `

in case, if you want to install in a directory of your choice.   In this case, after your successful numpy build, you have to export PYTHONPATH environment variable pointing to your install folder.

```\$export PYTHONPATH=<install_dir>/lib64/pythonx.x/site-packages

```

Build and Install SciPy

Compile and install SciPy with the Intel Compilers, for 64 bit builds:

```
\$python setup.py config --compiler=intelem --fcompiler=intelem build_clib --compiler=intelem --fcompiler=intelem build_ext --compiler=intelem --fcompiler=intelem install

```

and for the ia32 builds:

```\$python setup.py config --compiler=intel --fcompiler=intel build_clib --compiler=intel --fcompiler=intel build_ext --compiler=intel --fcompiler=intel install

```

Setup Library path for Intel MKL and Intel Compilers

If you build NumPY/SciPy for Intel64 bit platforms:

```\$export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018/linux/mkl/lib/intel64/:/opt/intel/compilers_and_libraries_2018/linux/lib/intel64:\$LD_LIBRARY_PATH

```

If you build NumPY for ia32 bit platforms:

`\$export LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2018/linux/mkl/lib/ia32/:/opt/intel/compilers_and_libraries_2018/linux/lib/ia32:\$LD_LIBRARY_PATH`

It is possible that LD_LIBRARY_PATH causes a problem, if you have installed Intel MKL and Intel Composer XE in other directories than the standard ones. The only solution we ha've found that always works is to build Python, NumPy and SciPy inside an environment where you've set the LD_RUN_PATH variable, e.g: for ia32 platform:

```
\$export LD_RUN_PATH=/opt/intel/compilers_and_libraries_2018/linux/mkl/lib/ia32/:/opt/intel/compilers_and_libraries_2018/linux/lib/ia32:\$LD_LIBRARY_PATH```

Note:We recommend users to use arrays with 'C' ordering style which is row-major, which is default than Fortran Style which is column-major, and this is because NumPy uses CBLAS and also to get better performance.

Appendex A: Example:

Please see below an example Python script for matrix multiplication that you can use Numply installed with Intel MKL which has been provided for illustration purpose.

```import numpy as np
import time
N = 6000
M = 10000

k_list = [64, 80, 96, 104, 112, 120, 128, 144, 160, 176, 192, 200, 208, 224, 240, 256, 384]

def get_gflops(M, N, K):
return M*N*(2.0*K-1.0) / 1000**3

np.show_config()

for K in k_list:
a = np.array(np.random.random((M, N)), dtype=np.double, order='C', copy=False)
b = np.array(np.random.random((N, K)), dtype=np.double, order='C', copy=False)
A = np.matrix(a, dtype=np.double, copy=False)
B = np.matrix(b, dtype=np.double, copy=False)

C = A*B

start = time.time()

C = A*B
C = A*B
C = A*B
C = A*B
C = A*B

end = time.time()

tm = (end-start) / 5.0

print ('{0:4}, {1:9.7}, {2:9.7}'.format(K, tm, get_gflops(M, N, K) / tm))

```

Appendix B: Performance Comparison

Please click Examples.py to download the examples for LU, Cholesky and SVD.

Please note all the charts in this article were generated with the Intel MKL 11.1 update 1 version.

Appendix 1:

Known issues:

When the -O3 or -O2(default) compiler flags (more aggressive compiler optimization flags) used for the ifort, one of the scipy tests may fail and it is a known corner case, to avoid this, as a workaround you can use -O1.

Building with GNU Compiler chain:

Make modifications to MKL section in the site.cfg as mentioned above. To build numpy and scipy with Gnu compilers, in the site.cfg file, you must link with mkl_rt only and any other linking method will not work.

Export the compiler flags as:

```\$export CFLAGS="-fopenmp -m64 -mtune=native -O3 -Wl,--no-as-needed"
\$export CXXFLAGS="-fopenmp -m64 -mtune=native -O3 -Wl,--no-as-needed"
\$export LDFLAGS="-ldl -lm"
\$export FFLAGS="-fopenmp -m64 -mtune=native -O3"

```

Then run the config, build, install commands for both numpy and scipy from their respective source folders.

If you want to use GNU OpenMP instead of Intel OpenMP, you should set MKL_THREADING_LAYER=GNU.

Since both numpy and scipy has the linear algebra functions, users can call either numpy BLAS or scipy BLAS, but not both. Usage of both at the same time is not supported by MKL and may lead to crashes.

When using scipy BLAS, you must set MKL_INTERFACE_LAYER=GNU.

The above environment various MKL_THREADING_LAYER and MKL_INTERFACE_LAYER are supported only from Intel MKL 11.1 update 3 and above.

Related Link and Trouble Shooting

Building Numpy/Scipy with Intel® MKL and Intel® Compilers on Windows

Categories:
Tags:
For more complete information about compiler optimizations, see our Optimization Notice.

### Dear all,

Dear all,

I have macOS 10.13 high sierra, mkl lib 2017.1, Intel-compilers also 2017.1 and specified inside site.cfg follows:

[mkl]
library_dirs = /opt/intel/compilers_and_libraries_2017.1.126/mac/mkl/lib
include_dirs = /opt/intel/compilers_and_libraries_2017.1.126/mac/mkl/include
lapack_libs =

Next, when numpy installation has finished and I ran tests I got follows:

```python3 runtests.py
Building, see build.log...
Build OK
Traceback (most recent call last):
File "/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
from . import multiarray
ImportError: dlopen(/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so, 2): Symbol not found: _main
Referenced from: /Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so
Expected in: flat namespace
in /Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "runtests.py", line 492, in <module>
main(argv=sys.argv[1:])
File "runtests.py", line 297, in main
__import__(PROJECT_MODULE)
File "/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
from . import add_newdocs
File "/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/__init__.py", line 26, in <module>
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: dlopen(/Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so, 2): Symbol not found: _main
Referenced from: /Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so
Expected in: flat namespace
in /Users/artem.samtsevich/dev/numpy/build/testenv/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-darwin.so```

Actually, I have seen a message about this error a little bit below. But no one has an idea how to solve it.

Hope for your help

### I was able to build numpy

I was able to build numpy with 2015 compiler but not scipy. Just an fyi , in case anyone tries building with older Intel compiler.

### Thanks for the great article.

Thanks for the great article.

I built the mkl on OSX 10.12 with python3, following this guide everything seemed just fine until I imported numpy and I got

In [1]: import numpy
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/core/__init__.py in <module>()
15 try:
---> 16     from . import multiarray
17 except ImportError:

ImportError: dlopen(/usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/core/multiarray.cpython-36m-darwin.so, 2): Symbol not found: _main
Referenced from: /usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/core/multiarray.cpython-36m-darwin.so
Expected in: flat namespace
in /usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/core/multiarray.cpython-36m-darwin.so

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-5a0bd626bb1d> in <module>()
----> 1 import numpy

/usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/__init__.py in <module>()
140         return loader(*packages, **options)
141
--> 142     from . import add_newdocs
143     __all__ = ['add_newdocs',
144                'ModuleDeprecationWarning',

11 from __future__ import division, absolute_import, print_function
12
---> 13 from numpy.lib import add_newdoc
14
15 ###############################################################################

/usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/lib/__init__.py in <module>()
6 from numpy.version import version as __version__
7
----> 8 from .type_check import *
9 from .index_tricks import *
10 from .function_base import *

/usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/lib/type_check.py in <module>()
9            'common_type']
10
---> 11 import numpy.core.numeric as _nx
12 from numpy.core.numeric import asarray, asanyarray, array, isnan, \
13                 obj2sctype, zeros

/usr/local/lib/python3.6/site-packages/numpy-1.12.1-py3.6-macosx-10.12-x86_64.egg/numpy/core/__init__.py in <module>()
22 files not under version control).  Otherwise reinstall numpy.
23 """
---> 24     raise ImportError(msg)
25
26 for envkey in env_added:

ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Can someone help me on that? Thanks in advance.

### This does not work for me.

This does not work for me. Here is my site.cfg. I am having composer xe 2013.

When I run numpy.test() I get undefined symbol __intel_sse2_strcpy

[DEFAULT]
library_dirs = /usr/local/lib:/usr/lib
include_dirs = /usr/local/include:/usr/include

[mkl]
library_dirs = /opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.3.174/compiler/lib/intel64
include_dirs = /opt/intel/composer_xe_2013.3.174/compiler/include/intel64
mkl_libs = mkl_def, mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5
lapack_libs = mkl_lapack95_lp64

[umfpack]
library_dirs = /usr/lib
include_dirs = /usr/include/suitesparse
umfpack_libs = umfpack

[amd]
library_dirs = /usr/lib
include_dirs = /usr/include/suitesparse
amd_libs = amd

### There's some pretty hefty

There's some pretty hefty complaining about the -openmp option with recent Intel compilers (16.0.3 at least) -- might consider changing the instructions:

"ifort: command line remark #10411: option '-openmp' is deprecated and will be removed in a future release. Please use the replacement option '-qopenmp'"

I think I may have figured it out the -xSSE4.2, BTW: it's in the numpy/distutils/fcompiler/intel.py file. The instructions above say "If you are using the latest source, this is already modified in intel.py." But that's only partially true.

### Ryan, scipy should inherit

Ryan, scipy should inherit whatever settings numpy was compiled with so you could look there. My guess is that numpy (which uses -xhost in master) ended up being effectively compiled with -xSSE4.2 and that got passed to scipy, but I don't know.

Vipin Kumar E K, FYI the -O3 error persists in the recently released 2016.2 verison, and -O1 must still be used with the Fortran compiler to get scipy tests to pass.

### When building SciPy, I notice

When building SciPy, I notice the compiler using -xSSE4.2 instead of -xhost. There's no intel.py that I can see in SciPy. What's the story there?

### After installing numpy 1.10.1

After installing numpy 1.10.1 following the instructions on this page I've run into a problem with scipy 0.16.1.  I'm running Linux Mageia 5, with Intel Composer 2013_sp1.3.174.

The error message is the following:

```icc: build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c
In file included from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h(1781),
from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h(18),
from /usr/lib64/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h(4),
from build/src.linux-x86_64-2.7/fortranobject.h(13),
from build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.c(19):
/usr/lib64/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h(15): warning #1224: #warning directive: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
#warning "Using deprecated NumPy API, disable it by " \
^

creating build/lib.linux-x86_64-2.7/scipy/fftpack
/usr/bin/gfortran -Wall -g -Wall -g -shared -Wl,-gc-sections -Wl,-s build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/zfft.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/drfft.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/zrfft.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/zfftnd.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/src/dct.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/src/dst.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/fortranobject.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-2.7 -ldfftpack -lfftpack -lpython2.7 -lgfortran -o build/lib.linux-x86_64-2.7/scipy/fftpack/_fftpack.so
/usr/bin/ld: build/temp.linux-x86_64-2.7/libdfftpack.a(dffti1.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
collect2: error: ld returned 1 exit status
/usr/bin/ld: build/temp.linux-x86_64-2.7/libdfftpack.a(dffti1.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
collect2: error: ld returned 1 exit status
error: Command "/usr/bin/gfortran -Wall -g -Wall -g -shared -Wl,-gc-sections -Wl,-s build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/_fftpackmodule.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/zfft.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/drfft.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/zrfft.o build/temp.linux-x86_64-2.7/scipy/fftpack/src/zfftnd.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/src/dct.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/scipy/fftpack/src/dst.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/fortranobject.o -L/usr/lib64 -Lbuild/temp.linux-x86_64-2.7 -ldfftpack -lfftpack -lpython2.7 -lgfortran -o build/lib.linux-x86_64-2.7/scipy/fftpack/_fftpack.so" failed with exit status 1```

With earlier versions of scipy I've not had this problem.  Any help appreciated, thanks.

### @Raffaella. I've documented

@Raffaella. I've documented how I installed Numpy/Scipy with openmp/icc/mkl_rt on a source-based GNU/Linux distribution but I can't get the URL past this site's spam filter. If you google `gentoo numpy multiprocessing',  it's the first hit. I know of no non-suboptimal alternatives to mkl_rt.