Forthcoming Intel® MKL 11.0 (Expected date of release: Autumn 2012)

•  Intel® MKL 11.0 will have backward incompatibility with Intel® MKL 10.2 update 3

•  Open MP static library on Microsoft* Windows

•  GMP* Arithmetic functions

•  Intel® MKL Reference Manual will be removed from product package

•  Intel® Pentium III will no longer be supported

•  Red Hat* EL4 and PGI* Fortran 10.x support will be dropped

•  PGI* Fortran 77 support will be removed

• a powerful N-dimensional array object
• sophisticated (broadcasting) functions
• tools for integrating C/C++ and Fortran code
• useful linear algebra, Fourier transform, and random number capabilities.

Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data.

For more information on NumPy, please visit http://NumPy.scipy.org/

Version Information

This application note was created to help NumPy users to make use of the latest versions of Intel MKL on Linux platforms.

The instructions given in this articles apply to Intel MKL 10.3 and above and Intel Compiler 11.0 and above.

 
Step 2 - Downloading NumPy Source Code

The NumPy source code can be downloaded from:

http://NumPy.scipy.org/

Prerequisites

Intel MKL can be obtained from the following options:

Download a FREE evaluation version of the Intel MKL product.
Download the FREE non-commercial* version of the Intel MKL product.

All of these can be obtained at: Intel® Math Kernel Library product web page.

Intel® MKL is also bundled with the following products

Intel® Parallel Studio XE 2011
Intel Parallel Composer XE 2011
Intel Cluster Studio 2011

Step 3 - Configuration

Use the following commands to extract the NumPy tar files from the downloaded NumPy-x.x.x.tar.gz.

1.	$gunzip numpy-x.x.x.tar.gz 
2.	$tar -xvf numpy-x.x.x.tar 

 
The above will create a directory named numpy-x.x.x

Make sure that C++ and FORTRAN compilers are installed and they are in PATH. Also set LD_LIBRARY_PATH to your compiler (C++ and FORTRAN), and MKL libraries.

Step 4 - Building NumPy

Change directory to numpy-x.x.x
Create a site.cfg from the existing one
 
Edit site.cfg as follows:

Add the following lines to site.cfg in your top level NumPy directory to use Intel® MKL, if you are building on Intel 64 platform:

[mkl]
library_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2011.sp1.6.233/mkl/include
mkl_libs = mkl_rt
lapack_libs =

If you are building NumPy for 32 bit, please add as the following

[mkl]
library_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/ia32
include_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/include
mkl_libs = mkl_rt
lapack_libs = 

Modify cc_exe in numpy/distutils/intelccompiler.py to be something like:

self.cc_exe = 'icc -O3 -g -fPIC -fp-model strict -fomit-frame-pointer -openmp -xhost'

Here we use, -O3, optimizations for speed and enables more aggressive loop transformations such as Fusion, Block-Unroll-and-Jam, and collapsing IF statements, -openmp for OpenMP threading and -xhost option tells the compiler to generate instructions for the highest instruction set available on the compilation host processor. If you are using the ILP64 interface, please add -DMKL_ILP64 compiler flag.

Run icc --help for more information on processor-specific options, and refer Intel Compiler documentation for more details on the various compiler flags.

Compile and install NumPy with the Intel compiler: (on 64-bit platforms replace "intel" with "intelem")

python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install

If you build NumPY for Intel64 bit platforms:

$export LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64:/opt/intel/composer_xe_2011_sp1.6.233/compiler/lib/intel64:$LD_LIBRARY_PATH

If you build NumPY for ia32 bit platforms:

$export LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/ia32:/opt/intel/composer_xe_2011_sp1.6.233/compiler/lib/ia32:$LD_LIBRARY_PATH

It is possible that LD_LIBRARY_PATH causes a problem, if you have installed MKL and Composer XE in other directories than the standard ones. The only solution I've found that always works is to build Python, NumPy and SciPy inside an environment where you've set the LD_RUN_PATH variable, e.g:

$export LD_RUN_PATH=~/opt/lib:~/intel/composer_xe_2011_sp1.6.233/compiler/lib:~/intel/composer_xe_2011_sp1.6.233/mkl/lib/ia32

Note: We recommend users to use arrays with 'C' ordering style which is row-major, which is default than Fortran Style which is column-major, and this is because NumPy uses CBLAS and also to get better performance.

Appendex A: Example:

Please see below an example Python script for matrix multiplication that you can use Numply installed with Intel MKL which has been provided for illustration purpose.

import numpy as np
import time

N = 6000
M = 10000

k_list = [64, 80, 96, 104, 112, 120, 128, 144, 160, 176, 192, 200, 208, 224, 240, 256, 384]

def get_gflops(M, N, K):
	return M*N*(2.0*K-1.0) / 1000**3

np.show_config()

for K in k_list:
	a = np.array(np.random.random((M, N)), dtype=np.double, order='C', copy=False)
	b = np.array(np.random.random((N, K)), dtype=np.double, order='C', copy=False)
	A = np.matrix(a, dtype=np.double, copy=False)
	B = np.matrix(b, dtype=np.double, copy=False)

	C = A*B

	start = time.time()

	C = A*B
	C = A*B
	C = A*B
	C = A*B
	C = A*B

	end = time.time()

	tm = (end-start) / 5.0

	print "{0:4}, {1:9.7}, {2:9.7}".format(K, tm, get_gflops(M, N, K) / tm)

Appendix B: Performance Comparison

Please click Examples.py to download the examples for LU, Cholesky and SVD.



 

Below each error is described in details:

0: no error

-1: Input inconsistency

• Incorrect stage number for PARDISO was called.
• Incorrect PARDISO calling sequence, e.g. run stage > 1 without initialization results in error reporting.
• Incorrect number of matrices to be solved was set (PARDISO can be used for solving several matrices with the same sparsity structure at once, taking into account that their maximum number was defined previously. Setting the number of matrices to be solved outside the range of 1≤ … ≤maxfct results in error).
• PARDISO checks the parameters at each stage for consistency with the parameters at the previous stages. Every disagreement results in error reporting.

-2: not enough memory

This error value is returned in the case of any problem with memory allocation inside PARDISO.

PARDISO messages:

"*** Error in PARDISO memory allocation: [STRUCTURE NAME], size to allocate: %d bytes"

"total memory wanted here: %d kbyte"

"symbolic (max): %d symbolic (permanent): %d"

"real(including 1 factor): %d"

It describes the issue that arises on allocation of STRUCTURE_NAME inside PARDISO. A additional information about current memory usages is also printed.

-3: reordering problem

Returned for any problem on a reordering stage (phase 11)

PARDISO messages:

"*** error PARDISO: reordering, symbolic factorization"

-4: zero pivot, numerical factorization or iterative refinement problem

Let us start with a citation of the Intel MKL manual:

Using phase =33 results in an error message (error =4, should be -4 * ) if the stopping criteria for the Krylow-Subspace iteration cannot be reached.

If phase= 23, then the factors L, U are recomputed for the matrix A and the error flag error=0 in case of a successful factorization. If phase =33, then error = -4 signals the failure

If the solver detects a zero or negative pivot for these matrix types, the factorization is stopped, PARDISO returns immediately with an error (error = -4) and iparm(30) contains the number of the equation where the first zero or negative pivot is detected.

The error returned in the case of any problem at the factorization stage (phase 22) or at the iterative refinement stage of solution.

PARDISO messages:

"*** Error in PARDISO: cgs error iparam(20) %d" – prints iparm(20) – see CG / CGS diagnostics in the Intel MKL manual

"*** error PARDISO: iterative refinement"

" contraction rate is greater than 0.9, interrupt" – rate of contraction is too small

"*** error PARDISO: iterative refinement"

" exceeds max. iteration number %d" - prints abs(iparm(8))

"*** Error in PARDISO: internal error, insufficient memory factorization" – looks like a problem at the factorization stage except for pivoting issues (see errors below)

"*** Error in PARDISO: zero or negative pivot, A is not SPD-matrix" – original matrix (almost) not SPD one (probably due to computer arithmetic inaccuracies)

"*** Error in PARDISO: zero pivot" – the same as above but for other matrix types

-5: unclassified (internal) error

This error value is not used currently and is reserved for the future use.

-6: preordering failed (matrix types 11, 13 only)

This error value is returned in the case of any problem at the stage of preparation for reordering (in matching algorithm).

PARDISO messages:

"*** Error in PARDISO: preordering failed after %d neqns out of %d"

"structure singular or input/parameter problem (matrix type 11,13)"

Looks like the message provides no meaningful information.

-7: diagonal matrix problem

PARDISO prints no messages.

-8: 32-bit integer overflow problem

This error value is returned on 32-bit architecture for big matrices when indices become greater than the maximal integer value on this platform.

PARDISO messages:

"*** error PARDISO: reordering, symbolic factorization"

-9: not enough memory for OOC

Let us start with a citation of the Intel MKL manual:

Note that if iparm(60) is equal to 1 or 2, and the total peak memory needed for strong local arrays is more than MKL_PARDISO_OOC_MAX_CORE_SIZE, the program stops with error -9. In this case, increase of MKL_PARDISO_OOC_MAX_CORE_SIZE is recommended.

This error value is returned when amount of memory available for PARDISO (defined by MKL_PARDISO_OOC_MAX_CORE_SIZE, by default 2000 Mb) is not enough to solve the current matrix. The issue can be resolved by increasing the value for available memory.

-10: problems with opening OOC temporary files

This error value is returned when PARDISO can’t create / open temporary files for storing OOC arrays, e.g. in the case of wrong permissions or when files were removed or blocked or not released after the previous steps.

-11: read/write problems with the OOC data file

This error value is returned when some problems appear in the process of working with files, e.g. in the case of no space left on device or problems with read / write operations because of algorithm issues.

* - the documentation error. Will be fixed in the version 10.3 Update3.

** - available memory means the RAM system's memory which is available at the moment of starting the calculations.

Introduction : 

By default the Intel(R) Debugger requires the ability to open either a GUI or at least a dedictaed separate xterm window. This article describes how to launch a command line verion of MPI enabled IDB in a command line only terminal window.


Version :
This article applies to the Intel(R) Debugger for Linux* v11.0 and higher
The proposed command ine and testing of this approach was based on Intel(R) MPI Library 3.x, but newer versions should work as well.
The screen command needs to be available

Configuration Set Up :
The usage model this article applies to is as follows:

- network based Linux MPI cluster with application possibly running distributed over several nodes

- only available access to the cluster is via remote terminal login (puTTY, ssh, ...)

- no X window output redirection to the login machine is possible 

- Please ensure to IDB_PARALLEL_SHELL set to enable multi-node debugging

"export IDB_PARALLEL_SHELL=/usr/bin/ssh"

Solution:

- Edit mpiexec.py launch script in the bin directory of your Intel(R) MPI Library installation
- Launch debugger with "screen mpiexec -idb -n <node number> <debuggee executable>"

Python Script Changes :

1. In the line that defines dbg_cmd replace "xterm" with "screen" and remove "-e" in mpiexec.py. In addition remove the "&" at the end of the line.

E.g. replace the line

dbg_cmd = 'xterm -e ' + tv_exe + ' -pid ' + `os.getpid()` + ' -mpi2 -parallel /usr/bin/python &'

with

dbg_cmd = 'screen' + tv_exe + ' -pid ' + `os.getpid()` + ' -mpi2 -parallel /usr/bin/python'

2. Replace the line

os.system(dbg_cmd)

with the following lines:

cpid = os.fork()

if (cpid == 0):
os.system(dbg_cmd)
sys.exit(0)

Question and Answers:

Q: If I install Intel® Fortran Composer XE, will my old version still be there?

A: If by old version you mean 11.1, etc., then yes. The Visual Studio integration for Intel® Visual Fortran Composer XE 2011 gives you access to versions 11.0 and 11.1 through Tools > Options > Intel Visual Fortran > Compilers. Note that there are separate entries for 32-bit and 64-bit development.

On Linux* and Mac OS X*, the new installation will default to /opt/intel. Under this directory you will see your older 11.x compilers under the Compiler/11.1/xxx/ directories. The new Intel® Composer XE will have physical directory /opt/intel/composerxe-2011.12.1.122 (or similar): deciphering "12.1.122" is major version 12, Update 1, minor version 122. Along with this directory, you will see the new generic directory link "composerxe/" which links this generic path to the last version of the compiler installed on the system. These generic paths allow one to 'source /opt/intel/composerxe/bin/compilervars.sh intel64' and be assured that this path will exist and will link to the latest version compiler installed on the system. This provides a version-independent, default path to compilervars.sh for those who don't want to rewrite their makefiles or shell startup files for each new version installed on the system.



Q: I have been using Fortran for scientific applications since the 1980s, but am not a professional programmer. What is the most efficient way to learn how to properly use new features?
Q: Do you provide some level of guidance to write a parallel processing program?
Q: Is there a refresher course available to bring me up to speed from earlier versions of Fortran to the latest version?

A: Intel does not offer training on the Fortran language. There are several good books on the language which are more tutorial in nature. A list of books we have seen can be found in the compiler documentation under Introduction > Related Information



Q: Would it be possible to change the default N of images (Coarray Fortran) to 1, instead of the number of available threads?

A: Yes. This can be specified as an option (-coarray-num-images=1 on linux or /Qcoarray-num-images:1 for Windows) when compiling the main program or, before the program is run, by setting the environment variable FOR_COARRAY_NUM_IMAGES to be the number of images you want. This can be more or fewer than the number of available threads.



Q: Will there be extensions to Intel Fortran to permit programming GPUs using Fortran?

A: Intel will be extending its Fortran and C++ compilers to encompass Intel's future "Many Integrated Core" products. We do not have plans at this time to support GPUs from other vendors.



Q: What is the relationship between OpenMP and Coarrays?
Q: How did you implement coarrays? Does it use OpenMP?

A: We use Intel® MPI Library as the underlying support for coarrays, both for shared memory and distributed memory configurations. Intel MPI Library is a highly optimized implementation of MPI that supports a wide variety of interconnect types.

At present, we do not support the use of other parallel methods such as OpenMP or direct MPI calls in an application that uses coarrays. We recognize that customers may want to use multiple technologies and will be testing this in the future. You are welcome to try it yourself, but we don't promise it will work.



Q: Is it possible to have Intel Visual Fortran in Windows without Microsoft Visual Studio*?

A: Intel Visual Fortran relies on tools and libraries that come from Microsoft* Visual Studio*. If a system does not have a supported version of Visual Studio installed (2005, 2008 or 2010), we will install an environment based on Visual Studio 2008 Shell which supports Fortran development. You do not need to use the Visual Studio graphical development environment – you can still use the command line to build applications – but the only debugging environment supported is Visual Studio's.



Q: Is this compatible with the Intel® Cluster Studio XE?

A: Intel Cluster Studio XE 2011 includes Intel [Visual] Fortran Composer XE with full capabilities, including support for distributed memory coarrays.



Q: Can we expect improvements to the debugging experience when moving to VS2010 and Intel Visual Fortran Composer XE?

A: The debugging experience is improved continually, and we have resolved some known issues in Intel Visual Fortran Composer XE. A move to Visual Studio 2010, on its own, has no specific benefits here.



Q: Does Intel® Parallel Studio support Intel Visual Fortran?

A: Intel Parallel Studio is a C++ product, and its analysis tools do not support Fortran. Intel Parallel Studio XE includes both Intel C++ and Intel Fortran, as well as analysis tools that support both languages.



Q: I have found polymorphic entities are not very useful w/o sourced allocation. Any idea when it might come out.?

A: We can't say at this time when this feature will be added, but we are working on it.



Q: Is there any possibility of getting the documentation in pdf files?
A: Not at this time.



Q: Is Intel Visual Fortran compatible with Visual Studio 2005*?
A: Yes.



Q: Who besides Intel is involved in developing the Fortran standards?

A: You can view the membership of the US Fortran Standards committee at http://www.j3-fortran.org/. Vendors who are members of this committee are Cray, IBM, Intel and Oracle (Sun). A representative from NAG is a committee member, but not formally representing NAG. A member of the gfortran development team regularly attends meetings as well.



Q: Are there any plans to support Intellisense features in Fortran Composer XE?

A: We have had requests for some Intellisense features. It would help us if you would let us know, either in the user forum or an Intel® Premier Support issue, which specific features you are interested in.



Q: Are there any specific performance reasons for upgrading from 11.1 to the new version (Performance improvements - speed up execution that is)?
Q: Has performance been improved with v12 (assuming the same options are used)?

A: Yes, the version 12 compiler (Intel Fortran Composer XE) has numerous optimization improvements, even using the same options. Of course, your specific results may vary. As an example, loops with expressions with mixed data types (single and double precision, for example) are candidates for vectorization in version 12.



Q: How long is the Fortran standard going to be extended? Will there be an end to further standards? What is the long term strategy?

A: The Fortran standards committee is currently working on interpretations to the newly-approved Fortran 2008 standard. A revision, currently referred to as Fortran 2013, is being planned, but it is unclear what the scope of that will be. A majority of the committee wants it to be a minor update along the lines of Fortran 95.



Q: What are the plans for the editor enhancements? This has been asked a lot of times on the forum.
Q: Any plans for enhancements in Visual Studio (e.g., type viewer, module viewer), important for large projects
Q: Will there be regions so that we can hide program units that are not being developed?

A: We are working on enhancements to the Visual Studio environment, but can't comment on specifics.



Q: Does the new Intel Fortran Compiler support ubuntu 10.04?
A: Yes.



Q: Are there any plans for supporting the simple idb -gui debugging tool on OSX. Xcode is nice, but it requires a bit of investment in time to use.

A: We don't currently have plans for extending the idb debugger with a GUI interface at this time. There is a workaround to use the graphfical DDD front-end with Intel® Debugger. This workaround is documented in this forum post: http://software.intel.com/en-us/forums/showthread.php?t=60060.



Q: When will the IMSL libraries be included
A: Expect to see these available in late January or perhaps early February.



Q: When will defined type IO be added. I currently use namelists of temporary defined types.
A: We can't comment on a timeframe for this, but it will be there eventually.



Q: What about coarrays for Mac OS X*?
A: We have no plans for that at this time.



Q: Are there plans to revive Array Visualizer?
A: This is still being considered.



Q: How many multithreading methodologies does Intel Fortran Composer XE have? and What are they?
A: In the product itself, the compiler supports:
• Automatic parallelization (/Qparallel or –parallel), along with Guided Auto Parallelization to help tune the application for parallelism.
• OpenMP
• Coarrays
• DO CONCURRENT regions along with –parallel compiler option
Using external libraries, you can also use MPI or the operating system's threading calls.



Q: Any support for the old source browser option?
A: Work is being done in this area, but we can't comment further at this time.



Q: What is the difference between /fp:precise switch and the new improved switch?

A: As explained in the presentation, the /fp (-fp-model) switch is very different from the new option to select among alternate math libraries. The new options allow you to choose a variant of the math library that is faster but less precise, or one that delivers the same results on all processor types. The fp-model can affect how expressions are evaluated, and can disable/enable certain optimizations. The new libraries merely replace the default libimf library which contains intrinsics such as exp, sin, cos, etc. So these new libraries should be considered for use along with the –fp-model settings. For example, for very strict numeric results that you'd like to replicate on multiple cpu architectures: -fp-model exact –fimf-arch-consistency=true could be used together to insure the best possible results and consistency across many cpu generations. See the –fimf-* options. This could be a topic we revisit in a separate dedicated webinar next year.