Developer Reference

  • 2020
  • 09/11/2020
  • Public Content

Environment Variables for Memory Policy Control

Intel® MPI Library supports non-uniform memory access (NUMA) nodes with high-bandwidth (HBW) memory (MCDRAM) on Intel® Xeon Phi™ processors (codenamed Knights Landing). Intel® MPI Library can attach memory of MPI processes to the memory of specific NUMA nodes. This section describes the environment variables for such memory placement control.
I_MPI_HBW_POLICY
Set the policy for MPI process memory placement for using HBW memory.
Syntax
I_MPI_HBW_POLICY=<user memory policy>[,<mpi memory policy>][,<win_allocate policy>]
In the syntax:
  • <user memory policy>
    - memory policy used to allocate the memory for user applications (required)
  • <mpi memory policy>
     - memory policy used to allocate the internal MPI memory (optional)
  • <win_allocate policy>
     - memory policy used to allocate memory for window segments for RMA operations (optional)
Each of the listed policies may have the values below:
Arguments
<value>
The memory allocation policy used.
hbw_preferred
Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
hbw_bind
Allocate only the local HBW memory for each process.
hbw_interleave
Allocate the HBW memory and dynamic random access memory on the local node in the round-robin manner.
Description
Use this environment variable to specify the policy for MPI process memory placement on a machine with HBW memory.
By default, Intel MPI Library allocates memory for a process in local DDR. The use of HBW memory becomes available only when you specify the
I_MPI_HBW_POLICY
variable.
Examples
The following examples demonstrate different configurations of memory placement:
  • I_MPI_HBW_POLICY=hbw_bind,hbw_preferred,hbw_bind
    Only use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local HBW memory internally allocated in Intel® MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library.
  • I_MPI_HBW_POLICY=hbw_bind,,hbw_bind
    Only use the local HBW memory allocated in user applications and window segments for RMA operations. Use the local DDR internally allocated in Intel MPI Library.
  • I_MPI_HBW_POLICY=hbw_bind,hbw_preferred
    Only use the local HBW memory allocated in user applications. Use the local HBW memory internally allocated in Intel MPI Library first. If the HBW memory is not available, use the local DDR internally allocated in Intel MPI Library. Use the local DDR allocated in window segments for RMA operations.
I_MPI_BIND_NUMA
Set the NUMA nodes for memory allocation.
Syntax
I_MPI_BIND_NUMA=
<value>
Arguments
<value>
Specify the NUMA nodes for memory allocation.
localalloc
Allocate memory on the local node. This is the default value.
Node_1,…,Node_k
Allocate memory according to
I_MPI_BIND_ORDER
on the specified NUMA nodes.
Description
Set this environment variable to specify the NUMA node set that is involved in the memory allocation procedure.
I_MPI_BIND_ORDER
Set this environment variable to define the memory allocation manner.
Syntax
I_MPI_BIND_ORDER=
<value>
Arguments
<value>
Specify the allocation manner.
compact
Allocate memory for processes as close as possible (in terms of NUMA nodes), among the NUMA nodes specified in
I_MPI_BIND_NUMA
. This is the default value.
scatter
Allocate memory among the NUMA nodes specified in
I_MPI_BIND_NUMA
using the round-robin manner.
Description
Set this environment variable to define the memory allocation manner among the NUMA nodes specified in
I_MPI_BIND_NUMA
. The variable has no effect without
I_MPI_BIND_NUMA
set.
I_MPI_BIND_WIN_ALLOCATE
Set this environment variable to control memory allocation for window segments.
Syntax
I_MPI_BIND_WIN_ALLOCATE=
<value>
Arguments
<value>
Specify the memory allocation behavior for window segments.
localalloc
Allocate memory on the local node. This is the default value.
hbw_preferred
Allocate the local HBW memory for each process. If the HBW memory is not available, allocate the local dynamic random access memory.
hbw_bind
Allocate only the local HBW memory for each process.
hbw_interleave
Allocate the HBW memory and dynamic random access memory on a local node in the round-robin manner.
<NUMA node id>
Allocate memory on the given NUMA node.
Description
Set this environment variable to create window segments allocated in HBW memory with the help of the
MPI_Win_allocate_shared
or
MPI_Win_allocate
functions.
MPI_Info
You can control memory allocation for window segments with the help of an
MPI_Info
object, which is passed as a parameter to the
MPI_Win_allocate
or
MPI_Win_allocate_shared
function. In an application, if you specify such an object with the
numa_bind_policy
key, window segments are allocated in accordance with the value for
numa_bind_policy
. Possible values are the same as for
I_MPI_BIND_WIN_ALLOCATE
.
A code fragment demonstrating the use of
MPI_Info
:
MPI_Info info; ... MPI_Info_create( &info ); MPI_Info_set( info, "numa_bind_policy", "hbw_preferred" ); ... MPI_Win_allocate_shared( size, disp_unit, info, comm, &baseptr, &win );
Note
When you specify the memory placement policy for window segments, Intel MPI Library recognizes the configurations according to the following priority:
  1. Setting of
    MPI_Info.
  2. Setting of
    I_MPI_HBW_POLICY
    , if you specified
    <win_allocate policy>.
  3. Setting of
    I_MPI_BIND_WIN_ALLOCATE.
Syntax
Arguments
Description

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804