Hydra Environment Variables

I_MPI_HYDRA_HOST_FILE

Set the host file to run the application.

Syntax

I_MPI_HYDRA_HOST_FILE=<arg>

Arguments

<arg>

String parameter

<hostsfile>

The full or relative path to the host file

Description

Set this environment variable to specify the hosts file.

I_MPI_HYDRA_HOSTS_GROUP

Set node ranges using brackets, commas, and dashes.

Syntax

I_MPI_HYDRA_HOSTS_GROUP=<arg>

Argument

<arg>

Set a node range.

Description

Set this variable to be able to set node ranges using brackets, commas, and dashes (like in Slurm* Workload Manager). For example:

I_MPI_HYDRA_HOSTS_GROUP=”hostA[01-05],hostB,hostC[01-05,07,09-11]”

You can set node ranges with the -hosts-group option.

I_MPI_HYDRA_DEBUG

Print out the debug information.

Syntax

I_MPI_HYDRA_DEBUG=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on the debug output

disable | no | off | 0

Turn off the debug output. This is the default value

Description

Set this environment variable to enable the debug mode.

I_MPI_HYDRA_ENV

Control the environment propagation.

Syntax

I_MPI_HYDRA_ENV=<arg>

Arguments

<arg>

String parameter

all

Pass all environment to all MPI processes

Description

Set this environment variable to control the environment propagation to the MPI processes. By default, the entire launching node environment is passed to the MPI processes. Setting this variable also overwrites environment variables set by the remote shell.

I_MPI_JOB_TIMEOUT

Set the timeout period for mpiexec.hydra.

Syntax

I_MPI_JOB_TIMEOUT=<timeout>

I_MPI_MPIEXEC_TIMEOUT=<timeout>

Arguments

<timeout>

Define mpiexec.hydra timeout period in seconds

<n>0

The value of the timeout period. The default timeout value is zero, which means no timeout.

Description

Set this environment variable to make mpiexec.hydra terminate the job in <timeout> seconds after its launch. The <timeout> value should be greater than zero. Otherwise the environment variable setting is ignored.

Note

Set this environment variable in the shell environment before executing the mpiexec.hydra command. Setting the variable through the -genv and -env options has no effect.

I_MPI_JOB_STARTUP_TIMEOUT

Set the mpiexec.hydra job startup timeout.

Syntax

I_MPI_JOB_STARTUP_TIMEOUT=<timeout>

Arguments

<timeout>

Define mpiexec.hydra startup timeout period in seconds

<n>0

The value of the timeout period. The default timeout value is zero, which means no timeout.

Description

Set this environment variable to make mpiexec.hydra terminate the job in <timeout> seconds if some processes are not launched. The <timeout> value should be greater than zero.

I_MPI_JOB_TIMEOUT_SIGNAL

Define the signal to be sent when a job is terminated because of a timeout.

Syntax

I_MPI_JOB_TIMEOUT_SIGNAL=<number>

Arguments

<number>

Define the signal number

<n> > 0

The signal number. The default value is 9 (SIGKILL)

Description

Define a signal number to be sent to stop the MPI job if the timeout period specified by the I_MPI_JOB_TIMEOUT environment variable expires. If you set a signal number unsupported by the system, the mpiexec.hydra command prints a warning message and continues the task termination using the default signal number 9 (SIGKILL).

Note

Set this environment variable in the shell environment before executing the mpiexec.hydra command. Setting the variable through the -genv and -env options has no effect.

I_MPI_JOB_ABORT_SIGNAL

Define a signal to be sent to all processes when a job is terminated unexpectedly.

Syntax

I_MPI_JOB_ABORT_SIGNAL=<number>

Arguments

<number>

Define signal number

<n> > 0

The default value is 9 (SIGKILL)

Description

Set this environment variable to define a signal for task termination. If you set an unsupported signal number, mpiexec.hydra prints a warning message and uses the default signal 9 (SIGKILL).

Note

Set this environment variable in the shell environment before executing the mpiexec.hydra command. Setting the variable through the -genv and -env options has no effect.

I_MPI_JOB_SIGNAL_PROPAGATION

Control signal propagation.

Syntax

I_MPI_JOB_SIGNAL_PROPAGATION=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Turn on propagation

disable | no | off | 0

Turn off propagation. This is the default value

Description

Set this environment variable to control propagation of the signals (SIGINT, SIGALRM, and SIGTERM). If you enable signal propagation, the received signal is sent to all processes of the MPI job. If you disable signal propagation, all processes of the MPI job are stopped with the default signal 9 (SIGKILL).

Note

Set this environment variable in the shell environment before executing the mpiexec.hydra command. Setting the variable through the -genv and -env options has no effect.

I_MPI_HYDRA_BOOTSTRAP

Set the bootstrap server.

Syntax

I_MPI_HYDRA_BOOTSTRAP=<arg>

Arguments

<arg>

String parameter

ssh

Use secure shell. This is the default value

rsh

Use remote shell

pdsh

Use parallel distributed shell

pbsdsh

Use Torque* and PBS* pbsdsh command

fork

Use fork call

slurm

Use SLURM* srun command

ll

Use LoadLeveler* llspawn.stdio command

lsf

Use LSF* blaunch command

sge

Use Univa* Grid Engine* qrsh command

jmi

Use Job Manager Interface (tighter integration)

Description

Set this environment variable to specify the bootstrap server.

Note

Set the I_MPI_HYDRA_BOOTSTRAP environment variable in the shell environment before executing the mpiexec.hydra command. Do not use the -env option to set the <arg> value. This option is used for passing environment variables to the MPI process environment.

I_MPI_HYDRA_BOOTSTRAP_EXEC

Set the executable file to be used as a bootstrap server.

Syntax

I_MPI_HYDRA_BOOTSTRAP_EXEC=<arg>

Arguments

<arg>

String parameter

<executable>

The name of the executable file

Description

Set this environment variable to specify the executable file to be used as a bootstrap server.

I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS

Set additional arguments for the bootstrap server.

Syntax

I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=<arg>

Arguments

<arg>

String parameter

<args>

Additional bootstrap server arguments

Description

Set this environment variable to specify additional arguments for the bootstrap server.

Note

If the launcher (blaunch, lsf, pdsh, pbsdsh) falls back to ssh, pass the arguments with the invocation of ssh.

I_MPI_HYDRA_BOOTSTRAP_AUTOFORK

Control the usage of fork call for local processes.

Syntax

I_MPI_HYDRA_BOOTSTRAP_AUTOFORK = <arg>

Arguments

<arg>

String parameter

enable | yes | on | 1

Use fork for the local processes. This is default value for ssh, rsh, ll, lsf, and pbsdsh bootstrap servers

disable | no | off | 0

Do not use fork for the local processes. This is default value for the sge bootstrap server

Description

Set this environment variable to control usage of fork call for the local processes.

Note

This option is not applicable to slurm, pdsh, and persist bootstrap servers.

I_MPI_HYDRA_RMK

Use the specified value as the resource management kernel to obtain data about available nodes, externally set process counts.

Syntax

I_MPI_HYDRA_RMK=<arg>

Arguments

<arg>

String parameter

<rmk>

Resource management kernel. The supported values are slurm, ll, lsf, sge, pbs, cobalt.

Description

Set this environment variable to use the resource management kernel.

I_MPI_HYDRA_PMI_CONNECT

Define the processing method for PMI messages.

Syntax

I_MPI_HYDRA_PMI_CONNECT=<value>

Arguments

<value>

The algorithm to be used

nocache

Do not cache PMI messages

cache

Cache PMI messages on the local pmi_proxy management processes to minimize the number of PMI requests. Cached information is automatically propagated to child management processes.

lazy-cache

cache mode with on-demand propagation.

alltoall

Information is automatically exchanged between all pmi_proxy before any get request can be done. This is the default value.

Description

Use this environment variable to select the PMI messages processing method.

I_MPI_PERHOST

Define the default behavior for the -perhost option of the mpiexec.hydra command.

Syntax

I_MPI_PERHOST=<value>

Arguments

<value>

Define a value used for -perhost by default

integer > 0

Exact value for the option

all

All logical CPUs on the node

allcores

All cores (physical CPUs) on the node. This is the default value.

Description

Set this environment variable to define the default behavior for the -perhost option. Unless specified explicitly, the -perhost option is implied with the value set in I_MPI_PERHOST.

Note

When running under a job scheduler, this environment variable is ignored by default. To be able to control process placement with I_MPI_PERHOST, disable the I_MPI_JOB_RESPECT_PROCESS_PLACEMENT variable.

I_MPI_JOB_TRACE_LIBS

Choose the libraries to preload through the -trace option.

Syntax

I_MPI_JOB_TRACE_LIBS=<arg>

Arguments

<arg>

String parameter

<list>

Blank separated list of the libraries to preload. The default value is vt

Description

Set this environment variable to choose an alternative library for preloading through the -trace option.

I_MPI_JOB_CHECK_LIBS

Choose the libraries to preload through the -check_mpi option.

Syntax

I_MPI_JOB_CHECK_LIBS=<arg>

Arguments

<arg>

String parameter

<list>

Blank separated list of the libraries to preload. The default value is vtmc

Description

Set this environment variable to choose an alternative library for preloading through the -check_mpi option.

I_MPI_HYDRA_BRANCH_COUNT

Set the hierarchical branch count.

Syntax

I_MPI_HYDRA_BRANCH_COUNT =<num>

Arguments

<num>

Number

<n> >= 0

  • The default value is -1 if less than 128 nodes are used. This value also means that there is no hierarchical structure

  • The default value is 32 if more than 127 nodes are used

Description

Set this environment variable to restrict the number of child management processes launched by the mpiexec.hydra operation or by each pmi_proxy management process.

I_MPI_HYDRA_PMI_AGGREGATE

Turn on/off aggregation of the PMI messages.

Syntax

I_MPI_HYDRA_PMI_AGGREGATE=<arg>

Arguments

<arg>

Binary indicator

enable | yes | on | 1

Enable PMI message aggregation. This is the default value.

disable | no | off | 0

Disable PMI  message aggregation.

Description

Set this environment variable to enable/disable aggregation of PMI messages.

I_MPI_HYDRA_GDB_REMOTE_SHELL

Set the remote shell command to run GNU* debugger.

Syntax

I_MPI_HYDRA_GDB_REMOTE_SHELL=<arg>

Arguments

<arg>

String parameter

ssh

Secure Shell (SSH). This is the default value

rsh

Remote shell (RSH)

Description

Set this environment variable to specify the remote shell command to run the GNU* debugger on the remote machines. You can use this environment variable to specify any shell command that has the same syntax as SSH or RSH.

I_MPI_HYDRA_IFACE

Set the network interface.

Syntax

I_MPI_HYDRA_IFACE=<arg>

Arguments

<arg>

String parameter

<network interface>

The network interface configured in your system

Description

Set this environment variable to specify the network interface to use. For example, use "-iface ib0", if the IP emulation of your InfiniBand* network is configured on ib0.

I_MPI_HYDRA_DEMUX

Set the demultiplexer (demux) mode.

Syntax

I_MPI_HYDRA_DEMUX=<arg>

Arguments

<arg>

String parameter

poll

Set poll as the multiple I/O demultiplexer (demux) mode engine. This is the default value.

select

Set select as the multiple I/O demultiplexer (demux) mode engine

Description

Set this environment variable to specify the multiple I/O demux mode engine. The default value is poll.

I_MPI_TMPDIR

Specify a temporary directory.

Syntax

I_MPI_TMPDIR=<arg>

Arguments

<arg>

String parameter

<path>

Temporary directory. The default value is /tmp

Description

Set this environment variable to specify a directory for temporary files.

I_MPI_JOB_RESPECT_PROCESS_PLACEMENT

Specify whether to use the process-per-node placement provided by the job scheduler, or set explicitly.

Syntax

I_MPI_JOB_RESPECT_PROCESS_PLACEMENT=<arg>

Arguments

<value>

Binary indicator

enable | yes | on | 1

Use the process placement provided by job scheduler. This is the default value

disable | no | off | 0

Do not use the process placement provided by job scheduler

Description

If the variable is set, the Hydra process manager uses the process placement provided by job scheduler (default). In this case the -ppn option and its equivalents are ignored. If you disable the variable, the Hydra process manager uses the process placement set with -ppn or its equivalents.

I_MPI_GTOOL

Specify the tools to be launched for selected ranks. An alternative to this variable is the -gtool option.

Syntax

I_MPI_GTOOL=“<command line for a tool 1>:<ranks set 1>[=exclusive][@arch 1]; <command line for a tool 2>:<ranks set 2>[=exclusive][@arch 2]; … ;<command line for a tool n>:<ranks set n>[=exclusive][@arch n]”

Arguments

<arg>

Parameters

<command line for a tool>

Specify a tool's launch command, including parameters.

<rank set>

Specify the range of ranks that are involved in the tool execution. Separate ranks with a comma or use the '-' symbol for a set of contiguous ranks. To run the tool for all ranks, use the all argument.

Note

If you specify incorrect rank index, the corresponding warning is printed and the tool continues working for valid ranks.

[=exclusive]

Specify this parameter to prevent launching a tool for more than one rank per host. This parameter is optional.

[@arch]

Specify the architecture on which the tool runs (optional). For a given <rank set>, if you specify this argument, the tool is launched only for the processes residing on hosts with the specified architecture. This parameter is optional.

Description

Use this option to launch the tools such as Intel® VTune™ Amplifier XE, Valgrind*, and GNU* Debugger for the specified processes.

Examples

The following command line examples demonstrate different scenarios of using the I_MPI_GTOOL environment variable.

Launch Intel® VTune™ Amplifier XE and Valgrind* by setting the I_MPI_GTOOL environment variable:

$ export I_MPI_GTOOL="amplxe-cl -collect hotspots -analyze-system -r result1:5,3,7-9=exclusive@bdw;\
valgrind -log-file=log_%p:0,1,10-12@hsw"
$ mpiexec.hydra -n 16 a.out

This command launches amplxe-cl for the processes that are run on the Intel® microarchitecture codenamed Broadwell. Only one copy of amplxe-cl is launched for each host, the process with the minimal index is affected. At the same time, Valgrind* is launched for all specified processes that are run on the Intel® microarchitecture codenamed Haswell. Valgrind's results are saved to the files log_<process ID>.

Launch GNU* Debugger (GDB*) by setting the I_MPI_GTOOL environment variable:

$ mpiexec.hydra -n 16 -genv I_MPI_GTOOL="gdb:3,5,7-9" a.out

Use this command to apply gdb to the given rank set.

Note

The options and the environment variable -gtool, -gtoolfile and I_MPI_GTOOL are mutually exclusive. The options -gtool and -gtoolfile are of the same priority and have higher priority than I_MPI_GTOOL. The first specified option in a command line is effective and the second one is ignored. Therefore, use I_MPI_GTOOL if you do not specify -gtool or -gtoolfile.

I_MPI_HYDRA_TOPOLIB

Set the interface for topology detection.

Syntax

I_MPI_HYDRA_TOPOLIB=<arg>

Arguments

<arg>

String parameter

hwloc

The hwloc* library functions are invoked for topology detection.

Description

Set this environment variable to define the interface for platform detection. The hwloc* interface is utilized if the variable is set explicitly: I_MPI_HYDRA_TOPOLIB=hwloc. Otherwise, the native Intel® MPI Library interface is used, which is the default behavior.

I_MPI_PORT_RANGE

Specify a range of allowed port numbers.

Syntax

I_MPI_PORT_RANGE=<range>

Arguments

<range>

String parameter

<min>:<max>

Allowed port range

Description

Set this environment variable to specify a range of the allowed port numbers for the Intel® MPI Library.

I_MPI_SILENT_ABORT

Control abort warning messages.

Syntax

I_MPI_SILENT_ABORT=<arg>

Argument

<arg>

Binary indicator

enable | yes | on | 1

Do not print abort warning message

disable | no | off | 0

Print abort warning message. This is the default value

Description

Set this variable to disable printing of abort warning messages. The messages are printed in case of the MPI_Abort call.  

You can also disable printing of these messages with the -silent-abort option.

I_MPI_HYDRA_NAMESERVER

Specify the nameserver.

Syntax

I_MPI_HYDRA_NAMESERVER=<arg>

Argument

<arg>

String parameter

<hostname>:<port>

Set the hostname and the port.

Description

Set this variable to specify the nameserver for your MPI application in the following format:

I_MPI_HYDRA_NAMESERVER = hostname:port

You can set the nameserver with the -nameserver option.

Optimization Notice: 

standard

For more complete information about compiler optimizations, see our Optimization Notice.
Select sticky button color: 
Orange (only for download buttons)