Intel Software Network articles Feed

The mflops_intel_mkl test module hangs during execution on Scyld Clusterware 5.4

Wed, 03 Feb 2010 22:00:00 -0800

Symptom

Intel® Cluster Checker hangs during the execution of the mflops_intel_mkl test module on clusters running Penguin Computing* Scyld Clusterware* 5.4.

In addition, inactive or zombie processes named dgemm_mflops may be present on the nodes. The dgemm_mflops binary is a DGEMM benchmark optimized with the Intel® Math Kernel library. It is packaged with Intel® Cluster Checker.

Debug output provides no other information.

Cause

The root cause of this error is undetermined. It appears to be an incompatibility between the binary version of dgemm_mflops included with Intel® Cluster Checker and Scyld ClusterWare 5.4. It only occurs when executing over non-interactive SSH.

Resolution

Configure Intel® Cluster Checker to build dgemm_mflops from source rather than using the prebuilt binary. This is accomplished using the configuration tag.

The Intel Math Kernel Library must be in the linker path or the option must also be set. The option can also be set using the global configuration capability. The GNU C Compiler (gcc) must also be present, so it is recommended to add the corresponding test module as a dependency.

The following is an example of the updated test module configuration to work around the issue:

gcc

/opt/intel/cmkl/10.1

Troubleshooting the dmidecode check: Wake-up Type

Wed, 16 Dec 2009 22:00:00 -0800

Symptom

The dmidecode module of Intel® Cluster Checker reports inconsistencies in the “Wake-Up Type” value. The error may appear similar to the following example:

SMBIOS/DMI Uniformity,(dmidecode).............................................FAILED
subtest 'System Information (0x0002): Wake-up Type' failed
- failing hosts node-1, node-2 returned: 'AC Power Restored'
- failing host node-0 returned: 'Power Switch'

Modifying the Power Restore Policy or Restore on AC Power Loss setting, using either the IPMI/BIOS configuration interface or the Save and Restore System Configuration utility (syscfg), has no effect on this error.

Cause

The “Wake-Up Type” value does not represent the current setting for power restore or power-on response. It describes how the system was powered-on after the last power off, regardless of restore settings. This value may change after each boot, depending on what method was used to power-on the individual node.

Resolution

This value stores dynamic data and compute nodes are not expected to have identical data for this value. It can be ignored for the purposes of Intel® Cluster Ready certification and verification.

Exclude the value by adding an exclude element to the dmidecode block in the Intel® Cluster Checker configuration file. For example:

System Information (0x0002): Wake-up Type

In Intel® Cluster Checker 1.4, the exclude element has been enhanced to support regular expression matching. In this case, the following option is sufficient:

Wake-up Type

For more information on troubleshooting the dmidecode check, see this article.

Random fabric errors on Red Hat Enterprise Linux 5.4

Tue, 01 Dec 2009 22:00:00 -0800

Problem:

The Intel® MPI Library fails intermittently when run over the RDSSM or RDMA devices. Approximately 5-10% of runs fail on RHEL (Red Hat Enterprise Linux) 5.4, but this problem does not occur on earlier versions of RHEL.

When reviewing the debug output, the following error is seen during the Intel MPI Library operations:

setup_listener Cannot assign requested address

Environment:

Red Hat Enterprise Linux 5.4 only

Root Cause:

This error occurs with the Intel MPI Library and the version of OFED (Open Fabrics Enterprise Distribution) included with RHEL 5.4. There is a potential port space conflict with RDS (reliable datagram sockets) and when this port space conflict occurs, uDAPL does not resolve it correctly.

By default, the Intel MPI Library uses its process ID to define its port number. In RHEL 5.4, the process ID can occasionally match a port number that the RDS driver has already allocated, which creates a port space conflict. Currently, uDAPL will reply with the wrong return code to the Intel MPI Library and communication will fail.

Resolution:

As a temporary workaround, set the following environment variable on all nodes:

$ export I_MPI_RDMA_CREATE_CONN_QUAL = 0

After setting this variable, the Intel MPI Library will not define its port number from its process ID.

This error is resolved in DAPL 2.0.25, to be included in Open Fabrics Enterprise Distribution (OFED) 1.5. Status of the resolution can be found in the latest OFED release notes.

How to Deal with file_tree issues in Intel® Cluster Checker 1.3

Mon, 28 Sep 2009 22:00:00 -0700

Note: this article describes behavior in Intel® Cluster Checker version 1.3 Update 2 or earlier.

The file_tree Intel® Cluster Checker test module verifies that a consistent set of files is present on all cluster nodes. This consistency is a requirement of the Intel® Cluster Ready Specification. However, some files are expected to contain data unique to each node, such as hostname and IP address. While the file_tree test automatically excludes many common examples of such files from its consistency check, special cases exist that are not automatically handled (see the end of this article for a list).

If the file_tree test identifies files that are not consistent across the cluster, you should manually determine whether the reported difference must be resolved by applying the following rules:

Does the reported file contain unique information to each node, such as a time stamp or network configuration? If so, manually verify that other than than the node unique data, the file is identical on all nodes.
Does the file name itself or the path to the file contain node unique information? If so, are similar files present on all the nodes?
Is the file dynamically generated or modified, e.g., a log file, compiled from source on each node, or modified by the prelink utility, on each node? If so, a time stamp or other node unique information may be embedded.

If you can determine why the file is different on each node and verify that the differences are not material, then you may ignore the reported errors. If you are certifying a cluster design as Intel® Cluster Ready, you should include a brief description of why you believe the reported errors are immaterial with your submitted Intel® Cluster Checker output logs. You should document the files you manually resolved with your cluster design so that other engineers and technicians at your company understand that they can ignore file_tree errors with these files, but should not ignore errors reported for other files.

Forthcoming versions of Intel® Cluster Checker will have the capability to configure which files should be automatically excluded.

Files that are known to differ from node to node not automatically handled by Intel® Cluster Checker 1.3

/opt/mlnx-ofed/src/*
/opt/rocks/lib/graphviz/config
/opt/torque/lib64/xpbs*
/opt/torque/mom_logs/*
/usr/java/jdk1.6.0_14/jre/lib/servicetag/registration.xml
/usr/java/jdk1.6.0_14/register.html
/usr/java/jdk1.6.0_14/register_ja.html
/usr/java/jdk1.6.0_14/register_zh_CN.html
/usr/java/i386/jre1.6.0_12/lib/i386/client/classes.jsa