Intel Software Network articles Feed

Using Cluster SSH with Intel® Cluster Checker

Wed, 03 Feb 2010 22:00:00 -0800

Symptom

The process_check module of Intel Cluster Checker sometimes reports an issue related to Cluster SSH:

Stale Process Check, (process_check)................................................................FAILED
subtest 'Percent cpu usage is greater than 5%' failed
- failing host compute-00-14 returned: 'pid=8736 (cssh)'

Resolution

Configure Intel® Cluster Checker to ignore the CPU usage of Cluster SSH:

cssh

How to resolve dat_conf issues affecting Platform MPI

Wed, 27 Jan 2010 22:00:00 -0800

Symptom

The Intel® Cluster Checker test dat_conf verifies that the file /etc/dat.conf does not contain any entries known to affect some versions of Platform MPI* (formerly HP MPI*). The error may appear similar to the following example:

Valid dat.conf entries, (dat_conf).....................................FAILED
subtest 'Device: ehca0' failed
- failing hosts compute-00-00 - compute-00-05 returned: '/etc/dat.conf contains Interface Adapter entries known to trigger a fault in HP MPI version 2.2.5 or earlier. Please contact Hewlett-Packard for more information.'

Cause

Some versions of the Open Fabrics Enterprise Distribution (OFED) insert example entries into /etc/dat.conf. However, some older versions of HP MPI / Platform MPI assume that all /etc/dat.conf entries are valid and active.

Resolution

Remove or comment out all entries in /etc/dat.conf that are not valid.

These lines must be removed/commented out like so:

For example, the following two entries have been commented out by inserting the '#' character at the beginning of the line:

# OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
# OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 2" ""

Troubleshooting InfiniBand connection issues using OFED tools

Wed, 20 Jan 2010 22:00:00 -0800

The Open Fabrics Enterprise Distribution (OFED) package has many debugging tools available as part of the standard release. This article describes the use of those tools to troubleshoot the hardware and firmware of an InfiniBand fabric deployment.

First, the /sys/class sub-system should be checked to verify that the hardware is up and connected to the InfiniBand fabric. The following command will show the InfiniBand hardware modules recognized by the system:

ls /sys/class/infiniband

This example will use the module mlx4_0, which is typical for Mellanox ConnectX* series of adapters. If this, or a similar module, is not found, refer to the documentation that came with the OFED package on starting the OpenIB drivers.

Next, check the state of the InfiniBand port:

cat /sys/class/infiniband/mlx4_0/ports/1/state

This command should return “ACTIVE” if the hardware is initialized, and the subnet manager has found the port and added the port to the InfiniBand fabric. If this command returns “INIT” the hardware is initialized, but the subnet manager has not added the port to the fabric yet.

If necessary, start the subnet manager:

/etc/init.d/opensmd start

Once the port on the head node is in the “ACTIVE” state, check the state of the InfiniBand port on all the compute nodes to ensure that all of the Infiniband hardware on the compute nodes has been initialized, and the subnet manager has added all of the compute nodes ports on to the fabric. This article will use the pdsh tool to run the command on all nodes:

pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/state

All nodes should report “ACTIVE”. If a node reports it cannot find the file, ensure the OpenIB drivers is loaded on that node. Refer to the documentation that came with the OFED package on starting the OpenIB drivers.

Once all of the compute nodes report that port 1 is “ACTIVE”, verify the speed on each port using the following commands:

cat /sys/class/infiniband/mlx4_0/ports/1/rate
pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/rate

This is a good first check for a bad cable or connection. Each port should report the same speed. For example, the output for double data rate (DDR) InfiniBand cards will be similar to “20 Gb/sec (4X DDR)”.

Once the above basic checks are complete, more in-depth troubleshooting can be performed. The main OFED tool for troubleshooting performance and connection problems is ibdiagnet. This tool runs multiple tests, as specified on the command line during the run, to detect errors related to the subnet, bad packets, and bad states. These errors are some of the more common seen during initial setup of Infiniband fabrics.

Run ibdiagnet with the following command line options:

ibdiagnet –pc –c 1000

The output will be similar to this:

Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
Loading IBDM from: /usr/lib64/ibdm1.2
-W- A few ports of local device are up.
Since port-num was not specified (-p option), port 1 of device 1 will be
used as the local port.
-I- Discovering ... 17 nodes (1 Switches & 16 CA-s) discovered.

-I---------------------------------------------------
-I- Bad Guids/LIDs Info
-I---------------------------------------------------
-I- No bad Guids were found

-I---------------------------------------------------
-I- Links With Logical State = INIT
-I---------------------------------------------------
-I- No bad Links (with logical state = INIT) were found

-I---------------------------------------------------
-I- PM Counters Info
-I---------------------------------------------------
-I- No illegal PM counters values were found

-I---------------------------------------------------
-I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list)
-I---------------------------------------------------

-I---------------------------------------------------
-I- IPoIB Subnets Check
-I---------------------------------------------------
-I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
-W- No members found for group

-I---------------------------------------------------
-I- Bad Links Info
-I- Errors have occurred on the following links
(for errors details, look in log file /tmp/ibdiagnet.log):
-I---------------------------------------------------
Link at the end of direct route "1,5"
----------------------------------------------------------------
-I- Stages Status Report:
STAGE Errors Warnings
Bad GUIDs/LIDs Check 0 0
Link State Active Check 0 0
Performance Counters Report 0 0
Partitions Check 0 0
IPoIB Subnets Check 0 1
Link Errors Check 0 0

Please see /tmp/ibdiagnet.log for complete log
----------------------------------------------------------------
-I- Done. Run time was 9 seconds.

The warning “No members found for group” can safely be ignored.

In this example, a bad link was found: “Link at the end of direct route “1,5”.” "1,5" refers to the LID numbers associated with the individual ports. The following commands can be used to identify the LID numbers associated with each port:

cat /sys/class/infiniband/mlx4_0/ports/1/lid
pdsh –a /sys/class/infiniband/mlx4_0/ports/1/lid

This command generates a list of LIDs associated with nodes. In the output of the above command, locate the entries for 0x1 and 0x5. 0x1 is likely the head node. For errors of this type, reseat or replace the InfiniBand cable connecting the node corresponding to LID 0x5.

Finally, run ibdiagnet once more time to verify there are no errors, and then to check the error state of each port. Each test should pass.

ibdiagnet –pc –c 1000
ibcheckerrors.

Configuration of threshold values in the HPCC test module

Sun, 22 Nov 2009 22:00:00 -0800

The HPC Challenge (HPCC) benchmark suite is a common method to gauge the performance of a cluster. HPCC consists of seven benchmarks that measure a spectrum of system characteristics. The hpcc module for Intel® Cluster Checker runs the HPCC benchmark suite on the cluster and reports ‘Succeeded' or ‘Failed' based on the outcome of the tests.

This article does not cover descriptions or definitions of the individual HPCC benchmarks. For more information about the HPCC benchmarks, see http://icl.cs.utk.edu/hpcc/.

Module configuration affects the results of the hpcc tests

Whether the hpcc module succeeds or fails depends on the configuration of the module in the Intel® Cluster Checker configuration file. The module will execute the HPCC benchmark over each network fabric that is configured in the hpcc module block in the input configuration file. For each network fabric configured, the individual HPCC benchmark test can optionally configure a performance threshold value that must be achieved for a successful result. If a performance threshold is not set, then success of a test is based solely on the benchmark running to completion.

Results when threshold values are configured

When threshold values are set, a benchmark must meet or exceed the configured performance value. Depending on the benchmark, that may mean a result that is equal to or greater than the configured threshold OR a result that is equal to or less than the configured threshold.

hpcc module configuration tag	Measurement unit	Output characteristics	Passing result
bandwidth	GB/s	Higher is better	Equal or greater
dgemm	GFLOPS	Higher is better	Equal or greater
fft	GFLOPS	Higher is better	Equal or greater
hpl	TFLOPS	Higher is better	Equal or greater
latency	µs	Lower is better	Equal or less
ptrans	GB/s	Higher is better	Equal or greater
randomacess	GUPs	Higher is better	Equal or greater
stream	GB/s	Higher is better	Equal or greater

If one of the benchmarks does not meet the configured threshold value, the module will report a failing result identifying the network fabric and the individual failing benchmark(s). For example, using the following configuration, the hpcc module reported the following failure:

      /opt/intel/cce/11.0.069/



            0.003

            sock

            5.76

            0.4

            0.04

            40

            0.10

            0.008

            1.4



      /opt/intel/cmkl/10.1.0.015/

      /opt/intel/impi/3.2/

      8

      1

HPC Challenge Benchmark (Intel(R) C++ Compiler, Intel(R) MPI

Library, Intel(R) Math Kernel Library), (hpcc)

Attention: this check may take a long time to complete......FAILED

subtest 'PTRANS, GB/s (device = sock)' failed

- failing All hosts returned: '0.0817186'

The module reported a failure because the result of running the PTRANS test was 0.0817186 GB/s which did not meet or exceed the configured value of 0.10 GB/s.

What do failures to meet thresholds mean?

Many system characteristics affect the results of the HPCC benchmark suite, and a reported test failure does not necessarily indicate an under-performing or malfunctioning cluster. Processor speeds, network characteristics, and memory architecture, for instance, all factor into the measured results. Changes in the characteristics of any of those components or sub-systems can affect the outcome of the tests. Therefore, a failure to meet a threshold may be the result of a value configured too high for the characteristics of a particular cluster. The thresholds can be reset to levels that are more appropriate for the specific system to resolve the issue.

A cluster that has historically passed the hpcc module testing where threshold values were configured but begins to fail the test consistently may indicate a problem with one or more components in the system. Make sure that Intel® Cluster Checker was the only application running on the system; other applications running concurrently are likely to impact the measured results of the benchmarks. If failures to meet thresholds persist and there have been no changes to the hardware characteristics of the cluster, then there may be an issue causing the system to exhibit degraded performance that should be resolved.

Intermittent failures to meet threshold values may be the result of threshold levels that are set too high to account for the natural fluctuations in performance of the system. For example, with the PTRANS configuration above, the threshold is set to 0.10. A given cluster may exhibit performance that routinely yields 0.11 GB/s but has fluctuations ranging from 0.095 to 0.12 GB/s. Any fluctuations that dip below the 0.10 threshold will be flagged as a failure. Threshold values should be configured to account for some fluctuations in results, so a better threshold for this example may be 0.09 GB/s.

How to Deal with file_tree issues in Intel® Cluster Checker 1.3

Mon, 28 Sep 2009 22:00:00 -0700

Note: this article describes behavior in Intel® Cluster Checker version 1.3 Update 2 or earlier.

The file_tree Intel® Cluster Checker test module verifies that a consistent set of files is present on all cluster nodes. This consistency is a requirement of the Intel® Cluster Ready Specification. However, some files are expected to contain data unique to each node, such as hostname and IP address. While the file_tree test automatically excludes many common examples of such files from its consistency check, special cases exist that are not automatically handled (see the end of this article for a list).

If the file_tree test identifies files that are not consistent across the cluster, you should manually determine whether the reported difference must be resolved by applying the following rules:

Does the reported file contain unique information to each node, such as a time stamp or network configuration? If so, manually verify that other than than the node unique data, the file is identical on all nodes.
Does the file name itself or the path to the file contain node unique information? If so, are similar files present on all the nodes?
Is the file dynamically generated or modified, e.g., a log file, compiled from source on each node, or modified by the prelink utility, on each node? If so, a time stamp or other node unique information may be embedded.

If you can determine why the file is different on each node and verify that the differences are not material, then you may ignore the reported errors. If you are certifying a cluster design as Intel® Cluster Ready, you should include a brief description of why you believe the reported errors are immaterial with your submitted Intel® Cluster Checker output logs. You should document the files you manually resolved with your cluster design so that other engineers and technicians at your company understand that they can ignore file_tree errors with these files, but should not ignore errors reported for other files.

Forthcoming versions of Intel® Cluster Checker will have the capability to configure which files should be automatically excluded.

Files that are known to differ from node to node not automatically handled by Intel® Cluster Checker 1.3

/opt/mlnx-ofed/src/*
/opt/rocks/lib/graphviz/config
/opt/torque/lib64/xpbs*
/opt/torque/mom_logs/*
/usr/java/jdk1.6.0_14/jre/lib/servicetag/registration.xml
/usr/java/jdk1.6.0_14/register.html
/usr/java/jdk1.6.0_14/register_ja.html
/usr/java/jdk1.6.0_14/register_zh_CN.html
/usr/java/i386/jre1.6.0_12/lib/i386/client/classes.jsa

Troubleshooting the kernel_parameters check

Wed, 19 Aug 2009 22:00:00 -0700

When clusters consist of homogeneous compute nodes, Linux kernel parameters should be consistent across the cluster. However, when there are differences in the hardware characteristics of computes nodes, the kernel parameters may not be consistent. In some cases, this may be the result of a failure or issue that needs to be addressed to ensure proper operation of the cluster. A bad block of memory or inconsistent BIOS configuration may alter the kernel parameters of a compute node. These issues could potentially affect the overall performance of the cluster. Intel® Cluster Checker detects differences in the kernel parameters and reports differences between the nodes in the cluster.

Variability in the characteristics of compute nodes may occur for many reasons. Some clusters are designed with a few nodes that contain larger amounts of memory and/or storage. Clusters that have nodes replaced or added to the system after the initial deployment may include servers that have subtle hardware differences. These variances may be acceptable differences but can manifest in inconsistent kernel parameters.

The kernel_parameters module can be configured to ignore these differences. An tag can be used for each kernel parameter that varies between the nodes. The parameter value is the name of the kernel parameter that is allowed to vary between nodes. By using this tag, the module will not report differences in the specified parameter across the cluster.

For example, a compute node has been replaced. The original compute nodes each contain a DVD drive, but the replacement node does not. The kernel_parameters module detects differences related to this variance in compute nodes.

Linux Kernel Runtime Parameters, (kernel_parameters)...................FAILED
subtest 'dev.cdrom.autoclose' failed
- failing hosts compute-00-00 - compute-00-06 returned: '1'
- failing host compute-00-07 returned: 'undefined value'
subtest 'dev.cdrom.autoeject' failed
- failing hosts compute-00-00 - compute-00-06 returned: '0'
- failing host compute-00-07 returned: 'undefined value'

If this difference is determined to be acceptable, the following configuration block will tell this check to ignore the inconsistencies reported above.

dev.cdrom.autoclose
dev.cdrom.autoeject

Excluding kernel parameters should be used with some discretion, however, as this can mask errors or differences that may not be desired. The best option is to resolve inconsistencies and only exclude specific kernel parameters as a last resort.

Troubleshooting STREAM bandwidth issues

Wed, 19 Aug 2009 22:00:00 -0700

Intel® Cluster Checker uses the STREAM benchmark to verify the memory performance of each cluster node. STREAM consists of several individual benchmark tests; the memory_bandwidth_stream check only uses the 'Triad' benchmark. When running the memory_bandwidth_stream check, you may encounter a diagnostic message similar to the following:

Single-node Memory Bandwidth (STREAM), (memory_bandwidth_stream).......FAILED
subtest 'Triad' failed
- failing host compute-00-03 returned: '15977.7 MB/s'
- failing host compute-00-01 returned: '15997.1 MB/s'
- failing host compute-00-00 returned: '16004.3 MB/s'
- failing host compute-00-02 returned: '16126.5 MB/s'

The failure reported above occurs because the measured bandwidth of the STREAM Triad benchmark on the nodes is less than the threshold value configured by the tag in the configuration file:

27000

The performance for the STREAM benchmark is sensitive to the characteristics of the processor(s), motherboard, and memory used in the system. Failures to achieve the configured performance threshold may not actually be a system fault. It’s possible that the threshold value is set or tuned for higher performing processors and memory.

If all of the following statements are true, then it likely indicates the performance threshold needs to be tuned for the performance of the individual cluster nodes:

The memory_bandwidth_stream check has always reported a failure to achieve the configured performance threshold on the system (no previous runs ever met the specified performance threshold)
All the nodes in a cluster fail to achieve the configured performance threshold
All the nodes achieve relatively similar performance levels

When setting the performance threshold, it is suggested to use a value that is 90% of the lowest measured performance. This will allow for some normal fluctuation in the results.

Memory performance may also be sensitive to where memory is located on the motherboard and the BIOS settings. If all the nodes have the identical hardware but one node is consistently reporting lower performance, verify the same memory slots are populated and the BIOS memory options are set consistently.

If the memory performance is inconsistent from run to run, there may be other processes on the node consuming resources. Verify that no other programs are running on the nodes before starting this check.

If the cluster has heterogeneous hardware, a single performance threshold may not be appropriate for all the nodes. This knowledge base articledescribes how to configure Intel® Cluster Checker for heterogeneous clusters.

Troubleshooting the dmidecode check

Wed, 19 Aug 2009 22:00:00 -0700

The dmidecode check uses the dmidecode utility to retrieve system information from the SMBIOS tables and checks for consistency across the cluster. Each subtest examines a particular SMBIOS field. If any of the fields differ across the cluster nodes, then the subtest reports a failing result. There are many fields that are compared by the dmidecode check. Here are some examples of how the check reports differences found between nodes.

subtest 'BIOS Information (0x0000): Release Date' failed
- failing hosts compute-00-00 - compute-00-05, compute-00-07 returned: '03/09/2009'
- failing host compute-00-06 returned: '03/12/2009'

In this example, above, the BIOS release date is reported to be different on one of the compute nodes. This implies the BIOS versions are not the same in all the compute nodes. Note that the dmidecode check does not assume that either of the BIOS release dates is correct, only that the field is not consistent across the cluster.

subtest 'Base Board Information (0x0200): Product Name' failed
- failing hosts compute-00-00 - compute-00-02, compute-00-07 returned: '0AJ123A'
- failing hosts compute-00-03 - compute-00-06 returned: '0AK456A'

The second example indicates that there are two types of motherboards used in the cluster. This can result from the compute nodes using differing model motherboards or simply different generations of the same motherboard.

subtest 'Memory Device (0x1100): Asset Tag' failed
- failing hosts compute-00-00, compute-00-02 - compute-00-06 returned: '0108001D'
- failing hosts compute-00-01, compute-00-07 returned: '01082803'

The output from the third example shows varying results about the memory used in the nodes. Two of the nodes have a different memory configuration than the rest of the cluster. Using a mix of memory sizes, performance characteristics, etc. can adversely and unexpectedly impact overall cluster performance.

Issues reported by the dmidecode check do not necessarily mean a cluster will exhibit any issues or failures; however, consistency across cluster nodes is highly desirable. If there are legitimate or desired differences between compute nodes, the Intel® Cluster Checker group capability and/or individual subtest exclusions can be configured to instruct the dmidecode check to ignore specific differences between nodes.

Specifying a group in the dmidecode configuration restricts the consistency check to defined subsets of nodes, and only differences between nodes within a group are reported as issues. For instance, the following configuration option causes the dmidecode check to compare all nodes defined in the mthrbd2 group to each other, but a node in the mthrbrd2 group would not be compared to a node that is not in this group.

For more information on the groups capability, see this article and the Intel® Cluster Checker User's Guide.

The other way to instruct the dmidecode check to ignore specific differences between nodes is to use the exclude configuration option. This option ignores a specific field for all nodes but still checks all the remaining fields for consistency across the whole cluster.

Memory Device (0x1100): Asset Tag

For more information on the exclude option, see the dmidecode section of the Intel® Cluster Checker Module Reference.

Troubleshooting the uid_sync check

Thu, 30 Jul 2009 22:00:00 -0700

The uid_sync check verifies that the user and group data is consistent across the cluster. A user or group present on some nodes but others will cause this check to fail. Here is an example of how the check reports differences found between nodes:

User and Group Uniformity, (uid_sync)...............................................................FAILED
subtest 'gid = 1313' failed
- failing hosts node00001 - node00128 returned: 'clck:x:1313:'
- failing host node00129 returned: 'group does not exist'
subtest 'uid = 1313' failed
- failing hosts node00001 - node00128 returned: 'clck:x:1313:1313:::cluster checker:/home/clck:/bin/bash'
- failing host node00129 returned: 'user does not exist'

The ID of the user or group is displayed as the subtest description: subtest 'gid = 1313' failed. The output under the heading identifies the nodes on which the user exists and provides the corresponding entry from the user or group database (e.g., /etc/group or /etc/password).

This check most often fails when there are mismatches or conflicts with NIS or the local group or user databases. Depending on the method you employ, you should ensure that NIS is properly configured on the inconsistent node(s) or that the /etc/group and /etc/password files are correctly propagated to the inconsistent node(s).