Intel Software Network articles feed http://software.intel.com/en-us/articles/intel-cluster-ready-kb/errors-diagnostics/ en-us Configuration of threshold values in the HPCC test module The HPC Challenge (HPCC) benchmark suite is a common method to gauge the performance of a cluster.  HPCC consists of seven benchmarks that measure a spectrum of system characteristics.  The hpcc module for Intel® Cluster Checker runs the HPCC benchmark suite on the cluster and reports ‘Succeeded' or ‘Failed' based on the outcome of the tests.

This article does not cover descriptions or definitions of the individual HPCC benchmarks.  For more information about the HPCC benchmarks, see http://icl.cs.utk.edu/hpcc/.

Module configuration affects the results of the hpcc tests

Whether the hpcc module succeeds or fails depends on the configuration of the module in the Intel® Cluster Checker configuration file.  The module will execute the HPCC benchmark over each network fabric that is configured in the hpcc module block in the input configuration file.  For each network fabric configured, the individual HPCC benchmark test can optionally configure a performance threshold value that must be achieved for a successful result.  If a performance threshold is not set, then success of a test is based solely on the benchmark running to completion.

Results when threshold values are configured

When threshold values are set, a benchmark must meet or exceed the configured performance value.  Depending on the benchmark, that may mean a result that is equal to or greater than the configured threshold OR a result that is equal to or less than the configured threshold.

hpcc module configuration tag

Measurement unit

Output characteristics

Passing result

bandwidth

GB/s

Higher is better

Equal or greater

dgemm

GFLOPS

Higher is better

Equal or greater

fft

GFLOPS

Higher is better

Equal or greater

hpl

TFLOPS

Higher is better

Equal or greater

latency

µs

Lower is better

Equal or less

ptrans

GB/s

Higher is better

Equal or greater

randomacess

GUPs

Higher is better

Equal or greater

stream

GB/s

Higher is better

Equal or greater


If one of the benchmarks does not meet the configured threshold value, the module will report a failing result identifying the network fabric and the individual failing benchmark(s).  For example, using the following configuration, the hpcc module reported the following failure:

<hpcc>

        <cc-path>/opt/intel/cce/11.0.069/</cc-path>

        <fabric>

              <bandwidth>0.003</bandwidth>

              <device>sock</device>

              <dgemm>5.76</dgemm>

              <fft>0.4</fft>

              <hpl>0.04</hpl>

              <latency>40</latency>

              <ptrans>0.10</ptrans>

              <randomaccess>0.008</randomaccess>

              <stream>1.4</stream>

        </fabric>

 

        <mkl-path>/opt/intel/cmkl/10.1.0.015/</mkl-path>

        <mpi-path>/opt/intel/impi/3.2/</mpi-path>

        <process-number>8</process-number>

        <thread-number>1</thread-number>

</hpcc>

 

HPC Challenge Benchmark (Intel(R) C++ Compiler, Intel(R) MPI

Library, Intel(R) Math Kernel Library), (hpcc)

Attention: this check may take a long time to complete......FAILED

subtest 'PTRANS, GB/s (device = sock)' failed

  - failing All hosts returned: '0.0817186'

 
The module reported a failure because the result of running the PTRANS test was 0.0817186 GB/s which did not meet or exceed the configured value of 0.10 GB/s.

What do failures to meet thresholds mean?

Many system characteristics affect the results of the HPCC benchmark suite, and a reported test failure does not necessarily indicate an under-performing or malfunctioning cluster.  Processor speeds, network characteristics, and memory architecture, for instance, all factor into the measured results.  Changes in the characteristics of any of those components or sub-systems can affect the outcome of the tests.  Therefore, a failure to meet a threshold may be the result of a value configured too high for the characteristics of a particular cluster.  The thresholds can be reset to levels that are more appropriate for the specific system to resolve the issue.

A cluster that has historically passed the hpcc module testing where threshold values were configured but begins to fail the test consistently may indicate a problem with one or more components in the system.  Make sure that Intel® Cluster Checker was the only application running on the system; other applications running concurrently are likely to impact the measured results of the benchmarks.  If failures to meet thresholds persist and there have been no changes to the hardware characteristics of the cluster, then there may be an issue causing the system to exhibit degraded performance that should be resolved.

Intermittent failures to meet threshold values may be the result of threshold levels that are set too high to account for the natural fluctuations in performance of the system.  For example, with the PTRANS configuration above, the threshold is set to 0.10.  A given cluster may exhibit performance that routinely yields 0.11 GB/s but has fluctuations ranging from 0.095 to 0.12 GB/s.  Any fluctuations that dip below the 0.10 threshold will be flagged as a failure.  Threshold values should be configured to account for some fluctuations in results, so a better threshold for this example may be 0.09 GB/s.

]]> http://software.intel.com/en-us/articles/configuration-of-threshold-values-in-the-hpcc-test-module Mon, 23 Nov 2009 12:40:48 -0800 http://software.intel.com/en-us/articles/configuration-of-threshold-values-in-the-hpcc-test-module#comments http://software.intel.com/en-us/articles/configuration-of-threshold-values-in-the-hpcc-test-module Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base How to Deal with file_tree issues in Intel® Cluster Checker 1.3 Note: this article describes behavior in Intel® Cluster Checker version 1.3 Update 2 or earlier.

The file_tree Intel® Cluster Checker test module verifies that a consistent set of files is present on all cluster nodes.  This consistency is a requirement of the Intel® Cluster Ready Specification.  However, some files are expected to contain data unique to each node, such as hostname and IP address.  While the file_tree test automatically excludes many common examples of such files from its consistency check, special cases exist that are not automatically handled (see the end of this article for a list). 

If the file_tree test identifies files that are not consistent across the cluster, you should manually determine whether the reported difference must be resolved by applying the following rules:

  • Does the reported file contain unique information to each node, such as a time stamp or network configuration?  If so, manually verify that other than than the node unique data, the file is identical on all nodes. 
  • Does the file name itself or the path to the file contain node unique information?  If so, are similar files present on all the nodes?
  • Is the file dynamically generated or modified, e.g., a log file, compiled from source on each node, or modified by the prelink utility, on each node?  If so, a time stamp or other node unique information may be embedded.

If you can determine why the file is different on each node and verify that the differences are not material, then you may ignore the reported errors.  If you are certifying a cluster design as Intel® Cluster Ready, you should include a brief description of why you believe the reported errors are immaterial with your submitted Intel® Cluster Checker output logs.  You should document the files you manually resolved with your cluster design so that other engineers and technicians at your company understand that they can ignore file_tree errors with these files, but should not ignore errors reported for other files. 

Forthcoming versions of Intel® Cluster Checker will have the capability to configure which files should be automatically excluded.

Files that are known to differ from node to node not automatically handled by Intel® Cluster Checker 1.3

  • /opt/mlnx-ofed/src/*
  • /opt/rocks/lib/graphviz/config
  • /opt/torque/lib64/xpbs*
  • /opt/torque/mom_logs/*
  • /usr/java/jdk1.6.0_14/jre/lib/servicetag/registration.xml
  • /usr/java/jdk1.6.0_14/register.html
  • /usr/java/jdk1.6.0_14/register_ja.html
  • /usr/java/jdk1.6.0_14/register_zh_CN.html
  • /usr/java/i386/jre1.6.0_12/lib/i386/client/classes.jsa
]]> http://software.intel.com/en-us/articles/how-to-deal-with-file_tree-issues-in-intel-cluster-checker-13 Tue, 29 Sep 2009 15:05:16 -0700 http://software.intel.com/en-us/articles/how-to-deal-with-file_tree-issues-in-intel-cluster-checker-13#comments http://software.intel.com/en-us/articles/how-to-deal-with-file_tree-issues-in-intel-cluster-checker-13 Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Troubleshooting the kernel_parameters check
Variability in the characteristics of compute nodes may occur for many reasons. Some clusters are designed with a few nodes that contain larger amounts of memory and/or storage. Clusters that have nodes replaced or added to the system after the initial deployment may include servers that have subtle hardware differences. These variances may be acceptable differences but can manifest in inconsistent kernel parameters.

The kernel_parameters module can be configured to ignore these differences. An <exclude> tag can be used for each kernel parameter that varies between the nodes.  The parameter value is the name of the kernel parameter that is allowed to vary between nodes.  By using this tag, the module will not report differences in the specified parameter across the cluster.

For example, a compute node has been replaced. The original compute nodes each contain a DVD drive, but the replacement node does not. The kernel_parameters module detects differences related to this variance in compute nodes.

Linux Kernel Runtime Parameters, (kernel_parameters)...................FAILED
subtest 'dev.cdrom.autoclose' failed
- failing hosts compute-00-00 - compute-00-06 returned: '1'
- failing host compute-00-07 returned: 'undefined value'
subtest 'dev.cdrom.autoeject' failed
- failing hosts compute-00-00 - compute-00-06 returned: '0'
- failing host compute-00-07 returned: 'undefined value'

If this difference is determined to be acceptable, the following configuration block will tell this check to ignore the inconsistencies reported above.

<kernel_parameters>
  <exclude>dev.cdrom.autoclose</exclude>
  <exclude>dev.cdrom.autoeject</exclude>
</kernel_parameters>

Excluding kernel parameters should be used with some discretion, however, as this can mask errors or differences that may not be desired.  The best option is to resolve inconsistencies and only exclude specific kernel parameters as a last resort.
]]> http://software.intel.com/en-us/articles/troubleshooting-the-kernel_parameters-check Wed, 05 Aug 2009 08:46:44 -0700 http://software.intel.com/en-us/articles/troubleshooting-the-kernel_parameters-check#comments http://software.intel.com/en-us/articles/troubleshooting-the-kernel_parameters-check Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Troubleshooting the uid_sync check uid_sync check verifies that the user and group data is consistent across the cluster.  A user or group present on some nodes but others will cause this check to fail.  Here is an example of how the check reports differences found between nodes:

User and Group Uniformity, (uid_sync)...............................................................FAILED
subtest 'gid = 1313' failed
- failing hosts node00001 - node00128 returned: 'clck:x:1313:'
- failing host node00129 returned: 'group does not exist'
subtest 'uid = 1313' failed
- failing hosts node00001 - node00128 returned: 'clck:x:1313:1313:::cluster checker:/home/clck:/bin/bash'
- failing host node00129 returned: 'user does not exist'

The ID of the user or group is displayed as the subtest description: subtest 'gid = 1313' failed.  The output under the heading identifies the nodes on which the user exists and provides the corresponding entry from the user or group database (e.g., /etc/group or /etc/password). 

This check most often fails when there are mismatches or conflicts with NIS or the local group or user databases.  Depending on the method you employ, you should ensure that NIS is properly configured on the inconsistent node(s) or that the /etc/group and /etc/password files are correctly propagated to the inconsistent node(s). 
]]> http://software.intel.com/en-us/articles/troubleshooting-the-uid_sync-check Fri, 31 Jul 2009 13:02:27 -0700 http://software.intel.com/en-us/articles/troubleshooting-the-uid_sync-check#comments http://software.intel.com/en-us/articles/troubleshooting-the-uid_sync-check Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Troubleshooting STREAM bandwidth issues Intel® Cluster Checker uses the STREAM benchmark to verify the memory performance of each cluster node.  STREAM consists of several individual benchmark tests; the memory_bandwidth_stream check only uses the 'Triad' benchmark.  When running the memory_bandwidth_stream check, you may encounter a diagnostic message similar to the following:

Single-node Memory Bandwidth (STREAM), (memory_bandwidth_stream).......FAILED
subtest 'Triad' failed
- failing host compute-00-03 returned: '15977.7 MB/s'
- failing host compute-00-01 returned: '15997.1 MB/s'
- failing host compute-00-00 returned: '16004.3 MB/s'
- failing host compute-00-02 returned: '16126.5 MB/s'

The failure reported above occurs because the measured bandwidth of the STREAM Triad benchmark on the nodes is less than the threshold value configured by the <bandwidth> tag in the configuration file:

<memory_bandwidth_stream>
  <bandwidth>27000</bandwidth>
</memory_bandwidth_stream>

The performance for the STREAM benchmark is sensitive to the characteristics of the processor(s), motherboard, and memory used in the system. Failures to achieve the configured performance threshold may not actually be a system fault. It’s possible that the threshold value is set or tuned for higher performing processors and memory. 

If all of the following statements are true, then it likely indicates the <bandwidth> performance threshold needs to be tuned for the performance of the individual cluster nodes:
  • The memory_bandwidth_stream check has always reported a failure to achieve the configured performance threshold on the system (no previous runs ever met the specified performance threshold)
  • All the nodes in a cluster fail to achieve the configured performance threshold
  • All the nodes achieve relatively similar performance levels
When setting the performance threshold, it is suggested to use a value that is 90% of the lowest measured performance. This will allow for some normal fluctuation in the results.

Memory performance may also be sensitive to where memory is located on the motherboard and the BIOS settings.  If all the nodes have the identical hardware but one node is consistently reporting lower performance, verify the same memory slots are populated and the BIOS memory options are set consistently.

If the memory performance is inconsistent from run to run, there may be other processes on the node consuming resources.  Verify that no other programs are running on the nodes before starting this check.

If the cluster has heterogeneous hardware, a single performance threshold may not be appropriate for all the nodes.  This knowledge base articledescribes how to configure Intel® Cluster Checker for heterogeneous clusters. 

]]> http://software.intel.com/en-us/articles/troubleshooting-stream-bandwidth-issues Fri, 22 May 2009 09:50:01 -0700 http://software.intel.com/en-us/articles/troubleshooting-stream-bandwidth-issues#comments http://software.intel.com/en-us/articles/troubleshooting-stream-bandwidth-issues Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Troubleshooting the dmidecode check The dmidecode check uses the dmidecode utility to retrieve system information from the SMBIOS tables and checks for consistency across the cluster. Each subtest examines a particular SMBIOS field. If any of the fields differ across the cluster nodes, then the subtest reports a failing result. There are many fields that are compared by the dmidecode check. Here are some examples of how the check reports differences found between nodes.

subtest 'BIOS Information (0x0000): Release Date' failed
- failing hosts compute-00-00 - compute-00-05, compute-00-07 returned: '03/09/2009'
- failing host compute-00-06 returned: '03/12/2009'

In this example, above, the BIOS release date is reported to be different on one of the compute nodes. This implies the BIOS versions are not the same in all the compute nodes.  Note that the dmidecode check does not assume that either of the BIOS release dates is correct, only that the field is not consistent across the cluster.

subtest 'Base Board Information (0x0200): Product Name' failed
- failing hosts compute-00-00 - compute-00-02, compute-00-07 returned: '0AJ123A'
- failing hosts compute-00-03 - compute-00-06 returned: '0AK456A'

The second example indicates that there are two types of motherboards used in the cluster.  This can result from the compute nodes using differing model motherboards or simply different generations of the same motherboard.

subtest 'Memory Device (0x1100): Asset Tag' failed
- failing hosts compute-00-00, compute-00-02 - compute-00-06 returned: '0108001D'
- failing hosts compute-00-01, compute-00-07 returned: '01082803'

The output from the third example shows varying results about the memory used in the nodes. Two of the nodes have a different memory configuration than the rest of the cluster. Using a mix of memory sizes, performance characteristics, etc. can adversely and unexpectedly impact overall cluster performance.

Issues reported by the dmidecode check do not necessarily mean a cluster will exhibit any issues or failures; however, consistency across cluster nodes is highly desirable.  If there are legitimate or desired differences between compute nodes, the Intel® Cluster Checker group capability and/or individual subtest exclusions can be configured to instruct the dmidecode check to ignore specific differences between nodes.

Specifying a group in the dmidecode configuration restricts the consistency check to defined subsets of nodes, and only differences between nodes within a group are reported as issues. For instance, the following configuration option causes the dmidecode check to compare all nodes defined in the mthrbd2 group to each other, but a node in the mthrbrd2 group would not be compared to a node that is not in this group.

<dmidecode>
  <group name=”mthrbd2”/>
</dmidecode>

For more information on the groups capability, see this article and the Intel® Cluster Checker User's Guide.

The other way to instruct the dmidecode check to ignore specific differences between nodes is to use the exclude configuration option.  This option ignores a specific field for all nodes but still checks all the remaining fields for consistency across the whole cluster.

<dmidecode>
  <exclude>Memory Device (0x1100): Asset Tag</exclude>
</dmidecode>

 
For more information on the exclude option, see the dmidecode section of the Intel® Cluster Checker Module Reference.

]]> http://software.intel.com/en-us/articles/troubleshooting-the-dmidecode-check Fri, 01 May 2009 11:48:39 -0700 http://software.intel.com/en-us/articles/troubleshooting-the-dmidecode-check#comments http://software.intel.com/en-us/articles/troubleshooting-the-dmidecode-check Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base