Intel Software Network articles Feed

Intel Cluster Checker 1.8 Execution Time

Sat, 08 Oct 2011 20:00:00 -0700

This article shows reference times for a full execution on different node counts, similar to the one required for Intel Cluster Ready architecture compliance. It is expected that a simple interpolation of the provided values will help to roughly estimate execution time during troubleshooting.

The executed command line is shown below; it uses an almost empty configuration file, only having the node list file location. This configuration selects default values for all checks. MPI-related tests and benchmarks are executed over the best available messaging fabric.

$ cluster-check config.xml --certification

On a reference system the wall time was:

64 nodes: 2097 seconds (about 0.58 hours)
128 nodes: 2825 seconds (about 0.78 hours)
256 nodes: 5655 seconds (about 1.57 hours)
320 nodes: 6915 seconds (about 1.92 hours)

This complete check covers different tests: hardware and software uniformity, health and functional wellness behavior, individual node and cluster wide performance, etc. Check the product documentation to find out how to run a different set of test modules; in order to have a lighter or deeper coverage with a reduced or increased execution time, respectively.

If your system has hundred of nodes check this article for more details.
The details of the system used to gather reference data can be found here.

Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

Understanding How Intel Speedstep® Technology and Intel® Hyper-Threading Technology Are Verified by Intel® Cluster Checker

Wed, 20 Jan 2010 22:00:00 -0800

Intel Speedstep® Technology and Intel® Hyper-Threading Technology settings can impact the overall performance of a cluster. Intel® Cluster Checker verifies these settings in the speedstep and core_count checks, respectively.

For the Intel® Xeon® processor 5500 series and later, by default, the core_count check verifies that Intel® Hyper-Threading Technology is enabled. On earlier processors, the default core_count behavior is to verify that Intel® Hyper-Threading Technology is disabled. For any processor, the state to be verified can be configured. For example, to always verify that Intel® Hyper-Threading Technology is enabled:

on

By default, Intel® Cluster Checker verifies that Intel Speedstep® Technology is disabled. The speedstep check can be configured to verify either state. For example, to verify that Intel Speedstep® Techology is enabled:

on

Note: Refer to the documentation provided by your hardware vendor for more information on how to enable or disable Intel Speedstep® Technology or Intel® Hyper-Threading Technology.

Understanding how Intel® MPI Library is verified by Intel® Cluster Checker

Wed, 13 Jan 2010 22:00:00 -0800

Intel® Cluster Checker contains 4 checks that verify the basic functionality of Intel® MPI Library using a MPI Hello World program. The names of the checks are: intel_mpi, intel_mpi_internode, intel_mpi_rt, and intel_mpi_rt_internode. The differences between these checks are summarized in the following table:

Check	Verifies each node individually	Verifies the whole cluster	Intel® MPI Library Runtime Edition	Intel® MPI Library
intel_mpi	X			X
intel_mpi_internode		X		X
intel_mpi_rt	X		X	X
intel_mpi_rt_internode		X	X	X

The runtime (rt) checks, intel_mpi_rt and intel_mpi_rt_internode, use a pre-compiled MPI Hello World binary. Hence, the runtime checks may be used to verify Intel® MPI Library Runtime Edition or Intel MPI Library. The intel_mpi and intel_mpi_internode checks compile MPI Hello World from source as one of their verification steps. Hence, these checks require the MPI compiler wrappers (e.g., mpicc) that are only part of the Intel® MPI Library.

The internode checks run a single MPI Hello World job with at least one process on each cluster node to verify internode network communication. The non-internode checks start multiple MPI jobs, each one limited to a single node, to verify the basic functionality of MPI and related system components.

By default, Intel® Cluster Checker runs the intel_mpi_rt and intel_mpi_rt_internode checks. The intel_mpi and intel_mpi_internode checks are recommended to also be included if the cluster has Intel® MPI Library installed. Refer to the Intel® Cluster Checker documentation for more information on how to include and configure these checks.

Configuration of threshold values in the HPCC test module

Sun, 22 Nov 2009 22:00:00 -0800

The HPC Challenge (HPCC) benchmark suite is a common method to gauge the performance of a cluster. HPCC consists of seven benchmarks that measure a spectrum of system characteristics. The hpcc module for Intel® Cluster Checker runs the HPCC benchmark suite on the cluster and reports ‘Succeeded' or ‘Failed' based on the outcome of the tests.

This article does not cover descriptions or definitions of the individual HPCC benchmarks. For more information about the HPCC benchmarks, see http://icl.cs.utk.edu/hpcc/.

Module configuration affects the results of the hpcc tests

Whether the hpcc module succeeds or fails depends on the configuration of the module in the Intel® Cluster Checker configuration file. The module will execute the HPCC benchmark over each network fabric that is configured in the hpcc module block in the input configuration file. For each network fabric configured, the individual HPCC benchmark test can optionally configure a performance threshold value that must be achieved for a successful result. If a performance threshold is not set, then success of a test is based solely on the benchmark running to completion.

Results when threshold values are configured

When threshold values are set, a benchmark must meet or exceed the configured performance value. Depending on the benchmark, that may mean a result that is equal to or greater than the configured threshold OR a result that is equal to or less than the configured threshold.

hpcc module configuration tag	Measurement unit	Output characteristics	Passing result
bandwidth	GB/s	Higher is better	Equal or greater
dgemm	GFLOPS	Higher is better	Equal or greater
fft	GFLOPS	Higher is better	Equal or greater
hpl	TFLOPS	Higher is better	Equal or greater
latency	µs	Lower is better	Equal or less
ptrans	GB/s	Higher is better	Equal or greater
randomacess	GUPs	Higher is better	Equal or greater
stream	GB/s	Higher is better	Equal or greater

If one of the benchmarks does not meet the configured threshold value, the module will report a failing result identifying the network fabric and the individual failing benchmark(s). For example, using the following configuration, the hpcc module reported the following failure:

      /opt/intel/cce/11.0.069/



            0.003

            sock

            5.76

            0.4

            0.04

            40

            0.10

            0.008

            1.4



      /opt/intel/cmkl/10.1.0.015/

      /opt/intel/impi/3.2/

      8

      1

HPC Challenge Benchmark (Intel(R) C++ Compiler, Intel(R) MPI

Library, Intel(R) Math Kernel Library), (hpcc)

Attention: this check may take a long time to complete......FAILED

subtest 'PTRANS, GB/s (device = sock)' failed

- failing All hosts returned: '0.0817186'

The module reported a failure because the result of running the PTRANS test was 0.0817186 GB/s which did not meet or exceed the configured value of 0.10 GB/s.

What do failures to meet thresholds mean?

Many system characteristics affect the results of the HPCC benchmark suite, and a reported test failure does not necessarily indicate an under-performing or malfunctioning cluster. Processor speeds, network characteristics, and memory architecture, for instance, all factor into the measured results. Changes in the characteristics of any of those components or sub-systems can affect the outcome of the tests. Therefore, a failure to meet a threshold may be the result of a value configured too high for the characteristics of a particular cluster. The thresholds can be reset to levels that are more appropriate for the specific system to resolve the issue.

A cluster that has historically passed the hpcc module testing where threshold values were configured but begins to fail the test consistently may indicate a problem with one or more components in the system. Make sure that Intel® Cluster Checker was the only application running on the system; other applications running concurrently are likely to impact the measured results of the benchmarks. If failures to meet thresholds persist and there have been no changes to the hardware characteristics of the cluster, then there may be an issue causing the system to exhibit degraded performance that should be resolved.

Intermittent failures to meet threshold values may be the result of threshold levels that are set too high to account for the natural fluctuations in performance of the system. For example, with the PTRANS configuration above, the threshold is set to 0.10. A given cluster may exhibit performance that routinely yields 0.11 GB/s but has fluctuations ranging from 0.095 to 0.12 GB/s. Any fluctuations that dip below the 0.10 threshold will be flagged as a failure. Threshold values should be configured to account for some fluctuations in results, so a better threshold for this example may be 0.09 GB/s.

Setting the Processor Frequency for Intel® Cluster Checker Performance Tests

Wed, 19 Aug 2009 22:00:00 -0700

Note: This article is only applicable to clusters on which Enhanced Intel Speedstep® Technology is enabled.

Enhanced Intel Speedstep Technology allows systems to achieve very high performance while also conserving power. However, clusters using Enhanced Intel Speedstep Technology may experience intermittent performance failures when checked with Intel Cluster Checker. Due to the ability of the cluster nodes to dynamically adjust processor frequency, performance results may be lower than expected. This issue affects nearly all the checks that verify cluster performance, including mflops_intel_mkl, hpcc, and imb_pingpong_intel_mpi.

The recommended method to resolve this issue, without disabling Enhanced Intel Speedstep Technology, is to manually set the processor frequency on all cluster nodes. Processor frequency should be statically configured until Intel Cluster Checker execution is complete. Most versions of Linux provide utilities to control dynamic processor frequency. Two common utilities are cpuspeed and powersaved.

Follow the following steps:

Manually set all processors in the cluster to maximum frequency. This is normally done executing the CPU frequency scaling utility with the correct options. The command must be executed on all nodes using the cluster-wide execution capability required by Intel Cluster Ready.
Run Intel Cluster Checker.
Once Intel Cluster Checker has completed successfully, restore dynamic CPU frequency scaling on all nodes.

For example, to set maximum processor frequency on a cluster using cpuspeed and pdsh:

pdsh –a "killall -SIGUSR1 cpuspeed"

To return the processor frequency scaling to automatic mode:

pdsh –a "killall -SIGHUP cpuspeed"

With the introduction of Intel® Turbo Boost Technology in Intel Core™ i7 processors, processor frequency can exceed the base operating frequency. Additional steps may be required to force the processors to run at nominal base frequency, instead of maximum frequency.

More information on Enhanced Intel Speedstep Technology can be found at
http://software.intel.com/en-us/articles/enhanced-intel-speedstepr-technology-and-demand-based-switching-on-linux/

The following article discusses how to implement processor frequency control for high performance computing clusters:
http://software.intel.com/en-us/articles/using-enhanced-intel-speedstep-features-in-hpc-clusters/

Using Intel® Cluster Checker Generic Tests

Wed, 19 Aug 2009 22:00:00 -0700

Background

Intel® Cluster Checker includes almost 80 different modules that test the functionality, consistency, and performance of Intel® Cluster Ready clusters. However, your cluster may have capabilities that need to be checked that are not currently included the set of provided modules.

The checking capabilities of Intel Cluster Checker can be extended by either creating a custom module using the Intel Cluster Checker API or by using the built-in generic module capabilities. This article discusses the use of generic modules. More information on developing custom modules can be found in the Intel Cluster Checker Developers Guide.

Intel Cluster Checker includes two modules that provide a simple and easy way to extend the checking capabilities: generic_correctness and generic_uniformity.

generic_correctness: execute a command on all nodes and evaluate that the output exactly matches an expected result.
generic_uniformity: execute a command on all nodes and confirm that the output is identical on all nodes.

These modules can be used to implement checks specific to a cluster or site, and are especially useful for temporary, or limited time, checks.

Modifying Nodes on which Generic Tests are Executed

By default, generic modules execute on all nodes. This can be modified using the standard settings available to all modules: , , and , and . In order to run tests on specific nodes, you will need to assign those nodes to a group. More information on these settings is contained in the Intel Cluster Checker Users Guide.

Note: It is not currently possible to create two generic tests in the same generic module with different execution parameters. For example, you cannot have one generic_correctness test run on the head node only and a second generic_correctness test run on all nodes.

Executing Generic Tests as Root

Due to the potential damage that a poorly designed could cause, running generic tests as the system superuser is disabled by default. It is important to understand that any command can be executed by the generic tests, including those that permanently modify the filesystem or security parameters.

In order to run generic modules as a superuser, the module must be configured to include the tag. For example, in order to test that root authorized keys are identical on all nodes:

md5sum /root/.ssh/authorized_keys

Examples

The following examples are provided to demonstrate how the generic modules may be used. These tests are only examples and should not be considered the best or only means to perform these tests.

Confirm that there no errors exist in the MCE logs:

mcelog; echo $?
0

Confirm that the second Ethernet port is disabled on all compute nodes:

off

/sbin/ifconfig eth1 | grep -c UP
0

Confirm that the ahci.ko module is configured identically in modprobe.conf:

grep "options[[:space:]]*ahci" /etc/modprobe.conf | sed 's/[[:space:]]*//g'

Confirm that the DHCP server daemon is running on the head node:

off

/etc/init.d/dhcpd status | grep –c running
1

New Features in Intel® Cluster Checker 1.3 Update 2

Wed, 19 Aug 2009 22:00:00 -0700

Intel® Cluster Checker 1.3 Update 2 is an update release of the Intel® Cluster Checker. This release includes several new and updated features compared to Intel® Cluster Checker 1.3 (see product documentation for more details).

The highlights of this release include:

Optimized the HPCC* benchmark for the Intel(R) Xeon(TM) Processor 5500 series.
Identification of outlier nodes with respect to performance.
Option to automatically retry failing performance checks. The performance check is repeated until it passes or the specified number of attempts is exceeded.
Additional options to allow exceptions to the hardware homogeneity checks. Also, additional fields that are known to vary from node to node are automatically excluded (e.g., the apic_id field in /proc/cpuinfo).
Support for hierarchical configuration files using XInclude*.
Extended support for groups. Logical combinations of node groups may be defined using ‘AND’ and ‘OR’. The matrix (all-to-all) check class can now take advantage of groups.
Extended operating system support for the --sales-report command line option.