Intel Software Network articles Feed http://software.intel.com/en-us/articles/intel-cluster-checker-kb/type/tips-and-techniques/ en-us How to configure the clean_ipc test module Background

The Intel® Cluster Checker clean_ipc test module is used to verify that facilities available in the X/Open System Interface (XSI) Inter-Process Communication (IPC) subsystem are not in use. This is a “housekeeping” check used to evaluate that unnecessary IPC facilities are not active and that the cluster is ready for use.

By default, if the ipcs command reports that any shared memory segments, semaphore arrays, or message queue facilities are in use, the test module will report an error. An example of this error looks like this:

System V Interprocess Communication, (clean_ipc).......................FAILED
subtest 'Semaphore Arrays' failed
- failing All hosts returned: 'found 5 entries, target was 0'

Several common software applications use the IPC subsystem, including Apache web server. In addition, it’s common for the head node to use more facilities than compute nodes. This can lead to issues on clusters where the head node is also a compute node. The module does not normally execute on dedicated head nodes.

The clean_ipc test module can be configured to look for a specific number of entries, but it cannot be configured to look for a minimum or maximum number of entries, nor can any facility tests be excluded. This makes it unsuitable for use if any processes use the IPC subsystem in a non-deterministic manner.

Configuration

If the exact number of entries for each subtest are known and will not change during cluster operation, these can be defined in the configuration file. You can determine the current number in use with the ipcs command.
The following is example output from the command ipcs -au | grep 'allocated':

segments allocated 12
pages allocated 960
allocated semaphores = 5
allocated queues = 0

This information can then be used to update the configuration file. For the example output above, the new configuration file should look like this:

<clean_ipc>
  <shm_segments>12</shm_segments>
  <sem_arrays>5</sem_arrays>
  <msg_queues>0</msg_queues>
</clean_ipc>

Any facility not defined in the configuration file will default to zero (0).

The clean_ipc test module should be excluded from the configuration file if the cluster is using any processes for which the number of IPC facilities in use cannot be pre-determined. This includes processes that may use a fixed number of IPC facilities when started, but allocate additional facilities or release facilities during execution.

For clusters where the head node is also a compute node, the mixed head node and compute nodes may report different values for IPC facilities in use. In this case, either disable this test module on the head node using the check_head tag or use Intel® Cluster Checker's group capability to set separate configuration values for the head node and compute nodes.
]]> http://software.intel.com/en-us/articles/how-to-configure-the-clean_ipc-test-module/ Wed, 24 Feb 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/how-to-configure-the-clean_ipc-test-module/#comments http://software.intel.com/en-us/articles/how-to-configure-the-clean_ipc-test-module/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base How to configure the hdparm test module for diskless, multi-drive, or external storage nodes hdparm test module is to evaluate the primary disk drive on each node. By design, it’s only capable of testing performance on a single disk drive per node.

Configuration of this test module is dependent on the type of storage devices used for each node. Many clusters will implement multiple storage devices; for example, all nodes may include a local hard drive and also access shared, external storage. In this case, hdparm should be configured to test the local drive.  Other cases and their configurations are described below.

Diskless clusters

This test module can be excluded for diskless clusters. If the head node contains a disk drive, Intel® Cluster Checker can be configured to evaluate just that device. For example:

<hdparm>
  <check_compute>0</check_compute>
</hdparm>

Multi-drive nodes

hdparm is capable of evaluating a single drive. For nodes with multiple hard drives, it’s beneficial to evaluate the disk drive with the most activity or impact on application performance. This is most likely the hard drive containing the /tmp directory or partition, or the drive used for application scratch space.  For example, if the primary application scratch space is located on the device /dev/sdb:

<hdparm>
  <device>/dev/sdb</device>
</hdparm>

External storage system

If shared, external storage is the only storage medium used for nodes, the module should be excluded. The hdparm test module is not capable of evaluating devices that do not support kernel-level ioctl() operations. If the external storage system appears to the Linux kernel as an ATA, SATA, or SCSI local storage device, hdparm may support it.
]]> http://software.intel.com/en-us/articles/how-to-configure-the-hdparm-test-module-for-diskless-multi-drive-or-external-storage-nodes/ Wed, 24 Feb 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/how-to-configure-the-hdparm-test-module-for-diskless-multi-drive-or-external-storage-nodes/#comments http://software.intel.com/en-us/articles/how-to-configure-the-hdparm-test-module-for-diskless-multi-drive-or-external-storage-nodes/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Using Cluster SSH with Intel® Cluster Checker Symptom

The process_check module of Intel Cluster Checker sometimes reports an issue related to Cluster SSH:

Stale Process Check, (process_check)................................................................FAILED
subtest 'Percent cpu usage is greater than 5%' failed
- failing host compute-00-14 returned: 'pid=8736 (cssh)'

Resolution

Configure Intel® Cluster Checker to ignore the CPU usage of Cluster SSH:

<process_check>
  <exclude>cssh</exclude>
</process_check>

]]> http://software.intel.com/en-us/articles/using-cluster-ssh-with-intel-cluster-checker/ Wed, 03 Feb 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/using-cluster-ssh-with-intel-cluster-checker/#comments http://software.intel.com/en-us/articles/using-cluster-ssh-with-intel-cluster-checker/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base How to resolve dat_conf issues affecting Platform MPI Symptom

The Intel® Cluster Checker test dat_conf verifies that the file /etc/dat.conf does not contain any entries known to affect some versions of Platform MPI* (formerly HP MPI*).  The error may appear similar to the following example:

Valid dat.conf entries, (dat_conf).....................................FAILED
subtest 'Device: ehca0' failed
- failing hosts compute-00-00 - compute-00-05 returned: '/etc/dat.conf contains Interface Adapter entries known to trigger a fault in HP MPI version 2.2.5 or earlier. Please contact Hewlett-Packard for more information.'

Cause

Some versions of the Open Fabrics Enterprise Distribution (OFED) insert example entries into /etc/dat.conf.  However, some older versions of HP MPI / Platform MPI assume that all /etc/dat.conf entries are valid and active.  

Resolution

Remove or comment out all entries in /etc/dat.conf that are not valid.

These lines must be removed/commented out like so:

For example, the following two entries have been commented out by inserting the '#' character at the beginning of the line:

# OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
# OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 2" ""



]]> http://software.intel.com/en-us/articles/how-to-resolve-dat_conf-issues-affecting-platform-mpi/ Wed, 27 Jan 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/how-to-resolve-dat_conf-issues-affecting-platform-mpi/#comments http://software.intel.com/en-us/articles/how-to-resolve-dat_conf-issues-affecting-platform-mpi/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Understanding How Intel Speedstep® Technology and Intel® Hyper-Threading Technology Are Verified by Intel® Cluster Checker Intel Speedstep® Technology and Intel® Hyper-Threading Technology settings can impact the overall performance of a cluster.  Intel® Cluster Checker verifies these settings in the speedstep and core_count checks, respectively. 

For the Intel® Xeon® processor 5500 series and later, by default, the core_count check verifies that Intel® Hyper-Threading Technology is enabled.  On earlier processors, the default core_count behavior is to verify that Intel® Hyper-Threading Technology is disabled.  For any processor, the state to be verified can be configured.  For example, to always verify that Intel® Hyper-Threading Technology is enabled:

<core_count>
  <hyper-threading>on</hyper-threading>
</core_count>

By default, Intel® Cluster Checker verifies that Intel Speedstep® Technology is disabled.  The speedstep check can be configured to verify either state.  For example, to verify that Intel Speedstep® Techology is enabled:

<speedstep>
  <state>on</state>
</speedstep>

Note:  Refer to the documentation provided by your hardware vendor for more information on how to enable or disable Intel Speedstep® Technology or Intel® Hyper-Threading Technology.

]]> http://software.intel.com/en-us/articles/understanding-how-intel-speedstep-technology-and-intel-hyper-threading-technology-are-verified-by-intel-cluster-checker/ Wed, 20 Jan 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/understanding-how-intel-speedstep-technology-and-intel-hyper-threading-technology-are-verified-by-intel-cluster-checker/#comments http://software.intel.com/en-us/articles/understanding-how-intel-speedstep-technology-and-intel-hyper-threading-technology-are-verified-by-intel-cluster-checker/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base How To Utilize the Intel® Cluster Checker Level Command Line Parameter
The downside of running some test modules that provide exhaustive coverage of issues is that time it can take to run those modules. As such, it may be desired to limit when the high level tests (Levels 4 and 5) are executed to save time.

By default, Intel® Cluster Checker executes Level 3 tests and below. This behavior can be modified using the --level command line option to specify the top level to execute. This causes the tool to only run included tests that are of the specified level or lower. For instance, the following command will execute test modules that are Levels 1-4:

cluster-check config.xml --level 4

Similarily, the following will execute test modules that are Levels 1-2:

cluster-check config.xml --level 2

Examples of modules classified at each level:
  • Level 1: ping, kernel
  • Level 2: intel_cce_rtl, process_check
  • Level 3: imb_pingpong_intel_mpi
  • Level 4: hpcc
  • Level 5: intel_mpi_testsuite
One of the advantages of running lower levels is the tests will complete relatively quickly. Administrators can utilize varying levels to make quick daily checks of basic cluster functionality combined with less frequent higher level checks that are more exhaustive. The longer higher level checks can be scheduled for low resource usage times. See the Intel® Cluster Checker Advanced Guide for more information. ]]> http://software.intel.com/en-us/articles/how-to-utilize-the-intel-cluster-checker-level-command-line-parameter/ Wed, 20 Jan 2010 00:00:00 -0800 http://software.intel.com/en-us/articles/how-to-utilize-the-intel-cluster-checker-level-command-line-parameter/#comments http://software.intel.com/en-us/articles/how-to-utilize-the-intel-cluster-checker-level-command-line-parameter/ Intel® Cluster Ready Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Debugging with the --include_only command line parameter --include_only command line parameter will save time by focusing in on the relevant issue.

Once you know the name of the desired module to execute, it is simple to launch the tool to execute up to that module and stop. For instance, if it is desired to run just the intel_mpi_rt module, the following command line can be used:

cluster-check config.xml --include_only mflops_intel_mkl

This instructs Intel® Cluster Checker to run the test module specified and include only any other modules that are upstream in the dependency list or have been explicitly included in the configuration file.

Executing the above command will then only include the following tests to be run:

Exclusively including modules (at user request):
'mflops_intel_mkl'

Including modules (to meet dependency requirements):
'genuine_intel'
'mount_proc'
'ping'
'sh'
'ssh'
'tmp'


Now assume for this example that config.xml contains the following tag to explicitly include a module:

<include_module>openib</include_module>

Executing the same command above with this tag in the configuration file will cause the explicitly included module to execute as well:

Exclusively including modules (at user request):
'mflops_intel_mkl'

Including modules (at user request):
'openib'

Including modules (to meet dependency requirements):
'genuine_intel'
'mount_proc'
'ping'
'sh'
'ssh'
'tmp'

Notice the inclusion of the openib test module even though it is not in the dependency list of the mflops_intel_mkl test module. If the configuration file contains many tags that are explicitly included, it may be beneficial to create a temporary configuration file and remove those items. This can save time running specific test modules when debugging or checking a single component of the cluster.

Using the --include_only option is an effective debugging tool that saves time troubleshooting and resolving minor issues since full Intel® Cluster Checker runs are not needed to test a single component. Once an individual issue is resolved and the corresponding test module is passing, then the full test suite can be run to verify the cluster is compliant and running correctly.

]]> http://software.intel.com/en-us/articles/debugging-with-the-include_only-command-line-parameter/ Wed, 13 Jan 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/debugging-with-the-include_only-command-line-parameter/#comments http://software.intel.com/en-us/articles/debugging-with-the-include_only-command-line-parameter/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Setting the Intel® Cluster Checker Environment clckvars.sh and clckvars.csh.  Among other things, having a proper environment ensures that the license file can be located, man pages are available, and the cluster-check program is part of your PATH. 

If the environment is not set, Intel® Cluster Checker may not function properly.

The script that you use depends on your shell.  

For Bourne-derivative shells (bash, ksh, sh):

source /opt/intel/clck/1.4/clckvars.sh

For C shell (csh or tcsh): 

source  /opt/intel/clck/1.4/clckvars.csh

If you are unsure what shell you are using, run the following command:

ps -p $$

Rather than setting the environment each time you login, consider adding the appropriate command to your shell startup file (e.g., .bashrc, .cshrc).

Note: the examples in this article are based on Intel® Cluster Checker 1.4.  For other versions, you will need to adjust the path.
]]> http://software.intel.com/en-us/articles/setting-the-intel-cluster-checker-environment/ Wed, 13 Jan 2010 22:00:00 -0800 http://software.intel.com/en-us/articles/setting-the-intel-cluster-checker-environment/#comments http://software.intel.com/en-us/articles/setting-the-intel-cluster-checker-environment/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Troubleshooting the dmidecode check: Wake-up Type Symptom

The dmidecode module of Intel® Cluster Checker reports inconsistencies in the “Wake-Up Type” value. The error may appear similar to the following example:

SMBIOS/DMI Uniformity,(dmidecode).............................................FAILED
subtest 'System Information (0x0002): Wake-up Type' failed
- failing hosts node-1, node-2 returned: 'AC Power Restored'
- failing host node-0 returned: 'Power Switch'

Modifying the Power Restore Policy or Restore on AC Power Loss setting, using either the IPMI/BIOS configuration interface or the Save and Restore System Configuration utility (syscfg), has no effect on this error.

Cause

The “Wake-Up Type” value does not represent the current setting for power restore or power-on response. It describes how the system was powered-on after the last power off, regardless of restore settings. This value may change after each boot, depending on what method was used to power-on the individual node.

Resolution

This value stores dynamic data and compute nodes are not expected to have identical data for this value. It can be ignored for the purposes of Intel® Cluster Ready certification and verification.

Exclude the value by adding an exclude element to the dmidecode block in the Intel® Cluster Checker configuration file. For example:

<dmidecode>
  <exclude>System Information (0x0002): Wake-up Type</exclude>
</dmidecode>

In Intel® Cluster Checker 1.4, the exclude element has been enhanced to support regular expression matching. In this case, the following option is sufficient:

<dmidecode>
  <exclude>Wake-up Type</exclude>
</dmidecode>

For more information on troubleshooting the dmidecode check, see this article.  ]]> http://software.intel.com/en-us/articles/troubleshooting-the-dmidecode-check-wake-up-type/ Wed, 16 Dec 2009 22:00:00 -0800 http://software.intel.com/en-us/articles/troubleshooting-the-dmidecode-check-wake-up-type/#comments http://software.intel.com/en-us/articles/troubleshooting-the-dmidecode-check-wake-up-type/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base Configuring Intel® Cluster Checker for Intel® Server System SR1680MV Background

The Intel® Server System SR1680MV is a modularized rack server system providing two independent dual-socket, 18 DIMM nodes in a 1U chassis. Each node features an independent, flexible, and pluggable I/O tray supporting a PCI Express* 2.0 X8 riser.

The design of this server system has minor differences in the PCI Express riser between the nodes on either side of the chassis.  This impacts add-in cards, such as InfiniBand* HCAs.  An add-in card in one server board will use a different PCI Express slot number than an add-in card in the other server board in the same chassis.

The Intel® Cluster Checker pci module checks that add-in cards are inserted into the same slot on every node, so this slot number inconsistency will produce errors. This is by design, since performance characteristics may differ between PCI Express slots.

Configuration

The pci module must be resolved separately for the left and right sides of the chassis. This can be easily accomplished using group functionality in Intel® Cluster Checker. The node list and configuration file must be updated to reflect the different board layouts.

First, update the node list to create two groups, one each for the left and right-side server boards. When complete, the node list should appear similar to this example.

cluster # group:right type:head
node-1  # group:left
node-2  # group:right
node-3  # group:left
node-4  # group:right
node-5  # group:left

Next, update the pci module configuration in the XML configuration file to identify each group.

<pci>
  <group name="left"/>
  <group name="right"/>
</pci>

The pci module will run on all nodes in the cluster, but PCI device consistency will be evaluated on the left and right groups independently. If excluding specific PCI devices, both the exclude and group elements must be direct children of the pci parent element.

If similar consistency errors appear in other modules, such as dmidecode, add the group element to the configuration of those modules to separate the tests for the left and right nodes.

]]> http://software.intel.com/en-us/articles/configuring-intel-cluster-checker-for-intel-server-system-sr1680mv/ Thu, 10 Dec 2009 22:00:00 -0800 http://software.intel.com/en-us/articles/configuring-intel-cluster-checker-for-intel-server-system-sr1680mv/#comments http://software.intel.com/en-us/articles/configuring-intel-cluster-checker-for-intel-server-system-sr1680mv/ Intel® Cluster Checker Knowledge Base Intel® Cluster Ready Knowledge Base