Analysis

Given a populated database (see the Data Collection chapter), Intel® Cluster Checker analyzes the data to identify issues, diagnose problems, and in some cases, provide recommendations on how to repair the cluster. Invoke the clck-analyze program to perform analysis or clck to perform both collection and analysis. The analysis evaluates the collected data using an embedded expert system.

Running Analysis

There are two ways to analyze data using Intel® Cluster Checker. The clck command runs data collection followed by analysis, while clck-analyze immediately analyzes data in the database. Analysis does not require any command line options, and Intel® Cluster Checker will analyze all nodes in the database by default. To analyze a subset of nodes or to assign node roles, provide a nodefile using the -f command line option. For more information about writing a nodefile, see the Selecting Nodes section in the Data Collection chapter. For details about the available command line options, see Configuring Intel® Cluster Checker. A typical use of the analysis command is:

clck-analyze -f nodefile

Or:

clck -f nodefile

The output to the screen will provide a brief summary of any issues found. Further details will be written to the log file.

Each issue has a message ID, a severity, a list of relevant nodes, and potentially a database row ID and a remedy. The message id is an unique identifier for the issue type. All issues have a primary id; some issues may also have an optional sub-id appended with a colon. That is, id:sub-id. The message id can be used to suppress the issue, if desired (see the Suppressions section).

There are three severity levels: informational, warning, and critical. Critical issues are the most severe, and higher severity levels indicate a higher significance of the identified issues on the cluster. Lower severity levels, like informational, indicate that the issue is less of a problem.

A list of nodes will be displayed with the issue, indicating the nodes in the system to which the issue applies. Node names displayed in parentheses indicate that the issues applies to a pair of nodes, like MPI latency between a pair of nodes.

The database row id is a list of database entries containing the raw data that led to the issue. Database row ids are only included when debug output is enabled (see Configuring Intel® Cluster Checker).

Some issues may recommend a remedy to resolve the issue. Depending on the cluster configuration, some remedies may not be appropriate. If unsure, review any suggested remedies with an expert before implementing them. Some remedies may require privileged cluster access.

Issues fall into one of two categories:

  • Diagnoses - Diagnoses describe the root cause of an issue. For example, MPI performance is substandard because some network setting is misconfigured. The typical process to reach a diagnosis is by combining one or more observations. In this example, an observation for substandard MPI performance and another observation for a misconfigured network setting.

  • Observations - An observation is an objective fact about the cluster based on collected data. For example, a cluster's memory may not be uniform. An observation may or may not contribute to a diagnosis.

Each reported issue should be investigated and either resolved or suppressed (see the Suppressions section). Once the issue is resolved, fresh data should be collected and the analysis repeated. When no issues are reported, the cluster has been successfully verified with Intel® Cluster Checker.

Selecting Nodes

By default, clck-analyze will analyze all nodes in the database, and clck will use Slurm to auto-detect nodes. If a nodefile is supplied, then the list of nodes contained in the nodefile will be used instead. The nodefile contains a list of line-separated cluster node hostnames. Additional, optional nodefile annotations can also be specified and may alter the analysis (see the Selecting Nodes section in the "Data Collection" chapter). For example, some rules may only apply to compute nodes and ignore non-compute nodes.

Framework Definition (FWD) Selection

Framework Definitions, further detailed in the Framework Definitions chapter, can be used to select which group of providers will run during data collection and which analyzer extensions and knowledge base modules will run during analysis.

Framework Definitions can be specified through the command line by using the -F / --framework-definition command line option.

-F FWD/ --framework-definition FWD

For instance, the following command would run myFramework.xml:

clck -f nodefile -F /path/to/myFramework.xml

Custom FWDs can also be specified in the configuration file /opt/intel/clck/201n/etc/clck.xml. The following example shows how to declare the use of two custom definitions:

<configuration>
   <analyzer>
      <framework_definitions>
         <framework_definition>/path/to/CustomFWD1.xml
            </framework_definition>
         <framework_definition>/path/to/CustomFWD2.xml
            </framework_definition>
      </framework_definitions>
      ...
      ...
   </analyzer>
   ...
</configuration>

For more information on Framework Definitions, see the Included Framework Definitions section in the Appendix.

 

Filtering

When troubleshooting an issue, it can be useful to filter issues so that only the issue at hand is displayed. To efficiently triage issues, determining at what severity issues cause a result of FAIL can also be useful.

Several command line options are available to filter the reported issues, including:

-n node / --node-include node

  • Include only issues for the specified node.

 

-z / --pass-level severity

  • Determine the severity level at which found issues cause a result of FAIL. For example, -z warning would cause any issue with a severity level of warning or higher to result in a FAIL.

 

Suppressions

In some cases, while the issue may be correct, the behavior is actually intended and should not be flagged. Such issues can be suppressed by adding an entry to the configuration file.

The base suppression format is:

<configuration>
   <analyzer>
      ...
      <suppressions>
         <suppress>
            <id>string</id>
            <node_id>hostname</node_id>
            <severity>num</severity>
         </suppress>
         ...
      </suppressions>
      ...
   </analyzer>
   ...
</configuration>

Multiple suppressions may be specified.

<id>string</id>

  • Suppress all issues matching the specified message id string. The default is empty, meaning suppress all message ids that match the other tags.
  • If the message id includes a sub-id and only the primary id is used, then all messages with the same primary id will be suppressed regardless of the sub-id.

 

<node_id>hostname</node_id>

  • Suppress all issues corresponding to the specified node. The default is empty, meaning suppress all nodes that match the other tags.

 

<severity>num</severity>

  • Suppress all issues with a severity level less than the specified value. The default is 0. Severity levels occur in a range of 0 to 100. Issues with a severity of 0-24 are informational, 25-74 are warning, and 75+ are critical.

 

If a tag is omitted, then the default value is used. There is implicit AND logic among tags within a suppression.

The following example will suppress all issues from node4, any issues with message id example-id and with a confidence level of less than 50% on any node, as well as any issues with message id network:eth1 or network:eth2 but not other sub-id values.

<configuration>
   <analyzer>
      ...
      <suppressions>
         <suppress>
            <node_id>node4</node_id>
         </suppress>
         <suppress>
            <confidence>50</confidence>
            <id>example-id</id>
         </suppress>
         <suppress>
            <id>network:eth1</id>
         </suppress>
         <suppress>
            <id>network:eth2</id>
         </suppress>
      </suppressions>
      ...
   </analyzer>
   ...
</configuration>

 

Snapshots

The snapshot functionality allows for comparison between two sets of data for certain framework definitions. Currently, only the framework definitions files_snapshot, hardware_snapshot, and rpm_snapshot support snapshot comparisons. The -M or --mark-snapshot command line option takes a snapshot of the data used for analysis. Take, for example, the following command:

clck-analyze -M snapshot1 -F rpm_snapshot

This command performs analysis, taking a snapshot of the data and marking it with the string snapshot1.

The -W or --with-snapshot command line option compares the data from two snapshots. Consider the following command:

clck-analyze -W snapshot1,snapshot2 -F rpm_snapshot

This command performs analysis, comparing data from snapshot1 with data from snapshot2.

 

Configuration Options

Intel® Cluster Checker contains both command line options and a configuration file to allow for configuration of the tool. The chapter Configuring Intel® Cluster Checker contains a complete list of command line options and an explanation of the config file. 

The config file is in an XML format, and a variety of XML tags are available to configure the behavior of Intel® Cluster Checker. Below is a list of configuration tags that affect analysis.

cluster-mode-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of cluster mode entries across the cluster.

XML syntax:

<config>
   <cluster-mode-uniformity-threshold>NUMBER
   </cluster-mode-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same cluster mode entry value is above the value specified for the cluster-mode-uniformity-threshold tag, then that value is considered uniform in that cluster. If the percentage of nodes that share the same cluster mode entry value is below the uniformity threshold, then a sign is generated.

The value should be a floating point value between 0 and 1.

 

data-age-threshold

 

Specify the maximum age of data points, in seconds, before a data point is considered too old for relevant analysis.

XML syntax:

<config>
   <data-age-threshold>NUMBER
   </data-age-threshold>
</config>

 

The value should be an integer value greater than 0. The default value is 604800 seconds (1 week).

 

data-source-time-difference

 

Specify the maximum time difference allowed between timestamps for two data sources that contribute to the same analysis sign.

XML syntax:

<config>
   <data-source-time-difference>NUMBER
   </data-source-time-difference>
</config>

 

Currently this is only enabled for the dgemm sign substandard-dgemm-due-to-offline-cores.

The value should be an integer value greater than 0. The default value is 900 seconds (15 minutes) for dgemm.

 

dgemm-number-of-mad

 

Specify the number of median absolute deviations (MADs) allowed before a dgemm value is considered an outlier.

XML syntax:

<config>
   <dgemm-number-of-mad>NUMBER
   </dgemm-number-of-mad>
</config>

 

The value should be an integer value greater than 0.

 

dgemm-peak-fraction

 

Specify the minimum value of the ratio between the measured dgemm performance and theoretical peak performance value.

XML syntax:

<config>
   <dgemm-peak-fraction>NUMBER
   </dgemm-peak-fraction>
</config>

 

Any value below this will generate a sign.

The value should be a floating point value between 0 and 1.

 

environment-blacklist

 

Specify the environment variable patterns that will be ignored for uniformity comparison across the cluster.

XML syntax:

<config>
   <environment-blacklist>
      <entry>PATTERN</entry>
      <entry>PATTERN</entry>
   </environment-blacklist>
</config>

 

The value within each entry tag is interpreted as a POSIX matching regular expression. If this value is not a valid POSIX regular expression, then no filtering will be done.

The entry tag can be repeated multiple times.

Note that to exactly match meta characters, (^[.*(${\()+|?<>), they should be escaped.

 

hpl-number-of-mad

 

Specify the number of median absolute deviations (MADs) allowed before an HPL value is considered an outlier.

XML syntax:

<config>
   <hpl-number-of-mad>NUMBER
   </hpl-number-of-mad>
</config>

 

The value should be an integer value greater than 0.

 

imb-pingpong-number-of-mad

 

Specify the number of median absolute deviations (MADs) allowed before a PingPong latency or bandwidth value is considered an outlier.

XML syntax:

<config>
   <imb-pingpong-number-of-mad>NUMBER
   </imb-pingpong-number-of-mad>
</config>

 

The value should be an integer value greater than 0.

 

iozone-number-of-mad

 

Specify the number of median absolute deviations (MADs) allowed before an iozone value is considered an outlier.

XML syntax:

<config>
   <iozone-number-of-mad>NUMBER
   </iozone-number-of-mad>
</config>

 

The value should be an integer value greater than 0.

 

kernel-blacklist

 

Specify the kernel parameter patterns that will be ignored for uniformity comparisons across the cluster.

XML syntax:

<config>
   <kernel-blacklist>
      <entry>PATTERN</entry>
      <entry>PATTERN</entry>
   </kernel-blacklist>
</config>

 

The value within each entry tag is interpreted as a POSIX matching regular expression. If this value is not a valid POSIX regular expression, then no filtering will be done.

The entry tag can be repeated multiple times.

Note that to exactly match meta characters, (^[.*(${\()+|?<>), they should be escaped.

 

kernel-param-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of kernel parameters across the cluster, that is, sysctl entries.

XML syntax:

<config>
   <kernel-param-uniformity-threshold>NUMBER
   </kernel-param-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same kernel parameter entry value is above the value specified for the kernel-param-uniformity-threshold tag, then that value is considered uniform in that cluster. If the percentage of nodes that share the same kernel parameter entry value is below the uniformity threshold, then a sign is generated.

The value should be an floating point value between 0 and 1.

 

logical-cores-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of logical cores across the cluster.

XML syntax:

<config>
   <logical-cores-uniformity-threshold>NUMBER
   </logical-cores-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same setting is above the value specified for the logical-cores-uniformity-threshold tag, then it is considered uniform on the cluster. If the percentage of nodes that share the same number of logical cores is below the uniformity threshold, then a sign is generated.

The value should be a floating point value between 0 and 1. The default value is 0.9.

 

lshw-blacklist

 

Specify the lshw output patterns that will be ignored for uniformity comparison across the cluster.

XML syntax:

<config>
   <lshw-blacklist>
      <entry>PATTERN</entry>
      <entry>PATTERN</entry>
   </lshw-blacklist>
</config>

 

The value within each entry tag is interpreted as a POSIX matching regular expression. If this value is not a valid POSIX regular expression, then no filtering will be done.

The entry tag can be repeated multiple times.

Note that to exactly match meta characters, (^[.*(${\()+|?<>), they should be escaped.

 

lshw-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of lshw entries across the cluster.

XML syntax:

<config>
   <lshw-uniformity-threshold>NUMBER
   </lshw-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same lshw entry value is above the value specified for the lshw-uniformity-threshold tag, then that value is considered uniform in that cluster. If the percentage of nodes that share the same lshw entry value is below the uniformity threshold, then a sign is generated.

The value should be an floating point value between 0 and 1.

 

memory-mode-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of memory mode entries across the cluster.

XML syntax:

<config>
   <memory-mode-uniformity-threshold>NUMBER
   </memory-mode-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same memory mode entry value is above the value specified for the memory-mode-uniformity-threshold tag, then that value is considered uniform in that cluster. If the percentage of nodes that share the same memory mode entry value is below the uniformity threshold, then a sign is generated.

The value should be an floating point value between 0 and 1.

 

memory-uniformity-threshold

 

Specify the maximum allowable deviation, in bytes, from the median memory size before a memory size is considered non-uniform.

XML syntax:

<config>
   <memory-uniformity-threshold>NUMBER
   </memory-uniformitythreshold>
</config>

 

Any value greater than 0 can be used for this tag. The default value is 268435456 bytes (256 MB).

 

ntp-offset-threshold

 

Specify the maximum offset value an NTP peer can have before a sign is generated.

XML syntax:

<config>
   <ntp-offset-threshold>NUMBER
   </ntp-offset-threshold>
</config>

 

Any floating point value can be used for this tag.

 

outlier-max-median-mad-dist

 

Specify the maximum distance, in orders of magnitude, between the median and median absolute deviation (MAD) for the MAD outlier algorithm to be used.

XML syntax:

<config>
   <outlier-max-median-mad-dist>NUMBER
   </outlier-max-median-maddist>
</config>

 

If the allowable distance is exceeded, then the MAD outlier algorithm is disabled and a fallback algorithm (controlled by the outlier-median-pct tag) is used for outlier rules.

The following describes the test controlled by the outlier-max-median-mad-dist tag:

if ( |median - MAD| < 10^outlier-max-median-mad-dist )
   then <use MAD outlier algorithm>
   else <use fallback outlier algorithm>

Any value greater than 0 can be used for this tag. The default value is 2.5.

 

outlier-median-pct

 

Percentage of the median used to calculate outliers by the fallback algorithm.

XML syntax:

<config>
   <outlier-median-pct>NUMBER
   </outlier-median-pct>
</config>

 

The outlier-median-pct determines the distance from the median that a sample value is allowed to be before it is considered an outlier in the fallback outlier algorithm. The outlier-median-pct value is divided by 100 and multiplied by the median to get an allowable distance. If the sample value is further away from the median than the allowable distance, the sample value is considered an outlier.

The following describes the fallback outlier algorithm controlled by the outlier-median-pct tag:

if ( |median - sample_value| >
      (median * (outlier-median-pct / 100) )
   then <the sample_value is an outlier>
   else <the sample_value is not an outlier>

Any value between 0 and 100 can be used. The default value is 5.

 

preferred-cluster-mode

 

Specify the preferred cluster mode for Intel® Xeon Phi™ processor.

XML syntax:

<config>
   <preferred-cluster-mode>MODE
   </preferred-cluster-mode>
</config>

 

Valid values for MODE are All2AllSNC2SNC4Hemisphere and Quadrant.

 

preferred-memory-mode

 

Specify the preferred memory mode for Intel® Xeon Phi™ processor.

XML syntax:

<config>
   <preferred-memory-mode>MODE
   </preferred-memory-mode>
</config>

 

Valid values for MODE are FlatCacheHybrid25 and Hybrid50.

 

preferred-tickless-cores

 

Specify the list of cores for the nohz_full kernel parameter for the Intel® Xeon Phi™ processor.

XML syntax:

<config>
   <preferred-tickless-cores>core list
   </preferred-tickless-cores>
</config>

 

128-2551,2,7-91,6,9 are examples of valid values.

 

preferred-turbo-status

 

Specify the preferred Intel® Turbo Boost Technology status for the processor.

XML syntax:

<config>
   <preferred-turbo-status>STATUS
   </preferred-turbo-status>
</config>

 

The valid values are enabled and disabled.

 

rpm-uniformity-threshold

 

 

Specify the threshold ratio for checking whether each rpm file installed on a node is uniform across the cluster.

XML syntax:

<config>
   <rpm-uniformity-threshold>NUMBER
   </rpm-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same rpm file is above the value specified for the rpm-uniformity-threshold tag, then that rpm file is considered uniform on the cluster. If the percentage of nodes that share the same rpm file is below the uniformity threshold, then a sign is generated.

The value should be a floating point value between 0 and 1.

 

storage-max-used-pct

 

Specify the maximum percentage of space that can be used on a disk partition.

XML syntax:

<config>
   <storage-max-used-pct>NUMBER
   </storage-max-used-pct>
</config>

 

If the percentage is exceeded on a disk partition, then a sign is emitted.

Any value between 0 and 100 can be used for this tag. The default value is 85.

 

stream-number-of-mad

 

Specify the number of median absolute deviations (MADs) allowed before a stream value is considered an outlier.

XML syntax:

<config>
   <stream-number-of-mad>NUMBER
   </stream-number-of-mad>
</config>

 

The value should be an integer value greater than 0.

 

threads-per-core-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of threads available per core across the cluster.

XML syntax:

<config>
   <threads-per-core-uniformity-threshold>NUMBER
   </threads-per-core-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same setting is above the value specified for the threads-per-core-uniformity-threshold tag, then it is considered uniform on the cluster. If the percentage of nodes that share the same threads available per core is below the uniformity threshold, then a sign is generated.

The value should be a floating point value between 0 and 1. The default value is 0.9.

 

turbo-status-uniformity-threshold

 

Specify the threshold ratio for checking the uniformity of Intel® Turbo Boost Technology status (enabled or disabled) on a set of nodes within the cluster.

XML syntax:

<config>
   <turbo-status-uniformity-threshold>NUMBER
   </turbo-status-uniformity-threshold>
</config>

 

If the percentage of nodes that share the same Intel® Turbo Boost Technology status is above the value specified for the turbo-status-uniformity-threshold tag, then Intel® Turbo Boost Technology status is considered uniform on the cluster; otherwise a sign is generated.

The value should be a floating point value between 0 and 1. The default value is set to 0.9.

For more complete information about compiler optimizations, see our Optimization Notice.