Debugging with Intel® Cluster Checker

By Christopher Heller

Intel® Cluster Checker can detect and report a number of issues relating to hardware, software configuration, network performance, or other matters. Many times, you can resolve the issue by looking at the console output or by going into the .out or .xml output files from the run.

The first step in debugging a problem is to utilize the --include_only flag to examine the subset of the test suite that is having issues. The --include_only command line parameter is designed for situations where you’re debugging a known issue or want to test a specific component in the cluster. Issuing a command with the form --include_only tells Intel Cluster Checker to run the specified test module and include any other modules that you specify or that are upstream in the dependency list.



The verbosity flag lets you control the amount of output you’ll receive from the cluster checker. The format for the flag is --verbose.

The value can be anywhere from 1 to 5, with higher values providing more output:

1. reports the overall success or failure only with no information on the status of the test modules.
   
2. is the default verbosity level. It reports the success or failure of each test module and the overall success or failure, and provides additional output about ailing or indeterminate test modules.
   
3. provides the same output as Level 2, but also prints the name of the failing test modules that cause a check to be skipped.
   
4. reports the success or failure of each test module and the overall success or failure, and prints additional output for all test modules, regardless of their success or failure status.
   
5. reports all the information as 4, and also displays the version of each module.
   

For example, if you’re having disk performance issues and want to debug your hdparm values with as much output from Intel Cluster Checker as possible, you would use the following command:

cluster-check myconfig.xml --include_only=hdparm --verbose 5

This will run hdparm and its prerequisites and provide screen and log output.

Going Further with Debug

If the include_only and verbosity flags were unable to shed light on your problem, Intel Cluster Checker can provide detailed debug output to assist with trouble-shooting. You can enable debug mode either through use of the --debug command line option for all checks, or on a per-check basis using the element tag. Debug output is provided in one or more files named using the format: -.debug. The resulting .debug file will show the commands executed to the compute nodes in the test or tests specified and will include the results of those tests. This should give you more diagnostic information to debug the issue. 



Check Out the Knowledge Base

There’s a wealth of valuable information in the Intel Cluster Ready Knowledge Base. Look for additional information on the topics I’ve covered here at the links below. Then, I hope you’ll look around and see what else you can find to increase your expertise and help you get the most out of the Intel Cluster Ready resources.

Debugging with the --include_only command line parameter: http://software.intel.com/en-us/articles/debugging-with-the-include_only-command-line-parameter/

Running debug: http://software.intel.com/en-us/articles/understanding-intel-cluster-checker-debug-output-and-the-secure-shell-ssh/

See you next issue.
______

Christopher Heller is an HPC customer support and development engineer within the Intel Cluster Ready program. He's based in Champaign, IL and has worked at Intel for 5 years.