Partner Newsletter Q2 2011 Intel Cluster Ready Articles 2

Tips & Tricks
Automated MPI Performance Tuning for Clusters and Applications

By Gergana Slavova
Technical Consulting Engineer
Intel® Cluster Studio

Are you impressed (or slightly baffled) by the fact that there are seven different algorithms for MPI_Allgather you can utilize within the Intel® MPI Library? Wondering how to best pick the size of your DAPL buffers?

While the sheer number of library settings can be intimidating, there’s no need to be overwhelmed. With the mpitune utility, we provide a simple way to make those decisions for you while still giving you the flexibility to customize it to your cluster or application. You may also find that the application experiences a reduction in its total run-time as a result.

The purpose of the mpitune utility is to help you find the optimal values for the large set of Intel MPI Library tuning knobs available. The utility operates in two modes: cluster- and application-specific:

  • In cluster mode, the tuner evaluates the execution environment and uses a set of benchmarks to determine best performance. By default, it uses the ever-popular Intel® MPI Benchmarks. This is, of course, customizable, and you can replace those with any other benchmark of your choosing. We recommend you run this once after installation or following a cluster configuration change to get the best combination of settings.
  • The application-specific mode tunes the Intel MPI Library by focusing only on the MPI routines and variables that are particular to your application. For example, if you have a lot of reduction operations in your code mainly sending small messages across, mpitune would look at the best algorithms for MPI_Reduce with message shorter than 1Kb.
Both methods yield a set of configuration files that you can:

  • Apply to all applications running locally (when doing cluster-specific tuning) OR
  • Take with you across a variety of hardware (using the application-specific tuning parameters)
To find out more, check out the Automatic Tuning Utility section in the Intel® MPI Library Reference Manual, or come talk to us at the Intel® Clusters and HPC Technology forum.

Using Intel® Cluster Checker for Post-Sales Support

By Jeremy Siadal
Senior Technical Consultant
Intel Cluster Ready Program

Intel® Cluster Checker is a versatile
tool for certifying and verifying an Intel Cluster Ready certified cluster prior to delivery. Since Intel Cluster Checker ships with each certified cluster, it also offers a great way for you to streamline and simplify remote support after clusters are shipped and installed. Using Intel Cluster Checker, support engineers can more easily identify problems, determine the sources of those problems, and get their customers back to work quickly.
Intel Cluster Checker 1.7 is now available and we’ll refer to capabilities in that version in the discussion below.

A Modular, Flexible Tool
Intel Cluster Checker provides a method to test against a known and verified configuration. As a certified configuration, the cluster was fully functional and met performance expectations when it left the factory. However, once the cluster is at the end-user’s site, those users are likely to install new software and make changes to the configuration. Problems can occur and performance can be compromised.
Intel Cluster Checker can help reveal changes to configuration or performance since the cluster was delivered. Intel Cluster Checker provides several capabilities that make it suitable for working remotely with customers:

  • Input and output files. Since Intel Cluster Checker supports editable input files and plain text output files, you can perform remote support through email.
  • Custom configurations. You can create new input files based on the user’s description of the problem or the results of initial Intel Cluster Checker runs.
  • Modularity. Test modules can be switched on and off as needed. You can create special tests with generic test modules, which execute the commands you specify. You can even create your own test modules.
  • Debug output. Detailed output from all of the cluster nodes can be captured to text files, including commands issued, standard output, and error messages.
Intel Cluster Checker provides a rich set of diagnostics capabilities. For example, the test module will list every change to RPM packages from the original installation. The test module can flag any file differences. This can help identify possible issues with file or version conflicts. Test modules such as or can identify performance degradation. In addition, you can enable debug output or adjust the amount of detail (verbosity) that Intel Cluster Checker reports. Even if you don’t enable increased verbosity, you’ll find much additional information in the XML output file.

Easy for Customers, Easy for Support Engineers
Let’s look at a typical support operation using Intel Cluster Checker. Let’s say the customer reports that the Intel® MPI library is no longer working correctly when InfiniBand* fabric is used. As a support engineer working remotely, you first have the customer run Intel Cluster Checker in deployment mode and email the resulting output files to you. As it executes, Intel Cluster Checker will perform multiple checks in a much simpler fashion than asking the customer execute multiple command lines. For this scenario, the test module will identify if base functionality with the Intel MPI Library over InfiniBand fabric is working. If this test module passes, other test modules will then check inter-node operation and performance.

A quick look at the output file is often enough to identify the cause of the problem. Problems causing the test module to fail may be identified in more basic or prerequisite test modules. A required RPM that was removed will be found by the test module. A disconnected cable may be found by the test module. If a kernel driver failed to load on a node, the test module will flag this.

If the problem isn’t clearly identified by the initial cluster check, you can modify the input configuration file and forward it to the customer for another test. Using the tag will limit execution to specific test modules. You can also select just the specific nodes you want test. Using an XML editor, you can easily create multiple scenarios and perform regression testing. The customer can run these additional tests and forward the results to you, providing a support cycle that can quickly isolate the source of the problem (see Figure 1).

Figure 1

You can obtain much more detailed information about execution and error messages by enabling debug on a test module. If you need even more information, you may be able to use the tag to set up additional environment options. In this scenario, where has failed, you would use to set the I_MPI_DEBUG environment variable. This will provide much more information on Intel MPI Library execution. You must also set the flag on that test module to capture all the output.

For this scenario, let’s say the cause of the error was using the wrong device in the dat.conf file. The additional output from the Intel MPI Library, captured using the flag, could identify the problem. You can have the customer run Intel Cluster Checker again to confirm that an attempted fix was successful.

Let us know what you think
The Intel Cluster Ready program is about helping the HPC industry to come together for our customers’ success and our own. If you’re not already using it, Intel Cluster Checker can help streamline many support operations, so your customers can get the most from their certified clusters. It’s a great example of Intel Cluster Ready value, and we’re improving Intel Cluster Checker based on feedback from our customers. Please let us know if you have feedback or recommendations.

Contact us at or come talk to us at the Intel® Clusters and HPC Technology forum.


For more complete information about compiler optimizations, see our Optimization Notice.