Intel® Cluster Ready www.intel.com/go/cluster is designed to create predictable expectations for users and providers of high-performance technical computing (HPTC) clusters, primarily targeting customers in the commercial and industrial sectors.  These are not experimental “test-bed” clusters used for computer science and computer engineering research, or high-end “capability” clusters closely targeting their specific computing requirements that power the high-energy physics at the national labs or other specialized research organizations. Intel Cluster Ready seeks to advance HPC clusters used as computing resources in production environments by providing cluster owners with have a high degree of confidence that the clusters they deploy will run the applications their scientific and engineering staff rely upon to do their jobs.  It achieves this by providing cluster hardware, software, and system providers a precisely defined basis for their products to meet their customers’ production cluster requirements.  Intel Cluster Ready is not an HPC-oriented Linux* distribution; nor is it simply yet another Linux cluster software stack. Instead, Intel Cluster Ready complements existing Linux distributions and cluster stacks, simplifying the selection and deployment process for those who prioritize deployment of multiple applications over the total flexibility of building their own cluster to their specifications.

The primary objective of Intel Cluster Ready is to make clusters easier to specify, easier to buy, easier to deploy, and easier to develop applications that run on them.  A key feature of ICR is the concept of “application mobility”, which we define as the ability of a registered Intel Cluster Ready application—more correctly, the same binary—to run correctly on any certified Intel Cluster Ready cluster.  Clearly, application mobility is important for users, software providers, hardware providers, and system providers.

• Users want to know the cluster they choose will reliably run the applications they rely on today, and will rely on tomorrow
• Application providers want to satisfy the needs of their customers by providing applications that reliably run on their customers’ cluster hardware and cluster stacks
• Cluster stack providers want to satisfy the needs of their customers by providing a cluster stack that supports their customers’ applications and cluster hardware
• Hardware providers want to satisfy the needs of their customers by providing hardware components that supports their customers’ applications and cluster stacks
• System providers want to satisfy the needs of their customers by providing complete cluster implementations that reliably run their customers’ applications

Wi thout application mobility, each group above must either try to support all combinations, which they have neither the time nor resources to do, or pick the “winning combination(s)” that best supports their needs, and risk making the wrong choice.

The Intel Cluster Ready definition of application portability supports all of these needs by going beyond pure portability, (i.e., re-compiling and linking a unique binary for each platform), to application binary mobility, (i.e., running the same binary on multiple platforms), by more precisely defining the target system.

A further aspect of application mobility is to ensure that registered Intel Cluster Ready applications do not need special programming or alternate binaries for different message fabrics. Intel Cluster Ready accomplishes this by providing an MPI* implementation supporting multiple fabrics at runtime; through this, registered Intel Cluster Ready applications obey the “message layer independence property”.

Stepping back, the unifying concept of Intel Cluster Ready is “one-to-many,” that is,

• One application will run on many clusters
• One cluster will run many applications

How is one-to-many accomplished?  Looking at Figure 1, on the left you see the abstract Intel Cluster Ready “stack” components that always exist in every cluster, i.e., one or more applications, a cluster software stack, one or more fabrics, and finally the underlying cluster hardware. The remainder of that picture (to the right) shows the components in greater detail.

Figure 1: One to Many

Applications, on the top of the stack, rely upon the various APIs, utilities, and file system structure presented by the underlying software stack.  Registered Intel Cluster Ready applications are always able to rely upon the APIs, utilities, and file system structure specified by the Intel® Cluster Ready Specification; if an application requires software outside this “required” set, then Intel Cluster Ready requires the application to provide that software as a part of its installation.  To ensure that this additional per-application software doesn’t conflict with the cluster stack or other applications, Intel Cluster Ready also requires the additional software to be installed in application-private trees, so the application knows how to find that software while not interfering with other applications.  While this may well cause duplicate software to be installed, the reliability provided by the duplication far outweighs the cost of the duplicated files.  A prime example supporting this comparison is the removal of a common file (library, utility, or other) that is unknowingly needed by some other application—such errors can be insidious to repair even when they cause an outright application failure.

Cluster platforms, at the bottom of the stack, provide the APIs, utilities, and file system structure relied upon by registered applications.  Certified Intel Cluster Ready platforms ensure the APIs, utilities, and file system structure are complete per the Intel Cluster Ready Specification; certified clusters are able to provide them by various means as they deem appr opriate.  Because of the clearly defined responsibilities ensuring the presence of all software required by registered applications, system providers have a high confidence that the certified clusters they build are able to run any certified applications their customers rely on.  In addition to meeting the Intel Cluster Ready requirements, certified clusters can also provide their added value, that is, other features and capabilities that increase the value of their products.

At its heart, Intel® Cluster Ready is:

• A definition of the cluster as parallel application platform
• A tool to certify an actual cluster to the definition
• Example descriptions of how to build certified clusters (a.k.a. recipes)
• Documentation, support, and training.
• Conforming (certified) hardware and (registered) software

Let’s look at each of these in more detail, to understand their motivations and benefits.

A definition of the cluster as parallel application platform.   The Intel Cluster Ready Specification is very much written as the requirements for, not the implementation of, a platform upon which parallel applications, more specifically MPI applications, can be built and run.  As such, the specification doesn’t care whether the cluster is diskful or diskless, fully distributed or single system image (SSI), built from “Enterprise” distributions or community distributions, fully open source or not.  Perhaps more importantly, with one exception, the specification doesn’t have any requirements on how the cluster is built; that one exception is that compute nodes must be built with automated tools, so that new, repaired, or replaced nodes can be rebuilt identically to the existing nodes without any manual interaction, other than possibly initiating the build process.

Some items the specification does care about include:

• the ability to run both 32- and 64-bit applications, including MPI applications and X-clients, on any of the compute nodes,
• the ability to build both 32- and 64-bit applications, including MPI applications and X-clients, on any of the compute nodes,
• consistency among  the compute nodes’ configuration, capability, and performance,
• the identical accessibility of libraries and tools across the cluster,
• the identical access by each compute node to permanent and temporary storage, as well as users’ data,
• the identical access to each compute node from the head node,
• the MPI implementation provides fabric independence, and
• all nodes support network booting and provide a remotely accessible console,

The specification also requires that the runtimes for specific Intel software products are installed on every certified cluster:

• Intel® Math Kernel Library
• Intel® MPI Library Runtime Environment
• Intel® Threading Building Blocks

This requirement does two things.  First and foremost, mainline Linux distributions do not necessarily provide a sufficient software stack to build an HPC cluster—such specialization is beyond their mission.  Secondly, the requirement ensures that programs built with this software will always work on certified clusters and enjoy simpler installations.  As these runtimes are directly available from the web for download (see http://www.intel.com/ for more information), the requirement does not cause additional costs to certified clusters.  It is also very important to note that this does not require certified applications to use these libraries nor does it preclude alternate libraries, e.g., other MPI implementations, from being present on certified clusters.  Quite clearly, an application that requires, e.g., an alternate MPI, must also provide the runtimes for that MPI as a part of its installation.

A tool to certify an actual cluster to the definition. The Intel® Cluster Checker, included with every certified Intel Cluster Ready implementation, is used in four modes in the life of a cluster:

• To certify a system provider’s prototype cluster as a valid implementation of the specification
• To verify to the owner that the just-delivered cluster is a “true copy” of the certified prototype
• To ensure the cluster remains fully functional, reducing service calls not related to the applications or the hardware
• To help software and system provides diagnose and correct actual problems to their code or their hardware.

While these are critical capabilities, in all fairness, this greatly understates the capabilities of Intel Cluster Checker.  The tool will not only verify the cluster is performing as expected.  To do this, per-node and cluster-wide static and dynamic tests are made of the hardware and software.

Figure 2 Intel® Cluster Checker

The static checks ensure the systems are configured consistently and appropriately.  As one example, the tool will ensure the systems are all running the same BIOS versions as well as having identical configurations among key BIOS settings.  This type of problem—differing BIOS versions or settings—can be the root cause of subtle problems such as differing memory configurations that manifest themselves as differing memory bandwidths only be seen at the application level as slower than expected overall performance.  As many readers of this site probably know, parallel program performance can be very much governed by the performance of the slowest components, not the fastest.  In another static check, the Intel Cluster Checker will ensure that the expected tools, libraries, and files are present on each node, identically located on all nodes, as well as identically implemented on all nodes.  This ensures that each node has the minimal software stack specified by the specification, as well as identical software stack among the compute nodes.

A typical dynamic check ensures consistent system performance, e.g., via the STREAM* benchmark.  This particular test ensures processor and memory performance is consistent across compute nodes, which, like the BIOS setting example above, can be the root cause of overall slower application performance.  An additional check with STREAM can be made if the user configures an expectation of benchmark performance; this check will ensure that performance is not only consistent across the cluster, but also meets expectations.  Going beyond processor performance, the Intel® MPI Benchmarks are used to ensure the network fabric(s) are performing properly and, with a configuration that describes expected performance levels, up to the cluster provider’s performance expectations.  Network inconsistencies due to poorly performing Ethernet* NICs, InfiniBand* HBAs, faulty switches, and loose or faulty cables can be identified.

Our experience is the Intel Cluster Checker identifies many common sources of cluster malfunction; we expect this to substantially reduce support calls to systems providers and ISVs by identifying underlying structural problems, so that when a service call is made, it is for an actual application or hardware problem.

Finally, the Intel Cluster Checker is extensible, enabling additional tests to be added supporting additional features and capabilities.  This enables the Intel Cluster Checker to not only support the minimal requirements of the Intel Cluster Ready Specification, but the full cluster as delivered to the customer.

Example descriptions of how to build certified clusters. We provide step-by-step descriptions, (a.k.a., recipes) along with a bill of materials, showing exactly how to build a cluster that can be certified Intel Cluster Ready.  While larger system providers will already have their own existing cluster offerings that may only need a few small changes to add a level of conformance so that they can be certified as Intel Cluster Ready, smaller system providers, who don’t already have an established cluster practice, or perhaps don’t consistently build clusters, will benefit, as these recipes provide a solid basis for building properly configured clusters able to support registered applications.  While these recipes can’t guarantee maximal performance for all registered applications, they do provide for a functional cluster meeting basic performance expectations.  It is then up to the system provider to make sure the cluster’s actual performance meets their customer’s requirements, which, beyond application performance, will likely include budget, power/cooling, and space, and may also include software, hardware, as well as various other operational and implementation requirements.

The first two Intel Cluster Ready recipes document the construction of clusters based on our uni-socket and dual-socket systems, S3000PT and S5000PAL, respectively, with both Ethernet and InfiniBand network fabrics, using the Platform* Open Cluster Stack* http://www.platform.com and Red Hat* Enterprise Linux* http://www.redhat.com.

Documentation, support, and training. While the Intel Cluster Ready Specification, the Intel Cluster Checker, recipes, and other documents and procedures get you a fairly long way to registered applications and certified clusters, you can’t necessarily drop these documents on somebody’s desk and expect applications to be registered and clusters to be certified.  Beyond the documentation, Intel provides direct training and support to hardware, software, and system providers to get their products certified and registered as appropriate.

Figure 3 Intel® Cluster Ready Program

Certifying a cluster requires the system provider to build their candidate cluster according to their proposed practices, then use the Intel Cluster Checker in its certification mode to verify compliance.  Once the cluster has passed the certification process, the cluster is certified and can then be sold as a certified Intel Cluster Ready cluster.  Clusters delivered under that certification to customers may vary within narrow bounds; the primary variation being the number of nodes.  The Intel Cluster Ready Specification permits the number of computer nodes to be doubled from the certified size, e.g., a certified cluster with one head node and 4 compute nodes can be shipped with up to 8 compute nodes.

Registering an application requires the application to be tested on an Intel-provided certified cluster built specifically to test the prospective application.  ISVs and software providers will have exclusive access to the registration cluster via an SSH* connection; they can then install their application, run tests their validation tests, and complete the registration process.

Conforming (certified) hardware and (registered) software. The preceding was primarily related to the builders of certified clusters and the developers of registered applications.  For end users, that want to purchase a certified cluster to run registered applications, the ability to identify registered applications and certified clusters is most important, as that will reduce their effort to evaluate, acquire, and deploy the clusters that run their applications, and then keep that computing resource operating properly, with full performance, directly increasing their productivity.

Intel Cluster Ready is a program and technology package making it easier to design, build, sell, acquire, deploy and develop application programs for clusters built with Intel components.

Without Intel Cluster Ready, software providers are forced to consider each cluster implementation as a unique environment, and deal with the variations and unique operational issues as best they can; hardware and system providers must also work with each software provider whose applications are important to their customers; finally, users must learn the finer details of cluster computing to ensure the clusters they deploy are able to reliably run the applications they rely upon to do their scientific and engineering work.

With Intel Cluster Ready, IDC http://www.idc.com wrote "Intel Cluster Ready:"

The Intel Cluster Ready program aims to address this dilemma by making available a reference architecture for Intel-based systems that hardware vendors can use to certify their configurations and that ISVs can use to test and register their applications. The chief goal of this voluntary compliance program is to ensure fundamental hardware/software integration, so that end users can get their work done even in cases where no HPC-knowledgeable IT staff are available to help. The Intel Cluster R eady program intends to allow overworked IT departments to respond more readily to end users’ HPC requirements, driving technical cluster resources further down into the organization and freeing up IT staff to focus on mainstream enterprise applications (e.g., payroll, HR, sales, CRM). The Intel Cluster Ready program wants to prevent technical computing end users from having to become, in effect, their own system integrators.

64-bit computing on Intel architecture requires a computer system with a processor, chipset, BIOS, operating system, device drivers and applications enabled for Intel® 64 architecture. Performance will vary depending on your hardware and software configurations. Consult with your system vendor for more information.