Intel® Cluster Checker Release Notes and New Features

Linux* Release Notes

• Update SQLite3* to latest version (version 3.40.1)
• Update IOzone3 to latest version (version 494)

Linux* Release Notes

• Minor bug fixes and improvements
• Update SQLite3* to latest version (version 3.39.4)
• Update libxml2 to latest version (version 2.10.3)
• Intel® Cluster Checker 2021.7.2 may not include all the latest functional and security updates.
  • A new version of Intel® Cluster Checker is targeted to be released by March 2023 and will include additional functional and security updates. Customers should update to the latest version as it becomes available.

Linux* Release Notes

• ​​​​​​New Intel Simplified Software License (ISSL) version
• Update libxml2 to latest version (version 2.10.2)

Linux* Release Notes​​​​​​

• ​​​​​Support added for the LSF scheduler
• Improved handling of invalid postprocessor extension input
• More concise output of numerical node names
• Improvements to the Select Solutions analysis and reporting
• Minor fixes and tweaks to provider functionality and parameters
• Update SQLite3* to latest version (version 3.39.2)
• Update libxml2 to latest version (version 2.9.14)

Linux* Release Notes​​​​​​

• Added support for directories as valid arguments for the '-o' command line option
• Support for Intel® Select Solutions for Professional Visualization 2022.0 including new framework definitions:
  • select_solutions_provis_benchmarks_base_2022.0
  • select_solutions_provis_benchmarks_plus_2022.0
  • select_solutions_provis_user_base_2022.0
  • select_solutions_provis_user_plus_2022.0
• Added support for checking allocated logical cores using the Intel(R) MPI Library cpuinfo tool
• Minor updates to providers for more robust functionality or to provide more information on runtime issues
• Updates to provider parsing for tools with updated output.
• Update SQLite3*(v3.38.0), libxml2(v2.9.13), IOzone(v493) to latest version.

Linux* Release Notes

• Added checks to handle library name/version changes for the Intel® oneAPI Runtime Libraries.
  • Also updated related output messages to provide better clarity on what major components were not found.
• Updated linking to Intel® Math Kernel Library used by HPCG providers to work with the version included by Intel® oneAPI Toolkit 2022.1 release.
  • Added better dependency error reporting in HPCG provider scripts.
• Update SQLite3* to latest version (versions 3.37.2)

Linux* Release Notes

• Added support to discover the type(s) of fabric in use on a system and update the Cluster Checker environment and/or configuration variables accordingly
• Relocatable execution warning file; will now go into the same directory as the log output file specified with the '-o' option
• Command line options added for collection methods/extensions
  • '-p' flag added for setting collection method
  • '-i' flag added for checking which collection methods are available in the installation
• Minor bug fixes and output improvements

Linux* Release Notes

• Added a dependency tree for framework definitions to display the hierarchy of included framework definitions
  • Added tree at the top of '-X <FWD name>' output
  • Added tree as a section in the results output file
• Environment variable path expansion handling added to Framework Definitions
• Bug fixes and improvements
• Improved providers and parsing to account for newest output changes
• Updated third-party SQLite3*, IOzone*, CZMQ*, and ZeroMQ* libraries to latest stable versions
• A CVE with SQLite3 was patched and fixed, CVE-2021-36690

Linux* Release Notes

• Intel® Cluster Checker 2021.3 includes new enhancements and bug fix for:
  • Improvements to the environment modulefile and vars.sh script
  • Fixed a bug related to MPI workload execution. Benchmarks using Intel(R) MPI Library or MPICH will maintain the environment (variables) when running with the MPI collect extension
  • Cleanup of unneeded environment variables from providers:
    • CLCK_PROVIDER_*_DAPL_PROVIDER_LIST (for setting I_MPI_DAPL_PROVIDER_LIST)
    • CLCK_PROVIDER_*_DAPL_PROVIDER (for setting I_MPI_DAPL_PROVIDER)
    • CLCK_PROVIDER_*_OFA_ADAPTER_NAME (for setting I_MPI_OFA_ADAPTER_NAME)
    • CLCK_PROVIDER_*_TMI_PROVIDER (for setting I_MPI_TMI_PROVIDER)
  • Software Development Kit: Added missing header file to SDK include directory
  • Latest libxml2 2.9.12 included addressing CVE-2021-3541 & previously patched CVE items.
  • Analysis of IMB_pingpong extension uses latest collected data for analysis
  • Environment propagation no longer propagates the local-only PROFILEREAD (readonly variable) and LC_* (locale settings) environment variables​

Linux* Release Notes

• This latest Intel® Cluster Checker release adds support for:
  • Support for Third Generation Intel® Xeon® Scalable processors
  • Support for Intel® HPC Platform Specification 2.0
  • Support for Intel® Select Solutions for Simulation and Modeling 2021.0 including new framework definitions:
    • select_solutions_network_performance
    • select_solutions_sim_mod_benchmarks_plus_2021.0
    • select_solutions_sim_mod_priv_plus_2021.0
    • select_solutions_sim_mod_user_plus_2021.0
  • A CVE with SQLite3 was patched and fixed, CVE-2021-20227.

Linux* Release Notes

• Improved execution of HPL and HPCG with mpirun launch command on workload managed systems (e.g. Slurm)
• Intel(R) Ethernet 800 series support (intel_ethernet800_* frameworks)
  • driver checks and version uniformity
  • required libraries
  • memlock unlimited check
  • PCIe checks
• setvars scripts compatible with mksh environments

Linux* Release Notes

Intel(R) Cluster Checker Version 2021 Update 1 represents several significant improvements from the last major revision of Intel(R) Cluster Checker Version 2019. Many of the modifications and improvements listed below have been introduced  in subsequent updates to Intel(R) Cluster Checker Version 2019 over the past year, and included in beta releases for Version 2021, the following is a compilation of all the significant changes, organized by category, not by update or beta release.

• Improvements to Output Formatting, Error Reports and JSON Support
  • Intel(R) Cluster Checker has greatly improved its output formatting and error reporting. These changes have been introduced over the past year in the Update packages, and during the beta release for oneAPI.  Intel(R) Cluster Checker has adopted a new summarized report upon completion, with details from the analysis available in the clck_report.log (or designated -o <filename>), and details about the execution environment for Intel(R) Cluster Checker in the clck_execution_warnings.log file.
• New Support for OSU and Intel(R) MPI workloads and analysis
  • Intel(R) Cluster Checker now provides a wide array of workloads and test suites available for the evaluation and continued performance analysis of the compute environment. Support is now extended to the execution of Ohio State University (OSU) workloads. While the specific OSU workloads are not provided as part of the standard Intel(R) Cluster Checker installation, the OSU workloads can be installed separately and used by Intel(R) Cluster Checker for the evaluation and analysis of the environment. Many of the OSU workloads support the MPICH messaging protocols.
• New and modified framework definitions
  • Several new framework definitions have been added to Intel(R) Cluster Checker to aid in the validation and analysis of clustered environments. While others have been cleaned up or modified to enable a cleaner and better understanding of the frameworks. These new and modified framework definitions include the following:
    • health_base
    • health_user
    • health_exteded_user
    • health_admin
    • health_extended_admin
    • osu_*
  • To get a full listing of the available framework definitions execute the following on the command line: 'clck -X list'
  • For details about a specific framework definiton either refer to the reference section of the User's Guide or perform the following on the command line: 'clck -X <framework definition name>'
    • example: 'clck -X health_base'
• New Feature for Node Group Analysis and Reporting
  • Intel(R) Cluster Checker has introduced the ability to analyze discrete groups of compute nodes. The new feature utilizes a command line option, -g, and an xml file designating the grouping of the compute nodes. The grouping could be based upon physical characteristics of the nodes, location, or other featuers. A compute node can belong in multiple node groups at the same time. The new feature only applies to the analysis of the nodes. Collection of the data for the specific framework definitions is across al designated nodes. Please see the User's Guide for more details on implementing this feature, the xml file setup and example, and interpretation of the analysis results.
• Support for Linux Environment Modules
  • Intel(R) Cluster Checker now supports the use of Linux environment modules for setting up the proper runtime environment. The environment modules support provide an alternative to the current setup requirements found in the clckvars.[sh|csh] script. Please see the User's Guide for addtional details and setup requirements.
• Improved Support for ksh and zsh shells
  • Intel(R) Cluster Checker has expanded the number of shell environments that can be utilized for the collection and analysis of clustered systems by supporting ksh (korn shell) and zsh (z-shell). Specific details, setup, and exemptions are included in the User's Guide for Intel(R) Cluster Checker.
• Propagation of Environment Variables
  • Intel(R) Cluster Checker has now added environment variable propagation with its default pdsh-based remote execution on compute nodes. This enables proper operation of of data collection on compute nodes within consistent environment settings.
  • A new command line option '-e' was added to turn off the environment variable propagation for backward compatibility.
• Changes to Default Temporary File Permissions
  • Intel(R) Cluster Checker has modified its default temporary file permissions to remove the "world-writable" flag on all files and the temporary directory used by Intel(R) Cluster Checker to store temporary files before analysis. This change is to provide better security to users and ensure only those authorized to access Intel(R) Cluster Checker results are allowed to do so. Intel(R) Cluster Checker still allows users and groups authorized to execute the tools to read and write files, but limits access by others.
• Update to checks on $TMPDIR
  • Intel(R) Cluster Checker has modified the check for file permissions to not require any 'world' permissions for $TMPDIR.
• Updates of 3rd Party Components to Correct CVEs
  • Intel(R) Cluster Checker leverages several third-party components to assist in the collection, analysis and generation of reports. To maintain security commitments to Intel(R) and our customers, Intel(R) Cluster Checker is reguarly scanned using industry standard tools for exposure of Common Vunerable Exploits in the third-party components. It is important for customers using Intel(R) Cluster Checker to maintain their implementations at the latest level to not only receive the latest improvements to the collection and analysis, but also to receive the most recent updates to these third-party components.

Linux* Release Notes

• Added feature to propagate all user environment variables to all nodes when using pdsh for remote execution, with the exception of the following system variables: $HOSTNAME, $HOST, $HOSTTYPE, $MACHTYPE, $OSTYPE,  $PMI_RANK, $PMI_SIZE, $PMI_FD, $MPI_LOCALRANKID, $MPI_LOCALNRANKS, $DISPLAY, $SHLVL, $_, $BASH_FUNC. This is the default setting.
  • To suppress propagation of any environment variables to any nodes when using pdsh add command line option ‘-e’ or 'export CLCK_TURN_OFF_ENV_PROPAGATION=true'
  • the framework definition 'environment_variables_uniformity' can be used to inspect a list of all environment variables found on all nodes. (clckdb -D <path>/clck.db –provider printenv)
• Correct paths for the Intel® MPI Benchmarks providers to locate IMB binaries and removed imb_wrapper in imb_pingpong as it was not being used.
• Modifications to the messages provided when *-data-missing is reported by the analyzer. This is to improve the recommended remedies.
• Fix to correctly check for Intel Python 3, dropping check for Intel Python 2 with Intel® Parallel Studio XE 2020 runtimes or later.
• Fix for false positive signs from clck-analyzer about missing libraries for offload. These libraries are no longer required.
• Fix to execution of the data collection for the framework definition "mpi_local_functionality".
• Fix to standalone execution of framework definition "infiniband_admin"
• Extended node groups feature to work on relative framework definition paths
• Improvements to several messages provided on 'data-is-too-old' or '*-data-missing-remedy' as reported by the analyzer.
• Corrected online documentation for hpcg_cluster framework definition.
• Improvement to the data collection of the "df" data provider.
• Added environment support for zsh and ksh in vars.sh script.
• Update SQLite to version 3.32.3 to fix:
  • CVE-2020-15358
• Update libxml2 to fix:
  • CVE-2020-24977
• Update zeromq to version 4.3.3 to fix:
  • CVE-2020-15166

Linux* Release Notes

• Further enhancements and improvements to the console output and reports were made including the following:
  • Additional details are provided when using the -n / --include-node analyzer filter parameter.
  • The output formatting reverts to previous versions when the new beta feature for node group analysis is not activated, -g / --groupfile.
  • When using the new beta feature for node group analysis filtering, the ordering of groups will be alphabetical. Numerics will be treated as a string value and generated node groups will be printed last.
• Further improvements to the node group feature output were made including the following:
  • Better debugging and error messages for configuration set up.
  • Removed superfluous spacing.
  • Draw more attention to the line listing any runtime issues.
  • Removed duplicated information in node group specific output and added descriptors on the node group sections.
• Replace example node group configuration file with a better example.
• The '-h' help flag output has been revised
  • The '-n' flag has a changed description in the help output. Provided examples in the man pages.
• Better handling of imb and osu providers.
• Added a configurable timeout to collect extensions.
  • Timeout can be set as an environment variable or in the main configuration file.
  • Also documented the default timeout value in the configuration file.
• Added linking via RPATH:
  • Distributed common libraries moved to a seperate 'lib_common' folder.
• Update SQLite to version to fix:
  • CVE-2020-11655 and CVE-2020-11656
  • CVE-2020-13434 and CVE-2020-13435
  • CVE-2020-13630, CVE-2020-13631 and CVE-2020-13632

Linux* Release Notes

• Extensive changes made to the formatting of the output to enhance readability and parsing of the analysis.
  • Grouping of issues by type; functionality, performance, uniformity
  • Summary report on the number of each type of issue
  • Separation of Cluster Checker execution issues from system environmental issues
  • Detailed analysis and recommendations are detailed in an analysis log file
• An option to redirect the generated reports from Cluster Checker into a JSON formatted file to enable separate reporting and analysis by user tools
  • To enable, include clck_json as part of the block of the cluster checker configuration XML file (default file: $INSTALL_PATH/etc/clck.xml)
• Extended the collection and analysis capabilities of Intel® Cluster Checker to include the OSU Micro-Benchmark Suite (must be downloaded and installed separately: https://mvapich.cse.ohio-state.edu/benchmarks/).
  • The OSU Collectives Benchmark Suite includes point-to-point, blocking and non-blocking collectives benchmark tests that verifies the functionality of the MPI functions.
• Introduction of new Intel® MPI blocking and non-blocking collectives bench-mark tests for the verification of functionality of Intel® MPI. The results are collected and analyzed.
• Introduction of a new beta feature to perform analysis on specific node groups defined by the user or system administrator in a nodegroup file. This beta feature only applies to the analysis of the nodes, not the collection of data on the nodes. An example node group file is available in /etc/example_group_file.xml. The documentation provides further details on this new beta feature.
• CVE fixes for libxml2 - CVE-2020-7595 and CVE-2019-20388

Linux* Release Notes

• Fix to correctly identify Intel Python 3.7.4.
• Improved error messaging covering:
  • when no high speed fabrics are detected for MPI.
  • incorrect detection during analysis for select solutions priv plus.
  • time out of collecting information using mpi.so collector.
• Common Vulnerabilities and Exposures (CVE) for SQLite3: Updated SQLite3 package to sqlite3.31.0 to address a group of CVEs identified in late 2019.

Linux* Release Notes

• Added environment modules support for Intel® Cluster Checker.
• Added patch to address SQLite CVE-2019-9937.
• Improved clarity and response output for certain checks.
• Improved handling and analysis of data from multiple databases. Analysis is performed on the most recent data collected when there exists multiple collections of the same data types.

Linux* Release Notes

• New default tests with faster execution.
• New predefined in-depth test sets made for user or admin specific analysis.
• Enhanced summary output lists brief facts on nodes and issues.
• Troubleshooting tests on prerequisites for Intel(R) MPI Library.
• Verifies the uniformity of the BIOS and management firmware settings for the Intel(R) Server Boards through Intel's System Configuration Utility (syscfg).
• Support for the latest Intel processors (Intel(R) Xeon(R) Platinum 9200 Processor Family).
• Support for validation of Intel(R) Select Solutions for Simulation & Visualisation.
• New -t option to set data age threshold.
• Bug fixes and improvements, CVE updates.

Linux* Release Notes

• Enhanced functionality for testing memory uniformity.
• Added flexibility on checks for memlock limits to InfiniBand and Intel® Omni-Path Architecture (Intel® OPA) checks.
• Improved support for diskless clusters.
• Improved messages and bug fixes.

Linux* Release Notes

• Added support for checking of second-generation Intel® Xeon® Scalable processors by privileged or non-privileged users.
• Updated support for validation of Intel® Select Solutions for Simulation and Modeling to include the second-generation Intel® Xeon® Scalable Processor solution.
• Added support for checking Intel® Optane™ DC persistent Memory configurations and uniformity.
• Included support for second-generation Intel Xeon Scalable processor with the Intel HPC Platform Specification.
• Added checking for the Intel® Parallel Studio XE 2019.0 runtimes.

Linux* Release Notes

• Updated support for validation of Intel® Select Solutions for Simulation and Modeling.
• Added in support for Intel HPC Platform Specification 2018.0.
• Intel® Cluster Checker 2019 Update 2.1 includes functional and security updates. Users should update to the latest version.

Linux* Release Notes

• Intel® Cluster Checker 2019 Update 2 includes functional and security updates. Users should update to the latest version.

Linux* Release Notes

• New clck command simplifies execution with a single command.
• Added improved output messaging:
  • New compact summary output provided on screen.
  • Details of analysis provided in the output logfile.
  • Simplified scheme to assess issues as CRITICAL, WARNING, or INFORMATIONAL
  • Added –R option for specifying where results are written.
  • Changed -o option to specify where log output is written.
• Added performance threshold checking for Intel® Xeon Phi™ processor x205 product family.
• Added -X command line option, allowing a user to obtain a list of available framework definitions and their respective descriptions on data collected and analysis tests.
• Added the ability to mark two snapshots of a cluster state to identify changes. Currently supported with the following framework definitions: rpm_snapshot, hardware_snapshot, files_snapshot.
• Added a user option to collect any missing or old data before analysis.
• Added ability to collect data on a cluster that does not have pdsh if the Intel® MPI Library is installed.
• Added the ability to collect data without specifying a node file if nodes are allocated through SLURM.
• Added in support for validation of Intel Select Solutions for Simulation and Modeling.
• New Intel® Cluster Checker API.

Linux* Release Notes

• Resolved an issue with running lshw without privileged access.

Linux* Release Notes

• Added support for Intel® Select Solutions Framework Definitions.

Linux* Release Notes

• Enabled provider configuration as part of Framework Definitions.
• Enabled latest Intel® Xeon® processors.
• Renamed predefined Framework Definition names to be more informative.
• Added support for the SGEMM benchmark.
• Improved memory checking.
• Enhanced user visible message output.
• Added checks for Intel® Omni-Path Fabric (Intel® OP Fabric) subnets.
• Bug fixes.

Linux* Release Notes

• Added support for Intel® Xeon® Scalable processors.
• Added Framework Definition feature to allow for customization of analysis.
• Added support for Intel® Turbo Boost Technology validation.
• Added support for analysis from multiple database sources.
• Updated samples and SDK.
• Converted documentation to online format.
• Enhanced Intel® Omni-Path Architecture (Intel® OPA) validation.
• Added OpenFabrics Interfaces support.
• Enhanced user viewable message output.
• Bug fixes

Linux* Release Notes

• Additional support for Intel® Xeon Phi™ product family x200 processors.
• Additional support for Intel® Omni-Path Architecture.
• Bug fixes.

Linux* Release Notes

• Added Intel® Scalable System Framework support.
• Deprecated support for Intel® Cluster Ready.
• Added additional support for Intel® Xeon Phi™ product family x200 processors.
• Removed heartbeat functionality.
• Bug fixes.

Linux* Release Notes

• Added support for Intel® Xeon Phi™ product family x200 processors.
• Added support for the Lustre* file system.
• Significantly improved analysis performance for large databases.
• Extended the API to provide the ability to collect data.
• The separate analyzer (clck.xml) and data collection (clckd.xml) configuration files were merged into a single file (clck.xml). Previous configuration files are not compatible.
• Allows more granular suppressions. See the section on suppressions in the User's Guide for more information.
• The asynchronous data collection daemons, clckd and clck-serverd, have been replaced by a plugin to the Open Resilient Cluster Manager* (ORCM). See orcm/README for more information. The ORCM plugin is a technical preview feature; please see the Known Limitation section for its current limitations.
• Databases from previous versions of the product are incompatible with version 2017 due to database schema changes.
• The samples have been moved online and are no longer distributed as part of the product. They are now available at Intel® Software Development Products Samples and Tutorials