Introducing...Concurrent Collections for distributed memory systems (distCnC)

Intel has just released Intel® Concurrent Collections for C++ update 0.5. One of the interesting features in this update is an optional extension/plugin to the “normal” runtime system that allows the CnC graph to be executed on distributed memory systems. This article shows you how to enable your CnC program to run on a distributed memory system.

In principle, any correct shared memory CnC program should also run correctly with distCnC. The mechanics of data distribution etc. are handled almost entirely inside the distCnC runtime and the programmer does not need to be concerned with the details. Of course, there are a few minor changes needed to make a program distCnC-ready, but once that's done, it will run on distributed CnC as well as on the shared memory model ("normal") CnC.

A sample program using distCnC, mandel, is included in the 0.5 Cnc update.

Inter-process communication

Conceptually, distCnC allows data and computation distribution across any kind of network; but currently only a socket-based communication model is provided. In the future, MPI-based communication model might be supported.

Linking for distCnC

distCnC is included in the 0.5 CnC update: the download comes with the necessary communication libraries (cnc_socket). The communication library is loaded on demand at runtime, hence you do not need to link against extra libraries to create distCnC-ready application. Just link your binaries like a shared memory model CnC application. Details about linking are available in the CnC User Guide.

Making your program distCnC-ready

1. Include the distributed version of CnC headers

A distributed version of a CnC program needs to #include a different version of the CnC headers. If you want to be able to create optimized binaries for shared memory and distributed memory from the same source, you might consider protecting distCnC specifics like this:

#ifdef _DIST_
# include "cnc/dist_cnc.h"
# include "cnc/cnc.h"

2. Modify the main procedure to initialize all CnC contexts used in the program.
In "main", initialize an object CnC::dist_cnc_init before anything else; parameters should be all context-types used in the program:

#ifdef _DIST_
CnC::dist_cnc_init _dinit;

Please note:

Special handling is needed if your items and/or tags are non-standard data types, in this case, the compiler will notify you about the need for serialization.. See the runtime_api document for more information about serialization.

Step instances are distributed across clients and the host. By default, they are distributed in a round-robin fashion. Note that every process can put tags (and so prescribe new step instances). The round-robin distribution decision is made locally on each process (not globally).

If the same tag is put multiple times, it might be scheduled for execution on different processes and the preserveTags attribute of tag_collections will then not have the desired effect. See the runtime_api document for more information about how to provide a tuner to control the distribution strategy.


• Pointers as tags are not yet supported by distCnC. However, it is possible to implement a serializable wrapper for pointers.
• Global variables must not be used within the execution scope of steps.
• "Global" attributes of collections (e.g. size(), iteration, etc.) must not be used while steps are being executed (e.g. within the dynamic scope of step-code).

Special host and client start-up requirements for using distCnC over sockets

A sample distCnC startup script is provided in the 0.5 CnC update to start the distributed host and client applications. You need to modify the startup script to set the number of clients and replace "localhost" with the names of the machines you want the application(-clients) to be started on. On Linux, it requires password-less login via ssh. It also gives some details of the start-up procedure. For windows, the script "start.bat" does the same, except that it will start the clients on the same machine without ssh or alike. Adjust the script to use your preferred remote login mechanism.

Concurrent Collections for C++ is under active development at Intel and we are looking for users in the field to try it out and give us some feedback, while it is still in the experimental stage. The download is free.