A CnC program run can be traced with Intel(R) Trace Collector (ITC) and post-morten analyzed with Intel(R) Trace Analyzer (ITA) (http://software.intel.com/en-us/intel-trace-analyzer)
The CnC header files are instrumented with hooks into ITC; when compiling with CNC_WITH_ITAC being defined the compiler will insert calls into ITC. Those work in shared and distributed mode, with distCnC you'll not only see the step-execution but also the communication. If you construct your collections with names (string parameters to their constructors) then the instrumentation will use those names and create a meaningful trace without further manual instrumentation.
The CnC runtime will initialize ITC automatically. In shared memory this will happen when the first context is created, in distributed memory right when dist_cnc_init is created. You can call VT_INIT earlier in the code if you want to to see ITC things before the first context creation (shared memory).
Please see Instrumenting with/for ITAC on how to instrument your code further.
After instrumentation/recompilation the execution of your applicaton "app" will let ITC write a tracefile "app.stf". To start analysis, simply execute "traceanalyzer app.stf".
For a more convinient file handling it is recommended to let ITC write the trac in the "SINGLESTF" format: simply set VT_LOGFILE_FORMAT=SINGLESTF when runnig your application.
Usually CnC codes are threaded. ITC will create a trace in which every thread is shown as busy from it's first acitivity to its termination, even if it is actually idle. To prevent this set VT_ENTER_USCERCODE=0 at application runtime.
The switch "-trace" to mpi[exec|run|cc] does not work out-of-the-box. However, if using the instrumented version ITC will load automatically. For this, compile with -DCNC_WITH_ITAC and link against ITAC (see Compiling and linking with ITAC instrumentation). To load ITC in a "plain" MPI environment (without recompiling or -linking), just preload the following libraries:
or in debug mode
Apparently you need a working isntallation of ITAC. When compiling add the include-directory of ITAC to your include path (-I$VT_ROOT/include). When linking, add the slib directory to the lib-path and link against the following libraries