A CnC program run can be traced with Intel(R) Trace Collector (ITC) and post-morten analyzed with Intel(R) Trace Analyzer (ITA) (http://software.intel.com/en-us/articles/intel-trace-analyzer/)
The CnC header files are instrumented with hooks into ITC; when compiling with CNC_WITH_ITAC being defined the compiler will insert calls into ITC. Those work in shared and distributed mode, but most (if not all) of the provided instrumentations is for distCnC.
While the runtime provides ITAC versions of the different communication libraries you have to call VT_initialize manually to enable ITC without distCnC. Everything else is handled by the runtime/compiler.
After instrumentation/recompilation the execution of your applicaton "app" will let ITC write a tracefile "app.stf". To start analysis, simply execute "traceanalyzer app.stf".
For a more convinient file handling it is recommended to let ITC write the trac in the "SINGLESTF" format: simply set VT_LOGFILE_FORMAT=SINGLESTF when runnig your application.
Usually CnC codes are threaded. ITC will create a trace in which every thread is shown as busy from it's first acitivity to its termination, even if it is actually idle. To prevent this set VT_ENTER_USCERCODE=0 at application runtime.
The switch "-trace" to mpi[exec|run|cc] does not work out-of-the-box. However, if using the instrumented version ITC will load automatically. For this, compile with -DCNC_WITH_ITAC and link against ITAC (see Compiling and linking with ITAC instrumentation). To load ITC in a "plain" MPI environment (without recompiling or -linking), just preload the following libraries:
env LD_PRELOAD="libcnc.so:libcnc_mpi_itac.so:libVT.so" env DIST_CNC=MPI mpiexec -n ...
or in debug mode
env LD_PRELOAD="libcnc_debug.so:libcnc_mpi_itac_debug.so:libVT.so" env DIST_CNC=MPI mpiexec -n ...
Apparently you need a working isntallation of ITAC. When compiling add the include-directory of ITAC to your include path (-I$VT_ROOT/include). When linking, add the slib directory to the lib-path and link against the following libraries