Papi on Xeon Phi?

Papi on Xeon Phi?


I was trying to use PAPI on Xeon Phi. The papi bins for mic can be found in 

Then I compiled my programt for mic using:

icpc -mmic -O3 -o run-mic program.cpp -openmp -Wall -Werror -ansi-alias -ip -opt-subscript-in-range -opt-prefetch=4 -fomit-frame-pointer -funroll-all-loops -restrict -vec-report -parallel  -lpapi -lpfm

When I run the executable I got the following error:

Error: @€

I do not know what does that mean!

Is there anyone who can help me with this?


7 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Your compiler invocation contains -mmic, indicating cross compilation, but nothing in your note indicates what you did to get the resulting execution file or any libraries associated with PAPI onto the coprocessor for native execution.  Native execution WAS what you were intending, yes?  If you could share more details, someone could probably provide more help.

Thanks Robert.

I have uploaded a sample program in

To compile for CPU, I used

icpc -O3 -o fwp-cpu FW_papi.cpp -Wall -Werror -ansi-alias -ip -opt-subscript-in-range -opt-prefetch=4 -fomit-frame-pointer -funroll-all-loops -restrict -vec-report -parallel -I$TACC_PAPI_INC -L$TACC_PAPI_LIB -DUSEPAPI -lpapi -lpfm -DTEST -DTIMEIT

To run it use:

fwp-cpu >>out-cpu.txt

Alternatively, for Xeon Phi:

icpc -mmic -O3 -o fwp-mic FW_papi.cpp -Wall -Werror -ansi-alias -ip -opt-subscript-in-range -opt-prefetch=4 -fomit-frame-pointer -funroll-all-loops -restrict -vec-report -parallel -I$TACC_PAPI_INC -L$TACC_PAPI_LIB -DUSEPAPI -lpapi -lpfm -DTEST -DTIMEIT fwp-mic >>out-mic.txt

It runs for CPU propely, but gives error 5 or other errors.


I think you might spend some time with our documentation to understand better the models for programming in the presence of our coprocessor.  You have provided above more detail about the compilation process but little more about how you execute.  However the hints may be present to help understand your predicament.  Given those compilations described above, you would end up with two execution modules, fwp-cpu, a normal compile that should work as is, invoked on the host by a command line issued on the host.

However, the line for fwp-mic is the equivalent for cross-compiling the code for standalone execution on the coprocessor.  If you were then to try to execute that execution module by the same means as the host version, the host program loader will not recognize the Intel Xeon Phi coprocessor native execution code, and may blow up in many interesting ways.  This execution module and any nonstandard shared libraries it needs for execution on the coprocessor NEED TO BE TRANSPORTED TO THE COPROCESSOR to execute in native mode. That means something like the following (though there are many ways to get the execution file where it needs to be for successful execution):

% scp fwp-mic `hostname`-mic0:/tmp
% scp .... # more copies to move libraries like papi to appropriate locations within the coprocessor virtual file system
% ssh `hostname`-mic0 /tmp/fwp-mic    # remotely launches fwp-mic on the coprocessor

If data are required for execution then they also may need to be copied to the coprocessor virtual file system, or made available to the coprocessor via NFS or other file sharing means.  Trying to run a cross-compiled coprocessor program by invoking it directly on the host is bound to lead to frustration.    I encourage you to spend more time with our documentation to understand how the shared execution model works for the Intel Xeon Phi coprocessor.

On the TACC Stampede system we have the host systems set up to run (most) Xeon Phi native binaries transparently.  (We use a "magic number" hack to identify Xeon Phi binaries, then use the "micrun" script to grab the current directory and environment and reproduce it on the Xeon Phi as the arguments to an ssh command.)

But even though we have a "papi-mic" module on the Stampede system, it does not appear that it is set up correctly.  I get failures in the "PAPI_library_init" call that seem to indicate a version mismatch.

Several of the PAPI example codes attempt to print an uninitialized string when they see this error, which could account for the strange output. 
Modifying the code to print the PAPI_library_init return code instead shows "-1", which corresponds to PAPI_EINVAL, which is set in PAPI_library_init if the value of the preprocessor variable PAPI_VER_CURRENT does not match the value that was used when the library was compiled.

John D. McCalpin, PhD
"Dr. Bandwidth"

In my last reply I did consider whether it was worth mentioning this capability, which I've seen discussed in other forums.  However, at this point in this thread I didn't want to confuse the issue by revealing such possibilities.  I'm still trying to explore how much Jahan understands about coprocessor execution basics.  

Thanks to Robert and John.

@Robert. I ran fwd-mic on mic in native mode. I did not run fwd-mic on host!

I think John has observed similar things in his experiments. Thanks for sharing. May be installing a local version of papi-mic will help.

Best Regards,


Leave a Comment

Please sign in to add a comment. Not a member? Join today