Intel® Vtune™: “Error connecting to MIC card”, Why?

Recently, while profiling a workload on the Intel® Xeon Phi™ coprocessor using Intel Vtune, I ran across the following error:

Quote:

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection…

ERROR connection to MIC card <1:1046>, errno 111

 

Before I tell you how to get rid of this error, let’s take a step back and root cause it. Most of you might know that Intel Vtune profiles your application by periodically collecting the hardware event data captured by the Performance Management Units (PMUs) located deep within the CPU’s hardware. To enable Intel Vtune to directly access these hardware units, a part of the analyzer needs to have kernel privileges. We commonly know this part of Intel Vtune as the SEP driver. In the case of the Intel Xeon Phi coprocessor, this driver needs to be up and running on the coprocessor for the collection to succeed. Generally, we don’t encounter this error because this driver is automatically loaded every time you start the Intel® Many Core System Stack (Intel MPSS).  However, sometimes the Intel MPSS fails to load the SEP driver, resulting in the above error. So in essence, all that Intel Vtune is telling you to do is to load the SEP driver on the coprocessor.

This error can be fixed by trying the following workarounds.

Workaround# 1:

You can try the simplest workaround first: Restart the Intel MPSS and see if it can successfully load the driver and thus solve the problem.  

$service mpss restart
$micctrl –w

Workaround# 2:

If workaround 1 doesn’t work then the next possibility is that your configuration files are not set up correctly and hence every time your Intel MPSS starts, it does not or cannot load the SEP driver. You can fix the SEP configuration files by going through the following steps:

$cd <install_dir>/bin64/k1om
$./sep_micboot_install.sh
$service mpss restart
$micctrl –w 
 

Workaround# 3:

In the worst case, if the above workarounds still don’t fix the problem then it is possible that the SEP driver configuration files for the Intel Xeon Phi coprocessor were never generated or were generated incorrectly or just got messed up somehow (just the way most things find a way to get messed up!).  The way to fix this would be to clean the SEP driver and set up the configurations files again:

$ cd  <vtune_install_dir>/bin64/k1om
$ sudo ./sep_micboot_uninstall.sh  --clean
$ sudo ./sep_micboot_create.sh
$ sudo ./sep_micboot_install.sh
$ sudo  service mpss restart
$micctrl –w 

* VTune is a trademark of Intel Corporation in the U.S. and/or other countries.

** Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries

For more complete information about compiler optimizations, see our Optimization Notice.

4 comments

Top
Surya Narayanan N.'s picture

Am using mic in network mode. So after the admin restarts the PC i am able to run my test program 

amplxe-cl -collect knc-general-exploration  ssh mic0 "/home/snataraj/test/nsleep_mic"

after that, when i try to run some of my benchmark something goes wrong and my vtune fails to connect to mic0.

If I try my previous sample program again I get the error

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.
amplxe: Internal Error

 

And I don't have admin rights to restart mpss service. What kind of modification should my admin make in my account to get rid of this issue?

Surya Narayanan N.'s picture

Am using mic in network mode. So after the admin restarts the PC i am able to run my test program 

amplxe-cl -collect knc-general-exploration  ssh mic0 "/home/snataraj/test/nsleep_mic"

after that, when i try to run some of my benchmark something goes wrong and my vtune fails to connect to mic0.

If I try my previous sample program again I get the error

ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection...
ERROR connecting to MIC card <1:1046>, errno 111amplxe: Error: Problem accessing the sampling driver. The driver may need to be (re)started. See Installing the Sampling Driver help topic to learn how to configure the sampling driver.
amplxe: Internal Error

 

And I don't have admin rights to restart mpss service. What kind of modification should my admin make in my account to get rid of this issue?

lcicca's picture

Hi,
I've tried all the workarounds but I'm still having the same problem. Does anybody know how can I verify the MIC status? I'm sure the MIC is there as I can SSH it.
Thanks in advance.

a.kaliazin's picture

Hi
Workaround #3 is bit outdated:

# ./sep_micboot_create.sh
This script has been deprecated.
Please directly run "sep_micboot_install.sh" to install the configuration files.

But the rest - did work for me, with an additional effort: before starting up the mpss I had to rebuild the mic*.image files using 'micctrl --updateramfs' as I have done 'micctrl --rootdev=StaticRamFS' before.

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.