Intel® Vtune™: “Error connecting to MIC card”, Why?

Recently, while profiling a workload on the Intel® Xeon Phi™ coprocessor using Intel Vtune, I ran across the following error:


ERROR connecting to MIC card, make sure sep_mic_server is running on the card, retrying connection…

ERROR connection to MIC card <1:1046>, errno 111


Before I tell you how to get rid of this error, let’s take a step back and root cause it. Most of you might know that Intel Vtune profiles your application by periodically collecting the hardware event data captured by the Performance Management Units (PMUs) located deep within the CPU’s hardware. To enable Intel Vtune to directly access these hardware units, a part of the analyzer needs to have kernel privileges. We commonly know this part of Intel Vtune as the SEP driver. In the case of the Intel Xeon Phi coprocessor, this driver needs to be up and running on the coprocessor for the collection to succeed. Generally, we don’t encounter this error because this driver is automatically loaded every time you start the Intel® Many Core System Stack (Intel MPSS).  However, sometimes the Intel MPSS fails to load the SEP driver, resulting in the above error. So in essence, all that Intel Vtune is telling you to do is to load the SEP driver on the coprocessor.

This error can be fixed by trying the following workarounds.

Workaround# 1:

You can try the simplest workaround first: Restart the Intel MPSS and see if it can successfully load the driver and thus solve the problem.  

$service mpss restart
$micctrl –w

Workaround# 2:

If workaround 1 doesn’t work then the next possibility is that your configuration files are not set up correctly and hence every time your Intel MPSS starts, it does not or cannot load the SEP driver. You can fix the SEP configuration files by going through the following steps:

$cd <install_dir>/bin64/k1om
$service mpss restart
$micctrl –w 

Workaround# 3:

In the worst case, if the above workarounds still don’t fix the problem then it is possible that the SEP driver configuration files for the Intel Xeon Phi coprocessor were never generated or were generated incorrectly or just got messed up somehow (just the way most things find a way to get messed up!).  The way to fix this would be to clean the SEP driver and set up the configurations files again:

$ cd  <vtune_install_dir>/bin64/k1om
$ sudo ./  --clean
$ sudo ./
$ sudo ./
$ sudo  service mpss restart
$micctrl –w 

* VTune is a trademark of Intel Corporation in the U.S. and/or other countries.

** Intel, Xeon, and Intel Xeon Phi are trademarks of Intel Corporation in the U.S. and/or other countries