Disabling instruction cache snooping on the Intel® Xeon Phi™ Coprocessor

Warning: The following documents a control flag users may experiment with; however, it's not an official feature supported by the Intel® Xeon Phi™ Coprocessor.

What is instruction cache snooping?
Instruction cache (Icache) snoop is used for invalidating forbidden sharing of data and instruction cachelines to properly handle Self Modifying Code (SMC) and Cross Modifying Code (CMC). When instructions are being evicted by data in the L2 cache, because the L1 cache is inclusive, it is should also get evicted from the L1 Icache. Disabling the snoop, however, means that if the instructions are evicted from L2, no snoop of the L1 will happen to see if those lines are also present there, and therefore the instructions do not get evicted from the L1 Icache losing coherency and potentially opening up for SMC to pass data for instructions.

Why disable instruction cache snooping on Intel Xeon Phi?
The Icache snoop implementation in Intel Xeon Phi is not optimal performance-wise ("slow/er") so disabling it can potentially gain some 10% performance from bypassing the check for L1 Icache coherency. Codes may exist to trip precisely the purpose for its existence causing unknown behavior. On Xeon, Icache snoop is always enabled and this is the default behavior for IA. Since Intel Xeon Phi is supposed to align with the IA default, disabling Icache snoop in Intel Xeon Phi is not a supported feature and has not undergone full validation. If disabling Icache snoop is solely limited to running only a specific application, and the user takes the effort to validate their results, then the downsides are unlikely. 

The bottom line remains -- "caveat emptor."

To disable Icache snooping on Intel Xeon Phi:

  1. Add icache_snoop_off to the ExtraCommandline configuration variable located in /etc/sysconfig/mic/default.conf.  The contents of ExtraCommandline are passed as boot parameters to the Linux kernel that is booted on the Intel Xeon Phi.  
  2. Reboot Xeon Phi for the new parameter to take effect: micctrl --reboot

That's it.  Now your Xeon Phi is booted with the icache_snoop_off flag.  You can verify this by reading /proc/cmdline on the device.  To put it back, simply remove it from your config file and reboot the device again.