Each core has its own L1 and L2 caches, and makes accesses from them. So in the case you discuss some line will be evicted from the local cache to make room for the newly required data.

Similarly, if another core later requires the same line it will also make space for it in its local cache, and then bring a copy to there (from one of the other L2s if it's already present on chip). So the same, unmodified, data can be present in each of the L2 caches on the chip.

The way to think of the machine is that each core has its own cache, and that all those caches are maintained coherent, not that there is one, large, shared L2 cache. (People familiar with Xeon, which does have a large, shared, L3 cache sometimes say that the Intel(r) Xeon Phi(tm) coprocessor "doesn't have a last-level cache", which is clearly wrong by definition :-), but they are right that there is no shared last level cache).

I know of no way to change the hardware cache policies.

However, you can use the compiler's nontemporal hint to ask it to attempt to avoid placing some variables in the cache or, if you're using assembler code the streaming store operations yourself. This discusses when the compiler chooses to use those itself, and shows you the intrinsics if you want to go to that level.

The nontemporal hint is much more frequently useful on MIC than on host, so you may require e.g.

#if defined __MIC__

#pragma vector [aligned] nontemporal

#endif

so as to instruct the compiler to use this for MIC only.  The compiler should be smart enough to apply nontemporal only to eligible arrays in the designated loop, but it's the programmer's responsibility to find out whether immediate cache eviction is wanted.  We've seen cases where this speeds up MIC KNC by 30% but if applied on host slows it down by as much as 10x.

I suspect pragma nontemporal doesn't work with CEAN array assignments.

One more point is that you can also use the _mm_clevict intrinsic to force data out of the cache.