Our program is multithreaded. However, only one thread is performance critical. So what we do is use the "isolcpus" kernel parameter to isolate cores from the kernel process scheduler. We launch the application as normal, but programmatically pin the performance-critical thread to one of the isolated cores (pthread_setaffinity_np()).

Because we are only interested in the performance of that one thread (and not the program as a whole), we are not using pcm.x. Instead, we modified our code to collect PCM stats on just that one thread. We didn't collect absolute cache misses, only hit ratios, and cpu cycle lost ratios.

The way we are looking at it, we assumed that a low hit ratio would imply a high number of absolute misses. The programs and OS config (save the kernel command line) are identical. Is it incorrect to assume the caching behavior should be similar on the two machines? If so, why?

I'll add that, along with the lousy 17% L3 hit ratio, we also see nearly 40% of CPU cycles lost due to L3 cache misses.

Versus Westmere, where we had 73% L3 hit ratio, and only 8% of CPU cycles lost due to L3 misses.

We can further modify our code to include the absolute number of cache misses. But in the meantime, I'm interested in knowing what additional information that will provide. (Not trying to be argumentative, these are honest questions!)

Thanks!

Hi, we modified our program to collect additional PCM stats. We don't use pcm.x because we are only concerned with the performance of one thread (which is running on an isolated core).

Here is one run of SNB versus WSM:

Sandy Bridge:
EXEC: 0.7332
IPC: 0.6451
FREQ: 1.1366
AFREQ: 1.1379
L3MISS: 89948
L2MISS: 164103
L3HIT Ratios: 0.4800
L2HIT Ratios: 0.6134
L3CLK: 0.1474
L2CLK: 0.0327

Westmere:
EXEC: 0.5902
IPC: 0.5594
FREQ: 1.0550
AFREQ: 1.0555
L3MISS: 54319
L2MISS: 253901
L3HIT: 0.7625
L2HIT: 0.5025
L3CLK: 0.0755
L2CLK: 0.0591

Note that is for 1020 iterations of this thread. The same exact binary, same operating system, same CPU isolation scheme, same input data. Only difference is CPU architecture.

Sandy Bridge had 48% L3 hit ratio, 15% of CPU cycles lost due to L3 misses, and 90k L3 misses.
Westmere had 75% L3 hit ratio, 8% of CPU cycles lost due to L3 misses, and only 54k L3 misses.

Thanks!
Matt

By the way, one idea was that perhaps SNB's fancy new pre-fetching was simply not working for our particular code. The BIOS for my Sandy Bridge machine (Dell R620) has four options for pre-fetching:

Adjacent Cache Line Prefetch
Hardware Prefetcher
DCU Streamer Prefetcher
DCU IP Prefetcher

I repeated my test for every possible combination of these settings enabled/disabled (16 total combinations). With any one option disabled, the L3 hit ratios actually got worse still.

Quote:

Roman Dementiev (Intel) wrote:

This means SNB executes more instructions per second than the Westmere. But your program runs slower on SNB! This makes me think that on SNB you execute (slightly) different code path compared to Westmere. You really need a profiler to see what code is executed and hot both on Westmere and on SNB and compare.

This is a relatively short code path that we are studying: maybe a dozen or so screens worth of code. I really don't think an alternate code path is possible, as the program wouldn't be correct.

We have done profiling as you suggest. We put about a dozen rdtscp calls in this code path, stored them in a table, and output them at the end of execution. We were hoping to isolate one section of code that was particularly slow on SNB versus WSM. But, it was just a general overall slowness. All of our "stopwatch" points were just a little bit higher on SNB. In other words, the added execution time was for the most part evenly distributed.

Any other ideas?

Thanks,
Matt

Quote:

Roman Dementiev (Intel) wrote:

This is a relatively short code path that we are studying: maybe a dozen or so screens worth of code. I really don't think an alternate code path is possible, as the program wouldn't be correct.

does your code execute any spin locks or other thread synchronization primitives that may have different number of executed iterations in spin loops?

Yes, that is a good point that I didn't originally consider. We use Linux pthreads, which IIRC do use some spin locks "under the covers".

Also: this is an event-driven thread. That thread is spinning on a SysV message queue waiting for the next event. Note that I also tested without the busy event waiting: this has virtually no impact on the L3 hit ratio of Westmere, but drops the SNB L3 hit ratio even more.

Thank you, I appreciate your feedback.
-Matt