SANDY BRIDGE ADDRESS MAPPING SCHEME

SANDY BRIDGE ADDRESS MAPPING SCHEME

Hi,

    I am investigating the memory address mapping scheme in Sandy Bridge MC.

    My platform has two sockets and total system RAM of 64 GB. 

    My question is regarding experimental values obtained from memory controller PCI registers

    which help in system address decoding by the MC

    Below are the readings of DRAM RULE register & DRAM INTERLEAVE register

###################################
  PCI DEVICE 12, FN : 0 BUS 1 (There are 10 DRAM Rules)
###################################

DRAM RULE REGISTER 0 (0x82c3)
--------------------------------

LIMIT                                      :       0x20b (64 MB Granularity )  ~ 32 GB
ATTRIBUTE FOR DRAM RULE :       DRAM
INTERLEAVE MODE                :       0    Address Bits <8,7,6> ^ Address Bits <18,17,16>
RULE_ENABLE                        :       1

DRAM INTERLEAVE LIST 0 REGISTER (0)
---------------------------------

PACKAGE 8               :       0
PACKAGE 7               :       0
PACKAGE 6               :       0
PACKAGE 5               :       0
PACKAGE 4               :       0
PACKAGE 3               :       0
PACKAGE 2               :       0
PACKAGE 1               :       0

DRAM RULE REGISTER 1 (102c3)
--------------------------------

LIMIT                                      :       0x40b (64 MB Granularity)  ~ 64 GB
ATTRIBUTE FOR DRAM RULE :       DRAM
INTERLEAVE MODE                :       0  Address Bits <8,7,6> ^ Address Bits <18,17,16>
RULE_ENABLE                        :       1

DRAM INTERLEAVE LIST 1 REGISTER (0x9090909)
-----------------------------------------

PACKAGE 8               : 1
PACKAGE 7               : 1
PACKAGE 6               : 1
PACKAGE 5               : 1
PACKAGE 4               : 1
PACKAGE 3               : 1
PACKAGE 2               : 1
PACKAGE 1               : 1

All the rest registers are diabled.

The DRAM Interleave Register for First 32 GB has only 0 entries. I beleive it indicates socket No 1.

The DRAM Interleave Register for the next 32 GB (till 64 GB ) has only 1 as entry. I believe it indicates socket No 2.

Does this mean that there is no interleaving across Nodes for either the first 32 GB or the next 32 GB  ?

However If I use the Performance Monitor Software, I observe interleaving across nodes even within the first 32 GB.

I am not able to explain the results.

Any suggestions are welcome.

Thank you

2 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Hello Saptarshi,

I'm not familiar with all the dram setup registers but I have a question. Is NUMA enabled?

If NUMA is enabled, then (if I understand correctly) the memory attached to node 0 should all belong to node 0 and node 1's memory belongs to node 1.

If NUMA is not enabled then the memory is interleaved at the cacheline level.

Even if NUMA is enabled, depending on how you run (for example, if you are not pinned to the node on which you allocate memory) you can still see cross node memory accesses.

Pat

Leave a Comment

Please sign in to add a comment. Not a member? Join today