Quick Start Guide Part 3: Windows Provisioning of Intel® Optane™ DC Persistent Memory

Автор Joseph E Oster

Introduction

Intel® Optane™ persistent memory (PMem) is a disruptive technology that creates a new tier between memory and storage. Intel Optane™ PMem modules support two modes: Memory Mode for volatile use cases and App Direct Mode that provides byte-addressable persistent storage. 

In this portion of the Quickstart Guide to Provisioning Pmem, we describe some methods of provisioning from within Microsoft Windows Server OS, by providing a tutorial for configuring and managing Intel® Optane™ persistent memory (PMem) modules in Memory Mode, App Direct Mode or mixed-mode, using Windows Server PowerShell commands.

This quick start guide (QSG) has three sections:

  1. Quick Start Guide Part 1: Persistent Memory Provisioning Introduction
  2. Quick Start Guide Part 2: Provisioning for Intel® Optane™ Persistent Memory on Linux
  3. Quick Start Guide Part 3: Provisioning for Intel® Optane™ Persistent Memory on Microsoft Windows

Windows Server Support for Intel® Optane™ PMem

This section provides an overview of the software tools for provisioning PMem on Microsoft Server 2019.

Device Manager View

Device Manager provides a list of PMem devices available in the system.

Software: PowerShell cmdlets

Powershell has the following cmdlets for managing PMem. 

  • Get-PmemDisk
  • Get-PmemPhysicalDevice
  • New-PmemDisk
  • Remove-PmemDisk
  • Get-PmemUnusedRegion

Getting Help

For more information on managing PMem on Windows, see Understanding and Deploy PMem

Tutorial: Provisioning Intel Optane Persistent Memory

The tutorial provides instructions for configuring and managing Intel® Optane™ persistent memory (PMem) modules in App Direct Mode using ipmctl and use the PowerShell utility to finish the configuration.

All the commands described in this section are demonstrated on a two-socket system with six terabytes (TB) of configured PMem, and 384 gigabytes (GB) of DRAM installed.

This tutorial considers that an operating mode is configured already by the Original Device Manufacturer (ODM), which is common in new systems received by customers. Typically, we see systems configured in App Direct Mode or Memory Mode. 

Note: Ipmctl snippets and screenshots below are for ipmctl 2.0 

Prerequisites

  • Windows Server 2019
  • Intel Optane PMem Modules are installed
  • The ipmctl utility is used to configure and manage PMem modules. 
    • To determine if ipmctl is already installed, type ipmctl into the PowerShell window
    • If ipmctl is not installed, please download and refer to the instructions in the ipmctl README Tutorial

Step 1: PMem Discovery

This tutorial considers that an operating mode is pre-configured by the system provider. New systems are commonly pre-configured in App Direct Mode or Memory Mode based on the customer requirement. 

  1. ipmctl show -dimm command displays the PMem modules discovered in the system and verifies that software can communicate with them. Among other information, this command outputs each DIMM ID, capacity, health state, and firmware version. The PMem module firmware and BIOS releases are available from the system provider
    PS C:\Windows\system32> ipmctl show -dimm
    DimmID | Capacity    | LockState | HealthState | FWVersion
    ===============================================================
     0x0001 | 126.422 GiB | Disabled  | Healthy     | 01.02.00.5435
     0x0011 | 126.422 GiB | Disabled  | Healthy     | 01.02.00.5435
     0x0021 | 126.422 GiB | Disabled  | Healthy     | 01.02.00.5435
    ….
     0x1121 | 126.422 GiB | Disabled  | Healthy     | 01.02.00.5435

     

  2. ipmctl show -topology command identifies which memory slots the DDR and PMem modules are installed in​
    PS C:\Windows\system32> ipmctl show -topology
     DimmID | MemoryType                  | Capacity    | PhysicalID| DeviceLocator
    ================================================================================
     0x0001 | Logical Non-Volatile Device | 126.422 GiB | 0x0026    | CPU1_DIMM_A2
     0x0011 | Logical Non-Volatile Device | 126.422 GiB | 0x0028    | CPU1_DIMM_B2
     0x0021 | Logical Non-Volatile Device | 126.422 GiB | 0x002a    | CPU1_DIMM_C2
     0x0101 | Logical Non-Volatile Device | 126.422 GiB | 0x002c    | CPU1_DIMM_D2
     0x0111 | Logical Non-Volatile Device | 126.422 GiB | 0x002e    | CPU1_DIMM_E2
     0x0121 | Logical Non-Volatile Device | 126.422 GiB | 0x0030    | CPU1_DIMM_F2
     0x1001 | Logical Non-Volatile Device | 126.422 GiB | 0x0032    | CPU2_DIMM_A2
     0x1011 | Logical Non-Volatile Device | 126.422 GiB | 0x0034    | CPU2_DIMM_B2
     0x1021 | Logical Non-Volatile Device | 126.422 GiB | 0x0036    | CPU2_DIMM_C2
     0x1101 | Logical Non-Volatile Device | 126.422 GiB | 0x0038    | CPU2_DIMM_D2
     0x1111 | Logical Non-Volatile Device | 126.422 GiB | 0x003a    | CPU2_DIMM_E2
     0x1121 | Logical Non-Volatile Device | 126.422 GiB | 0x003c    | CPU2_DIMM_F2
     N/A    | DDR4                        | 32.000 GiB  | 0x0025    | CPU1_DIMM_A1
     N/A    | DDR4                        | 32.000 GiB  | 0x0027    | CPU1_DIMM_B1
     N/A    | DDR4                        | 32.000 GiB  | 0x0029    | CPU1_DIMM_C1
     N/A    | DDR4                        | 32.000 GiB  | 0x002b    | CPU1_DIMM_D1
     N/A    | DDR4                        | 32.000 GiB  | 0x002d    | CPU1_DIMM_E1
     N/A    | DDR4                        | 32.000 GiB  | 0x002f    | CPU1_DIMM_F1
     N/A    | DDR4                        | 32.000 GiB  | 0x0031    | CPU2_DIMM_A1
     N/A    | DDR4                        | 32.000 GiB  | 0x0033    | CPU2_DIMM_B1
     N/A    | DDR4                        | 32.000 GiB  | 0x0035    | CPU2_DIMM_C1
     N/A    | DDR4                        | 32.000 GiB  | 0x0037    | CPU2_DIMM_D1
     N/A    | DDR4                        | 32.000 GiB  | 0x0039    | CPU2_DIMM_E1
     N/A    | DDR4                        | 32.000 GiB  | 0x003b    | CPU2_DIMM_F1
    

     

  3. ipmctl show -memoryresources will report information about the current configuration if it has been configured. If no configuration has been created, ipmctl will return a message similar to "One or more PMem modules have invalid PCD data. A platform reboot is recommended to restore valid PCD data, then try again." In the output below, the MemoryCapacity and AppDirectCapacity values are used to determine if the system was configured in Memory mode, App Direct Mode, or mixed mode, here are a few examples:
    • If the system is in Memory Mode, you will see something like this (see Volatile row):
      PS C:\Windows\system32> ipmctl show -memoryresources
       MemoryType	| DDR         | PMemModule       | Total
      ==========================================================
       Volatile    	| 189.500 GiB | 1512.000 GiB 	| 1512.000 GiB
       AppDirect   	|           - | - 			    | -
       Cache       	| 0.000 GiB   | -            	| 0.000 GiB
       Inaccessible	| 2.500 GiB   | 5.066 GiB		| 7.566 GiB
       Physical    	| 192.000 GiB | 1517.066 GiB 	| 1709.066 GiB 
      

       

    • If the system is in AppDirect, you will see something like this (see App Direct Row): 
      MemoryType	| DDR         | PMemModule       | Total
      ==========================================================
       Volatile    	| 189.500 GiB | 0.000 GiB    	| 189.500 GiB
       AppDirect   	|           - | 1512.000 GiB 	| 1512.000 GiB
       Cache       	| 0.000 GiB   | -            	| 0.000 GiB
       Inaccessible	| 2.500 GiB   | 5.066 GiB		| 7.566 GiB
       Physical    	| 192.000 GiB | 1517.066 GiB 	| 1709.066 GiB 
      

       

    • If the system is in Mixed Mode (50/50), you will see something like this (see volatile and AppDirect rows):
      MemoryType	| DDR         | PMemModule       | Total
      ==========================================================
       Volatile    	| 189.500 GiB | 758.000 GiB 	| 758.000 GiB
       AppDirect   	|           - | 758.000 GiB     | 758.000 GiB 
       Cache       	| 0.000 GiB   | -           	| 0.000 GiB
       Inaccessible	| 2.500 GiB   | 5.066 GiB		| 7.566 GiB
       Physical    	| 192.000 GiB | 1517.066 GiB 	| 1709.066 GiB 
      

       

Step 2: Determine a Configuration Goal 

At this point, if your system is configured as you need it, consider the following step as just information. However, if you need to make changes to the configuration, this section explains the basics. If you need to change the platform configuration, such as Memory mode to App Direct Mode, see appendix A  on resetting the system.

When setting a goal, we set the Intel Optane PMem modules to be used in one of the three standard modes: Memory Mode, App Direct, and mixed. Once the mode is set, the system needs to be rebooted for the changes to go into effect.

The first thing you need to do is decide which of the three modes you require and run the appropriate configuration command

  • Memory Mode - 100% of PMem capacity across sockets can be provisioned in Memory mode, as described below. In this example, 100% of the available PMem capacity is to be provisioned in Memory mode. You can always use the -f option to overwrite any existing goal. The system must comply with recommended DRAM to PMem capacity ratios. At a minimum, a system must have at least a 1:4 ratio and no more than 1:16 per CPU socket. For example, a system with 192GB of PMem should have at least 768GB of PMem. A warning may be reported by ipmctl and the BIOS at boot time if the configuration does not meet the requirements.
     
  • App Direct Mode - PMem can be provisioned in App Direct Mode with interleaving enabled or disabled. As described in the QSG Guide Part 1,  interleaving increases the throughput of reads and writes to PMem.
     
  • Mixed Modes - A percentage of PMem capacity can be provisioned in Memory mode, as described below. Any remaining PMem capacity can either be reserved or used to create an AppDirect region. 

Step 3: Configure your Platform

At this point, we have three options, choose the sub-tutorial below to configure your system.

Memory Mode Example

In this example, 100% of the available PMem capacity is to be provisioned in Memory Mode. 

  1. Enter the following on the command line ipmctl create -goal memorymode=100
    PS C:\Windows\system32> ipmctl create -goal memorymode=100
    
    The following configuration will be applied:
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0001 | 502.0 GiB | 0.0 GiB        | 0.0 GiB
     0x0000   | 0x0011 | 502.0 GiB | 0.0 GiB        | 0.0 GiB
     0x0000   | 0x0021 | 502.0 GiB | 0.0 GiB        | 0.0 GiB
    ...
     0x0001   | 0x1111 | 502.0 GiB | 0.0 GiB        | 0.0 GiB
     0x0001   | 0x1121 | 502.0 GiB | 0.0 GiB        | 0.0 GiB

     

  2.  A reboot is required to process new memory allocation goals.

App Direct Example

In this example, we configure the system into App Direct Mode.

  1. Enter the following on the command line  ipmctl create -goal persistentmemorytype=appdirect
    PS C:\Windows\system32> ipmctl create -goal persistentmemorytype=appdirect
    
    The following configuration will be applied:
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0011 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
     0x0000   | 0x0021 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
     0x0000   | 0x0001 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
    ...
     0x0001   | 0x1121 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
     0x0001   | 0x1101 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
    Do you want to continue? [y/n] y
    Created following region configuration goal
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0011 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
     0x0000   | 0x0021 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
     0x0000   | 0x0001 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
    ...
     0x0001   | 0x1121 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB
     0x0001   | 0x1101 | 0.0 GiB    | 502.0 GiB      | 0.0 GiB

     

  2. A reboot is required to process new memory allocation goals.

Mixed Mode Example

In this example, 60% of the available PMem capacity is to be provisioned in Memory Mode.

  1. Enter the following on the command line ipmctl create -goal memorymode=60
    PS C:\Windows\system32> ipmctl create -goal Memorymode=60
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0011 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
     0x0000   | 0x0021 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
     0x0000   | 0x0001 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
    ...
     0x0001   | 0x1121 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
     0x0001   | 0x1101 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
    Do you want to continue? [y/n] y
    Created following region configuration goal
     SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size
    ==================================================================
     0x0000   | 0x0011 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
     0x0000   | 0x0021 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
     0x0000   | 0x0001 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
    ...
     0x0001   | 0x1121 | 310.0 GiB | 192.0 GiB      | 0.0 GiB
     0x0001   | 0x1101 | 310.0 GiB | 192.0 GiB      | 0.0 GiB

     

  2. A reboot is required to process new memory allocation goals.

Step 4: Provisioning of Intel Optane Persistent Memory Namespaces 

This step of the tutorial is for continuing the provisioning process of creating PMem disks (equivalent to namespaces in Linux) for App Direct Mode and mixed-mode. 

Now we need to create PMem disks to be used by an application(s). The disk size can be any % of the overall GiB capacity of the installed PMem modules. Please run the following commands to configure the system.

  1. Open Powershell as Administrator and enter the following commands
    1. Get-PmemUnusedRegion 
      • This command Is used to list the regions for PowerShell. If the output of this command is empty, no regions exist on the PMem modules. Recall that ipmctl is used to create regions. 
      • After creating regions using ipmctl and then rebooting, the newly created interleave sets are represented as PMem unused regions. These are the regions that are not assigned to a logical persistent memory device on the system
        PS C:\Windows\system32> Get-PmemUnusedRegion   
          RegionId    TotalSizeInBytes    DeviceId
          ----------  -----------------  ------------
             1          2156073582592   {1, 101, 11, 111}  

         

    2. New-PmemDisk
      • This command allows Windows PowerShell to create a PMem Disk to be created on a region, and the capacity to be visible to the Windows operating system and used by applications. Just as an SSD can be carved into partitions, PMem disks represent the unit of storage that appears as a device that can be used for I/O
      • FSDax is the mode used by default
      • Creating a new persistent memory disk. This operation may take a few moments
    3. Initialize each newly created PMem disk by going to the Computer Management > Storage > Disk Management console to view each new disk and initialize it.
      • Initialize the disk using MBR or GPT partitioning before the logical disk manager can access it.
    4. Format the disk using NTFS  (right-click on disk > Format…)
    5. Get-pmemdisk    
      • Verifies that PMemDisks was created and initialized
        PS C:\Windows\system32> Get-pmemdisk                                                                                    
        DiskNumber Size    HealthStatus AtomicityType CanBeRemoved PhysicalDeviceIds UnsafeShutdownCount
        ---------- ----    ------------ ------------- ------------ ----------------- -------------------
        1                  2008 GB   Healthy            None              True                      {1, 101, 11, 111}                       2
        PS C:\Windows\system32> Get-physicaldisk | where Mediatype -Eq SCM                                                      
        Number    FriendlyName           SerialNumber                                                            MediaType CanPool OperationalStatus HealthStatus
        ------          ------------                  ------------                                                                          --------- ------- ----------------- ------------
        1              Persistent memory disk 03018089022aedcb17c3fe499155db38a408de05 SCM       True    OK                Healthy
        
        

         

We have come to the end of the tutorial for provisioning PMEM within a Windows Server 2019 environment. See below for a link to other articles in this series.

Resources: 

Appendix

Appendix A: Misc commands

PS C:\Windows\system32> get-pmemdisk | Remove-PmemDisk                                                                  

  • This command will remove the persistent memory disk(s) from the system and results in data loss.

Appendix B: Sector Mode PMem Disk (storage over App Direct Mode)

For old file systems or applications that require write sector atomicity, create a PMem Disk in sector mode.
PMem-based storage can perform I/O at byte, or more accurately, cache line granularity. However, exposing such storage as a traditional block device does not guarantee atomicity and requires the block drivers for PMem to provide this support.

Traditional SSDs typically provide protection against torn sectors in hardware to complete in-flight block writes. Whereas, if a PMem write is in progress and a power failure happens, the block may contain a mix of old and new data. Existing applications may not be prepared to handle such a scenario since they are unlikely to protect against torn sectors or metadata.

The Block Translation Table (BTT) provides the atomic sector update semantics for PMem devices so that applications that rely on sector writes not being torn can continue to do so.

As shown in the diagram below, the BTT manifests itself as a stacked block device and reserves a portion of its metadata's underlying storage. At the heart of it is an indirection table that re-maps all of the blocks on the volume. It can be viewed as a straightforward file system that only provides atomic sector updates. The DAX feature is not supported in this mode.


 

Информация о продукте и производительности

1

Производительность зависит от вида использования, конфигурации и других факторов. Дополнительная информация — по ссылке: www.Intel.com/PerformanceIndex.