Enabling Virtual Machine Control Structure Shadowing On A Nested Virtual Machine With The Intel® Xeon® E5-2600 V3 Product Family

Introduction

For those interested in creating a nested virtual machine setup, this blog covers some of the details that one should keep in mind when sequencing the set-up, as well as commands that are useful to inspect and adjust configuration. Recently I set up a nested VM in order to test a new feature known as Virtual Machine Control Structure (VMCS) Shadowing, which is available on the Intel® Xeon® Processor E5-2600 V3 Product Family.  Nested virtualization allows a root Virtual Machine Monitor (VMM) to support guest VMMs. However, additional Virtual Machine (VM) exits can impact performance. VMCS shadowing directs the guest VMM VMREAD/VMWRITE to a VMCS shadow structure. This reduces nesting induced VM exits. As a result, VMCS shadowing increases efficiency by reducing virtualization latency.

 

Figure 1 – Reduction of VM Exits with VMCS Shadowing

 

So, why might you want to set up a nested virtual machine?  A nested VM creates an additional layer of abstraction.  This extra level of abstraction can be useful in separating the nested VM from the hypervisor and the cloud infrastructure.  Infrastructure as a Service (IaaS) is one of the areas where nested VMs can be of benefit, as the VMs can more easily transfer between different clouds and different third-party hypervisors.    

When describing the setup I will use the following terminology:

  • L0 refers to the host operating system

  • L1 refers to the first guest virtual machine that is created

  • L2 refers to the nested guest virtual machine that is created inside of L1.

For my particular situation I was using the following operating system versions.

L0 = Red Hat Enterprise Linux* 7 (3.17.4-2.el7.repo.x86_64)

L1 = Red Hat Enterprise Linux* 7 (3.10.0-123.9.3.el7.x86_64)

L2 = Red Hat Enterprise Linux* 7 (3.10.0-123.9.3.el7.x86_64)

 

Setting up the Host (L0)

The nested and VMCS Shadowing features need to be enabled at the host level (L0).  To verify this is the case the following commands can be used.

To check if VMCS Shadowing is enabled.

L0 Command Line:cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs

You should get a response of ‘Y’

To check if the Nested feature is enabled.

L0 Command Line:cat /sys/module/kvm_intel/parameters/nested

You should get a response of ‘Y’

If you get a response of N for either parameter you can setup a configuration file for the module.

L0 Command Line:cd /etc/modprobe.d/

Create a file called kvm-intel.conf (if it doesn’t exist already).  Edit kvm-intel.conf to include the following:

options kvm-intel nested=1

options kvm-intel enable_shadow_vmcs=1

save the file and reboot the host.

You will also need to make sure that the VMX feature is enabled.  It should be enabled by default, but if you want to check you will find it listed as one of the cpu flags.

L0 Command Line:cat /proc/cpuinfo

 

Setting Up the First Guest VM (L1)

You will need to make sure the VMX feature is passed through to L1.  If after you install L1 you find this is not the case use the virsh tool to edit the configuration file for the L1 guest.

L0 Command Line:virsh edit /etc/libvirt/qemu/L1_guest.xml

Alter the CPU information so that it appears as follows and then save the file.

 <cpu mode='host-passthrough'>

 </cpu>

You should be able to see vmx as one of the listed cpu flags for L1.

L1 Command Line:cat /proc/cpuinfo

 

Setting Up the Nested VM (L2)

There are a few different ways you might want to set up L2.  You could set up as an image file, you could set it up as a raw partition, or you might want to set it up with an image file but then add in access to a raw partition.  One of the reasons you may choose raw partitions over the default image file might be if your goal is to test performance.

Regardless of how you choose to set up your Nested VM (L2) you will need to set up a network bridge.

 

Setting Up a Network Bridge for the Nested VM (L2)

You will need to set up a network bridge for the nested VM (L2) to the first guest VM’s (L1) network adapter.  If you do not do this you will get errors during the installation of the operating system.

In my particular case, I accomplished this through the virtual machine manager under Gnome on L1.

System Tools – Virtual Machine Manager - Edit – Connection Details – Virtual Networks

Click the + symbol in the bottom left corner. 

By default the network used by my guest VM was 192.168.122.0/24. 

I created an additional network with the following configuration.

Name: nested bridge

Device: virbr1

Active

On Boot

IPv4 Forwarding: NAT to eth0

IPv6 Forwarding: Isolated network, routing disabled

Network: 192.168.166.0/24

DHCP start: 192.168.166.128

DHCP end: 192.168.166.254

When installing your Nested VM you will want to choose the nested bridge for the network interface.

 

Setting Up L2 As An Image File

When you are installing the L2 Nested VM under L1 VM, if you want to install the L2 VM as an image file you can use the default options for the hard drive settings.  You will of course still want to use the nested bridge for the network interface that you set up in the previous step.

 

Setting Up L2 As a Raw Partition

Setting Up a Raw Partition For Installation of L2

Before installing the VM to a raw partition you will need to make sure you have a partition created using fdisk.  Let’s assume that the drive (/dev/sda) where L0 is installed has extra free space that could be used for this purpose.

L0 Command Line:fdisk /dev/sda

n (to setup a partition)

p (to make a primary partition)

When asked about where to start the first cylinder, just hit return to accept the default

On defining the Last cylinder, just hit return to accept the default (make sure that there is enough space for your VM)

w command to write the changes, then exit fdisk.

If L0 was installed on /dev/sda1 then the partition that is available for L2 will be /dev/sda2.  Note we will not be formatting the volume at this time, this will be taken care of during the installation of the VM.

Modifying L1 Configuration File

Before doing the installation of L2 you will need to make sure that the raw partition is accessible to L1.  To do this, modify L1’s configuration file to include the location that contains the partition that you want to install L2 on.

L0 Command Line:virsh edit /etc/libvirt/qemu/L1_guest.xml

<controller type='scsi' model='virtio-scsi'/>
<disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/sda'/>
      <target dev='sda' bus='scsi'/>
</disk>

Save the file.

Now when you install the operating system for L2 you will be able to access /dev/sda2.

Installing L2 On a Raw Partition

Below is an example of creating a nested VM on a raw partition from the command line of L1.

L1 Command Line:virt-install -n nested-L2 -r 16000 --vcpus=4 -l /RHEL-7.0-Server-x86_64-dvd1.iso –disk path=/dev/sda2 -w network=nested_bridge

Where:

-n is the name of the nested vm

-r  is the memory (16GB in this case)

-vcpus is how many cpus the nested VM will have

-l is the location of the ISO image for installing Redhat

-disk is the path location to install; in this case we are pointing to the direct access partition that we set up.

-w is the bridged network we previously set up. During this process you may get a message about the partition not having enough space and if you want to reclaim it.  You’ll need to reclaim the space in order to do the installation.  Just make sure you are pointing to the correct partition for your particular setup before proceeding!

 

Setting Up L2 As An Image But Adding a Partition For Direct Access

Under L0 you will want to use fdisk to setup an EXT3 partition to be used by L2.  In this case let’s assume that you have a second drive (/dev/sdb) that will be used strictly for this purpose.

L0 Command Line:fdisk /dev/sdb

n (to setup a partition)

p (to make a primary partition)

Question on first cylinder, just hit return to accept the default

Question on the Last cylinder, just hit return to accept the default (make sure that there is enough space for your VM)

w command to write the changes, then exit fdisk.

Then use this command to format the disk

L0 Command Line:mkfs.ext3 /dev/sdb1
 

Once you have your formatted partition set up, do not mount it under the host OS (L0) or the first guest VM (L1).  This is to avoid issues that can be created with permissions between the different levels.

In order to make sure that L2 can access the EXT3 partition that you have set up, you will need to make sure that both L1 and L2 can see the drive where the partition is located.  To do this you will need to edit the configuration files.  Edit your guest VM configuration file with virsh edit.  This time you need to add the following.

L0 Command Line:virsh edit /etc/libvirt/qemu/L1_guest.xml

<controller type='scsi' model='virtio-scsi'/>
<disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/sdb'/>
      <target dev='sdb' bus='scsi'/>
</disk>

Save the file.

L1 Command Line:virsh edit /etc/libvirt/qemu/L2_guest.xml

<controller type='scsi' model='virtio-scsi'/>
<disk type='block' device='lun'>
      <driver name='qemu' type='raw' cache='none'/>
      <source dev='/dev/sdb'/>
      <target dev='sdb' bus='scsi'/>
</disk>

Save the file.

L2 Command Line: mount /dev/sdb1 /mnt

You now have an EXT3 volume under the /mnt directory that you can directly access from the nested VM (L2).

 

Working With the VM’s

Here are some common tools that you may need to use when interacting with the different VM’s.  They are included for your convenience.

Moving Files Between VM’s

You can use the scp command to move files back and forth between the VMs.

To move a file from the system you are currently logged into, to a different system (a push method) use:

scp ~/my_local_file.txt user@remote_host.com:/some/remote/directory

To move a file from a different system to the system you are currently logged into (a pull method) use:

scp user@192.168.1.3:/some/path/file.txt .

Registering From the Command Line

Registering your VMs can make it easier to get updates via yum.  You may need to set the proxy server if you have one prior to registration

subscription-manager config --server.proxy_hostname=name_of_your_proxy_server--server.proxy_port=port_number_for_your_proxy_server

To register your VM

subscription-manager register --usernameyour_account_name--password your_password--auto-attach

Determining the IP Address Of Your VM

If you are using ssh to interact with a VM instead of the GUI interface and need to know the network address that was assigned to your VM (if you did not set a static IP) you can find out by using the following command.

L1 Command Line: nmap -v -sP 192.168.166.0/24

This will show you a list of the active connections and allow you to identify the IP address of L2.

 

Collecting Data On VM Exits

If you want to look at the data associated with VM exits you can do the following.

Find the 1st Guest VM process ID Number

L0 Command Line:pgrep qemu-kvm

Run the perf tool and record the events from L0: perf kvm stat record -pqemu-kvm_ID_number

Important note: You must use Control-C to terminate the data collection process.  Any other method will result in corrupt data.

The output file will be created in the directory where the perf command was run from.  It will generate a perf.data.guest file.  These files can get large, so be aware of that you might need to delete them after you are done with them or move them off the system if you need to archive.

Once you have the output file if you want to get the stats there are some of the different ways to do it depending on what you want to look at.  Run this command from the same directory where the output file is located.

L0 Command Line:perf kvm stat report --event=vmexit

L0 Command Line:perf kvm stat report --event vmexit --key=time

L0 Command Line:perf kvm stat report --event vmexit --key=time --vcpu=0

There are other more general stats for the VM’s that be of interest in some cases.  You can look at them by doing the following.

Find the 1st Guest VM process ID Number

L0 Command Line:pgrep qemu-kvm

L0 Command Line:perf stat –e ‘kvm:*’–p qemu-kvm_ID_number

 

Conclusion

Creating a nested VM can have its challenges depending on what you are trying to accomplish and what type of configuration you desire.  I hope if you are trying to do this that some of the information provided here is of benefit to you. 

 

Resources

http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/intel-vmcs-shadowing-paper.pdf

 

The Author:  David Mulnix is a software engineer and has been with Intel Corporation for over 15 years.  His areas of focus have included software automation, server power and performance analysis, and cloud security.  

*Other names and brands may be claimed as the property of others.

For more complete information about compiler optimizations, see our Optimization Notice.

1 comment

Top
sarfraz s.'s picture

This patent https://www.google.ch/patents/US7725895 explains it in some detail

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.