Optimizing Virtual Environments on Intel® Virtualization Technology-enabled Platforms

By Ken Strandberg

• Introduction
• Core Effects
• 64-bit Deployments
• Processor Optimization
  • Hyper-Threading
  • Processor Affinity
    • No Affinity
    • Pinned
    • VMware Virtual SMP*
• Memory
• VMware Resource Pools, Shares, Reservations, and Limits
• Storage and Virtual Machines
• Conclusion

Optimizing virtual environments can have an important impact on the overall effectiveness of running virtual machines (VMs) instead of native environments in a data center. Optimization is not an exact science. It depends on many factors, both specific to the virtual machine manager (VMM) being deployed, types of users, and the particular workloads being run, and general optimization parameters available on the servers.

At Intel, we constantly test VMM deployment scenarios and optimization of virtual environments for benchmarking on platforms with Intel® Virtualization Technology. Our testing helps us develop some valuable techniques for optimizing virtual environments. These techniques can be extended to general use in the data center.

This paper offers recommendations we have found useful for optimizing virtual environments on Intel Virtualization Technology-based platforms. Depending on your particular usages, users, and workloads, you might find that further tuning will provide even better performance.

When deploying a virtualized server, administrators should consider recommendations of the VMM developer. VMware,* for example, provides many optimization software notes and white papers for deploying and optimizing Virtual Infrastructure 3*, ESX Server,* and other VMware products. XenSource,* Novell,* Red Hat*, and Virtual Iron* also offer application notes for deploying their products.

Several core platform and processor architecture features bring the same native environment benefits to virtual environments. 

• Front-side bus (FSB) and core frequency effects - FSB and core frequencies scale effectively in virtual environments, like they do in native deployments. Higher frequencies here improve performance in VMs.
• Cache Effects - Virtualization deployments benefit from larger cache. However, just as in a native environment, benefits differ over different workloads.
• Prefetcher Effects - The prefetch settings that provide the best performance for a native operating system (OS) usually provide the best performance with VMs, too.

With some VMMs, in order to run 64-bit guest OSs, Intel® Virtualization Technology must be enabled in the BIOS. Check the capabilities of the VMM you are running.

How VM tasks are scheduled on platform processor cores can have a significant impact on VM performance. Hyper-Threading (HT) technology and processor affinity are two configurable options for optimizing processor performance.

Hyper-Threading

Most VMMs support HT technology. The HT technology setting that provides the best performance for a native environment usually provides the best performance in VMs. However, there are drawbacks to using HT technology. 

If the OS dispatches two threads that contend for identical resources in the CPU, HT technology could reduce performance. Table 1 shows the results we’ve seen with HT technology under the SysBench benchmark running on a dual-processor platform with HT technology.

Table 1. Effect of HT Technology Under Sysbench

If you deploy VMware’s Virtual SMP* on HT technology-enabled platforms, see the VMware white paper “Best Practices Using VMware Virtual SMP” on the VMware web site for tuning recommendations.

Processor Affinity

With multi-core processors and multi-processor platforms, you now have more flexibility in how you choose to assign a VM’s virtual CPUs (vCPUs) to physical cores (pCPUs) or threads (logical CPUs). How you assign vCPUs can impact VM and platform performance. You have the following configuration options when setting up the VM:

• No Affinity - The VMM scheduler’s algorithms automatically assign vCPUs to pCPUs and logical CPUs.
• Affinity or Pinned – You over-ride the VMM’s scheduler and assign, or pin, vCPUs to one or more pCPUs or logical CPUs. If a VM is pinned to more than one core, the locations of the cores determine if the cores are pinned within the same socket (package) or across sockets. Figure 1 illustrates these affinity options.

Figure 1 vCPU to pCPU Affinity Options

No Affinity

VMMs typically configure newly created VMs to run their vCPUs with no affinity. Often, running with no affinity is the best option. No affinity allows the VMM scheduler to balance all vCPU loads across all of the available pCPUs.  This is particularly beneficial when the total active vCPUs is greater than the pCPUs, or the pCPUs are overcommitted.

Pinned

When vCPUs are pinned to a pCPU, the VMM scheduler runs them only on the specific pCPUs. Pinning vCPUs can resul t in improved performance when the characteristics of a particular workload can benefit from executing it on specific cores. An example of such a workload characteristic is the impact of the pCPU caches on the workload performance. Always executing on the same core and its various caches can minimize cache misses per instruction, improving performance.

If a VM is pinned to more than one pCPU, pinning the pCPUs within a socket or across a socket can also impact performance. This performance impact is best determined by experimentation. Note that different platform BIOSs can number the pCPUs differently. Before experimenting, you need to be sure of the mapping between BIOS core numbers and the cores’ physical locations on the system board. Figure 2 illustrates how this mapping can differ among platforms.

Figure 2. BIOS Number Mapping to Physical Cores

Intel provides a simple utility that will report the OS numbering compared to the physical location (socket) of the CPU. Visit the following web page:

Detecting Multi-Core Processor Topology in an IA-32 Platform

VMware Virtual SMP*

VMware Virtual SMP* allows you to implement multiple vCPUs in a VMware ESX* server-based VM. For configuring vCPUs under Virtual SMP, refer to the recommendations in the VMware white paper “Best Practices Using VMware Virtual SMP” on the VMware web site.

Since the memory allocated for a VM comes from the system physical memory, you must account for the memory to be used. A good memory allocation practice is illustrated in Figure 3.

Total Active VM memory ? .8 * (System Memory – VMM/Host OS Memory)

Figure 3. Active VMM Memory Allocation

Active VM memory is the important component. You can create many VMs, some active and inactive, with inactive VMs replacing active VMs when the active tasks are complete. The total memory required by all VMs can be more than the allocated 80 percent, but if too many active VMs try to use more than the allocated memory, the host begins swapping and performance for all VMs suffers.

VMware Virtual Infrastructure 3* allows you to set CPU and memory resource limits and guarantees for VMs and pools of VMs, plus set priorities for how these resources are applied when conflicts arise. How you set up these parameters can not only affect performance, but actually prevent VMs from starting. A useful VMware lab from VMWORLD 2006* offers recommendations on how to make sure you are allocating resources adequately. Visit the following web address:

Storage is a very broad topic when it comes to both native and virtual environments. Because there are many storage architectures, technologies, and deployment methods, optimizing storage for VMs is beyond the scope of this paper. However, the following re commendations will help improve your virtual environment performance:

When you build your virtual environment, it is best to build the VMM on one hard drive and all VMMs on as many different drives as are available. You will achieve the best performance by having a separate physical disk for each VM, utilizing more spindles.

If you do not have enough physical drives for such a deployment, our experience has shown that deploying up to two VMs per spindle delivers acceptable performance in non-disk intensive VMs.

Beyond these limitations, you should consider adding physical drives.

Optimizing virtual environments can offer significant benefits to your virtual infrastructure. Some of the platform’s benefits experienced in native environments apply to virtualized systems. But, making adjustments in processor affinity, memory, and how you deploy VM disk storage can help improve performance further. You should also consider recommendations by your VMM developer, and possibly experiment on non-production environments to achieve added performance benefits for your particular workloads, users, and system hardware.

We apply the techniques outlined in this paper for many benchmarks. The Intel paper “Performance Impacts with Optimized Virtual Environments on Intel® Virtualization Technology-based Platforms” shows the results we are able to achieve on standard benchmarks.

Ken Strandberg is a technical writer and training developer, who has written on subjects such as high performance computing, enterprise, AdvancedTCA-based modular platforms, converged communications, and processor architectures. Ken can be reached at ken@kenstrandberg.com or visit his website at www.kenstrandberg.com.