Verifying ISV Applications Running Under Virtualization

Virtualization has become ubiquitous in general IT datacenter environments. Existing applications are already being run in production virtualized modes today with many more running virtualized in non-production testing environments.

In some cases, this situation has arisen without the ISV having to acknowledge their customer’s virtualized use of their product. However, many ISVs are beginning to consider their strategies around virtualization, including testing, verification, performance validation, licensing, delivery, support, etc.

Most end-users customers agree that they want their ISVs to “support” their applications whether they are deployed virtualized or not.  Thus, ISVs are working to set the level of support they will provide to virtualzed mode deployments. 

For verification and validation, in which the software product is tested to ensure it meets is functionality and performance specification, how tenaciously will an ISV proactively resolve issues by verification and validation testing before they release their software (versus resolving issues reactively)?

Intel has the viewpoint that in the near future, all enterprise software will see pervasive use under a Guest OS (OS Partitions, Virtual Machine Manager, etc.). Thus software stacks need to be tested in these virtual environments before they are launched. For an ISV, adding some up front proactive testing will reduce the ISV’s cost of support as well as the number customer quality issues which are raised back to the ISV.

ISV’s can validate their application behavior in the virtual environment using several methodologies. In this paper, we review some of the considerations and make prescriptive suggestions on verification and validation.  In other words, as an ISV, how and when should you look at validation, what steps should you use to validate, what should you consider, what issues are likely to arise, etc.

Applications running in native mode have the whole machine to utilize. The native OS will manage the resources and scheduling for the application software stack.

Virtualization is designed to “transparently” present generalized hardware to a Guest OS thru a Virtual Machine Monitor (VMM).

Thus the general guidelines for the validation of application software can also be applied to virtual mode.   Our first key piece of advice is captured in the following principle:

Treat the Platform/ VMM/ GuestOS combination just as you would another OS/Platform and apply your normal Quality Assurance (QA) process

Following this advice will minimize the cost and additional overhead in dealing with Virtualized software stacks.

So, where does this advice breakdown?

First, in a virtualized mode, there is a layer of abstraction of the underlying platform hardware by VMM or Hypervisor. The VMM will present an abstraction of the underlying hardware to the Guest OS. The VMM will mange re source-allocations and scheduling for each of the Virtual Machines hosted on top of the native machine. This additional layer may remove special hardware features or introduce extra resource contention among multiple virtual machines being hosted. These two may introduce unexpected test failures into your validation.

Second, in virtual environments, there are unique configurations that are either not needed or not possible in native environments. For example, the number of VMs and their resource allocations must be pre-defined. Some VMMs allow direct assignments of devices to particular VMs such as using Intel® Virtualization Technology (Intel® VT) for Directed I/O (Intel® VT-d) for added performance. These need to be understood in your test planning. 

Finally, virtualization can also bring new uses cases and application programming interfaces (APIs) to test. For instance, virtualization landscape management can quickly become a value-add feature that one may want to support in an application.  HyperVisor API calls can give applications (and/or their management consoles or installers) ways to do GuestOS provisioning, management, monitoring, etc. These need to be added to your test plans.

Application functionality in virtual environment can be verified and validated using similar methodologies to those used in native environments.

To verify an application in a Virtualized environment, suggested high level steps are:

• Decide use-case(s) to test; As appropriate add new consolidation uses cases and uses cases to support use of virtualization as a feature,
• Decide workload(s) and tests which simulated the important use-cases. Decide key functionality and performance indicators needed to indicate success
• Decide all software stacks to be used, including VMM and OS
• Decide precise System Under Test (SUT) configurations to use in testing
• Use exploratory runs to set VMM parameters
• Apply current Unit Testing and QA setups to VMM/Guest OS combinations
• Use/Add performance metrics and Quality-of-Service (QoA) indicators and validate that virtualized performance has not degraded over native performance

See the Appendix for an example of applying these seven steps to an Enterprise Resource Planning (ERP) application.

There are several challenges in validating applications in virtual environments. Some considerations are:

• Are there any “special features” of the h/w used that may not react the same way when virtualized?  For instance, some CPUID leaves may not pass unmodified thru the VMM.  Some I/O features may demonstrate reduced performance, etc.
• Are there any limits imposed by current generation VMM which might impact your application?  For instance, current MSFT Viridian has a 1 vCPU and 3.6Gb memory limit per GuestOS. 
• What is the application architecture? Multi-tier? Will the entire application stack be virtualized and consolidated onto the same platform?
• How does the application software behave in a multi VM heterogeneous configuration? Does having totally different VM/application-software running on same machine effect our application of interest?
• What are the minimal resource requirements for application software when run in a GuestOS (and also along with other GuestOS on same machine)?
  • How much network bandwidth/latency required for application to perform satisfactorily
  • Does the application software require guarantee of minimal/maximum resources? How many shares of
    • CPU, Memory, Network and disk/IO resources does it require?
• How does the application software behave when resources are overcommitted (eg: have only 2 physical CPUs, but 4 VMs are hosted; same for other resources like memory, network and disk). Generally, over committing memory is the most likely of these to lead to catastrophic performance issues.

Minimal Virtual Hardware Landing Zone

Many ISVs release their application software with a pre-defined “minimal software and hardware requirements” for their application to land successfully/ satisfactorily. A similar approach needs to be used for virtual mode also. For virtual environments, ISVs’ need to define the set of minimal requirements in terms of VMM specific resource requirements. These may be VMM specific. For example VMware has concept of “shares” for all the hardware resources maintained: CPU shares, memory shares, IO etc.

Most ISV’s have their own existing Unit Test Frameworks and QA processes for validating their application software stacks. The same can applied and used to validate configurations in virtualized environments.

An excellent practical approach to utilize in these testing frameworks is to collect important unit-level code and business transaction-level details as part of the system design. Collecting details of application specific transactions (their start/end times, times thru critical sections, etc.) can prove invaluable as an analysis tool as functional or performance issues are uncovered during testing. This data is equally as useful under virtualized system testing.  The only caveat with this approach is when taking measurements over very small spans of time (small enough that you might be tempted to use the CPU’s timestamp counters to get accurate enough measurements):  you need to be aware of the classic VM “time drift” problem. (Some other excellent whitepapers talk about this issue in more detail and review various techniques to overcome it).

These test frameworks may test functionality but may not test the performance metrics and quality of services requirements of the application. However, we believe these should be augmented with performance readings as performance measurement testing is critical for virtualization validation.

At least for the next few years, the biggest difference of a virtualized hardware platform from the native hardware platform will remain their difference in performance (at least on certain types of workloads).  For these next few years, customers will always compare virtualized environments to native and will always expect leading performance on their current hardware.

This leads us to another virtualized-mode verification principle:

A major component of an ISVs valid ation of virtualized mode must be a comparative performance reference back to the native hardware

In general, if the performance degradation is too great, then that is a guarantee that the customer will raise this as an issue to the ISV. Thus the ISV should have prior knowledge that the issue (might) exist and have suggested approach(es) to mitigate the issue.

We treat the important issues of testing of performance on virtualized system under test in a companion paper

Use of Other Software Benchmarking

ISV’s can take advantage of existing industry standard Virtualization benchmarks to assess the performance of VMMs with particular OS and application types. One such benchmark is vConsolidate developed by Intel.

ISVs can also take advantage of traditional micro benchmarks used in native environment to asses the Platform/OS combination and apply it to platform/ VMM /GuestOS combination. These experiments can give good understanding of component performance areas such as network and disk IO latency and throughput.

Current generation Virtualization technology has challenges in efficiently virtualizing IO hardware resources. Several VMMs currently use some form of para-virtualization to efficiently virtualize IO resources in one form or other. This process has resulted in not-so-efficient IO latency/throughput.

ISVs should validate their application software stacks for strict Quality-of-Service (QoS) requirements when virtualized. Application software response times and latency should be measured under different VT configuration and usage models (e.g.: Multiple GuestOSs).

Many new usage models can arise in virtual mode validation. Please refer to the uses cases paper for several cases of homogenous and heterogeneous multi-VM configurations.

For this example, we have a 3-Tier (GUI, Business Logic application servers, and Database) application that does Enterprise Resource Planning (ERP) transactions. Its standard system-wide testing procedure is to setup 2 identical Dual-Core Intel® Xeon® processor platforms, one each for the Database and Application Logic tiers. A test harness drives the system to load and verifies transactions complete appropriately.  In addition, the test harness records the maximum transaction achievable per second and the average user response time.  These then define both a final functional and a performance validation assessment for the application.

Following our seven steps,

• Decide use-cases to test.  To test virtualization, we have decided to add one new use case, in which the two DP platforms are “virtualized” and replaced by a single platform running two Guest OSs with the software stacks and configurations of the standard two platform test setup.
• Decide workloads. We will use the standard workload and test harness.
• Decide all software. Each Guest OS is the same as the native 2 platform setup.  Since our application runs over Linux, we will use Xen as the VMM.
• Decide SUT hardware configurations. A single Multi-Core Intel® Xeon® Processor 5000 Sequence will replace the two DP boxes. The MP box is configured with twice the memory of each DP box and the disks from the DP database platform are attached to the MP box.
• Set VMM parameters: Since the two DP resource partition the MP box, we assign half the processors and memory to on VM/ GuestOS and half to the other.  We use Intel® VT-d assignment to attach the database disks to the VM hold the database stack.
• Apply current test setups.  We can now run our standard tests.
• Performance & QoS indicators:  Our standard test already had these defined.  In our example, after our first run, we find the single MP box had no functional errors reported by the test harness and it was able to handle 90% of the transaction per second of the two DP boxes.  However, upon analysis, we find that the database was beginning to have memory buffer cache misses due to lack of available OS memory while the application tier never reached more than 70% memory saturation. Also the average CPU utilization of the database was never more that 30%, while in the application tier the average CPU utilization was often pegged its OS counter at 100%.  If desired, we can now iterate our performance validation by repeating step 5 of our procedure and adding more memory to the database VM and oversubscribe CPUs by adding one more vCPU to the application tier VM.