Configure and Attest Confidential Virtual Machines in Kubernetes with KubeVirt and Intel® Trust Domain Extensions

author-image

By

By Ruomeng Hao, Lei Zhou, Ken Lu, Haidong Xia, Yanhui Zhao

 

In the rapidly expanding world of cloud computing, Kubernetes has become the de facto standard for cloud native application management. However, not all applications are suitable for the transition from monolithic applications to cloud native microservices, requiring customers to manage both monolithic applications running on virtual machines and microservices applications running in cloud native environments. KubeVirt was designed to meet this challenge by enabling the coexistence of containerized and virtualized workloads within the same Kubernetes cluster. However, there is a need to address security in VMs managed by Kubernetes.  

In this article, we show how to enable Intel® Trust Domain Extensions (Intel® TDX) in KubeVirt to support running confidential computing workloads in hardware-isolated virtual machines called trust domains (TD), managed by Kubernetes. We also describe how to use the remote attestation process in KubeVirt to prove the integrity of TDs using an Intel® Trust Authority TD client and attestation service.  

Prerequisites

Here are the hardware and software prerequisites required to enable Intel TDX and perform remote attestation for VMs in KubeVirt. Note that Intel Trust Authority is a subscription-based service. For more information about gaining access to the service, visit the Intel Trust Authority website. If you don’t have access to Intel Trust Authority, you can’t perform remote attestation as described here, but you can still enable Intel TDX in KubeVirt and run Kubernetes-managed TDs. 

Hardware

Intel TDX is available on 5th Gen Intel® Xeon Scalable processors. You’ll need access to the host machine. If you don’t have access to a server, you can apply for access to a TDX bare metal server through Intel Developer Cloud (IDC). Learn more about Intel TDX on IDC at Intel® Trust Domain Extensions (Intel® TDX).  

Software

This article doesn’t cover enabling Intel TDX on your processor, but you can install one of the two Linux distributions listed in the Software section to enable it. To enable Intel TDX on your processor, you have two options:  

  1. Red Hat and Intel are offering a preview of the Intel-TDX virtualization stack on CentOS Stream 9. For more information, see the Red Hat blog Enabling hardware-backed confidential computing with a CentOS SIG. 

  1. Ubuntu has partnered with Intel to offer Ubuntu users a customized build derived from Ubuntu 23.10 and encompassing all the essential components required to construct TDX confidential workloads. For more information, see the Ubuntu blog Intel® TDX 1.0 technology preview available on Ubuntu 23.10. 

Solution Overview

Once prerequisites are met, you’ll build the solution using KubeVirt, Intel TDX, and Intel Trust Authority, if remote attestation will be configured. 

KubeVirt

KubeVirt is a Cloud Native Computing Foundation (CNCF) incubating project that extends Kubernetes to manage virtual machines. It allows users to run and manage virtual machines alongside containerized workloads in a Kubernetes environment. By leveraging KubeVirt, cloud service providers (CSP) can extend the flexibility and scalability of Kubernetes to achieve faster and more efficient provisioning of VMs to tenants and improve compute resource utilization. With KubeVirt, users can easily create, manage, and orchestrate virtual machines using Kubernetes tools and workflows. KubeVirt makes it easier for CSPs to adopt a hybrid infrastructure approach, combining containers and virtual machines in a unified platform. 

Intel ® Trust Domain Extensions (Intel ® TDX)

Intel TDX includes Intel Virtual Machine Extensions (VMX), instruction-set-architecture (ISA) extensions, Intel® Total Memory Encryption – Multi-Key (Intel® TME-MK) technology, and a CPU-attested software module. 

Intel TDX provides a hardware-isolated virtual machine called a trust domain (TD). The TD runs in a new CPU mode called Secure-Arbitration Mode (SEAM), in which the CPU only allows SEAM memory range access to software executing within the memory range. All other software and direct memory access (DMA) from devices to this memory range are aborted. Thus, Intel TDX can help provide memory and CPU state confidentiality and integrity by protecting TD memory content and CPU state from any other software, including the virtual machine monitor (VMM). For more detailed information on Intel TDX, download and refer to Intel TDX White Papers. 

Intel TDX Remote Attestation

Attestation is a process for establishing trust between a confidential computing workload and one or more relying parties. Generally, attestation involves different roles in the process below. 

  1. A challenger generates an attestation request to an attester, which refers to a confidential computing environment. 

  1. The attester collects measurements and cryptographic evidence and returns it to the challenger. 

  1. The challenger verifies the evidence itself or by using an attestation service. 

Intel TDX supports remote attestation to give a relying party (owner of a workload or sensitive data) increased confidence that the workload is running inside a TD, on a genuine Intel TDX system, and at a given trusted computing base (TCB) version. At a high level, the relying party requests attestation proof – a TD quote — and then verifies the quote. Among other information, the TD quote contains TD measurements (static and runtime), metadata associated with the TD, and security version numbers (SVN) of elements in the TDX TCB. For more details on the Intel TDX end-to-end remote attestation flow, read Runtime Integrity Measurement and Attestation in a Trust Domain. 

Remote Attestation with Intel® Trust Authority

Intel Trust Authority is software as a service (SaaS) that remotely attests to the authenticity and integrity of the confidential computing environment and related workloads, regardless of workload location. Intel Trust Authority runs all attestation-related processes within a trusted execution environment (TEE) and provides cryptographic evidence that the attestation verification logic is authentic, unaltered, and auditable. On top of this, the Intel Trust Authority attestation process provides assurance and endorsement to any relying party attesting that the TEE and any data and workloads within it have not been compromised. 

While Intel Trust Authority focuses on providing attestation service for the verification phase, it also provides a client and Intel TDX CLI for an attester to get a TD quote. The Intel Trust Authority attestation service will evaluate the TD quote, and an attestation token will be returned. The attestation token can be used by the relying party to decide whether it should trust the attester. 

How to Support Intel TDX in KubeVirt

Kubernetes offers the ability to extend its architecture through custom resources, which enable a user to create a new endpoint in the Kubernetes API for storing and retrieving a collection of API objects of a specific type. KubeVirt leverages this capability by using CustomResourceDefinitions (CRDs) and controllers to represent and manage traditional virtual machines alongside containers. The primary CRD utilized by KubeVirt is the virtual machine (VM) resource, which governs the lifecycle of a virtual machine instance (VMI) object, representing an individual virtualized workload. This approach allows KubeVirt to integrate virtual machines seamlessly into the Kubernetes environment. For more details, please refer to the KubeVirt user guide. 

KubeVirt has gone to great lengths to make it as easy to manage virtual machines as it is to manage pods in the cloud native environment. Figure 1 illustrates KubeVirt architecture with the components described here: 

  1. virt-api: This component links to the Kubernetes API server and offers an HTTP RESTful entry point that enables the management of virtual machines within the cluster. 

  1. virt-controller: This component acts as a Kubernetes controller, responsible for taking notice and creating the pod in which the VM will run. When the pod is scheduled on a particular node, the virt-controller updates the VM object with the node name and hands off further responsibilities to a node-specific KubeVirt component, virt-handler. 

  1. virt-handler: A daemon that runs on each Kubernetes node, this component is responsible for monitoring the state of VMIs according to Kubernetes and ensuring that the corresponding virtual machines transition to the desired state. The virt-handler has a communication channel with each virt-launcher, which manages the lifecycle of the libvirt process within the virt-launcher pod.  

  1. virt-launcher: This is a core component on which the virtual machine runs. This component directly manages the lifecycle of the libvirt process within the pod. 

A diagram showing the KubeVirt control plane and worker node architecture.

Figure 1 KubeVirt Architecture

KubeVirt Implementation for Intel TDX

The reason for supporting Intel TDX in KubeVirt is that typical VMs do not have memory encryption in Kubernetes, which leads to the possibility of data in memory being compromised. As users increasingly choose to rent virtual machines from CSPs, virtual machine security has become more of a concern and a pain point that must be addressed urgently. Without the support of memory encryption technology, an administrator or malicious software deployed on the platform can easily access a running VM, possibly leading to data leakage.  

Intel TDX technology can effectively solve this problem. It transparently encrypts memory using a key unique to each virtual machine. In addition, it provides an attestation mechanism to allow VM owners to verify the security of their VMs. 

In KubeVirt, feature gates provide a way to enable or disable specific features. They are inspired by and aligned with Kubernetes feature gates. Feature gates are typically used to test new features or disable features not yet production ready. Using feature gates, users can selectively enable or disable specific features to customize their KubeVirt deployment according to their requirements. Because Intel TDX is a cutting-edge technology, few worker nodes in the cluster will support it. Additionally, the Linux Intel TDX stack is not mature, so TDX has been added as an alpha feature that can be optionally turned on by the end user using KubeVirt feature gates.  

The node-labeller component in KubeVirt will check all nodes for features supported by the CPU, and now supports detection of the Intel TDX feature with the help of libvirt. The Intel TDX label is then applied to the node. 

The architecture of KubeVirt for Intel TDX is illustrated in Figure 2. For more details, please see KubeVirt for Intel® TDX. 

A diagram showing KubeVirt for Intel TDX control plane and worker node architecture.

Figure 2 KubeVirt for Intel® TDX Architecture


There are two parts to enabling Intel TDX in KubeVirt. The first part is to implement Intel TDX-specific APIs and the second is to support key management for Intel TDX VMs.  

Implement Intel TDX-specific APIs

Intel TDX is a very complex technology, and much work has been done at the Qemu/KVM level to make it easier to use. However, some usage changes are still required, such as the need to set up LaunchSecurity, the file descriptor-based private guest memory that is unique to confidential VMs. For end users, only a few Intel TDX-specific configurations are exposed at the API level, including Quote Generation Service (QGS) configuration for attestation and MRCONFIGID for configuration of the guest TD. Most of the Qemu-specific parameters required by TD are hidden from the user and are pre-configured in the backend to minimize complexity for the user. In other words, the virt-launcher service will be responsible for converting the Intel TDX configuration into parameters in the libvirt Domain XML to launch TD. The sample configuration  in Figure 3 demonstrates how to set the QGS and MRCONFIGID parameters. 

launchSecurity: 

      tdx:  

        QGS: vsock:2:4050 

        MRCONFIGID: <sha384_hash(48 Bytes)>

Figure 3 Sample configuration that demonstrates setting QGS and MRCONFIGID parameters


There are several parameters that need attention from end users. The first is QGS, a service providing TD quote generation from Intel. The QGS field is a string to specify QGS in Qemu socket address format. The parameter is used for communicating with QGS for remote attestation. Examples of the supported format include "vsock:2:1234", "unix:/run/qgs", and "localhost:1234".  

The second parameter is MRCONFIGID, which allows the user to assign a hash value that describes TD configuration. It can be parsed from the TD report and used for subsequent measurement to verify the trustworthiness of the TD. 

Support Key Management for Intel TDX VMs

For traditional VMs, KubeVirt only needs to know the basic CPU, memory, and disk configuration and can then use Kubernetes to configure these VMs to achieve a better allocation of resources. However, this is not the case for confidential VMs, which require a unique key to encrypt their memory, for which Intel TDX uses Intel TME-MK. Intel TME-MK is an extension of Intel® Total Memory Encryption (Intel® TME) that adds support for multiple keys. It also addresses data protection needs by encrypting memory content at runtime. The software can configure processors supporting Intel TME-MK to encrypt any memory page using any subset of the available keys. By using unique keys, Intel TDX helps ensure that the cryptographic boundaries are maintained, preventing unauthorized access or interference between VMs.  

These keys must be treated as finite resources to enable TD scheduling at the Kubernetes level. Key management is implemented at the CGROUP level, but Kubernetes’ built-in resource management currently has only a few basic features. Therefore, key usage must be exposed to Kubernetes as a third-party resource, like a device plugin resource. This ensures that the node assigned to launch a TD supports Intel TDX and manages the remaining keys.  

Remote Attestation for KubeVirt TDs with Intel Trust Authority

Like TDs managed by a traditional VM hypervisor, a TD managed by KubeVirt also needs remote attestation to prove its integrity in a Kubernetes cluster.  

Intel Trust Authority provides a Linux system client called Intel® Trust Authority TDX CLI to assist with Intel TDX remote attestation. The client uses  Intel® Software Guard Extensions Data Center Attestation Primitives (Intel® SGX DCAP) to collect the TD quote. 

The attestation workflow in a KubeVirt-booted TD is the same as in a typical libvirt-booted TD. The user does not need to perform additional steps in the TD. However, administrators who provide KubeVirt-booted TDs must add an additional configuration step when deploying TDs. This is because TD is implemented to support two configurations for attestation — VSOCK mode or TDVMCALL mode. 

Step-by-Step Remote Attestation Example

The steps to run remote attestation in a TD managed by KubeVirt are listed below. For more details, please refer to Launch TD Guest and Attestation in the GitHub intel/kubevirt-tdx wiki. 
 

  1. Add an additional configuration parameter to the TD yaml file for remote attestation. Use either VSOCK or TDVMCALL. 

 

  1. VSOCK configuration —Normally, KubeVirt will not expose a VSOCK device to the container; this device needs to be attached manually. 

devices: 

      autoattachVSOCK: true

 

  1. TDVMCALL configuration —Set up the channel to connect with the QGS backend.

 

launchSecurity: 

      tdx:  

        QGS: vsock:2:4050

 

  1. Deploy the VirtualMachine object to launch the TD. You can also check Intel TDX key usage in the Kubernetes node resource illustrated in Figure 4. 


 

Command line output showing Kubernetes node resources with TDX key usage highlighted.

Figure 4 Intel TDX key usage in a Kubernetes node resource

 

  1. Run the Intel Trust Authority TDX CLI for remote attestation. Perform the following steps to run remote attestation using trustauthority-client (currently available in Golang). For more details on trust authority-cli usage, please refer to the Intel Trust Authority TDX CLI. 
     

    1. Install Intel Trust Authority TDX CLI in TD 
    2. Create RSA key pair 
    3. Get TD quote 
    4. Get token 

Summary

KubeVirt meets the need for unified management of hybrid resources for virtual machines running traditional applications and containerized environments running cloud native applications. With the implementation of KubeVirt for Intel TDX, TEE technology such as Intel TDX is supported in Kubernetes environments. TDs can be created and managed by KubeVirt along with traditional VMs. The Intel Trust Authority SaaS provides remote attestation for Intel TDX. Running remote attestation using Intel Trust Authority in a TD managed by KubeVirt ensures the integrity of the TEE for any relying party in a cloud native environment. 

About the Author

Ruomeng Hao is a cloud software engineer at Intel and specializing in software stack design and integration for confidential computing technology such as Intel TDX. She also has experience in Intel TDX technology adoption in cloud native environments. 

Lei Zhou is a coud software engineer at Intel, specializing in virtualization and container technologies. He is responsible for Intel TDX enabling in KubeVirt and is skilled in confidential computing and zero-trust solutions in the cloud-native environment. 

Ken Lu is a cloud native software architect at Intel focused on confidential computing technology enabling and adoption of Kubernetes framework for confidential clusters and containers. Ken has initiated several open source projects, including CCNP (Confidential Cloud Native Primitives) and CNAP (Cloud Native AI Pipeline). 

Haidong Xia is a principal engineer at Intel, and lead architect for Intel Trust Authority. He also manages the Intel Trust Authority architecture team. Haidong has been working on security for 20+ years, with experience in cloud technologies and platforms. 

Yanhui Zhao is a cloud software architect at Intel, working on enabling differential security solutions based on Intel hardware. He has worked on TEE technologies across different platforms and architectures and driven several related product and service launches over the past few years. 

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.