Finn Wong, Intel Corporation
Edward Wu, Smellyriver
VR, as a new form of interaction between humans and the virtual world, is able to immerse users by making them feel like they are experiencing the simulated reality. However, it’s not easy to achieve considering the frame rendering budget for VR is 11.1ms per frame (90fps), and you need to render the entire scene twice (once for each eye). In this session, we’ll focus on both performance and user experience optimizations for VR games, introducing techniques we used in an arena Premium VR game called “Code51” to minimize motion sickness and increase user playtime in VR, as well as what optimizations and differentiations have been done in Code51 to increase both the user experience of players and audience throughout the game.
Code51 is the first worldwide mech arena VR game supporting Oculus Rift, HTC Vive, PSVR and Pico VR. It allows up to four versus four combat among worldwide players and is specifically tailored for VR e-sports, with a nausea minimized gameplay design and a built-in VR spectator mode. Code51 has already been released in over 3000 VR Arcades and experience centers in China (ZamerVR & VRLe), and is targeted to be released on PlayStation Store, Oculus Store and Steam in Q2’18.
Intel worked closely with Smellyriver to optimize the user experience and performance of the game. Moreover Intel and Smellyriver added richer visual and audial enhancements enabled by Intel Core i7 processors, including features like 3D audio, object destruction, enhanced CPU particles and additional background objects.
In this article, we’ll describe seven design points in Code51 that can help increase both the immersion and user experience of VR games.
Currently there are four classes of immersive motion tracking systems to drive player movements in the virtual world of a game, which are:
All of these solutions have different pros and cons. For Code51, virtual cockpit was chosen to be the way to move in VR due to the following reasons:
Fig. 1. Code51 supports various VR helmets. Left: HTC Vive (6 DoF). Right: Pico VR (3 DoF).
Motion sickness is one of the main factors preventing users from experiencing VR for a relatively long period of time. There are several factors causing it, including:
To address the factors leading to dizziness, several approaches were adopted in Code51 to minimize these effects from various aspects
Fig. 2. Dynamic FOV adopted in Code51 to eliminate the optical flows at the edges of the screen. It helps reduce the motion sickness induced to the end users.
For VR application, it’s also important to minimize network latency in order to have a fluent gameplay experience without lagging, for it might generate dizziness if not optimized. In Code51, all motion of mechs are predicted locally, and then being refined through synchronization with the server in a low frequency pace to avoid frequent interruption during bad network condition. If synchronization was failed, the client would extrapolate previous trajectory to get the new position.
To create a better viewing experience for e-Sports audience, a built-in VR spectator mode was implemented in Code51. The spectator mode acts like a client in the game and allows audience to watch a live battle in either stereo or mono mode. Spectators can use VR HMDs or not, and view from any angles and positions they want through keyboard, mouse or gamepad control, as shown in Fig. 3.
Fig. 3. VR spectator mode in Code51. End users can view a live battle in VR or non-VR mode.
Since the PPD (Pixels Per Degree) of current VR HMDs is still low as compared to conventional displays, it’s critical for VR apps to preserve the sharpness of the rendered scene as much as possible to minimize VR blur and reduce dizziness. Currently there are 3 types of Anti-Aliasing (AA) approaches available to the UE4 developers:
CPU optimization is also critical to the VR experience to ensure consistent and fast submits to the graphics pipe, and to prevent stalling the VR scene rendering. To minimize the CPU boundedness of DX11 VR game made with UE4, reduce render thread workload as much as possible. The RHI threads for DX11 in UE4 (4.20+) will help reduce the CPU render thread overhead through D3D11 deferred rendering context, and it should be adopted whenever possible.
In addition, there are various approaches to optimize the render thread workload in UE4, where some of them are deployed in Code51:
Fig. 4. An asset was modified from the left one to the right one to avoid culling calculation for objects behind the asset which can be seen through in the original one.
Last but not least, one key approach to optimize the user experience of VR apps is to utilize all available computational resources on a hardware platform as much as possible in order to deliver the best experience on that platform. Having well optimized CPU compute and higher thread count CPUs enables developers to employ the CPU to deepen the VR immersion experience. For example, a user with an Intel Core i7-7700K has more CPU threads and resources to use than a user with an Intel Core i5-4590 only (the minimum CPU spec. of Oculus Rift without using ASW). Thus the experience of a VR app can be improved if extra CPU intensive visual and audial features are added to the app to consume those extra resources.
In Code51, we implemented several CPU enhanced features to better utilize the CPU resources available on high end Intel Core i7 CPUs. These features include 3D audio, object destruction, CPU particle enhancement and additional background effects, as shown in the following video:
Most of the CPU computation of the visual and audial enhancements is offloaded to the worker threads or the audio simulation thread in UE4, including physical simulation performed by PhysX and ray tracing for physically-based sound propagation performed by the Steam Audio plugin (occlusion and environmental audio). These kinds of features significantly increase the immersion of Code51 on high end CPUs without performance drop since most of the computation is offloaded to the idle CPU cores, where the impact to the critical path of the rendering (CPU render thread and GPU) is minimized.
Fig. 5 shows the frame rate data of Code51 accordingly. The game is able to run smoothly on both Intel Core i7-7700K and Intel Core i7-7820HK with enhanced features on, but drops frames significantly on Intel Core i5-4590 with the same setting, implying that the i5 CPU is not able to handle all the enhanced features in 11.1ms per frame. Instead, for users with the min spec CPUs, they can still maintain a good user experience by turning off all the enhanced CPU features (low quality setting). According to the performance data, Intel Core i7-7700K shows 27% performance benefit against Intel Core i5-4590 in terms of fps(frame per second) when all CPU enhanced features are turned on in Code51.
Fig. 5. The frame rate of Code51 running on different CPUs (Intel Core i5 and i7 processors) and quality settings, where Ultra High is the one with CPU enhanced features and Low is the one without. Test systems: Intel Core i5-4590, NVIDIA GTX1080, 2x4GB DDR3-1600, Windows 10 version 1703; Intel Core i7-7700K, NVIDIA GTX1080, 4x4GB DDR4-2400, Windows 10 version 1703; Intel Core i7-7820HK, NVIDIA GTX1080, 4x4GB DDR4-2400, Windows 10 version 1703.
Fig.6 and 7 show the GPUView screenshots of Code51 in Ultra High quality running on Intel Core i7-6800K with CPU cores set to 4C4T and 6C12T respectively using msconfig, running at the same CPU frequency. It’s obvious from the charts that on 4C4T configuration (a proxy to 4C4T Intel Core i5 processors), the rendering time of a frame cannot reach 11.1ms and frame dropping happens in this case, while it can run smoothly on 6C12T configuration.
Here are the percentages of the CPU computation increased on different threads of Code51 accordingly, running on the same Intel Core i7-6800K in two configurations (6C12T and 4C4T) and observed in Windows Performance Analyzer (WPA):
Total CPU workload: 43%↑
Render thread: 44%↑
Driver thread: 10%↑
Game thread: 13%↑
Worker threads: 89%↑
A significant amount of the CPU works of the enhanced features was offloaded to the worker threads in Code51, leading to the largest computation increase on these threads.
Fig. 6. A GPUView screenshot of Code51 on Ultra High setting running on Intel Core i7-6800K and GTX1080. To demonstrate the scalable experience behavior of Code51, the CPU cores were set to 4C4T by msconfig in this case. Test system: Intel Core i7-6800K, NVIDIA GTX1080, 4x4GB DDR4-2400, Windows 10 version 1703.
Fig. 7. A GPUView screenshot of Code51 on Ultra High setting running on Intel Core i7-6800K and GTX1080. There is no modification on CPU cores (6C12T) in this case, and Code51 employs the additional CPU compute resources to deliver a richer immersive experience. Test system: Intel Core i7-6800K, NVIDIA GTX1080, 4x4GB DDR4-2400, Windows 10 version 1703.
Making VR users feel comfortable and alleviating motion sickness are two critical factors to the success of an e-Sports VR game. Only games with a long playtime and user addiction can succeed in e-Sports. Code51 adopted various approaches descripted in this article to reduce motion sickness, and to deepen the immersive experience, while at the same time create a spectator mode that helps to improve the viewing experience for audience. For performance optimization, render thread bound is usually the main CPU bottleneck in UE4 DX11 VR apps according to GPUView and WPA, standard game optimization methods and offloading tasks from the render thread to worker threads help in this case, it’s also beneficial to leverage all available CPU resources as much as possible to make your game stand out from the crowd, through adding more CPU intensive features such as those implemented in Code51.
Follow us on Twitter: https://twitter.com/smellyriver
Subscribe our YouTube channel: Code51 Smellyriver
 Shibata, Takashi, et al. "Visual discomfort with stereo displays: effects of viewing distance and direction of vergence-accommodation conflict." Stereoscopic Displays and Applications XXII. Vol. 7863. International Society for Optics and Photonics, 2011.
 Fernandes, Ajoy S., and Steven K. Feiner. "Combating VR sickness through subtle dynamic field-of-view modification." 3D User Interfaces (3DUI), 2016 IEEE Symposium on. IEEE, 2016.
 Finn Wong, “Performance Analysis and Optimization for PC-Based VR Applications: From the CPU’s Perspective,” 2016.
 Finn Wong, “Unreal* Engine 4 VR应用的CPU性能优化和差异化,” 2017.
Finn Wong is a senior software engineer in the Intel Developer Relations Division (DRD). He is responsible for VR content enabling and technical collaboration since 2015, helping VR developers to optimize CPU performance and differentiate CPU contents to deliver a truly immersive VR experience to end users. Finn was also invited to deliver tech talks in various VR conferences including CGDC, VRCORE, Tencent GDOC, Unreal Circle, Unity Unite and Vision AR/VR Summit, etc. Before that, Finn worked on the performance optimization in H.264/H.265 and RealSense applications at Intel and has over 10 years of expertise in the fields of video coding, video analytics, computer vision, algorithms and performance optimization, with several academic papers published in the literature as well. Finn holds a bachelor's degree in electrical engineering and a master's degree in communication engineering, all from National Taiwan University.
Edward Wu is the CEO and co-founder of Smellyriver Game Studio. He had many years of in-depth game development experience and led the development of the well-known VR game - Code51, starting from 2016.
Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision #20110804