The purpose of this paper is to introduce developers to the features of the RealSense Plugin for Unreal Engine 4, including its underlying architecture and design principles. The plugin was developed at Intel in support of the Intel® RealSense™ SDK. The intended audience for this paper includes experienced UE4 developers and developers who are familiar with RealSense technology. If you would like to get started learning UE4, check out the documentation and tutorials at https://wiki.unrealengine.com/Main_Page.
You can also download the Epic Games Launcher here and browse through some sample projects in the “Learn” tab.
For more information about RealSense and the Intel® RealSense™ SDK, visit https://software.intel.com/en-us/realsense/home.
Background and Scope
Unreal Engine 4 is the fourth-generation game engine from Epic Games. It was first unveiled to a limited audience at the 2012 Game Developers Conference. In 2014, its source code was made available to all developers for a small monthly subscription. This fee was abolished in 2015, making the complete engine source code available to all developers for free.
The engine source code is written in C++, and in UE4, Epic introduced the concept of Blueprint Visual Scripting, a successor to UE3’s Kismet scripting system. Figure 1 shows an example of a Blueprint script.
Figure 1: Example of a Blueprint script in Unreal Engine* 4.
Developers can create Blueprint nodes for events, functions, variables, macros, and so on, and string them together with logical flow lines. Blueprints help reduce the barrier to entry for developers who have limited programming experience, and they facilitate extremely rapid prototyping.
Many UE4 games are implemented as a combination of C++ programming and Blueprints scripting. Visual scripting can be used to drive animations, define complex AI behavior trees, and implement all kinds of game logic.
The Intel RealSense SDK Plugin
The Intel RealSense SDK is natively implemented in C++, making it easy for developers to access its features from game code also written in C++. This plugin enhances the UE4 developer experience by exposing these same SDK features through Blueprints scripts.
At the time of this writing, the officially supported features of the plugin are the following:
- Raw Color and Depth Camera Streaming
- 3D Face and Object Scanning
Two other features are currently in-progress:
- Head Tracking
- Scene Perception
The ultimate goal for this plugin is to support each of the Gold features of the Intel RealSense SDK that are applicable to gaming. A complete list of the RealSense SDK features broken down by camera and maturity can be found in the RealSense SDK release notes, which can be downloaded here: https://software.intel.com/en-us/intel-realsense-sdk/download
The plugin source code is available here: https://github.com/GameTechDev/UE4RealSensePlugin.
The sample project has a map dedicated to demonstrating each feature of the plugin, with commented scripts detailing the intended usage of the RealSense Components.
If you are interested in developing with any of the experimental features of the plugin, check out the additional branches in the plugin repository.
Also included in the project directory is a Getting Started Guide that walks through the steps to add the plugin to a new project. The following tutorial video will walk you through these steps as well:
The RealSense Components
This section provides a more in-depth look at each of the RealSense components.
The Unreal Engine has a special type of object called a “Component” that is designed to be attached to other objects. Components are typically small and self-contained, providing a limited set of functions to the object to which they are attached. They can be attached to almost any object, including player characters, static objects, vehicles, lights, and HUD elements and are intended to provide the same functionality to dissimilar classes of objects.
To give an example of the power of components, consider the Scene Component, which provides a transform to the object to which it is attached. A transform is a representation of location, rotation, and scale, and it allows a developer to position an object somewhere in the game world. This component makes it easy to add a transform to anything, even lights and sounds.
The RealSense Plugin is architected as a set of components, each of which exposes a specific set of functions from the RealSense SDK. For example, the Camera Streams Component gives the developer access to the raw color and depth streams from the camera. In the UE4 Editor, components are listed in the Add Component drop-down menu in the Components window under the category RealSense, as shown in Figure 2.
Figure 2: RealSense Plugin components as seen in the Component window of the Unreal Engine* 4 editor.
This component is the abstract parent class of all other RealSense components. It provides common functionality required by each component but cannot actually be attached to any object in Unreal.
The data exposed by this class includes the following (see Figure 3).
float ColorHorizontalFOV; // Horizontal Field of View of Color Camera float ColorVerticalFOV; // Vertical Field of View of Color Camera float DepthHorizontalFOV; // Horizontal Field of View of Depth Camera float DepthVerticalFOV; // Vertical Field of View of Depth Camera ECameraModel CameraModel; // Connected camera model: F200, R200, or SR300 FString CameraFirmware; // Firmware version of the connected camera
Figure 3: Real Sense Component member variables.
The RealSense Component also provides the following functions (see Figure 4).
void StartCamera(); // Turns on the RealSense camera void StopCamera(); // Turns off the RealSense camera bool IsCameraRunning(); // Returns true if the camera is running // Get and Set the resolution of the color camera FStreamResolution GetColorCameraResolution(); virtual void SetColorCameraResolution(EColorResolution); // Get and Set the resolution of the depth camera FStreamResolution GetDepthCameraResolution(); virtual void SetDepthCameraResolution(EDepthResolution); // Returns true if the input resolutions represent a valid set. bool IsStreamSetValid(EColorResolution, EDepthResolution);
Figure 4: Real Sense Component member functions.
Camera Streams Component
This component provides access to the raw color and depth images from the Intel RealSense camera, as well as convenient Texture objects to display these images.
The data exposed by this class includes the following (see Figure 5):
TArray<FSimpleColor> ColorBuffer; // Array of 32-bit colors (RGBA) TArray<int32> DepthBuffer; // Array of 16-bit depth values (in mm) UTexture2D* ColorTexture; // Size is equal to the color camera’s resolution. UTexture2D* DepthTexture; // Size is equal to the depth camera’s resolution.
Figure 5: Camera Streams Component member variables.
The Camera Streams Component also provides the following functions, which are overridden from the RealSense Component:
// Overrides the RealSense Component function and intiailizes the ColorBuffer and ColorTexture to the correct size. virtual void SetColorCameraResolution(EColorResolution) override; // Overrides the RealSense Component function and intiailizes the DepthBuffer and DepthTexture to the correct size. virtual void SetDepthCameraResolution(EDepthResolution) override;
There is small library of helper functions called the RealSense Utilities. One of these functions takes as input the Color Buffer and the Color Texture and fills the Texture with data from the Buffer. There is a companion function that operates on the Depth Buffer and Depth Texture as well. Figure 6 shows a sample usage of this function.
Figure 6: Sample usage of the Color Buffer to Texture function.
Watch this short tutorial on how to use the RealSense Plugin's Camera Streams Component to access the color camera stream and display it on screen:
Scan 3D Component
This component provides access to the 3D Scan middleware module in the Intel RealSense SDK. Currently, the plugin supports face and object scanning but does not support outputting textures, only vertex colors.
The data exposed by this class includes the following (see Figure 7):
TArray<FSimpleColor> ScanBuffer; // Array of 32-bit colors (RGBA) representing a preview of what the 3D scan middleware has scanned. UTexture2D* ScanTexture; // Size is equal to the size of the ScanBuffer TArray<FVector> Vertices; // Array of Vector3s representing mesh vertices TArray<int32> Triangles; // Array of vertex indices representing triangles TArray<FColor> Colors; // Array of colors representing vertex colors
Figure 7: Scan 3D Component member variables.
The Scan 3D Component also provides the following functions (See Figure 8):
// Sets the scanning mode (face or object) and a flag specifying whether or not to produce a solidified mesh void ConfigureScanning(EScan3DMode, bool); void StartScanning(); // Starts the scanning module void StopScanning(); // Stops the scanning module void ResetScanning(); // Resets the scanning module bool IsScanning(); // Returns true if the scanning module is running void SaveScan(FString); // Save the scan to a .OBJ file with the specified filename. void LoadScan(FString); // Load a scan from the specified .OBJ file.
Figure 8: Scan 3D Component member functions.
Finally, the Scan 3D Component provides one Blueprint event (See Figure 9):
OnScanComplete(); // Triggered when the scanning module finishes saving
OnScanComplete()Event Blueprint node.
SaveScan()function instructs the scanning module to save and then immediately returns. Sometime later (within one new camera frame) the RealSense Plugin thread will perform the save operation. Developers should use the
OnScanComplete() event to be notified when the scan has finished saving.
The RealSense Session Manager
In UE4, when an object attaches a component, it receives a new instance of that component class. The same component class can be instanced many times and attached to many objects in the same project. This presents a problem for the RealSense components, however. Logically, there is only one Intel® RealSense™ camera that a developer needs to control. But because it’s possible to create multiple instances of the same component class, every instance of a RealSense component will get its own copy of camera data.
To address this issue, the RealSense Plugin uses an object called the RealSense Session Manager, a singleton that is automatically instantiated when the first RealSense component is added to an object in the game. All subsequent RealSense component instances receive a reference to the Session Manager when they are initialized. The RealSense Session Manager owns the master copy of all Intel RealSense SDK data and each RealSense Component reads from and writes to it.
The RealSense Session Manager is responsible for managing a dedicated thread for the RealSense Plugin. It enables the Intel RealSense SDK modules based on which RealSense Component classes are being used in the game. This thread is where the SDK’s
ReleaseFrame() loop runs and each of the enabled modules process the newly acquired frame.
The RealSense Session Manager uses a triple buffer to control data sharing between the RealSense Plugin thread and the main UE4 Game thread. The RealSense Plugin thread writes all new data to a background buffer. After calling
ReleaseFrame(), the background buffer is swapped with a mutex-locked middle buffer. Then, on every tick of the game thread, the foreground frame is swapped with the middle buffer, providing fresh data to the Session Manager and all RealSense components.
The RealSense Plugin gives UE4 developers an easy-to-use interface to the Intel RealSense SDK. It enables rapid development of these features by exposing the SDK to the Blueprints scripting system. And using an architecture built on components, it allows developers to add 3D sensing capabilities to almost any object in their game.
About the Author
Geoffrey Douglas is a Games Engineer in the Developer Relations Division at Intel.