It’s been a few years since Virtual Reality (VR)* re-entered the public consciousness and reignited the matrix-like fantasies of the 90s and early 2000s. There are now dozens of devices for experiencing VR, and VR platforms have seen their first wave of content released into the wild. However in spite of all of the progress made on the consumer side, the developer side of VR has remained mostly unchanged. Some new SDKs have been released, but the process for creating content for VR is still exactly the same. Though this problem isn’t discussed much, it has a huge effect on the quality and quantity of content that is created for the medium.
In this article, I discuss some of the challenges involved in creating content for VR from a developer’s perspective and look at some ways that our team is changing the process we use so that we can create content more efficiently, more effectively, and more collaboratively.
Flat Tools for 3D Content
For many years now, we’ve been using digital media to tell stories, play games, train ourselves, and communicate. Industries have been built and methodologies have been defined to support the creation of applications that occupy nearly every part of our lives. However VR represents a different paradigm for human-computer interaction, and much of the existing knowledge and infrastructure around creating digital media content doesn’t apply when creating content for VR.
The root of this change stems from the addition of the third spatial dimension that VR enables. We’ve been digital flatlanders and have now been given eyes to see and work within a space that was previously inaccessible.
From The Fourth Dimension: A Guided Tour of the Higher Universes by Rudy Rucker
But we’re still using the tools that were designed for creating flat media, and it turns out that creating 3D media is quite different. Scale transforms from the screen to the head-mounted display, and detail (or lack of) is much more apparent, accessible, and dynamic. Psychology is also a factor, because VR content evokes feelings of personal space and a sense of presence. Interactions with the world happen in three dimensions instead of two. Even visual language is completely different, and things like the “rule of thirds” and simple composition get turned on their heads.
Smoothing out sharp, pointy roots for our music game Shapesong. In VR getting near these felt pretty uncomfortable.
Here are some of the common pain points we’ve encountered when creating VR content:
- Objects appear larger in VR than they do when designing on a flat screen. This means that we have to do a lot of jumping into VR and then back to the screen to get things feeling right.
- Details that are not usually visible become very apparent in VR. For example, users in VR can get inches away from objects, revealing low-resolution textures or lack of depth. They can also look at an object, including its insides, from angles that are completely overlooked on a screen.
- Objects with hard edges seem not only unnatural, but dangerous.
- User interfaces require a completely different design than those for flat media. Photoshop* or other image editing software is much less effective as a result.
- Interactions with VR content and environments require different design, which has yet to be discovered or well defined.
- VR devices isolate the user from the world. This means that when reviewing features or content for VR with a team, you cannot just look over someone’s shoulder to see how things look.
While any of these pain points are immediately apparent the moment you see them in VR, it’s easy to overlook them when designing on a screen. All of this amounts to a “translation” process that has to happen for VR designers, and this translation takes time, effort, and resources. In lieu of VR content creation tools, our goal is to reduce or eliminate the flat-to-3D translation so that we can work, as we have been doing with flat media, in the same space our content will be consumed in.
As described above, the biggest challenge in creating VR content is working around the flat-to-3D translation that must happen, since content creation tools are designed for flat interfaces. Our approach tries to eliminate any translation by shifting, wherever possible, our design process from traditional flat tools to 3D tools. Because no 3D tools have yet been released that are designed specifically for creating professional, production-ready content, we’ve had to get creative with what is currently available.
The following describes the current, high-level process we use.
Our VR content design process: paper -> VR prototyping/mockup -> polish/technical setup
Starting on “paper” is still the fastest way to create an initial concept. This approach is familiar to everyone, can be shared easily, and is supported by a plethora of tools. For us, paper designing means sketching onto notepads, writing descriptions for look and feel, gathering reference, and blocking out initial concepts. This is how we do ideation.
A concept sheet for the Tree Harp instrument in Shapesong. We created the concept here first before bringing it into Tilt Brush* to block out size and layout.
Because tools designed for creating production-ready 3D content do not yet exist, for final authoring we must still rely on their flat counterparts. Though some tools can get us close, they’re still in nascent stages and they leave a lot to be desired. Models created in VR tend to be a bit rough and unoptimized, while tools for other parts of the content creation process (rigging, animating, UV mapping, texturing, and so on) simply don’t exist.
So while we aren’t yet designing content entirely in VR, we are doing a lot of prototyping, mockups, and blocking with VR, which has helped to eliminate the flat-to-3D translation process.
The three tools we’re using are Tilt Brush*, Blocks*, and Medium*. Each of these tools has different strengths and weaknesses that lend themselves to creating certain types of content or design. Below, Jarrett Rhodes and Brandon Austin, two of our team members, share their personal experiences using these tools.
Tilt Brush* by Google
Jarrett Rhodes, Digital Artist
A Tilt Brush* sketch of the Tree Harp instrument from Shapesong. With this, we were able to quickly create designs for a number of layouts to play test in VR.
Using Tilt Brush is a great way to explore and conceptualize flow lines for interactive set pieces for VR. Working with lines makes it act like a sketchbook for getting down ideas and visualizing the placement of elements in a virtual space, which can then be exported as a guide to help you lay out your assets when creating them in a traditional digital workspace.
With Tilt Brush, it’s easy to start throwing down flow lines to get a feel for the interactive space intended for player use when it comes to set pieces. Also, working with lines makes it easy to add instructional lines and notes that describe how a concept is intended to function when another user enters this scene. This ability is great for communication, because it can be difficult to explain specific intent when someone else is in the scene with a headset, which creates a barrier between you and them.
When it comes to concepting a scene as opposed to an object, dynamic brushes allow you to add particle effects that can help push the overall feel of the scene further. Dynamic brushes also react to any music you have playing in the background, which helps make the scene feel more alive as it pulsates to the music.
Because you primarily work with lines, it can be difficult to create volume. I found that creating a wireframe first and then painting around in a paper-mache fashion with a wider-textured brush can get the job done, but it is a bit more time consuming. Also, when creating lines, you cannot erase parts of a line if you overdraw it or want to clean it up. You have to undo the entire line and then try to re-create it on the next or subsequent passes, which is time consuming.
Although you have many brush types and colors to choose from, exporting a scene with a lot of linetype variety can muddy up the scene with a plethora of shaders when you import it into a traditional digital media program such as Maya*. However, usually the intention is not to clean up the mesh, but to use it as a guide to build around to ensure interactive items translate better when used in VR.
Finally, if you don’t already have an idea of what you want to create before jumping in, it can be difficult to get started. Currently, you cannot import any reference images to bypass this issue, which is a sizable disadvantage. Although you can import scenes from the social network, it is unlikely that you’ll find precisely what you’re looking for.
Although it can be difficult to create a full-bodied asset from just line work within VR, especially if you don’t know in advance what you want to create, using Tilt Brush to create concepts within a virtual space has its advantages. It’s one thing for a 3D asset to look good from a 2D screen, but once a player can interact with it on a 1:1 scale, concepting out a rough sketch for more accurate proportions is quite useful.
Blocks* by Google
Jarrett Rhodes, Digital Artist
Google Blocks allows you to easily create simple geometric volumes to create low-poly assets in VR and block out rough concepts. Working with simple shapes is a more minimalist approach, which forces the user to think more about the core design and structure to create an asset rather than focus on the minute details. Even if you’ve never before worked with primitive shapes in any other 3D modeling program, Blocks is quite easy to learn.
Not only are you working with primitive geometrical shapes, you are also using just six simple tools: shape, stroke, paint, modify, grab, and erase. Before creating a shape or stroke you can select how many faces or sides it has, essentially choosing shapes from pyramids and triangles to near spheres and octagons. You can also move each shape you make individually or even select a few and move them as a group. Being able to separate and edit an individual shape is an advantage in being able to perfect a single object. And having a three-axis grid to snap allows you to organize and edit objects more linearly.
Blocks is a great VR design tool to use if you’re making low-poly assets, because of its foundation of primitive low-poly shapes. However you aren’t restricted to making only primitive shapes. With the modify tool you can add extra edges to round out your designs or the stroke tool to make curving geometric shapes such as tubes. The modify tool also lets you move, rotate, scale, and extrude faces, vertices, and edges, which follows a similar workflow to box modeling. Being low-poly means that any creation you export will probably require minimal cleanup after exporting to another 3D modeling program.
Also, Blocks is free!
If you want to make something more detailed to compare to current-gen game models, Blocks might not be capable enough for you. While it can still be useful as a concepting tool to use as a reference for a higher poly model, other VR design tools, such as Medium (see the next section), cater more to that workflow.
Another minor drawback is that you can’t quickly select the tool you want to use. You have to select the one you want by grabbing it from the menu or palette rather than being able to swipe the trackpad as you can with some other VR tools. And even though using the grid can help you do more precise editing, without the grid, modifying shapes can be a bit difficult and give awkward results when you want a linear modification.
If you are a low-poly artist, using Blocks is ideal. It follows a similar workflow working with primitives, and being able to import with minimal clean up means you can spend more time creating assets. Regardless of your skill set, it’s easy to learn, and though selecting tools might be a little clunky, for a free VR creation tool you can’t go wrong.
Medium* by Oculus
Brandon Austin, 3D Artist
A Medium* sculpt of a virtual avatar we worked on. We were able to take the final version of this and make only minor modifications in Maya before using it in one of our apps.
Oculus Medium is a fast and intuitive way to generate high-quality 3D sculpts from within a VR environment. The use of a natural interaction system that seeks to mimic real-world sculpting eliminates the barrier of clunky mouse and keyboard commands and 2D abstraction. The typical workflow begins with spraying clumps of clay into a basic form, and then carving away fine details until finally applying a rough paint job and materials.
Medium let you create basic shapes and forms in a matter of minutes. The gestural nature of holding virtual sculpting tools and moving them in 1:1 3D space means assets will have a more natural look, and creating complex shapes such as spirals or curves becomes trivial. This ability also leads to models having more of a “sketchy” feel as the strokes pick up minute hand movements.
The seamless scaling of the workspace by pulling apart/pushing hands together allows for fine-detail work as well as an overview of the whole mesh. There is also the advantage of being able to walk around and view the mesh naturally in a 3D environment.
The built-in materials and paint tools allow for rapid iteration of different colors and textures without bringing the model into a separate tool. You can use multiple resolution levels to block out a basic mesh, and then sample it up to add detail. Saving “stamps” at a high resolution allows you to easily fill an object with reusable samples like leaves, scales, or even skin pores.
Medium offers a variety of sharing tools for exporting videos, images, and recordings that other Medium users can play back. This is a great feature for portfolios or dev logs when accompanied by Medium’s fantastic user feedback and sound effects.
When using Medium, it’s important to realize that having to work within a VR headset for an extended period of time is more stressful on the eyes and arms than sitting at a traditional modeling workstation PC. Because of this increased strain, frequent breaks are absolutely required for most users.
The screen resolution on most VR headsets leads to a more blurry appearance than on a traditional screen. Resolution has a big impact on the visibility of small details and picking out minor errors with the sculpt.
Reference image importing is supported, but setting up orthographic references for front, side, and top has proven difficult. This problem is slightly alleviated by importing low poly base meshes from another program as a “stamp.”
It’s easy to accidentally rotate or scale the workspace, which can throw off the mirror tool or the pottery-style turntable. Separating the mesh into layers helps segment the piece and minimizes the impact of user errors.
Finally, Medium uses a proprietary file format, but can export as FBX in multiple resolution levels. However, the exported FBX will likely still require cleanup in a separate modeling program. I have found that using the retopology tools in Zbrush* worked the best, since it is already tailored to working with large vertex count files, but Maya should suffice for most models.
Medium is a great tool for throwing a rough idea into a 3D space and seamlessly iterating on it. Since hard-surface modeling can be a challenge, mechanical models are still easier to create in traditional tools like Maya or 3ds Max*. For artists used to working with physical media or working in ZBrush, this could be the next step in speeding up the creative process and experimenting with a unique workflow.
Working in VR is an inevitable outcome for many professions and trades. The power of the medium to give us complete control over our environments and the things inside them, as well as communicate with each other over great distances naturally, will be too great to ignore once the medium has matured. We’re creating digital domains for ourselves in which we will be gods.
But we’re not quite there yet. We still need to create the tools that will help us build the worlds we will inhabit. Without them, we have to think creatively about the best ways to leverage what we have now, so that we can build what we’ll use tomorrow.
People like to say that you need to “get your VR legs” before you can really enjoy VR properly. I think that sentiment also reflects our current state and that of VR itself. As virtual explorers we are evolving and just emerging from a lifetime of ocean dwelling with our flat media. There is a whole new digital, 3D world to discover, and we’ve only just begun to adjust to walking in it. The possibilities for this frontier are profound and completely open, and they lay just beyond the horizon. All we have to do is pick a direction and start walking.