By Matthew Baxter-Reynolds
Download Article Or View Video
Download The Human Touch: Building Ultrabook™ Applications in a Post-PC Age [PDF 715KB]
Watch How Multi-Region User Experience Influences Touch on Ultrabook
We’re used to interacting with our smart phones and Apple iPads* using touch. Yet, when we use a desktop or a notebook, we typically use a keyboard or mouse. With touch capability becoming available in more and more Ultrabook devices, Intel undertook a research program to better understand if and how people might use touch capabilities in more traditional, notebook form-factor devices.
To spoil the ending, the results were positive-very positive, in fact. Users who were presented with a way to interact with their computers via touch, keyboard, and mouse found it an extremely natural and fluid way of working. One user described it using the Italian word simpatico-literally, that her computer was in tune with her and sympathetic to her demands.
Let’s look at the research Intel’s team undertook, and then at how we can bring the lessons learned from that information forward into our applications.
Scoping the Problem
It’s easy enough to be dazzled by the appeal of developing applications for smart phones or building a new, snazzy website. However, the reality is that most of the software development that happens in the corporate arena relates to internal, proprietary, line-of-business applications. Being able to optimize these existing applications for touch can realize real benefits, as can being able to engineer new applications with touch capability in mind.
Who and Where
The research was led by Daria Loi, Ph.D. Daria holds the position of UX Innovation Manager at Intel and has a specialist interest in how people use and adapt to new technologies in user interaction.
Daria’s research took place with individuals in the United States, Brazil, Italy, and China. In each case, she and her team took pains to reproduce the subject’s environment as closely as possible. For example, in China, it’s typical for users to use their device in bed, so the test environment was rearranged to simulate a bedroom. The subject was taken through the same suite of exercises-specifically, changing settings, creating a Microsoft Office PowerPoint* slide, writing an email message, browsing and searching, online purchasing, video calling, and cropping an image. Subjects were a mix of those who had experience with touch on a day-to-day basis (e.g., with a smart phone or tablet) and those who were new to working with touch.
The team found that users would naturally move among using touch, mouse, or trackpad entry, or keyboard entry depending on the task at hand. This finding was really the key takeaway of the research. Unconscious cues instructed the subject as to the optimal approach on a step-by-step basis. This behavior occurs in normal, non-touch scenarios anywhere. Consider situations in which you might press a keyboard shortcut to save a document but use the mouse to click an icon on a toolbar. Introducing touch brings in another vector that users can choose at each step in a complex task. Touch is a basic, organic thing to do, making it sympathetic to users’ needs. Users will conceive of a need to activate a given button; they can then simply reach out and touch it, much as they might an object in the real world, if that approach suits them at the time.
Remember that although people often talk about touch as “sophisticated,” it’s not. It’s the most basic of our capabilities as humans.
Likely because touch is such a natural thing to use, users reported that touch transforms the notebook from a work device into a play device. Daria and her team felt that this was because touch enables people to feel immersed, transforming a laptop into something delightful to use. (Despite the implied association with work, they still wanted the keyboard.) Figure 1 shows the breakdown among interaction types.
Figure 1. Device interaction by input type
Touch also allows a new form of interaction: “flicking.” Using “momentum scrolling,” a fast and urgent flick of the finger can scroll a list a great distance compared to using the wheel on a mouse. Scrolling long distances using a mouse wheel is often tedious. Using a scroll bar with a trackpad is often both tedious and fiddly. However, flicking is neither. It’s simple and effective at moving large distances quickly.
Importantly for this class of devices, users reported that although they would be willing to get rid of their trackpads-and to an extent, mouse devices-they were not willing to get rid of their keyboards. With the Ultrabook devices, Intel has been working to make the notebook proposition even more portable. Touch helps with this. Rather than having to transport the notebook as well as a mouse, the user just carries the notebook. (This is, of course, analogous to the way that keyboards on laptops have always worked.) Intel’s researchers had found in the past that users are generally hostile toward trackpads because of the fussy and unnatural way in which they operate. Trackpads represent a compromise in notebook form factors, whereas touch is not. Touch is a first-class input capability, just as the keyboard is.
As part of this element of the work, the team wanted to understand how subjects would feel about convertibles and, by extension, tablet computers. The subjects were generally not keen to do away with their keyboards, even if it was just a temporary disconnection from the device. Convertible devices, although not directly tested, were considered appropriate by the test subjects. What was clear is that the subjects generally wanted to keep the work productivity space, yet have it enriched.
Conventional wisdom when approaching touch suggests that it’s a failure right out of the gate because people will find it uncomfortable to keep reaching up to the screen. The team found that this was actually not the case, and all of the subjects adapted their behavior to support this new input approach by shifting their body around to be comfortable, leaning on the available work surface, and so on. In fact, because of the duality of working with touch or the mouse or trackpad, subjects were able to switch between one mode and the other depending on how they felt. Far from introducing so-called gorilla arm, users reported dramatically increased comfort levels working in this way.
Another piece of conventional wisdom that Daria was keen to explore was the idea that tapping the screen would tip the whole device back. What the team actually found was that subjects were careful, yet confident and would touch with appropriate pressure to register their intent without tipping the device back or pushing the screen back on its hinges.
Before looking at how to apply touch optimization to applications, let’s look at some quotes from the subjects:
“I just LOVE the interaction without having all these peripherals (mouse, touchpad) . . . . You gotta make me take this home with me!” (Marcus, 49-United States)
As she points at the touchscreen: “I like touch so much that I found every once in a while I forget I can’t do that on my computer.” (Susan, 33-United States)
“I like the scrolling, [because] you can kind of flick your hand and go quicker.” (Betty, 22-United States)
“WOW . . . this is easy . . . . It’s almost reading your mind. You think of it and do it. Just touch it.” (Pamela, 49-United States)
“It’s easier to go directly there and click with your finger.” (Franco, 21-Italy)
“There is something about the interaction of actually touching the screen and pressing a button that feels more engaging.” (Sonia, 26-United States)
“More immediate, faster . . . more amiable.” (Alessandra, 44-Italy)
“Having a laptop with touch is having a laptop with an extra gear.” (Pino, 35-Italy)
“I like it actually . . . it’s freeing.” (Kyle, 30-United States)
“It seems more . . . connected.” (Jacob, 39-United States)
“It makes the experience more interactive.” (Marcus, 49-United States).
“It’d be nice to kind of have a little variety . . . switch around . . . more than being stuck in one spot” (as he points at the mouse) “. . . the whole time” (Mark, 37-United States)
“It seems almost more comfortable . . . using the mouse I feel it might be a little strain on my wrist.” (Kyle, 30-United States)
“Even like, ergonomically . . . feels better reaching in front of you than constantly moving on the side.” (Jodie, 25-United States)
“I actually prefer [the screen] being propped up as opposed to being flat (as a tablet). It just creates a more ergonomic landscape.” (Meredith, 32-United States)
“I get really frustrated with the mouse, so any opportunity to use touch . . . I’ll use it.” (Sonia, 26-United States)
“I like touch, but for typing I prefer the feel of the keyboard.” (Marcus, 49-United States).
“I think trackpads are kind of a pain.” (Jacob, 39-United States)
“I’ve been using THAT [trackpad], and it’s very tedious.” (Pamela, 49-United States)
“When you and I first sat down here, I thought I’d use the mouse more than I am doing. I am surprised that I am not.” (Alma, 47-United States)
Applications in the Enterprise
Enterprises generally have a mix of applications that are developed in-house or purchased from third-party vendors (perhaps with customization) and those applications that are web-based intranet/extranet applications or desktop applications. The ability to optimize the applications for touch varies by application class. In addition, applications have different audiences within the business who in turn have access to different forms of hardware. Employees who spend a lot of time on the road usually have notebooks. Employees who work primarily from the office typically have desktops.
An important secondary consideration is whether the audience for the application consists entirely of people who use notebooks or whether there is a mix of notebook and desktop. If the usage is exclusively notebook based, you don’t have to consider being able to turn on or off touch optimization.
Touch optimization is not necessarily about making something touchable, however. For example, when using touch, the user interface (UI) “targets” (buttons, fields, etc.) need to be larger to compensate for the lack of accuracy. But, when you make things bigger, you lose real estate. In applications that need a mixed approach to input, it’s desirable to turn on touch optimization (for example, make everything bigger) or turn it off.
Whether the application is a legacy application or a new engineering effort has no direct relevance on touch optimization work. You’ll be able to construct a quantified business case around touch optimization regardless of whether you’re modifying existing software or undertaking a new project.
The next two sections look at ways to introduce touch optimization to desktop applications and web applications.
Touch Capability in Desktop Applications
Looking back at the way the research subjects approached touch capability, they tended to move between touch and the mouse depending on subtle factors outside of conscious awareness. What this tells us is that the most comfortable mode of operation is not one where we are insisting that they use one method or another; rather, it is indeterminate. A user may click a button on a form using the mouse 80 percent of the time and use touch 20 percent of the time one day and entirely reverse this pattern the following day. Whatever you design has to suit either input mode. Of course, if you fail in your design and end up making the user’s input decision for him or her, frustration will certainly ensue.
Incidentally, the choice of platform and framework used in the application’s construction is probably not relevant when talking about touch optimization. For example, Java* and Microsoft .NET are equally capable of supporting developers introducing touch in that neither platform considers touch a first-class way of providing input into an application. (At the time of writing, that’s true. Over time, this view is likely to change as touch works its way backwards from post-PC devices to PC hardware.) Hence, any work you have to do you’ll have to do yourself, although such work tends not to be onerous. Really, all you’re trying to do is be sympathetic to the user.
For desktop applications, the major consideration is target size. Many business applications are form based, and such, applications tend to have a reputation for being tightly packed.
Touch relies on the operating system being able to intuit what you wanted to click by considering the area you press with your finger “fuzzy.” Whereas a mouse always registers an exact, single-pixel coordinate under the hotspot of the cursor, the operating system assumes the input from a touch screen is likely to be inaccurate and compensates accordingly. For example, if the touch screen reports that you pressed just outside of a field, the operating system may choose to interpret this as a click in the field, as you are more likely to want to click in the field than you are to click outside of the field. Therefore, one way in which you can help the operating system is to space the fields so that presses outside of fields can be more reliably mapped as intentions to click in fields. The same applies to buttons.
Another way to help the operating system is to make the fields larger, for the simple reason that a larger field is easier to hit. Generally though, the standard presentation of buttons and fields is of an appropriate size for touch or mouse operation.
Oftentimes, forms get cramped, because software engineers want to get everything onto a single form; so, it’s easy to find applications that have fussy or fiddly forms. When touch-optimizing this sort of form, you may find it appropriate to break out individual functionality into tabs. Of course, you also have to make the tabs an appropriate size. By default, tabs are typically too small, probably because designers generally considered them a secondary UI element. For touch optimization, you’re turning that secondary element back into a primary element by making it larger.
Toolbars and menus are another consideration. Menus are usually managed by the operating system, and as such, you can assume that the operating system will interpret touch in an optimal way when working with menus. That said, because it’s likely the user will have a keyboard, you may choose to make some of your touch optimization work include increasing keyboard accelerators for driving the menu, thereby generally reducing the need to use the menu.
In terms of toolbars, you may find that the buttons are too small. Toolbars are necessary, but developers tend to make them as small as possible, because they are a secondary feature compared to the display of the data you’re working on.
The bigger problem with toolbars is tooltips. Introduced with Microsoft Windows* 95, these tips allow users to use the mouse to obtain a deeper understanding of what a button does. (Although a picture speaks a thousand words, toolbar icons are often cryptic.) The team’s research didn’t include a test as to whether users took advantage of tooltips, but it’s easy to assume that users wanting to know what a button did might well hold it down and expect more information to appear. This functionality is certainly doable, but consider that the user’s finger and hand will be obscuring the screen around the button, etc. Therefore, you will need to consider where you display the tooltip. As tooltips are a standard feature of most toolbar libraries, this change of presentation may require additional work.
Traditional desktop applications often rely on pop-up forms and dialog boxes. The only consideration here is that people often want to move and resize these items. The operating system will, of course, do this for you. Moreover, the operating system should do a good job by itself of handling touch interaction on captions and borders. Therefore, verify that your application works adequately in this regard out of the box.
Other Types of Applications
So much for basic forms applications. What about those applications with more novel, specialized interfaces?
From time to time, developers build custom interfaces with the expectation that users will drag items around on their surfaces or scroll around. Gantt views and schedule views are good examples of this sort of interface. In the first instance, you must consider target size, just as you would a normal form.
In the second instance, you may run into a problem where the operating system is limited in the help it can give you in interpreting touch placement. When the operating system is asked to display a form, it has an understanding of the construction of that form and therefore is able to make decisions like, “The user clicked in a blank area; I suppose he or she actually wanted to click this field.” If you custom draw the whole interface, the operating system just sees a rendering surface without deeper meaning. As such, you may need to replicate the fuzzy matching that the operating system does on your own custom controls.
Zooming and Scrolling
Operating systems typically interpret scroll intentions properly. A word of caution, though. If you use third-party control libraries in your application, you may run into problems. Although the operating system knows that one of its list box controls is a list box and can infer a scroll intention, that may not be a given for a third-party control. That said, it should generally be the case that scrolling works alright, as even third-party controls tend to build on and extend the regular operating system controls.
One area where you may find yourself working harder is zooming. “Pinch to zoom” on touch devices is natural, but there’s no built-in support for it on the standard controls.
Note:As a developer, if you’re working on touch optimization, you don’t have the freedom of moving between touch and mouse. It’s a given that whatever you do will work with a mouse. You need to keep on top of whether an action works with touch.
Touch Capability in Web Applications
When considering touch in web applications, generally speaking we’re talking about the same rules as for desktop applications with regard to target size and placement. Fields need to be larger and more widely spaced. From an engineering perspective, however, it’s likely to be easier to make such modifications, because HTML layout is designed to be more dynamic and judicious adjustments to the cascading style sheet (CSS) may require little work to actually implement the change.
Web applications tend to be more simplistic than desktop applications-at least for the sorts of applications you find in enterprises. More inventive ways of building interfaces can be found in Web 2.0 applications. Enterprise-type web applications are typically forms based without terribly sophisticated, complex, custom UIs. As such, after you’ve negotiated optimizing against the target size and placement vector, you should be fairly well set.
There are some wrinkles in these applications, though. A major one is that it’s unlikely the browser of choice will be touch optimized, particularly if the organization has standardized on an old browser for compatibility reasons. Windows* Internet Explorer* 10-the default browser that will be supplied with Windows* 8 and Windows* RT-will be touch optimized. Older versions of Internet Explorer are not. Other browsers probably will not be. This suggests that testing and tweaking any touch optimization work is likely to be more difficult with web applications.
The biggest problem with touch-optimizing a web application is that links tend to be too small for users to touch with confidence. In situations where the user is using a touch-optimized browser, it’s likely the browser will do a decent job of interpreting the user’s touch intentions. However, if the user is not confident that his or her touch will have the desired effect, this will damage the experience and limit the value from the work done. Plain links should generally be avoided in web applications and replaced with buttons. (These buttons don’t necessarily need to be that much larger. You’re looking to increase confidence primarily. Spacing remains important, however.) Again, thanks to CSS, links can often be automatically stylized as buttons.
Dialog boxes and pop-up windows do appear in web applications from time to time but with less frequency than their desktop cousins. The same rules apply here, too, though. Make sure that people can move them around and resize them. Even with a touch-optimized browser, you’re unlikely to get much help supporting such functionality, but it is easier to make artifacts such as captions different sizes compared to desktop applications.
It’s becoming increasingly common to find people working in software development talking about “user experience.” Far from being an idea about fashion or a sop to allow developers to ignore what users need, increased understanding of the impact software has on users is a factor that comes about through maturation of the industry. Developers and project sponsors tend to be more involved and sympathetic to how users feel about the software they are asked to use.
What’s clear from the research that Daria and her team undertook is that the user experience can be improved-dramatically-by involving touch. It turns a work device into a play device, and in a commercial setting, that’s not a bad thing. Instead of thinking about it being play, think about it as being the software getting out of the way and allowing the user’s creativity to come to the fore.
This article looked at the research and its conclusion, but it also explored how legacy and new applications can be optimized for touch. Users will tend to switch between touch and mouse operations without conscious consideration. You can deliver value within the application by making targets larger and working with space. Remember to keep your testing efforts focused on touch, trusting that more traditional manipulation with the mouse will work regardless. Either way, with a bit of effort, you can use touch to deliver better, more engaging, and more compelling applications to your user base.
More About Daria Loi
Daria Loi is a designer, user experience architect, and creative lead, with a passion for participatory design, ethnographic and practice-based inquiry, natural UIs, and creative management. At Intel, she holds a UX Innovation Manager position in the PC Client Solutions Division. Prior to this role, she worked as a research scientist in Intel Labs and Intel Digital Home Group. Before Intel, she worked as an architect in Italy, a senior research fellow at RMIT University, and lead researcher in the Interactive Information Institute in Australia. Daria holds a Ph.D. in Management and a BArch with specialization in industrial design.