Using Sensors and Location Data for Cutting-edge User Experiences in Mobile Applications

Using Sensors and Location Data for Cutting-edge User Experiences in Mobile Applications (PDF 1.44MB)

Overview

Modern mobile devices like tablets are equipped with various sensors (accelerometer, gyroscope, magnetometer, GPS, etc.). To fully explore the use of the fused motion and location data, this article provides information on the best practices and importance of integrating/filtering/tuning the sensor data in end-user applications. The ultimate goal is to empower application developers to create cutting-edge user experiences in their applications.

This article discusses:

  • Overview of sensors, data provided by Windows* Runtime APIs, usage scenarios, and challenges on using the sensor data in applications
  • Illustrative examples on how to integrate, filter, and tune sensor data in user applications for an immersive, seamless end-user experience
  • Optimizing the location user-experience by tuning location parameters and filtering location data noise

Sensor Overview

The types and form factors of devices available with sensors is continuing to expand. Various phone designs, a range of tablet sizes, UltraBook™ devices, convertible tablets, etc. are just a few examples of the many form factors available with sensors. These unique devices open up interesting opportunities for developers to create sensor-based applications.

In addition to new form factors, the types and capabilities of the sensors available to consumers and application developers are ever increasing. Most mobile devices today are equipped with accelerometers, gyrometers, magnetometers, GPS, Wi-Fi*, touch displays, Bluetooth*, NFC, and more.

Sensors that were once available only internally to the device such as thermometers are now making their way to device APIs. The additional sensors available to application developers create new opportunities for developers to create compelling applications.

Motion Sensors in Windows 8 Runtime

New for Windows 8 UI application development is a powerful set of APIs for using sensors. The APIs include a set of common interfaces for most of the sensor components and provide both event and polling-based mechanisms for retrieving data.

The following image shows an overview of the different components involved in using the Windows 8 Runtime for motion sensor application development.

Inputs to the Windows 8 Runtime displayed are the raw sources of motion sensor data. The Ambient Light Sensor is shown for completeness since it uses the same common API for data retrieval. Outputs of the Runtime expose several components available to you as an application developer.

These include components that use multiple sensors to provide sensor reading data such as the Compass, Inclinometer, and Orientation Sensor.

Windows.devices.sensors is the namespace you will find the APIs to retrieve sensor event data. The following snippet is C# code showing an example of how to retrieve the accelerometer present in the system and setup a method AccelerometerReadingChanged that will be called whenever the accelerometer data is updated by the system.

Accelerometer acc = Windows.Devices.Sensors.Accelerometer.GetDefault();
if (acc != null) {
acc.ReadingChanged +=new typedEventHandler(AccelerometerReadingChanged);} 

The interval at which the event will be called is configurable. The property minimumReportInterval will tell you what the highest possible frequency is for this sensor. This value changes from device to device and is also influenced by the driver.

Location Sensors in Windows 8 Runtime

The location sensor APIs have a similar event-based interface and provide a common framework for retrieving location data that could come from a variety of location data sources.

The diagram below illustrates how Windows 8 Runtime uses a variety of sources of location data, multiplexes the geolocation data from all available location sensors, and returns the “best” location data.

The Windows 8 Runtime namespace windows.devices.geolocation is used to access the geolocation information. Creation of a location object is done with the following code:

Geolocator geo = new Geolocator();
To poll and detect the device’s current location use Geolocator.GetGeopositionAsync(). 
if (geo != null) {
IGeoposition pos = await geo.GetGeopositionAsync();
}

This could be a lengthy operation and is denoted as being asynchronous. The use of the await keyword makes this easy to handle in your UI code. By using await, the calling thread is not blocked while this call is made and once GetPositionAsync returns, code execution continues after the async function. To respond to location updates, hook an event callback geo_PositionChanged as below:

if (geo != null) { 
geo.PositionChanged += new TypedEventHandler(geo_PositionChanged); } 

Additionally, tunable options for controlling how often you are notified about events include desired accuracy, report time interval, and movement threshold.

Motion Sensors

Accelerometer-, compass-, and gyroscope-based apps are some of the most popular and have a variety of applications including:

  • Augmented Reality
  • Games
  • Orientation
  • Pedometer
  • Navigation
  • Remotely-controlled Devices
  • Biofeedback

The Windows 8 runtime provides robust APIs and sensor fusion to incorporate in your applications, but sometimes, applications need to build on top of this for the best user experience. The following sections go into detail about how some of the common motion sensors work, what can be done to improve the data of the sensor, and what are some of the limitations you might run into.

Accelerometer

The accelerometer sensor measures how fast the velocity of the device is changing over time. This could be measuring the device shaking, moving from point A to B, or rotating. The following image displays some characteristics of the accelerometer as the device is rotated by an angle counterclockwise.

Using gravity to our advantage, we can quickly determine the angle of rotation by looking at the angle between the X,Y accelerometer values and the –Y axis. The alternative way to determine the rotation would be to isolate gravity and measure just the changes in the accelerometer values that occurred during the rotation. You would then integrate the values to give you velocity and once again to give you distance travelled.

One major issue with both of these methods is that the data source of the accelerometer is rather noisy. The following graph looks at the raw sensor readings as the device is rotated 90 degrees.

Starting at the left of the graph, the device is in a landscape position with the -Y axis pointing straight down. Here you see the effect of gravity with a -1 value. After the rotation is complete, the –X axis is pointing straight down and gravity is now reporting -1 g’s, while the Y axis no longer has any gravity component. The other notable characteristic of the graph is the amount of noise or jitter in the lines. Using this raw data directly in your application may not produce ideal results. One straightforward algorithm to deal with the jitter is a smoothing function that looks like this:

yi=a*xi+?(1-a)*yi(i-1) a= .15

Corresponding C# code might look like the following:

CalcRollingValue(double val, double prevVal)
{
 	return val * .15 + .85f * prevVal;
} 

This rolling average function removes high frequency noise by only allowing 15% of the current value to be used at any new data point. The downside is that a large change in the accelerometer values that you cared about would not be fully realized for several samples. As a result, the smaller the alpha value the larger the lag introduced. Here is the same graph of a rotation with the smoothing filter applied:

Immediately visible is that the noise visible in the original graph is much more subtle in the filtered graph.

One other way to use the smoothing algorithm is to assist in isolating the gravity component of acceleration values. To do this, subtract the rolling value from the instantaneous value and you are left with only the change in value from the average. The following code and graph illustrate what this looks like with the same rotation of 90 degrees:

float accelX = currentAccel.x - rollingValueX;
float accelY = currentAccel.y - rollingValueY;

A further improvement on the rolling filter is to use more advanced algorithms such as a Kalman filter. The filter has a couple of interesting properties that allow it to provide better results. First, the algorithm is able to use more than one source of data allowing it to use both inputs as sources of measurement data and provide more accurate results. Second, the algorithm has a two-phase approach that allows it to adapt based on the measurement noise and then update and correct the prediction for the next measurement. The following diagram illustrates the approach:

The following graph shows the accelerometer output of a game that requires a quick response in order to play well. The time span of all the motion data is only about 800 ms and varies quite a bit in that short time. You can see in this case, the Kalman filter does a pretty good job of capturing the large changes in accelerometer data but still creates smooth data values.

Knowing the limitations of the accelerometer data available to the application is important. The following example of one of the limitations comes from an attempt to build an application that detects directional tap on the device using the accelerometer. In the hardware accelerometer the pulse caused by tapping the side of the device might look something like this to the sensor:

A large pulse followed by the signal quickly dampening, using this level of detail an algorithm could be created to figure out that the device was tapped and what direction it came from. However, the fidelity of the data at the application level is not this good and the following images illustrate what happens as the sampling rate is increased and the samples don’t align with the pulse as expected:

With even higher sample rates, writing an algorithm to detect the pulse is going to be difficult:

Gyrometer

The gyrometer measures rotational velocity in radians or degrees per second. To use the sensor to measure rotation of the device, the time between readings must be stored and multiplied by the velocity in order to calculate distance travelled.

double angle = -reading.AngularVelocityZ * secDelta;

The angle could then be used directly in your application.

The major drawback of the gyrometer is that it knows nothing about orientation of the device so over time any error in the gyrometer values shows up as drift in your application.

Sensor Fusion

Sensor fusion offers the best solution by using multiple sources of data to report the orientation more accurately. Accelerometer and gyrometer data are both used as inputs to the fused orientation output. Depending on your application, the orientation can be consumed as pitch, yaw, roll angles via the inclinometer or more advanced representations in the form of quaternions or a rotation matrix.

In Windows 8 some of the aggregation and processing of sensor data is done in hardware, which provides for greater accuracy and less power consumption. Using the inclinometer’s yaw reading, the application can measure the same rotation of the device.

float angle = reading.YawDegrees;

The angle of rotation around the z-axis is given as yaw in degrees.

Location Sensors

Most modern tablets, mobile phones, and PCs have location sensor capabilities. These devices normally use multiple positioning methods to provide different granularities of location data. The sources of position data vary in terms of accuracy, startup time, and power signature and include the following:

  • GPS
  • A-GPS
  • Cell tower triangulation
  • Wi-Fi triangulation
  • IP Address

Applications that make use of location sensors range from games to navigation and some examples are shown below:

  • Search for Points of Interest (POIs)
  • Geotagging ? adding location information to files such photos
  • Games ? Geocaching
  • Outdoors and fitness
  • Pedestrian or vehicle navigation

A few common sources of location data inaccuracy include IP resolution over corporate intranets, Wi-Fi AP moves, and cell tower ID movement. The following image shows what happens when you are indoors using only IP resolution and network traffic is being routed through corporate headquarters:

Although the device was physically located in Arizona, the location data indicates positions in Arizona and California.

Additional sources of location issues arise from GPS data error caused by situations such as being indoors, atmospheric delays, and building or tree interference. The following image shows a route being walked through the front lawn when the actual path never strayed from the sidewalk.

The following sections look at possible methods to increase the accuracy of the data used in location-aware applications.

Filtering Location Data

geocoordinate.accuracy is used to retrieve the accuracy of the location in meters—the larger the value, the less accurate the data. A simple filtering mechanism is to throw out values that are not useful, use the geocoordinate.accuracy value to filter out the location sensor data that exceeds the accuracy threshold.

if (position.coordinate.accuracy > X) { 
mark the location data invalid;
}  

In situations where the device is moving, we can identify some GPS data errors using the following algorithm that looks at the distances between neighboring points. The following pseudo-code demonstrates the details:

if (distance (Pk, Pk+1) > Th1 meters && Distance (Pk, Pk+2) < Th2 meters) { 
	mark Pk+1 invalid; 
} 

Th1 = Speed1*TimeDelta1*C Th2=Speed1*TimeDelta1*C

C is a tuning constant based on the characteristics of your application. While the device is moving, points are collected and the distances between two successive points are compared. If the distance travelled by the first point is larger than some threshold but the distance travelled by next point is within the threshold, the first point can be ignored.

An algorithm that is best in cases where your application is used in a stationary position is to look at the centroid computed by a series of points. With a computed centroid, outliers can easily be culled by comparing against a radius from the centroid.

Conclusion

We have shown a variety of possible filters to use in creating a sensor-driven mobile application. These filters may be what you need to add another level of refinement to your application and create a better user experience. Knowing the limitations and accuracy of the sensors you are using will help in understanding what performance you can expect in your application.

As more devices are available with sensors tightly integrated, developers will always be ready to create new features and come up with exciting possibilities for applications built around sensors. Access to lower level, more accurate sensor data is ever increasing and will allow developers to create even more exciting applications.

Additional information about the topics covered in this article:

– Ultrabook™ and Tablet Windows 8* Sensors Development Guide - http://software.intel.com/en-us/articles/ultrabook-and-tablet-windows-8-sensors-development-guide/

– Using Accelerometer in Windows* Style App and a Case Study of Tap Detection - http://software.intel.com/en-us/articles/using-accelerometer-in-windows-8-metro-style-app-and-a-case-study-of-tap-detection/

– MSDN - http://msdn.microsoft.com/en-us/library/windows/apps/

– J. Kerr, F. Raab, E. Ramirez, “Best Practices in Data Reduction and Analysis of GPS Data” - http://216.92.169.205/files/DataReduction_GPS_Workshop.pdf

– Greg Welch and Gary Bishop, “An Introduction to the Kalman Filter” - http://www.cs.unc.edu/~welch/media/pdf/kalman_intro.pdf

Acknowledgements

Special thanks to Miao Wei and Sushu Zhang for contributing content to this article.

About the Author

Nathan Totura is an application engineer in Intel's Software and Services Group. Currently working on the Intel® Atom™-enabling team he helps connect software developers with Intel technology and resources. Primarily these technologies include tablets and handsets on the Android*, Windows 8 and Tizen* platforms.

Notices

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations, and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license.

Intel, Ultrabook, and the Intel logo are trademarks of Intel Corporation in the US and/or other countries.

Copyright © 2012 Intel Corporation. All rights reserved.

*Other names and brands may be claimed as the property of others.

This sample source code is released under the Intel Sample Source Code License Agreement

如需更全面地了解编译器优化,请参阅优化注意事项