Data Synchronization: Which Technology?

Product Index
Search

Intel® Software Network

Language:
- English

	November 4, 2008 11:00 PM PST

Mark D. Sutton, Content Master Ltd

Introduction

This paper introduces data synchronization techniques for mobilized software applications. The aim is to help you identify, at an early design stage, which technology is appropriate for a given application, without going into the detail of how an implementation is coded. To achieve this aim, each technique is described here in turn, with its advantages and disadvantages.

At the conclusion, you must weigh the pros and cons of each technique to arrive at the most appropriate data synchronization technology to go forward and implement in your mobile solution. This paper is the first in a series, and forms an overview. Other papers in the series cover the implementation details that you will need, after you have decided which data synchronization technique to investigate further.

Definitions

Mobilized software: Mobilized software contains a set of features designed to insulate users from network dependencies by making data and applications available when the user is offline. Upon re-connection to the network, information is automatically updated through a data synchronization process. This application architecture allows users to work productively while enjoying a mobile work and life style.

802.11 LAN: Local Area Network based on the IEEE 802.11 standard for wireless networks.

Internet Information Services* (IIS): This is the name of the Windows* Web server.

Extensible Markup Language (XML): A standard text format for the transmission of data and its structure.

Web Service Security (WS-Security): A security system for Web Services.

Simple Object Access Protocol (SOAP): An XML protocol for object based network communication.

Distributed Component Object Model (DCOM): A specification for object based communication between components on a network.

Data Modification Language (DML): A subset of SQL (Structured Query Language) that allows data to be inserted, updated, and deleted.

Secure Sockets Layer (SSL): An encryption method for data transfer on the Internet.

Offline or on: Your App has to Work

One of the most basic requirements of any mobile application is the need for offline capability. You generally consider some form of local data cache, and you must assess the three facets of offline capability; data synchronization, security, and manageability. You need to consider the problems of deciding how often and how much data to cache on the mobile computer. Then, consider the way to synchronize changes back to the master data store on the server in a way that is both scalable and manageable. Finally, how the communication of data is secured and authenticated.

The following sections describe a number of techniques for data synchronization. Each includes a description of the technology, and an outline of its advantages and disadvantages. Aspects of performance, the proportion of time that a connection has to be available, and the complexity of im plementation are considered.

Direct Access to SQL Server*

If your application has access to the LAN, or to a VPN over the Internet for remote users, then you should consider implementing direct access to SQL Server*. An application that uses direct access to SQL Server requires a connection to be available to perform a database task, so this technique belongs to the environment of “mostly connected computing.” This technique does not offer a way of caching data locally for offline working, or for queuing updates while the client is disconnected so that updates can be applied when connectivity is restored.

Advantages

The advantages of direct access to SQL Server are: speed of implementation, security, performance, and the immediate update of data on the server.

To implement direct access to SQL Server in a .NET application, you use the objects in the System.Data.SqlClient namespace, and these exist in both the full and compact .NET Frameworks*. Therefore, similar code works for laptops and Tablet PCs running applications written using the full .NET Framework, and handheld devices running applications written using the .NET Compact Framework.

Visual Studio .NET* provides a SQL Server Explorer window that allows you to manipulate the database from within the development environment. Common database tasks such as creating a connection, retrieving data, and updating data are achieved easily by dragging data objects from the SQL Server Explorer window and dropping onto the application, resulting in faster development for data driven applications.

When your application updates data, it is updated directly on the server database, so it is immediately available to others. You do not need to program a data synchronization process for changes made locally into the server database. In addition, you can make use of database transactions within an update batch to ensure data integrity. These transactions run on the server without impacting the local device.

Performance can be good using direct access to SQL Server. No large local cache of data is required because the server data is kept accessible. Therefore, data retrieval can be focused; you need only download the data that is actually required for a specific application function. Thus, network traffic is kept to a minimum, and the local memory required for the data is kept at a minimum. You can also fully leverage the SQL Server query processing abilities: complex data calculations, loops and transactions are efficiently handled on the server.

Disadvantages

Paradoxically, both the advantages and the disadvantages of direct access to SQL Server stem from its close integration with SQL Server. Applications that rely on direct SQL access are unable to operate in an intermittently connected environment, and provide little value when network connections are unavailable.

Additionally, there is a negative side to the immediate update of data provided by direct access to SQL Server: you must plan for problems arising from multiple users trying to update the same data concurrently. Also, although there may be no overall data cache, your application will display data in controls on its user interface for t he user to read. So your implementation must deal with the consequences of other users updating this data on the server, while your application is still displaying the old values to the user.

For details of how to implement this technology see the white paper Direct Access to SQL Server Databases from Wireless Clients.

XML Web Services

XML Web Services use standard Web protocols to provide methods that can be invoked across the network, and which return data, or perform a specific function. XML Web Services are hosted by an IIS* server, and implemented as components using the full capabilities of the .NET Framework.

Advantages

The advantages of using XML Web Services are ease of use, inbuilt security, and its use of standard protocols which allow operation between computers running different operating systems communicating across the Internet.

XML Web Services use IIS anonymous or basic authentication options. You can also use SSL encryption to secure the transmission of passwords and data. In the near future you will be able to use WS-Security to encrypt and prevent interference with SOAP messages.

Note: WS-Security is currently only available for applications built with the full .NET Framework and not for applications built for handheld devices with the .NET Compact Framework.

Both the full .NET Framework and the .NET Compact Framework provide support for specifying the network credentials for the client application. The rest of the security setup is done quickly and easily using standard Windows administration tools and SQL Server tools.

Web service servers may be implemented on many different software platforms using software available from all the major software vendors. On Windows, Web service servers are implemented using the full power of the .NET Framework, and are able to send and retrieve data from a database such as SQL Server, and then to send that data to a Web service client for processing.

The client application part of a web services solution on Microsoft platforms can be written using either the full .NET Framework (on laptop or tablet PCs), or the .NET Compact Framework (handheld devices), as both support the System.Web.Services namespace. Visual Studio .NET provides tools to query a Web service server and automatically generate the code necessary to access the web service in the client application. Overall development speed for a Web service solution is high; both for writing the server, which can use the close integration of Visual Studio.NET with SQL Server, and for creating the client application.

Data retrieved from a Web service may be cached on the mobile client as XML, or may be stored in a client-side database such as SQL Server CE. The data can be modified in the application running on the client before being sent back to the server through another Web service method call.

Disadvantages

The disadvantages of a Web Service are that it offers poorer performance than many of the other methods described in this paper, and that it has no built-in support for merge conflict resolution. An XML Web Service is purely a mechanism for transferring data between a client and Web server, so there is no sophisticated built-in support for data synchronization, and you will have to code your own logic in the Web Service application on the server to handle merge conflicts (where two or more clients update the same data items simultaneously).

XML Web Services encode data using standards such as the SOAP protocol and XML. This use of accepted standards gives the benefit of interoperability between systems built using different software technologies, but it is somewhat 'verbose'. It does mean that the amount of network traffic required to transfer a piece of data is much more than if a proprietary binary protocol is used, with the consequent impact on performance.

.NET Remoting

.NET Remoting is the .NET technology that allows objects to be used remotely across the network. It is similar to DCOM, and you should also compare it with XML Web Services. However, unlike Web Services, the communication is not cross-platform as both the client and the server must be implemented in .NET.

Advantages

The main advantage of .NET Remoting over XML Web Services is performance. You can configure .NET Remoting to transfer data using a binary format, not XML and SOAP, which is the only option available to Web Services (although you can use XML and SOAP with .NET Remoting, if you prefer) . If both machines involved in the communication are running the .NET Framework and you only need to transfer a small amount of data, you should consider using .NET Remoting.

Disadvantages

The two main disadvantages of .NET Remoting are lack of support in the .NET Compact Framework, and (as for XML Web Services) no built-in support for data synchronization. As .NET Remoting is not supported in the .NET Compact Framework, you cannot use this technique for applications built for handheld devices such as Pocket PCs.

As for XML Web Services, .NET Remoting is purely a technique for synchronizing data between a backend server and a mobile client, so you will have to program all the logic to handle data extraction, data updates, and conflict resolution yourself. Nonetheless, this technique should be considered for the transfer of data between two .NET Framework machines.

SQL Server CE and Remote Data Access (RDA)

SQL Server CE is a separate product for handheld devices such as Pocket PCs that provides the opportunity for a fully functional local cache of data, and for extensive local data processing on a mobile device. SQL Server Remote Data Access* (RDA) is an Internet-based service for downloading data to a SQL Server CE database from a central SQL Server database. RDA is also capable of automatically tracking updates to downloaded data in the local SQL Server CE database, and sending updates to the server for merging back into the central server data. In addition you can send DML commands to the central SQL Server database using RDA.

Your mobile application will be implemented using the .NET Compact Framework that supports the System.Data.SqlServerCe namespace used to access the local SQL Server CE database.

Advantages

The advantages of RDA are its excellent support for intermittent network connections, the richness of functionality provided by SQL Server CE, and its simple setup. Using the SQL Server CE database means your application can run disconnected most of the time, and local database manipulation can be implemented quickly. The product provides setup wizards that make configuration of the server components relatively easy.

Disadvantages

The main disadvantage of RDA is in its update simplicity; it has no built-in conflict resolution. This means that if one client sends updates for a particular server data row, and then a second client updates the same data row, then the first changes are lost. So unless your application avoids such update conflicts, you may have to implement additional logic to avoid one client overwriting changes made by another.

If your target mobile device is a handheld with no extra storage card installed, then a possible further disadvantage would be the approximately 1.5 MB memory requirement for SQL Server CE and the memory requirement for its database. This can be a significant loading for a handheld device, which typically come equipped with 32MB or 64MB RAM. To minimize the database size, you should select carefully the server data to download to SQL Server CE.

SQL Server Merge Replication

Merge Replication is a technique that can be used to synchronize data between a central SQL Server database (the 'publisher') and a database on a client computer (the 'subscribers'). The mobile client application may be on a laptop or Tablet PC using a local SQL Server database such as MSDE* or SQL Server Personal Edition*, or it may be an application written using the .NET Compact Framework running on a handheld device, using the SQL Server CE database.

For an OCC application using the .NET Compact Framework, when the simplicity of SQL Server CE and RDA becomes a restriction, you should consider using SQL Server CE and Merge Replication.

SQL Server Merge Replication allows you to synchronize a central SQL Server publisher database with your application’s local subscriber database. The first time you synchronize your local database with the publisher, replication handles the initial download of the data. On subsequent synchronizations, replication merges locally made changes back to the publisher, and uploads changes made by any other subscriber clients. All clients maintain synchronized copies of the published data.

Advantages

The advantages of using Merge Replication are its excellent support for sporadic network connectivty, the richness of functionality provided by SQL Server CE, good security, and its full merging capabilities.

The security features handle client authentication and authorization through to the Publication database, and additional security can be provided by using SSL to encrypt data being transmitted.

As with RDA, once data has been downloaded to the SQL Server CE database, it can be modified disconnected, then updated data can be merged back to the central database. The .NET Compact Framework supports the System.Data.SqlServerCe namespace that you can use to modify the data locally, and the SqlCeReplic ation class that you can use to control merge replication. On a full PC client such as a laptop or tablet PC, you use the System.Data.SqlClient namespace in your application to manipulate the replicated copy of data in the local database, while the synchronization schedule is managed by tools such as SQL Server Enterprise Manager.

Disadvantages

The main disadvantage of SQL Server Merge Replication is that it is more complex to setup than other technologies mentioned. Also, as for RDA, if the target mobile device is a handheld with no extra storage card installed, then the memory requirements of SQL Server CE and its data could be a problem.

Conclusion

You must weigh the pros and cons of each technique presented in this paper to arrive at the most appropriate data synchronization technology to implement in your application. The following tables should help summarize the information previously presented:

Technology	Pros	Cons
Direct Access to SQL Server	Fast development (integrated data development environment) High Performance (better than .NET Remoting) Low network traffic (focused data retrieval) Immediate update Leverage full SQL Server functionality	Unable to operate with intermittent network connectivity Implementation must deal with update conflicts
.NET Remoting	Performance (better than XML Web Services)	Not supported by the .NET Compact Framework, so cannot be used with handheld clients Implementation must deal with update conflicts
XML Web Services	Cross platform availability May be secured using SSL and HTTP Authentication, or secured with WS-Security Good support for developer (both on client and web service)	Slow Performance (slower than .NET Remoting) No inbuilt support for synchronizing data Data transfers as XML so not as efficient as binary formats used by RDA or Merge Replication
SQL Server CE and RDA	Simple to setup Rich functionality provided by SQL Server CE	SQL Server CE takes up memory No merging functionality built-in (you will have to implement it)
SQL Server Merge Replication	Full merging functionality Rich functionality provided by SQL Server CE	SQL Server CE takes up memory More complex than RDA to setup

The following table should help you shortlist the most appropriate synchronization technology, from the requirements of your application:

Application	Consider Technology	Consider Rejecting
Suitable for intermittent network connectivity	XML Web Services .NET Remoting (when cross platform support is not required) SQL Server CE and RDA SQL Server Merge Replication	Direct Access to SQL Server (requires mostly connected)
High performance	Direct Access to SQL Server .NET Remoting	XML Web Services (too slow)
Minimum footprint on mobile device essential	Direct Access to SQL Server XML Web Services .NET Remoting	SQL Server CE and RDA (requires SQL Server CE) SQL Server Merge Replication (requires SQL Server CE)
Ease of Programming to track and synchronize updates	SQL Server Merge Replication SQL Server CE and RDA (although no update conflict resolution)	Direct Access to SQL Server (n/a) XML Web Services (no inbuilt programming support for synchronization) .NET Remoting (no inbuilt programming support for synchronization)
Cross-platform support required	XML Web Services	all others
Data shared by many clients with multiple updaters	Direct Access to SQL Server SQL Server Merge Replication	SQL Server CE and RDA (no merge support) XML Web Services and .NET Remoting – data updating code must be custom written