December 9, 2010

A Web of Sync

I’m excited about how the rise of HTML5 (and related technologies) will change the way we think about the web, and how they will further encourage developers to build richer applications instead of mere document-oriented sites.

(At Sencha, for example, we spend a lot of time thinking about what this really means, and how it changes the skills, tools and frameworks we’ll all need to be effective web practitioners)

But here, I’d like to consider three particular changes that are happening, and one particularly overlooked impact they might have.

The first is the rise of powerful application architectures on the client side. Sencha has added Model-View-Controller support in both Ext JS 4 and Sencha Touch, for example, and this makes it possible to implement entire data schemas, rich business logic, and well-structured user interfaces – entirely on the client side. (Other frameworks are following a similar path: SproutCore, JavaScriptMVC and so on)

Secondly, the way in which HTML5 provides support for local and offline storage makes the autonomy of these apps a real possibility. Viewing, filtering and manipulating data can still be performed in the browser, even when completely disconnected from the server (or the entire network, in the case of a mobile device that loses coverage).

Finally, there are the possibilities of being able to use Javascript on the server-side of the web. The rise of interest in node.js, for example, is largely prompted by the opportunity to be able to use the same programming language throughout the entire web stack.

But such changes mean that this stack is changing shape.

Historically, the entire logic of delivering web-based services to browsers was built on the server-side. Databases, security layers, business logic, user interface creation – all of this was within the realm of the web server, and the browser merely took the resulting HTML and dumbly rendered it for the user.

Of course, this is incredibly inefficient: the entire stack is executed (and a whole page of HTML generated) for almost every single user interaction. To mitigate this, web developers turned to AJAX & AHAH to start placing more of the user interface logic in the client’s runtime environment, and to reduce the size of the payloads involved. The stack started to span both the server and client sides of the picture.

But with client-side MVC and support for local storage techniques, we can take this stack evolution a whole step further. With Sencha Touch, for example, an application constructs its entire user interface from scratch, and can synthesize the entire business logic, validation ruleset, and storage schema that would have traditionally been encapsulated on the server side.

Now, islands of independent data on each and every user’s device might make for a useful service, but it is far more likely that most applications will continue to operate in conjunction with a server – which, with the appropriate layer of security to protect access to it, will provide a central storage repository.

Our stack has now been neatly split in two.

(Note that, in the same way that we’ve long learnt to run both client-side and server-side validation of data submitted from HTML forms, it will probably remain wise to mirror business logic on both sides of the network to prevent inconsistency. While the rise of Javascript on the server theoretically makes it possible to do this with one set of common code, in reality, most server stacks will remain non-JS based for some time.)

All well and good. But in one very particular respect, an architecture like this is deeply ambitious. We have added a significant new programming challenge into the mix: and that is one of data synchronization.

When the browser is merely displaying read-only HTML or AHAH, the definitive copy of the business data is one place only: the server. But in this new architecture, we have the same logical data in two places – and, assuming we’re allowing users to interact with their copy of it, quite possibly off-line, we have to deal with the matter of keeping the two (or N) data sets consistent.

At a protocol level, this is simple enough. We can easily use JSON to transmit data bidirectionally over an HTTP wire, and Ext JS and Sencha Touch’s model store classes, for example, take care of all the plumbing for us. But when we should do so, what the payload should be – and how we reconcile any conflicts between client and server – can quickly become interesting design questions.

Consider a web-based corporate price list application, build using a client-side MVC pattern, and providing up-to-date information about a company’s products to its mobile salesforce. On its first instantiation on a mobile device, say, such an application might easily pull the entire list of product records from the server over JSON and store them locally for fast, possibly off-line, access.

So far, so easy. But of course the records might frequently change on the server: the company creates new products, discontinues old ones, and often changes their prices. How should the ‘create’, ‘delete’ and ‘update’ operations be applied to the mobile device’s version of the data? Should it regularly refresh the whole data set again? Should it request all changes since the last sync and then incrementally apply those changes to its local copy? If certain data is not updated for a certain length of time, should the client application mark it as stale?

These are already thoughtful decisions for the application developer to have to make.

But now imagine that the company’s product managers are given more powerful rights within the same application, and they can actually edit the product descriptions on their mobile devices. When connected, these changes can be immediately sent directly back to the server. How can we ensure that other users’ apps can have that change broadcast to them?

What about when the product manager is off-line, perhaps editing the portfolio on a plane? Should the application prohibit updates to the local data? Preferably not, but if changes can be made, how should the application poll to see when it is reconnected, and send those changes back in the most efficient way?

Taking it to extreme, what happens when two product managers edit the same pieces of data? What happens when each have added a new product to which their client applications have assigned the same supposedly unique ID? The system needs to reconcile those potentially conflicting changes – and very quickly this becomes a non-trivial task.

In the broader computing world, these challenges are common and have been frequently, and successfully, addressed. Many proven solutions exist for synchronization in the realms of distributed file systems, versioned source control, disk, database and site mirroring and so on – and even to manage our MP3 collections. But what is new is that this will soon becoming a challenge that web developers will need to tackle, and it is a challenge that might be unfamiliar and daunting to many.

In the general case, of course, there is no single approach that will answer all these questions. Effective synchronization across a ‘split stack’ architecture will depend very much upon the sort of data that is being manipulated and what the application is setting out to achieve.

Frameworks will certainly provide us with all the tools we need to implement the choices we make. But considering these synchronization-related scenarios and use-cases will be an increasingly important part of bringing compelling applications to users in increasingly diverse contexts. Reconciliation will lie at the heart of how we bring complex applications to life – not just an after-thought.

The web is no longer one of documents or of simple uni-directional data: it is fast becoming a web of synchronization. And I fear it might not be as easy as we think. As developers, we all need to go into this new world with our eyes wide open and be prepared for new challenges around the corner.

Comments (20)

  1. December 9, 2010
    Branden Faulls said...

    localStorage and browser based SQL are quantum leaps forward for mobile application builds. But those of us at the coal-face of this new style of development are already yearning for in-built mechanisms for dealing with the sort of concurrency problems outlined here.

    The next step in this evolution is undoubtedly going to be the embedding of by-the-web/for-the-web managed NoSQL solutions like CouchDB which handle these sorts of data synchronisation issues by design.

    Great post and really useful illustration.

  2. December 9, 2010
    Ronan Cremin said...

    While I agree that app-like sites offer some great advantages for both user and developer, I wonder how quick and how extensive the transition from doc-like sites to app-like sites will be? We’ve had first-class examples of app-like sites for at least 6 years now (an eternity in web time) in the form of Gmail and Google Maps, and yet I can think of only a handful of really popular sites that are predominantly app-like today (new Twitter, Facebook) — the vast majority are still doc-like in their makeup.

    The advent of smarter frameworks such as Sencha, backbone.js and SpoutCore will undoubtedly help ease developers into this new world but I can’t help but feel that save for light AJAX touches here and there, doc-like sites will remain the dominant paradigm for the time being. There will be exceptions to this of course, specially in the mobile world where we’re finally seeing the demise of the iPhone monoculture, and app-makers will increasingly see HTML5 apps as their passport to cross-platform nirvana.

  3. December 10, 2010
    Stephen Stewart said...

    I agree with Ronan, doc like will be the predominant model because the step from doc to app in terms of web sites is fairly large and acts as a not insignificant barrier to web developers as a whole. I’ll also add that the advantages of web based apps seem to have been embraced by the investment banking world (amongst others) so I forsee a very clear financial (in terms of reward) and educational (in terms of training) divide between the two kinds of developer, not least because once you start developing web apps at that level you usually forgo frameworks (not always for the right reasons, I might add) and have to start from scratch. Professionally I think the road forks here, so to speak, and I would recast the final paragraph of the article in that light.

  4. December 10, 2010
    James said...

    Seems like I’ve been drinking more Minority Report kool-aid :-/

  5. June 17, 2011

    Its like you read my mind! You appear to know so much about this, like you wrote the book in it or something. I think that you can do with a few pics to drive the message home a bit, but other than that, this is magnificent blog. A fantastic read. I’ll certainly be back.

  6. June 17, 2011

    Really great website! :) Regards!

  7. June 21, 2011
    Jerold Hacker said...

    14. Hi, i feel that i saw you visited my weblog thus i came to ¡§go back the favor¡¨.I am trying to find issues to improve my site!I suppose its adequate to make use of a few of your ideas!!

  8. August 19, 2011
    Chris Aizak said...

    Chris Aizak Id just like to tell you how much I learnt from reading wisdom Bookmarked u.Hope to be back again for some more good articles…

    Id just like to tell you how much I learnt from reading wisdom Bookmarked u.Hope to be back again for some more good articles…

  9. September 1, 2011
    Robert said...

    Great insights in this post. Thanks for sharing it with us.

  10. September 1, 2011
    Playboy Mansion said...

    Your site is really a great help and it’s worth the money! I can highly recommend it!

  11. September 30, 2011
    Stephen Kaiser said...

    Thanks for the nice write-up, James. What about things like CouchDB to solve the data-synch/replication issues? It’s document-oriented and it seems like its real strength (replication) is what you were expressing the most concern about. And people are doing some amazing things with it – http://www.couchbase.com/case-studies/dimagi

  12. December 2, 2011
  13. December 9, 2011
    Minimize Pores said...

    Thanks man, real helpful insights.

  14. April 22, 2012
    Wedding Favors said...

    Almost a year and a half after this post and we still haven’t seen a surge of app like sites over doc based sites. Our site is up for a redesign and we are thinking of making it more app like. With so many new sites like pinterest and instagram coming up, it would seem like the natural progression for all sites including e-commerce based ones to make these changes.

  15. June 7, 2012
    Retha Mandler said...

    Incredible! This website appears to be exactly like my own one! It is actually on a entirely different area of interest nonetheless it has virtually the identical page layout and design. Superb choice of colors!

  16. June 14, 2012
    Government Bank said...

    Bangladesh bank, the central bank and monetary authority of the country. It came into existence under the Bangladesh Bank Order 1972 (Presidential Order No. 127 of 1972) which took effect on 16 December 1971 and operate the bank of Pakistan .the bank go of the Pakistan but Pakistan thing not development in Bangladesh .From introduction to end this order, the entire operation of the former State Bank of Pakistan in the eastern wing was transferred to Bangladesh Bank . Bangladesh
    http://bankinfodesk.com/page/2/

  17. June 15, 2012
    juwelsun HeLtH said...

    >>Hair Loss Easy Control Tips~
    Hair Loss Easy Control Tips : Every day a human being loses 50-100 hairs and it is normal that it is a regular process of hair renewal. When a man looses his hairs it becomes a painful blow for him
    http://medicalladvice.com/page/2/

  18. January 20, 2014

    I simply want to mention I’m new to blogging and site-building and really enjoyed your page. Likely I’m want to bookmark your blog post . You actually come with awesome stories. Thanks for sharing your webpage.

Leave a Reply