James Pearce » Programming http://tripleodeon.com Living the mobile dream Mon, 30 Aug 2010 06:42:30 +0000 en hourly 1 Mobile Analytics With Python, Django, ASP.NET, Java, and node.js http://tripleodeon.com/2010/08/mobile-analytics-with-python-django-asp-net-java-and-node-js/ http://tripleodeon.com/2010/08/mobile-analytics-with-python-django-asp-net-java-and-node-js/#comments Mon, 30 Aug 2010 06:42:30 +0000 James http://tripleodeon.com/?p=380 A few weeks ago, the PercentMobile team came to me to see if I could help write some new libraries for them. Contemporary web and mobile web sites are written on a vast array of different platforms… and obviously the more that PercentMobile supports, the better.

Something I love about programming is that there are so many languages to choose from – why restrict yourself to learning or becoming an expert at one when the same problems are also being solved in other, sometimes better, ways? Each language or platform has strengths and weaknesses of course. But I believe that understanding the differences – and more often, the similarities – makes one a better programmer.

So, generous polyglot that I am, I took the challenge, and plunged in.

(Sign up for the free PercentMobile service to get the libraries described in this post.)

Python & Django

I love Python – its readability, its libraries, and its overall philosophies. In a web server environment, Python can be used in a fairly raw way, behind a generic Web Service Gateway Interface (WSGI), but there are also a number of powerful web application frameworks using the language – most notably Django. PercentMobile want to provide painless Django support, but also to make sure that other Python server environments were not precluded.

percentmobile.py is the single file that provides everything you need.

If you are running code in a WSGI environment, you will have a handle to the WSGI environment. This is normally called environ by convention, and is passed in to your application via your top-level WSGI callable. To make the PercentMobile tracking code work, you need only make one call to the percentmobile.tracker_cookie_insert function. It returns two values: the cookie that you’ll need to set in the HTTP response headers, and the HTML that you should insert into your page.

Say, for example, you had a very simple WSGI app. This responds to requests with an HTTP 200 status code, a single header and some simple HTML:

def my_app(environ, start_response):
    status = '200 OK'
    response_headers = [('Content-type','text/html')]
    start_response(status, response_headers)
    return ["Hello world"]

To add PercentMobile tracking to this code, you need to firstly import the tracker library of course, and then call the tracker_cookie_insert function:

import percentmobile
cookie, insert = percentmobile.tracker_cookie_insert(environ, '1234555')

(Of course you should replace the final string with your own site ID!)

The cookie return value is actually a dictionary of the different parts needed to construct its string serialization. This will allow you to splice together multiple cookies, or alter the expiry time, path scope and so on. To convert the dictionary to a string, and to get the Set-Cookie header sent back in the request headers, the following code will suffice:

cookie = "%s=%s; expires=%s; path=%s" % (
    cookie['name'],
    cookie['value'],
    cookie['expires'],
    cookie['path']
)
response_headers = [('Content-type','text/html'), ('Set-Cookie', cookie)]

Finally, you need to make sure the HTML fragment, in the insert variable, gets placed in the HTML response:

return ["<html><body>Hello world %s</body></html>" % insert]

And that’s a wrap. The whole, PercentMobile-tracked WSGI application looks like this:

def my_app(environ, start_response):
    cookie, insert = percentmobile.tracker_cookie_insert(environ, '1234555')
    cookie = "%s=%s; expires=%s; path=%s" % (
        cookie['name'],
        cookie['value'],
        cookie['expires'],
        cookie['path']
    )
    status = '200 OK'
    response_headers = [('Content-type','text/html'), ('Set-Cookie', cookie)]
    start_response(status, response_headers)
    return ["<html><body>Hello world %s</body></html>" % insert]

Pretty easy, huh? Well, not as easy as tracking an app if you’re using the amazing Django framework! Contained in the same percentmobile.py file is a class that can be used as Django middleware. It uses the same underlying function as the WSGI implementation, but also takes care of the headers and insertion for you.

Assuming you’ve placed the library file in your Python or Django path, you simply need to add two lines of code in the settings file. Firstly add the percentmobile.PercentMobileDjangoMiddleware class to your list of middleware:

MIDDLEWARE_CLASSES = (
    ...
    'percentmobile.PercentMobileDjangoMiddleware'
)

And secondly, add your PercentMobile site ID to the settings file too:

PERCENTMOBILE_SITE_ID = '1234555'

Um… that’s it. The middleware will intercept the request, figure out the cookie that will need to be sent in the response, create the insertion code, and place it just before </body> in the response. You’re golden. I love Django.

ASP.NET: C# & VB.NET

Switching over to another world altogether, let’s take a quick look at the ASP.NET implementation of the tracking code. One of the great things about .NET is that you can choose between all sorts of different languages to write your applications and pages in. I decided to write the tracking library in C#, but you can use it declaratively in your page code, or programmatically from C#, VB.NET, or any other supported language.

The library is implemented as a User Control – that is, as a .ascx file. You need to download and add PercentMobile.ascx to your web application project.

To embed the tracking logic into a page (or probably preferably, a master page), you simply register the user control as residing in that file:

<%@ Register TagPrefix="pm" TagName="Tracker" Src="~/PercentMobile.ascx" %>

And then embed the control straight into the .aspx file contents, wherever you want it to be:

<pm:Tracker runat="server" SiteId="1234555" />

So a simple, tracked .aspx file might look something like this:

<%@ Page Language="C#" AutoEventWireup="true" %>
<%@ Register TagPrefix="pm" TagName="Tracker" Src="~/PercentMobile.ascx" %>
<!DOCTYPE html>
<html>
    <head><title>Hello World</title></head>
    <body>
        <form id="form1" runat="server">
            <div>Hello World</div>
            <pm:Tracker runat="server" SiteId="1234555" />
        </form>
    </body>
</html>

To be honest, this is so simple that I would expect most people to use the control declaratively. But if, for some reason, the control needs to be inserted programmatically, that’s pretty easy too. You need to use the Reference directive, instead of Register, at the top of the file, but otherwise it’s not much harder. Here, we’re using VB.NET to programmatically achieve exactly the same result as above:

<%@ Page Language="VB" AutoEventWireup="true" %><%@ Reference Control="~/PercentMobile.ascx" %>

<script runat="server">
    Private tracker As PercentMobile.Tracker
    Protected Sub Page_Load(ByVal sender As Object, ByVal e As EventArgs)
        tracker = CType(LoadControl("~/PercentMobile.ascx"), PercentMobile.Tracker)
        tracker.siteId = "1234555"
        form1.Controls.Add(tracker)
    End Sub
</script>
<!DOCTYPE html>
<html>
    <head><title>Hello World</title></head>
    <body>
        <form id="form1" runat="server">
            <div>Hello World</div>
        </form>
    </body>
</html>

(It should be fairly straightforward to see how to do this in other .NET languages – PercentMobile includes some examples in their install instructions.)

Java

Onwards. Let’s take a look at the tracking code for Java. There are numerous ways in which Java can be used for web or application server environments. To keep things simple, I decided that supporting JSP was more or less the most familiar and reusable approach.

JSP doesn’t enjoy the rich page event model that I was able to use in ASP.NET to intercept headers, write cookies, and insert HTML, all with one include. <jsp:include> is OK for adding the HTML snippets, but wouldn’t let me access the HTTP headers. <jsp:forward> does, but would only work if you weren’t planning to emit any HTML of your own after the tracking code – something of a radical assumption.

So I settled for an approach where an include directive creates an instance of an inner PercentMobile class defined within the JSP class. To cut a long story short, this means you merely add a reference to the library file at the top of the JSP file:

<%@ include file="percentmobile.jsp" %>

And then call the track method on that instance, somewhere within the page:

<%percentMobile.track("1234555");%>

The included JSP file takes care of instantiating the percentMobile object and giving it references to the request and response stream. This means it can read and write cookies, and later emit HTML. Your tracked Hello World in Java then? It’s as simple as this:

<%@ include file="percentmobile.jsp" %>
 <html>
  <body>
    Hello World
    <%percentMobile.track("1234555");%>
  </body>
</html>

node.js

OK, OK. Python, ASP.NET, Java. No big deal, right? That’s all so 2003 or so, right?

Well, there’s an alternative future for web server technologies. It’s one that’s blazingly fast, lightweight, event-driven, and… where you write your server logic in Javascript.

node.js is one of the most exciting things I’ve seen for a long time, and I know I’m not alone. Fresh, fashionable, and somewhat unproven, admittedly, it’s had a lot of gushing coverage. But since it turns the web server model (almost literally) inside out, I do believe there’s something important going on.

Would it be possible to write a web app with node.js and still have it tracked by PercentMobile? My challenge.

I decided to rely on Connect, a middleware framework for node.js, which, if you are writing web applications, provides a whole host of other helpful web logic. Using Connect to create a simple node.js app is extremely easy:

var Connect = require('connect');
Connect.createServer(
  function (req, res, next) {
    res.simpleBody(200,
      "<html><body>Hello World" +
      "</body></html>",
      {
        "Content-Type": "text/html"
      }
    );
  }
).listen(88);

To add PercentMobile tracking to this application firstly requires you to pull in the PercentMobile module:

var PercentMobile = require('./percentmobile');

(Where the percentmobile.js file has been placed in your node.js environment or in the common modules location.)

The module needs to be initialized as a piece of Connect middleware in the createServer function:

PercentMobile.init('1234555')

There are then two module functions, cookie and html, which both take a reference to the response object, and which return the cookie string and HTML to insert, respectively. These can be used in Connect’s simpleBody function, for example, meaning that our tracked application is as simple as this:

var Connect = require('connect');
var PercentMobile = require('./percentmobile');
Connect.createServer(
  PercentMobile.init('1234555'),
  function (req, res, next) {
    res.simpleBody(200,
      "<html><body>Hello World" +
        PercentMobile.html(res) +
      "</body></html>",
      {
        "Content-Type": "text/html",
        "Set-Cookie": PercentMobile.cookie(res)
      }
    );
  }
).listen(88);

That’s a wrap

So that’s it. A whistle-stop tour of the new languages and frameworks supported by PercentMobile. I’d love to hear your feedback on how easy (or hard!) these new APIs are to use.

And, oh… who will be the first to build a mobile web app on node.js? :-)

]]> http://tripleodeon.com/2010/08/mobile-analytics-with-python-django-asp-net-java-and-node-js/feed/ 0 WhitherApps http://tripleodeon.com/2010/08/whitherapps/ http://tripleodeon.com/2010/08/whitherapps/#comments Sun, 08 Aug 2010 03:10:20 +0000 James http://tripleodeon.com/?p=363 Why spend your life recompiling native client apps when you have HTML5 and the mighty mobile web at your disposal? WhitherApps sets out to debunk some contemporary mobile myths.

WhitherApps is a bandwagon-busting experiment. I believe there are far too many native client apps which could have been far better written as mobile web apps. What I’m trying to do with the project is take a few particularly obvious examples, apply a little reverse-engineering, and rewrite them, warts and all, with web technologies.

I’m focussing on native client apps that are free to download. These wouldn’t have suffered commercially had they been written this way in the first place. Obviously, I’ll also focus on feasible genres of apps. No 3D games or background apps, for example. To start with, informational media apps (such as newspapers and broadcasters) will be in the spotlight.

But otherwise, I may not need to pull too many punches. Offline access? Through the wonders of HTML5, that should be OK. Native device API access? Well, BONDI and proprietary libraries permitting, I’ll have a go – and there’s always PhoneGap too.

Each post on the site will be a walkthrough (or part of a walkhrough) of the process. As I get my hands dirty, from the protocol-sniffing to the pixel-jiggling, you can follow the steps I took.

Stay tuned and wish me luck!

]]> http://tripleodeon.com/2010/08/whitherapps/feed/ 0 The WordPress Mobile Pack is back! And here’s how. http://tripleodeon.com/2009/12/the-wordpress-mobile-pack-is-back-and-heres-how/ http://tripleodeon.com/2009/12/the-wordpress-mobile-pack-is-back-and-heres-how/#comments Fri, 18 Dec 2009 19:13:09 +0000 James http://tripleodeon.com/?p=348 Just a quick post to confirm that the WordPress Mobile Pack plugin is back online at wordpress.org.

Most WordPress plugins, if ‘GPL-compatible’, are hosted on wordpress.org – it provides an easy way for people to view and upgrade them. The increasingly popular Mobile Pack is no exception, and we’ve been happily listed there for over 6 months.

But what do you do when it suddenly disappears? About 3 days ago, that’s exactly what happened. I’m the administrator of the plugin and could see it when I logged in. But suddenly none of my collaborators – nor the general public! – could see or download the plugin.

Our sole channel for software distribution had mysteriously dropped into a black hole – without a word of notification from WordPress themselves.

  • Assuming it could only therefore be a bug with the wordpress.org web site, I filed a number of support tickets. No response.
  • Assuming it could only therefore be a permissions or licence issue, I raised a number of forum threads on the site (as, kindly, did Dennis Bournique, over at WAP Review). No response.
  • Assuming I was somehow falling between the cracks, I even emailed Matt Mullenweg. Less surprisingly, no response. UPDATE: I received an apology shortly after this post.

Hmm. All rather unsatisfactory.

Well, this morning, a breakthrough. My WPMP partner in crime, Andrea Trasatti thought of joining the WordPress IRC channel. (Obviously he’s a bit more old school than me). But finally found someone who would be prepared to answer the puzzle.

At this point, I need to point out there there are (true!) other mobile plugins out there. One particularly popular one has been Andy Moore‘s. He hadn’t hosted his on wordpress.org due, I guess, to his choice of license, and his – disclosed – use of automatic ads in the resulting pages, which WordPress does not permit.

For whatever reason, Andy recently closed down his plugin’s page, and had kindly redirected it to ours. This probably accounts for our nice increase in traffic in mid-November. Thanks Andy!

However, one of Andy’s previous plugin downloaders apparently complained about an error message they had received. The WordPress administrators investigated by visiting Andy’s site, and of course got redirected to our Mobile Pack page. Thinking our plugin was at fault, they then disabled our plugin for violating the no-ads rule.

Oh, and no-one thought to drop me an email to tell me.

But Andrea’s investigations and protestations prompted us to get re-approved, and we’re back!

(In the process of investigating this, it also turns out that when WordPress says ‘GPL-compatible’ and points you to a list of licenses, that doesn’t mean much at all. You also have to work out that they actually mean ‘GPL2-compatible’, and not ‘GPL3-compatible’, as our more permissive Apache license is. So I guess we’ll have to change our license too – which is a shame.)

I don’t think WordPress covered themselves with glory here. In the interests of constructive criticism, I would suggest the company should:

  • Inform a plugin owner whenever there is a complaint raised against it. I would have been able to point out the complaint was aimed at a different piece of software.
  • Inform a plugin owner when – or better, before – disabling a plugin from the listing. I could have had a grace period to have investigated any issues, rather than hear it first as complaints from users.
  • Respond to ‘site bug’ tickets raised on the site.
  • Make the licensing requirements for plugins more clear. The phrase ‘GPL-compatible’ here, (and its link), is certainly not precise enough.
  • Remember that plugin authors work hard to support your business of running the most popular blog platform in the world. Those individuals’ reputations, to a fair extent, rest on your responsible syndication of that code, and they get the grief when you pull the plug.

Anyway, thanks guys (and Mark in particular) for putting it back. And special thanks to Andrea, who now overtakes me in the chocolate-for-favours race.

Now… back to the roadmap of Mobile Pack coolness. Stay tuned.

]]> http://tripleodeon.com/2009/12/the-wordpress-mobile-pack-is-back-and-heres-how/feed/ 3 WordPress Mobile Pack v1.1 http://tripleodeon.com/2009/07/wordpress-mobile-pack-v11/ http://tripleodeon.com/2009/07/wordpress-mobile-pack-v11/#comments Wed, 15 Jul 2009 10:14:09 +0000 James http://tripleodeon.com/?p=322 The dotMobi WordPress Mobile Pack is a complete toolkit to help mobilize your WordPress site and blog.

We just launched v1.1, compatible with the latest version of WordPress. Download it, read the changelog, read about the features in full, or even join the team.

The plugin includes a mobile switcher to select themes based on the type of user that is visiting the site, a selection of mobile themes, extra widgets, device adaptation and a mobile administration panel to allow users to edit the site or write new posts when out and about.

Features include:

I have a lot of fun writing this plugin and there’s a bunch of cool ideas we have for the future. Check out the Assembla project to see the work as it happens – and maybe toss us some new ideas too.

It’s always a work in progress, but thousands of users seem happy enough so far. See what you think…

]]> http://tripleodeon.com/2009/07/wordpress-mobile-pack-v11/feed/ 4 Simple ‘on-server’ mobile AJAX in Django http://tripleodeon.com/2009/06/simple-on-server-mobile-ajax-in-django/ http://tripleodeon.com/2009/06/simple-on-server-mobile-ajax-in-django/#comments Wed, 17 Jun 2009 11:30:52 +0000 James http://tripleodeon.com/?p=318 –or–

“How your mobile site may not be much more complicated than building one with a data API or AJAX”

I am working on a project called ‘IvyRoot’ that is predominantly mobile and which is going to make me fabulously rich and famous.

OK, OK. It’s predominantly mobile, at least. It’s all in Django.

There’s a ‘desktop’ web site to go with it, and this contains panels within pages. Some of these panels might take a few moments to process and so I use XHR to call back to get them once the main page has loaded, displaying a pretty spinning icon in the meantime.

I also use XHR to populate in-page tables which might have many rows – so that pagination and sorting are fast without having to load the whole page again.

I also want to expose a number of data APIs (providing XML, JSON, basic HTML and the like) so that people can build 3rd party clients, their own consuming sites, and widgets.

Finally, I want to provide a mobile web version of the site. I’ll probably do some sort of device detection to refine the behaviour, but I should assume that plenty of browsers will not be able to do the XHR bit, and should have the page (containing all the panels) in one go – admittedly after a slightly longer wait.

Well, it occurs to me that these requirements can all be solved at the same time. Switching templates based on URL (or other HttpRequest clues) is pretty easy, so one thing I could do is have a filename extension convention for all my URLs. Imagine a ‘things’ view that lists things.

  • http://ivyroot.com/things – this is the human-consumable page: the view surrounded by menus, logos, graphics and the like. Knows how to switch outer template, based on various things, between desktop and mobile.
  • http://ivyroot.com/things.html – this is the table at the heart of the page, perhaps using unstyled markup containing the list of things itself.
  • http://ivyroot.com/things.xml, http://ivyroot.com/things.json etc – data APIs that list things in various serializations.

So perhaps we can set up URL routes like this:

...
(r'^things$', things),
(r'^things.(?P<mode>[a-z0-9]+)$', things),
...

And have a view that looks something like this:

def things(request, mode=''):
    if not mode:
        mode = desktop_or_mobile()
    template = "things.%s" % mode
    ...

(Or, I suppose we could pass the mode into the template via the context and have some sort of a switch statement in the template itself. It might be good to pass it in, anyway, in case there is any further refinement required within each template.)

Anyway, let’s also assume that the things list is very long and takes a while to generate. For the desktop users, we might want to either load the table as HTML asynchronously using XHR, or paged into a jQuery grid control. For these two approaches, I can call back to /things.html or /things.json respectively (and do a little extra plumbing to make sure the JSON is in the right format for the grid control to consume).

But what of mobile? Well, for many devices, I can’t expect any XHR activity to be supported, so we need to insert the table HTML into the body of the things.mobile page before it departs for the client – and we may need to pre-page it too (if it’s long). But the good news is that, if I made it simple enough, the HTML generated by things.html might be good enough for either type of browser.

(And if not, I could easily do a little adaptation on that view too – perhaps turn my table into a list.)

So what I would like to have a is a way of embedding the output of one view (or even URL) into another, whilst the response is still being put together on the server. Try as I could, there didn’t seem to be anything in Django to do this.

Enter my first DjangoSnippet!

This basically allows me to put a template tag into the mobile template that pulls the output from one of the other views (identified by view name, with arguments; or a verbatim URL which would be the same as the one used in the desktop’s XHR request – nifty!)

It also makes sure that the context for that view is sandboxed with what it needs to render itself – so that I don’t have to worry about getting the context ready in the ‘outer’ view. That’s not necessary in our simple example above – but it might be useful later on if I am dynamically choosing which panels to place on a page, and I want them to gather together their own context prior to template rendering.

Now, I’ll need to be careful to make sure that any interaction elements within the table – say, some actions available on each item in the list, or a paginator – behave suitably for their location. But this may be as simple as toggling between further XHR requests (for desktop) and simple anchor links (for mobile). Forms-in-panels will be harder but similar.

So… there we are, a glimpse at the inner workings of a mystery application that’s many weeks away from seeing the light of day.

But hopefully I’ve raised a few thoughts about how you can treat the mobile version of your site as being a cousin of the data APIs that you might have written anyway. And also how you might be able to sneakily bring some of the funky XHR – that you wanted to impress your friends with on the desktop – up into the server to keep your mobile pals happy too.

]]> http://tripleodeon.com/2009/06/simple-on-server-mobile-ajax-in-django/feed/ 6