CodeLab
23Feb/11Off

Simple HTTP streaming with Twisted & Javascript

When writing web based apps a recurring problem is updating page contents in response to server side events. A classical pattern to solve it is to do periodic pooling of the server using XMLHttpRequests, however this often leads to making far more HTTP requests than what is really needed. It would be best to be able to originate the update from the server. Emerging technologies such as HTML5 websockets may provide a neat solution in the future, but until they become widely supported by all major browsers your path will be filled of all sort of "workarounds" to emulate the websockets on all those still catching-up. However a simple solution may be at hand in some particular cases: enter the world of html streaming.

The pattern is relatively simple:

  1. Upon a request an html page is sent to the browser,
  2. but instead of closing the connection, you keep it open.
  3. Then you can append script tags that are evaluated as soon as they arrive to the browser, updating the page content.

To implement it, we will look at a very basic HTML page with a small JavaScript code that implements updating a list (appending li elements). Then, a simple Twisted-based web server takes care of handling the requests and pushing data in our example.

Let's start with the HTML, save the following as index.html:

<html>
<head>
</head>
<body>
	<ul id="log">
	</ul>	
</body>
</html>

Now let's add the JavaScript that will update our UI:

<script type="text/javascript">
var webPush = function(args){
  var wp = {
	init      : function(){
		  this.ul =  window.document.getElementById(args.id);
		  this.lastScript = null
	},
	addLi		   : function(content){
		  var newli       = document.createElement('li');
		  newli.innerHTML = content;
		  this.ul.appendChild(newli);
		  this.ul.scrollTop = this.ul.scrollHeight;
                  this.removeScript()
	},
	scriptsCount  : function(){
		  scriptsL  = window.document.getElementsByTagName('SCRIPT');
		  this.lastScript = scriptsL[ scriptsL.length - 1 ]
	  	  return scriptsL.length
	},
	removeScript  : function(){
                  this.scriptsCount();
		  if (this.lastScript)
			this.lastScript.parentNode.removeChild(this.lastScript)
	}
  };

  wp.init();
  return wp
};

var pushHandler;
</script>

Here we defined a constructor function, webPush that takes the argument "id" and returns an object. We place this script in the head of our html. Later, in the body, just after adding the UL that we are going to use to display the incoming data, we create an instance of the object, pointing to our target UL id: log (lines 10-15 ):

<html>
<head>
<script type="text/javascript">
...
</script>
</head>
<body>
	<ul id="log">
	</ul>	
	<script type="text/javascript">
		/* INIT */
		pushHandler = webPush({id: 'log'});
		pushHandler.addLi('Log started')
		/* This script will remove itself */
	</script>
</body>
</html>

At the end of the method addLi, a call to removeScript prevents unnecessary accumulation of the script snippets after the new list items are appended.

We are now set to start. All that is remaining is to get the server to send scripts snippets making calls on our object:

<script type="text/javascript">
   pushHandler.addLi('hello world')
</script>

Let's look at a basic web server that completes our pattern:

#!/usr/bin/env python
from twisted.protocols import basic
from twisted.internet  import reactor, protocol, defer
import datetime

class WebPUSH(basic.LineReceiver):
    logTemplate = '''
      <script type="text/javascript">
         pushHandler.addLi('%s')
      </script>
    '''
    def __init__(self):
        self.gotRequest = False

    def lineReceived(self, line):
        if not self.gotRequest:
            self.startResponse()
            self.gotRequest = True
            
    def startResponse(self):
        self.sendLine('HTTP/1.1 200 OK')
        self.sendLine('Content-Type: text/html; charset=utf-8')
        self.sendLine('')
        with open('index.html', 'r') as f:
            self.transport.write( ''.join(f.read()) )
        
        self.logTime()

    def logTime(self):
        self.sendLine( self.logTemplate % datetime.datetime.now() )
        reactor.callLater(2, self.logTime)

if __name__ == '__main__':
    f = protocol.ServerFactory()
    f.protocol = WebPUSH
    reactor.listenTCP(8080, f)
    reactor.run()

The server relies on a basic.LineReceived-based protocol: WebPUSH. Upon receiving the first line of the request, we send the html to the browser (lines 15-26). Then a call to logTime sends the first data stream and registers a timer to repeat the call in 2 seconds (lines 29-31).

That's it! Save the server. Run it, point your browser to http://localhost:8080 and watch the timestamps appear every 2 seconds ;-)

Comments (3) Trackbacks (0)
  1. thanks for the code! But i have one problem, the browser doesn’t display anything until i shutdown the twisted server (ctrl+c).
    Any hints how i can solve this?

  2. Thanks for the code!
    But i have a problem, the browser still loads and loads the page but display’s nothing. Even if i cancel the server, then the browser display all results.

    Any hints whats going wrong?

    • The browser will seem to be loading the page forever since we are on purporse keeping the connection open so we can “stream” chunks of javascript. If you can not see anything in the browser try using firebug and look for javascript errors. Also pay attention to what messages are you trying to send (ie. any special characters, etc that may break the js?)

Trackbacks are disabled.