gregw

Jetty 6.0 Continuations - AJAX Ready!

The 6.0.0alpha3 release of Jetty is now available and provides a 2.4 servlet server in 400k jar, with only 140k of dependencies (2.6M more if you want JSP!!!). But as well as being small, fast, clean and sexy, Jetty 6 supports a new feature called Continuations that will allow scalable AJAX applications to be built, with threadless waiting for asynchronous events.

Thread per connection

One of the main challenges in building a scalable servlet server is how to handle Threads and Connections. The traditional IO model of java associates a thread with every TCP/IP connection. If you have a few very active threads, this model can scale to a very high number of requests per second.

However, the traffic profile typical of many web applications is many persistent HTTP connections that are mostly idle while users read pages or search for the next link to click. With such profiles, the thread-per-connection model can have problems scaling to the thousands of threads required to support thousands of users on large scale deployments.

Thread per request

The NIO libraries can help, as it allows asynchronous IO to be used and threads can be allocated to connections only when requests are being processed. When the connection is idle between requests, then the thread can be returned to a thread pool and the connection can be added to an NIO select set to detect new requests. This thread-per-request model allows much greater scaling of connections (users) at the expense of a reduced maximum requests per second for the server as a whole (in Jetty 6 this expense has been significantly reduced).

AJAX polling problem

But there is a new problem. The advent of AJAX as a web application model is significantly changing the traffic profile seen on the server side. Because AJAX servers cannot deliver asynchronous events to the client, the AJAX client must poll for events on the server. To avoid a busy polling loop, AJAX servers will often hold onto a poll request until either there is an event or a timeout occurs. Thus an idle AJAX application will have an outstanding request waiting on the server which can be used to send a response to the client the instant an asynchronous event occurs. This is a great technique, but it breaks the thread-per-request model, because now every client will have a request outstanding in the server. Thus the server again needs to have one or more threads for every client and again there are problems scaling to thousands of simultaneous users.

Jetty 6 Continuations

The solution is Continuations, a new feature introduced in Jetty 6. A java Filter or Servlet that is handling an AJAX request, may now request a Continuation object that can be used to effectively suspend the request and free the current thread. The request is resumed after a timeout or immediately if the resume method is called on the Continuation object. In the Jetty 6 chat room demo, the following code handles the AJAX poll for events:

    private void doGetEvents(HttpServletRequest request, AjaxResponse response)
    {
        member = (Member)chatroom.get(request.getSession(true).getId());

        // Get an existing Continuation or create a new one if there are no events.
        boolean create=!member.hasEvents();
        Continuation continuation=ContinuationSupport.getContinuation(request,create);
        
        if (continuation!=null)
        {
            if(continuation.isNew())
	        // register it with the chatroom to receive async events.
                member.setContinuation(continuation);
        
            // Get the continuation object. The request may be suspended here.
            Object event= continuation.getEvent(timeoutMS);
        }
        
        // send any events that have arrived
        member.sendEvents(response);
        
        // Signal for a new poll
        response.objectResponse("poll", "");
    }

When another user says something in the chat room, the event is delivered to each member by another thread calling the method:

  class Member
  {
       public synchronized void addEvent(Event event)
       {
         _events.add(event);
         if (_continuation!=null)
           // resume requests suspened in getEvents
           _continuation.resume(event); 
       }
       ...
  }

How it works

Behind the scenes, Jetty has to be a bit sneaky to work around Java and the Servlet specification as there is no mechanism in Java to suspend a thread and then resume it later. The first time the request handler calls continuation.getEvent(timeoutMS) a RetryReqeuest runtime exception is thrown. This exception propogates out of all the request handling code and is caught by Jetty and handled specially. Instead of producing an error response, Jetty places the request on a timeout queue and returns the thread to the thread pool.

When the timeout expires, or if another thread calls continuation.resume(event) then the request is retried. This time, when continuation.getEvent(timeoutMS) is called, either the event is returned or null is returned to indicate a timeout. The request handler then produces a response as it normally would.

Thus this mechanism uses the stateless nature of HTTP request handling to simulate a suspend and resume. The runtime exception allows the thread to legally exit the request handler and any upstream filters/servlets plus any associated security context. The retry of the request, re-enters the filter/servlet chain and any security context and continues normal handling at the point of continuation.

Furthermore, the API of Continuations is portable. If it is run on a non-Jetty6 server it will simply use wait/notify to block the request in getEvent. If Continuations prove to work as well as I hope, I plan to propose them as part of the 3.0 Servlet JSR.

Posted at 09:54AM Sep 17, 2005 by gregw in General  |  Comments[2]