Thursday, June 28, 2007

Tiny ASP.NET Ajax Framework Contest

 

Contest context

More than two years ago we've built a small web application used to collect and print documents required to participate in secondary school final examinations in Poland. The application is used quite extensively by people from all over the country.

The application, maturzysta.vulcan.pl, is located here.

From the web developer's point of view there is one interesting aspect of the main form of the application: there are about 30 dropdown lists with complicated logic between their selections: when you change the selection in one list, few others should be cleared or repopulated with context-bound data. There is also quite a lot of validation of various dependencies between user's selections.

The first version of the application was built as a server-side application: all SelectedIndexChanged events were auto-postbacked and processed on the application server. This caused a lot of trouble since the heavy form had to be sent to and from the server.

The year later we've completely redesigned the application, moving the interface logic to the javascript. The web traffic problem has been solved but two other appear. First of all - no one likes javascript, it's clumsy and error-prone. Then - we had to disable ASP.NET page validation mechanism because otherwise the page content modified on the client-side would be rejected on the server-side as potentially dangerous.

In the meantime, few promising AJAX frameworks appeared and we thought that it would be possible to go back to the initial version of the application but instead of expensive postbacks we could take the AJAX approach and replace postbacks with callbacks. This could solve all problems: reduce the traffic and still keep the application logic on the server-side.

Contest participants

Three AJAX frameworks were chosen for the contest:

The first and the third one were chosen because it's fairly straightforward to extend existing application with AJAX capabilities. The second one is perhaps one of the most promising AJAX frameworks out there, designed to fill the gap between Windows.Forms and Web.Forms programming.

In order to make the application compatible with participants, following steps have been taken:

  • in case of ASP.NET AJAX, the ScriptManager has been put on the form page and the whole page content has been put into the UpdatePanel. The concept of several small UpdatePanels with triggers between them has been rejected because of the complexity of the application logic and dependencies between controls.
  • in case of the Ajaxium framework, the Ajaxium initialization routine has been put into the OnBeginRequest event of the global application class as well as into the Page_Load method of the form.
  • in case of the VisualWebGUI framework, the application has been completely rewritten using the VWG library and components.

Compare the layout of the WebForms/AJAX application (save images to see them full size)

with the layout of the rewritten VWG version

 The judge

Microsoft Application Center Test has been choosen as the contest judgement tool. Tests were performed on the Windows XP machine serving as the application server. Exactly 200 sessions have been simulated for 5 concurrent users. Each session has been recorded as the complete user session - forms have been filled with data and dropdown selections have been made.

Each session ended with the application showing the print-ready document.

The results

The results are available here.

Observations:

  • both the Microsoft AJAX.NET and the Ajaxium frameworks are not capable of delivering dynamically created content to the web browser from within the AJAX callback [or at least we do not know how to do that]. This means that when the application produces, let say, XML data file and sends it to the application, instead of a typical "Save - Open - Cancel" prompt, you get ... nothing. Fortunately, this was not considered as a test case.
  • the original Web.Forms server-side application, numer (1) in the result chart, has been able to process 213 requests per second and ended with almost 8 MB of data sent to the server and 80 MB of data received from the server. These results has been considered as reference results.
  • Microsoft ASP.NET AJAX application, numer (2) in the result chart, has been able to process 222 requests per second with 5 MB of data sent to the server and 129 MB (!) of data received from the server.
  • The Ajaxium version, number (3) in the result chart, has been able to process only 111 request per second. It sent 5 MB and received 47 MB of data.
  • The VWG version, numer (4), did 422 requests per second with 2 MB sent and 23 MB received (!)
  • The javascript version, numer (5), did 400 requests per second, with 2 MB sent to the server and 37 MB received.

Results:

  • despite its simplicity, Ajaxium did unexpectably bad. We belive that the framework initialization takes just too much time (look at the "time to first byte") and this downgrades the performance.
  • Microsoft ASP.NET AJAX didn't really help regarding server-side resource consumption. It also sends large amount of data to the client - in our case this could be because large form has been put into single UpdatePanel. In overall it did a nice job of ajaxing the application but nothing else.
  • The javascript version was considered as an easy winner and, to our suprise, it sends quite a lot of data to the client. Well then, scripts are over 90 kB. Multiply it by 200 clients and you have the bandwitch consumption.
  • The VWG is an undoubt winner. It did a great job of serving the highest number of requests per second, 422, and is also a clear winner regarding the amount of data sent to and from the server. The only problem with VWG is that it is still in beta version and is not reliable for the time (it still has stability and compatibility problems, let's hope it will get better every other release).

What could be interesting is that the VWG version seems to be the slowest one from the end-user's point of view. It just feels sluggish inside the web browser. We belive that this is caused by the constant callbacks to the server and as you can see it does not slow down the processing pipeline at all - it is just slow inside your web browser.

I would like to thank my friend Krzysiek Owczarek for helping me to perform the tests.

 

Update 2007-07-15

Mark Krapovek, Ajaxium Support Manager, made a comment on the testing environment:

"The evaluation edition available freely on vendor's website DOES NOT SUPPORT CONCURRENT CONNECTIONS. That's just a limitation of the evaluation edition. Before testing the component for "200 sessions simulated for 5 concurrent users" you had to ask vendor for a version with all limitations removed.
So: if Ajaxium Evaluation Edition was able to serve more than 100 requests per second, being limited for only 1 connection at a time, it is the fastest component among all compared."

Mark, thank you for the information.

8 comments:

Nelson said...

Hi
Nice information test, i am working with VWG too and I would like to know how did you do this kind of tests??

Thanks a lot

Wiktor Zychla said...

Test were performed using Microsoft Application Center Test, the tool shipped with Visual Studio .NET 2003 Enterprise Architect.

I belive that other smiliar tools exist but never tried anything other than MACT.

Regards,
Wiktor

Unknown said...

You were unable to test Ajaxium in the environment you described.

The evaluation edition available freely on vendor's website DOES NOT SUPPORT CONCURRENT CONNECTIONS. That's just a limitation of the evaluation edition. Before testing the component for "200 sessions simulated for 5 concurrent users" you had to ask vendor for a version with all limitations removed.

So: if Ajaxium Evaluation Edition was able to serve more than 100 requests per second, being limited for only 1 connection at a time, it is the fastest component among all compared.

Regards,
Mark Krapovek,
Ajaxium Support Manager

Wiktor Zychla said...

mark,

thank you for the very important information. I will update the article to include your remark so that readers could be aware of it.

regards,
Wiktor

Рома said...

It's strange that javascript version downloaded more than VWG. You say that 90k * 200 will be 37 MBs (it's not really so) but you also say that you have only 5 concurrent users. So looks like web browser had to cache scripts after the first request and reuse them. And if the javascript version uses jason it should send/recieve only actual data - in this case VWG result looks a bit like a magic.

Рома said...

Hey! Have you actually seen your test results? Only 22% of VWG requests were successful. That's why "The VWG is an undoubt winner". It did nothing in near 80% cases - just did a request and got error response.

Wiktor Zychla said...

that's an interesting note and I cannot explain it. during the recording phase of the test the application performed correctly under vwg.

however I admit that I did not analyzed how many request ended with 400 status code during recording of the test.

I think then that there are two possibilities:

- the test results are correct and the 400 is somehow valid or at least non-cricital to vwg
- the test resulsts are invalid because Microsoft Application Center Test could not correctly handle concurrent requests to vwg application.

Unfortunately, I am not able to reproduce any of the tests and frankly, I somehow lost the interest in vwg. I respect the general idea very seriously, however the amount of technical problems hidden inside the framework is still too significant to use it in mission critical applications.

ajax tab container said...

It seems so easy to learn and apply. It sure is a wonderful thing to learn different kinds of application in order to make the best website.