Wednesday, July 1, 2009

ASP.NET WebServices two-way (Response and Request) compression - a general solution

Objective

I will present a general solution to the two-way compression of WebServices" issue. The issue can be critical in a complicated SOA solution where a lot of data is passed not only from the server to the client but also from the client to the server. The solution is a part of a Data Service component I currently work on.

Motivation

On one hand, the application server can compress the HTTP data sent to the client "out-of-the-box", just enable "HTTP Compression". While such solution seems attractive, it does not provide two-way compression. It's nice to have a server's response compressed, however uploading huge amount of data is still a problem.

On the other hand, a generic solution already exists and consists in writing a custom SOAP extension. The solution by Saurabh Nandu dates 2002 is described here. There are however two issues with that approach. A small issue is that you have to apply a custom attribute to both server and client code. While applying a custom attribute to a server method is not an issue, the client proxy class is regenerated each time you update the reference which means that you have to remember to manually edit the proxy class after you update the reference. Unfortunately, there's also a big issue. We've observed that the solution causes random OutOfMemory exceptions in a production server environment! The randomness was a complete disaster for our proprietary software!

Towards the solution

Let's start with a basic code we'll enhance during this tutorial. Open Visual Studio, create a new solution with two projects: a console applications and an ASP.NET Web Service application.

Implement a method on the WebService.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.Services;
 
namespace WebService1
{
    [WebService( Namespace = "http://tempuri.org/" )]
    [WebServiceBinding( ConformsTo = WsiProfiles.BasicProfile1_1 )]
    [System.ComponentModel.ToolboxItem( false )]
    public class Service1 : System.Web.Services.WebService
    {
 
        [WebMethod]
        public string HelloWorld( string Request )
        {
            return Request;
        }
    }
}

Note that the method has an input parameter and just returns it's value to the client. We are going to pass really long strings here and we'll gonna see how much data is actually passed to and from the server.


In a console application, add a web service reference and following code:



using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
namespace ConsoleApplication60
{
    class Program
    {
        static void Main( string[] args )
        {
            MyService.Service1 s = new ConsoleApplication60.MyService.Service1();
            s.Url = "http://localhost.:3112/Service1.asmx";
 
            StringBuilder sb = new StringBuilder();
 
            for ( int i=0; i < 10000; i++ )
                sb.Append( "qwertyuiopasdfghjklzxcvbnmqwertyuiopasdfghjklzxcvbnm" );
 
            Console.WriteLine( s.HelloWorld( sb.ToString() ) );
 
            Console.ReadLine();
        }
    }
}

Note that I manually set the Url of the WebService and what I actually did was to add a dot after the "localhost". This is to be able to use a HTTP sniffer, Fiddler.


Run the application and inspect the HTTP session in Fiddler. Please take a look at "Content-Length" headers for both the request and the response, the content length of the request is 520318 bytes, and the content length of the response is 520352 bytes.



One way compression (compression of the response)

The OutOfMemory exceptions caused by the solution based of a custom SOAP Extensions made us to search for another approach and lead us to a common and well known "HttpCompressionModule on a server - HttpWebResponseDecompressed on a client proxy". This solution solves only the half of the compression issue - the data sent from the server to the client is compressed.


Start with a HttpCompressionModule: add a HttpCompressionModule.cs file into the WebService project:



using System;
using System.IO.Compression;
using System.Web;
using System.Web.Security;
 
public class HttpCompressionModule : IHttpModule
{
    private bool _isDisposed = false;
 
    public void Init( HttpApplication context )
    {
        context.BeginRequest += new EventHandler( context_BeginRequest );
    }
 
    void context_BeginRequest( object sender, EventArgs e )
    {
        HttpApplication app = sender as HttpApplication;
        HttpContext ctx = app.Context;
 
        if ( !ctx.Request.Url.PathAndQuery.ToLower().Contains( ".asmx" ) )
            return;
 
        if ( IsEncodingAccepted( "gzip" ) )
        {
            app.Response.Filter = new GZipStream( app.Response.Filter,
      CompressionMode.Compress );
            SetEncoding( "gzip" );
        }
        else if ( IsEncodingAccepted( "deflate" ) )
        {
            app.Response.Filter = new DeflateStream( app.Response.Filter,
      CompressionMode.Compress );
            SetEncoding( "deflate" );
        }
    }
    private bool IsEncodingAccepted( string encoding )
    {
        return HttpContext.Current.Request.Headers["Accept-encoding"] != null &&
          HttpContext.Current.Request.Headers["Accept-encoding"].Contains( encoding );
    }
    private void SetEncoding( string encoding )
    {
        HttpContext.Current.Response.AppendHeader( "Content-encoding", encoding );
    }
    private void Dispose( bool dispose )
    {
        _isDisposed = dispose;
    }
    ~HttpCompressionModule()
    {
        Dispose( false );
    }
    public void Dispose()
    {
        Dispose( true );
    }
}

Now, make the module active by adding



<httpModules>
    ...
    <add name="HttpCompressionModule" type="HttpCompressionModule" />
</httpModules>

to your web.Config.


Now go back to the client console application and add two methods to the partial proxy class. Assuming that the proxy's class name is MyService, add MyService.cs:



public partial class Service1 
 {
     #region WebResponseCompress
 
     protected override WebRequest GetWebRequest( Uri uri )
     {
         HttpWebRequest request = (HttpWebRequest)base.GetWebRequest( uri );
         request.Headers.Add( "Accept-Encoding", "gzip, deflate" );
     
         return request;
     }
 
     protected override WebResponse GetWebResponse( WebRequest request )
     {
         return new HttpWebResponseDecompressed( request );
     }
 
     #endregion
 }

Note that what this does is to make sure that the correct header is sent to the server and that the response is actually decompressed on the client side. The HttpWebResponseDecompressed is a common class but it's not in the Base Class Library so here it is:



public class HttpWebResponseDecompressed : System.Net.WebResponse
{
    private HttpWebResponse response;
 
    public HttpWebResponseDecompressed( WebRequest request )
    {
        try
        {
            response = (HttpWebResponse)request.GetResponse();
        }
        catch ( WebException ex )
        {
            response = (HttpWebResponse)ex.Response;
        }
    }
    public override void Close()
    {
        response.Close();
    }
    public override Stream GetResponseStream()
    {
        if ( response.ContentEncoding == "gzip" )
        {
            return new GZipStream( response.GetResponseStream(), 
                CompressionMode.Decompress );
        }
        else if ( response.ContentEncoding == "deflate" )
        {
            return new DeflateStream( response.GetResponseStream(), 
                CompressionMode.Decompress );
        }
        else
        {
            if ( response.StatusCode == 
                    HttpStatusCode.InternalServerError )
                return new GZipStream( response.GetResponseStream(), 
                    CompressionMode.Decompress );
 
            return response.GetResponseStream();
        }
    }
    public override long ContentLength
    {
        get { return response.ContentLength; }
    }
    public override string ContentType
    {
        get { return response.ContentType; }
    }
    public override System.Net.WebHeaderCollection Headers
    {
        get { return response.Headers; }
    }
    public override System.Uri ResponseUri
    {
        get { return response.ResponseUri; }
    }
} 

Now run the solution and inspect it with Fiddler:



Note that the content length of the server's response is not only 5771 bytes! (Note also that the 100:1 compression ratio is rather unusual and is caused by repetitive data I send to the server from the console application!)


Two-way solution (compression of requests and responses)

We are now ready to enhance the partial solution and enable a two-way compression (and this is my contribution to the issue).


First, go to the HttpCompressionModule and add a line:



...
 
void context_BeginRequest( object sender, EventArgs e )
{
    HttpApplication app = sender as HttpApplication;
    HttpContext ctx = app.Context;
 
    if ( !ctx.Request.Url.PathAndQuery.ToLower().Contains( ".asmx" ) )
        return;
 
    if ( IsEncodingAccepted( "gzip" ) )
    {
        /* INSERTED LINE HERE! 
           We add a filter to decompress incoming requests.
         */
 
        app.Request.Filter  = 
            new System.IO.Compression.GZipStream( 
                app.Request.Filter, CompressionMode.Decompress );
 
        app.Response.Filter = new GZipStream( app.Response.Filter,
  CompressionMode.Compress );
        SetEncoding( "gzip" );
    }
    else if ( IsEncodingAccepted( "deflate" ) )
    {
        app.Response.Filter = new DeflateStream( app.Response.Filter,
  CompressionMode.Compress );
        SetEncoding( "deflate" );
    }
}

Then, go back to the client application, and modify the partial class definition:



public partial class Service1 
{
     #region WebResponseCompress
 
     protected override WebRequest GetWebRequest( Uri uri )
     {
         HttpWebRequest request = (HttpWebRequest)base.GetWebRequest( uri );
         request.Headers.Add( "Accept-Encoding", "gzip, deflate" );
     
         // create compressed request
         return new HttpWebRequestCompressed( request );
     }
 
     // THIS WILL NOT BE NEEDED ANYMORE
     //protected override WebResponse GetWebResponse( WebRequest request )
     //{
     //    return new HttpWebResponseDecompressed( request );
     //}
 
     #endregion
}

and add the HttpWebRequestCompressed class:



public class HttpWebRequestCompressed : System.Net.WebRequest
{
    private HttpWebRequest request;
 
    public HttpWebRequestCompressed( WebRequest request )
    {
        this.request = (HttpWebRequest)request;
    }
 
    public override WebResponse GetResponse()
    {
        return new HttpWebResponseDecompressed( this.request );
    }
 
    public override Stream GetRequestStream()
    {
        return new GZipStream( request.GetRequestStream(), CompressionMode.Compress );
    }
 
    public override string Method
    {
        get
        {
            return request.Method;
        }
        set
        {
            request.Method = value;
        }
    }
 
    public override WebHeaderCollection Headers
    {
        get
        {
            return this.request.Headers;
        }
        set
        {
            this.request.Headers = value;                     
        }
    }
 
    public override string ContentType
    {
        get
        {
            return this.request.ContentType.ToString();
        }
        set
        {
            this.request.ContentType = value;
        }
    }
}

And that's it! Note that the HttpWebRequestCompressed uses HttpWebResponseDecompressed when getting the response back. This is why we do not need to override the GetWebResponse in the proxy class.


Now inspect the application with Fiddler:



Note that this time the content length of the request is 6130 bytes (and the request is unreadable in Fiddler since it's compressed) and the response's length is still 5771 bytes.


Source code

Please download the source code for this tutorial here. If you find any bugs or have any other related comments, please feel free to comment this post.

24 comments:

Anonymous said...

yes.
Found a bug when there's a network lag or when working thread it's put to sleep.

it throws an error in WebHeaderCollection Headers NullReferenceExcepction

Diana

coriscow said...

Thanks Wiktor, you really saved my life with that post. It was difficult to find you, but it was worth.

For those of you who are not using CF 3.5 (I'm on 2.0), and thus do not have GzipStream, use sharplib. It's free and efficient.

Regards!

Wiktor Zychla said...

Thanks. I have this working in a production environment and it behaves quite reliable.

Unknown said...

I'm trying to implement the solution here, but I'm having an issue with the GZipStream request side. I did some packet sniffing and noticed the request is GZip encoded, but it's the same size as my non-encoded request? Has anyone run into this issue?

Windows Phone Application Development said...

Thanks for the nice information. I am sure, I will tweet this to my twitter account. This will help a lot of users.

web developers said...

Thanks for sharing the code with us !
great job !

Hire magento designer said...

I like your blog, It is very good. I am very happy to leave comment here for you!

Ecommerce developer said...

Excellent and useful blog.This blogs are good way to explain a site.Thanks for making a wonderful site.

Rubem Rocha said...

How to get the WSDL page for a compressed webservice coded with the illustrated techniques of this article?

[]s
Rubem Rocha
Manaus, AM - Brazil

Wiktor Zychla said...

@Rubem Rocha: the easiest way would be to add a stronger condition to the first "if" of the BeginRequest method of the compression module. Just check if the url contains ".asmx" but DOES NOT contain "?wsdl".

Rubem Rocha said...

I'm trying to implement this techniques on my webservice, but it's not working. First, I'm having 500 internal error messages. I've add the compression module in web.config like this:






I'm using VS 2010. I want to allow tests on webbrowser, but the communication with pocket PC client application and webservice must be compressed. How can I accomplish this?

Since now, thanks for your help!

[]s
Rubem Rocha
Manaus, AM - Brazil

Ohmnibus said...

Thanks for the post, very useful.

Just wondering why the method GetResponseStream() in HttpWebResponseDecompressed attempt to decompress a gzip stream when the StatusCode is "Internal Server Error".

Anonymous said...

Hello,

Thanks for sharing your code. The attached project is working fine but when I tried to enable compression in my webservice. But having the following exception.

System.InvalidOperationException: Client found response content type of '', but expected 'text/xml'.
The request failed with an empty response.

I followed the following steps.

In Web Services:
1. Add the class(HttpCompressionModule) to webservices project
2. Add the line in the web.config

And In Client:
1. Add Two Classes HttpWebResponseDecompressed and HttpWebRequestCompressed in the LCMS Client
2. Make changes in the Partial class of the webservice with the same name as present in the reference.cs and override the GetWebRequest Method.

Marcin Smółka said...

It is very nice solution, hoever I have few notes:
1. It is not workng when BufferResponse is set to false
2. If I would like to disable it temporary on server the client will stop work, which is a bit dangerous.

Nick G said...
This comment has been removed by the author.
Nick G said...

I found several bugs with this. It seems to try and unzip the request unconditionally (when the request was not actually GZIP'd). It also does not maintain the content-encoding header if an exception occurs on the webservice. In order to fix these errors I had to make a few changes.

I've posted an updated version of the code here:
http://pastebin.com/Aak9FUiw

Please contact me if you find any bugs.

Unknown said...

Hi, thank you for such wonderful codes. I am facing some problem when trying to consume the webservice from my console app, I have got an error saying "Response is not well-formed XML." Any idea what could causing this?

Unknown said...

hi, I manage to solve my problem above by enabling web service extension for ASP.NET 4 (previously only enable for ASP.NET 2), but however when I check with Fiddler, the compression ratio is 26:1, as contrast to 100:1, i use back the same string, what possible cause for this compression ratio difference?

Wiktor Zychla said...

@HonWaiLai: I would not be concerned by different compression ratio. This heavily depends on actual data and the implementation of the compression algorithm. You run it on .net 4 while my tests were performed on .net 2.

Unknown said...

I see. After some research plan to use serialization instead of compression, way to go!!!

Unknown said...

Hi,

when the client sends the gzipped request I think this line should be added in the 2nd implementation of the partial class Service1:

request.Headers.Add("Content-Encoding", "gzip");

Regards,
Falk

Unknown said...

Thank you so much! Now I would like to consume the same web service with Java Client. Hwo can I handle responses and requests?

Daniel

Wiktor Zychla said...

@Daniel, unfortunately I don't know the answer to your question. Regards!

Unknown said...

Hello,
firstly, thank you for share.

i want to get response length from serverside.
but when i tried context.response.outputstream.length ,
i getting "is not supported"

How can i get response length server side ?