Friday, June 2, 2017

HTTP/2, Keep-Alive and Safari browsers DDoSing your site

A supposedly common scenario where you have a HTTP/2 enabled reverse proxy layer in front of your application servers can have unexpected feature we have lately stumbled upon.

The problem is described by users of Safari browsers (regardless of the actual device, it affects iPhones, iPads as well as the desktop Mac OS) as "the page load time is very long". And indeed, where the page loads in miliseconds on any other browser on any other device, this combination of Apple devices and Safari browsers makes users wait for like 5 to 30 seconds before the page is rendered.

Unfortunately, the issue has a disastrous impact on the actual infrastructure - it is not the page that just loads long. It's much worse. It is the browser that makes as many requests as it is physically able to until it is finally satisfied and renders the page. If a client-side console is consulted in a browser - you see a single request. But from the server's perspective, it's the browser making hundreds of requests per second! During our tests, we have logged Safari waiting about 30 seconds before it rendered the page, making approximately 15000 requests to the server.

A complete disaster, a handful of Safaris could possibly take down a smaller site and even for us it was something noticeable.

We lost like few days tracking the issue, capturing the traffic with various HTTP debuggers and TCP packet analyzers on various devices and trying to narrow the issue to see a pattern behind it.

What it finally turned out to be was the combination of HTTP/2, Reverse proxy layer and specific configuration behind it.

It all starts with the common Connection: keep-alive header that is used to keep the TCP connection so that the browser doesn't open new connections when requesting consecutive resources.

The problem with this header is that since it is a default setting for HTTP/1.1, different servers treat it differently. In particular, the IIS doesn't send it by default while Apache always sends it by default.

And guess what, since Apache sends it, you can have an issue where it is the Apache that sits behing a HTTP/2 enabled proxy.

The scenario is as follows. The browser makes a request to your site and is upgraded to HTTP/2 by your front servers. Then, the actual server is requested and if it is the Apache (or any other server that sends the header) but HTTP/1.1 enabled, the actual server will return the Connection header to the proxy server. And if you are unlucky (as we were), the front server will just proxy the header to the client, together with the rest of the response.

So what, you'd ask. What's wrong with the header?

Well, the problem is, it is considered unnecessary for HTTP/2. The docs says:

The Connection header needs to be set to "keep-alive" for this header to have any meaning. Also, Connection and Keep-Alive are ignored in HTTP/2; connection management is handled by other mechanisms there.

The actual HTTP/2 RFC states otherwise

   This means that an intermediary transforming an HTTP/1.x message to
   HTTP/2 will need to remove any header fields nominated by the
   Connection header field, along with the Connection header field
   itself.  Such intermediaries SHOULD also remove other connection-
   specific header fields, such as Keep-Alive, Proxy-Connection,
   Transfer-Encoding, and Upgrade, even if they are not nominated by the
   Connection header field.

What could be considered unnecessary and be ignored, for some (hello Apple) is considered illegal. By analyzing TCP packets we have found out that Safaris internally fail with Error: PROTOCOL_ERROR (1) and guess what, they repeat the request over and over.

The issue has finally been narrowed down and it turned out others have identified it too in a similar scenario.

A solution is then to clear the Connection header from the outgoing response so that the browser never gets it. This potentially slows down browsers that can't make use of HTTP/2 connection optimizations but luckily, most of modern browsers support HTTP/2 for some time.

A happy ending and, well - it looks it is a bug at the Apple's side that should be taken care of to make it more compatible with other browsers that just ignore the issue rather than fail so miserably on it.

Monday, April 24, 2017

Using ngrok to expose any local server to the Internet

Ngrok is here for some time and it's a great tool that makes it possible to expose a website from local machine to the Internet by creating a tunnel from ngrok's infrastructure to your machine. This post adds 2 cents to this story. The story begins when you have a web app but not necessarily on your local machine but somewhere nearby at your local network. Sure, you can log into the local server and run ngrok there. But another approach is to have a HTTP proxy, from your machine to the local server. One of simplest and cleanest proxies that do the job would be this tiny node.js snippet
var httpProxy = require('http-proxy');

httpProxy.createProxyServer({
    target       : 'http://a.server.in.your.local.network:80', 
    changeOrigin : true, 
    autoRewrite  : true 
})
.listen(3000);
The two options make sure that both host and location headers are correctly rewritten. Please consult the list of other available options of the http-proxy module if you need more specific behavior. Then just invoke the ngrok as usual
ngrok http 3000
navigate to the address returned by ngrok and you will get the response from the server behind the proxy.

Friday, March 10, 2017

Visual Studio 2017 doesn't seem to support Silverlight but Silverlight itself is supported till 2021

This has been a great week as VS2017 has finally got its RTM version. Unfortunately, for some developers news aren't that great.

It seems the new VS drops the support for the Silverlight:

Silverlight projects are not supported in this version of Visual Studio. To maintain Silverlight applications, continue to use Visual Studio 2015.

Great. Note, though, that Silverlight 5 support lasts till 2021.

This effectively means that starting from now, till 2021, people will have to stick with VS2015 for some projects, despite VS2017, VS2019, VS2021 or any other newer version is released.

Nice one, Microsoft!

Friday, February 3, 2017

Calling promises in a sequence

One of the top questions about promises I was asked lately is a question on how to call promises in a sequence so that the next starts only after the previous one completes.

The question is asked mostly by people with C# background where a Task has to be started in an explicit way. In contrast, in Javascript, as soon as you have the promise object, it the execution has already started. Thus, following naive approach doesn’t work:

function getPromise(n) {
    return new Promise( (res, rej) => {
        setTimeout( () => {
            console.log(n);
            res(n);
        }, 1000 );
    });
}

console.log('started');

// in parallel
Promise.all( [1,2,3,4,5].map( n => getPromise(n)) )
    .then( result => {
        console.log('finished with', result);
    });

The above code yields 1,2,3,4,5 as soon as 1 second passes and there are no further delays, although people expect them here. The problem with this approach is that as soon as getPromise is called, the promise code starts. There are then 5 timers in total, started in almost the very same time, the time it takes the map function to loop over the source array.

In order to call promises in a sequence we have to completely rethink the approach. Each next promise should be invoked from within the then handler of the previous promise as then is called only where a promise is fulfilled. A simple structural trick is to use map.reduce in order to loop over the array so that each time we have the access to the previous “value” (the previous promise) and the current value (current number/whatever from the array). The map.reduce needs an “initial value” and it should be a promise so we just feed it with a dummy, resolved promise.

function getPromise(n) {
    return new Promise( (res, rej) => {
        setTimeout( () => {
            console.log(n);
            res(n);
        }, 1000 );
    });
}

console.log('started');

// in a sequence
[1,2,3,4,5].reduce( (prev, cur) => {
    return prev.then( result => {
        return getPromise(cur);
    });
}, Promise.resolve() )
    .then( result => {
        console.log('finished with', result);
    });

A neat trick worth of remembering.

Thursday, January 12, 2017

Node.js 7.x async/await example with express and mssql

Async/await has finally found its home in node 7.x and it’s just great, we’ve been waiting for ages. Not only it works great, with Visual Studio Code you can write and debug async/await like any other code. Just make sure you have runtimeArgs in your launch.json as a temporary fix until it’s not required somewhere in future:

{
    // Use IntelliSense to learn about possible Node.js debug attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "type": "node",
            "request": "launch",
            "name": "Launch Program",
            "program": "${workspaceRoot}/app.js",
            "cwd": "${workspaceRoot}",
            "runtimeArgs": [
                "--harmony"
            ]
        },
        {
            "type": "node",
            "request": "attach",
            "name": "Attach to Process",
            "port": 5858
        }
    ]
}

Async/await could possibly sound like a ephemeral alien until you realize that a lot of good code has already been written with Promises async/await is a sugar over. In an example below I read the data from a simple Sql Server table using node’s mssql module:

CREATE TABLE [dbo].[Parent](
 [ID] [int] IDENTITY(1,1) NOT NULL,
 [ParentName] [nvarchar](150) NOT NULL,
 CONSTRAINT [PK_Parent] PRIMARY KEY CLUSTERED
(
 [ID] ASC
)
) ON [PRIMARY]

I also need a connection string somewhere

// settings.js
module.exports.connectionString = 'server=.\\sql2012;database=databasename;user id=user;password=password';

Note that since tedious doesn't support integrated auth at the moment, you are stuck with sql's username/pwd authentication. The mssql on the other hand uses tedious internally.

Here goes an example of the "old-style" code, written with promises  
   var conn = new sql.Connection(settings.connectionString);
   conn.connect()
     .then( () => {

         var request = new sql.Request(conn);

         request.query('select * from Parent')
              .then( recordset => {

                  recordset.forEach( r => {
     // do something with single record
                  });

    conn.close();
              })
              .catch( err => {
                  console.log( err );
              });
     })
     .catch( err => {
         console.log(err);
     });  
(note that you could possibly avoid nesting thens by just returing the request.query promise so that the next, chained then would refer to it.) However, with async/await the same becomes
try {
 var conn = new sql.Connection(settings.connectionString);
 await conn.connect();
 
 var request = new sql.Request(conn);
 var recordset = await request.query('select * from Parent') 
 
 recordset.forEach( r => {
  // do something with single record
 });
 
 conn.close();
}
catch ( err ) {
 console.log( err );
}
Pretty impressive, if you ask me. If you don't mind occasional awaits, the code is clean, no .thens, no .catches. Remember that express middlewares can be async too
var app = express();

app.get('/', async (req, res) => {

      // code that awaits

});
All this means the callback hell is hopefully gone forever.

Friday, December 16, 2016

Combining sync/async WCF client/server that share an interface-based contract

Having a shared interface between WCF client and server is handy, often you just put such contract in a shared assembly and you just build your service out of the contract and then use the ChannelFactory to automatically create clients.

// shared contract
[ServiceContract]
public interface IService1
{
  [OperationContract]
  string DoWork( string s );
}
// server
public class Service1 : IService1
{
  public string DoWork( string s )
  {
     return s + " from wcf";
  }
}
// client
var binding  = new BasicHttpBinding();
var endpoint = new EndpointAddress( "http://localhost:55748/Service1.svc" );
 
var factory = new ChannelFactory<IService1>( binding, endpoint );
var channel = factory.CreateChannel();
 
var result  = channel.DoWork("foo");

The problem starts when you decide to go async at the server or the client but in the same time keep the other sync – in MVC or in Windows.Forms you just change the method from Foo Method() to async Task<Foo> Method(), here in WCF you just can’t change the signature of the interface method as it affects both the client and the server.

For example, I you to be async at the server side so that you switch to the async interface of your data provider, and you change the interface to

[ServiceContract]
public interface IService1
{
  [OperationContract]
  Task<string> DoWork( string s );
}

// oops, the client doesn't compile anymore as the supposed contract has just changed
// the client is not "smart enough" to notice that in fact this is still the same contract 
// at the conceptual level

Fortunately, a simple solution exists, mentioned here. The solution is to have two interfaces, a sync one and an async one. Both interfaces have to share the name, however they can exist in the very same assembly, in adjacent namespaces. It turns out that both the client and the server could implement any of the two interfaces and wiring still succeeds.

namespace Sync
{
    // shared contract
    [ServiceContract]
    public interface IService1
    {
        [OperationContract]
        string DoWork( string s );
    }
}

namespace Async
{
    [ServiceContract]
    public interface IService1
    {
        [OperationContract]
        Task<string> DoWork( string s );
    }
}

An example async client, using the async interface, would be

  
private async void button1_Click( object sender, EventArgs e )
{
    var binding  = new BasicHttpBinding();
    var endpoint = new EndpointAddress( "http://localhost:55748/Service1.svc" );

    var factory = new ChannelFactory<Async.IService1>( binding, endpoint );
    var channel = factory.CreateChannel();

    var result = await channel.DoWork( "foo" );

    MessageBox.Show( result );
}

An example async server would be

public class Service1 : IService1
{
    public async Task<string> DoWork( string s )
    {
        await Task.Delay( 5000 );
        return s + " from wcf";
    }
}

The conclusion is: the shared assembly should just provide both interfaces, the sync and the async one, which lets both the client and the server freely decide which interface they implement. In a sense, the client and the server still share the same interface although technically these are two different interfaces.

Thursday, August 25, 2016

SignedXml::CheckSignature and dreadful “The profile for the user is a temporary profile” on .NET 4.6.2

Sometimes, out of nowhere, your well-tested and working-for-years code just stops working just because something changes somewhere you can really have control over.

This post documents one of such cases. The case is about checking digital signatures of signed XML documets. For years we’ve been creating and validating documents with self-contained signatures. I’ve blogged on that in a three part tutorial. In our scenario, IIS hosted ASP.NET applications were receiving signed XML documents and processing them.

This worked as a charm until .NET 4.6.2. In .NET 4.6.2 the SignedXml::CheckSignature( X509Certificate2 ) method we’ve been using for years doesn’t work anymore in our scenario, it throws “The profie for the user is a temporary profile” exception.

What is a temporary profile? You see, when an app pool is configued in IIS to use a custom account (a domain account in our case) and the Load User Profile is turned on, the ASP.NET worker process is supposed to load the profile of the app pool account. The existence of the profile changes some details on how temporary paths are resolved and how keys are imported to cert stores. IIS has an unfortunate way of handling this requirement – instead of creating a regular profile, it creates a temporary profile. People often ask what these profiles are and how to turn them off.

Unfortunately, there is no easy way to change this behavior of creating temporary profiles rather than regular profiles (or we don’t know how to do it). A workaround exists – you can just log into the server as the app pool account with an interactive session (a direct login or a remote desktop session). A local profile is then created at the server and further loading the profile by ASP.NET use the newly created, valid profile rather than a temporary one (e.g. the workaround is mentioned here at Paul Stovell blog). But suppose you have a farm where servers are cloned on demand (we use the Web Farm Framework). Suppose also you have multiple apps. We have like ~150 servers and ~50 apps on each, this means logging 150*50 times to different servers only to have profiles created correctly.

Unfortunately also, to be able to use cryptographic API, you often just have to turn on loading user profiles, otherwise the crypto system just doesn’t work sometimes (private keys cannot be accessed for certs that are loaded from local files). In our case, turning off the profile raises an instant exception “Object is in invalid state”.

What all it means is that .NET 4.6.2 changes the way SignedXml::CheckSignature( X509Certificate2 ) works and changes it in a fundamental way. Before 4.6.2 the method works always, regardless of whether the profile is a temporary profile or not, as long as the profile is loaded (Load User Profile = true). In .NET 4.6.2 the method doesn’t work if the profile is a temporary profile.

One of our first attempts was to follow the previously mentioned workaround and somehow automate the profile creation so that profiles are there when IIS requests them. This however doesn’t solve the issue on already existing servers.

But then another approach was tested. Because we check signatures on self-contained signed documents, X509Certificate2 instances we pass to the method are retrieved from the SignedXml:

// Create a new SignedXml object and pass it  // the XML document class.
SignedXml signedXml = new SignedXml( xd );
 
// Load the first <signature> node.   signedXml.LoadXml( (XmlElement)messageSignatureNodeList[0] );

// load certificate 
foreach ( KeyInfoClause clause in signedXml.KeyInfo ) 

    if ( clause is KeyInfoX509Data ) 
    { 
        if ( ( (KeyInfoX509Data)clause ).Certificates.Count > 0 ) 
        { 
            certificate = (X509Certificate2)( (KeyInfoX509Data)clause ).Certificates[0]; 
        } 
    } 
}

if ( certificate != null ) 

    // Check the signature and return the result. Throws the “user profile is a temporary profile” 
    return signedXml.CheckSignature( certificate, true ); 

else 
    return false;

The exception possibly is related to the possible leakage of private keys when temporary profiles are involved. Maybe one of .NET BCL developers was just oversensitive here, implementing a bunch of new features.

There are no private keys, however, in our certificates! Since certificates are here to verify signatures, private keys are not included, we only have public keys. How about using the overload of the CheckSignature that only expects an AsymmetricAlgorithm?

// Create a new SignedXml object and pass it 
// the XML document class.
SignedXml signedXml = new SignedXml( xd );

// Load the first  node.  
signedXml.LoadXml( (XmlElement)messageSignatureNodeList[0] );

AsymmetricAlgorithm rsa = null;

// load certificate
foreach ( KeyInfoClause clause in signedXml.KeyInfo )
{
	if ( clause is KeyInfoX509Data )
	{
        	if ( ( (KeyInfoX509Data)clause ).Certificates.Count > 0 )
	        {
        	      certificate = ((KeyInfoX509Data)clause).Certificates[0] as X509Certificate2;
	        }
        }
}

if ( certificate == null )
{
	Message = "No KeyInfoX509Data clause in the signature";
        return false;
}

if ( certificate.PublicKey == null || certificate.PublicKey.Key == null )
{
	Message = "The KeyInfoX509 clause doesn't contain any valid public key";
        return false;
}

rsa = certificate.PublicKey.Key;

// Check the signature and return the result. 
return signedXml.CheckSignature( rsa );

Yes, as you can expect, this works correctly, again regardless on whether the profile is a regular profile or a temporary one.