Tutorial: using url rewriting to build an application using multiple independent data sources

Download the source code for this tutorial 

As you are happy with your web application deployed on the application server, serving the data to your clients, someone asks you to launch yet another instance of your application to serve data for another group of clients. Then you repeat the process, create new copies of the application in your application server and new databases.

Then suddenly you wake up having few, few dozens or few hundreds copies of the same application deployed in your application server. And someone asks you to upgrade the application. You copy files to their directories, sign clickonce applications and convert databases. The upgrade takes forever to complete or worse - crashes before it's complete.

Sounds familiar?

I belive that this is one of the issues not-so-widely discussed (at least not so widely I would expect) and in the same time - it's one of the issues which occur not during the development phase but rather when the application is used for quite a time.

The intention of this post is to give you some basic ideas of architectural patterns used to build applications that are easily maintainable in many independent instances.

Assumptions

First of all - I do not assume that independent datasources can easily be merged into a single huge datasource. This is not the right way to go - large datasources are hard to maintain and debug and, moreover, you can consider putting all the data from different clients into a single database insecure.

However, there's no point of having copies of the same application in the application server. As you will see, it's fairly easy to be able to run multiple independent data sources using a single application.

So, how different datasources are distinguished by end-users? Again, it's fairly simple: by using virtual url addresses which are specific to data sources.

Suppose that you have your http://yourwebsite.com/start.aspx webpage, served for all users. Users running the datasource instance1 will use http://yourwebsite.com/instance1/start.aspx address while users running instance2 will use http://yourwebsite.com/instance2/start.aspx but both addresses will be virtual meaning that the only resource physically existing in your web server will be the http://yourwebsite.com/start.aspx webpage.

Goal

The goal of this tutorial is then to write an application which can "virtually" address nonexistent "instances" of independend datasources thus giving access to different datasources to different users. However, as the datasources have to be physically separated, we still want to have just a single instance of the application deployed on the application server.

Towards the solution

It seems that implementing such virtual addressing used to pick the datasource is relatively easy - all you have to do is to plug into the ASP.NET processing pipleline early enough to be able to perform url rewriting. There are two possible ways to rewrite urls in ASP.NET, however I prefer to use modules, specifically - the Global.asax, global application class.

It turns out that no matter what ASP.NET resource is actually accessed and whether the resource exists or not, you can plug into the begin request event and use HttpContext's RewritePath method to perform url rewriting.

In real life applications you would probably prefer to match addresses using regular expressions, however, for this simple tutorial it's sufficient to manually parse the request address to find out which datasource the user is actually accessing.

   1: void Application_BeginRequest( object sender, EventArgs e )

   2: {

   3:     HttpApplication app = (HttpApplication)this;

   4:     HttpContext ctx = app.Context;

   5:  

   6:     string LocalPath = 

   7:         Helpers.ToAppRelativeWithQueryString( ctx.Request.Url.PathAndQuery );

   8:     string[] Segments = LocalPath.Split( '/' );

   9:  

  10:     if ( Segments.Length > 2 )

  11:     {

  12:         string Instance = Segments[Segments.Length - 2].Replace( "/", "" );

  13:  

  14:         /* remember curren instance */

  15:         ctx.Items["INSTANCE"] = Instance;

  16:         /* remember that url has been rewritten */

  17:         ctx.Items["REWRITE"] = true;

  18:  

  19:         string NewUrl = app.Request.Url.PathAndQuery.Replace( "/" + Instance, "" );

  20:  

  21:         app.Context.RewritePath( NewUrl );

  22:     }

  23: }

 

Let's start with lines 6/7. You see, as the actual request comes to the application server, the PathAndQuery differs depending on whether the application is located under the root directory of the applicaiton server or in the application server's "application directory". Suppose that users refers to the start.aspx page. In the former example the PathAndQuery is just ~/start.aspx. However, in latter case, when the application is put inside IIS's application, the PathAndQuery is ~/ApplicationName/start.aspx.

To be able to omit the part of the PathAndQuery which corresponds to the IIS's application name, I use magical

   1: VirtualPathUtility.ToAppRelative( ... )

from the standard library, however, this little shiny gem does not work with paths that contain nonempty querystrings! This is why I wrap it with my custom ToAppRelativeWithoutQueryString.

Then, in line 8 I split the PathAndQuery to check whether it refers to the virtual address of nonexistent instance of the application. As I said, you usually use regular expressions there.

If there are more than 2 segments, I know that the address user refers to is of the form http://yourservername.com/instance1/start.aspx. In line 15 I store the instance name in the temporary Items container and in line 17 I store the information about the rewriting being made.

Two notes here. First - as you'll find out in a minute, in this tutorial I store the information about the instance users selects (with the virtual address) in the server's session container. However - this is not the best way to go as it raises few technical issues. For example, the session container is not available during the BeginRequest phase of the pipeline.

So I store the information in the Items container and just to copy it to session container after it becomes available:

   1: void Application_AcquireRequestState( object sender, EventArgs e )

   2: {

   3:     HttpApplication app = (HttpApplication)this;

   4:     HttpContext ctx = app.Context;

   5:  

   6:     /* copy from items to session */

   7:     if ( ctx.Items["INSTANCE"] != null )

   8:         ctx.Session["INSTANCE"] = ctx.Items["INSTANCE"];

   9: }

Yet another note about the line 17 of the previous code snippet - we'll make use of the fact that the url is rewritten in the login page.

In the line 19 I build the new url by removing the virtual part of it which refers to the instance name and in line 21 I rewrite the url to the existing one.

The rewriting itself is really powerful operation. It changes the internal address of the request while maintaining all the POST parameters. This is why it's possible to build applications that rewrite urls - without POST parameters passed to rewritten address, you'd for example end up with no events raised from any controls!

Tweaks in the login page

Url rewriting is just not enough, what we have to do is to tweak the forms authentication. Normally, when users access pages they are not authorized to access, they are redirected to the login page. The problem is that there can be only a single login page specified in the forms authentication's section of the web.config. How do we then redirect some users to http://yourservername.com/instance1/loginpage.aspx and other users to http://yourservername.com/instance2/loginpage.aspx?

Well, that's easy again. Put this into the Page_Load of the login page:

   1: public partial class LoginPage : System.Web.UI.Page

   2: {

   3:     protected void Page_Load(object sender, EventArgs e)

   4:     {

   5:         /* if there is an instance and url has NOT been rewritten yet */

   6:         if ( this.Session["INSTANCE"] != null &&

   7:              this.Context.Items["REWRITE"] == null

   8:             )

   9:         {

  10:             this.Response.Redirect( 

  11:                 Helpers.CreateVirtualUrl( this.Request.Url.PathAndQuery ) );

  12:         }

  13:     }

  14: }

Take a look at lines 6/7. If the users uses a virtual instance (the information for which we store in the Session container) and there has been no rewriting during processing of current request - we redirect the login page to another login page, the virtual one. And, as the first request to the login page will be of the form http://youraddress.com/LoginPage.aspx?ReturnUrl=Default.aspx, we redirect to http://youraddress.com/instance1/LoginPage.aspx?ReturnUrl=Default.aspx. This new, virtual login page, is then processed by the BeginRequest, where we rewrite it back again to the former form but this time there is no redirect back in the LoginPage's Page_Load since the Items container stores the information of the rewriting beeing made.

Storing the instance name user selects with his first request

As I said, in this tutorial we store the information of the user's virtual context in the Session container, however cookies seem much more practical solution. There's only one important security issue.

Suppose users opens http://youraddress.com/instance1/Default.aspx, gets redirected to http://youraddress.com/instance1/LoginPage?ReturnUrl=Default.aspx and provides his/her valid credentials. Then he types http://youraddress.com/instance2/Default.aspx into his/her browser and in the same time he deletes the cookie pointing the instance he/she has selected before from his/her browser.

Can you see what happens? Your BeginRequest happily assigns new cookie which stores the information about instance2 as the current selection, however, the Forms Authentication cookie is still there! And the user gets the access to private data of another instance of the datasource. Disaster.

I solve this issue by calling FormsAuthentication.SignOut() in BeginRequest each time I find out that there's no cookie pointing to a selected instance. Do allow users to change datasource instances in the same browser's window but allow them access only public resources for which a forms cookie is not required.

I also store the instance name in the UserData part of the Forms Authentication cookie and each time BeginRequest processes a virtual address, I check if the address of the virtual instance matches the one stored in the UserData.

Handling IIS's 404 error pages

There's yet a minor issue regarding the way IIS processes requests. You see, if there's the .aspx extension in your request, the IIS always passes such request to ASP.NET. However, if an user just navigates to http://youraddress.com/instance1/ (which is his/her legitimate address!), IIS happily returns 404 page, since there's no physical /instance1 resource available on the server.

The solution to this issue is to redefine the page IIS sends back when 404 occurs. Change the standard page to your custom page and do what you want to make user's browser ask for anything ending with .aspx.

For example, put:

   1: <html>

   2: <head>

   3: </head>

   4: <script language="javascript" type="text/javascript">
   1:  
   2: function redirect404()
   3: {
   4:   var infix = '';
   5:   if ( location.href.toString().charAt( location.href.toString().length - 1 ) != '/' )
   6:     infix = '/';
   7:   location.href = location.href + infix + "start.aspx";  
   8: }
</script>

   5: <body onload="redirect404()">

   6: </body>

   7: </html>

into the _404.htm file and configure IIS to serve _404.htm in response to 404 error.

Where to go now from there

The basics are behind, let's discuss three issues.

First of all - you'll notice that you create a lot of relative/absolute links in your code which must be modified so that they point to /instance1/thelinkcontent instead just /thelinkcontent. It does not take a lot of time but the whole application has to be carefully examined and tested.

And another issue - you end up with a single application talking to many independed datasources. In case of databases - you have to be sure that the structure of all databases match the object model in your application.

This is where I recommend a very handy pattern - do not convert databases manually but rather write a code which converts the database and execute the code each time you handle the first request to a database. This way databases are automatically coverted when they are accessed for the first time after you deploy a new version of your web application.

And the last issue - your HTTP forms will be rendered with invalid action value. This issue can be resolved using standard techniques used by any ASP.NET url rewriting frameworks. You can either create your own Form Control Adapter or change the way forms are rendered.

If you are interested in url rewriting itself, please take a look here.

Tortoise SVN - tag & merge tutorial

Since it's quite confusing how to branch/tag and then merge a branch with the production code - here's a short tutorial on the issue. Note that the strategy of tagging and merging presented below is probably not the only possible, however it works in practice.

I assume that a SVN repository is created, the SVN server works and Tortoise is installed on the client machine.

Step 1 - Import local files into the repository

Start from creating a local directory with application files in it.

  

You need to "put" the local directory into the SVN server. In SVN language this is called "import". Click the Application directory with the right mouse button and choose TortoiseSVN/Import from the shell context menu.

After you click "OK", the local directory is imported into the SVN. Note, that I use /trunk subdirectory to actually store files there. This is because we will create other subdirectories in the same repository and these subdirectories will contain tags of the code.

Step 2 - Checkout managed repository from SVN back to your local machine

Now you need a new, fresh location on your local filesystem dedicated to work as a local image of the repository. Create a fresh directory, let's call it ApplicationCode, right click on it and choose "SVN Checkout" from the context menu.

The Checkout operation retrieves the contents of the repository and creates a local image of it which can be then synchronized with the repository using the "Update" and "Commit" operations.

Local filesystem after the checkout operation looks like this:

Delete the /Application folder, it's no longer needed.

Step 3 - Continue working on the /trunk line of the code

This is what people find confusing.

Should I continue to develop application using the /trunk line? Or should I rather create a new /development line?

Well, there's no single answer to this question, however the advice is as follows: use the /trunk line to make any changes you want and whenever you make a release to clients - create a tag.

Let's then continue to work on the /trunk line for a while - create a new document "c.txt" in the /ApplicationCode directory and commit changes into the repository (right click on the /ApplicationCode and select "Commit" option).

Step 4 - Tag whenever you release your application to clients

A great day comes and you release your application to your clients. Click the /ApplicationCode with the right mouse button and select TortoiseSVN/Branch-tag option from the context menu. This invokes the "Copy (Branch/Tag)" window where you should give a name to the tag's directory.

Let's assume that the tag's name is "tag1" (this is where it turns out that putting files into /trunk subdirectory is important - both /trunk and /tag1 will be subdirectories of the same repository directory on the server).

Click "OK" and SVN will happily inform you that "your working copy remains on the previous path. If you want your next changes to be in the just created copy then you need to switch over to that copy path. Use the Switch command to do that."

Step 5 - Oops, a bug's been found. Go back to the Tag and fix the bug

As you continue your work on the /ApplicationCode directory and Update/Commit changes into the /trunk subdirectory in the SVN, a day comes when a bug is found in the code released to your clients. This is where you need to go back to the code, make changes and synchronize changes with the /trunk line.

As the message from the previous step says, you could use context menu's "TortoiseSVN/Switch" command to switch the code in the local /Application directory. However, this is not required!

Instead, I prefer to restore the image of the SVN's /tag1 into just another, fresh directory in the local filesystem. Let's then create /ApplicationCode_Tag1 directory in the local filesystem and checkout the /tag1 into it (just like we did in Step 2).

Now, let's fix the bug in the code retrieved from the /tag1: we'll modify the b.txt and create d.txt.

Commit changes from the /ApplicationCode_Tag1 local directory. Note that changes are stored only in the /tag1/ subdirectory in the SVN.

Step 6 - Merge the patched code with the /trunk line

This is the most fun part - we'll merge changes applied to the /tag1/ line of the code with the /trunk line of the code.

First, invoke the TortoiseSVN/Merge function from the context menu on the /ApplicationCode directory.

Note that the text in the lower groupbox correctly informs you that the result of the merge will be stored in the /ApplicationCode folder which contains an image of the /trunk folder. However, what you are supposed to do is to specify the From: and To: sources correctly.

Let's leave the To: untouched and click the "Show log" beside the "Revision" in the From: section.

Here you see that one of the revisions is printed with blue color - this is the number of revision which had been active when the tag has been made. Remember the number (1101 in my case), cancel the log window and put that number in the Revision textbox in the Merge window:

Do a "Dry run" where you can inspect changes which are about to be made to the /trunk subdirectory. In our example you'll learn that the b.txt is going to be updated and d.txt is going to be added. Click "Merge" and inspect the contents of the /ApplicationCode folder:

Happy SVNing!

Application Architecture Guide 2.0 from Patterns & Practices

The Microsoft's Patterns and Practices group continue their great work on the Application Architecture Guide. You should definitely check out the newly released Beta 2 version.

http://www.codeplex.com/AppArchGuide

log4net and appenders for different levels

The log4net uses different logging levels, ALL, DEBUG, INFO, WARN, ERROR and FATAL.

What I wanted to have is to log debugging, information and warnings into an appender and errors and fatal errors to another appender.

Here's how to define two appenders for different levels:

   1: <?xml version="1.0" standalone="yes"?>

   2: <log4net>

   3:     <appender name="RSLogFileAppenderInfo" type="log4net.Appender.RollingFileAppender">

   4:         <file value="PATHTOFILE\info.log" />

   5:         <appendToFile value="true" />

   6:         <rollingStyle value="Size" />

   7:         <filter type="log4net.Filter.LevelRangeFilter">

   8:             <acceptOnMatch value="true" />

   9:             <levelMin value="DEBUG" />

  10:             <levelMax value="WARN" />

  11:         </filter>

  12:         <maxSizeRollBackups value="10" />

  13:         <maximumFileSize value="10MB" />

  14:         <staticLogFileName value="true" />

  15:         <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />

  16:         <layout type="log4net.Layout.PatternLayout">

  17:             <conversionPattern value="%newline%date [%thread] %-5level - %message" />

  18:         </layout>

  19:     </appender>

  20:     <appender name="RSLogFileAppenderFatal" type="log4net.Appender.RollingFileAppender">

  21:         <file value="PATHTOFILE\error.log" />

  22:         <appendToFile value="true" />

  23:         <rollingStyle value="Size" />

  24:         <filter type="log4net.Filter.LevelRangeFilter">

  25:             <acceptOnMatch value="true" />

  26:             <levelMin value="ERROR" />

  27:             <levelMax value="FATAL" />

  28:         </filter>

  29:         <maxSizeRollBackups value="10" />

  30:         <maximumFileSize value="10MB" />

  31:         <staticLogFileName value="true" />

  32:         <lockingModel type="log4net.Appender.FileAppender+MinimalLock" />

  33:         <layout type="log4net.Layout.PatternLayout">

  34:             <conversionPattern value="%newline%date [%thread] %-5level - %message" />

  35:         </layout>

  36:     </appender>

  37:     <root>

  38:         <level value="ALL" />

  39:         <appender-ref ref="RSLogFileAppenderInfo" />

  40:         <appender-ref ref="RSLogFileAppenderFatal" />

  41:     </root>

  42: </log4net>

Note that there are two appenders with different filters, the former raning from DEBUG to WARN and the latter ranging from ERROR to FATAL.

The simple helper class I've found somewhere and use to simplify the logging:

   1: public class Log

   2: {

   3:     public static ILog For( object LoggedObject )

   4:     {

   5:         if ( LoggedObject != null )

   6:             return For( LoggedObject.GetType() );

   7:         else

   8:             return For( null );

   9:     }

  10:  

  11:     public static ILog For( Type ObjectType )

  12:     {

  13:         if ( ObjectType != null )

  14:             return LogManager.GetLogger( ObjectType );

  15:         else

  16:             return LogManager.GetLogger( string.Empty );

  17:     }

  18: }

To test different levels defined for these two appenders just call:

   1: class LogTest

   2: {

   3:   public void Test()

   4:   {

   5:       Log.For( this ).Fatal( "fatal" );

   6:       Log.For( this ).Error( "error" );

   7:       Log.For( this ).Warn ( "warn" );

   8:       Log.For( this ).Info ( "info" );

   9:       Log.For( this ).Debug( "debug" );

  10:   }

  11: }

  12: ...

  13: LogTest test = new LogTest();

  14: test.Test();

Note that logs with different levels are correctly directed to appropriate appenders.

ASPNET user account folder and files

Did you ever thought about the folder and files created by Windows XP for the ASPNET user account?

"The ASPNET user account is an internal account and no one is able to log using this identity"

This is the most common answer for my question. And yes, the answer is true. At least it touches the most important part. But there's more.

You see, since Windows XP invokes ASP.NET processes using ASPNET account, it also automatically creates a user folder with typical structure (Application Data, Cookies, My Documents etc.) for this account.

And here comes the strange part: the folder structure for the ASPNET user account is created under "Documents and Settings" but instead of username as a folder name, the machinename/username is used.

Suppose now that your machine's name is XYZ and you use a user account named XYZ. Guess what - you'll see ASPNET user local folder inside your own user folder.

Something tells me that this is not quite right. I should not be able to peek into other user's files but since the folder sits inside my own local folder, I am able to browse it with no restrictions.

I would say that this is rather insecure, no matter if it can be easily misused or not.

The "cookie" property of mshtml.IHtmlDocument2 does not work anymore

In one of our applications we host the Internet Explorer ActiveX control in a window and manually create parameters for navigation (including the body and http headers). To correctly inject the ASP.NET session id cookie into such requests (the page we navigate to comes from the ASP.NET server) we have to be able to somehow inject the ASP.NET_SessionId cookie into the navigation context.

The application was written two (or three) years ago, and we've been injecting the cookie using the cookie property of the mshtml.IHtmlDocument2 interface.

/* we've retrieving the reference to the document */

mshtml.IHtmlDocument2 doc = ...; 

/* and we've been setting the aspnet session id cookie */

doc.cookie = ...;

This worked like a charm, up to this year - the application is heavily used but only in september (it's responsible for gathering and processing some data available in september and then the data is passed to another appliaction) and this year we've been surprised to see that the cookie property does not work (at least in Internet Explorer 7). Both accessors (set and get) seem to have no effect and you always get the null value from the getter.

Pretty annoying. I guess it's somehow related to security issues but surely it broke the backward compatibility, at least for us.

The solution is to use another method which can inject cookies into the http processing chain:

/* injects cookies into the http processing for

   current process */

[DllImport("wininet.dll")]

public static extern bool InternetSetCookie( 

    [MarshalAs(UnmanagedType.LPStr)]

    string Url,

    [MarshalAs(UnmanagedType.LPStr)]

    string CookieName,

    [MarshalAs(UnmanagedType.LPStr)]

    string CookieData

    );

...

    /* first inject the cookie */

    InternetSetCookie(

        "url",

        "ASP.NET_SessionId",

        "theaspnetcookievalue_nomatterhowyougetit" );

    /* then navigate */

    webBrowser1.Navigate( "url" );

    /* the cookie is correctly set for the navigation.

       although you STILL cannot see it using mshtml.IHtmlDocument2.cookie,

       the http debugger reveals that it's really there 

    */

SmtpClient exception - An invalid character was found in the mail header

This hit me today as I tried to move a configuration section to another configuration file. It seems that the SmtpClient class does not like non-english letters in email header.

MailMessage mail = new MailMessage();

mail.From = new MailAddress( 

    "noreply@noreply.com", 

    "ąęłóżśńć", // <- the cause of the exception

    Encoding.UTF8 ); 

What's really scary is that the code above actually works - the exception occured only when the subject name was read from the configuration file!

After an intense struggle, I've found the solution. Just add

mail.SubjectEncoding = Encoding.UTF8;

somewhere around. It works, at least for me.