Friday, May 24, 2013

Public Suffix List

The most obvious aspect of the Public Suffix List is that your website can’t issue cookies for any of these effective top-level domains.

For example, your web site cannot issue cookies for wroclaw.pl (since it IS on the list) but you can issue cookies for koszalin.pl (since it is NOT on the list).

In theory, the list is available from Mozilla’s site and the only question is “do all browsers respect the list”?

Wednesday, May 15, 2013

Yet Another ORM Micro Benchmark, part 3/3, results

This is the last post of the series. We introduced a micro orm benchmark with few contestants and we implemented all required tests. In this post we analyze results.

Complete set of results

Let us start from a complete set of results (all numbers are miliseconds so for example 531 means 0.531 sec and 6502 means 6.502 sec):

category

name

totalms

ADO.NET

TOP 1

144.0144

Linq2Sql

TOP 1

526.0526

EF ModelFirst NoTracking

TOP 1

531.0531

EF ModelFirst Tracking

TOP 1

551.0551

EF CodeFirst NoTracking

TOP 1

1063.1063

EF CodeFirst Tracking

TOP 1

1069.1069

nHibernate StateLess

TOP 1

654.0654

nHibernate StateFull

TOP 1

658.0658

ADO.NET

TOP 10

111.0111

Linq2Sql

TOP 10

574.0574

EF ModelFirst NoTracking

TOP 10

499.0499

EF ModelFirst Tracking

TOP 10

627.0627

EF CodeFirst NoTracking

TOP 10

1046.1046

EF CodeFirst Tracking

TOP 10

1247.1247

nHibernate StateLess

TOP 10

1194.1194

nHibernate StateFull

TOP 10

1305.1305

ADO.NET

TOP 100

270.027

Linq2Sql

TOP 100

710.071

EF ModelFirst NoTracking

TOP 100

675.0675

EF ModelFirst Tracking

TOP 100

1596.1596

EF CodeFirst NoTracking

TOP 100

1210.121

EF CodeFirst Tracking

TOP 100

2836.2836

nHibernate StateLess

TOP 100

5699.5699

nHibernate StateFull

TOP 100

6502.6502

ADO.NET

TOP 1000

1327.1327

Linq2Sql

TOP 1000

2064.2064

EF ModelFirst NoTracking

TOP 1000

1885.1885

EF ModelFirst Tracking

TOP 1000

10700.0699

EF CodeFirst NoTracking

TOP 1000

2365.2365

EF CodeFirst Tracking

TOP 1000

18658.8657

nHibernate StateLess

TOP 1000

51017.1012

nHibernate StateFull

TOP 1000

58400.8395

ADO.NET

TOP 10000

10878.0877

Linq2Sql

TOP 10000

19527.9526

EF ModelFirst NoTracking

TOP 10000

13496.3495

EF ModelFirst Tracking

TOP 10000

122345.2333

EF CodeFirst NoTracking

TOP 10000

13797.3796

EF CodeFirst Tracking

TOP 10000

210554.0533

nHibernate StateLess

TOP 10000

546423.6369

nHibernate StateFull

TOP 10000

564736.468

The most interesting comparison to me was to see how retrieving data works for different sizes of datasets. Here are results presented on charts:

General observations:

  • bare ADO.NET is of course always the fastest but for the largest dataset, EF with tracking turned off is almost as fast (which is amazing)
  • Linq2Sql is outperformed by EF Model First No Tracking and, for the largest dataset, even EF Code First No Tracking is way faster than Linq2SQL
  • EF Model First is always faster than EF Code First
  • EF with tracking is always much slower than EF without tracking. For 1000 or more records tracking can probably hurt the performance badly if you are only reading the data
  • nHibernate StateLess is always faster than nHibernate StateFull (which is expected)
  • nHibernate is surprisingly slow when larger sets are retrieved (which is rather unexpected and frankly, I wonder if there is a way to speed it up as it looks rather disappointing)

Note that benchmaring ORMs should also involve inserts/updates/deletes in various scenarios. Please consider my experiments as valid only in this specific context (data retrieval).

I would also like to hear a comment or two about this. I am especially interested in the underperformance of nHibernate for large sets of data – I wonder what else could be done to speed it up.

Yet Another ORM Micro Benchmark, part 2/3, implementation

As the principles of the benchmark have been laid, I can now move to implementations. What follows are code snippets of each test case, with comments where necessary.

ADO.NET

The ADO.NET test is straightforward:

public class PerformanceADOSelect : IPerformanceTest
{
    public PerformanceADOSelect( int NumberOfItems, int NumberOfTests )
    {
        this._NUMBEROFITEMS = NumberOfItems;
        this._NUMBEROFTESTS = NumberOfTests;
    }
 
    int _NUMBEROFITEMS;
    int _NUMBEROFTESTS;
 
    public void PerformTest( out DateTime StartedAt, out DateTime EndedAt )
    {
        StartedAt = DateTime.Now;
 
        for ( int i = 0; i < _NUMBEROFTESTS; i++ )
        {
            StringBuilder sb = new StringBuilder();
            using ( SqlConnection conn = 
               new SqlConnection( ConfigurationManager.ConnectionStrings["ParentChildConnectionString"].ConnectionString ) )
            {
                conn.Open();
                using ( SqlCommand cmd = 
                   new SqlCommand( string.Format( "SELECT TOP {0} * FROM Child", _NUMBEROFITEMS ), conn ) )
                using ( SqlDataReader dr = cmd.ExecuteReader() )
                {
                    while ( dr.Read() )
                    {
                        Child child = new Child();
 
                        child.ID        = (int)dr["ID"];
                        child.ID_PARENT = (int)dr["ID_PARENT"];
                        child.ChildName = (string)dr["ChildName"];
 
                        sb.Append( child.ChildName );
                    }
                }
            }
            sb = null;
        }
 
        EndedAt = DateTime.Now;
    }
 
    public string Category
    {
        get { return "ADO.NET"; }
    }
 
    public string Name
    {
        get { return string.Format( "SELECT TOP {0} - {1} TIMES", _NUMBEROFITEMS, _NUMBEROFTESTS ); }
    }
}
 
public class Child
{
    public int ID { get; set; }
    public string ChildName { get; set; }
    public int ID_PARENT { get; set; }
}

Note that the materialization is done manually but in addition, a dummy StringBuilder accumulates some data for each read record. This string builder is not important with this ADO.NET test but in later tests the intention of it will be to simulate that the data is actually used somehow (not only read from the database).

Linq2SQL

This time also the test is straightforward. The model has been generated automatically, both the data context and entities.

public class PerformanceLinq2SqlSelect : IPerformanceTest
{
    public PerformanceLinq2SqlSelect( int NumberOfItems, int NumberOfTests )
    {
        this._NUMBEROFITEMS = NumberOfItems;
        this._NUMBEROFTESTS = NumberOfTests;
    }
 
    int _NUMBEROFITEMS;
    int _NUMBEROFTESTS;
 
    public void PerformTest( out DateTime StartedAt, out DateTime EndedAt )        
    {
        // warmup
        using ( ParentChildDataContext ctx = new ParentChildDataContext() )
        {
            var children = ctx.Childs.ToList();
        }
 
        StartedAt = DateTime.Now;
 
        for ( int i = 0; i < _NUMBEROFTESTS; i++ )
        {
            StringBuilder sb = new StringBuilder();
            using ( ParentChildDataContext ctx = new ParentChildDataContext() )
            {
                foreach ( var child in ctx.Childs.Take( _NUMBEROFITEMS ) )
                    sb.Append( child.ChildName );
            }
            sb = null;
        }
 
        EndedAt = DateTime.Now;
    }
 
    public string Category
    {
        get { return "Linq2Sql"; }
    }
 
    public string Name
    {
        get { return string.Format( "SELECT TOP {0} - {1} TIMES", _NUMBEROFITEMS, _NUMBEROFTESTS ); }
    }
}

Note that this time I perform a “warmup” – this is to make sure that no internals of the ORM depending on the “first run” will influence the measurement.

EF5 Model First

This test is similar to the previous one – model is generated with the help of the designer, both object context and entity classes are available.

public class PerformanceEFModelFirstSelect : IPerformanceTest
{
    public PerformanceEFModelFirstSelect( bool Tracking, int NumberOfItems, int NumberOfTests )
    {
        this.Tracking       = Tracking;
        this._NUMBEROFITEMS = NumberOfItems;
        this._NUMBEROFTESTS = NumberOfTests;
    }
 
    bool Tracking;
    int _NUMBEROFITEMS;
    int _NUMBEROFTESTS;
 
    public void PerformTest( out DateTime StartedAt, out DateTime EndedAt )        
    {
        using ( ParentChildEntities ctx = new ParentChildEntities() )
        {
            var children = ctx.Child.ToList();
            var parents  = ctx.Parent.ToList();
        }
 
        StartedAt = DateTime.Now;
 
        for ( int i = 0; i < _NUMBEROFTESTS; i++ )
        {
            StringBuilder sb = new StringBuilder();
            using ( ParentChildEntities ctx = new ParentChildEntities() )
            {
                if ( !this.Tracking )
                    ctx.Child.MergeOption = System.Data.Objects.MergeOption.NoTracking;
 
                foreach ( var child in ctx.Child.Take( _NUMBEROFITEMS ) )
                    sb.Append( child.ChildName );
            }
            sb = null;
        }
 
        EndedAt = DateTime.Now;
    }
 
    public string Category
    {
        get { return string.Format( "EF ModelFirst {0}", Tracking ? "Tracking" : "NoTracking" ); }
    }
 
    public string Name
    {
        get { return string.Format( "SELECT TOP {0} - {1} TIMES", _NUMBEROFITEMS, _NUMBEROFTESTS ); }
    }
}

What is interesting is that this time I consider tracking as an important factor and I make sure that it can be turned off. Turining tracking off has some serious consequences for change tracking but since we only select the data, the only observable difference would be that turning tracking off will make the execution much faster.

I also make sure that views are pregenerated (by forcing the pregeneration with the EF Power Tools).

EF5 Code First

For Code First approach, we create entities and the dbcontext manually

public partial class Child
{
    public int ID { get; set; }
    public int ID_PARENT { get; set; }
    public string ChildName { get; set; }
    public virtual Parent Parent { get; set; }
}
 
public partial class Parent
{
    public Parent()
    {
        //this.Children = new List<Child>();
    }
 
    public int ID { get; set; }
    public string ParentName { get; set; }
    public virtual ICollection<Child> Children { get; set; }
}
 
public partial class ParentChildContext : DbContext
{
    static ParentChildContext()
    {
        Database.SetInitializer<ParentChildContext>(null);
    }
 
    public ParentChildContext()
        : base( "Name=ParentChildConnectionString" )
    {
        this.Configuration.AutoDetectChangesEnabled = false;
    }
 
    public DbSet<Child> Children 
    { 
        get; 
        set; 
    }
 
    public DbSet<Parent> Parents 
    { 
        get; 
        set; 
    }
 
    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Child>()
            .ToTable("Child");
 
        // Relationships
        modelBuilder.Entity<Child>()
            .HasRequired( t => t.Parent )
            .WithMany( t => t.Children )
            .HasForeignKey( d => d.ID_PARENT );
 
        modelBuilder.Entity<Parent>()
            .ToTable( "Parent" );
    }
}

The test now becomes

public class PerformanceEFCodeFirstSelect : IPerformanceTest
 {
     public PerformanceEFCodeFirstSelect( bool Tracking, int NumberOfItems, int NumberOfTests )
     {
         this.Tracking       = Tracking;
         this._NUMBEROFITEMS = NumberOfItems;
         this._NUMBEROFTESTS = NumberOfTests;
     }
 
     bool Tracking;
     int _NUMBEROFITEMS;
     int _NUMBEROFTESTS;
 
     public void PerformTest( out DateTime StartedAt, out DateTime EndedAt )
     {
         #region warmup
         using ( ParentChildContext ctx = new ParentChildContext() )
         {
             var parents = ctx.Parents.ToList();
             var children = ctx.Children.ToList();
         }
         #endregion
 
         StartedAt = DateTime.Now;
 
         for ( int i = 0; i < _NUMBEROFTESTS; i++ )
         {
             StringBuilder sb = new StringBuilder();
             using ( ParentChildContext ctx = new ParentChildContext() )
             {
                 if ( Tracking )
                     foreach ( var child in ctx.Children.Take( _NUMBEROFITEMS ) )
                         sb.Append( child.ChildName );
                 else
                     foreach ( var child in ctx.Children.AsNoTracking().Take( _NUMBEROFITEMS ) )
                         sb.Append( child.ChildName );
             }
             sb = null;
         }
 
         EndedAt = DateTime.Now;
 
     }
 
     public string Category
     {
         get { return string.Format( "EF CodeFirst {0}", Tracking ? "Tracking" : "NoTracking" ); }
     }
 
     public string Name
     {
         get { return string.Format( "SELECT TOP {0} - {1} TIMES", _NUMBEROFITEMS, _NUMBEROFTESTS ); }
     }
 }

Note that in this case there is no easy way of turing tracking off on the db context, instead, an extension AsNoTracking is used.

nHibernate

For nHibernate, model has to be created manually

public class Child
{
    public virtual int ID { get; set; }
    public virtual int ID_PARENT { get; set; }
    public virtual string ChildName { get; set; }
    public virtual Parent Parent { get; set; }
}
 
public class Parent
{
    public Parent()
    {
    }
 
    public virtual int ID { get; set; }
    public virtual string ParentName { get; set; }
    public virtual IList<Child> Children { get; set; }
}

as well as mappings

<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping 
    xmlns="urn:nhibernate-mapping-2.2" 
    assembly="Performance.NH" 
    namespace="Performance.NH.Model"    >
    <class name="Child" 
           optimistic-lock="none" dynamic-update="true">
        <id name="ID">
            <generator class="native" />
        </id>
        <property name="ChildName" />
        <many-to-one name="Parent" column="ID_PARENT" />
    </class>
</hibernate-mapping>
 
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-mapping xmlns="urn:nhibernate-mapping-2.2" 
    assembly="Performance.NH" namespace="Performance.NH.Model">
    <class name="Parent" table="`Parent`" 
           optimistic-lock="none" dynamic-update="true">
        <id name="ID">
            <generator class="native" />
        </id>
        <property name="ParentName" />
        <bag name="Children">
            <key column="ID_PARENT" />
            <one-to-many class="Child"/>
        </bag>
    </class>
</hibernate-mapping>
 
<?xml version="1.0" encoding="utf-8" ?>
<hibernate-configuration xmlns="urn:nhibernate-configuration-2.2">
    <session-factory>
        <property name="connection.provider">NHibernate.Connection.DriverConnectionProvider</property>
        <property name="dialect">NHibernate.Dialect.MsSql2008Dialect</property>
        <property name="connection.driver_class">NHibernate.Driver.SqlClientDriver</property>
        <property name="connection.connection_string_name">ParentChildConnectionString</property>
    </session-factory>
</hibernate-configuration>

and the test is

public class PerformanceNHSelect : IPerformanceTest
{
    public PerformanceNHSelect( bool StateLess, int NumberOfItems, int NumberOfTests )
    {
        this._StateLess = StateLess;
        this._NUMBEROFITEMS = NumberOfItems;
        this._NUMBEROFTESTS = NumberOfTests;
    }
 
    bool _StateLess;
    int _NUMBEROFITEMS;
    int _NUMBEROFTESTS;
 
    public void PerformTest( out DateTime StartedAt, out DateTime EndedAt )        
    {
        Configuration cfg = new Configuration();
 
        cfg.Configure( typeof( PerformanceNHSelect ).Assembly, 
           "Performance.NH.Model.hibernate.cfg.xml" );
        cfg.AddAssembly( typeof( PerformanceNHSelect ).Assembly );
 
        var sessionFactory = cfg.BuildSessionFactory();
 
        #region warmup
        StringBuilder sbw = new StringBuilder();
        using ( var session = sessionFactory.OpenSession() )
        {
            session.DefaultReadOnly = true;
            session.FlushMode = FlushMode.Never;
            foreach ( var child in session.Query<Child>().Take( _NUMBEROFITEMS ) )
                sbw.Append( child.ChildName );
        }
        #endregion
 
        StartedAt = DateTime.Now;
 
        for ( int i = 0; i < _NUMBEROFTESTS; i++ )
        {
            StringBuilder sb = new StringBuilder();
 
            if ( _StateLess )
            {
                using ( var session = sessionFactory.OpenStatelessSession() )
                {
                    using ( var tran = session.BeginTransaction() )
                    {
                        foreach ( var child in session.Query<Child>().Take( _NUMBEROFITEMS ) )
                            sb.Append( child.ChildName );
                    }
                }
            }
            else
            {
                using ( var session = sessionFactory.OpenSession() )
                {
                    using ( var tran = session.BeginTransaction() )
                    {
                        foreach ( var child in session.Query<Child>().Take( _NUMBEROFITEMS ) )
                            sb.Append( child.ChildName );
                    }
                }
            }
 
            sb = null;
        }
 
        EndedAt = DateTime.Now;
    }
 
    public string Category
    {
        get { return string.Format( "nHibernate {0}", _StateLess ? "StateLess" : "StateFull" ); }
    }
 
    public string Name
    {
        get { return string.Format( "SELECT TOP {0} - {1} TIMES", _NUMBEROFITEMS, _NUMBEROFTESTS ); }
    }
}

What is interesting here is that I am trying to compare the statefull and stateless session with the hope that stateless sessions would somehow correspond to EF’s “no tracking” in a sense that a stateless session should fetch rows from the database faster.

In the next entry we will analyze results.