Wednesday, May 15, 2013

Yet Another ORM Micro Benchmark, part 3/3, results

This is the last post of the series. We introduced a micro orm benchmark with few contestants and we implemented all required tests. In this post we analyze results.

Complete set of results

Let us start from a complete set of results (all numbers are miliseconds so for example 531 means 0.531 sec and 6502 means 6.502 sec):

category

name

totalms

ADO.NET

TOP 1

144.0144

Linq2Sql

TOP 1

526.0526

EF ModelFirst NoTracking

TOP 1

531.0531

EF ModelFirst Tracking

TOP 1

551.0551

EF CodeFirst NoTracking

TOP 1

1063.1063

EF CodeFirst Tracking

TOP 1

1069.1069

nHibernate StateLess

TOP 1

654.0654

nHibernate StateFull

TOP 1

658.0658

ADO.NET

TOP 10

111.0111

Linq2Sql

TOP 10

574.0574

EF ModelFirst NoTracking

TOP 10

499.0499

EF ModelFirst Tracking

TOP 10

627.0627

EF CodeFirst NoTracking

TOP 10

1046.1046

EF CodeFirst Tracking

TOP 10

1247.1247

nHibernate StateLess

TOP 10

1194.1194

nHibernate StateFull

TOP 10

1305.1305

ADO.NET

TOP 100

270.027

Linq2Sql

TOP 100

710.071

EF ModelFirst NoTracking

TOP 100

675.0675

EF ModelFirst Tracking

TOP 100

1596.1596

EF CodeFirst NoTracking

TOP 100

1210.121

EF CodeFirst Tracking

TOP 100

2836.2836

nHibernate StateLess

TOP 100

5699.5699

nHibernate StateFull

TOP 100

6502.6502

ADO.NET

TOP 1000

1327.1327

Linq2Sql

TOP 1000

2064.2064

EF ModelFirst NoTracking

TOP 1000

1885.1885

EF ModelFirst Tracking

TOP 1000

10700.0699

EF CodeFirst NoTracking

TOP 1000

2365.2365

EF CodeFirst Tracking

TOP 1000

18658.8657

nHibernate StateLess

TOP 1000

51017.1012

nHibernate StateFull

TOP 1000

58400.8395

ADO.NET

TOP 10000

10878.0877

Linq2Sql

TOP 10000

19527.9526

EF ModelFirst NoTracking

TOP 10000

13496.3495

EF ModelFirst Tracking

TOP 10000

122345.2333

EF CodeFirst NoTracking

TOP 10000

13797.3796

EF CodeFirst Tracking

TOP 10000

210554.0533

nHibernate StateLess

TOP 10000

546423.6369

nHibernate StateFull

TOP 10000

564736.468

The most interesting comparison to me was to see how retrieving data works for different sizes of datasets. Here are results presented on charts:

General observations:

  • bare ADO.NET is of course always the fastest but for the largest dataset, EF with tracking turned off is almost as fast (which is amazing)
  • Linq2Sql is outperformed by EF Model First No Tracking and, for the largest dataset, even EF Code First No Tracking is way faster than Linq2SQL
  • EF Model First is always faster than EF Code First
  • EF with tracking is always much slower than EF without tracking. For 1000 or more records tracking can probably hurt the performance badly if you are only reading the data
  • nHibernate StateLess is always faster than nHibernate StateFull (which is expected)
  • nHibernate is surprisingly slow when larger sets are retrieved (which is rather unexpected and frankly, I wonder if there is a way to speed it up as it looks rather disappointing)

Note that benchmaring ORMs should also involve inserts/updates/deletes in various scenarios. Please consider my experiments as valid only in this specific context (data retrieval).

I would also like to hear a comment or two about this. I am especially interested in the underperformance of nHibernate for large sets of data – I wonder what else could be done to speed it up.

3 comments:

DCM said...

Your experiment, while valid in some contexts, may be a bit wrong in others. Ayende Rahien spoke about micro benchmarks once and I tend to agree on many if his points.
However, you do have a point and it's very good to have hard data like yours to look. Thanks very much :)

Wiktor Zychla said...

@David: I know this Ayende's opinion and still believe that my approach reveals some interesting data. Note that not only I do bare SELECTs but also I "warm-up" every engine by skipping few initial measurements just to minimize the risk of any unwanted penalties coming from things like metadata initializing (which slow down the very first SELECT but do not slow down consecutive operations in the same process). My biggest concern is the unexpected slowness of nHibernate and unfortunately after consulting few fellow developers who use NH daily, I am still unable to correctly diagnose and fix the issue (if this IS an issue).

Anonymous said...

Wiktor, do your tests discount the startup time? Iow, does the test app do some EF call before meeasuring the particular test so that building the edmx or code first model into the DbContext is discounted? Thanks, Dave