Disclaimer

These statistics were obtained over the internet using a 'single shot' of Patrick Meenan's most excellent Web Page Test service, using a publically available client in Sydney, using the 'Cable' performance model. This really isn't good enough to be statistically valid as there are too many possible interferences to provide clean, repeatable data. In addition, it's looking at the home page, whereas an in-depth analysis would also need to address performance of category and product pages as a bare minimum. However, these tests were carried out automatically, and my hand at no time tainted the data!

Even taking this into account, I feel that there is enough information here to highlight, and start some talking points about the rather unique problems of running an eCommerce server in the Antipodes

Intro

Using commercially available information, I generated a list of approximately 4,000 websites that are based in Australia and, over the period of a month or so, monitored the performance of the sites. In this post, I'm concentrating on the server side performance, as indicated by the 'Time to First Byte'. Whilst some servers do game this, most don't so it's a pretty good indicator of the amount of time that the web server takes to assemble and start delivering the html skeleton that makes up the web page. From this, I've looked at a few different statistics, and split them across web server software.

Results

I've included the most popular web servers (as found in my results), plus one specialist Magento provider


+---------------------+--------+------+-------+---------+--------------------+
| WebServer           | Sample | Min  | Max   | Average | Standard Deviation |
+---------------------+--------+------+-------+---------+--------------------+
| Apache              |   2110 | 0.11 | 32.34 |    2.05 |               2.25 |
| CloudFlare          |    230 | 0.16 | 21.61 |    2.13 |               2.76 |
| IIS                 |     39 | 0.27 |  8.28 |    1.85 |               1.65 |
| LiteSpeed           |    403 | 0.11 | 48.12 |    2.45 |               3.45 |
| MageStack-MagentoOS |      8 | 1.53 |  3.41 |    2.52 |               0.69 |
| Nginx               |    865 | 0.11 | 28.71 |    2.34 |               2.38 |
| Sucuri/Cloudproxy   |     28 | 0.33 | 11.27 |    2.40 |               2.21 |
+---------------------+--------+------+-------+---------+--------------------+


The interesting thing here is that they're not statistically different!

Next up, I quantified the TTFB as follows:

• < 200ms = Excellent
• 201-700ms = Good
• 701-1100ms = average
• 1101-1500ms = Poor
• and > 1500ms = Bad.

This could be conceived as being pretty harsh, but we are talking eCommerce here, and this is the front page:


+---------------------+-----------+------+---------+------+-----+
| WebServer           | Excellent | Good | Average | Poor | Bad |
+---------------------+-----------+------+---------+------+-----+
| Apache              |        31 |  402 |     353 |  293 | 995 |
| CloudFlare          |         2 |   54 |      38 |   34 | 101 |
| IIS                 |         0 |    8 |       6 |    6 |  18 |
| LiteSpeed           |         6 |   48 |      73 |   60 | 211 |
| MageStack-MagentoOS |         0 |    0 |       0 |    0 |   8 |
| Nginx               |         4 |  116 |     107 |  135 | 483 |
| Sucuri/Cloudproxy   |         0 |    4 |       3 |    5 |  15 |
+---------------------+-----------+------+---------+------+-----+


Maybe easier to read as percentages


+---------------------+-----------+------+---------+------+------+
| WebServer           | Excellent | Good | Average | Poor |  Bad |
+---------------------+-----------+------+---------+------+------+
| Apache              |       1.4 | 19.0 |    16.7 | 13.8 | 47.1 |
| CloudFlare          |       0.8 | 23.4 |    16.5 | 14.7 | 43.9 |
| IIS                 |         0 | 20.5 |    15.3 | 15.3 | 46.1 |
| LiteSpeed           |       1.4 | 11.9 |    18.1 | 14.8 | 52.3 |
| MageStack-MagentoOS |         0 |    0 |       0 |    0 |  100 |
| Nginx               |       0.4 | 13.4 |    12.3 | 15.6 | 55.8 |
| Sucuri/Cloudproxy   |         0 | 14.2 |    10.7 | 17.8 | 53.5 |
+---------------------+-----------+------+---------+------+------+


So what do we make of this?

Disclaimer repeated: this really isn't a statistically valid set of results, but the results are still food for thought.

My primary finding is that it doesn't really matter what technology stack you use, they all seem to perform just as well. In reality, it's showing that you can still poorly configure and incorrectly size your server, no matter what stack you choose.

(Interestingly though, the one specialist Magento provider performed worst... at a wild guess I'd say it's probably due to the distance from the target market as the company in question is UK based)

So the real finding of this post is that there is a step between developers declaring a website complete, and it being production ready. Any effort spent in addressing the performance of your website will be recouped in better user engagement, leading to an increase in sales.

As a final point, only Apache and Nginx are supported Web Servers for Magento ( 1 and 2 ). Whilst it is a moot point as to the value of this in an Open Source environment, it will place more of a burden on any provider not using either of these Web Servers, and should be covered during your due diligence.