Sunday, April 5, 2015

Web Framework Comparison Benchmarks

Introduction

I am a final year student in Software Development in the Institute of Technology Carlow, Ireland. For my final year project I decided to try and provide a way for developers to choose a web framework in an unbiased way.

In a recap / update from the last post, I have developed two applications using three different web frameworks; Chicago Boss, Ruby on Rails and Flask. The first is a blog application and the second is a simple CRUD application built around the MySQL Sakila database.

The aim of this project is to provide the testing mechanisms I used to developers, so they can then build the same applications in their chosen web framework and compare the results of the tests they run against my benchmarks provided below.

Application Tutorials


I have developed tutorials for building both applications using the three different web frameworks.

Blog application


Chicago Boss Blog: https://github.com/archerydwd/cb_blog
Ruby on Rails Blog: https://github.com/archerydwd/ror_blog
Flask Blog: https://github.com/archerydwd/flask_blog

Sakila application


Chicago Boss Sakila: https://github.com/archerydwd/cb_sakila
Ruby on Rails Sakila: https://github.com/archerydwd/ror_sakila
Flask Sakila: https://github.com/archerydwd/flask_sakila

I tested these six applications in their development and production modes. I also tested Ruby on Rails and Flask using the production server called Apache. Chicago Boss uses a web server called cowboy instead of apache. As Chicago Boss is built using Erlang OTP, it would be counterproductive to use  the apache web server.

Apache on Ubuntu Tutorial


Apache, Ruby on Rails & Passenger: https://github.com/archerydwd/Apache-Rails-Passenger-Ubuntu
Apache & Flask: https://github.com/archerydwd/Apache-Flask-Ubuntu

Testing Tutorials


For Testing I used Gatling (http://gatling.io/#/) and double checked the results by using Tsung (http://tsung.erlang-projects.org).

Gatling Testing: https://github.com/archerydwd/gatling-tests

For the blog application tests, I used 1000, 50000, 100000 and 300000 users making two get requests each and for the Sakila application I used 100 and 1000 users making five get requests each. When testing I completed three iterations of each test and derived the below charts.

Benchmarks


Blog application - development mode


When running the blog applications in development mode, I got the following results. Please note the lower the number the better. Flask was the fastest and only started having failures at 300000 users. Chicago Boss was the second fastest and started having failures at 100000 users and Ruby on Rails was the slowest and started having failures at 50000 users.

1000 users:

50000 users:
 100000 users:

300000 users:

Blog application - production mode


When running the blog applications in development mode, I got the following results. Please note the lower the number the better. Chicago Boss and Flask were very close in this set of iterations with Chicago Boss coming in a little bit faster and a lower fail rate at 300000 users than Flask's fail rate. Ruby on Rails was again the slowest with failures starting to occur at 50000 users.

1000 users:
50000 users:
 100000 users:
 300000 users:

Blog application - Production server mode


When running the blog applications with a production server (Apache and cowboy for Chicago Boss), I got the following results. Please note the lower the number the better. The fastest was Chicago Boss with no failures, second was Ruby on Rails with failures starting at 300000 users and the slowest was Flask with failures happening at 50000 users.

1000 users:
 50000 users:
 100000 users:
 300000 users:

Sakila application - development mode


When running the sakila applications in development mode, I got the following results. Please note the lower the number the better. Flask performed faster with 100 users and then Chicago Boss was faster with 1000 users although Flask had a lower failure percentage with 1000 users. I would have to give this round to Flask, closely followed by Chicago Boss which started getting failures at 1000 users. Ruby on Rails was the slowest in this round and started getting failures at 100 users.

100 users:
 1000 users:

Sakila application - production mode


When running the sakila applications in production mode, I got the following results. Please note the lower the number the better. Flask was the fastest with 100 users but as you can see below it started to suffer when higher numbers (1000) where introduced and got a higher failure rate. Chicago Boss was fastest with 1000 users and only had a small failure percentage at 1000 users. Ruby on Rails was the slowest this round and had a very high failure percentage starting at 100 users. 

100 users:
 1000 users:

Sakila application - Production server mode


When running the sakila applications with a production server (Apache and cowboy for Chicago Boss), I got the following results. Please note the lower the number the better. Flask was the fastest this round again getting a failure percentage starting at 1000 users. Chicago Boss performed better with 100 users than Ruby on Rails and only got a failure percentage starting at 1000 users, but Chicago Boss was slower than Ruby on Rails at 1000 users. Ruby on Rails had a failure percentage starting at 100 users. 

100 users:
 1000 users:



Thanks for reading and hopefully the above benchmarks will help when making a decision on your web framework.

Darren.



2 comments: