Sunday, April 5, 2015

Web Framework Comparison Benchmarks

Introduction

I am a final year student in Software Development in the Institute of Technology Carlow, Ireland. For my final year project I decided to try and provide a way for developers to choose a web framework in an unbiased way.

In a recap / update from the last post, I have developed two applications using three different web frameworks; Chicago Boss, Ruby on Rails and Flask. The first is a blog application and the second is a simple CRUD application built around the MySQL Sakila database.

The aim of this project is to provide the testing mechanisms I used to developers, so they can then build the same applications in their chosen web framework and compare the results of the tests they run against my benchmarks provided below.

Application Tutorials


I have developed tutorials for building both applications using the three different web frameworks.

Blog application


Chicago Boss Blog: https://github.com/archerydwd/cb_blog
Ruby on Rails Blog: https://github.com/archerydwd/ror_blog
Flask Blog: https://github.com/archerydwd/flask_blog

Sakila application


Chicago Boss Sakila: https://github.com/archerydwd/cb_sakila
Ruby on Rails Sakila: https://github.com/archerydwd/ror_sakila
Flask Sakila: https://github.com/archerydwd/flask_sakila

I tested these six applications in their development and production modes. I also tested Ruby on Rails and Flask using the production server called Apache. Chicago Boss uses a web server called cowboy instead of apache. As Chicago Boss is built using Erlang OTP, it would be counterproductive to use  the apache web server.

Apache on Ubuntu Tutorial


Apache, Ruby on Rails & Passenger: https://github.com/archerydwd/Apache-Rails-Passenger-Ubuntu
Apache & Flask: https://github.com/archerydwd/Apache-Flask-Ubuntu

Testing Tutorials


For Testing I used Gatling (http://gatling.io/#/) and double checked the results by using Tsung (http://tsung.erlang-projects.org).

Gatling Testing: https://github.com/archerydwd/gatling-tests

For the blog application tests, I used 1000, 50000, 100000 and 300000 users making two get requests each and for the Sakila application I used 100 and 1000 users making five get requests each. When testing I completed three iterations of each test and derived the below charts.

Benchmarks


Blog application - development mode


When running the blog applications in development mode, I got the following results. Please note the lower the number the better. Flask was the fastest and only started having failures at 300000 users. Chicago Boss was the second fastest and started having failures at 100000 users and Ruby on Rails was the slowest and started having failures at 50000 users.

1000 users:

50000 users:
 100000 users:

300000 users:

Blog application - production mode


When running the blog applications in development mode, I got the following results. Please note the lower the number the better. Chicago Boss and Flask were very close in this set of iterations with Chicago Boss coming in a little bit faster and a lower fail rate at 300000 users than Flask's fail rate. Ruby on Rails was again the slowest with failures starting to occur at 50000 users.

1000 users:
50000 users:
 100000 users:
 300000 users:

Blog application - Production server mode


When running the blog applications with a production server (Apache and cowboy for Chicago Boss), I got the following results. Please note the lower the number the better. The fastest was Chicago Boss with no failures, second was Ruby on Rails with failures starting at 300000 users and the slowest was Flask with failures happening at 50000 users.

1000 users:
 50000 users:
 100000 users:
 300000 users:

Sakila application - development mode


When running the sakila applications in development mode, I got the following results. Please note the lower the number the better. Flask performed faster with 100 users and then Chicago Boss was faster with 1000 users although Flask had a lower failure percentage with 1000 users. I would have to give this round to Flask, closely followed by Chicago Boss which started getting failures at 1000 users. Ruby on Rails was the slowest in this round and started getting failures at 100 users.

100 users:
 1000 users:

Sakila application - production mode


When running the sakila applications in production mode, I got the following results. Please note the lower the number the better. Flask was the fastest with 100 users but as you can see below it started to suffer when higher numbers (1000) where introduced and got a higher failure rate. Chicago Boss was fastest with 1000 users and only had a small failure percentage at 1000 users. Ruby on Rails was the slowest this round and had a very high failure percentage starting at 100 users. 

100 users:
 1000 users:

Sakila application - Production server mode


When running the sakila applications with a production server (Apache and cowboy for Chicago Boss), I got the following results. Please note the lower the number the better. Flask was the fastest this round again getting a failure percentage starting at 1000 users. Chicago Boss performed better with 100 users than Ruby on Rails and only got a failure percentage starting at 1000 users, but Chicago Boss was slower than Ruby on Rails at 1000 users. Ruby on Rails had a failure percentage starting at 100 users. 

100 users:
 1000 users:



Thanks for reading and hopefully the above benchmarks will help when making a decision on your web framework.

Darren.



Saturday, February 7, 2015

Web Framework Comparison Benchmark Specification

Project Specification

Hi, I am a final year student in Software Development in the Institute of Technology Carlow, Ireland. For my final year project I decided to try and provide a way for people to choose their web framework for their application in an unbiased way.
I decided to start with two web frameworks; Chicago Boss and Ruby on Rails. Chicago Boss is a web framework built on top of the Erlang programming language and Ruby on Rails is a web framework built on top of the Ruby programming language.

The first step will be building the same series of applications on both web frameworks. Next, I will be comparing the performance of these applications on both web frameworks with testing tools, such as JMeter and Tsung to get a benchmark for both web frameworks.

The series of applications that I have developed are available on GitHub along with the procedure I used to build them. The first is a simple blog application built in both Chicago Boss and Ruby on Rails.
Chicago Boss version:
Ruby on Rails version:
The second is an application that I built around an existing MySQL database including some large tables.
Chicago Boss version:
Ruby on Rails version:

So far I have built the applications and currently I'm in the process of learning Tsung and JMeter (or any other tools that I come across of any use). Once I have learnt the testing tools, I will provide the tests that I have run on the above applications along with a tutorial for using them. If all goes well I will expand the web frameworks I am using to include some more such as Flask.

To use the benchmarks that I will be providing you should develop the series of applications that I have created in your chosen web framework. Implement my tests and compare your results with my benchmarks. 
This should provide you with an unbiased way to choose a web framework.

Thanks for reading!

Darren Daly.