Introduction

Recently, I have implemented a new RabbitMQ infrastructure, for the company I regularly work,which is supposed to serve several different customers and services spacing from complex applications to CMS services to long lasting transaction systems.

What is extremely interesting of this exercise is that the request to have this new service was driven by developers, accepted by operations and commonly implemented. All the in all the solution is tailored for developers’ needs while it considers normal operational constraints of a production service (i.e. backup, monitoring etc).

 

What is RabbitMQ?

For those of you that don’t know the service, RabbitMQ is a free/open source implementation of the AMQP queue protocol.

The system is written in Erlang and runs on (in my specific case) a Redhat 7 Virtual Machine.

Out of curiosity, tales say that someone is running RabbitMQ on Microsoft platform somewhere in a dark wood. I usually don’t follow these sad stories and sincerely I don’t know how good it is on the windows OS. What I know is that RabbitMQ rocks on Linux systems and Linux is the OS that is meant to be used for production usage.

 

Tell me more about it

I suppose anyone into the IT business knows what is a messaging queue.  I am sure that a quick refresh of our knowledge will know hurt anyone btw.

A message queue is a system to let different applications or applications’ components to communicate in an asynchronous manner. All in all the world around us is asynchronous by definition and it is of paramount importance to be able to implement asynchronous solutions.

In this way, applications can easily perform tasks such as:

• scale depending on the need
• Implement the so-called “in flight transactions”
• Implement heterogeneous solutions taking the best of the breed the market can offer nowadays
• Offload frontend systems
• Implement microservices components specialized in specific tasks. Kill the monolith!
• Put what your imagination suggests!

A message queue has typically 3 actors involved:

• The provider, the guy/item which feeds the queue with messages
• The queue, the container of the messages
• The consumer, the actor(s) that takes the message and performs a given action

 

Note: Real, modern IT systems usually have more actors than this involved but the basic logic remains.

 

A practical example

Let me give you an example of message queue usage.

You are on a social platform and you want to upload your latest profile picture (the picture when you were on holiday last summer!)

What you do is usually to upload the picture file straight away from you camera. Modern cameras have billions of pixels.

The system accepts the picture but it needs to post-process it. Post Processing is typically an heavy task and might include:

• Resizing of the picture to a manageable format (DO I NEED A 20000X300000 resolution?)
• Creation of a thumbnail of your picture
• Duplication in different resolutions of the same picture for different devices (HD, non HD, Mobile devices, Search results list etc.)
• Create different versions of the picture depending on the theme of your system (black and white? Sepia? Guess it)
• Any other business (use your imagination 🙂 )

Now, we have two different way to accomplish it:

– The “1970 IT way”, you let the user wait for 20 minutes hoping that the last step does fail (only god knows why) and so the user has to start from scratch again
– The “nowadays IT way”, you queue n-messages for the postprocessing units. Each message will be handled by a different specialized  bot/algorithm which takes care of the process. If any of the steps fail, can be processed autonomously.

Which solution is the best? The 1970 one or the nowadays one?

 

Tell me more about its implementation

The implementation is rather easy. Our design of RabbitMQ was done bearing in mind the need of multi tenancy and service sharing in order to optimize costs and reduce the number of infrastructure components to be maintained. Having said this,  RabbitMQ has been configured in order to support multiple virtual hosts where each virtual hosts represent a logical split between customers/tenants (the server takes care of vhosts isolation).

The developer/application owners get an account that can spawn different entities into the virtual hosts such as queues, exchanges and messages.

Queues inside a given virtual host are controlled by the application, we (application hosting) don’t control the nature of the content (like we don’t enter the logic of other services) but we take care of the needed to guarantee the service to be fully operational.

For us a queue is a generic entity, a container of information. For you the queue is the aggregation of your application’s messages and all the messages together implement your business.

 

Why do we need it?

The question is not “why do we need it?”, the real question is “why not?”.  RabbitMQ makes the developer’s’ life easier, allows services decoupling, makes scalability easier and so on. The cost is ridiculously low in terms of hosting and the software comes for free even though offers a production level robustness.

Moreover, if we do not provide the service as a commodity hosting service, developers will do it anyway, in their own, unprotected/unruled way. This is already happening on tens of Feeder containers. Developers need it, it our duty to assist the developers in the most efficient and cost effective way possible.

 

Companies are moving to the cloud, why now?

When we will migrate to the cloud, depending on the provider, we will have an equivalent SAAS solution or we will have the ability to implement RabbitMQ on a virtual machine in our VPC. Think about AWS, the SQS system implements exactly the very same concepts we have been discussing so far and it does it in an decent way. It is not an AMQP compliant system but all in all, millions of developers live without it 🙂 We are not a special case (actually SQS does much more than this). If we want to stick with the standard AMQP, we can still implement rabbitMQ in the very same way we implemented it now. Nothing changes. Cloud does not change the services’ needs. It does change the way you satisfy the needs.

 

 

Conclusions

I strongly believe that anyway, nowadays, having the need to decouple service’s functions, would need to have a look to RabbitMQ.

The service is rock solid, easy to be installed and has a great maturity level.

For sure there are other competitors out there which do similar things. In my humble experience anyway, RabbitMQ looks to be easy and straight forward and can represent an excellent deal for your needs.