Message Brokers are solutions that help you address various challenges, such as connecting multiple infrastructures and broadcasting messages.

Typically, when developing frontend applications, we send HTTP requests to servers and receive responses. However, backend services also need to communicate with each other.

Managing connections between your applications can be challenging in terms of integrity and availability. For example, what if a message you sent to another endpoint takes time to respond because it needs to perform computations or ask other endpoints for results?

When developing web/mobile/backend applications, data from requests/response can be lost due to various reasons. It’s crucial to focus on the core functionality of your application, keeping it readable and upgradable without compromising its value to clients.

Taking extra care of messages can make your application more complex and bloated, leading to poor designs. I learned this lesson when I was developing a mobile application without any experience. I ended up creating numerous fallbacks that made the app unresponsive and unintuitive for users.

In your application, as everyone else, you wish to take care of the important stuff which is making your application work fluidly, be readable in terms of code, smoothly up-gradable by adding new components and most important of all have some value so it could give your clients the best experience as possible.

In this article, I want to share my experience with message brokers and how they transformed my approach to various solutions. I’ll mainly focus on RabbitMQ as an example of a message broker.

Hello Message Broker

A message broker is a simple solution which provide a robust and scalable producer-consumer architecture to many applications.

Before we dive into deep explanations I wish that we will firstly go over some basic concepts:

• Producer — Is an endpoint which sends any kind of data, that is stored inside the message broker to distribute.
• Consumer — Is an endpoint which asks from the message broker for data(messages).
• Queue — Is a data type which the message broker use to store messages inside, with the logic of FIFO(First in First out).
• Exchanger — A logical configuration or even entity, on top of the queues, which tells the message broker to create some sort of a group, which a consumers/producers can write or listen to, to send/receive messages.

Today there are many message brokers but the famous ones are RabbitMQ, Kafka(By Apache), Redis, SQS(By Amazon).

A message broker is a simple server which you communicate with the AMQP.
What we do with the AMQP server is simply create queues which will store messages on the server itself, each queue will be responsible to distribute these messages in some sort of behavior to single or multiple clients.

Each queue allows us to seperate application messages logic, so when an operation needs to get done, it gets enqueued to a queue which will allow consumers get those message by the same sequence which they got inside that queue.

The server stores the messages on the hard disk or in memory, so if the client which the message was ment to is not online, the message broker can send that message another possible client to process it, unless it is a direct message which is ment to be sent to a specific client by some sort of logic.

Persistence and Durability

One beautiful part of message broker is that they know to make sure that the data they received will be delivered no matter what.

This gives us a very huge relief because we know for sure, that any kind of request of our app that it needs to do, will be provided, ain’t that a charm? ☺

The persistence comes from a acknowledgement mechanism that each message broker provides.

For example, we can produce a message to some client, and the message broker will store it in his memory (if said so). When the consumer client will consume the message, even then(again if configured to do so) the message broker will still keep it in his memory.

When do the message broker will delete it you ask?
The message broker will wait for an acknowledge from the consuming client.
This can assure that the consuming client will send his acknowledge to delete the message, only when he truly finished his work with it.
Today, you can also do auto acknowledgement for that message, so you don’t need to send your acknowledge back to the message broker 🙂

The memory time gap of pain

In order to understand the following last important subject on the matter we need to do a short overview about Operating systems.

In operating systems, every thing is being loaded to main memory which is the name of RAM, from the hard disk.
Off course not every thing will be loaded but there are mechanisms to handle all of that operation.

We also have cache memory which is the fastest memory to access data. Each memory piece inside cache was recently fetched from memory, so we would have faster access to it, if we do access it quite frequently.

The cache memory is separated to small memory pieces which each one can be used for individual core on the CPU.
You might be wondering… how do the CPU sync these small parts so everyone will be in sync with the data present inside the cache memory in terms of all cores?

That’s a great question but I don’t want to “grow” any more out side of this article subject but I do hope you get the point 🙂

For the adventurers and tech lovers, read about cache coherence which is responsible to sync the cache memory, it’s an interesting subject.
MESI is an implementation of it.

Now that we understand how memory synchronization between multiple memory units more or so, we can continue with our main story.
In each message the message broker receives, he do not call the fsync system call, which is responsible to write data which is present in main memory and cache to the disk.

So wait… what do happen when my message broker has like 100 messages which he hasn’t written to the disk, and unfortunately he shuts down or receive some kind of exception?!
Well it’s kind of a bummer to tell you this but these messages would be lost.
On this specific subject I can tell you that RabbitMQ doesn’t issue as of today fsync on each arriving message, so if you wanna know about any other message broker, search it on their website or any other place on the internet, I’m sure someone have tried to find that out already 🙂

Architectural benefits

Message brokers can sometimes be used to connect between two micro services big applications, and you might be thinking why should I do that?
Why don’t we make an API call between them?

Great question, so lets imagine two big applications which each one is very different from one another, or even better lets take a obsolete micro services application, and say we wish to have some kind of events from it.

What your first response would be in terms of connecting them?
The first response might be creating an API for them, and that’s a good call but I have an improvement for you.

Instead of relying on developing on a full scale API, and update it periodically due to need of changes, why not take an independent entity which doesn’t know who is connected to who and simply pass data?

What it will allow us is that no matter the amount of services instances there are, our Message Broker will handle the messages from end to end for us, and each service could produce data to anyone regardless of concerns of knowing how to reach them for example, and there are many more.

Of course, there’s an elegant solution to every challenge, so I’m not trying to say you should always go with Message Broker for cases like these or similar, but you should consider them in your architecture design.

Conclusion

We’ve seen what is a message broker as of day and why we use them, because they are simply great!

Off course, you do need to make your homework when thinking on merging a message broker in your architecture, because each solution has it’s own best approach to solve it.

I had a chat with a friend of mine on the subject, which people has the thrive to implement an entire backend application, which all services communicates using the message broker, and he’ s the single point of failure of the application, but we can actually be done is use other applications like api gateways which as well can route our requests like message broker and even simplifies the code we write, but will not make our message persistence on the other hand.

I’m not saying the solution from before is bad or anything, I just think you should think on the big picture, depending on the tools you’re using what is the best tool to add to our toolbox which will give us the best effort out of it all.

As always, I do hope you enjoyed reading this piece, and if you have any suggestion to improve this article so everyone else will enjoy it and have a better experience, I’ll be very glad to hear 🙂

Thank you again and have a great day!