Simple Guide to Message Brokers

Estafet consultants occasionally produce short, practical tech notes designed to help the broader software development community and colleagues. 

If you would like to have a more detailed discussion about any of these areas and/or how Estafet can help your organisation and teams with best practices in improving your SDLC, you are very welcome to contact us at enquiries@estafet.com 

Introduction

In today’s fast-changing and dynamic world of IT, distributed computing systems can grow very rapidly and get extremely complex. Clear and secure communication between different components is vital to ensuring that all individual units are working together as part of a complete software solution. Inter-component communication is especially challenging in a microservice environment, as it requires a high degree of coordination and synchronisation across many services. Message brokers can help tackle this challenge by serving as an intermediary between various parts of your software infrastructure.

In this tech note, I will give a short description of what message brokers are, what their purpose is and when they are used. I will also list 4 of the most widely used message brokers today and look at their characteristics and capabilities. By the end of reading this, you will:

  • Learn what message brokers are and why they have an essential role in supporting a microservice architecture
  • Understand how message brokers work and what kind of message distribution models they offer
  • Familiarise yourself with 4 of today’s most popular message brokers and determine which one can fit your current or future needs

What Are Message Brokers and What Problems Do They Solve?

Below, you can see an example of how integration even between a handful of applications has potential to get really difficult to track and maintain:

Communication between 3 source apps and 3 target apps

This is an example of just 3 source apps having to communicate to 3 target apps and we are already seeing a “spider web” of integrations: complexity can only grow exponentially with the increase of components. To make things even more convoluted, each of these apps may be written in a different language and each integration might use a different communication protocol. On top of everything, in order to ensure resiliency if a target system is down, an individual fault tolerance mechanism would need to be implemented for each source app.

Message brokers offer a solution to this problem and make communication between components more manageable by acting as a mediator between applications. Brokers externalise the entire communication mechanism into a separate module which can be individually scaled, if needed. As seen below, there is no longer a dense web of connections between components: instead, they all “talk” to the common message broker, achieving loose coupling:

Using a message broker as an intermediary decouples the source and target systems

Some specific terminology is used when discussing message brokers: a piece of data that needs to be sent between systems is called a message. Source applications that send messages are called producers/publishers and target applications that need to accept these messages are called consumers/subscribers. After receiving a message, the broker adds it to an internal message queue (a FIFO-based data structure) until it’s consumed by the appropriate consumer(s).

Broker capabilities include translating messages between different messaging protocols, as well as validating, storing, routing and delivering these messages to the desired target system.

One of the most valuable mechanisms that message brokers enable is asynchronous messaging: when sending a message, producers are not blocked until they get acknowledgement from a consumer. Instead, producers and consumers can send and receive messages asynchronously with the guarantee that a message will be delivered even in the event of network connectivity issues. Thanks to asynchronous messaging, if a producer keeps publishing while a consumer is down, when it eventually comes up, the consumer will be able to catch up with all the messages it may have missed.

Message brokers can employ the following 2 basic message distribution models:

  • Point-to-point messaging model: used when producers have a one-to-one relationship with consumers. This model is utilised when a message needs to be processed only once by only one recipient. Financial transaction processing is a frequent use-case for the point-to-point messaging model.
  • Publish/subscribe messaging model: used when producers have a one-to-many relationship with consumers. One or more producers (or “publishers”) can publish messages on a specific topic and interested consumers (or “subscribers”) can subscribe to that topic in order to receive messages. Use-cases for the pub/sub messaging model include sending and processing event notifications, logging to multiple systems, etc.

Having established what message brokers are and what problems they solve, we will now explore a list of four popular message brokers.

Four Popular Message Brokers

Apache Kafka

Apache Kafka is an open source fault-tolerant and robust distributed event streaming platform originally developed by LinkedIn and publically released in 2011. According to the official website, Apache Kafka is used by thousands of companies including more than 80% of all Fortune 100 companies. Kafka can be run on-premise or in the cloud as a fully managed serverless service (as provided by Confluent, for example).

In Apache Kafka, a running server instance that can receive and send messages is called a broker. Kafka gives you the ability to  horizontally scale your system by running multiple brokers at the same time. Messages (also called “events”) are stored in topics. Topic partitioning, as illustrated below, is a distinguishing characteristic of Kafka:

A topic with 4 partitions and two different producers publishing to it
Source: Apache Kafka official website

 A topic is split into different “buckets” (or “partitions”) that are spread across multiple brokers. This feature is vital to Kafka’s scalability because it allows producers and consumers to both read and write data from/to many brokers simultaneously. Moreover, topics can be replicated across multiple brokers, making your data resilient and highly available. 

Apache Kafka is actually much more than a message broker: it is a fully fledged event streaming platform. In traditional message brokers, messages are removed from the queue after they have been consumed by a subscriber. This is not the case with Apache Kafka: while consumers can be configured to only read new messages from a topic as they arrive, you have control over the actual data retention. If configured, messages can stay persisted in storage for months or even years. Messages can also be “replayed” (consumed multiple times) by consumers.

Kafka leveredges 5 powerful core APIs:

  • Admin API: manage and inspect Kafka objects like topics, brokers, etc.
  • Producer API: publish messages to topics
  • Consumer API: subscribe to topics
  • Kafka Streams API: provides higher-level functions that can be used to read data from an input steam, transform that data and produce it into a resulting output stream
  • Kafka Connect API: enables the use of “connectors” which can read/write streams of data from/to external systems 

The reason Kafka is included in this list, even though it’s not a traditional message broker, is because many people recognize it as a tool that solves the same problems: it can be used as a messaging solution to decouple microservices. Where Kafka really shines though is in building event-driven architectures and real-time data pipelines. Apache Kafka is designed to achieve very high throughput with low latency and can handle a very large amount of messages per second, making it a great solution for doing things like tracking user clicks on a website.

RabbitMQ

RabbitMQ is an open source message broker originally released in 2007. The official website says that RabbitMQ is one of the most popular open source message brokers with tens of thousands of users – from small startups to large enterprises.

RabbitMQ originally only implemented the Advanced Message Queuing Protocol (AMQP), but has since been extended to support other protocols (STOMP, MQTT, etc.), as well as streams. It is lightweight, scalable and can be deployed both on-premise and in the cloud. RabbitMQ benefits from a huge open source community which continuously develops and releases plugins and tools that can be utilised by users.

The diagram below shows a simplified view of the AMQP 0-9-1 Model which is natively implemented by RabbitMQ:

AMQP 0-9-1 Model
Source: RabbitMQ official website

You may notice that instead of directly pushing messages to a queue, producers are publishing into an entity called an exchange instead. An exchange has the role of a post office: it distributes messages across queues by using rules called bindings. According to specific needs, RabbitMQ offers several types of exchanges which can be seen here, including:

  • fanout exchange: the exchange will multiply a message and send it out to all available queues
  • direct exchange: each message gets a routing key and the exchange will move the message to a specific queue according to that key
  • topic exchange: the exchange will send the message to multiple queues with a shared pattern

In addition to traditional message queue capabilities, the release of RabbitMQ 3.9 in 2021 introduced streams. Much like Apache Kafka, consuming from a stream in RabbitMQ will not remove messages from the queue after they are processed. RabbitMQ streams are applicable for use cases that require high throughput, large fan-outs (where many applications need to read the same message) or replay/time-travelling (the ability to re-read the same data from the start of a queue). This important new feature elevates RabbitMQ into not only being a message broker, but a viable event-streaming solution as well.

Amazon SQS

AWS offers over 200 cloud services and naturally, some of them are centred around message brokers and queues. These include Amazon MQ, Amazon SQS, Amazon SNS among others. In this tech note, we will focus on Amazon SQS (Simple Queue Service) which provides fully managed and secure message queuing for microservices, distributed systems and serverless applications. The diagram below gives a high-level view on how Amazon SQS works:

How Amazon SQS works
Source: Official AWS website

As with many other AWS services, elastic scaling and cost-effectiveness come inherently with SQS: you can send and receive any volume of messages and SQS will dynamically scale to handle the load without you having to do any preliminary capacity planning or pre-provisioning.

Being fully managed, SQS simplifies the task of using message queues to decouple and connect microservices: you are not required to manually manage a message broker of your own, AWS does it for you. In addition, the contents of SQS messages are encrypted with Server-Side-Encryption using keys in the AWS KMS (Key Management Service), thereby ensuring the security of the data inside the message queue.

Amazon SQS offers two queue types, depending on your application requirements:

  • Standard queues, which offer a nearly unlimited number of transactions per second, an at-least-once message delivery guarantee and best-effort message ordering (the order in which messages are delivered might occasionally be different than the one they were sent in)
  • FIFO queues, which offer high throughput plus a guarantee of exactly-once message processing and First-In-First-Out delivery

As with many other AWS services, SQS pricing is on a “pay only for what you use” basis, there is no minimum fee that customers have to pay. As of writing this in December 2023, the Amazon SQS Free Pricing Tier allows customers to make 1 million requests for free each month. If this limit is exceeded, customers can pay as little as $0.40 for up to 100 billion requests per month. This makes Amazon SQS a very popular message broker despite other free and open source alternatives.

Apache ActiveMQ

ActiveMQ is another entry on this list under the Apache umbrella: it’s an open source multi-protocol message broker which is written in Java, but can be connected to by clients written in different languages like JavaScript, C/C++, Python and others. It implements the Jakarta Messaging API (formerly known as Java Messaging Service/JMS) and supports a series of protocols including AMQP, OpenWire, STOMP and MQTT. A broker diagram of ActiveMQ can be found below:

High-level architecture of Apache ActiveMQ
Source: Official ActiveMQ documentation

ActiveMQ’s “pluggable” architecture is immediately noticeable – for example, users can choose from multiple message storage options like in-memory, on-disk or inside a relational database. You are also encouraged to write your own plugins if none of the existing ones suit your needs, making this message broker extremely customizable. In addition, an interesting feature of ActiveMQ is that besides in standalone mode, it can also run as an embedded broker inside of an application.

ActiveMQ was originally released in 2004 and has since been adopted as a messaging solution by many enterprises. Right now, ActiveMQ comes in two variations: Classic and Artemis, which are designed to be interchangeable, but use completely different code under the hood. The official description of the two products describes ActiveMQ Classic as “long established, endlessly pluggable architecture serving many generations of applications” and ActiveMQ Artemis as “high-performance, non-blocking architecture for the next generation of messaging applications”, giving the clear notion that Artemis is going to be the default “flavour” of ActiveMQ in the future.

Conclusion

As the complexity of modern distributed applications increases, so does the challenge of communication between the systems involved. Direct communication between a large number of components, especially if they use different technologies and programming languages, can become really hard to track and maintain. Synchronous communication can lead to microservices being blocked while waiting for a long-running task to complete or requests getting lost if a target system is down.

Message brokers can help tackle some of these problems by introducing asynchronous messaging between your system’s components. Depending on your needs, you can choose between two different message distribution models: point-to-point messaging (suitable for things like financial transaction processing where a message has to be processed exactly once) or publish/subscribe (suitable for things like processing event notifications where a single event needs to be broadcasted to multiple applications).

We have also looked at the characteristics of 4 popular message solutions:

  • Apache Kafka: a fully fledged event streaming platform. Perfect for event-driven architecture, extremely high throughput (like tracking website activities) or cases that require message retention
  • RabbitMQ: a more traditional message broker, but recent versions also support streams. Ideal if you need to apply specific routing rules for messages or if you need a lightweight message broker that can support streaming
  • Amazon SQS: fully managed message queuing in the AWS cloud. Great choice if you are in need of a message broker and your infrastructure is already on AWS
  • Apache ActiveMQ: long established message broker that is used by many enterprises. Suitable if you need to do JMS messaging or want to leverage the features of an embedded broker

These are just a handful of messaging technologies that can simplify the communication between components in your architecture – there are many more open source and proprietary solutions available for your specific needs, but this tech note has hopefully given you a good starting point for further exploring the topic of message brokers.

By Yoana Ivanova, Consultant at Estafet

References

  1. VMware – What is a Message Broker? https://www.vmware.com/topics/glossary/content/message-brokers.html
  2. Apache Kafka official website: https://kafka.apache.org/
  3. Confluent official website: https://www.confluent.io/
  4. RabbitMQ official website: https://www.rabbitmq.com/
  5. Amazon SQS – official AWS website: https://aws.amazon.com/sqs
  6. ActiveMQ official website: https://activemq.apache.org/
Stay Informed with Our Newsletter!

Get the latest news, exclusive articles, and updates delivered to your inbox.