Skip to content

RabbitMQ in an ASP .Net Core 3.1 Microservice

In my last posts, I created two microservices using ASP .net core 3.1. Today, I will implement RabbitMQ, so the microservices can exchange data while staying independent. RabbitMQ can also be used to publish data even without knowing the subscribers. This means that you can publish an update and whoever is interested can get the new information.

What is RabbitMQ and why use it?

RabbitMQ describes itself as the most widely deployed open-source message broker. It is easy to implement and supports a wide variety of technologies like Docker, .Net or Go. It also offers plugins for monitoring, managing or authentication. I chose RabbitMQ because it is well known, quickly implemented and especially can be easily run using Docker.

Why use a Queue to send data?

Now that you know what RabbitMQ is, the next question is: why should you use a queue instead of directly sending data from one microservice to the other one. There are a couple of reasons why using a queue instead of directly sending data is better:

  • Higher availability and better error handling
  • Better scalability
  • Share data with whoever wants/needs it
  • Better user experience due to asynchronous processing

Higher availability and better error handling

Errors happen all the time. In the past, we designed our systems to avoid errors. Nowadays we design our systems to catch errors and handle them in a user-friendly way.

Let’s say we have an online shop and the order services send data to the process order service after the customer placed his order. If these services are connected directly and the process order service is offline, the customer will get an error message, for example, “An error occurred. Please try it again later”. This user probably won’t return and you lost the revenue of the order and a customer for the future.

If the order service places the order in a queue and the order processing service is offline, the customer will get a message that the order got placed and he or she might come back in the future. When the order processing service is back online, it will process all entries and the queue. You might know this behavior when booking a flight. After booking the flight it takes a couple of minutes until you get your confirmation per email.

Better scalability

When you place messages in a queue, you can start new instances of your processor depending on the queue size. For example, you have one processor running, when there are ten items in the queue, you start another one and so on. Nowadays with cloud technologies and serverless features, you can easily scale up to thousands of instances of your processor.

Share data with whoever wants/needs it

Most of the time, you don’t know who wants to process the information you have.

Let’s say our order service publishes the order to the queue. Then the order processing service, reporting services, and logistics services can process the data. Your service as a publisher doesn’t care who takes the information. This is especially useful when you have a new service in the future which wants your order data too. This service only has to read the data from the queue. If your publisher service sends the data directly to the other services, you would have to implement each call and change your service every time a new service wants the data.

Better user experience due to asynchronous processing

Better user experience is the result of the three other advantages. The user is way less likely to see an error message and even when there is a lot of traffic like on Black Friday, your system can perform well due to the scalability and the asynchronous processing.

Implement RabbitMQ with .Net Core

You can find the code of  the finished demo on GitHub.

RabbitMQ has a lot of features. I will only implement a simple version of it to publish data to a queue in the Customer service and to process the data in the Order service.

Implementation of publishing data

I am a big fan of Separation of Concerns (SoC) and therefore I am creating a new project in the CustomerApi solution called CUstomerApi.Message.Send. Next, I install the RabbitMQ.Client NuGet package and create the class CustomerUpdateSender in the project. I want to publish my Customer object to the queue every time the customer is updated. Therefore, I create the SendCustomer method which takes a Customer object as a parameter.

Publish data to RabbitMQ

Publishing data to the queue is pretty simple. First, you have to create a connection to RabbitMQ using its hostname, a username and a password using the ConnectionFactory. With this connection, you can use QueueDeclare to create a new queue if it doesn’t exist yet. The QueueDeclare method takes a couple of parameters like a name and whether the queue is durable.

Creating a connection and a queue in RabbitMQ
Creating a connection and a queue in RabbitMQ

After creating the connection to the queue, I convert my customer object to a JSON object using JsonConvert and then encode this JSON to UTF8.

Convert the customer object to json and then encode it to UTF8
Convert the customer object to json and then encode it to UTF8

The last step is to publish the previously generated byte array using BasicPublish. BasicPublish has like QueueDeclare a couple of useful parameters but to keep it simple, I only provide the queue name and my byte array.

Publish the customer to the RabbitMQ queue
Publish the customer to the queue

That’s all the logic you need to publish data to your queue. Before I can use it, I have to do two more things though. First, I have to register my CustomerUpdateSender class in the Startup class. I am also providing the settings for the queue like the name or user from the appsettings. Therefore, I have to read this section in the Startup class.

Read the configs for the RabbitMQ queue and register the service
Read the configs for the queue and register the service

The last step is to call the SendCustomer method when a customer is updated. This call is in the Handle method of the UpdateCustomerCommandHandler. If you commented out the line before to test the application without RabbitMQ, you have to uncomment the call now.

"<yoastmark

Implementation of reading data from RabbitMQ

Implementing the read functionality is a bit more complex because we have to constantly check the queue if there are new messages and if so, process them. I love .net core and it comes really handy here. .net core provides the abstract class BackgroundService which provides the method ExecuteAsync.  This method can be overriden and is executed regularly in the background.

In the OrderApi solution, I create a new project called OrderApi.Messaging.Receive, install the RabbitMQ.Client NuGet and create a class called CustomerFullNameUpdateReceiver. This class inherits from the BackgroundService class and overrides the ExecuteAsync method.

In the constructor of the class, I initialize my queue the same way as in the CustomerApi using QueueDeclere. Additionally, I register events that I won’t implement now but might be useful in the future.

Initialize the RabbitMQ queue
Initialize the RabbitMQ queue

Reading data from the queue

In the ExecuteAsync method, I am subscribing to the receive event and whenever this event is fired, I am reading the message and encode its body which is my Customer object. Then I am using this Customer object to call another service that will do the update in the database.

Read a message from the RabbitMQ queue and process it
Read a message from the queue and process it

That’s all you have to do to read data from the queue. The last thing I have to do is to register my CustomerFullNameUpdateReceiver class as a background service in the Startup class.

Register the class as background service
Register the class as background service

Run RabbitMQ in Docker

The publish and receive functionalities are implemented. The last step before testing them is to start an instance of RabbitMQ. The easiest way is to run it in a Docker container. If you don’t know what Docker is or haven’t installed it, download Docker Desktop for Windows from here or for Mac from here. After you installed Docker, copy the two following lines in Powershell or bash:

docker run -d --hostname my-rabbit --name some-rabbit -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:3-management
docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3-management

Don’t worry if you don’t understand them. Simplified these two lines download the RabbitMQ Docker image, start is as a container and configure the ports, the name, and the credentials.

Run RabbitMQ in a Docker container
Run RabbitMQ in a Docker container

After RabbitMQ is started, you can navigate to localhost:15672 and login with guest as user and guest as password.

Login into the RabbitMQ management portal
Login into the RabbitMQ management portal

Navigate to the Queues tab and you will see that there is no queue yet.

"<yoastmark

Now you can start the OrderApi and the CustomerApi project. The order how you start them doesn’t matter. After you started the CustomerApi, the CustomerQueue will be created and you can see it in the management portal.

"<yoastmark

Click on CustomerQueue and you will see that there is no message in the queue yet and that there is one consumer (the OrderApi).

Overview of the CustomerQueue and its Consumers
Overview of the CustomerQueue and its Consumers

Go to the Put action of the CustomerApi and update a customer. If you use my in-memory database you can use the Guid “9f35b48d-cb87-4783-bfdb-21e36012930a”. The other values don’t matter for this test.

Update a customer
Update a customer

After you sent the update request, go back to the RabbitMQ management portal and you will see that a message was published to the queue and also a message was consumed.

 

A message was published and consumed
A message was published and consumed

 

Shortcomings of my Implementation

I wanted to keep the implementation and therefore it has a couple of shortcomings. The biggest problem is that the OrderApi won’t start if there is no instance of RabbitMQ running. Also, the CustomerApi will crash when you try to update a customer and there is no instance of RabbitMQ running. There is no exception handling right now. Also if there is an error while processing a message, the message will be deleted from the queue and therefore be lost.

Another problem is that after the message is read, it is removed from the queue. This means only one receiver is possible at the moment. There are also no unit tests for the implementation of the RabbitMQ client.

Conclusion

This post explained why you should use queues to decouple your microservices and how to implement RabbitMQ using Docker and ASP .net core 3.1. Keep in mind that this is a quick implementation and needs some work to be production-ready.

You can find the code of  the finished demo on GitHub. In my next post, I will dockerize the application which will make it way easier to run and distribute the whole application.

Published inASP.NET MVC / CoreCloudDevOpsDockerProgramming

13 Comments

  1. Tomas Forsman Tomas Forsman

    For me the command had to be:
    docker run -d –hostname my-rabbit –name some-rabbit -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=password rabbitmq:3-management

    Note the double – for hostename and name.

    • Tomas Forsman Tomas Forsman

      I realize now that you had that as well but the formatting of the blog made it into a long – rather than two -.

      • Hi Tomas,

        I don’t know why the double – were displayed as one long one but I changed it and it should work now.

        Thank you for your feedback.

  2. Ricardo van Dijke Ricardo van Dijke

    the adress in line “After RabbitMQ is started, you can navigate to localhost:15762 and login with guest as user and guest as password.”

    is wrong, it should be localhost:15672 as shown in the picture

  3. Piotr Piotr

    Hi,
    How would you unit test the implementation of RabbitMQ? Is there a way to test it without actually using its dependencies?

  4. Matheswaran Matheswaran

    Hi,

    How does this receiver works in a multi threaded / concurrent environment.?

    when the receiver service is deployed in a Load balanced environment, there is a high possibility both receiver try to pick the same message from the queue.

    Do you agree that we need to add an additional thread safe mechanism like Semaphore, monitor etc to be implemented.?

    or does RabbitMQ internal mechanism handle this ?

    sorry if my question is stupid.

    Thanks,
    Matheswaran S

    • Hi Matheswaran,

      that’s a great question. You are right, this solution would lead to problems in a load balancer environment. I wanted to keep this demo as simple as possible, therefore I kept everything in one project. In a real-world project, I would use an Azure function to fetch data from the queue. The function gets triggered every time a new message is written. So I don’t have to deal with concurrency problems and additionally, I don’t have to worry about the performance of my website.
      In my demo, the website may be slowed down if there are too many messages to be handled, since it all runs in the same solution.

  5. Damjan Damjan

    Hi Wolfgang,

    I am wondering, what can we do here to implement the ability for ordering service to consume messages from, for example, one more queue?

    Thanks in advance

    • Hi Damjan,
      if you want to consume message from another queue, you could add another worker service. This service would run in the background and process the new messages. The problem with this approach is that it may slow down your order service if you have too much running at the same time.
      A better solution would be to place the background service in separate applications or to use Azure Functions. Alternatively, you could use Cron Jobs, for example, in Kubernetes to start the processing of the message periodically (every 5 or 15 minutes for example)

  6. Tiaan Krieg Tiaan Krieg

    This is an excellent writeup and that even impressed me more is the structuring of the 2 GitHub projects with a fully functioning real production scenario. The great breakup of the different concerns and class structures. Lost of great nuggets used.

Leave a Reply

Your email address will not be published. Required fields are marked *

RSS
Follow by Email
LinkedIn
Share