Microservice + RabbitMQ application - 5

In this post, we will analyze how we used rabbitmq in out microservice project. As i have mentioned in the previous posts, no matter what kind of system we design, we have to get to know rabbitmq well in order to use it properly.

We didn't have to dig deeper into the spring cloud tools this much. We used gateway and eureka and even though they have brilliant features, their main responsibility were emphasized more. Message brokers have a bit wider serving area since the connecting parties can be doing very different things. What i mean is: The message brokers that works with publishers and consumer by loosely coupling, are also trying to minimize the problems and the responsibilities of the publishers and consumers.

How did we use it

How did the rabbitmq work in our system: It was delivering the messages to mail service from MVC application with a queue and redirecting the dead ones to deadletter queue. Let's keep the big picture here as usual.

We haven't implemented any logic for any backup or load balancing. To be honest, i didn't quite like the node management features of rabbitmq and didn't get it much :D So the mail service became the listener of the rabbitmq. Boths sides of the message broker is in our ecosystem. Which means the publisher and the consumer are not any external system, they are interal. This could be different. For example mail service could have been a 3rd party docker image somewhere outside of our control. In this case, we would have had to deal with questions like when is it going to response or what kind of responses can it return.

We have used 3 exchanges and 3 queues for 3 functionalities. We have sent emails for pament events and event cancelations. There is not much critical case here. You should think thoroughly about how to use exchanges and queues for more complicated scenarios. Because it will be too expensive to change these configurations later. We have focused on something else. We have simulated errors while sending cancelation emails and and redirected the dead messages to deadletter queue. To carry out this functionality, we have put the responsibility to rabbitmq and we made all configurations there.

So basically we have used rabbitmq for its basic features. We haven't dug deeper into extra features of exchanges and queues, which i don't even have much knowledge about. We ahve used the topic logic wthout any obligation, we wanted to see it works. There isn't anything special about the listener methods in the mail service, except they have rabbitlistener on top. So both the MVC and the mail service side of this interaction are pretty simple. All we had to keep in mind was the configuration of rabbitmq. Doesn't it look like rabbitmq became way too important?

Possible problems and solutions

Let me begin with the first question that pops up in my mind. We use message brokers like rabbitmq for asynchronous operations. Moreover, these operations might be bound to some external 3rd party application too. We can use rabbitmq as an intermediary tool to wait fore messages instead of handling the network operations ourselves. That is why it becomes harder to answer questions like "why did rabbitmq do this" or "is it caused by rabbitmq or the external system" since it usually works asynchronous. No matter what tool you use for message broker, it would require a strong logging and monitoring system in place too.

The same situation is also occurs for the exception we throw from the mail service. We have seen the deadletter queue working when exception happen and it worked. But the MVC application has no idea about what happened. Nobody knows there are messages bunching up in the deadletter queue. Someone has to prepare a reporting system, whether it is inside rabbitmq or in some other service, and has to make sure it is working properly. You should also log these errors somewhere since the errors in a microservice is also distributed like microservice itself. This way, deadletter queue could make sense to use. We have seen how to use it, not how to ideally use it.

Another problem in this system: When rabbitmq service is stopped, the mvc application crashes with Connection refused: connect error, even if it is able to sell the ticket. Because we don't have any resiliency provider for this interaction. This is the 3rd problem, when rabbitmq stops for some reason. This doesn't mean our messages not reaching to related queue because of wrong routing keys. We will look at it later.

Spring amqp establishes an http connection to send messages but it throws an error if the connection fails. You need to log it somewhere but it won't be enough. You need to save the messages and re-send them later. This way, rabbitmq will be able to retrieve the messages that it doesn't even know they exist. I don't exactly have a solution for this situation but i think the concept of event sourcing comes into play here. After all, there is an interaction happening between 2 parts of the microservice and we could save it somewhere to repeat it i guess.

There is one more problem. If you don't document the rabbitmq configuration somewhere and change it one day, both sides of the communication must be aware of these changes. Both the publisher and receiver paries has to know where to send messages or receive them from. This is extra work for everyone. Furthermore, we have used direct exchange for cancelation mails. If you make a type in the routing key of this operation, the system doesn't throw an error :) This means looking for a problem in rabbitmq for hours when somebody changes a route configuration and forgets it and everybody realize errors after 3-4 days and configuration change is already forgotten. Since there is no undeliverable message, you can't see this in the deadletter queue either. So if i got it right, our message just vanishes. I think there is a ReturnHandler to notify this but you would still have to save your message and resend it later. So i think event sourcing again.

Of course rabbitmq developers have probably thought about these problems and there could be other solutions as well. I can investigate so much with my limited developer level of knowledge. But one thing is for sure: This tool is a beast dude.

Is it production ready?

Maybe some of you have "so tell me if i can use rabbitmq directly in me production environment?" kind of question in mind. You may not want to deal with errors and of an immature tool. Even though some people have been working on this tool for like 10 years, they have left a note in their website under the code examples they give:

Production [Non-]Suitability Disclaimer
Please keep in mind that this and other tutorials are, well, tutorials. They demonstrate one new concept at a time and may intentionally oversimplify some things and leave out others. For example topics such as connection management, error handling, connection recovery, concurrency and metric collection are largely omitted for the sake of brevity. Such simplified code should not be considered production ready.

In other words, you can't just get rabbitmq and use it in production since it is a vital part of the system and able to carry out lots of responbilities. You might have to consider reliability concerns and might have to make developments according to them even for very basic functionalities. Being a mature tool doesn't mean you can just take it and use it like a brand new car. You can refer to this page for more information about rabbitmq and reliability.

I hope the mssage is received :)

Well, i don't know if my messages are reaching you from a queue :D If you ask me, i may not use rabbitmq for very basic operations if there is no asynchronous request - response with an external system. If i don't have someone who knows a lot about rabbitmq working with me, it kind of looks like i am taking more risks and work load for simple functions. So why did you write 5 parts series of posts? I have learned it and embraced it. I have realized it is not a tool to be used for hype. If you know the detailed inner workings of rabbitmq, it looks like a strong and successful tool. I can say it re-defines the interaction between the parts of your microservice. I hope i got it right and said it right. We have reached the end of this series where we used rabbitmq in a microservice application. See you at the next post :)