Logstash – reading logs from RabbitMQ

1. Introduction

In my previous post, I’ve shown how to configure Logstash to parse logs from files. This is pretty useful however if your application is deployed on multiple servers, you usually log to some kind of central log storage – in my case to queue, RabbitMQ to be more specific. In this post, I will show how to configure Logstash so it reads the logs from that queue.

2. Preparing queue

Before we move to Logstash configuration, first of all, we have to prepare RabbitMQ test instance. If you don’t have RabbitMQ yet, go to this website and install the queue. Once installation is done, go to the installation folder (C:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.5\sbin in my case) and run in console

This command will prepare RabbitMQ management website, so it will be easier for us to see what is going on in given queue. In the next step, we have to prepare the queue, the logs will be sent to. You can do it via the website we’ve just enabled (http://localhost:15672/) or via RabbitMQ admin console. As I prefer to automate things as much as possible I will do it via command line. What is quite unusual when it comes RabbitMQ CLI is the fact that it is a python script you have to download and run locally (this is not an executable). The script can be found on management site under this address. Once the script is downloaded (in my case it is saved as rabbitmqadmin.py) you can start preparing necessary elements: exchange, queue and the binding.

As you can see I’ve created exchange called logger which is bound to MyAppLogginQueue queue using MyApp route. This means that every message with topic MyApp sent to logger exchange will be pushed to MyAppLogginQueue .

3. Preparing Logstash

Logstash configuration will be modified version of my previous config. I will just add another input source. Here is a basic usage

As you can see we will be consuming messages from MyAppLogginQueue which is deployed on localhost. For password and user properties use your own credentials. That is basically it, so now it is time to see if everything is working.

4. Testing coniguration

In order to test the configuration you have to run the Elasticsearch, Kibana and use new config for Logstash. I’ve shown how to do it in one of my recent post . For sending messages to the queue I will just use RabbitMQ management website API. The API exposes

endpoint accepting POST verbs which can be used for publishing messages to given exchange. In my case POST body will look as follows

and I will be sending it to

Note that I will be sending messages to the exchange, not to the queue itself. The exchange’s responsibility is to route the message to all bound queues. Here is how it looks in practice
As you can see our configuration is valid and messages are shown on Kibana’s dashboard almost in real time.

Full Logstash config can be found here

Logstash – reading logs from RabbitMQ