1. Introduction

In my previous post, I’ve shown how to configure Logstash to parse logs from files. This is pretty useful however if your application is deployed on multiple servers, you usually log to some kind of central log storage – in my case to queue, RabbitMQ to be more specific. In this post, I will show how to configure Logstash so it reads the logs from that queue.

2. Preparing queue

Before we move to Logstash configuration, first of all, we have to prepare RabbitMQ test instance. If you don’t have RabbitMQ yet, go to this website and install the queue. Once installation is done, go to the installation folder (C:\Program Files\RabbitMQ Server\rabbitmq_server-3.6.5\sbin in my case) and run in console

This command will prepare RabbitMQ management website, so it will be easier for us to see what is going on in given queue. In the next step, we have to prepare the queue, the logs will be sent to. You can do it via the website we’ve just enabled (http://localhost:15672/) or via RabbitMQ admin console. As I prefer to automate things as much as possible I will do it via command line. What is quite unusual when it comes RabbitMQ CLI is the fact that it is a python script you have to download and run locally (this is not an executable). The script can be found on management site under this address. Once the script is downloaded (in my case it is saved as rabbitmqadmin.py) you can start preparing necessary elements: exchange, queue and the binding.

As you can see I’ve created exchange called logger which is bound to MyAppLogginQueue queue using MyApp route. This means that every message with topic MyApp sent to logger exchange will be pushed to MyAppLogginQueue .

3. Preparing Logstash

Logstash configuration will be modified version of my previous config. I will just add another input source. Here is a basic usage

As you can see we will be consuming messages from MyAppLogginQueue which is deployed on localhost. For password and user properties use your own credentials. That is basically it, so now it is time to see if everything is working.

4. Testing coniguration

In order to test the configuration you have to run the Elasticsearch, Kibana and use new config for Logstash. I’ve shown how to do it in one of my recent post . For sending messages to the queue I will just use RabbitMQ management website API. The API exposes

endpoint accepting POST verbs which can be used for publishing messages to given exchange. In my case POST body will look as follows

and I will be sending it to

Note that I will be sending messages to the exchange, not to the queue itself. The exchange’s responsibility is to route the message to all bound queues. Here is how it looks in practice

As you can see our configuration is valid and messages are shown on Kibana’s dashboard almost in real time.

Full Logstash config can be found here

In my previous post I’ve shown how to configure Logstash so that, it would be able to parse the logs in custom format. Configuration presented in that post had one significant drawback – it wasn’t able to parse multiline log entries. This is a rather common scenario, especially when you log exceptions with a stack trace. Log entry, in that case may look as follows

Parsing these kind of messages using current configuration will result in grok parsing error

and we won’t be able to search against predefined fields. Fortunately Logstash allows you to configure something called input codecs which basically allows you to transform input data into some other form. One of those codes is multiline codec, which is responsible for “merging” multiline logs into one entry.
Here is example of codec configuration

The code above says that any line not starting with a “TimeStamp=timestamp value” should be merged with the previous line.
Multiline coded can be added to a variety of inputs. Here is how you can apply it to file input

Thanks to that change, Logstash is now able to correctly parse exceptions from our logs.

Source code for this post can be found here

1. Introduction

In one of my previous post I’ve shown how to improve logging in an application by tracking the flow of incoming requests. Now it is time to show the basics of Elastic stack to make searching across multiple log files/sources a piece of cake. Elastic stack (previously called ELK stack) is set of three tools which allows you to parse (Logstash), query (Elasticsearch) and visualize (Kibana) logs with ease.

2. Installation

First of all (as usual) we have to get the tools, so go to elastic.co and download apps mentioned before.
These are stand-alone applications so no installation is required. The only requirement is to have JAVA_HOME system variable pointing to your java directory. In my case, this looks as follows

3. Elasticsearch

Once all three applications are downloaded we can run Elasticsearch (as there is not additional configuration needed for basic usage) instance via an elasticsearch.bat file.

4. Logstash

Having our Elasticsearch instance up and running, now it is time configure Logstash so that it will be able to parse logs.
Configuration provided in next sections will be able to parse logs with the following format

4.1. Configuration

Logstash configuration is done via config file in specific format. You can read about that here
The very first step is to define input and output sections

As you can see the logs will be read from C:\logs directory and parsed content will be pushed to Elasticsearch instance and to the console output.
We can verify correctness of configuration calling

If you are making your config file for the first time it is good idea to add additional properties to the file section

This will force Logstash to reparse entire file when restarted and will also make sure that older files are not ignored (by default files modified more than 24 hours ago are ignored).

At this point, Logstash can read the log file but it doesn’t do anything special with it. Next step is to configure a pattern which will be used for log parsing. Logstash uses grok for defining the patterns. Long story short it is kind of a regex which can use predefined patterns. The easiest way to play around with it is to use grokconstructor, for list of ready to use patterns take a look at this

For parsing logs shown in previous section I’ve ended up with this grok pattern

Notice that you are able to give aliases for particular fields, for instance

means that everything that matches TimeStamp=%{TIMESTAMP_ISO8601} will be stored in logdate field.
Having defined our pattern, now we can add it to the config file. After modifications, it looks as follows

Once we run this config and start querying elastic search for list of all indices via

we will see that our logs were parsed and stored in index called logstash-2016.10.30

If we now go to

we will be able to see index information

4.2. Fixing the date fields

At this moment there are two main problems with our configuration. First of all, the indices are created based on read time from file. Second of all our logdate field is treated as string.

By default Logstash creates indices based on read time of the source. However in my opinion it is better to create index names based on time given event occurred.
In order to do that, we have to tell the Logstash which field is responsible for holding timestamp. In my case, this field is called logdate. All we have to do is to map this field into field via date filter

As you can see first argument is a filed name and rest of the arguments (you can specify more) are just date time formats. By default date filter maps a field from the match property into @timestamp field. So the config above equals this one

If we restart Logstash and get the indices, we will see something similar to that

Second problem can be handled in very similar way. We have to add second date filter and select as target longdate field

From now on logdate field will be treated as date so we will be able to filter the logs easily with Kibana

5. Running Kibana

Having all the configuration in place now we are ready to run Kibana. As in previous steps, no installation is needed so just run the Kibana.bat file

and go to

If you run the app for the first time you will be asked to configure the indices. You can use the default parameters and just click “Create” button.
Once the indices are setup you can start writing queries against the logs. By default entire message is searched for the search terms, however the real power comes with queries written against specific fields. For example, you can search for any errors in the application with this simple query

Thanks to date type fields, you can combine this query with date range selector and narrow down time to specific values, just like that

These are just the basic queries you can run in Kibana, for more advanced scenarios please visit website . I also strongly encourage you guys to take a look at other feature Elastic stack provides. Source code for this post can be found here