\nThese are stand-alone applications so no installation is required. The only requirement is to have JAVA_HOME<\/i> system variable pointing to your java directory. In my case, this looks as follows<\/p>\n
\r\nC:\\Program Files\\Java\\jre1.8.0_91\\\r\n<\/pre>\n3. Elasticsearch<\/h3>\n
Once all three applications are downloaded we can run Elasticsearch<\/i> (as there is not additional configuration needed for basic usage) instance via an elasticsearch.bat<\/i> file.
\n![\"elasticsearchweb\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/ElasticSearchWeb-1.png\")
<\/a><\/p>\n4. Logstash<\/h3>\n
Having our Elasticsearch<\/i> instance up and running, now it is time configure Logstash<\/i> so that it will be able to parse logs.
\nConfiguration provided in next sections will be able to parse logs with the following format<\/p>\n\r\nTimeStamp=2016-07-20 21:22:46.0079 CorrelationId=dc665fe7-9734-456a-92ba-3e1b522f5fd4 Level=INFO Message=About to start GET \/ request\r\n<\/pre>\n4.1. Configuration<\/h4>\n
Logstash<\/i> configuration is done via config file in specific format. You can read about that here<\/a>
\nThe very first step is to define input and output sections<\/p>\n\r\ninput {\r\n file {\r\n path => [\"C:\/logs\/*.log\"]\r\n start_position => beginning\r\n }\r\n}\r\noutput {\r\n elasticsearch {\r\n hosts => \"localhost:9200\"\r\n }\r\n stdout {}\r\n}\r\n<\/pre>\nAs you can see the logs will be read from C:\\logs<\/i> directory and parsed content will be pushed to Elasticsearch<\/i> instance and to the console output.
\nWe can verify correctness of configuration calling<\/p>\n\r\nlogstash.bat -f \"Path\\To\\Config\\File.conf\" -t\r\n<\/pre>\n![\"configurationtest\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/ConfigurationTest.png\")
<\/a>
\nIf you are making your config file for the first time it is good idea to add additional properties to the file <\/i>section<\/p>\n\r\nignore_older => 0\r\nsincedb_path => \"NUL\"\r\n<\/pre>\nThis will force Logstash<\/i> to reparse entire file when restarted and will also make sure that older files are not ignored (by default files modified more than 24 hours ago are ignored).<\/p>\n
At this point, Logstash<\/i> can read the log file but it doesn\u2019t do anything special with it. Next step is to configure a pattern which will be used for log parsing. Logstash<\/i> uses grok<\/i> for defining the patterns. Long story short it is kind of a regex which can use predefined patterns. The easiest way to play around with it is to use grokconstructor<\/a>, for list of ready to use patterns take a look at this <\/a><\/p>\nFor parsing logs shown in previous section I’ve ended up with this grok pattern<\/p>\n
\r\nTimeStamp=%{TIMESTAMP_ISO8601:logdate} CorrelationId=%{UUID:correlationId} Level=%{LOGLEVEL:logLevel} Message=%{GREEDYDATA:logMessage}\r\n<\/pre>\nNotice that you are able to give aliases for particular fields, for instance<\/p>\n
\r\nTimeStamp=%{TIMESTAMP_ISO8601:logdate} \r\n<\/pre>\nmeans that everything that matches TimeStamp=%{TIMESTAMP_ISO8601}<\/i> will be stored in logdate<\/i> field.
\nHaving defined our pattern, now we can add it to the config file. After modifications, it looks as follows<\/p>\n\r\ninput {\r\n file {\r\n path => [\"C:\/logs\/*.log\"]\r\n start_position => beginning\r\n }\r\n}\r\nfilter {\r\n grok {\r\n match => { \"message\" => \"TimeStamp=%{TIMESTAMP_ISO8601:logdate} CorrelationId=%{UUID:correlationId} Level=%{LOGLEVEL:logLevel} Message=%{GREEDYDATA:logMessage}\" }\r\n }\r\n}\r\noutput {\r\n elasticsearch {\r\n hosts => \"localhost:9200\"\r\n }\r\n stdout {}\r\n}\r\n<\/pre>\nOnce we run this config and start querying elastic search for list of all indices via<\/p>\n
\r\nhttp:\/\/localhost:9200\/_cat\/indices?v\r\n<\/pre>\nwe will see that our logs were parsed and stored in index called logstash-2016.10.30<\/i>
\n![\"indices\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/indices.png\")
<\/a>
\nIf we now go to <\/p>\n\r\nhttp:\/\/localhost:9200\/logstash-2016.10.30\r\n<\/pre>\nwe will be able to see index information
\n![\"indexinfo\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/IndexInfo-1.png\")
<\/a><\/p>\n4.2. Fixing the date fields <\/h4>\n
At this moment there are two main problems with our configuration. First of all, the indices are created based on read time from file. Second of all our logdate<\/i> field is treated as string.
\n![\"logdateasstring\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/logdateasstring.png\")
<\/a>
\nBy default Logstash<\/i> creates indices based on read time of the source. However in my opinion it is better to create index names based on time given event occurred.
\nIn order to do that, we have to tell the Logstash<\/i> which field is responsible for holding timestamp. In my case, this field is called logdate<\/i>. All we have to do is to map this field into field via date filter<\/p>\n\r\ndate {\r\n match => [ \"logdate\", \"yyyy-MM-dd HH:mm:ss.SSSS\" ]\r\n}\r\n<\/pre>\nAs you can see first argument is a filed name and rest of the arguments (you can specify more) are just date time formats. By default date filter maps a field from the match<\/i> property into @timestamp<\/i> field. So the config above equals this one<\/p>\n\r\ndate {\r\n match => [ \"logdate\", \"yyyy-MM-dd HH:mm:ss.SSSS\" ]\r\n target => \"@timestamp\"\r\n}\r\n<\/pre>\nIf we restart Logstash<\/i> and get the indices, we will see something similar to that
\n![\"multipleindices\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/multipleindices.png\")
<\/a>
\nSecond problem can be handled in very similar way. We have to add second date filter and select as target longdate<\/i> field<\/p>\n\r\ndate {\r\n match => [ \"logdate\", \"yyyy-MM-dd HH:mm:ss.SSSS\" ]\r\n target => \"logdate\"\r\n}\r\n<\/pre>\nFrom now on logdate<\/i> field will be treated as date so we will be able to filter the logs easily with Kibana<\/i>
\n![\"logdateasdate\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/logdateasdate.png\")
<\/a><\/p>\n5. Running Kibana<\/h3>\n
Having all the configuration in place now we are ready to run Kibana<\/i>. As in previous steps, no installation is needed so just run the Kibana.bat<\/i> file
\n![\"kibanastarted\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/kibanastarted.png\")
<\/a>
\nand go to <\/p>\n\r\nhttp:\/\/localhost:5601\/\r\n<\/pre>\nIf you run the app for the first time you will be asked to configure the indices. You can use the default parameters and just click “Create” button.
\nOnce the indices are setup you can start writing queries against the logs. By default entire message is searched for the search terms, however the real power comes with queries written against specific fields. For example, you can search for any errors in the application with this simple query<\/p>\n
\r\nlogLevel: (FATAL OR ERROR)\r\n<\/pre>\nThanks to date type fields, you can combine this query with date range selector and narrow down time to specific values, just like that<\/p>\n
\r\nlogdate:[2016-07-20 TO 2016-10-31] AND logLevel: (FATAL OR ERROR)\r\n<\/pre>\n![\"kibanasearchresults\"](\"http:\/\/tpodolak.com\/blog\/wp-content\/uploads\/2016\/10\/getting-started-elastic-stack\/kibanasearchresults-1.png\")
<\/a>
\nThese are just the basic queries you can run in Kibana, for more advanced scenarios please visit website <\/a>. I also strongly encourage you guys to take a look at other feature Elastic stack<\/i> provides. Source code for this post can be found here<\/a><\/p>\n\nPS
\nConfiguration presented in this post will not be able to parse multiline log entries e.g. exceptions. I will show you how to do it in the next post.\n<\/div>\n","protected":false},"excerpt":{"rendered":"1. Introduction In one of my previous post I’ve shown how to improve logging in an application by tracking the flow of incoming requests. Now it is time to show the basics of Elastic stack to make searching across multiple log files\/sources a piece of cake. Elastic stack (previously called ELK stack) is set of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[309],"tags":[311,312,310],"class_list":["post-855","post","type-post","status-publish","format-standard","hentry","category-elasticstack","tag-elasticsearch","tag-kibana","tag-logstash"],"_links":{"self":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts\/855","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/comments?post=855"}],"version-history":[{"count":33,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts\/855\/revisions"}],"predecessor-version":[{"id":923,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts\/855\/revisions\/923"}],"wp:attachment":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/media?parent=855"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/categories?post=855"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/tags?post=855"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}