{"id":895,"date":"2016-11-03T20:26:28","date_gmt":"2016-11-03T20:26:28","guid":{"rendered":"http:\/\/tpodolak.com\/blog\/?p=895"},"modified":"2016-11-03T20:26:28","modified_gmt":"2016-11-03T20:26:28","slug":"logstash-parsing-multiline-log-entries","status":"publish","type":"post","link":"https:\/\/tpodolak.com\/blog\/2016\/11\/03\/logstash-parsing-multiline-log-entries\/","title":{"rendered":"Logstash – parsing multiline log entries"},"content":{"rendered":"

In my previous post<\/a> I’ve shown how to configure Logstash<\/i> so that, it would be able to parse the logs in custom format. Configuration presented in that post had one significant drawback – it wasn’t able to parse multiline log entries. This is a rather common scenario, especially when you log exceptions with a stack trace. Log entry, in that case may look as follows<\/p>\n

\r\nTimeStamp=2016-11-01 02:08:02.9086 CorrelationId=69dd69bb-3457-4440-8f77-88b1fd39255a Level=ERROR Message=System.InvalidOperationException: Operation is not valid due to the current state of the object.\r\n   at AspNetMvcLoggingWithCorrelationId.Controllers.HomeController.d__2.MoveNext() in E:\\Tomek\\Programowanie\\BlogSrc\\AspNetMvcLoggingWithCorrelationId\\AspNetMvcLoggingWithCorrelationId\\Controllers\\HomeController.cs:line 24\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Mvc.Async.TaskAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass37.b__36(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.b__3d()\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.b__3f()\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass33.b__32(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.b__1c()\r\n   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.b__1e(IAsyncResult asyncResult)\r\n<\/pre>\n
\r\ncodec => multiline {\r\n                       pattern => \"^TimeStamp=%{TIMESTAMP_ISO8601}\"\r\n                       negate => true\r\n                       what => \"\r\n}\r\n<\/pre>\n
The code above says that any line not starting with a “TimeStamp=timestamp value” should be merged with the previous line.
\nMultiline coded can be added to a variety of inputs. Here is how you can apply it to file<\/i> input<\/p>\n
\r\ninput {\r\n    file {\r\n        path => [\"E:\/Tomek\/Programowanie\/BlogSrc\/ElasticStackParsingMultilineEntries\/logs\/*.log\"]\r\n        start_position => beginning\r\n        ignore_older => 0\r\n        sincedb_path => \"NUL\"\r\n        codec => multiline {\r\n                               pattern => \"^TimeStamp=%{TIMESTAMP_ISO8601}\"\r\n                               negate => true\r\n                               what => \"previous\"\r\n        }\r\n    }\r\n}\r\n<\/pre>\n
Thanks to that change, Logstash<\/i> is now able to correctly parse exceptions from our logs.
\n\"fixed2\"<\/a><\/p>\n
Source code for this post can be found here<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In my previous post I’ve shown how to configure Logstash so that, it would be able to parse the logs in custom format. Configuration presented in that post had one significant drawback – it wasn’t able to parse multiline log entries. This is a rather common scenario, especially when you log exceptions with a stack […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[309],"tags":[311,312,310],"class_list":["post-895","post","type-post","status-publish","format-standard","hentry","category-elasticstack","tag-elasticsearch","tag-kibana","tag-logstash"],"_links":{"self":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts\/895","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/comments?post=895"}],"version-history":[{"count":16,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts\/895\/revisions"}],"predecessor-version":[{"id":934,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/posts\/895\/revisions\/934"}],"wp:attachment":[{"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/media?parent=895"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/categories?post=895"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tpodolak.com\/blog\/wp-json\/wp\/v2\/tags?post=895"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Parsing these kind of messages using current configuration will result in grok parsing error
\n\"grokfailure\"<\/a>
\nand we won’t be able to search against predefined fields. Fortunately Logstash<\/i> allows you to configure something called input codecs<\/i> which basically allows you to transform input data into some other form. One of those codes is multiline codec<\/i>, which is responsible for “merging” multiline logs into one entry.
\nHere is example of codec configuration<\/p>\n