Logstash Grok Filters for Mule Logs in PatternLayout - Part II

byGonzalo Marcos -

In part I of the series, we’ve seen how to create a Grok filter in Logstash to parse the default Mule logs in PatternLayout. We’ve learnt what the PatternLayout and Grok filters are, and how to match the PatternLayout conversion specifiers to Grok patterns.


With that, we’ve got everything we need to know to create now our custom format for our logs in PatternLayout and create the corresponding Grok filter. Let’s see it:

Create a Mule App

First, let’s create a new Mule Project. Then, create a new flow with an HTTP Listener for GET /hello and a Logger Processor



Define our custom log format

Next, let’s define what we want to include in our logs. First, we’d like to include some of the default fields that come out of the box in a mule app:

  • log level
  • Date & Time
  • Thread name
  • thread id
  • correlation id
  • Processor name
  • processor path
  • logger name
  • log message

We’d also like to add some contextual information, so that we can identify from which app we’re logging and some details about the app, such as the version, the runtime or the environment. For that, we’ll use the POM file, picking up some existing properties.

  • Application name - We can pick up the existing ${project.name}
  • Application version - From the maven coordingates ${project.version}

and adding some of our own. Edit the POM file and add:

<project>
    ...
    <properties>
        ...
        <app.runtime>4.8.0</app.runtime>
        <app.environment>DEV</app.environment>
        <api.layer>SYSTEM</api.layer>
    </properties>
    ...
</project>

Where you can replace the values of the runtime, environment and API layer with your own.


Enable Resource Filtering

In order to be able to use the properties from our POM file in the log4j2.xml, we need to enable the Maven Resource Filtering plugin. Check out our post - How to pass values from the POM file to the Mule logs to know more about how it works. 

Modify the pom file to enable the resource filtering ONLY to our log4j2.xml file by adding the following to the build section of the POM file:

<?xml version="1.0" encoding="UTF-8"?>
<project>
    ...
    <groupId>com.mycompany</groupId>
    <artifactId>custom-log-format</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>mule-application</packaging>

    <name>pom-to-logs</name>

    <properties>
        ...
        <app.runtime>4.8.0</app.runtime>
        ...
    </properties>

    <build>
        <resources>
            <!-- Enable filtering for specific resources -->
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
                <includes>
                    <include>log4j2.xml</include>
                </includes>
            </resource>
            <!-- Disable filtering for everything else -->
            <resource>
                <directory>src/main/resources</directory>
                <filtering>false</filtering>
                <excludes>
                    <exclude>log4j2.xml</exclude>
                </excludes>
            </resource>
            ...
        </resources>
        ...
   </build>
   ...
</project>


Define the PatternLayout

With that, let’s define now our custom PatternLayout to include all this info. Our log format, with the PatternLayout should be something like this

[LOG_LEVEL][DATE&TIME] app-details: [APP_NAME, APP_VERSION, API_LAYER, MULE_RUNTIME, ENVIRONMENT] thread: [THREAD_ID, THREAD_NAME] event: [CORRELATION_ID] processor: [PROCESSOR_NAME] logger: [LOGGER_NAME] - message: [LOG_MESSAGE]

For that we’ll need the following conversion specifiers in PatternLayout syntax:

SEMANTICPatternLayout Conversion Specifier
Log level%-5p
Date & Time%d
Application Name${project.name}
Application Version${project.version}
API Layer${api.layer}
Mule Runtime Version${app.runtime}
Environment${app.environment}
Thread Name%t
Thread ID%T
Correlation ID%X{correlationId}
Processor%X{processorPath}
Logger Name%c
Logger Message%m


Putting all together:

%-5p - %d - app-details:[${project.name}, ${project.version}, ${app.runtime}, ${app.environment}] - thread:[%t, %T] - event:%X{correlationId} - processor: %X{processorPath} - logger: %c - message: %m%n


Modify the log4j2 configuration

Open the log4j2.xml file. We’ll use the Console appender, just for testing, so that we can directly see our custom log format in the Anypoint Studio console when we’ll run the app. If we wanted to use it in our apps we’ll just need to replace the PatternLayout of your app with this one.

To define the Console appender and add our custom PatternLayout, add the following within the Appenders section:

<Console name="Console" target="SYSTEM_OUT">
    <PatternLayout pattern="%-5p %d app-details: [${project.name}, ${project.version}, ${api.layer}, ${app.runtime}, ${app.environment}] thread: [%T, %t] event: [%X{correlationId}] processor: %X{processorPath} logger: %c - message: %m%n " />
 </Console>


Add the Appender to the AsyncRoot element of the Logger section:

<Configuration>
    ...
    <Loggers>
        ...
        <AsyncRoot level="INFO">
            <AppenderRef ref="file" />
            <AppenderRef ref="Console" />
        </AsyncRoot>
    </Loggers>
</Configuration>


Our final log4j2.xml will look something like this:


Run the app

Let’s run the app from Anypoint Studio and generate some logs. Send some test requests and have a look at the Console. Our logs should now follow the layout we’ve created:

Here’s a full sample of a log event:

INFO 2024-12-02 11:28:50,582 app-details: [custom-log-format, 1.0.0-SNAPSHOT, SYSTEM 4.8.0, DEV] thread: [47, [MuleRuntime].uber.04: [custom-log-format].custom-log-format.CPU_LITE @75d9618d] event: [9a89e900-b0a0-11ef-b8c7-16946c581286] processor: custom-log-format/processors/0 logger: org.mule.runtime.core.internal.processor.LoggerMessageProcessor - message: Hello Gon!


Send the logs to Logstash

Once we know that our custom layout works, let’s modify again the log4j2.xml to add an HTTP appender to send the logs to Logstash. Get back to the log4j2.xml land set up a new appender by adding the following lines within the Appenders section:

<Configuration>
    <Appenders>
        ...
        <Http name="logstash" url="http://[YOUR_LOGSTASH_SERVER]:8080">
            <PatternLayout
                pattern="%-5p %d app-details: [${project.name}, ${project.version}, ${api.layer}, ${app.runtime}, ${app.environment}] 
                thread: [%T, %t] event: [%X{correlationId}] processor: %X{processorPath} logger: %c - message: %m%n " />
        </Http>
    </Appenders>
    ...
</Configuration>

Make sure you use the custom PatternLayout we defined in this post.

After that, add the appender to the root logger by adding the following lines within the Loggers section:

<Configuration>
    ...
    <Loggers>
        ...
        <AsyncRoot level="INFO">
            <AppenderRef ref="file" />
            <AppenderRef ref="logstash" />
        </AsyncRoot>
    </Loggers>
</Configuration>


Set up the Logstash Pipeline

For the Logstash Pipeline we’ll create a .conf file with input, filter and output:

Input - For the input of our Logstash Pipeline we will use the HTTP input plugin. Check out our post How to Send Mule Logs to Logstash with a Log4j HTTP Appender to see how to install it.


Once we’ve got the plugin, we can set up the input of our conf file like this:

input {
    http {
        host => "0.0.0.0"
        port => 8080
    }
}

Where:

  • host - represents the interfaces of our Logstash machine on which we’ll be listening. (All of them with the value 0.0.0.0)
  • port - Will be the port on which Logstash will be listening for events. Put here your preferred port.


Filter
In the filter section of our pipeline, we’ll add a grok filter to parse the different entities/fields we’ve defined in our custom log4j PatternLayout. Chek out the first post of this series to know more about Grok filters. These are the grok patterns we’ll use for each conversion specifier:

SEMANTICPATTERNLAYOUT CONVERSION SPECIFIERGROK PATTERN
Log level%-5p%{LOGLEVEL:log_level}
Date & Time%d%{TIMESTAMP_ISO8601:time_stamp}
Application Name${project.name}%{DATA:app_name}
Application Version${project.version}%{DATA:app_version}
API Layer${api.layer}%{DATA:api_layer}
Mule Runtime Version${app.runtime}%{DATA:mule_runtime}
Environment${app.environment}%{DATA:environment}
Thread Name%t%{DATA:thread_name}
Thread ID%T%{DATA:thread_id}
Correlation ID%X{correlationId}%{DATA:correlation_id}
Processor%X{processorPath}%{DATA:processor}
Logger Name%c%{DATA:logger_name}
Logger Message%m%{GREEDYDATA:log_message}


And the filter we need to match all our custom PatternLayout will be:

filter {
    grok {
        match => {"message" => "%{LOGLEVEL:log_level}%{SPACE}%{TIMESTAMP_ISO8601:time_stamp}%{SPACE}app-details:%{SPACE}\[%{DATA:app_name}, %{DATA:app_version}, %{DATA:api_layer}, %{DATA:mule_runtime}, %{DATA:environment}\]%{SPACE}thread:%{SPACE}\[%{NUMBER:thread_id}, %{DATA:thread_name}\]%{SPACE}event: \[%{DATA:correlation_id}\]%{SPACE}processor:%{SPACE}%{DATA:processor}%{SPACE}%{JAVACLASS:logger_name}%{SPACE}-%{SPACE}message:%{GREEDYDATA:log_message}"}
    }
}


Output: Here, we’ll define two outputs: 

  • The terminal, so that we can debug in real time the events that are coming from our mule app
  • A file, where we’ll set Logstash to write our mule app logs. 

For that, the output of our pipeline conf file will be:

output {
    stdout {
        codec => rubydebug
    }
    file {
        path => "/home/ubuntu/mule-app.log"
    }
}


Run the Logstash Pipeline and the Mule App

Time to see if our customizations work. First, open a terminal in your Logstash server and run the Logstash Pipeline with the previous command:

sudo /usr/share/logstash/bin/logstash -f [YOUR_CONF_FILE]

Once you see in the terminal that Logstash is listening for incoming events, then head back to Anypoint Studio and run the app. In a few seconds, right after the app is deployed to Studio, you will see in the Terminal of the Logstash server that Logstash starts to process events.


Verify

Lastly, after the Mule app has been deployed and is running, send some requests to the test endpoint (with curl or postman) and check the logstash terminal. 

First, verify we’re getting the log message that we set up in our Mule app


You will see that, thanks to the Grok filter, we’ve got now all the contextual information we added to the patternLayout in JSON fields. That’s going to be very useful for the logging aggregation system we’ll be sending these logs, because we’ll be able to build filters, queries, dashboards based on these fields

And secondly, let’s have a look at the file where we were writing the logs and check:

Tags

You may like these posts

Previous Post Next Post