How to Externalize Logs with Log4j in Runtime Fabric

Externalizing the mule apps logs to platforms like Splunk, ELK or Datadog is a very common (and good) practice for your Logging Strategy. One of the best options to send these logs to an external logging system is to use the Log4j framework and create appenders for our log destinations.

In Runtime Fabric, the log4j customizations in our apps are not enabled by default, we need to change the configuration of our RTF instance to allow these customizations.
In this post, we will see step by step how set up our RTF instance to externalize logs using log4j.

Prerequisites

To show the full configuration, we will see in this post how to setup Runtime Fabric to send the logs to Splunk. The RTF configuration will be the same for any other appender, so if you are sending the logs to ELK, Datadog or any other logging aggregation the only difference will be the configuration of the log appender for that log destination.
To follow this tutorial we will need:

An instance of Runtime Fabric - You can use AKS, EKS, GKE, Openshift... For testing purposes the quickest way to get an RTF instance would be to use a K3s cluster. Have a look at this post to see How to Install Mulesoft Runtime Fabric on K3s
An instance of Splunk - For testing purposes you can quickly get your Splunk trial up and running using Docker. Check out this post to see How to install Splunk Enterprise in a Docker Container

Create a Mule app for testing

We’ll create a very basic app for testing with the following elements:

An HTTP listener - A simple GET /hello
Two Logger processors to show how the app writes to the log

First flow to inform of the start of the flow
Second flow to inform of the end of the flow

For that:

Head over to Anypoint Studio
Create a new flow. Drag and drop from the Mule palette to the canvas: an HTTP listener and two Loggers.
Configure the HTTP listener to be listening on the endpoint http://localhost:8081/hello for GET requests.
For the loggers

The first logger will include the custom message - “The flow has started”
The second logger will include the custom message - “The flow has ended”

Log4j Configuration

We’ll modify the Log4j configuration of our app to add a new appender for Splunk. We can use a generic HTTP appender or the specific Splunk appender. Check out this post if you want to know the differences between both appenders.
The configuration of these appenders is something we’ve previously done in previous posts.

Follow the steps in the post How to setup log appenders for Splunk - HTTP Appender to configure your app with the HTTP appender
Follow the steps in the post How to setup log appenders for Splunk - Splunk Appender to configure your app with the Splunk appender

Test the log forwarding in Anypoint Studio

Before we deploy our app to Runtime Fabric verify that logs are sent to Splunk when you run your project in Anypoint Studio and send requests to the /hello endpoint of our app.

If we see the logs in Splunk, our app is ready to be deployed to Runtime Fabric

RTF Configuration

Before we deploy our app to Runtime Fabric we need to enable the log forwarding in our RTF instance. For that, follow the next steps:

Open a terminal session and make sure your kubectl is pointing to your RTF K8s cluster.
Run the command:

kubectl config view

Verify that the current context of the kube-config is pointing to our RTF cluster.
In this example, I’m logged in to the controller node and I can see the current-context is pointing to its own.

To enable log forwarding in RTF we need to modify the secret called custom-properties. This secret in RTF stores some configuration details of the RTF instance. We’ll change the value of the CUSTOM_LOG4J_ENABLED field from false to true.
To modify the secret run the command

kubectl edit secret -n rtf custom-properties

You will see the following values:

Secrets in K8s are stored base64-encoded, so you will notice that your current value is ZmFsc2U= which is false in base64.
Edit that value and replace it with dHJ1ZQ== which is true in base64.

Save and exit the file
Next, restart the Runtime Fabric Agent pod by running the command

kubectl rollout restart deployment agent -n rtf

Deploy the app

We’re now ready to deploy our app. For that, go to Runtime Manager > Applicatons and click on Deploy application
Upload the jar file of our app, select your RTF instance as Deployment target and set up the rest of the configuration you want for your app.

Lastly, click on the Monitoring tab. You will notice there’s an option for Forward application logs to Anypoint Platform. In RTF, you can only send logs to Anypoint if you are on the Titanium subscription. So, if you’re not on Titanium, don’t worry if you don’t see the logs in Anypoint even if you check this option.

But if you do have the Titanium subscription the best practice is to activate this option. This will keep your log forwarding to Splunk but it will also add the appender to Cloudhub so that you can have an additional copy of your logs in the Anypoint platform.

Final Test

Once your app is deployed in Runtime Fabric it’s time for the final test. Hit your /hello endpoint in your RTF app and verify you’re getting the logs in Splunk.