How to send Mule logs to SQS

Forwarding logs to a message broker is a solid architectural choice for ensuring reliability, scalability, and flexibility in a distributed environment. However, the decision depends on your system's complexity, performance requirements, and budget.

Here are some scenarios when this is a good fit:

• High-Volume Logging: When dealing with large log volumes that could overwhelm a centralized logging system.
• Distributed Systems: For logs generated across multiple services or microservices.
• Reliability and Durability: When you need to ensure that logs are not lost due to temporary failures.
• Multiple Consumers: When logs need to be processed by different systems (e.g., real-time monitoring, long-term storage, and analytics).

In this post we will see an example of that scenario. We will see, step by step, how to forward the logs of our Mule Apps using a log4j appender to an AWS SQS Queue. 

Set up SQS

Before configuring our Mule App we will have to get our AWS SQS ready to get our logs. For that we’ll have to create a queue and a user with permissions to send messages to that queue in our AWS account. 

Create IAM user

• Open the AWS Management Console and go to the IAM service. Then, under the Access Management section click on users and Create User

• In Step 1 - Provide a name for our sqs user (in our example we’ll name it mule-sqs). No need to provide access to the management console. Click Next

• In Step 2 - Set Permissions choose the Attach policies directly option and select the AmazonSQSFullAccess policy. Click Next

• In Review and Create click on Create user
• Once the user is created, click on the user name on the list of users.
• Next, click on Create access key

• Choose the Application running outside AWS option Click on Next

• Provide a tag if you wish
• After that click on Create access key
• Lastly, take note of the Access key and Secret access key. We’ll use them in the next steps. Click Done

Lastly, from the summary, take note as well of the ARN of the user. We’ll need it in to set up our queue.

Create a Queue for the mule logs

Now, let’s create the Queue in SQS where we will send our Mule Logs. From the AWS Management Console go to the Simple Queue Service and click on Create queue

Next, provide a name for our queue. We’ll name it mule-logs in our example:

Leave the Configuration and Encryption sections with the default values for now. 
Modify the Access Policy - In this section, we’ll choose the Basic method and specify the user that we created as the user that can send messages to the queue. 

This is the user we’ll set up later on in our Mule app. This way, following the Principle of Least Privilege (PoLP) we’re making sure our user can only interact with SQS and in particular with ONLY this queue.

Create the Mule App

Now, let’s head over to Anypoint Studio and create an app for testing. For this tutorial, we’ll use Mule Runtime version 4.6.8 and Java 17. Create a New Mule Project and drag & drop the following elements to our flow:

• An HTTP listener - A simple GET /hello
• A Logger processor to show how the app writes to the log. Write any text in the message that can help you identify the log is coming from this component when we’ll see the logs in SQS
• A Set Payload processor to create a response for our test endpoint. Enter any text that confirms the app is running well

Modify POM

The SQS appender that we will use in this tutorial has been developed by Avioconsulting. The appender will make use of additional java libraries that we need to import to our project. For that, we just need to add the dependency of the solution. Open the pom.xml file of the project and add the following under the dependencies section:

<project>
    <dependencies>
        ...
        <dependency>
            <groupId>com.avioconsulting</groupId>
            <artifactId>log4j2-sqs-appender</artifactId>
            <version>1.1.0</version>
        </dependency>
        ...
    </dependencies>
    ...
</project>

At the time of writing this post, the latest version available in the Maven Central repository is 1.1.0. Double check if there’s a newer version available before adding the dependency.

Configure the Appender

The last step of the configuration is the Appender. Open the log4j2.xml file, located at src/main/resources in your Mule project and do the following edits:
Add the package of the dependency we added to the Configuration element:

<Configuration packages="com.avioconsulting.log4j" >

Add a new Appender element within the Appenders section with the following configuration:

<Appenders>
    ...
    <SQS name="SQS" awsAccessKey="[YOUR_AWS_ACCESS_KEY]"
        awsRegion="[YOUR_AWS_REGION]" awsSecretKey="[YOUR_AWS_SECRET]"
        maxBatchOpenMs="10000" maxBatchSize="5"
        maxInflightOutboundBatches="5" queueName="[YOUR_SQS_QUEUE]">
        <PatternLayout pattern="%d{ISO8601} %-5p PID[%X{correlationId}][%c{1}] - %m %n" />
    </SQS>
</Appenders>

Where:

• awsAccessKey and awsSecretKey - Paste here the AccessKey and Secret of the mule-sqs user we created in the first step of this tutorial
• awsRegion - The AWS region where we’ve created the queue
• queueName - The name of the queue we created in the previous step.
• For now, just for testing purposes, we’ll be using the PatternLayout, although it would probably be better the JSONLayout in a production environment.

Lastly, link this appender to the root logger. Add the following in the Loggers section

<Loggers>
    ...
    <AsyncRoot level="INFO">
        <AppenderRef ref="file" />
        <AppenderRef ref="sqs_cloud" />
    </AsyncRoot>
</Loggers>

With that, our full log4j2.xml file should look like this:

Test

Save the Mule Project and Run it. Verify the apps builds and deploy correctly and then send some request to the test endpoint.
Now, head back to the AWS management Console > SQS and click on the mule-logs queue. We should now see that there are new messages in the queue.