How to choose the right destination for our Mule Logs

In MuleSoft, logs can be sent to different destinations. Choosing the right destination for our logs is a key element of our Logging Strategy. In this post, we will see some of the key aspects we should consider to define the right destination for the logs of our mule apps. 

Here’s a list of requirements to take into consideration to choose the most suitable destination for our logs:

Infrastructure available

Many times, in organizations there is an existing global logging strategy with a predefined set of tools. In these cases we’ll have to adapt our Mule logging strategy to that and send the logs to the existing repository for all logs.

Centralization

In distributed architectures such as microservices or api led connectivity, it’s a good practice to send the logs of all the app to the same location so that they can be analyzed as a whole. Using correlation ids for our logs will allow us to track end-to-end a transaction in we have all the logs centralized.

Log Analysis and Searching capabilities

Our log destination is not only a repository for our logs. It will be also the place where we’ll be analyzing the info in our logs for debugging and troubleshooting. So, it’s important that our logs destination provides us with basic features like filtering, aggregation, tagging or more advanced features like a search engine, a query syntax etc. It’s also important to evaluate the performance of these features - how complex are the queries we’ll use and how performant they are. Is it fast enough with the volume of your logs to do troubleshooting/debugging?

Retention Policy and Automation

Many companies have policies to 

• keep the logs a minimum period of time 
• delete the logs after a period of time
• archive/move logs to another tier/destination

So, if that’s the case we’ll have to check the capabilities of our logging system to set up retention policies and provide some automation for the deletion, archiving or moving of our logs.

Security

The information in our logs is a very valuable asset for an organization, and as such it has to be protected. 

• Access to logs should be restricted only to the IT operators that will work with them. 
• Also, logs might contain sensitive information. If that’s the case we’ll need to encrypt or mask that info. 

For these reasons, plan ahead and check if your destination for logs provides the right level of security.

Performance

Make sure your logs destination is able to handle all the traffic that will be coming from your mule apps and/or other systems. If not, we’ll be creating a point of failure and eventually losing the logs.

Reliability

What happens if your logs destination become unavailable? How reliable is the network segment between your mule apps and your logs destination? Can we afford not having logs for a period of time? In these scenarios, we need to think of the reliability mechanisms we can provide to our logs:

• Protect your store logs with a backup mechanism
• Having multiple logs destinations
• Downloading logs from Anypoint periodically as a backup
• Shipping logs asynchronously

Alerting

With a proper definition of what info our mule apps will be writing to the logs and when they will do it, it can be possible to track key metrics about the health of your apps and trigger alerts when something is not going well. Does your logs destination have the ability to set up this kind of alerts?

Cost

Logs will consume space in our local or cloud storage. Storage comes always with a cost, so it’s important that we estimate how much storage we’ll need for our mule apps according to our log retention policies and how much that will cost.

With all of this in mind, in the next post, we’ll review some of the most common types of destinations/systems for our logs. All of the aspects seen above will be now what will determine what the most suitable destination is for our logs.