Typically, they would customize their log4j config file, adding their own
appenders. And even if they've tested it from Anypoint Studio and verified
that the application running locally sends logs to their external system, when
they deploy the app to Cloudhub they just don't get the logs. To be more
specific, they get logs but only in Anypoint, no matter if they've added or
not the Cloudhub appender.
So, what's going on? why does it work locally but not in CH? The appenders
configuration looks good, actually it works locally.
Don't panic, let's try to break it down...
How logging works in Cloudhub 1.0
If you're reading this, you probably know what log4j is. It is the logging
framework, written in Java, that Mule uses in all the apps.
The framework is highly customizable and is configured with an XML file, the
log4j2.xml located under src/resources.
This file specifies the configuration of the three main components of the
framework:
- Loggers - Responsible for capturing logging information
- Appenders - where we define the different destinations for our logs
- Layout - where we define the style and format of our logs
When we deploy an app to Cloudhub 1.0, the log4j.xml of your app is
replaced by another one, which follows a template. This template basically
specifies the configuration of the appender for Cloudhub. That's the reason
why, no matter what you put in your log4j2.xml (even if you don't include an
appender for Cloudhub), the application, by default will send the logs
exclusively to the Anypoint platform. It's an internal mechanism that
Mulesoft uses to ensure the apps will always send logs to Anypoint.
Ok, got it. So, does that mean I can't externalize logs?
No, not at all. Log externalization is supported (and recommended!) for
Cloudhub apps but it requires some extra steps.
The key thing to understand here is that for each application we want to
externalize logs, we need to tell Cloudhub not to replace your custom
log4j2.xml. And for that, we only need to enable one option.
Go to Runtime Manager > Applications and click on the app you want to
externalize the logs from. In the settings page of your app, Runtime tab,
you need to select the option Disable Cloudhub logs in the Runtime options
section.
Don't go crazy if you don't see that option in your Anypoint platform,
that's normal. To make this option available you need to request it on
Help Center and
the support team will enable it for you.
After you click on that option, you'll get a window with a couple of warnings
from Mulesoft. Don't worry, if you know what you're doing you can tick both
checks and Apply Changes. This is just a safe harbour statement from
Mulesoft making you aware that if you decide to do that the responsibility
of getting and maintaining logs will be transferred from Mulesoft to you.
This means:
- You will now be responsible of setting up correctly your log4j configuration, logging somewhere and maintaining that.
- If your logs are outside the Anypoint platform, the Mule Support team cannot have access to them, so troubleshooting will be more difficult. They will probably ask you to provide the logs.
But don't worry, even if you select Disabling Cloudhub logs you can
still send your logs to Anypoint, you just need to include the Cloudhub
appender into your log4j2.xml file.
It might seem a little bit confusing. You just need to remember that the
Disabling Cloudhub logs only means that Cloudhub will not replace
your log4j configuration.
What does Mulesoft recommend to do?
As we mentioned before, log externalization is, in many cases, a best practice that Mulesoft encourages you to do for your apps.
If you do so, and you've got your own logging management system, where you'll be sending the mule apps logs, the other recommendation is that you also send those logs to Anypoint.
One of the cool features of the log4j framework is that it allows you to define multiple appenders, meaning you can send your logs simultaneously to multiple locations.
So, make sure that along with the appender for your preferred logging system you include the appender for Cloudhub. This way, troubleshooting with Mulesoft will be much easier with the support team.
My personal recommendation is that if you're planning to externalize logs for all your applications, you include the configuration of the appenders in a template for your developers.
If you do so, and you've got your own logging management system, where you'll be sending the mule apps logs, the other recommendation is that you also send those logs to Anypoint.
One of the cool features of the log4j framework is that it allows you to define multiple appenders, meaning you can send your logs simultaneously to multiple locations.
So, make sure that along with the appender for your preferred logging system you include the appender for Cloudhub. This way, troubleshooting with Mulesoft will be much easier with the support team.
My personal recommendation is that if you're planning to externalize logs for all your applications, you include the configuration of the appenders in a template for your developers.
Example - Externalize logs in Cloudhub 1.0 to Splunk
We’ve seen in other posts of this blog how to set up log4j Appenders for Splunk. Let’s put all together and see what the full log4j2.xml file for an app in Cloudhub 1.0 should look like to send logs to Splunk and Cloudhub.
First, have a look at the necessary steps to configure Splunk:
First, have a look at the necessary steps to configure Splunk:
- How to set up your Splunk instance to get your Mule logs
- How to setup log appenders for Splunk - HTTP Appender
- How to setup log appenders for Splunk - Splunk Appender
Here below you can find a basic log4j configuration with two appenders:
- An HTTP Appender to send logs to Splunk
- The Cloudhub Appender to get the logs also in the Anypoint control plane
<?xml version="1.0" encoding="utf-8"?>
<Configuration status="INFO" name="cloudhub" packages="com.mulesoft.ch.logging.appender,com.splunk.logging,org.apache.logging.log4j">
<Appenders>
<Http name="Splunk"
url="http://[YOU_SPLUNK_SERVER]:8088/services/collector/raw">
<Property name="Authorization"
value="Splunk [YOUR_SPLUNK_TOKEN]"></Property>
<PatternLayout pattern="%m%n"></PatternLayout>
</Http>
<Log4J2CloudhubLogAppender name="CLOUDHUB"
addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
appendMaxAttempts="${sys:logging.appendMaxAttempts}"
batchSendIntervalMs="${sys:logging.batchSendInterval}"
batchMaxRecords="${sys:logging.batchMaxRecords}"
memBufferMaxSize="${sys:logging.memBufferMaxSize}"
journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
journalMaxFileSize="${sys:logging.journalMaxFileSize}"
clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</Log4J2CloudhubLogAppender>
</Appenders>
<Loggers>
<AsyncRoot level="INFO">
<AppenderRef ref="Splunk" />
<AppenderRef ref="CLOUDHUB" />
</AsyncRoot>
</Loggers>
</Configuration>