How to externalize logs in Cloudhub 2.0

In a previous post, we learnt How to Externalize logs in Cloudhub 1.0. You might be wondering, what about Cloudhub 2.0? Do we have to follow the same procedure? The answer is no, the way CH2.0 manages log externalization has changed. But that’s all good news, it changed for better. Now it’s easier and quicker.

Why Changing the externalization procedure in Cloudhub 2.0?

When we discussed CH1.0 we saw that during the deployment of our app, Cloudhub automatically overrides our log4j configuration and sets up only one appender to Cloudhub. We also saw that, in order for us to change that, we needed to open a support ticket and ask the Mulesoft team to enable the option of disabling the log forwarding to Cloudhub.
it might not make sense at first, but we learnt that the reason why Cloudhub does it is to ensure that logs are always available in Cloudhub.

After a few years with that procedure, here at Mulesoft we realized that the process was a little bit confusing for the customers. This has been taking into account in the design of Cloudhub 2.0 and now the process is much more intuitive and self-managed. Let’s see that in detail:

How Cloudhub 2.0 has improved the procedure

Keeping the logs in Cloudhub is a good practice. For many reasons. First because, even though you might me sending your logs to a logging aggregation system like ELK, Splunk or Datadog, you will always have another copy of your logs in the Anypoint platform. With searching capabilities and integrated in the platform. With no effort or extra cost. And secondly because it will help a lot our Mulesoft support team when you would be opening a ticket, as they will have access to your logs. That’s why CH1.0 was overriding your log4j configuration.

In Cloudhub 2.0, we still want to ensure that we can keep the logs in the platform. This is how it works:

When you deploy an app in CH2.0, you will see a checkbox for Log Forwarding in the Monitoring tab:

The option is checked by default.
If the option is selected, Cloudhub will keep all the configuration you set up in the log4j2.xml file of your app but it will add the Cloudhub appender. So, now, Cloudhub does not replace your log4j2.xml, it merges it with its own template.
This way, your appenders for externalization will be kept and whether or not you added the Cloudhub appender, the final log4j2.xml will have it.
On the other hand, if you uncheck the option, Cloudhub won’t do anything and your log4j configuration will remain as it is in your mule project.
So, in summary, this is what you need to know:

You can set up and test your log4j appenders in dev. If that works in Anypoint Studio, it will work in CH2.0
You don’t have to worry about the Cloudhub appender. No need to add it to your log4j2.xml file anymore. At deployment time, you just decide if you want to have logs in Cloudhub or not with the checkbox
Good news - You don’t have to open a support ticket you disable cloudhub logs. No need for that, only the checkbox
And lastly, remember - same as we saw for CH1.0: if you decide not to send logs to Cloudhub, the logs will be your entire responsibility. If there’s an issue with your logs destination and you lose some logs, Mulesoft won’t be able to help you.

Example - Externalizing logs in Cloudhub 2.0 to Splunk

Let’s just see a practical example to understand how things work in real life. In these previous posts, we saw how to set up log appenders to send our mule logs to Splunk:

Now, If your final tests with those appender were successful you're ready to deploy your app to Cloudhub 2.0. Go to Runtime Manager, deploy the app and select Cloudhub 2.0 as your deployment target. Provide your own settings and remember to verify the Monitoring tab.

By default, you’ll see the Log forwarding option activated. Try to deploy a sample app with this option activated and verify that you can see the logs in Splunk and in Cloudhub, even though you didn’t include the cloudhub appender in your log4j configuration.

In Cloudhub:

In Splunk:

And lastly, redeploy the app, but this time with the log forwarding option deactivated. Verify that now you only see the logs in Splunk

As a reference, I’ll leave the full log4j2.xml configuration file for both the http and the splunk appender.

Log4j configuration with HTTP appender for Cloudhub 2.0:

<?xml version="1.0" encoding="utf-8"?>
<Configuration>

    <Appenders>
    <!--  This is the default appender to write logs in a local file -->
        <RollingFile name="file"
            fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log-forwarding-ch1-http-appender.log"
            filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log-forwarding-ch1-http-appender-%i.log">
            <PatternLayout
                pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n" />
            <SizeBasedTriggeringPolicy size="10 MB" />
            <DefaultRolloverStrategy max="10" />
        </RollingFile>
        
    <!-- Here's the HTTP appender for Splunk.Replace with the hostname, port and token of your Splunk instance -->
        <Http name="Splunk"
            url="http://[YOUR_SPLUNK_SERVER]:8088/services/collector/raw">
            <Property name="Authorization"
                value="Splunk [YOUR_HEC_TOKEN]"></Property>
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
        </Http>
    </Appenders>

    <Loggers>
        
        <AsyncLogger name="org.mule.service.http" level="WARN" />
        <AsyncLogger name="org.mule.extension.http" level="WARN" />

        <!-- Mule logger -->
        <AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO" />

        <AsyncRoot level="INFO">
            <AppenderRef ref="file" />
            <AppenderRef ref="Splunk" />
        </AsyncRoot>
        
    </Loggers>

</Configuration>

Log4j configuration with Splunk appender for Cloudhub 2.0:

<?xml version="1.0" encoding="utf-8"?>
<Configuration status="INFO" name="cloudhub" packages="com.splunk.logging,org.apache.logging.log4j">

    <Appenders>
        <!--  This is the default appender to write logs in a local file -->
        <RollingFile name="file" fileName="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log-forwarding-ch1-splunk-appender.log"
                 filePattern="${sys:mule.home}${sys:file.separator}logs${sys:file.separator}log-forwarding-ch1-splunk-appender-%i.log">
            <PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n"/>
            <SizeBasedTriggeringPolicy size="10 MB"/>
            <DefaultRolloverStrategy max="10"/>
        </RollingFile>
        
        <!-- Here's the Splunk appender. Replace it with the hostname, port and token of your Splunk instance -->
        <SplunkHttp name="SPLUNK"
            source="${env:APP_NAME}"
            host="${env:POD_NAME}"
            sourceType="mule-app"
            url="http://[YOUR_SPLUNK_SERVER]:8088/"
            token="[YOUR_HEC_TOKEN]"
            index="[YOUR_SPLUNK_INDEX]">
            <PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
        </SplunkHttp>
    </Appenders>

    <Loggers>
        <!-- Http Logger shows wire traffic on DEBUG -->
        <!--AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger" level="DEBUG"/-->
        <AsyncLogger name="org.mule.service.http" level="WARN"/>
        <AsyncLogger name="org.mule.extension.http" level="WARN"/>

        <!-- Mule logger -->
        <AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>

        <AsyncRoot level="INFO">
            <AppenderRef ref="file"/>
            <AppenderRef ref="SPLUNK" />
        </AsyncRoot>
    </Loggers>

</Configuration>