How to Set Up Elastic for our Mule Logs

In our previous posts, we learnt how to install Elasticsearch and Kibana to be used for the observability of our Mule apps. But before we send any metrics or logs, there are a few previous configuration steps we need to do on our Elastic and Kibana instances.

We need to:

Create an Index for the Mule Logs
Create a Role and a User for our Mule Apps
Test the Index and User
Create a Data View in Kibana

Check out our previous posts if you need to get your Elastic stack:

Let’s dive in!

Create an Index for the Mule Logs

An index in Elasticsearch is a data structure used to store and organize documents. It is analogous to a database in a traditional relational database management system (RDBMS). In Elasticsearch, the index is the highest-level entity under which all data is grouped and stored. In our case, for our Mule apps, we will create an index, which will be like the database where we store our logs.

From the home page go to Stack Management

Next, from the left panel, go to Index Management and click on Create Index

Provide a descriptive name for the index. We’ll name it mule-logs

Alternatively, we can also create the index using the Elasticsearch REST API, sending the following request (in curl)

curl --request PUT 'http://[YOUR_SERVER]:9200/mule-logs' -u elastic:[YOUR_PASSWORD]

Create a Role and a User for our Mule Apps

Next, we will create a specific user to be used by our Mule Apps to connect to elasticsearch and ingest the logs. We don’t want to use the elastic superuser for this. As a best practice, we should use a user with the minimum permissions required to write ONLY to the index we’ve created. For that, we need to do it in two steps - first create a custom role with those permissions and second create a user and assign it to that custom role

Create a Role

First we’ll create a custom role with permissions only to our dedicated index for the mule logs. From the home page go to Stack Management and then click on Roles from the left panel. Next, click on Create role

In the

Create Role page, provide a name and description to the new role. Then, select the index we’ve created for our mule logs and add read and write privileges

Create User

Go to Stack Management and then, under the Security section, click on Users from the left panel. Next, click on Create user.

On the Create user page, provide a name and password for the user. Then, select the custom role we’ve created in the previous step in the Privileges/Roles dropdown. Click Create user

Create Role and User via REST API

Alternatively, we can create the Role and the User using the Elasticsearch REST API.

Create a Custom Role:

curl -X POST "http://localhost:9200/_security/role/[MULE_CUSTOM_ROLE]" -H "Content-Type: application/json" -u elastic:[YOUR_PASSWORD] -d '{
  "indices": [
    {
      "names": [ "[MULE_CUSTOM_ROLE]" ],   # Specify the index to apply this role
      "privileges": [ "read", "write" ] # Grant read and write privileges
    }
  ]
}'

Create a user and assign it the custom role

curl -X POST "http://localhost:9200/_security/user/[MULE_USER]" -H "Content-Type: application/json" -u elastic:[YOUR_PASSWORD] -d '{
"password" : "user_password",
"roles" : [ "MULE_CUSTOM_ROLE" ],
"full_name" : "Full Name of User",
"email" : "user@example.com“
}‘

Test the Index and User

Let’s verify our new User works. From postman, let’s create a new entry for our mule-logs index. For that, send the following request:

Method: POST
URL: https://[YOUR_SERVER]:9200/[MULE_LOGS_INDEX]/_doc
Set up Basic Auth with the credentials of the user we’ve created
In the body, add a valid json

{
    "message": "It works!"
}

Try also to modify the username or the password and you’ll see you get an authentication error. That means our user/role we created is working.
Let’s now try to retrieve the entries of our mule logs index. For that, create a new Postman request:

Method: GET
URL: https://[YOUR_SERVER]:9200/[MULE_LOGS_INDEX]/_search?pretty
Set up Basic Auth with the credentials of the user we’ve created

We should verify that there’s one hit, corresponding to the element we’ve just added.

Create a Data View in Kibana

Let’s now see the entries of our index in Kibana. For that we need to first create a Data View for our mule-logs index. This will be the Data view that we’ll use to see the logs coming from our Mule apps.
Go to the menu button on the top left corner and click on Dashboards

Next click on Create data view. (If you don’t see this option, that’s very likely because you still don’t have any data in your Elasticsearch instance. Use the previous POST request in postman to create a test entry and get back)

Next, provide a name for this view and enter the name of the index we created for our mule logs in the Index pattern textbox. Click on Save data view to Kibana

Now, let’s see if we can see the content of our mule-logs index. Go to Analytics > Discover and you should see now the message we created from Postman

With that, we’re ready to send logs from our mule apps.

In the next post, we will see how to configure our Mule Apps to send logs to our Elastic instance.