How to Install Logstash on Linux

Logstash is an open-source, server-side data processing pipeline tool developed by Elastic. It is part of the Elastic Stack(formerly known as the ELK Stack), which includes Elasticsearch, Kibana, and Beats. Logstash is designed to collect, process, and forward data from various sources to destinations such as Elasticsearch, databases, or file systems.

For us, Mulesoft Architects and Developers, it’s a solution that can help us with the logs of our Mule Apps.

Logstash can significantly enhance how we handle and analyze MuleSoft logs by centralizing, processing, and transforming the data for better observability. These are some use cases in which Logstash can help us with the Mule logs:

Centralized Log Collection: MuleSoft generates logs across different applications and services. Logstash can aggregate these logs from multiple MuleSoft instances or servers into a single, centralized location.
Log Parsing and Transformation: MuleSoft logs are often in plain text or JSON format. Logstash filters (like grok or json) can parse these logs to extract meaningful fields (e.g., timestamp, error code, application name).
Data Enrichment: Logstash can enrich MuleSoft logs with additional metadata, such as:

Host information
Environment tags (e.g., dev, staging, production)
IP address details

Sending Data to Elasticsearch for Analysis: After processing, Logstash can forward the enriched logs to Elasticsearch, making them searchable and visualizable in Kibana.
Correlating Logs Across Systems: If MuleSoft is part of a larger system architecture, Logstash can correlate MuleSoft logs with logs from other systems (e.g., databases, APIs) to provide a holistic view of end-to-end transactions.

In this post, we will see how to install Logstash on Ubuntu Server so that we can create different scenarios with Mulesoft logs in further posts

Prerequisites

Software requirements

An instance of Elasticsearch running on a separate server. We’ll keep Logstash and Elasticsearch on different machines for better performance. Optionally (like in this tutorial) we can use Kibana installed on the Elasticsearch server to have a web UI. If not, Elasticsearch can be managed via REST API.

Hardware requirements

The hardware requirements for installing and running the ELK Stack (Elasticsearch, Logstash, and Kibana) depend on your use case, data volume, and expected workload. Below are minimum and recommended resource requirements for each component of the ELK Stack.

Minimum Requirements (Small-Scale Testing) - For development or testing purposes with low data volume (less than 1,000 events per second):

CPU: 2 cores
RAM: 4 GB
Disk Space: 20 GB (SSD recommended)
Network: 1 Gbps

Recommended Requirements (Production) - For production environments handling moderate to large data ingestion (10,000 - 20,000 events per second):

CPU: 4-8 cores (more for complex pipelines or higher throughput)
RAM: 8-16 GB (or more, depending on load)
Disk Space: 100+ GB (SSD strongly recommended)
Network: 1 Gbps or higher

High-Performance/Enterprise Setup - For large-scale data processing or highly complex pipelines (High-throughput ingestion (50,000+ events per second), complex transformations, and enrichment):

CPU: 8-16 cores or more
RAM: 32-64 GB (consider higher for very large pipelines)
Disk Space: 500 GB - 1 TB (SSD or NVMe recommended)
Network: 10 Gbps

This tutorial is just a proof of concept, we’ll install Logstash on an Ubuntu Server 24.04 LTS with 2 cores and 4 GB RAM.

Install Java JDK

Logstash uses Java, so the first thing we’ll install will be the Java JDK. Elastic 8 requires min Java 17.
If you need more details check out this post on How to install Java on Ubuntu Server

Run the commands

sudo apt-get update
sudo apt-get install openjdk-17-jdk

Install Logstash

To install Logstash, we need to add the Elastic repository to our system, so that APT can find the binaries of the ELK stack. For that, follow the next steps:

Download and install the public signing key. A public signing key is cryptographic key used to verify the authenticity and integrity of software packages retrieved from an APT repository. It ensures that the packages come from a trusted source and have not been tampered with during transit.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Install the apt-transport-https packages. The apt-transport-https package provides support for fetching APT repository files over HTTPS

sudo apt-get install apt-transport-https

Save the repository definition

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Now, we can Install the Logstash Debian package from its repository. As a best practice, make sure your version of Logstash matches the version of Elasticsearch, otherwise we might get issues when connecting with each other.

Before downloading the Logstash package, let’s have a look at the versions available for Logstash in our package manager

sudo apt list kibana -a

sudo apt-get update && sudo apt-get install logstash=[VERSION]

where VERSION is the Logstash version to install. In this tutorial we’re using version 1:8.15.3-1, as this is the version of our Elasticsearch instance.

Create a Role and a User for Logstash

Next, we will create a specific user to be used by Logstash to connect to elasticsearch and ingest the logs. We don’t want to use the elastic superuser for this. As a best practice, we should use a user with the minimum permissions required to write ONLY to the index created for Logstash. For that, we need to do it in two steps - first create a custom role with those permissions and second create a user and assign it to that custom role

Create a Role

First we’ll create a custom role with permissions only to create indexes and send logs to those indices. From the home page go to Stack Management and then click on Roles from the left panel. Next, click on Create role

In the Create Role page, provide a name and description to the new role

For our Logstash instance we need the following privileges (according to Elastic official docs)
Cluster Privileges

manage_index_templates
manage_ilm
monitor

Index Privileges - As index we’l use the wildcard *

create
create_index
manage
manage_ilm
write
auto_configure

Alternatively, to create the custom role, we can use the Elastic REST API. For that, run the following command to create a role with the required permissions.

With CURL:

curl -X POST 'http://[YOUR_ELASTICSEARCH_SERVER]:9200/_security/role/[MULE_LOGSTASH_WRITER]' \
-u [YOUR_ELASTIC_ADMIN_USER]:[YOUR_ADMIN_PASSWORD] \
--header 'Content-Type: application/json' \
--data '{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"],
  "indices": [
    {
        "names": ["filebeat-*"],
      "privileges": ["create", "auto_configure", "create_index", "manage", "manage_ilm", "write"]
    }
  ]
}'

Where:

YOUR_ELASTICSEARCH_SERVER - Hostname/DNS name that identifies your elasticsearch server in the network
MULE_LOGSTASH_WRITER - The name of the custom role you want to create
Default port for Elasticsearch is 9200. Change it if you’re using another port in your Elasticsearch instance.
[YOUR_ELASTIC_ADMIN_USER] - An admin user with permissions to create roles and users on elastic. In here, we can use the elastic superuser
[YOUR_ADMIN_PASSWORD] - The password of your admin user
The body of the request will contain all the Privileges we defined previously for our logstash custom role

Create a User and assign the custom role

Next, we’ll create a user and assign the custom role we’ve defined, in our case mule-filebeat
With CURL:

curl -X POST 'http://[YOUR_ELASTICSEARCH_SERVER]:9200/_security/user/[MULE_LOGSTASH_WRITER]' \
--header 'Content-Type: application/json' \
-u [YOUR_ELASTIC_ADMIN_USER]:[YOUR_ADMIN_PASSWORD] \
--data '{
  "password": "[LOGSTASH_USER_PASSWORD]",
  "roles": ["LOGSTASH_CUSTOM_ROLE"],
  "full_name": "Logstash User",
  "email": "logstash@mulesoft.com"
}'

Where:

MULE_LOGSTASH_WRITER - The name of the user you want to create
Default port for Elasticsearch is 9200. Change it if you’re using another port in your Elasticsearch instance.
[YOUR_ELASTIC_ADMIN_USER] - An admin user with permissions to create roles and users on elastic. In here, we can use the elastic superuser
[YOUR_ADMIN_PASSWORD] - The password of your admin user
The body of the request will contain:

password: Sets the password for the user.
roles: Assign the custom role we’ve created
full_name : Provides a descriptive/display name of the user

Check the connection to Elasticsearch

Another thing we should do before setting up Logstash is to verify that our Logstash server can connect to our Elasticsearch instance. This way we can make sure the connection details are correct. For that, from the Logstash server run the curl command:

curl http://[YOUR_ELASTICSEARCH_SERVER]:9200 -u elastic:[YOUR_PASSWORD]

Test Logstash

Before we start using Logstash, it’s a good idea to run some simple tests to verify we’ve installed it successfully and it processes data correctly:. In order to do so, we’ll run a couple of tests. The first one to verify Logstash has been installed successfully and can run. The second one to verify Logstash can send data to Elasticsearch

Test 1: Simple Pipeline

Let’s create a simple pipeline that read from the terminal input and writes to the terminal output. Create a simple Logstash config file (test.conf):

sudo nano /etc/logstash/conf.d/test.conf

Paste the following content:

input {
  stdin {}
}

output {
  stdout {
    codec => rubydebug
  }
}

Run Logstash with the Test Configuration - Start Logstash manually to process input from the terminal:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

Expected behavior:

Logstash should start and wait for input.
Type a simple message, e.g., Hello, Logstash!, and press Enter.

Expected output:

{
    "message" => "Hello, Logstash!",
    "@version" => "1",
    "@timestamp" => "2024-01-01T00:00:00.000Z",
    "host" => "your_hostname"
}

Press Ctrl+C to stop Logstash.

Test 2: Pipeline to connect to Elasticsearch

Let’s now create a simple pipeline with elasticsearch in the output. This way we can verify that Logstash can create indices and send messages to Elasticsearch. For that, create a new test-elastic.conf file and add the following:

input {
    stdin {}
}

output {
    elasticsearch {
        hosts => ["[YOUR_ELASTICSEARCH_SERVER]:9200"]
        user => "[YOUR_LOGSTASH_USER]"
        password => "[YOUR_PASSWORD]"
        index => "test-logstash"
    }
    stdout {
        codec => rubydebug
    }
}

Run Logstash with that test configuration file:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

Logstash should start and wait for input. Type a simple message, e.g., Hello, Logstash!, and press Enter. Verify, in the output of the terminal that Logstash processes the message.
Now, let’s head over to Kibana and we verify we’ve got a new index. Go to Management > Index Management

Next, create a data view for that Index. Go to Analytics > Discover

Save the data view and verify we can see the message from the terminal: