Install Elasticsearch and Kibana with Docker Compose

ELK (or Elastic) is one of the preferred platforms for the Observability of our Mule apps. The installation and setup of ELK is not an easy task and it can take time. Sometimes, as in my case, we need a quick way of testing some aspects of the integration between Mule and ELK.

In these cases, Docker is a great option to simplify the deployment of any of the components of the Elastic Stack. And with Docker Compose we can spin up all the containers in one go very quickly.

Docker Compose is a tool for defining and managing multi-container Docker applications. It uses a YAML file to define the services, networks, and volumes your application needs and provides commands to manage the lifecycle of the application.

In this tutorial, we’ll see how to deploy Elasticsearch and Kibana with Docker compose in very few steps, so that we can have a testing environment in minutes. We’ll leave for now Logstash, that’s for another post.

Prerequisites and Considerations

The main goal here is speed - it’s to be able to have both containers running and set up with only one command, just for testing. For that, we’ll disable the security pack for elasticsearch, so no authentication will be required to connect to elasticsearch. Again, not something for a production environment, but quick and easy to get.
If we needed a more custom or more production-like installation, then have a look at these posts

How to Install Elasticsearch and Kibana on Linux - Part I
How to Install Elasticsearch and Kibana on Linux - Part II
How to Install Elasticsearch on Docker
How to Install Kibana on Docker

For this tutorial we will need:

A Docker host where we’ll run the containers. It could be your laptop or a server in your network with Docker. Check out this post on How to install Docker CE on Ubuntu
Docker Compose - If you’re using a Desktop version it is normally bundled with it, but if not, you can install it separately using Docker Compose Installation Guide.

You can verify that with the commands:

docker --version
docker-compose --version

Installation

To use Docker compose we need to create a docker-compose.yml file where we’ll be defining the services, networks and volumes that our Elasticsearch and Kibana containers will use.

Copy and paste the following to your compose file:

version: '3.4'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.3
    container_name: elasticsearch
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - cluster.name=elk-mule
      - node.name=elk-mule-01
      - network.host=0.0.0.0
      - xpack.security.enabled=false
    networks:
      - elastic
  kibana:
    image: docker.elastic.co/kibana/kibana:8.15.3
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      - SERVER_PORT=5601
      - SERVER_HOST=0.0.0.0
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    networks:
      - elastic
    depends_on:
      - elasticsearch

volumes:
  es_data:

networks:
  elastic: {}

Some considerations:

Both Elasticsearch and Kibana use the same version (at the time of writing, the latest version available for elastic is 8.15.3). If you use diff version, make sure it’s the same for the image of both containers.
As we mentioned, no authentication is enabled for elasticsearch. This allows Kibana to connect to our Elasticsearch instance straight away, without any extra configuration step.
We’ve defined a docker volume for elasticsearch, so that our data can be persisted and we can use it after restarting the containers.
We’ve defined a docker network, elastic. This allows both containers to communicate to each other using the container name as dns name. This helps us as well if we needed to restart the containers.

Start the services

Run the following command in the directory containing docker-compose.yml

docker-compose up -d

This will:

Pull the elasticsearch and kibana images if they were not available locally
Create the docker network
Create the volume for elasticsearch
Make sure Kibana starts after the elasticsearc container is up
The -d flag will run the containers in detached mode.

Verify

First, lets list the running services:

docker-compose ps

Next, check the logs of the containers and verify there are not errors

docker logs elasticsearch
docker logs kibana

Verify your elasticsearch is working sending an HTTP request:

curl http://localhost:9200

Lastly, verify that you can access Kibana. From your web browser go to http://localhost:5601

Stop Services

Once you’re done with your testing you can quickly stop the containers by running the below command from the directory containing the docker-compose.yml file.

docker-compose down

If you needed to remove eveything - containers, networks and volumes add the -v flag

docker-compose down -v