Securing Prometheus and Node Exporter with TLS and Basic Authentication

In our last post, How to install Prometheus Node Exporter with systemd, we built a production-grade Node Exporter service managed by systemd. Prometheus scrapes it over plain HTTP on port 9100. That setup works, but it is not safe for a real production environment.

Two problems exist with our current configuration.

Problem 1 — No Authentication: Right now, any process or person with network access to port 9100 can scrape our Node Exporter metrics. In a Mule production environment, those metrics reveal CPU load, memory pressure, disk usage, and network throughput. That data helps an attacker understand exactly when and where our infrastructure is under stress. We must restrict who can scrape the endpoint.
Problem 2 — No Encryption: Our current setup sends all metric data as plain text over HTTP. Anyone who can intercept traffic between Prometheus and Node Exporter reads that data in full. We need to encrypt the connection so that intercepted packets are unreadable.

The solution to both problems is the same approach we use for any secured web service: TLS encryption and basic authentication. We configure TLS on Node Exporter so it speaks HTTPS instead of HTTP. We configure basic authentication so Prometheus must present a valid username and password on every scrape request.

What We Will Build

By the end of this post, our setup will look like this:

Node Exporter serves metrics over HTTPS on port 9100
Node Exporter requires a username and password before returning any data
Prometheus connects to Node Exporter over HTTPS
Prometheus presents valid credentials on every scrape

This is the minimum security baseline for any Prometheus target exposed on a network shared with other systems or teams.

Prerequisites

Ubuntu 20.04 or 22.04
Node Exporter installed and running as a systemd service (see How to install Prometheus Node Exporter with systemd)
Prometheus server configured to scrape Node Exporter (see How to install Node Exporter for Prometheus on Ubuntu Server)
sudo privileges on both machines
openssl installed (it is pre-installed on most Ubuntu systems — verify with openssl version)

Step 1 — Understand Self-Signed vs. CA-Signed Certificates

TLS requires a certificate. A certificate proves the identity of the server and contains the public key used to encrypt the connection.

We have two options:

Type	Issued By	Cost	Use Case
Self-signed	Us, using OpenSSL	Free	Internal services, labs, development
CA-signed	A Certificate Authority (e.g. Let's Encrypt)	Free to paid	Public-facing services

In this tutorial, we use a self-signed certificate. Our Prometheus server and Node Exporter are internal infrastructure. They do not face the public internet. A self-signed certificate gives us full encryption. The trade-off is that no external authority vouches for our identity — so tools like curl will warn us about the certificate. We will handle that in the Prometheus configuration.

Production note: If your Prometheus or Node Exporter instances sit behind a corporate network that has an internal Certificate Authority, use that CA instead. The steps for configuring the certificate files are identical.

Step 2 — Generate the TLS Certificate and Key

We generate our certificate on the Node Exporter machine. We create both files in our working directory first, then move them to a secure location.

Move to a temporary working directory:

cd /tmp

Run the OpenSSL command to generate a self-signed certificate valid for one year:

sudo openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 \
  -keyout node_exporter.key \
  -out node_exporter.crt \
  -subj "/C=US/ST=California/L=Oakland/O=MyOrg/CN=localhost" \
  -addext "subjectAltName = DNS:localhost"

Let us break down the important flags:

Flag	Purpose
`-new -newkey rsa:2048`	Generate a new 2048-bit RSA private key
`-days 365`	Certificate expires after one year
`-nodes`	Do not encrypt the private key with a passphrase (required for unattended service start)
`-x509`	Output a self-signed certificate, not a certificate signing request
`-keyout node_exporter.key`	Write the private key to this file
`-out node_exporter.crt`	Write the certificate to this file
`-subj`	Set the certificate subject fields without interactive prompts
`-addext subjectAltName`	Add a Subject Alternative Name — modern TLS clients require this

Why -nodes? If we encrypt the private key with a passphrase, systemd cannot start Node Exporter automatically because it cannot enter a passphrase at boot time. For managed services, we skip the passphrase and rely on filesystem permissions to protect the key instead.

Verify that both files were created:

ls -l /tmp/node_exporter.*

We expect two files:

The private key file should be readable only by root at this stage. We will set the final permissions after moving the files.

Step 3 — Create a Secure Configuration Directory

We store Node Exporter's TLS configuration and credentials under /etc, following the Linux convention for persistent configuration files.

Create the directory:

sudo mkdir -p /etc/node_exporter

Move the certificate and key into it:

sudo mv /tmp/node_exporter.crt /etc/node_exporter/
sudo mv /tmp/node_exporter.key /etc/node_exporter/

Set ownership so the node_exporter service user can read these files:

sudo chown -R node_exporter:node_exporter /etc/node_exporter

Lock down the directory so only the node_exporter user has access:

sudo chmod 750 /etc/node_exporter
sudo chmod 640 /etc/node_exporter/node_exporter.crt
sudo chmod 600 /etc/node_exporter/node_exporter.key

The private key (node_exporter.key) must be readable only by the node_exporter user. If any other process can read this file, the encryption is compromised.

Verify the permissions:

sudo ls -l /etc/node_exporter/

Expected output:

-rw-r----- 1 node_exporter node_exporter 1326 node_exporter.crt
-rw------- 1 node_exporter node_exporter 1704 node_exporter.key

Step 4 — Create the Node Exporter Web Configuration File

Node Exporter reads its TLS and authentication settings from a web configuration file. We create it now.

sudo nano /etc/node_exporter/config.yml

Add the TLS configuration:

tls_server_config:
  cert_file: /etc/node_exporter/node_exporter.crt
  key_file: /etc/node_exporter/node_exporter.key

We use absolute paths here. Node Exporter will run as a systemd service with a defined WorkingDirectory, but absolute paths remove all ambiguity.

Save and close the file. Set ownership:

sudo chown node_exporter:node_exporter /etc/node_exporter/config.yml
sudo chmod 640 /etc/node_exporter/config.yml

Step 5 — Test TLS Before Touching systemd

Before we update the systemd service, we test our TLS configuration manually. This step catches errors early, without a failed service restart making things harder to debug.

Run Node Exporter directly with the web config:

sudo -u node_exporter /usr/local/bin/node_exporter \
  --web.config.file=/etc/node_exporter/config.yml

The output should include this line:

msg="TLS is enabled." http2=true

You’ll probably see error messages telling you there was a handshake error. That’s because now, with this confirutation, our Prometheus server can’t verify the certificate we’ve just added. That’s expected.

You might need to stop the node_exporter service if it is already running with systemd - sudo systemctl stop node_exporter

In a second terminal, test the endpoint:

# This will fail — that is expected
curl https://localhost:9100/metrics

We will see a certificate error:

This error is correct. curl does not trust our self-signed certificate. We bypass verification with the -k flag for testing only:

curl -k https://localhost:9100/metrics | head -5

We should see metrics output. TLS is working.

Stop the manual process with Ctrl+C.

Step 6 — Set Up Basic Authentication

TLS encrypts the connection. Basic authentication controls who can use it. We add a username and hashed password to our Node Exporter configuration.

Generate a Hashed Password

We must store the password as a bcrypt hash — never as plain text. Install the apache2-utils package, which includes the htpasswd tool:

sudo apt install apache2-utils -y

Generate a bcrypt hash. The -nBC 12 flags tell htpasswd to output the hash without creating a file (-n), using bcrypt (-B), with a cost factor of 12 (-C 12). The cost factor controls how slow the hash computation is — higher means harder to brute-force.

htpasswd -nBC 12 "" | tr -d ':\n'


htpasswd

prompts us to enter and confirm a password. Choose a strong password and save it somewhere secure — we will need it later when configuring Prometheus.

The output looks like this:

$2y$12$qCGYWDaYc8FxentuP5A18eY8mn7tFzTAoh/N8evRraLYKK3VecnAq

Copy this hash. We’ll add it to the Node Exporter config file.

Add Authentication to the Config File

Open the Node Exporter configuration file:

sudo nano /etc/node_exporter/config.yml

Add the basic_auth_users block below the existing TLS configuration. Replace the hash below with the one we generated:

tls_server_config:
  cert_file: /etc/node_exporter/node_exporter.crt
  key_file: /etc/node_exporter/node_exporter.key

basic_auth_users:
  prometheus: $2y$12$qCGYWDaYc8FxentuP5A18eY8mn7tFzTAoh/N8evRraLYKK3VecnAq

The username is prometheus. This is the account our Prometheus server will use to authenticate. The value is the bcrypt hash — not the plain text password.

Save and close the file.

Step 7 — Update the systemd Service File

We need to tell the Node Exporter systemd service to load our web configuration file on start.

Open the service file:

sudo nano /etc/systemd/system/node_exporter.service

Update the ExecStart line to include the --web.config.file flag:

[Unit]
Description=Prometheus Node Exporter
Documentation=https://prometheus.io/docs/guides/node-exporter/
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter \
    --web.config.file=/etc/node_exporter/config.yml \
    --collector.disable-defaults \
    --collector.cpu \
    --collector.meminfo \
    --collector.diskstats \
    --collector.filesystem \
    --collector.netdev \
    --collector.loadavg \
    --collector.uname

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

The only change from our previous post is the addition of --web.config.file=/etc/node_exporter/config.yml as the first argument in ExecStart. All other lines remain the same.

Reload systemd and restart Node Exporter:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter

Check the service status:

sudo systemctl status node_exporter

Confirm it shows active (running). Check the logs for the TLS confirmation line:

sudo journalctl -u node_exporter -n 20 --no-pager

Look for:

msg="TLS is enabled." http2=true

Step 8 — Test Authentication

Verify that an unauthenticated request is now rejected:

curl -k https://localhost:9100/metrics

Expected response:

Unauthorized

Now test with valid credentials:

curl -k -u prometheus:YOUR_PLAIN_TEXT_PASSWORD https://localhost:9100/metrics | head -5

Replace YOUR_PLAIN_TEXT_PASSWORD with the password we used when generating the hash in Step 6. We should see metrics output. Node Exporter now requires both a valid TLS connection and valid credentials.

Step 9 — Copy the Certificate to the Prometheus Server

Prometheus needs to trust our Node Exporter certificate to establish a TLS connection. We copy the certificate file from the Node Exporter machine to the Prometheus server.

On the Node Exporter machine, use scp to transfer the certificate. Replace the placeholders with actual values:

scp /etc/node_exporter/node_exporter.crt \
  YOUR_USER@PROMETHEUS_SERVER_IP:/tmp/node_exporter.crt

On the Prometheus server, move the certificate to the Prometheus configuration directory:

sudo mv /tmp/node_exporter.crt /etc/prometheus/node_exporter.crt
sudo chown prometheus:prometheus /etc/prometheus/node_exporter.crt
sudo chmod 640 /etc/prometheus/node_exporter.crt

We copy only the certificate (.crt) — never the private key (.key). The private key must never leave the Node Exporter machine.

Step 10 — Update the Prometheus Scrape Configuration

On the Prometheus server, open the Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

Update the node_exporter job we created in this Post. We add three new settings:

scrape_configs:

  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    scheme: https
    basic_auth:
      username: prometheus
      password: YOUR_PLAIN_TEXT_PASSWORD
    tls_config:
      ca_file: /etc/prometheus/node_exporter.crt
      insecure_skip_verify: true
    static_configs:
      - targets: ['NODE_EXPORTER_HOST_IP:9100']

Here is what each new field does:

Field	Purpose
`scheme: https`	Tells Prometheus to use HTTPS when scraping this target
`basic_auth.username`	The username Node Exporter expects
`basic_auth.password`	The plain text password — Prometheus handles this securely in memory
`tls_config.ca_file`	The certificate file Prometheus uses to verify the Node Exporter connection
`insecure_skip_verify: true`	Required for self-signed certificates — skips hostname validation

Why insecure_skip_verify: true? Our certificate is self-signed. No Certificate Authority has verified it. Prometheus would reject the connection without this flag. If we switch to a CA-signed certificate in the future, we remove this line.

Save and close the file.

Reload Prometheus:

sudo systemctl restart prometheus

Step 11 — Verify the Secured Connection in the Prometheus UI

Open the Prometheus UI in a browser:

http://PROMETHEUS_SERVER_IP:9090/targets

The node_exporter job should show state UP with a green badge.

If it shows DOWN, check the error message in the UI. Common causes:

Error	Cause	Fix
`connection refused`	Node Exporter is not running	`sudo systemctl start node_exporter`
`401 Unauthorized`	Wrong password in `prometheus.yml`	Correct the `basic_auth.password`value
`certificate signed by unknown authority`	Wrong `ca_file` path	Verify the `.crt` file path in `prometheus.yml`
`connection timed out`	Firewall blocking port 9100	Run `sudo ufw allow 9100/tcp` on the Node Exporter host

For deeper diagnostics, check the Prometheus logs:

sudo journalctl -u prometheus -n 50 --no-pager

Summary

We moved our Prometheus-to-Node Exporter connection from plain HTTP with no access control to an encrypted, authenticated channel. Here is what we put in place:

A self-signed TLS certificate generated with OpenSSL
Certificate and key files stored in /etc/node_exporter with locked-down permissions
Node Exporter configured to serve HTTPS via a config.yml web configuration file
Basic authentication with a bcrypt-hashed password
The systemd service updated to load the web configuration at start
The Node Exporter certificate copied to the Prometheus server
The Prometheus scrape job updated to use https, basic_auth, and tls_config

This is the minimum security baseline for any Prometheus target running on a shared network. No unencrypted traffic. No unauthenticated access to metrics.