Protecting Runtime Fabric’s Persistence Gateway with PodDisruptionBudgets

In our previous post, we explored how PodDisruptionBudgets (PDBs) keep Kubernetes workloads safe during voluntary disruptions like node drains or upgrades. We saw how to protect a simple deployment using labels and a few lines of YAML.

Now, let’s take that same concept and apply it to Runtime Fabric. This time, we’ll protect a critical part of the platform: the Persistence Gateway.

What Is the Persistence Gateway?

The Persistence Gateway in MuleSoft Runtime Fabric handles communication between Mule applications and an external object store backend, PostgreSQL. It manages object store operations like reading and writing state. This is vital for app reliability, especially when using features like caching, clustering, or persistent queues.

Why Use a PDB with the Persistence Gateway?

If the persistence gateway pods are disrupted during maintenance or scaling, the entire runtime fabric can suffer. Apps may lose access to object store data. Performance may degrade. In worst cases, apps could fail due to timeouts or missing data.

By using a PodDisruptionBudget, we ensure that at least one persistence gateway pod always remains available—even during cluster changes. This adds a layer of resilience to your Runtime Fabric and helps avoid unexpected app failures.

How to add a PDB for the Persistence Gateway

When you deploy a PersistenceGateway custom resource in Runtime Fabric, Kubernetes creates a Deployment behind the scenes. That Deployment uses the label app=persistence-gateway

This means we can create a PDB that matches this label directly. We can verify this with the following command:

kubectl get deployment -n rtf -l app=persistence-gateway --show-labels

This command lists the deployment created for the persistence gateway in the rtf namespace and shows its labels. With that we can confirm that the label app=persistence-gateway appears in the output. This is the label we will use to target the pods in our PodDisruptionBudget. We’ll now walk through the full setup.

Step 1: Review the Existing Deployment

This is the default definition of the Persistence Gateway Custom Resource:

apiVersion: rtf.mulesoft.com/v1
kind: PersistenceGateway
metadata:
  name: default
  namespace: rtf
spec:
  objectStore:
    backendDriver: postgresql
    maxBackendConnectionPool: 20
    replicas: 2
    secretRef:
      name: rtf-pg-secret
    resources:
      limits:
        cpu: 250m
        memory: 250Mi
      requests:
        cpu: 200m
        memory: 75Mi

After deploying this in the rtf namespace, Kubernetes creates a Deployment with the label app=persistence-gateway.

Step 2: Create the PodDisruptionBudget

Create a file named persistence-gateway-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: persistence-gateway-pdb
  namespace: rtf
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: persistence-gateway

This definition ensures that at least 1 pod stays available at all times. Apply it:

kubectl apply -f persistence-gateway-pdb.yaml

Step 3: Verify the PDB Is Active

Check the status:

kubectl get pdb -n rtf

You should see something like:

NAME                       MIN AVAILABLE   ALLOWED DISRUPTIONS   AGE
persistence-gateway-pdb   1               1                     10s

This tells us: one pod may be disrupted at a time, but at least one must remain running. We can get more details:

kubectl describe pdb persistence-gateway-pdb -n rtf

Step 4: Test a Disruption Scenario

Identify which node is running a persistence gateway pod:

kubectl get pods -n rtf -l app=persistence-gateway -o wide

Then simulate a node drain:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Kubernetes will only allow this if it can maintain the budget (i.e., one pod stays running). Try draining a second node running the other pod—Kubernetes will block this action because the PDB would be violated.

Summary

Adding a PodDisruptionBudget for the Persistence Gateway brings fault tolerance to a critical part of Runtime Fabric. It prevents accidental downtime during rolling upgrades or node rebalancing. It's a simple YAML change with a big impact.
Start by identifying which components are vital in your Runtime Fabric setup. Then, protect them one by one with PDBs.