Disaster Recovery in Ceph with cephadm, Ceph- CSI, and RBD Mirror

1. Introduction

Ceph is a highly available, scalable, and resilient storage solution widely used in cloud and enterprise environments. However, even with its built-in redundancy, disaster recovery (DR) strategies are essential to ensure business continuity in case of data center failures, network outages, or hardware failures. Ceph provides robust disaster recovery options, including RBD mirroring, to replicate block storage volumes across geographically separated Ceph clusters.

With the introduction of cephadm, Ceph cluster management has become more straightforward, making it easier to deploy and maintain disaster recovery setups. Additionally, Ceph-CSI enables Kubernetes clusters to consume Ceph storage efficiently. In this blog post, we will explore how to set up disaster recovery in Ceph using cephadm, Ceph-CSI, and RBD Mirror to protect RBD volumes used by Kubernetes clusters deployed across two data centers.

2. Disaster Recovery Architecture

We have two geographically separated data centers:

• Primary Data Center (Production): Hosts a Kubernetes cluster and a Ceph cluster.
• Secondary Data Center (Disaster Recovery – DR): Hosts another Kubernetes cluster and a Ceph cluster where data is replicated using RBD mirroring.

Kubernetes workloads in the primary DC store their persistent data in Ceph RBD volumes via Ceph-CSI. These volumes are mirrored asynchronously to the secondary DC using RBD mirroring, ensuring data availability in case of a failure in the primary DC.

3. Deploying Ceph with cephadm in Both Data Centers

a) Bootstrap the Ceph Cluster

On each Ceph cluster (Primary and Secondary):

cephadm bootstrap --mon-ip <MONITOR_IP>

b) Add Additional Nodes

cephadm shell -- ceph orch host add <HOSTNAME> <IP_ADDRESS>

c) Deploy Required Services

ceph orch apply mon
ceph orch apply mgr
ceph orch apply osd --all-available-devices
ceph orch apply rbd-mirror

Ensure that the rbd-mirror daemon is running on both clusters:

ceph orch ps | grep rbd-mirror

d) Configure RBD Mirroring

On the primary Ceph cluster:

rbd mirror pool enable <pool-name> snapshot

Export and import the authentication key:

ceph auth get-key client.rbd-mirror > rbd-mirror.key
scp rbd-mirror.key <secondary-cluster>
ssh <secondary-cluster> 'ceph auth import -i rbd-mirror.key'

On the secondary Ceph cluster, add a peer connection:

rbd mirror pool peer add <pool-name> client.rbd-mirror@<primary-cluster-mon>

Verify peering status:

rbd mirror pool status <pool-name>

4. Installing Ceph-CSI in Kubernetes Clusters

Now we have ceph cluster ready and we can deploy ceph-csi on our k8s clusters. We need to deploy ceph-
csi in both locations, but

a) Deploy Ceph-CSI Driver

kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/devel/deploy/csi-rbdplugin.yaml
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/devel/deploy/csi-rbdplugin-provisioner.yaml

b) Enable RBD Mirroring on the Pool

rbd mirror pool enable <pool-name> snapshot

c) Configure StorageClass to Use the Mirrored Pool

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd-mirrored
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: <ceph-cluster-id>
  pool: <pool-name>
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete
allowVolumeExpansion: true

Apply this StorageClass:

kubectl apply -f storageclass.yaml

d) Create a PersistentVolumeClaim (PVC)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ceph-rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: ceph-rbd-mirrored

5. Failover Process (Switching to the Secondary Data Center)

a) Promote the Secondary Ceph Cluster

rbd mirror pool promote <pool-name>

b) Update ClusterID and PoolID Mappings

Ensure that the Kubernetes cluster in the DR site correctly maps the Ceph cluster’s ClusterID and PoolID using the predefined mapping.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
data:
  cluster-mapping.json: |-
    [
      {
        "clusterIDMapping": {
          "primary-cluster-id": "secondary-cluster-id"
        },
        "RBDPoolIDMapping": [
          {
            "1": "2"
          },
          {
            "11": "12"
          }
        ]
      }
    ]

Apply this updated mapping:

kubectl apply -f ceph-csi-config.yaml

c) Modify Ceph-CSI Config to Update Monitor Addresses on Secondary Cluster

To use a mirrored and promoted RBD image on a secondary site during a failover, you need to replace the primary monitor addresses with the IP addresses of the secondary cluster in ceph-csi-config. Otherwise, Ceph-CSI won’t be able to use the volumes, and application pods will become stuck in the ContainerCreating state. Thus, during failover, both clusters will have the same monitor IP addresses in csi-config on secondary site.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config
data:
  config.json: |-
    [
      {
      "clusterID": "ceph1",
      "rbd": {
          "radosNamespace": "<rados-namespace>",
      },
      "monitors": [
        "192.168.39.82:6789"
      ],
      "cephFS": {
        "subvolumeGroup": "<subvolumegroup for cephfs volumes>"
      }
      },
      {
      "clusterID": "ceph2",
      "rbd": {
          "radosNamespace": "<rados-namespace>",
      },
      "monitors": [
        "192.168.39.82:6789"
      ],
      "cephFS": {
        "subvolumeGroup": "<subvolumegroup for cephfs volumes>"
      }
      }
    ]

Apply the updated configuration

kubectl apply -f ceph-csi-config.yaml

d) Modify StorageClass to Point to the Secondary Cluster

parameters:
 clusterID: secondary-cluster-id

Apply the modified StorageClass:

kubectl apply -f storageclass.yaml

e) Restart Affected Workloads

kubectl rollout restart deployment <deployment-name>

f) Validate Data Accessibility

Ensure the applications can access data stored in the secondary Ceph cluster.

6. Failback Process (Restoring to the Primary Data Center)

a) Demote the Secondary Cluster and Re-enable Mirroring

rbd mirror pool demote <pool-name>

b) Update ClusterID and PoolID Mappings Back to Primary

apiVersion: v1
kind: ConfigMap
metadata:
  name: ceph-csi-config

c) Modify StorageClass to Point Back to the Primary Cluster

parameters:
 clusterID: primary-cluster-id

d) Restart Workloads to Use the Primary Cluster

kubectl rollout restart deployment <deployment-name>

e) Verify Mirroring and Data Integrity

rbd mirror pool status <pool-name>

7. Conclusion

By configuring ClusterID and PoolID mappings and ensuring proper Ceph monitor address updates during failover, you enable seamless disaster recovery for Kubernetes workloads using Ceph-CSI. This approach maintains data accessibility and consistency, facilitating a smoother failover and failback process. Using cephadm, deploying and managing mirroring has become significantly easier, enabling organizations to set up failover mechanisms efficiently. By following the above steps, you can ensure data integrity, minimize downtime, and enhance business continuity in the event of a disaster.