Skip to content

antony@notes:~/kubernetes$ cat "Test-Ceph-RBD-with-Talos-Kubernetes.md"

Test Ceph RBD with Talos Kubernetes

2023-12-23· kubernetes

Test Ceph RBD with Talos Kubernetes

Preface

本篇文章會主要會先在 Talos Kubernetes 上建立 Ceph-csi 和 Ceph StorageClass，再實作本次測試目標。

可以透過點擊以下目錄，選擇想看的內容，跳轉至特定章節

:::warning

:::spoiler {state=“open”} 目錄

[TOC]

:::

目標

測試 Deployment Object 可不可以共享同一個 Ceph RBD

測試環境 

$ kubectl get nodes -o wide
NAME   STATUS   ROLES           AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
m1     Ready    control-plane   2d6h   v1.28.3   192.168.61.31   <none>        Talos (v1.5.5)   6.1.61-talos     containerd://1.6.23
w1     Ready    <none>          2d6h   v1.28.3   192.168.61.34   <none>        Talos (v1.5.5)   6.1.61-talos     containerd://1.6.23
w2     Ready    <none>          2d6h   v1.28.3   192.168.61.35   <none>        Talos (v1.5.5)   6.1.61-talos     containerd://1.6.23

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
NAME   TAINTS
m1     [map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
w1     <none>
w2     <none>

Talos OS version: v1.5.5 K8S version: v1.28.3 Ceph Version: 18.2.0 Reef (stable) 使用 nodeName 可以無視節點上的 Taints，或是使用 toleration (容忍度)，可以讓 Pod 容忍節點的 Taints

Ceph 設定

1. 建立 Ceph Pool

在 Ceph Node 執行以下命令 :

$ ceph osd pool create kubernetes

# Use the rbd tool to initialize the pool
$ rbd pool init kubernetes

# 設定 pool 最大容量為 10GB
$ ceph osd pool set-quota kubernetes max_bytes $((10 * 1024 * 1024 * 1024))

Ceph pool 是否有其他 init 方式?

2. 建立 Ceph Client 身分驗證 

$ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'

螢幕輸出 :

[client.kubernetes]
        key = AQBqkoFlJoe0MxAAb3UMOPeQSsDWI3FTRj8UTw==

image

設定與部屬 Ceph-CSI RBD plugins

1. Download ceph-csi

在 Talos k8s 外部管理主機執行以下命令 :

$ cd ~ && git clone https://github.com/ceph/ceph-csi.git

2. 建立並切換至 csi-ceph namespace 

$ kubectl create ns csi-ceph
namespace/csi-ceph created

$ kubectl config set-context --current --namespace=csi-ceph
Context "admin@topgun" modified.

3. 配置 ceph-csi 設定檔

3.1. 獲取 Ceph monitor 和 fsid 資訊

在 Proxmox Ceph Node ( monitor ) 執行以下命令 :

$ ceph mon dump

螢幕輸出 :

epoch 4
fsid cd4925d8-31e9-4245-a392-469d661fedce
last_changed 2023-12-16T22:35:48.168822+0800
created 2023-11-22T09:44:04.138233+0800
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:192.168.61.11:3300/0,v1:192.168.61.11:6789/0] mon.pve
1: [v2:192.168.61.12:3300/0,v1:192.168.61.12:6789/0] mon.pve2
2: [v2:192.168.61.13:3300/0,v1:192.168.61.13:6789/0] mon.pve3
dumped monmap epoch 4

3.2. 設定 Ceph-csi configmap

在 Talos 外部管理主機執行以下命令 :

$ cd ~/ceph-csi/deploy/rbd/kubernetes

$ cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "cd4925d8-31e9-4245-a392-469d661fedce",
        "monitors": [
          "192.168.61.11:6789",
          "192.168.61.12:6789",
          "192.168.61.13:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
EOF

須設定 clusterID 和 monitors 的 IP Address

3.3. 設定 csidriver 的 pod 不要 Mount host 主機的 /etc/selinux 到 pods 裡面 

$ sed -i 's|seLinuxMount: true|seLinuxMount: false|g' csidriver.yaml
$ sed -i '/- mountPath: \/etc\/selinux/,+2d' csi-rbdplugin.yaml
$ sed -i '/- name: etc-selinux/,+2d' csi-rbdplugin.yaml

3.4. 將所有 Yaml 中定義的物件都更換為 csi-ceph Namespace 

$ sed -i 's|namespace: default|namespace: csi-ceph|g' *.yaml

3.5. 設定 csi-rbdplugin-provisioner 和 csi-rbdplugin 的 Pod 能夠在 Contorlplane Node 上執行 

$ sed -i '36i\      tolerations:\n        - operator: Exists' csi-rbdplugin-provisioner.yaml
$ sed -i '24i\      tolerations:\n        - operator: Exists' csi-rbdplugin.yaml

3.6. 產生 CEPH-CSI cephx secret 

$ cat <<EOF > ~/ceph-csi/examples/rbd/secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: csi-ceph
stringData:
  userID: kubernetes
  userKey: AQBqkoFlJoe0MxAAb3UMOPeQSsDWI3FTRj8UTw==

  # Encryption passphrase
  encryptionPassphrase: test_passphrase
EOF

須設定 namespace、userID 和 userKey 的值

3.7. 建立 CEPH-CSI cephx secret 

$ kubectl apply -f ~/ceph-csi/examples/rbd/secret.yaml

4. 部屬 ceph-csi

4.1. 設定 csi-ceph Namespace 中的 Pod 能夠擁有 privileged 權限 

$ kubectl label ns csi-ceph pod-security.kubernetes.io/enforce=privileged

csi-rbdplugin 和 csi-rbdplugin-provisioner 的 Pod 會需要 privileged 的權限。

Talos K8S PSA 預設所有的 Pod 不能有 privileged 的權限，此時可以透過幫 Namespace 貼上面這個 label ，就能夠覆蓋掉 K8S PSA 的設定。

4.2. 最新版本的 ceph-csi 還需要另一個 ConfigMap 物件來定義 Ceph 的設定資訊，以便新增至 CSI Container 內的 ceph.conf 檔案中 

$ kubectl apply -f ~/ceph-csi/deploy/ceph-conf.yaml

This is a sample configmap that helps define a Ceph configuration as required by the CSI plugins.

4.3. 開始部屬 ceph-csi 

$ cd ~/ceph-csi/examples/rbd

$ ./plugin-deploy.sh ~/ceph-csi/deploy/rbd/kubernetes

## 移除 vault
$ kubectl delete -f ../kms/vault/vault.yaml

4.4. 檢視 ceph-csi 部屬狀態 

$ kubectl get all

螢幕輸出 :

NAME                                            READY   STATUS    RESTARTS   AGE
pod/csi-rbdplugin-2dls2                         3/3     Running   0          111m
pod/csi-rbdplugin-j7fx6                         3/3     Running   0          111m
pod/csi-rbdplugin-l84w5                         3/3     Running   0          111m
pod/csi-rbdplugin-provisioner-ddb58c7f4-brz59   7/7     Running   0          111m
pod/csi-rbdplugin-provisioner-ddb58c7f4-jdwcr   7/7     Running   0          111m
pod/csi-rbdplugin-provisioner-ddb58c7f4-td422   7/7     Running   0          111m

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/csi-metrics-rbdplugin       ClusterIP   10.98.76.185    <none>        8080/TCP   111m
service/csi-rbdplugin-provisioner   ClusterIP   10.104.214.86   <none>        8080/TCP   111m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/csi-rbdplugin   3         3         3       3            3           <none>          111m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/csi-rbdplugin-provisioner   3/3     3            3           111m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/csi-rbdplugin-provisioner-ddb58c7f4   3         3         3       111m

4.5. 設定 StorageClass Yaml 檔 

$ cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: cd4925d8-31e9-4245-a392-469d661fedce
   pool: kubernetes
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: csi-ceph
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: csi-ceph
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: csi-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard
EOF

須設定 clusterID 和 pool 的值

4.6. 建立 StorageClass 

$ kubectl apply -f csi-rbd-sc.yaml

4.7. 檢視 StorageClass 

$ kubectl get sc csi-rbd-sc

螢幕輸出 :

NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   9s

以上環境部屬完畢，以下開始測試本次實作目標。

開始實作測試目標

1. 設定 PVC 

$ cat <<EOF > raw-block-pvc-rwx.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF

Note: Using ceph-csi, specifying Filesystem for volumeMode can support both ReadWriteOnce and ReadOnlyMany accessMode claims, and specifying Block for volumeMode can support ReadWriteOnce, ReadWriteMany, and ReadOnlyMany accessMode claims.

2. 建立 PVC Yaml 檔 

$ kubectl apply -f raw-block-pvc-rwx.yaml

3. 檢查 PVC 建立狀態 

$ kubectl get pvc

螢幕輸出 :

NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
raw-block-pvc   Bound    pvc-629444ef-f0c0-4afd-8263-443d6f4beb25   1Gi        RWX            csi-rbd-sc     3s

4. 設定 Deployment Object Yaml 檔 

$ cat <<EOF > raw-block-deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-with-raw-block-volume
  labels:
    os: alpine
spec:
  replicas: 3
  selector:
    matchLabels:
      os: alpine
  template:
    metadata:
      labels:
        os: alpine
    spec:
      containers:
      - name: alpine
        image: quay.io/cloudwalker/alp.base:latest
        command: ["/bin/sleep", "infinity"]
        volumeDevices:
          - name: data
            devicePath: /dev/xvda
        securityContext:
          capabilities:
            add: ["SYS_ADMIN"]
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: raw-block-pvc
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: os
                operator: In
                values:
                - alpine
            topologyKey: kubernetes.io/hostname
      tolerations:
      - operator: Exists
EOF

5. 建立 Deployment Object Yaml 檔 

$ kubectl apply -f raw-block-deployment.yaml

6. 檢查 Pods 運作狀態 

$ kubectl get pods -l os=alpine -o wide

螢幕輸出 :

NAME                                       READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
pod-with-raw-block-volume-8677b97b-m64wt   1/1     Running   0          26s   10.244.1.19   m1     <none>           <none>
pod-with-raw-block-volume-8677b97b-nnpt2   1/1     Running   0          26s   10.244.0.53   w1     <none>           <none>
pod-with-raw-block-volume-8677b97b-x2qh9   1/1     Running   0          26s   10.244.2.46   w2     <none>           <none>

7. 進入不同的 Pods 檢視已掛載的 Ceph Block Device 

$ kubectl exec pod-with-raw-block-volume-8677b97b-2dfj9 -- ls -l /dev/xvda

$ kubectl exec pod-with-raw-block-volume-8677b97b-ft4kw -- ls -l /dev/xvda

$ kubectl exec pod-with-raw-block-volume-8677b97b-sbp6m -- ls -l /dev/xvda

3 次執行的結果應該跟以下螢幕輸出類似 :

brw------- 1 root root 252, 0 Dec 19 09:25 /dev/xvda

8. 測試使用 XFS 檔案系統 Fotmat Ceph Block Device，並將 /ceph 目錄掛載上去，最後建立 test 空檔案 

$ kubectl exec -it pod-with-raw-block-volume-8677b97b-m64wt -- bash
pod-with-raw-block-volume-8677b97b-m64wt:/# apk add xfsprogs
pod-with-raw-block-volume-8677b97b-m64wt:/# mkfs.xfs /dev/xvda
pod-with-raw-block-volume-8677b97b-m64wt:/# mkdir /ceph
pod-with-raw-block-volume-8677b97b-m64wt:/# mount /dev/xvda /ceph
pod-with-raw-block-volume-8677b97b-m64wt:/# lsblk /dev/xvda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
rbd0 252:0    0   1G  0 disk /ceph
pod-with-raw-block-volume-8677b97b-m64wt:/# touch /ceph/test
pod-with-raw-block-volume-8677b97b-m64wt:/# exit
exit

9. 進到另外兩個 Pods 檢查是否有看到 test 這個空檔案 

$ kubectl exec -it pod-with-raw-block-volume-8677b97b-nnpt2 -- bash
pod-with-raw-block-volume-8677b97b-nnpt2:/# mkdir /ceph
pod-with-raw-block-volume-8677b97b-nnpt2:/# mount /dev/xvda /ceph
pod-with-raw-block-volume-8677b97b-nnpt2:/# ls -l /ceph/
total 0
-rw-r--r-- 1 root root 0 Dec 19 13:54 test
pod-with-raw-block-volume-8677b97b-nnpt2:/# exit
exit

$ kubectl exec -it pod-with-raw-block-volume-8677b97b-x2qh9 -- bash
pod-with-raw-block-volume-8677b97b-x2qh9:/# mkdir /ceph
pod-with-raw-block-volume-8677b97b-x2qh9:/# mount /dev/xvda /ceph
pod-with-raw-block-volume-8677b97b-x2qh9:/# ls -l /ceph/
total 0
-rw-r--r-- 1 root root 0 Dec 19 13:54 test
pod-with-raw-block-volume-8677b97b-x2qh9:/# exit
exit

10. 列出在 kubernetes pool 底下的 RBD images

在 Ceph Node 執行以下命令

$ rbd ls kubernetes

螢幕輸出 :

csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e

11. 查看 RBD Image 的詳細資訊 

$ rbd info kubernetes/csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e

螢幕輸出 :

rbd image 'csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 13c94844fea3c6
        block_name_prefix: rbd_data.13c94844fea3c6
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Tue Dec 19 21:50:23 2023
        access_timestamp: Tue Dec 19 21:50:23 2023
        modify_timestamp: Tue Dec 19 21:50:23 2023

清除測試環境 

## 在 Talos 外部管理主機執行以下命令
$ kubectl delete -f raw-block-deployment.yaml,raw-block-pvc-rwx.yaml
$ kubectl delete -f csi-rbd-sc.yaml,secret.yaml
$ kubectl delete -f ~/ceph-csi/deploy/ceph-conf.yaml
$ ./plugin-teardown.sh ~/ceph-csi/deploy/rbd/kubernetes/
$ kubectl label ns csi-ceph pod-security.kubernetes.io/enforce-
$ kubectl get all,configmap,secret
NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      20h
$ kubectl config set-context --current --namespace=default
$ kubectl delete ns csi-ceph
$ cd ~ && rm -r ceph-csi/

## 在 Ceph Node 執行以下命令
$ rbd -p kubernetes ls
$ ceph auth rm client.kubernetes
$ ceph osd pool rm kubernetes kubernetes --yes-i-really-really-mean-it

參考文件

Getting Started, configuring Backstage測試 K8S Pod 數量超過 110 會發生甚麼事?