antony@notes:~/rancher$ cat "Install-Longhorn-on-Talos-Kubernetes.md"

Install Longhorn on Talos Kubernetes

2024-03-07· rancher ·Longhorn

Install Longhorn on Talos Kubernetes

PreRequest

System Extensions

要額外在 Talos K8S 的 Worker Node 上安裝 iscsi-tools 和 util-linux-tools 這兩個套件

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools

siderolabs/iscsi-tools: this extension enables iscsid daemon and iscsiadm to be available to all nodes for the Kubernetes persistent volumes operations. siderolabs/util-linux-tools: this extension enables linux tool to be available to all nodes. For example, the fstrim binary is used for Longhorn volume trimming.

Pod Security

Longhorn 要求在 longhorn-system Namespace 要有 enforce: "privileged" 的權限

Data Path Mounts

You need provide additional data path mounts to be accessible to the Kubernetes Kubelet container.

These mount is necessary to provide access to the host directories and attaching volumes required by the Longhorn components.

machine:
  kubelet:
    extraMounts:
      - destination: /var/lib/longhorn
        type: bind
        source: /var/lib/longhorn
        options:
          - bind
          - rshared
          - rw

環境準備

獲得 Talos System Extensions 的 image ID

## create the schematic file
$ cat <<EOF > longhorn.yaml
customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/iscsi-tools
      - siderolabs/util-linux-tools
      - siderolabs/gvisor
      - siderolabs/qemu-guest-agent
EOF

## 到 Talos 官網的 Image Factory 查詢包含指定套件的 Image ID:
$ curl -X POST --data-binary @longhorn.yaml https://factory.talos.dev/schematics
{"id":"c527b6b20fb22847304656677e9bd4c4055dfcce95f3385da5db80e35f5fa1dc"}

Image Factory is a service that generates Talos boot assets on-demand. Image Factory allows to generate boot assets for the official Talos Linux releases and official Talos Linux system extensions.

Controlplane node patch yaml

以下是 Controlplane node 使用 machine config 檔需要的 patch yaml

machine:
  install:
    image: factory.talos.dev/installer/c527b6b20fb22847304656677e9bd4c4055dfcce95f3385da5db80e35f5fa1dc:v1.6.5
cluster:
    inlineManifests:
      - name: namespace-longhorn-system
        contents: |-
          apiVersion: v1
          kind: Namespace
          metadata:
             name: longhorn-system

Worker node patch yaml

以下是 Worker node 使用 machine config 檔需要的 patch yaml

machine:
    kubelet:
        extraMounts:
          - destination: /var/lib/longhorn
            type: bind
            source: /var/lib/longhorn
            options:
              - bind
              - rshared
              - rw
    install:
      image: factory.talos.dev/installer/c527b6b20fb22847304656677e9bd4c4055dfcce95f3385da5db80e35f5fa1dc:v1.6.5

透過 patch yaml 修改 machineconfig

## 修改 k1m1
$ talosctl machineconfig patch /home/bigred/k1/v1.6.5/controlplane.yaml --patch @/home/bigred/k1/v1.6.5/k1m1.patch --output /home/bigred/k1/v1.6.5/k1m1.yaml

## 修改 k1w1
$ talosctl machineconfig patch /home/bigred/k1/v1.6.5/worker.yaml --patch @/home/bigred/k1/v1.6.5/k1w1.patch --output /home/bigred/k1/v1.6.5/k1w1.yaml

## 修改 k1w2
$ talosctl machineconfig patch /home/bigred/k1/v1.6.5/worker.yaml --patch @/home/bigred/k1/v1.6.5/k1w2.patch --output /home/bigred/k1/v1.6.5/k1w2.yaml

建立 Talos K8S

$ 1m2w.sh
Talos Control Plane (172.22.1.11) config
waiting 120
k1m1(172.22.1.11) config
waiting 300
k1w1(172.22.1.15) config
k1w2(172.22.1.16) config
waiting 120

NAME   STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
k1m1   Ready    control-plane   5m56s   v1.29.2   172.22.1.11   <none>        Talos (v1.6.5)   6.1.78-talos     containerd://1.7.13
k1w1   Ready    worker          63s     v1.29.2   172.22.1.15   <none>        Talos (v1.6.5)   6.1.78-talos     containerd://1.7.13
k1w2   Ready    worker          56s     v1.29.2   172.22.1.16   <none>        Talos (v1.6.5)   6.1.78-talos     containerd://1.7.13

確認 System Extension 是否安裝成功

$ talosctl -n 172.22.1.11,172.22.1.15,172.22.1.16 --talosconfig=/home/bigred/k1/v1.6.5/talosconfig get extensions
NODE          NAMESPACE   TYPE              ID   VERSION   NAME               VERSION
172.22.1.11   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
172.22.1.11   runtime     ExtensionStatus   1    1         util-linux-tools   $VERSION
172.22.1.11   runtime     ExtensionStatus   2    1         schematic          613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
172.22.1.15   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
172.22.1.15   runtime     ExtensionStatus   1    1         util-linux-tools   $VERSION
172.22.1.15   runtime     ExtensionStatus   2    1         schematic          613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245
172.22.1.16   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
172.22.1.16   runtime     ExtensionStatus   1    1         util-linux-tools   $VERSION
172.22.1.16   runtime     ExtensionStatus   2    1         schematic          613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245

安裝 Longhorn

$ curl -s https://raw.githubusercontent.com/longhorn/longhorn/v1.6.0/deploy/longhorn.yaml | sed 's/numberOfReplicas: "3"/numberOfReplicas: "2"/' | kubectl apply -f -

檢查是否部屬成功

$ kubectl get pods --namespace longhorn-system
NAME                                                READY   STATUS    RESTARTS      AGE
csi-attacher-57689cc84b-btwsj                       1/1     Running   0             12m
csi-attacher-57689cc84b-h7zw9                       1/1     Running   0             12m
csi-attacher-57689cc84b-zsb4f                       1/1     Running   0             12m
csi-provisioner-6c78dcb664-pc2f5                    1/1     Running   0             12m
csi-provisioner-6c78dcb664-vjw8z                    1/1     Running   0             12m
csi-provisioner-6c78dcb664-xfvc6                    1/1     Running   0             12m
csi-resizer-7466f7b45f-b8qq9                        1/1     Running   0             12m
csi-resizer-7466f7b45f-pbhwp                        1/1     Running   0             12m
csi-resizer-7466f7b45f-pdqr5                        1/1     Running   0             12m
csi-snapshotter-58bf69fbd5-54w76                    1/1     Running   0             12m
csi-snapshotter-58bf69fbd5-8vznv                    1/1     Running   0             12m
csi-snapshotter-58bf69fbd5-ps7nj                    1/1     Running   0             12m
engine-image-ei-acb7590c-6jq2h                      1/1     Running   0             12m
engine-image-ei-acb7590c-bhjs5                      1/1     Running   0             12m
instance-manager-45ef10875bdd08b79ebfe014faeb1722   1/1     Running   0             12m
instance-manager-55c27a0ae5e3ee437940cd838ea3ccef   1/1     Running   0             12m
longhorn-csi-plugin-f8zqf                           3/3     Running   0             12m
longhorn-csi-plugin-g22hx                           3/3     Running   0             12m
longhorn-driver-deployer-576d574c8-j2lzq            1/1     Running   0             12m
longhorn-manager-2jf5d                              1/1     Running   1 (12m ago)   12m
longhorn-manager-tfh54                              1/1     Running   0             12m
longhorn-ui-556f7bb76c-fwf5d                        1/1     Running   0             12m
longhorn-ui-556f7bb76c-ljflg                        1/1     Running   0             12m

測試 mysql database 飄移在不同節點上不會遺失資料

使用以下 yaml 測試

$ cat <<EOF > test-mysql.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql # has to match .spec.template.metadata.labels
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      restartPolicy: Always
      containers:
      - image: mysql:5.6
        name: mysql
        livenessProbe:
          exec:
            command:
              - ls
              - /var/lib/mysql/lost+found
          initialDelaySeconds: 5
          periodSeconds: 5
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: changeme
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-volume
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-volume
        persistentVolumeClaim:
          claimName: mysql-pvc
EOF

建立 mysql

$ kubectl apply -f test-mysql.yaml

檢查部屬狀態

$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
mysql-7b79bcfb7c-2zrxh   1/1     Running   0          33s   10.244.2.18   k1w2   <none>           <none>

在 mysql 產生資料

$ mysql -u root -p -h 10.244.2.18
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.51 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> create database haha;
Query OK, 1 row affected (0.001 sec)

MySQL [(none)]> use haha;
Database changed
MySQL [haha]> CREATE TABLE customer (
    ->     ID int NOT NULL,
    ->     NAME varchar(100) NOT NULL,
    ->     AGE int,
    ->     ADDRESS varchar(255),
    ->     SALARY DECIMAL(10,2),
    ->     PRIMARY KEY (ID)
    -> );
Query OK, 0 rows affected (0.028 sec)

MySQL [haha]>  INSERT INTO customer
    -> VALUES (1,'Ramesh',32,'Ahmedabad','2000.00'),
    -> (2,'Khilan',25,'Delhi','1500.00'),
    -> (3,'kaushik',23,'kota','2000.00'),
    -> (4,'Chaitali',25,'Mumbai','6500.00'),
    -> (5,'Hardik',27,'Bhopal','8500.00'),
    -> (6,'Komal',22,'MP','4500.00'),
    -> (7,'Muffy',24,'Indore','10000.00');
Query OK, 7 rows affected (0.005 sec)
Records: 7  Duplicates: 0  Warnings: 0

MySQL [haha]> select * from customer;
+----+----------+------+-----------+----------+
| ID | NAME     | AGE  | ADDRESS   | SALARY   |
+----+----------+------+-----------+----------+
|  1 | Ramesh   |   32 | Ahmedabad |  2000.00 |
|  2 | Khilan   |   25 | Delhi     |  1500.00 |
|  3 | kaushik  |   23 | kota      |  2000.00 |
|  4 | Chaitali |   25 | Mumbai    |  6500.00 |
|  5 | Hardik   |   27 | Bhopal    |  8500.00 |
|  6 | Komal    |   22 | MP        |  4500.00 |
|  7 | Muffy    |   24 | Indore    | 10000.00 |
+----+----------+------+-----------+----------+
7 rows in set (0.001 sec)

將 Pod 轉移到其他節點上，並觀察資料是否還在

$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
mysql-7b79bcfb7c-2zrxh   1/1     Running   0          5m    10.244.2.18   k1w2   <none>           <none>
bigred@kube-kadm:~$ kubectl cordon k1w2
node/k1w2 cordoned
bigred@kube-kadm:~$ kubectl scale deploy mysql --replicas=0
deployment.apps/mysql scaled
bigred@kube-kadm:~$ kubectl cordon k1w2
node/k1w2 already cordoned
bigred@kube-kadm:~$ kubectl get pods -o wide
No resources found in default namespace.
bigred@kube-kadm:~$ kubectl scale deploy mysql --replicas=1
deployment.apps/mysql scaled

$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
mysql-7b79bcfb7c-fz7pg   1/1     Running   0          30s   10.244.1.24   k1w1   <none>           <none>
bigred@kube-kadm:~$ mysql -u root -p -h 10.244.1.24
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.51 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> select * from haha.customer;
+----+----------+------+-----------+----------+
| ID | NAME     | AGE  | ADDRESS   | SALARY   |
+----+----------+------+-----------+----------+
|  1 | Ramesh   |   32 | Ahmedabad |  2000.00 |
|  2 | Khilan   |   25 | Delhi     |  1500.00 |
|  3 | kaushik  |   23 | kota      |  2000.00 |
|  4 | Chaitali |   25 | Mumbai    |  6500.00 |
|  5 | Hardik   |   27 | Bhopal    |  8500.00 |
|  6 | Komal    |   22 | MP        |  4500.00 |
|  7 | Muffy    |   24 | Indore    | 10000.00 |
+----+----------+------+-----------+----------+
7 rows in set (0.005 sec)

Block Volume 測試

建立 pvc

$ echo 'apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-block-vol
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  storageClassName: longhorn
  resources:
    requests:
      storage: 3Gi' | kubectl apply -f -

建立 pod 並且掛載 pvc

$ echo 'apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-with-raw-block-volume
  labels:
    os: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      os: alpine
  template:
    metadata:
      labels:
        os: alpine
    spec:
      containers:
      - name: alpine
        image: taiwanese/alpine:stable
        imagePullPolicy: IfNotPresent
        command: ["/bin/sleep", "infinity"]
        volumeDevices:
          - name: data
            devicePath: /dev/sdd
        securityContext:
          privileged: true
        lifecycle:
            postStart:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - |
                   set -e
                   mkdir /longhorn
                   checkformat=$(blkid | grep -w /dev/sdd | cut -d ':' -f1)
                   [[ "$checkformat" != /dev/sdd ]] && (mkfs.ext4 /dev/sdd && mount /dev/sdd /longhorn) || mount /dev/sdd /longhorn
            preStop:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - |
                    umount /dev/sdd
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: longhorn-block-vol' | kubectl apply -f -

$ kubectl get po
NAME                                         READY   STATUS    RESTARTS   AGE
pod-with-raw-block-volume-57b7bf4d5d-nmrzn   1/1     Running   0          20s

$ kubectl exec -it pod-with-raw-block-volume-57b7bf4d5d-5wl9l -- bash
bash-5.1# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0     4K  1 loop
loop1    7:1    0    28K  1 loop
loop2    7:2    0   2.3M  1 loop
loop3    7:3    0  53.8M  1 loop
loop4    7:4    0    16M  0 loop
loop5    7:5    0     3G  0 loop
sda      8:0    0   800G  0 disk
├─sda1   8:1    0   100M  0 part
├─sda2   8:2    0     1M  0 part
├─sda3   8:3    0  1000M  0 part
├─sda4   8:4    0     1M  0 part
├─sda5   8:5    0   100M  0 part
└─sda6   8:6    0 798.8G  0 part /etc/resolv.conf
                                 /etc/hostname
                                 /dev/termination-log
                                 /etc/hosts
sdb      8:16   0   120G  0 disk
sdc      8:32   0   120G  0 disk
sdd      8:48   0     3G  0 disk /longhorn
sr0     11:0    1  81.4M  0 rom

bash-5.1# echo 123 > /longhorn/test

bash-5.1# cat /longhorn/test
123

測試 pod 移轉

$ kubectl get po -owide
NAME                                         READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
pod-with-raw-block-volume-57b7bf4d5d-5wl9l   1/1     Running   0          59s   10.244.2.24   k1w1   <none>           <none>

$ kubectl scale deploy pod-with-raw-block-volume --replicas=0

$ kubectl cordon k1w1

$ kubectl scale deploy pod-with-raw-block-volume --replicas=1

$ kubectl get no
NAME   STATUS                     ROLES           AGE   VERSION
k1m1   Ready                      control-plane   47m   v1.29.2
k1w1   Ready,SchedulingDisabled   worker          42m   v1.29.2
k1w2   Ready                      worker          42m   v1.29.2

$ kubectl uncordon k1w1

$ kubectl get po -owide
NAME                                        READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
pod-with-raw-block-volume-6cfcd9889-t6pzt   1/1     Running   0          25s   10.244.1.21   k1w2   <none>           <none>

$ kubectl exec -it pod-with-raw-block-volume-6cfcd9889-t6pzt -- cat /longhorn/test
123

參考資料

Introduction to SUSE Rancher Rancher Monitoring