antony@notes:~/misc$ cat "中華電信-POC.md"

中華電信 POC

2024-04-02· misc

中華電信 POC

驗測項目

一、安裝/管理議題：

底層作業系統為RHEL9主機 | 環境內連線需保持全程加密(SSL憑證) | 完成交付安裝建置文件

完全離線架構下完成 RKE 叢集安裝，2個叢集 < RKE2架構｜v1.25版本｜ 3 control + 2 worker >（需包含 Elasticsearch、Grafana、cert-manager、hpa、istio、servicemesh、kiali、nmstate）
透過 Racher 同時管理 2個最小架構 RKE2 ; 實現兩個叢集內同時部署 Deployment （基本 Nginx image）提供服務。
透過 Prometheus/Grafana 監控 RKE2 運作狀況 ; 透過 Elasticsearch(EFK) 收容日誌並分析 ingress/叢集狀態。
叢集/ingress 自動化憑證更新設置。
ingress 限制同時間 Session 連線數量為 2000。
RBAC 定義使用者角色 (Cluster-admin / project-admin / 限制project-admin無法查看secret ) 對應HTPASSWD帳號 ( rkeadmin / infra01 / ap001 )
多租戶隔離（負載管理/資源配額設定）
測試告警發信

二、備份/還原議題：

備份叢集組態資訊 (etcd)
串接環境內提供備份軟體 (Veritas/Veeam) 備份資料。 (可跳過)
還原叢集組態設定。

三、升級議題：

升級中透過迴圈持續打 nginx 網頁連線，測試網頁連線狀況。
進行叢集升級 v1.25 > v1.26 > v1.27 測試升級時間及相關網頁連線狀況。
退版測試 ( v1.27 > v1.26 )

實作

一、安裝/管理議題

1.1. 完全離線架構下完成 RKE 叢集安裝，2個叢集 < RKE2架構｜v1.25版本｜ 3 control + 2 worker >（需包含 Elasticsearch、Grafana、cert-manager、hpa、istio、servicemesh[x]、kiali、nmstate[x]）

1.1.1. 準備 Harbor 並將 Container Images 都上傳進去

一台機器，最低規格: 4 CORE, 8G RAM, 250G Disk (建議 300G 以上)
Disk 檔案系統為 Btrfs，需啟動 LVM，將 / 目錄的可用空間設為 270G ，/home 目錄 30G
須能上網
檢查 Docker 儲存 Image 的目錄應為 /var/lib/docker

# 確認 Docker 儲存 Image 的目錄
$ sudo docker info -f '{{ .DockerRootDir}}'

# 檢查目錄的儲存容量
$ lsblk -o NAME,SIZE,MOUNTPOINTS

# 檢查 Docker 儲存 Image 目前已使用多大的儲存空間
$ sudo docker system df

# Download Git Repo
$ git clone https://github.com/braveantony/Prepare-AirGap-Installer.git && \
cd Prepare-AirGap-Installer/ && \
chmod +x *.sh

# 準備 harbor、rke2 和 Rancher 全離線安裝包
./podman-prepare.sh all

# 檢查是否符合預期
ls -lh compressed_files/
total 27G
-rw-r--r-- 1 rancher users 1.3G Mar 15 13:01 harbor-offline-v2.10.0.tar.gz
-rw-r--r-- 1 rancher users  25G Mar 15 11:47 rancher-airgap-v2.8.2.tar.gz
-rw-r--r-- 1 rancher users 956M Mar 15 13:59 rke2-airgap-v1.27.11.tar.gz

# SSH 連線至 Harbor 主機
$ ssh rancher@192.168.11.191

# 解壓縮 Harbor 全離線安裝包
$ tar -zxvf harbor-offline-v2.10.0.tar.gz && cd harbor/v2.10.0

# 將 docker-compose 移至 PATH 目錄中，並修改權限
$ sudo mv docker-compose-linux-x86_64 /usr/bin/docker-compose && sudo chmod 755 /usr/bin/docker-compose

# 解壓縮 harbor-offline-installer
$ tar -xvf harbor-offline-installer-v2.10.0.tgz harbor/; cd harbor/

# Generate a Certificate Authority Certificate
## Generate a CA certificate private key.
$ openssl genrsa -out ca.key 4096

## Generate the CA certificate.
$ openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=TW/ST=Taiwan/L=Taipei/O=example/OU=lab/CN=harbor.example.com" \
 -key ca.key \
 -out ca.crt
 
# Generate a Server Certificate
## Generate a private key.
$ openssl genrsa -out server.key 4096

## Generate a certificate signing request (CSR).
$ openssl req -sha512 -new \
    -subj "/C=TW/ST=Taiwan/L=Taipei/O=example/OU=lab/CN=harbor.example.com" \
    -key server.key \
    -out server.csr

## Generate an x509 v3 extension file.
$ cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=harbor.example.com
DNS.2=example.com
DNS.3=harbor
EOF

## Use the v3.ext file to generate a certificate for your Harbor host.
$ openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in server.csr \
    -out server.crt

# Copy the server certificate and key into the certficates folder on your Harbor host.
$ sudo cp ca.crt /usr/share/pki/trust/anchors/ && sudo cp ca.crt /etc/pki/trust/anchors/ && sudo update-ca-certificates --fresh

# Convert server.crt to server.cert, for use by Docker.
$ openssl x509 -inform PEM -in server.crt -out server.cert

# Copy the server certificate, key and CA files into the Docker certificates folder on the Harbor host. You must create the appropriate folders first.
$ sudo mkdir -p /etc/docker/certs.d/example.com/ && \
sudo cp server.cert /etc/docker/certs.d/example.com/ && \
sudo cp server.key /etc/docker/certs.d/example.com/ && \
sudo cp ca.crt /etc/docker/certs.d/example.com/ && \
sudo systemctl restart docker

# 設定 harbor.yml
$ cp harbor.yml.tmpl harbor.yml && nano harbor.yml
5  hostname: harbor.example.com
17 certificate: /etc/docker/certs.d/example.com/server.cert
18 private_key: /etc/docker/certs.d/example.com/server.key

# 安裝 Harbor
$ sudo ./install.sh --with-trivy

# 檢查 Harbor 服務運作狀態
$ sudo docker-compose ps

# 設定 Harbor 在機器開機時自動啟動
$ cat <<EOF | sudo tee /usr/lib/systemd/system/harbor.service && sudo systemctl enable harbor --now 
[Unit]
Description=Harbor start at boot
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service

[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f /root/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /root/harbor/docker-compose.yml down

[Install]
WantedBy=multi-user.target
EOF

# Login Harbor
$ sudo docker login harbor.example.com

# Create Harbor Project
$ curl -k -u "admin:Harbor12345" -X POST \
-H "Content-Type: application/json" \
"https://172.20.0.40.nip.io/api/v2.0/projects" \
-d "{\"project_name\": \"rancher\", \"metadata\": {\"public\": \"true\"}, \"storage_limit\": -1}"

# 解壓縮 rancher 全離線安裝包
$ tar -xzvf ~/rancher-airgap-v2.8.2.tar.gz && \
cd rancher/v2.8.2/

# Push Images to Harbor
$ rancher_image=$(sudo docker images --format "{{.Repository}}:{{.Tag}}" | grep  '^harbor.example.com')
$ for r in $rancher_image; do sudo docker push "${r}" || ( echo "${r}" push fail >> push.log &&
exit 1); done

1.1.2. 安裝 Upstream Cluster - RKE2 and Rancher

# SSH 連線至 RKE2 主機
$ ssh rancher@192.168.11.185

# Configure DNS
$ sudo yast2 dns edit nameserver1=192.168.11.85

# 檢查 Harbor 和 Rancher 的名稱解析
$ dig @192.168.11.85 harbor.example.com +short
192.168.11.191

$ dig @192.168.11.85 rancher.example.com +short
192.168.11.185

# 解壓縮 rke2 全離線安裝包
$ tar -zxvf rke2-airgap-v1.27.11.tar.gz && cd rke2/v1.27.11/

# 將 RKE2 全離線安裝包的物件，搬移至 /root/rke2-artifacts/
$ sudo mkdir /root/rke2-artifacts && sudo mv * /root/rke2-artifacts/

# 切換為 root 使用者，並將工作目錄切換至 /root/rke2-artifacts/
$ sudo su

$ cd /root/rke2-artifacts/

# 執行 RKE2 安裝腳本
$ INSTALL_RKE2_ARTIFACT_PATH=/root/rke2-artifacts sh install.sh

# 設定 RKE2 Node 之基本組態檔
$ mkdir -p /etc/rancher/rke2/ && nano /etc/rancher/rke2/config.yaml
node-name:
  - "rms"
token: my-shared-secret

$ exit

# Setup kubeconfig and kubectl
$ cd ~ && mkdir ~/.kube && \
sudo cp /etc/rancher/rke2/rke2.yaml .kube/config && \
sudo chown rancher .kube/config && \
sudo cp /var/lib/rancher/rke2/bin/* /usr/local/bin/

# 檢視 node 狀態
$ kubectl get nodes

# 安裝 Rancher Prime v2.8.2
$ tar -zxvf rancher-airgap-v2.8.2.tar.gz && cd rancher/v2.8.2/

$ rm rancher-v2.8.2-image.tar.gz rancher-*-images.sh

$ tar -zxvf helm-v3.14.2-linux-amd64.tar.gz && \
sudo cp linux-amd64/helm /usr/local/bin/

$ helm template cert-manager ./cert-manager-v1.11.0.tgz --output-dir . \
    --namespace cert-manager \
    --set image.repository=rancher/cert-manager-controller \
    --set webhook.image.repository=rancher/cert-manager-webhook \
    --set cainjector.image.repository=rancher/cert-manager-cainjector \
    --set startupapicheck.image.repository=rancher/cert-manager-ctl \
    --set installCRDs=true
    
$ kubectl create ns cert-manager

$ kubectl apply -R -f cert-manager/

$ kubectl -n cert-manager get pods
NAME                                       READY   STATUS      RESTARTS   AGE
cert-manager-554d4bf957-fqmtw              1/1     Running     0          4m21s
cert-manager-cainjector-687f649756-2jf92   1/1     Running     0          4m21s
cert-manager-startupapicheck-88t2l         0/1     Completed   0          17m
cert-manager-webhook-5d6598d446-tgkb7      1/1     Running     0          4m21s

$ helm template --validate rancher ./rancher-2.8.2.tgz --output-dir . --no-hooks --namespace cattle-system     --set hostname=rancher.example.com     --set certmanager.version=1.11.0     --set rancherImage=rancher/rancher     --set systemDefaultRegistry=harbor.example.com     --set useBundledSystemChart=true     --set replicas=1

$ kubectl create namespace cattle-system

$ kubectl -n cattle-system apply -R -f rancher/

$ kubectl -n cattle-system get pods
NAME                               READY   STATUS      RESTARTS   AGE
helm-operation-84qp9               0/2     Completed   0          2m15s
helm-operation-8588v               0/2     Completed   0          2m46s
helm-operation-dssm7               0/2     Completed   0          2m8s
helm-operation-qksdk               0/2     Completed   0          99s
rancher-594796d65c-rk8nk           1/1     Running     0          4m46s
rancher-webhook-6fbb4fc4cc-8ts9m   1/1     Running     0          2m3s

1.1.3. 建立 DownStream Cluster

環境準備

# 永久關閉防火牆
$ sudo systemctl disable --now firewalld.service

# 設定 NTP Client
$ sudo sed -i '3s|^|# |g' /etc/chrony.conf
$ sudo sed -i '4iserver 192.168.11.85' /etc/chrony.conf
$ sudo systemctl enable --now chronyd.service
$ sudo systemctl restart chronyd.service
$ systemctl is-active chronyd.service
active
$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 192.168.11.85                11   6     1    13   -9053m[ -9053m] +/-   12ms

$ timedatectl
               Local time: Wed 2024-03-13 02:32:52 CST
           Universal time: Tue 2024-03-12 18:32:52 UTC
                 RTC time: Tue 2024-03-19 01:26:12
                Time zone: Asia/Taipei (CST, +0800)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

將註冊指命分別貼到 5 台 node 上

設定 kubectl

$ sudo cp /var/lib/rancher/rke2/bin/* /usr/local/bin/

$ mkdir -p $HOME/.kube && sudo cp /etc/rancher/rke2/rke2.yaml $HOME/.kube/config && sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.1.4. 準備和測試 elasticsearch

# Helm install elasticsearch
# prepare node
$ mkdir -p ~/work/elasticsearch/8.5.1 && cd ~/work/elasticsearch/8.5.1
$ helm repo add elastic https://helm.elastic.co && \
helm repo update

$ helm fetch elastic/elasticsearch

$ helm template elasticsearch-*.tgz | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g'
docker.elastic.co/elasticsearch/elasticsearch:8.5.1

$ sudo docker pull docker.elastic.co/elasticsearch/elasticsearch:8.5.1

$ sudo docker images docker.elastic.co/elasticsearch/elasticsearch:8.5.1
REPOSITORY                                     TAG         IMAGE ID      CREATED        SIZE
docker.elastic.co/elasticsearch/elasticsearch  8.5.1       9d4620cd27bd  16 months ago  1.29 GB

$ sudo docker save harbor.example.com/elasticsearch/elasticsearch:8.5.1 | gzip --stdout > el
asticsearch-images-8.5.1.tar.gz

$ ls -lh
total 588M
-rw-r--r-- 1 rancher users  29K Mar 19 10:16 elasticsearch-8.5.1.tgz
-rw-r--r-- 1 rancher users 588M Mar 19 10:41 elasticsearch-images-8.5.1.tar.gz

$ cd ../..; tar -czvf compressed_files/elasticsearch-airgap-8.5.1.tar.gz elasticsearch/8.5.1

# 先在 Harbor 上建立 elasticsearch 專案
$ sudo docker push harbor.example.com/elasticsearch/elasticsearch:8.5.1

$ scp compressed_files/elasticsearch-airgap-8.5.1.tar.gz rancher@192.168.11.180:~


# on kubectl node
$ tar -zxvf elasticsearch-airgap-8.5.1.tar.gz && \
cd elasticsearch/8.5.1/

$ helm template elasticsearch ./elasticsearch-8.5.1.tgz --output-dir . \
--namespace elasticsearch --create-namespace \
--set image=harbor.example.com/elasticsearch/elasticsearch \
--set replicas=2

$ sed -i 's|docker.elastic.co|harbor.example.com|g' ./elasticsearch/templates/test/test-elasticsearch-health.yaml

$ cat <<EOF > elasticsearch-pv.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
  name: elasticsearch-0
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/var/lib/elasticsearch-0"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-role.kubernetes.io/worker
          operator: In
          values:
          - 'true'
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: elasticsearch-1
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/var/lib/elasticsearch-1"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: node-role.kubernetes.io/worker
          operator: In
          values:
          - 'true'
EOF

$ kubectl create -f elasticsearch-pv.yaml

# 前往 worker 主機建 PV 宣告的目錄
$ ssh rancher@192.168.11.183 'sudo mkdir -p /var/lib/{elasticsearch-0,elasticsearch-1}'
$ ssh rancher@192.168.11.184 'sudo mkdir -p /var/lib/{elasticsearch-0,elasticsearch-1}'

$ kubectl apply -R -f elasticsearch/

$ kubectl replace -f elasticsearch/templates/test/test-elasticsearch-health.yaml --force

$ kubectl get pods
NAME                       READY   STATUS      RESTARTS   AGE
elasticsearch-jqhml-test   0/1     Completed   0          6s
elasticsearch-master-0     1/1     Running     0          4m8s
elasticsearch-master-1     1/1     Running     0          4m8s
elasticsearch-ppxis-test   0/1     Completed   0          6s
elasticsearch-zskgi-test   0/1     Completed   0          7s

RUN Local Elastic Package Registry

$ mkdir -p ~/work/elasticsearch-package-registry/8.5.3 && cd ~/work/elasticsearch-package-registry/8.5.3

# Pull the Docker image from the public Docker registry:
$ sudo docker pull docker.elastic.co/package-registry/distribution:8.5.3

$ sudo docker images docker.elastic.co/package-registry/distribution:8.5.3
REPOSITORY                                        TAG       IMAGE ID       CREATED         SIZE
docker.elastic.co/package-registry/distribution   8.5.3     e37a947c8f56   15 months ago   2.64GB

$ sudo docker tag docker.elastic.co/package-registry/distribution:8.5.3 harbor.example.com/package-registry/distribution:8.5.3

$ sudo docker save harbor.example.com/package-registry/distribution:8.5.3 | gzip --stdout > ../../compressed_files/package-registry-images-8.5.3.tar.gz

$ ls -lh ../../compressed_files/package-registry-images-8.5.3.tar.gz
-rw-r--r-- 1 rancher users 1.9G Mar 25 15:09 ../../compressed_files/package-registry-images-8.5.3.tar.gz

$ sudo docker load < package-registry-images-8.5.3.tar.gz

$ sudo docker push harbor.example.com/package-registry/distribution:8.5.3

$ sudo docker run -d -p 8080:8080 harbor.example.com/package-registry/distribution:8.5.3

1.1.5. 準備和測試 kiali

# Helm install kiali
$ helm repo add kiali https://kiali.org/helm-charts && \
helm repo update

$ helm fetch kiali/kiali-operator

$ helm template kiali-operator-*.tgz | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g'
quay.io/kiali/kiali-operator:v1.81.0

$ sudo podman pull quay.io/kiali/kiali-operator:v1.81.0

$ sudo docker images quay.io/kiali/kiali-operator:v1.81.0
REPOSITORY                     TAG       IMAGE ID       CREATED       SIZE
quay.io/kiali/kiali-operator   v1.81.0   1434ee8c1d9f   2 weeks ago   764MB

1.1.6. 準備和測試 Grafana

直接透過 Rancher 裝 Monitoring cluster tools

1.1.7. 準備和測試 istio、servicemesh、kiali

在 Rancher 介面安裝 Istio Cluster Tools 即可

1.1.8. 測試 hpa

# Run and expose nginx web server
$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - name: nginx
        image: harbor.example.com/rancher/library-nginx:1.19.2-alpine
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    run: nginx
spec:
  ports:
  - port: 80
  selector:
    run: nginx
EOF

# Create the HorizontalPodAutoscaler
$ kubectl autoscale deployment nginx --cpu-percent=50 --min=1 --max=10

$ kubectl get hpa

# Increase the load
$ kubectl create deployment load-generator --image=harbor.example.com/rancher/mirrored-library-busybox:1.36.1 -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://nginx; done"

$ kubectl scale deploy load-generator --replicas=10

$ kubectl scale deploy load-generator --replicas=20

$ kubectl scale deploy load-generator --replicas=30

$ kubectl scale deploy load-generator --replicas=40

$ kubectl scale deploy load-generator --replicas=50

$ kubectl scale deploy load-generator --replicas=60

# more replicas
$ kubectl get hpa nginx
NAME    REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
nginx   Deployment/nginx   29%/50%   1         10        2          36m

# Stop generating load
$ kubectl scale deploy load-generator --replicas=0

$ kubectl get hpa
NAME    REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
nginx   Deployment/nginx   0%/50%    1         10        1          89m

1.1.9. 準備和測試 nmstate

# on prepare node
$ mkdir -p ~/work/nmstate/v0.81.0 && cd ~/work/nmstate/v0.81.0

$ curl -# -OL https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.81.0/nmstate.io_nmstates.yaml && \
curl -# -OL https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.81.0/namespace.yaml && \
curl -# -OL https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.81.0/service_account.yaml && \
curl -# -OL https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.81.0/role.yaml && \
curl -# -OL https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.81.0/role_binding.yaml && \
curl -# -OL https://github.com/nmstate/kubernetes-nmstate/releases/download/v0.81.0/operator.yaml

$ cat ./* | grep image:
          image: quay.io/nmstate/kubernetes-nmstate-operator:v0.81.0
          
$ sudo docker pull quay.io/nmstate/kubernetes-nmstate-operator:v0.81.0

$ sudo docker tag quay.io/nmstate/kubernetes-nmstate-operator:v0.81.0 harbor.example.com/nmstate/kubernetes-nmstate-operator:v0.81.0

# 先在 Harbor 建立一個 nmstate 專案
$ sudo docker push harbor.example.com/nmstate/kubernetes-nmstate-operator:v0.81.0

$ sudo docker save harbor.example.com/nmstate/kubernetes-nmstate-operator:v0.81.0 | gzip --stdout > nmstate-images.v0.81.0.tar.gz

$ cd ../..; tar -czvf compressed_files/nmstate-airgap-v0.81.0.tar.gz nmstate/v0.81.0

$ ls -lh compressed_files/nmstate-airgap-v0.81.0.tar.gz
-rw-r--r-- 1 rancher users 55M Mar 19 15:56 compressed_files/nmstate-airgap-v0.81.0.tar.gz

# on kubectl node
$ tar -zxvf nmstate-airgap-v0.81.0.tar.gz && cd nmstate/v0.81.0

$ kubectl apply -f ./

# Configure Image Repo
$ sed -i 's|quay.io|harbor.example.com|g' operator.yaml

$ kubectl -n nmstate get pods
NAME                                READY   STATUS    RESTARTS   AGE
nmstate-operator-77c48bc498-jjntx   1/1     Running   0          16s

$ cat <<EOF | kubectl create -f -
apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate
EOF

$ kubectl -n nmstate get NMState
NAME      AGE
nmstate   36s

# Helm install kibana
# prepare node
$ mkdir -p ~/work/kibana/8.5.1 && cd ~/work/kibana/8.5.1
$ helm repo add elastic https://helm.elastic.co && \
helm repo update

$ helm fetch elastic/kibana

$ helm template kibana-*.tgz | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g'
docker.elastic.co/kibana/kibana:8.5.1

$ sudo docker pull docker.elastic.co/kibana/kibana:8.5.1

$ sudo docker images docker.elastic.co/kibana/kibana:8.5.1
REPOSITORY                        TAG       IMAGE ID       CREATED         SIZE
docker.elastic.co/kibana/kibana   8.5.1     1a5dbfe4bf27   16 months ago   707MB

$ sudo docker save docker.elastic.co/kibana/kibana:8.5.1 | gzip --stdout > kibana-images-8.5.1.tar.gz

$ cd ../..; tar -czvf compressed_files/kibana-airgap-8.5.1.tar.gz kibana/8.5.1

# Helm install logstash
# prepare node
$ mkdir -p ~/work/logstash/8.5.1 && cd ~/work/logstash/8.5.1
$ helm repo add elastic https://helm.elastic.co && \
helm repo update

$ helm fetch elastic/logstash

$ helm template logstash-*.tgz | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g'
docker.elastic.co/kibana/kibana:8.5.1

$ sudo docker pull docker.elastic.co/logstash/logstash:8.5.1

$sudo docker images docker.elastic.co/logstash/logstash:8.5.1
REPOSITORY                            TAG       IMAGE ID       CREATED         SIZE
docker.elastic.co/logstash/logstash   8.5.1     db6effbaf70b   16 months ago   746MB

$ sudo docker save docker.elastic.co/logstash/logstash:8.5.1 | gzip --stdout > logstash-images-8.5.1.tar.gz

$ cd ../..; tar -czvf compressed_files/logstash-airgap-8.5.1.tar.gz logstash/8.5.1

1.2. 實現兩個叢集內同時部署 Deployment （基本 Nginx image）提供服務。

$ sudo yum install git

# generate a private ssh key
$ ssh-keygen -t rsa -b 4096 -m pem -C "user@email.com" -P ''

$ kubectl create secret generic ssh-key -n fleet-default \
--from-file=ssh-privatekey=/home/rancher/.ssh/id_rsa  \
--from-file=ssh-publickey=/home/rancher/.ssh/id_rsa.pub \
--type=kubernetes.io/ssh-auth

$ mkdir -p ~/fleet && \
cd ~/fleet && \
git config --global user.name "bobo" && \
git config --global user.email "bobo@example.com" && \
git config --global core.editor "nano" && \
git config --list 

$ git init

$ cat <<EOF > deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: hello-nginx
spec:
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - name: nginx
        image: harbor.example.com/rancher/library-nginx:1.19.2-alpine
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m
EOF

$ cat <<EOF > namespace.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: hello-nginx
EOF

$ cat <<EOF > service.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    run: nginx
  namepsace: hello-nginx
spec:
  ports:
  - port: 80
  selector:
    run: nginx
EOF

$ git add . && git commit -m "init repo"

# Create Cluster Groups
1. 在左側選單點選 Cluster Groups
2. 點選 Create 按鈕
Cluster Group: Create page will be displayed.
3. For the Name, enter rke2-group
4. 點選 Add Rule 然後輸入以下訊息 : 
Key: provider.cattle.io
Operator: in list
Value: rke2
Notice the following message displayed:
Matches all 2 existing clusters
5. 點選 Create

# Configure a Git Repo
1. 在左側選單點選 Git Repos
2. 點選 Add Repository 按鈕
3. 輸入以下訊息
Name: hello-nginx
Repository URL: ssh://192.168.11.185/home/rancher/fleet
Branch Name master
Paths Add Path → hello-rancher
Deploy To rke-group
4. 點選 Create.
You will see the State starting with GitUpdating, then changing to Active.

1.3. 透過 Prometheus/Grafana 監控 RKE2 運作狀況 ; 透過 Elasticsearch(EFK) 收容日誌並分析 ingress/叢集狀態。

離線裝 Elasticsearch

點選 apps -> Charts -> Elasticsearch -> install

values.yaml

image: harbor.example.com/elasticsearch/elasticsearch
...
replicas: 2

離線安裝 kibana

點選 apps -> Charts -> kibana -> install

values.yaml

image: harbor.example.com/kibana/kibana # 全離線再改
...
kibanaConfig:
  kibana.yml: |
    xpack.fleet.registryUrl: http://192.168.11.191:8080
...
service:
  type: NodePort

安裝 Rancher Logging

點選 Cluster Tools -> Logging -> Install

values.yaml

fluentbit:
  ...
  inputTail:
    Buffer_Chunk_Size: 1MB
    Buffer_Max_Size: 5MB
  ...
  resources:
    limits:
      cpu: "2"
      memory: 2Gi
    requests:
      cpu: "1"
      memory: 1Gi
fluentd:
  ...
  resources:
    limits:
      cpu: "2"
      memory: 2Gi
    requests:
      cpu: "1"
      memory: 1Gi

Run nginx

$ kubectl run nginx --image=harbor.example.com/rancher/library-nginx:1.19.2-alpine -l run=nginx

$ kubectl get pods -l run=nginx -o wide
NAME    READY   STATUS    RESTARTS   AGE     IP           NODE   NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          2m23s   10.42.44.2   k2w1   <none>           <none>

$ NGINX_PodIP=$(kubectl get pods -l run=nginx -o custom-columns=POD_IP:.status.podIP --no-headers)

$ curl http://$NGINX_PodIP

建立與設定 ClusterOutput

Logging -> ClusterOutputs -> Create

Name: cluster-output-e
Output: Elasticsearch
Target.Scheme: https
Target.Host: elasticsearch-master.cattle-logging-system.svc.cluster.local
Target.Port: 9200
Index Name: nginx
Access.User: elastic
Access.Password from Secret: elasticsearch-master-credentials
Access.Key: password
Connection: 取消勾選 Verify SSL
點選 Create 按鈕

建立與設定 ClusterFlows

Logging -> ClusterFlows -> Create

Name: clusterflow-nginx
Matches.Pod Label Key: run
Matches.Pod Label Value: nginx
Outputs.Cluster Outputs: cluster-output-e
點選 Create 按鈕

等新建立的 rancher-logging-root-fluentd-configcheck-* Pod Completed 之後，

$ while true; do curl $NGINX_PodIP &> /dev/null; done &

# 停止 while true loop
$ kill $!

連線至 kibana

Service 找到 kibana-kibana ，直接點 NodePort，就會連線到 kibana 的網站。登入的帳號密碼存在 cattle-logging-system Namespace 底下的 elasticsearch-master-credentials 這個 Secret 中。要 Add Integrations -> 選擇 Kibana -> Settings -> Install

大概過 3 分鐘後，連到 kibana 網站，點選漢堡排 -> Stack Management -> Index Management

建立 kibana view data

漢堡排 -> Stack Management -> Kibana -> Data View -> Create Data view

Name: nginx
Index pattern: nginx*
按 Save data view to Kibana 按鈕

查詢 log

漢堡排 -> Dev Tools -> Console

POST /nginx/_search
{
  "query": {
    "match_all": {}
  }
}

1.4. 叢集/ingress 自動化憑證更新設置

安裝 Cert-manager 步驟: Apps -> Charts -> cert-manager -> install -> v1.11.0 -> 改 Image Tag 名稱 -> 改 installCRD = true -> Next -> install

# 使用 cert-manager SelfSigned 部屬 ClusterIssuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: default
spec:
  selfSigned: {}

## 檢查
$ kubectl get issuers -o wide
NAME                READY   STATUS   AGE
selfsigned-issuer   True             20s
  
# 生成憑證
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-ca
  namespace: default
spec:
  isCA: true
  commonName: test-ca
  subject:
    organizations:
      - ACME Inc.
    organizationalUnits:
      - Widgets
  secretName: test-ca-secret
  duration: 2h
  renewBefore: 1h
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io

# 透過 ca 憑證，產生 ingress 要用的憑證發布者
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-ca-issuer
spec:
  ca:
    secretName: test-ca-secret

# add an annotation indicating the issuer to use.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-nginx-ingress
  namespace: default
  annotations:
    # add an annotation indicating the issuer to use.
    cert-manager.io/issuer: test-ca-issuer
spec:
  ingressClassName: nginx
  tls:
  - hosts:
      - 192.168.11.174.nip.io
    secretName: echo-cert
  rules:
  - host: 192.168.11.174.nip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

# 匯出 CA 憑證
$ kubectl get secret test-ca-secret -o jsonpath='{.data.ca\.crt}' | base64 -d

# 裝 cmctl
$ tar xzf cmctl.tar.gz && sudo mv cmctl /usr/local/bin

$ echo | openssl s_client -connect 192.168.11.174.nip.io:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Mar 27 03:26:12 2024 GMT
notAfter=Jun 25 03:26:12 2024 GMT

$ kubectl get certificate
NAME        READY   SECRET           AGE
echo-cert   True    echo-cert        104m
test-ca     True    test-ca-secret   106m

$ kubectl get certificaterequest --sort-by=.metadata.creationTimestamp
echo-cert-nbd69   True                True    test-ca-issuer      system:serviceaccount:cert-manager:cert-manager   20s

$ echo | openssl s_client -connect 192.168.11.174.nip.io:443 2>/dev/null | openssl x509 -noout -dates

叢集自動更新

先檢查時間

sudo ./k8s-cert_check
client-admin.crt:              Mar 27 04:55:53 2025 GMT
client-auth-proxy.crt:         Mar 27 04:55:53 2025 GMT
client-ca.crt:                 Mar 18 08:48:11 2034 GMT
client-ca.nochain.crt:         Mar 18 08:48:11 2034 GMT
client-controller.crt:         Mar 27 04:55:53 2025 GMT
client-kube-apiserver.crt:     Mar 27 04:55:53 2025 GMT
client-kube-proxy.crt:         Mar 27 04:55:53 2025 GMT
client-rke2-cloud-controller.crt: Mar 27 04:55:53 2025 GMT
client-rke2-controller.crt:    Mar 27 04:55:53 2025 GMT
client-scheduler.crt:          Mar 27 04:55:53 2025 GMT
client-supervisor.crt:         Mar 21 08:26:55 2025 GMT
request-header-ca.crt:         Mar 18 08:48:11 2034 GMT
server-ca.crt:                 Mar 18 08:48:11 2034 GMT
server-ca.nochain.crt:         Mar 18 08:48:11 2034 GMT
serving-kube-apiserver.crt:    Mar 27 04:55:53 2025 GMT
client-ca.crt:                 Mar 18 08:48:11 2034 GMT
client-kubelet.crt:            Mar 27 04:55:55 2025 GMT
client-kube-proxy.crt:         Mar 27 04:55:53 2025 GMT
client-rke2-controller.crt:    Mar 27 04:55:53 2025 GMT
server-ca.crt:                 Mar 18 08:48:11 2034 GMT
serving-kubelet.crt:           Mar 27 04:55:54 2025 GMT

Rotate 叢集憑證
再執行一次腳本確認

1.5. ingress 限制同時間 Session 連線數量為 2000

1.5.1. Enable the annotation validation feature with ingress nginx controller

修改 ingress nginx controller 的 YAML 檔

Workloads -> DaemonSets -> kube-system -> Edit YAML

在 spec.template.spec.containers.args 底下新增 --enable-annotation-validation=true 參數，這樣可以開啟 annotations 功能，新增後按 Save

1.5.2. 測試單一 IP 位址允許的同時連線數

建立以下 Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/limit-connections: "1"
spec:
  ingressClassName: nginx
  rules:
  - host: 192.168.11.173.nip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

Curl 網站測試

$ while true; do curl -s http://192.168.11.173.nip.io | grep 503 >> check.txt; done &
[1] 25917

$ cat check.txt

$ while true; do curl -s http://192.168.11.173.nip.io | grep 503 >> check.txt; done &
[2] 28690

$ cat check.txt
<head><title>503 Service Temporarily Unavailable</title></head>

1.7. 多租戶隔離（負載管理/資源配額設定）

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: default
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 256Mi
    requests.nvidia.com/gpu: 1
    limits.cpu: "2"
    limits.memory: 1Gi
    limits.nvidia.com/gpu: 2

1.8. 測試告警發信

1.8.1. 建立 SMTP 的 TLS Secret

$ kubectl create ns cattle-monitoring-system

$ kubectl -n cattle-monitoring-system create secret generic smtp-tls-secret \
--from-file=tls.crt=/home/rancher/smtp/server.crt \
--from-file=tls.key=/home/rancher/smtp/server.key \
--from-file=ca.crt=/home/rancher/smtp/ca.crt

需準備 CA 憑證、還有 SMTP Server 的憑證和私鑰

1.8.2. 建立 Rancher Monitoring

點選 Cluster Tools -> 找到 Monitoring 點選 Install
勾選 Customize Helm options brfore install -> Next
展開 Additional Secrets 選單，選擇 smtp-tls-secret
Install
安裝成功畫面

1.8.2. 設定 Alert-manager 告警對象為 E-mail

點選左側選單 Storage -> Secrets -> 選擇 “cattle-monitoring-system” Namespace -> 找到 alertmanager-rancher-monitoring-alertmanager 這個 Secret -> 點選 Edit Config
複製以下 alertmanager.yaml 的內容，並再刪除所有原始的值後，貼到對應的 Value 欄位中

global:
  resolve_timeout: 5m
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: Mail Alert
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
  - match:
      alertname: DemoAlertName
    receiver: 'Mail Alert'

receivers:
- name: Mail Alert
  email_configs:
  - to: antony.chen@beeops.com.tw
    from: rancher@lab.com
    # Your smtp server address
    smarthost: smtp.example.com:587
    tls_config:
      ca_file: /etc/alertmanager/secrets/smtp-tls-secret/ca.crt
      cert_file: /etc/alertmanager/secrets/smtp-tls-secret/tls.crt
      key_file: /etc/alertmanager/secrets/smtp-tls-secret/tls.key
    headers:
      From: rancher@lab.com
      Subject: 'Demo ALERT'

確認設定檔是否修改成功，點選 Monitoring -> Alert Manager
點選 Status
檢視 Config 內容
打開信箱，看 Alert

如果沒有 Alert 的話，可以透過 curl api 發送一個測試的 Alert
$ curl -H 'Content-Type: application/json' -d '[{"labels":{"alertname":"myalert"}}]' http://10.43.130.225:9093/api/v1/alerts
10.43.130.225 為 Alertmanager Service 的 IP，執行以下命令可以找到 $ kubectl -n cattle-monitoring-system get svc -l app=rancher-monitoring-alertmanager

二、備份/還原議題：

1. 備份叢集組態資訊 (etcd)

rancher 介面 Take Snapshot

2. 串接環境內提供備份軟體 (Veritas/Veeam) 備份資料。 (可跳過)

3. 還原叢集組態設定。

rancher 介面 restore

三、升級議題：

升級中透過迴圈持續打 nginx 網頁連線，測試網頁連線狀況。
進行叢集升級 v1.25 > v1.26 > v1.27 測試升級時間及相關網頁連線狀況。
退版測試 ( v1.27 > v1.26 )

3.1. 建 nginx 並跑 while 迴圈

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      run: nginx
  template:
    metadata:
      labels:
        run: nginx
    spec:
      containers:
      - name: nginx
        image: harbor.example.com/rancher/library-nginx:1.19.2-alpine
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: 500m
          requests:
            cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    run: nginx
spec:
  ports:
  - port: 80
  selector:
    run: nginx
EOF

$ while true; do curl <service cluster ip>; done

3.2. TroubleShooting

問題描述

RKE2 從 v1.27.11 Restore 回 v1.26.11 的版本時，發現 m3 的 node 有異常

問題檢查

$ journalctl -u rke2-server.service --no-pager

看到以下錯誤訊息

...
{
  "level": "warn",
  "ts": "2024-03-20T15:54:37.011625+0800",
  "logger": "etcd-client",
  "caller": "v3@v3.5.9-k3s1/retry_interceptor.go:62",
  "msg": "retrying of unary invoker failed",
  "target": "etcd-endpoints://0xc000775880/127.0.0.1:2379",
  "attempt": 0,
  "error": "rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""
}
...
{
  "level": "info",
  "ts": "2024-03-20T14:50:38.133657+0800",
  "logger": "etcd-client",
  "caller": "v3@v3.5.9-k3s1/client.go:210",
  "msg": "Auto sync endpoints failed.",
  "error": "context deadline exceeded"
}
...
level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"

看 ETCD Member

$ kubectl -n kube-system exec -it etcd-k2m1 -- bash
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl member list -w table

螢幕輸出

+------------------+---------+---------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |     NAME      |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+
| 6fc66799bcea7735 | started | k2m1-3cc51c7e | https://192.168.11.170:2380 | https://192.168.11.170:2379 |      false |
| a7d16fd4a2d2688b | started | k2m2-308b0b3a | https://192.168.11.171:2380 | https://192.168.11.171:2379 |      false |
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+

Root cause

m3 node 的 etcd 要加入 etcd cluster 的時候超過 Timeout 時間，因為 etcd 根據給定的端點，對 etcd 成員的儲存空間進行碎片整理預設 Timeout 時間為 5 秒

問題修復

$ kubectl -n kube-system exec -it etcd-k2m1 -- bash
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl --command-timeout=30s defrag

螢幕輸出 :

Finished defragmenting etcd member[https://127.0.0.1:2379]

再看一次 ETCD Member

$ kubectl -n kube-system exec -it etcd-k2m1 $etcdpod -- bash
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl member list -w table

螢幕輸出

+------------------+---------+---------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |     NAME      |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+
| 66077b12da5276cb | started | k2m3-e637cb23 | https://192.168.11.172:2380 | https://192.168.11.172:2379 |      false |
| 6fc66799bcea7735 | started | k2m1-3cc51c7e | https://192.168.11.170:2380 | https://192.168.11.170:2379 |      false |
| a7d16fd4a2d2688b | started | k2m2-308b0b3a | https://192.168.11.171:2380 | https://192.168.11.171:2379 |      false |
+------------------+---------+---------------+-----------------------------+-----------------------------+------------+

成功加入~

註冊 RHEL Google

中華電信 POC

中華電信 POC

驗測項目

一、 安裝/管理議題：

二、備份/還原議題：

三、升級議題：

實作

一、 安裝/管理議題

1.1. 完全離線架構下完成 RKE 叢集安裝，2個叢集 < RKE2架構｜v1.25版本｜ 3 control + 2 worker >（需包含 Elasticsearch、Grafana、cert-manager、hpa、istio、servicemesh[x]、kiali、nmstate[x]）

1.1.1. 準備 Harbor 並將 Container Images 都上傳進去

1.1.2. 安裝 Upstream Cluster - RKE2 and Rancher

1.1.3. 建立 DownStream Cluster

環境準備

1.1.4. 準備和測試 elasticsearch

RUN Local Elastic Package Registry

1.1.5. 準備和測試 kiali

1.1.6. 準備和測試 Grafana

1.1.7. 準備和測試 istio、servicemesh、kiali

1.1.8. 測試 hpa

1.1.9. 準備和測試 nmstate

1.2. 實現兩個叢集內同時部署 Deployment （基本 Nginx image）提供服務。

1.3. 透過 Prometheus/Grafana 監控 RKE2 運作狀況 ; 透過 Elasticsearch(EFK) 收容日誌並分析 ingress/叢集狀態。

離線裝 Elasticsearch

離線安裝 kibana

安裝 Rancher Logging

Run nginx

建立與設定 ClusterOutput

建立與設定 ClusterFlows

連線至 kibana

建立 kibana view data

查詢 log

1.4. 叢集/ingress 自動化憑證更新設置

叢集自動更新

1.5. ingress 限制同時間 Session 連線數量為 2000

1.5.1. Enable the annotation validation feature with ingress nginx controller

1.5.2. 測試單一 IP 位址允許的同時連線數

1.7. 多租戶隔離 （負載管理/資源配額設定）

1.8. 測試告警發信

1.8.1. 建立 SMTP 的 TLS Secret

1.8.2. 建立 Rancher Monitoring

1.8.2. 設定 Alert-manager 告警對象為 E-mail

二、備份/還原議題：

1. 備份叢集組態資訊 (etcd)

2. 串接環境內提供 備份軟體 (Veritas/Veeam) 備份資料。 (可跳過)

3. 還原叢集組態設定。

三、升級議題：

3.1. 建 nginx 並跑 while 迴圈

3.2. TroubleShooting

問題描述

問題檢查

Root cause

問題修復

一、安裝/管理議題：

一、安裝/管理議題

1.7. 多租戶隔離（負載管理/資源配額設定）

2. 串接環境內提供備份軟體 (Veritas/Veeam) 備份資料。 (可跳過)