antony@notes:~/kubernetes$ cat "Step-by-step-install-HA-Kubernetes-with-Kubeadm-in-Disconnected-environment.md"
Step by step install HA Kubernetes with Kubeadm in Disconnected environment
Step by step install HA Kubernetes with Kubeadm in Disconnected environment
0. Preface
本篇文章會介紹，如何在全離線的環境下，在多台 VM 上使用 kubeadm 安裝 3 台 Control-plane Node 和 3 台 Worker Node 架構的 Kubernetes Cluster
0.1. 環境架構規劃
1. 準備 K8s 全離線安裝包
1.1. 準備 K8s 軟體套件
連線到一台 Disk 空間 70G 或以上, 可以透過網路正常連線到公有網路的 RHEL 9 電腦
註冊 RHEL 9
sudo rhc connect --activation-key $ACTIVATION-KEY --organization $ORGANIZATION設定 Kubernetes 和 CRI-O 版本
KUBERNETES_VERSION=v1.34 CRIO_VERSION=v1.34添加 Kubernetes yum repository
cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/rpm/repodata/repomd.xml.key EOF添加 CRI-O yum repository
cat <<EOF | sudo tee /etc/yum.repos.d/cri-o.repo [cri-o] name=CRI-O baseurl=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/ enabled=1 gpgcheck=1 gpgkey=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/repodata/repomd.xml.key EOF建立工作目錄
mkdir -p "$HOME"/k8s/{addon/{cni/cilium,csi/nfs-provisioner,ingress/ingress-nginx,metrics/metrics-server,logging/{fluentd,logging-operator},kube-vip/{ha-control-plane,k8s-lb-service}},img,pkg/{kubernetes,podman,skopeo},config}目錄結構如下
tree "$HOME"/k8s執行結果：
/home/bigred/k8s ├── addon │ ├── cni │ │ └── cilium │ ├── csi │ │ └── nfs-provisioner │ ├── ingress │ │ └── ingress-nginx │ ├── kube-vip │ │ ├── ha-control-plane │ │ └── k8s-lb-service │ ├── logging │ │ ├── fluentd │ │ └── logging-operator │ └── metrics │ └── metrics-server ├── config ├── img └── pkg ├── kubernetes ├── podman └── skopeo 21 directories, 0 files將必要套件載入指定目錄
sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/kubernetes -y \ crun \ runc \ cri-o \ kubeadm \ kubectl \ kubelet檢視已載入的套件
ls -lh "$HOME"/k8s/pkg/kubernetes執行結果：
total 78M -rw-r--r--. 1 root root 240K Oct 16 16:28 conntrack-tools-1.4.7-4.el9_5.x86_64.rpm -rw-r--r--. 1 root root 66K Oct 16 16:28 container-selinux-2.235.0-1.el9_6.noarch.rpm -rw-r--r--. 1 root root 21M Oct 16 16:28 cri-o-1.34.1-150500.1.1.x86_64.rpm -rw-r--r--. 1 root root 7.3M Oct 16 16:28 cri-tools-1.34.0-150500.1.1.x86_64.rpm -rw-r--r--. 1 root root 563K Oct 16 16:28 criu-3.19-1.el9.x86_64.rpm -rw-r--r--. 1 root root 33K Oct 16 16:28 criu-libs-3.19-1.el9.x86_64.rpm -rw-r--r--. 1 root root 244K Oct 16 16:28 crun-1.23.1-2.el9_6.x86_64.rpm -rw-r--r--. 1 root root 13M Oct 16 16:28 kubeadm-1.34.1-150500.1.1.x86_64.rpm -rw-r--r--. 1 root root 12M Oct 16 16:28 kubectl-1.34.1-150500.1.1.x86_64.rpm -rw-r--r--. 1 root root 13M Oct 16 16:28 kubelet-1.34.1-150500.1.1.x86_64.rpm -rw-r--r--. 1 root root 8.6M Oct 16 16:28 kubernetes-cni-1.7.1-150500.1.1.x86_64.rpm -rw-r--r--. 1 root root 60K Oct 16 16:28 libnet-1.2-7.el9.x86_64.rpm -rw-r--r--. 1 root root 26K Oct 16 16:28 libnetfilter_cthelper-1.0.0-22.el9.x86_64.rpm -rw-r--r--. 1 root root 26K Oct 16 16:28 libnetfilter_cttimeout-1.0.0-19.el9.x86_64.rpm -rw-r--r--. 1 root root 31K Oct 16 16:28 libnetfilter_queue-1.0.5-1.el9.x86_64.rpm -rw-r--r--. 1 root root 38K Oct 16 16:28 protobuf-c-1.3.3-13.el9.x86_64.rpm -rw-r--r--. 1 root root 3.6M Oct 16 16:28 runc-1.2.4-2.el9.x86_64.rpm -rw-r--r--. 1 root root 42K Oct 16 16:28 yajl-2.1.0-25.el9.x86_64.rpm下載並準備 podman pkg
sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/podman -y \ podman檢視已載入的套件
ls -l "$HOME"/k8s/pkg/podman執行結果：
total 24048 -rw-r--r--. 1 bigred bigred 902510 Oct 16 16:59 aardvark-dns-1.14.0-1.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 55519 Oct 16 16:59 conmon-2.1.12-1.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 158417 Oct 16 16:59 containers-common-1-117.el9_6.x86_64.rpm -rw-r--r--. 1 bigred bigred 67105 Oct 16 16:59 container-selinux-2.235.0-1.el9_6.noarch.rpm -rw-r--r--. 1 bigred bigred 576009 Oct 16 16:59 criu-3.19-1.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 33643 Oct 16 16:59 criu-libs-3.19-1.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 249255 Oct 16 16:59 crun-1.23.1-2.el9_6.x86_64.rpm -rw-r--r--. 1 bigred bigred 58706 Oct 16 16:59 fuse3-3.10.2-9.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 95573 Oct 16 16:59 fuse3-libs-3.10.2-9.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 8750 Oct 16 16:59 fuse-common-3.10.2-9.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 71022 Oct 16 16:59 fuse-overlayfs-1.14-1.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 61278 Oct 16 16:59 libnet-1.2-7.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 71992 Oct 16 16:59 libslirp-4.4.0-8.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 3846970 Oct 16 16:59 netavark-1.14.1-1.el9_6.x86_64.rpm -rw-r--r--. 1 bigred bigred 268131 Oct 16 16:59 'passt-0^20250217.ga1e48a0-10.el9_6.x86_64.rpm' -rw-r--r--. 1 bigred bigred 27658 Oct 16 16:59 'passt-selinux-0^20250217.ga1e48a0-10.el9_6.noarch.rpm' -rw-r--r--. 1 bigred bigred 17803252 Oct 16 16:59 podman-5.4.0-13.el9_6.x86_64.rpm -rw-r--r--. 1 bigred bigred 38224 Oct 16 16:59 protobuf-c-1.3.3-13.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 90389 Oct 16 16:59 shadow-utils-subid-4.9-12.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 50387 Oct 16 16:59 slirp4netns-1.3.2-1.el9.x86_64.rpm -rw-r--r--. 1 bigred bigred 42487 Oct 16 16:59 yajl-2.1.0-25.el9.x86_64.rpm下載並準備 skopeo pkg
sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/skopeo -y \ skopeo檢視已載入的套件
ls -l "$HOME"/k8s/pkg/skopeo執行結果：
total 9308 -rw-r--r--. 1 root root 9530108 Oct 17 10:19 skopeo-1.18.1-2.el9_6.x86_64.rpm
1.2. 準備 kubeadm 要安裝 K8s 所需的 images
列出安裝 kubeadm 要安裝 K8s 所需的 images
kubeadm config images list | tee ${HOME}/k8s/img/img-list.txt執行結果：
registry.k8s.io/kube-apiserver:v1.34.1 registry.k8s.io/kube-controller-manager:v1.34.1 registry.k8s.io/kube-scheduler:v1.34.1 registry.k8s.io/kube-proxy:v1.34.1 registry.k8s.io/coredns/coredns:v1.12.1 registry.k8s.io/pause:3.10.1 registry.k8s.io/etcd:3.6.4-0將 K8s images 備份到指定目錄中
for i in $(cat ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done podman save -m $(cat ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/all-K8s-Components-v1.34.1.tar拷貝 K8s 所有 components 的 images 到 Harbor 的 library project 中
for i in $(cat ${HOME}/k8s/img/img-list.txt); do skopeo copy docker://${i} docker://${i/registry.k8s.io/${HARBOR_FQDN}/library}; done
1.2.3. 準備 ingress-nginx v1.13.3 所需之 images 和安裝 yaml
找到 ingress-nginx
v1.13.3所需之 imagesINGRESS_NGINX_VERSION="v1.13.3" curl -s https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX_VERSION}/deploy/static/provider/cloud/deploy.yaml | grep image: | sort -u | tr -s " " | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39將 Ingress-nginx Controller image 備份到指定目錄
for i in $(grep ingress-nginx ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*};done podman save -m $(grep ingress-nginx ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/ingress-nginx-controller-v1.13.3.tar將安裝 yaml 下載至指定目錄
curl -so ${HOME}/k8s/addon/ingress/ingress-nginx/deploy.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX_VERSION}/deploy/static/provider/cloud/deploy.yaml
1.3. 準備 kube-vip v1.0.1 所需之 images
設定環境資訊
export VIP="172.20.6.110" export VIPSUBSET="16" export INTERFACE="ens18" KVVERSION="v1.0.1" alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"產生 kube-vip static pod yaml
kube-vip manifest pod \ --interface $INTERFACE \ --address $VIP \ --vipSubnet $VIPSUBSET \ --controlplane \ --arp \ --leaderElection | tee ${HOME}/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml上述參數沒有
--service是因為 control-plane 的 static pod 不需要管理 loadbalancer type service產生 kube-vip daemonset yaml
kube-vip manifest daemonset \ --interface $INTERFACE \ --vipSubnet $VIPSUBSET \ --services \ --arp \ --leaderElection \ --servicesElection | tee ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-ds.yaml下載 kube-vip Cloud Provider yaml 檔
curl -so ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-cloud-controller.yaml https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml下載 kube-vip daemonset 所需之 rbac.yaml
curl -so ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-rbac.yaml https://kube-vip.io/manifests/rbac.yaml檢視 Kubevip 執行所需之 images
find ${HOME}/k8s/addon/kube-vip -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
ghcr.io/kube-vip/kube-vip:v1.0.1 ghcr.io/kube-vip/kube-vip-cloud-provider:v0.0.12將 kube-vip images 備份到指定目錄
for i in $(tail -n 2 ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done podman save -m $(tail -n 2 ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/kube-vip-v1.0.1.tar
1.4. 準備 Metrics Server v0.8.0 所需之 images
定義 Metrics Server 版本變數
METRICS_SERVER_VERSION="v0.8.0"下載安裝 yaml 到指定路徑
curl -sLo ${HOME}/k8s/addon/metrics/metrics-server/components.yaml https://github.com/kubernetes-sigs/metrics-server/releases/download/${METRICS_SERVER_VERSION}/components.yaml檢視 Metrics Server 執行所需之 images
find ${HOME}/k8s/addon/metrics/metrics-server -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
registry.k8s.io/metrics-server/metrics-server:v0.8.0將 Metrics Server image 備份到指定目錄
for i in $(grep metrics-server ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done podman save -m $(grep metrics-server ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/metrics-server-${METRICS_SERVER_VERSION}.tar
1.5. 準備 Fluentd v1.19.0 所需之 images
定義 Fluentd 版本變數
FLUENTED_VERSION="v1.19.0-1.1"下載安裝 yaml 到指定路徑
curl -sLo ${HOME}/k8s/addon/logging/fluentd/fluentd-daemonset-syslog.yaml https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/refs/tags/${FLUENTED_VERSION}/fluentd-daemonset-syslog.yaml修改 yaml
sed -i "s|version: v1|version: ${FLUENTED_VERSION}|g" fluentd-daemonset-syslog.yaml sed -i "s|v1-debian-syslog|v1.19.0-debian-syslog-amd64-1.1|g" fluentd-daemonset-syslog.yaml sed -i 's|fluent/fluentd-kubernetes-daemonset|docker.io/fluent/fluentd-kubernetes-daemonset|g' fluentd-daemonset-syslog.yaml檢視 fluentd 執行所需之 images
find ${HOME}/k8s/addon/logging/fluentd -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
docker.io/fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-amd64-1.1將 fluentd image 備份到指定目錄
for i in $(grep fluentd ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done podman save -m $(grep fluentd ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/fluentd-kubernetes-daemonset-v1.19.0-debian-syslog-amd64-1.1.tar
1.6. 準備 cilium 1.18.2 所需之 images
定義 helm 和 cilium 版本變數
HELM_VERSION="v3.19.0" CILIUM_VERSION="1.18.2"下載 helm Binary
curl -sLo ${HOME}/k8s/addon/cni/cilium/helm-${HELM_VERSION}-linux-amd64.tar.gz https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz添加 Cilium Helm repository
helm repo add cilium https://helm.cilium.io/下載 cilium chart file
helm fetch cilium/cilium --version=${CILIUM_VERSION} -d ${HOME}/k8s/addon/cni/cilium/設定 helm values yaml
echo 'bpf: masquerade: true hubble: relay: enabled: true ui: enabled: true service: type: NodePort ipam: mode: kubernetes ipv4NativeRoutingCIDR: 10.244.0.0/16 k8sServiceHost: 172.20.6.10 k8sServicePort: 6443 kubeProxyReplacement: true' > ${HOME}/k8s/addon/cni/cilium/custom-values.yaml檢視 cilium 執行所需之 images
helm template ${HOME}/k8s/addon/cni/cilium/cilium-${CILIUM_VERSION}.tgz -f ${HOME}/k8s/addon/cni/cilium/custom-values.yaml | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g' | sort -u | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0 quay.io/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667 quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953 quay.io/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d quay.io/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805將 cilium images 備份到指定目錄
for i in $(grep "quay.io/cilium" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done podman save -m $(grep "quay.io/cilium" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/cilium-${CILIUM_VERSION}.tar下載 Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum下載 Hubble Client
HUBBLE_VERSION="$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)" HUBBLE_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum} sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum準備 cilium 測試網路效能 image
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin cilium connectivity perf --print-image-artifacts | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
quay.io/cilium/network-perf:1751527436-c2462ae@sha256:0c491ed7ca63e6c526593b3a2d478f856410a50fbbce7fe2b64283c3015d752f將 cilium 測試網路效能 images 備份到指定目錄
for i in $(grep "network-perf" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done podman save $(grep "network-perf" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/cilium-network-perf.tar
1.7. 準備 NFS subdir external provisioner v4.0.2 所需之 images
定義 NFS provisioner 版本變數
NFS_PROVISIONER_VERSION="v4.0.2"下載安裝 yaml 到指定路徑
git clone --no-checkout --depth=1 --filter=tree:0 https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner cd ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner git sparse-checkout set --no-cone /deploy git checkout mv ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner/deploy/ ${HOME}/k8s/addon/csi/nfs-provisioner/ sudo rm -r ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner sed -i "s|busybox:stable|docker.io/library/busybox:stable|g" ${HOME}/k8s/addon/csi/nfs-provisioner/deploy/test-pod.yaml建立
kustomization.yamlecho 'namespace: nfs-provisioner bases: - ./deploy resources: - namespace.yaml patchesStrategicMerge: - patch_nfs_details.yaml' > ${HOME}/k8s/addon/csi/nfs-provisioner/kustomization.yaml建立
namespace.yamlecho '# namespace.yaml apiVersion: v1 kind: Namespace metadata: name: nfs-provisioner' > ${HOME}/k8s/addon/csi/nfs-provisioner/namespace.yaml建立
patch_nfs_details.yamlecho 'apiVersion: apps/v1 kind: Deployment metadata: labels: app: nfs-client-provisioner name: nfs-client-provisioner spec: template: spec: containers: - name: nfs-client-provisioner env: - name: NFS_SERVER value: 172.20.6.9 - name: NFS_PATH value: /k8s volumes: - name: nfs-client-root nfs: server: 172.20.6.9 path: /k8s' > ${HOME}/k8s/addon/csi/nfs-provisioner/patch_nfs_details.yaml檢視 nfs-provisioner 執行所需之 images
find ${HOME}/k8s/addon/csi/nfs-provisioner -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt執行結果：
registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 docker.io/library/busybox:stable將 nfs-provisioner images 備份到指定目錄
for i in $(grep -E "nfs-subdir-external-provisioner|busybox" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done podman save -m $(grep -E "nfs-subdir-external-provisioner|busybox" ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/nfs-provisioner-${NFS_PROVISIONER_VERSION}.tar
1.8. 準備 Ingress NGINX v1.13.3 所需之 images 和 yaml 檔
定義 Ingress NGINX 版本變數
INGRESS_NGINX="v1.13.3"下載部署 Ingress NGINX YAML
curl -sLo ${HOME}/k8s/addon/ingress/ingress-nginx/deployment.yaml \ https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX}/deploy/static/provider/baremetal/deploy.yaml將 deployment 轉成 daemonset
nano daemonSet.yaml:::spoiler 檔案內容：
apiVersion: v1 kind: Namespace metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx name: ingress-nginx --- apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx namespace: ingress-nginx --- apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx namespace: ingress-nginx rules: - apiGroups: - "" resources: - namespaces verbs: - get - apiGroups: - "" resources: - configmaps - pods - secrets - endpoints verbs: - get - list - watch - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - coordination.k8s.io resourceNames: - ingress-nginx-leader resources: - leases verbs: - get - update - apiGroups: - coordination.k8s.io resources: - leases verbs: - create - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission namespace: ingress-nginx rules: - apiGroups: - "" resources: - secrets verbs: - get - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets - namespaces verbs: - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission rules: - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations verbs: - get - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: v1 data: null kind: ConfigMap metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-controller namespace: ingress-nginx --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http - appProtocol: https name: https port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: LoadBalancer --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-controller-admission namespace: ingress-nginx spec: ports: - appProtocol: https name: https-webhook port: 443 targetPort: webhook selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: ClusterIP --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-controller namespace: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx template: metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 spec: automountServiceAccountToken: true containers: - args: - /nginx-ingress-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key - --enable-metrics=false - --report-node-internal-ip-address=true - --watch-ingress-without-class=true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so image: registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: controller ports: - containerPort: 80 name: http protocol: TCP hostPort: 80 - containerPort: 443 name: https protocol: TCP hostPort: 443 - containerPort: 8443 name: webhook protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 100m memory: 90Mi securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - ALL readOnlyRootFilesystem: false runAsGroup: 82 runAsNonRoot: true runAsUser: 101 seccompProfile: type: RuntimeDefault volumeMounts: - mountPath: /usr/local/certificates/ name: webhook-cert readOnly: true dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission-create namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission-create spec: automountServiceAccountToken: true containers: - args: - create - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc - --namespace=$(POD_NAMESPACE) - --secret-name=ingress-nginx-admission env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39 imagePullPolicy: IfNotPresent name: create securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsGroup: 65532 runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure serviceAccountName: ingress-nginx-admission ttlSecondsAfterFinished: 0 --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission-patch namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission-patch spec: automountServiceAccountToken: true containers: - args: - patch - --webhook-name=ingress-nginx-admission - --namespace=$(POD_NAMESPACE) - --patch-mutating=false - --secret-name=ingress-nginx-admission - --patch-failure-policy=Fail env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39 imagePullPolicy: IfNotPresent name: patch securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsGroup: 65532 runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure serviceAccountName: ingress-nginx-admission ttlSecondsAfterFinished: 0 --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: nginx spec: controller: k8s.io/ingress-nginx --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.13.3 name: ingress-nginx-admission webhooks: - admissionReviewVersions: - v1 clientConfig: service: name: ingress-nginx-controller-admission namespace: ingress-nginx path: /networking/v1/ingresses port: 443 failurePolicy: Fail matchPolicy: Equivalent name: validate.nginx.ingress.kubernetes.io rules: - apiGroups: - networking.k8s.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - ingresses sideEffects: None:::
檢視 ingress nginx 執行所需之 images
find ${HOME}/k8s/addon/ingress/ingress-nginx \ -name "daemonSet.yaml" \ -type f \ -exec grep image: {} + | \ tr -s " " | \ cut -d " " -f 3 | \ sort -u | \ uniq | \ tee -a ${HOME}/k8s/img/img-list.txt執行結果：
registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39將 ingress nginx images 備份到指定目錄
for i in $(grep "ingress-nginx" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done podman save -m $(grep "ingress-nginx" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/ingress-nginx-controller-${INGRESS_NGINX}.tar
1.9. 準備 logging operator 6.1.0 所需之 images
定義 logging operator 版本變數
LOGGING_OPERATOR="6.1.0"下載 logging operator chart file
helm pull oci://ghcr.io/kube-logging/helm-charts/logging-operator \ --version ${LOGGING_OPERATOR} \ -d ${HOME}/k8s/addon/logging/logging-operator/檢視 logging operator 執行所需之 images
echo 'ghcr.io/kube-logging/logging-operator:6.1.0 ghcr.io/axoflow/axosyslog:4.18.1 ghcr.io/kube-logging/logging-operator/fluentd:6.1.0-full ghcr.io/kube-logging/logging-operator/syslog-ng-reloader:6.1.0 ghcr.io/kube-logging/logging-operator/config-reloader:6.1.0 ghcr.io/kube-logging/logging-operator/fluentd-drain-watch:6.1.0 ghcr.io/kube-logging/logging-operator/node-exporter:6.1.0 ghcr.io/axoflow/axosyslog-metrics-exporter:0.0.13' >> ${HOME}/k8s/img/img-list.txt將 logging operator images 備份到指定目錄
for i in $(grep -E "kube-logging|axoflow" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done podman save -m $(grep -E "kube-logging|axoflow" ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/kube-logging-${LOGGING_OPERATOR}.tar
1.10. 將整個工作目錄打包並匯入全離線的主機
壓縮出全離線安裝包
cd ~; tar -zcvf k8s-v1.34.1.tar.gz k8s/檢視全離線安裝包
ls -lh ${HOME}/k8s-v1.34.1.tar.gz執行結果：
-rw-r--r--. 1 bigred bigred 1.5G Oct 17 17:25 /home/bigred/k8s-v1.34.1.tar.gz將檔案上傳至所有全離線的 K8s 叢集主機，包含可上傳 image 至 Harbor 的主機
scp ${HOME}/k8s-v1.34.1.tar.gz user_name@vm_ipaddress:.
1.11. 將離線安裝所需之 Container Images 匯入 Harbor
:::info
先決條件：已安裝好 Harbor
:::
連線到可將 image tar 檔上傳至 Harbor 的主機
ssh user@ipaddress定義 Harbor FQDN
HARBOR_FQDN=harbor.example.com下載 Harbor CA 憑證
sudo curl -k \ -o /etc/pki/ca-trust/source/anchors/harbor-ca.crt \ https://${HARBOR_FQDN}/api/v2.0/systeminfo/getcert更新系統的 CA 信任清單，讓 HTTPS 連線時能信任此憑證
sudo update-ca-trust設定 podman 信任 Harbor 自簽 CA 憑證
sudo mkdir /etc/containers/certs.d/${HARBOR_FQDN} sudo curl -k \ -o /etc/containers/certs.d/${HARBOR_FQDN}/ca.crt \ https://${HARBOR_FQDN}/api/v2.0/systeminfo/getcert登入 harbor
podman login ${HARBOR_FQDN}執行結果：
Username: admin Password: Login Succeeded!上傳所有 container images 至 Harbor
cd ~/k8s/img; ./imgctl.sh正確執行結果：
Load All Images Successfully Currnt: 1/29 harbor.example.com/library/kube-apiserver:v1.34.1 push ok Currnt: 2/29 harbor.example.com/library/kube-controller-manager:v1.34.1 push ok Currnt: 3/29 harbor.example.com/library/kube-scheduler:v1.34.1 push ok Currnt: 4/29 harbor.example.com/library/kube-proxy:v1.34.1 push ok Currnt: 5/29 harbor.example.com/library/coredns/coredns:v1.12.1 push ok Currnt: 6/29 harbor.example.com/library/pause:3.10.1 push ok Currnt: 7/29 harbor.example.com/library/etcd:3.6.4-0 push ok Currnt: 8/29 harbor.example.com/library/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd push ok Currnt: 9/29 harbor.example.com/library/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39 push ok Currnt: 10/29 harbor.example.com/library/kube-vip/kube-vip:v1.0.1 push ok Currnt: 11/29 harbor.example.com/library/kube-vip/kube-vip-cloud-provider:v0.0.12 push ok Currnt: 12/29 harbor.example.com/library/metrics-server/metrics-server:v0.8.0 push ok Currnt: 13/29 harbor.example.com/library/fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-amd64-1.1 push ok Currnt: 14/29 harbor.example.com/library/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0 push ok Currnt: 15/29 harbor.example.com/library/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667 push ok Currnt: 16/29 harbor.example.com/library/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac push ok Currnt: 17/29 harbor.example.com/library/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953 push ok Currnt: 18/29 harbor.example.com/library/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d push ok Currnt: 19/29 harbor.example.com/library/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805 push ok Currnt: 20/29 harbor.example.com/library/sig-storage/nfs-subdir-external-provisioner:v4.0.2 push ok Currnt: 21/29 harbor.example.com/library/library/busybox:stable push ok Currnt: 22/29 harbor.example.com/library/kube-logging/logging-operator:6.1.0 push ok Currnt: 23/29 harbor.example.com/library/axoflow/axosyslog:4.18.1 push ok Currnt: 24/29 harbor.example.com/library/kube-logging/logging-operator/fluentd:6.1.0-full push ok Currnt: 25/29 harbor.example.com/library/kube-logging/logging-operator/syslog-ng-reloader:6.1.0 push ok Currnt: 26/29 harbor.example.com/library/kube-logging/logging-operator/config-reloader:6.1.0 push ok Currnt: 27/29 harbor.example.com/library/kube-logging/logging-operator/fluentd-drain-watch:6.1.0 push ok Currnt: 28/29 harbor.example.com/library/kube-logging/logging-operator/node-exporter:6.1.0 push ok Currnt: 29/29 harbor.example.com/library/axoflow/axosyslog-metrics-exporter:0.0.13 push ok
2. 建置 HA K8s 前置作業
2.1. 先決條件
- 已將 K8s 全離線安裝包(檔名:
k8s-v1.34.1.tar.gz)匯入 K8s 叢集中的每一台主機 - 已將 Harbor 自簽 root CA 憑證匯入匯入 K8s 叢集中的每一台 (Harbor 如果不是自簽 root CA 則不用做到此步驟)
2.2. 連線至 K8s 的第一台 Control-plane 節點
ssh 連線至第一台 Control-plane 節點
ssh user@ipaddress
2.3. 關閉 Firewalld Systemd Service
關閉 Firewalld Systemd Service
sudo systemctl disable --now firewalld.service
2.4. 調整 SELinux 為 permissive mode
調整 SELinux 為
permissivemode# Set SELinux in permissive mode (effectively disabling it) sudo setenforce 0 sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
2.5. 確認 SWAP 已關閉
確認 SWAP 已關閉
free -mh執行結果：
total used free shared buff/cache available Mem: 7.5Gi 530Mi 6.2Gi 8.0Mi 1.0Gi 7.0Gi Swap: 0B 0B 0B確認 swap 欄位的值都是
0B
2.6. 確認 Kernel 版本 >= 5.10
確認 Kernel 版本 >= 5.10
uname -r執行結果：
5.14.0-570.52.1.el9_6.x86_64
2.7. 添加必要 Kernel Modules
永久添加
br_netfilter和overlayKernel Modulescat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF立即啟用
br_netfilter和overlayKernel Modulessudo modprobe overlay sudo modprobe br_netfilter驗證已載入
br_netfilter和overlayKernel Modulessudo lsmod | grep -E "overlay|br_netfilter"執行結果：
br_netfilter 36864 0 bridge 417792 1 br_netfilter overlay 229376 0
2.8. 添加必要 Kernel Parameters
永久添加必要 Kernel Parameters
cat <<EOF | sudo tee /etc/sysctl.d/99-k8s-cri.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF立即載入 Kernel Parameters
sudo sysctl --system驗證
net.bridge.bridge-nf-call-iptables,net.bridge.bridge-nf-call-ip6tables, 和net.ipv4.ip_forward的值都設成1sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward執行結果：
net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1
2.9. 解壓縮 K8s 全離線安裝包
設定 K8s 全離線安裝包所在的目錄變數
PREFIX="$HOME"解壓縮 K8s 全離線安裝包
tar -zxvf "$PREFIX"/k8s-v1.34.1.tar.gz執行結果：
... tar: k8s/img/ingress-nginx-controller-v1.13.3.tar: time stamp 2025-10-20 14:18:13 is 166476.992186816 s in the future k8s/config/ tar: k8s/img: time stamp 2025-10-20 14:34:01 is 167424.992012128 s in the future
2.10. 安裝所有必要軟體套件
安裝所有必要軟體套件
sudo yum localinstall "$HOME"/k8s/pkg/kubernetes/*.rpm
2.11. 確認節點可以透過 DNS 解析到 harbor
定義 Harbor FQDN 變數
HARBOR_FQDN=harbor.example.com DOMAIN_NAME_SYSTEM_SERVER=172.20.6.11確認節點可以透過 DNS 解析到 harbor
dig @$DOMAIN_NAME_SYSTEM_SERVER $HARBOR_FQDN +short執行結果：
172.20.7.32
2.12. 啟動 CRI-O Service 並設為開機自動啟動
啟動 CRI-O Service 並設為開機自動啟動
sudo systemctl enable --now crio.service確認 CRI-O Service 狀態
sudo systemctl status crio.service執行結果：
● crio.service - Container Runtime Interface for OCI (CRI-O) Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; preset: disabled) Active: active (running) since Sat 2025-10-18 16:48:59 CST; 1min 18s ago Docs: https://github.com/cri-o/cri-o Main PID: 5734 (crio) Tasks: 7 Memory: 20.0M CPU: 126ms CGroup: /system.slice/crio.service └─5734 /usr/bin/crio Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314737723+08:00" le…cher" Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314774402+08:00" le…face" Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314808872+08:00" le…bled" Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314814672+08:00" le…ated" Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314820552+08:00" le… NRI" Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314823302+08:00" le…p..." Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314825973+08:00" le…s..." Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314833213+08:00" le…tate" Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314882773+08:00" le…bled" Oct 18 16:48:59 topgun-c1.kubeantony.com systemd[1]: Started Container Runtime Interface for OCI (…I-O). Hint: Some lines were ellipsized, use -l to show in full.
2.13. 設定 private registry 驗證
將 Harbor 自簽 ca 憑證上傳至主機家目錄
設定 Harbor 自簽 ca 憑證所在的目錄和 CA 憑證檔名為變數
CA_PREFIX="$HOME" CA_FILE_NAME="ca.pem"讓 RHEL 信任 Harbor 自簽 ca 憑證
sudo cp "$CA_PREFIX"/"$CA_FILE_NAME" /etc/pki/ca-trust/source/anchors/harbor-ca.crt sudo update-ca-trust編輯 CRI-O 設定檔，告訴它去哪裡讀取全域的認證資訊
sudo nano /etc/crio/crio.conf.d/10-crio.conf檔案內容：
新增
global_auth_file那一行：指定 CRI-O 讀取 Docker 格式的全域認證檔案路徑 還有pause_image這行，如果沒指定這行預設還是會到 K8s 的 image repo 去拉 還有conmon_cgroup和cgroup_manager也要指定在crio.runtime區塊底下[crio.image] signature_policy = "/etc/crio/policy.json" global_auth_file = "/etc/crio/registries.d/auth.json" pause_image="harbor.example.com/library/pause:3.10.1" [crio.runtime] default_runtime = "crun" conmon_cgroup = "pod" cgroup_manager = "systemd"建立存放認證檔案的目錄
sudo mkdir -p /etc/crio/registries.d/設定 Harbor Registry 的 FQDN (網域名稱) 變數，方便後續使用
HARBOR_FQDN=harbor.example.com使用 podman 登入 Harbor，並將產生的認證 token 儲存到 CRI-O 指定的 auth.json 檔案中
sudo podman login $HARBOR_FQDN --authfile /etc/crio/registries.d/auth.json驗證認證檔案是否已成功建立
sudo cat /etc/crio/registries.d/auth.json執行結果如下：
{ "auths": { "harbor.example.com": { "auth": "YWRtaW46SGFyYm9yMTIzNDU=" } } }編輯 system-wide 的 registry 設定檔，供 Podman, CRI-O, Skopeo 等工具共用
sudo nano /etc/containers/registries.conf檔案內容如下：
注意，Harbor FQDN 每個環境會不同。
[[registry]] location = "harbor.example.com" insecure = false重新啟動 CRI-O 服務，使其載入所有新的設定 (10-crio.conf 和 registries.conf)
sudo systemctl restart crio使用 crictl (CRI 的客戶端工具) 測試是否能成功從私有 Harbor 拉取映像檔
sudo crictl pull ${HARBOR_FQDN}/library/library/busybox:stable成功執行結果：
Image is up to date for harbor.example.com/library/library/busybox@sha256:f0d4f96113b0ebc3bc741b49cc158254ddd91bbba8dba20281b2dacd2da4a45f清理測試，移除剛剛拉取的映像檔
sudo crictl rmi ${HARBOR_FQDN}/library/library/busybox:stable
2.14. 設定 kubelet service node ip
指定節點 internal ip
INTERFACE=ens18 IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1) sudo sed -i \ -e '/^KUBELET_EXTRA_ARGS=$/s/$/\"--node-ip='"$IPV4_IP"'\"/' \ /etc/sysconfig/kubelet檢查是否正確修改
cat /etc/sysconfig/kubelet執行結果:
KUBELET_EXTRA_ARGS="--node-ip=172.20.6.111"
2.15. 在執行 kubeadm 之前啟用 kubelet 服務
啟動 kubelet 服務
sudo systemctl enable --now kubelet.service
2.16. 設定 NTP 校時
編輯 chrony 設定檔
sudo nano /etc/chrony.conf檔案內容如下：
#pool 2.rhel.pool.ntp.org iburst server 172.20.6.249 iburst重啟 chronyd 服務
sudo systemctl restart chronyd確認是否成功校時
sudo chronyc sources正確執行結果：
MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 172.20.6.249 3 6 377 20 -64us[ -237us] +/- 19ms
2.16. 在其他所有 K8s 節點重複執行所有步驟 2.2. ~ 2.16.
3. 透過 kubeadm 開始建立 HA K8s
3.1. 設定 kube-vip (單台 Control-plane 可跳過此步驟)
以可以執行 sudo 的 user 透過 ssh 連線至第一台 Control-plane node
ssh sudo_user@ipaddderss將 kube-vip static pod 拷貝到
/etc/kubernetes/manifests/sudo cp "$HOME"/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml /etc/kubernetes/manifests/kube-vip.yaml修改 kube-vip 的 Service Account，只需在第一台 Control-plane 需要修改
sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' \ /etc/kubernetes/manifests/kube-vip.yaml設定 harbor fqdn
HARBOR_FQDN=harbor.example.com修改 kube-vip 的 image repo
sudo sed -i "s|image: ghcr.io|image: ${HARBOR_FQDN}/library|g" \ /etc/kubernetes/manifests/kube-vip.yaml
$ sudo nano /etc/kubernetes/manifests/kube-vip.yaml
......
spec:
containers:
......
- name: vip_interface
value: ens18
- name: vip_subnet
value: "24"
......
- name: address
value: 10.10.7.313.2. 編輯 kubeadm 初始化設定檔
設定 kubeadm 初始化設定檔
nano "$HOME"/k8s/config/kubeadm-init.yaml注意每個環境需設定以下項目:
localAPIEndpoint.advertiseAddress: API Server 宣告(Advertise)給叢集其他成員的 IP 位址。nodeRegistration.criSocket: Kubelet 應使用的 CRI socket 路徑。nodeRegistration.name: 此節點在 Kubernetes 叢集中的唯一識別名稱 (通常使用主機名稱)。clusterName: K8s 叢集的名稱。controllerManager.extraVolumes: 掛載額外的磁碟區 (Volume) 到kube-controller-managerPod 中。imageRepository: 用於拉取 Kubernetes 核心組件 (如 API Server, etcd) images 的 Registry 位址。kubernetesVersion: 要部署的 Kubernetes 確切版本。networking.serviceSubnet: 分配給 Kubernetes Services 使用的虛擬 IP 位址範圍 (CIDR)。networking.podSubnet: 分配給叢集中所有 Pods 使用的 IP 位址範圍 (CIDR)。proxy.disabled: 停用節點上的kube-proxy組件 。controlPlaneEndpoint: HA 架構的 K8s LoadBalance IP，供所有節點和使用者存取 Kubernetes API Server 的 IP 或 FQDN。imageMaximumGCAge: 節點上未使用 images 在被垃圾回收 (GC) 前可保留的 最長 存留時間。imageMinimumGCAge: 節點上未使用 images 在被垃圾回收 (GC) 前必須保留的 最短 存留時間。shutdownGracePeriod: Kubelet 執行正常關機程序 (Node Shutdown) 的總寬限時間。shutdownGracePeriodCriticalPods: 在正常關機期間，Kubelet 等待「關鍵 Pods (Critical Pods)」優雅終止的專用寬限時間。systemReserved.memory: 為 OS 系統守護進程 (如 systemd, sshd) 保留的記憶體量，Kubelet 不會使用這部分。kubeReserved.memory: 為 Kubernetes 系統組件 (如 kubelet, 容器執行階段) 保留的記憶體量，Pod 無法使用這部分。
檔案內容如下 :
apiVersion: kubeadm.k8s.io/v1beta4 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 172.20.6.111 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/crio/crio.sock imagePullPolicy: IfNotPresent imagePullSerial: true name: topgun-c1.kubeantony.com taints: null timeouts: controlPlaneComponentHealthCheck: 4m0s discovery: 5m0s etcdAPICall: 2m0s kubeletHealthCheck: 4m0s kubernetesAPICall: 1m0s tlsBootstrap: 5m0s upgradeManifests: 5m0s --- apiServer: certSANs: - 127.0.0.1 apiVersion: kubeadm.k8s.io/v1beta4 caCertificateValidityPeriod: 87600h0m0s certificateValidityPeriod: 8760h0m0s certificatesDir: /etc/kubernetes/pki clusterName: topgun controllerManager: extraArgs: - name: bind-address value: "0.0.0.0" - name: secure-port value: "10257" extraVolumes: - name: tz-config hostPath: /etc/localtime mountPath: /etc/localtime readOnly: true dns: imageRepository: harbor.example.com/library/coredns encryptionAlgorithm: RSA-2048 etcd: local: dataDir: /var/lib/etcd extraArgs: - name: listen-metrics-urls value: http://0.0.0.0:2381 imageRepository: harbor.example.com/library kind: ClusterConfiguration kubernetesVersion: 1.34.1 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/16 podSubnet: 10.244.0.0/16 proxy: disabled: true scheduler: extraArgs: - name: bind-address value: "0.0.0.0" - name: secure-port value: "10259" controlPlaneEndpoint: 172.20.6.110:6443 --- apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s cgroupDriver: systemd clusterDNS: - 10.96.0.10 clusterDomain: cluster.local containerRuntimeEndpoint: "" cpuManagerReconcilePeriod: 0s crashLoopBackOff: {} evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMaximumGCAge: "168h" imageMinimumGCAge: "2m0s" kind: KubeletConfiguration logging: flushFrequency: 0 options: json: infoBufferSize: "0" text: infoBufferSize: "0" verbosity: 0 memorySwap: {} nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s shutdownGracePeriod: 30s shutdownGracePeriodCriticalPods: 10s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s maxPods: 110 systemReserved: memory: "1Gi" kubeReserved: memory: "2Gi"
3.3. 初始化第一台 Kubernetes control plane
Dry run set up the Kubernetes control plane，用以檢視是否設定檔有誤
sudo kubeadm init \ --dry-run \ --upload-certs \ --config="$HOME"/k8s/config/kubeadm-init.yaml正確執行結果 :
... Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.20.6.110:6443 --token 0n2iwr.ugdsni61mrtuqv4l \ --discovery-token-ca-cert-hash sha256:0e9dd65a4b3ba662a1a9b9f448cc1a632b808f863dd892d46fa4005b731da5bbSet up the Kubernetes control plane
sudo kubeadm init \ --upload-certs \ --config="$HOME"/k8s/config/kubeadm-init.yaml正確執行結果 :
W1018 22:31:59.203382 8373 initconfiguration.go:332] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta4", Kind:"ClusterConfiguration"}: strict decoding error: unknown field "apiServer.timeoutForControlPlane" [init] Using Kubernetes version: v1.34.1 [preflight] Running pre-flight checks [WARNING SystemVerification]: kernel release 5.14.0-570.52.1.el9_6.x86_64 is unsupported. Supported LTS versions from the 5.x series are 5.4, 5.10 and 5.15. Any 6.x version is also supported. For cgroups v2 support, the recommended version is 5.10 or newer [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local topgun-c1.kubeantony.com] and IPs [10.96.0.1 172.20.6.111 172.20.6.110 127.0.0.1] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost topgun-c1.kubeantony.com] and IPs [172.20.6.111 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost topgun-c1.kubeantony.com] and IPs [172.20.6.111 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "super-admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml" [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests" [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 501.238442ms [control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s [control-plane-check] Checking kube-apiserver at https://172.20.6.111:6443/livez [control-plane-check] Checking kube-controller-manager at https://0.0.0.0:10257/healthz [control-plane-check] Checking kube-scheduler at https://0.0.0.0:10259/livez [control-plane-check] kube-controller-manager is healthy after 1.860780055s [control-plane-check] kube-scheduler is healthy after 2.364803775s [control-plane-check] kube-apiserver is healthy after 4.00165287s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 681423a365cee2a5ff5b612831e42a97ff941c0c6ad0595298a6ff603b0fae5d [mark-control-plane] Marking the node topgun-c1.kubeantony.com as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node topgun-c1.kubeantony.com as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: z6kmo1.qp9d9u4xcb85xmjl [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of control-plane nodes running the following command on each as root: kubeadm join 172.20.6.110:6443 --token z6kmo1.qp9d9u4xcb85xmjl \ --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 \ --control-plane --certificate-key 681423a365cee2a5ff5b612831e42a97ff941c0c6ad0595298a6ff603b0fae5d Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.20.6.110:6443 --token z6kmo1.qp9d9u4xcb85xmjl \ --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50
3.4. 設定 kubeConfig
設定 kubeconfig
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config printf "\nsource <(kubectl completion bash)\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\n" >> ~/.bashrc source ~/.bashrc第一台 control plan 安裝好後再將
kube-vip權限調整回來$ sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml $ sudo systemctl daemon-reload $ sudo systemctl restart kubelet
3.5. 安裝 Cilium CNI
安裝 helm 指令
cd ${HOME}/k8s/addon/cni/cilium/ tar -zxvf helm-v3.19.0-linux-amd64.tar.gz sudo mv linux-amd64/helm /usr/local/bin/安裝 cilium 與 hubble client 指令
tar -zxvf cilium-linux-amd64.tar.gz -C /tmp tar -zxvf hubble-linux-amd64.tar.gz -C /tmp sudo mv /tmp/cilium /usr/bin/ sudo mv /tmp/hubble /usr/bin/編輯 cilium 設定檔
nano custom-values.yaml檔案內容：
operator: image: repository: harbor.example.com/library/cilium/operator tag: v1.18.2 useDigest: false image: repository: harbor.example.com/library/cilium/cilium tag: v1.18.2 useDigest: false envoy: image: repository: harbor.example.com/library/cilium/cilium-envoy tag: v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324 useDigest: false bpf: masquerade: true hubble: relay: enabled: true image: repository: harbor.example.com/library/cilium/hubble-relay tag: v1.18.2 useDigest: false ui: enabled: true service: type: NodePort frontend: image: repository: harbor.example.com/library/cilium/hubble-ui tag: v0.13.3 useDigest: false backend: image: repository: harbor.example.com/library/cilium/hubble-ui-backend tag: v0.13.3 useDigest: false ipam: mode: kubernetes ipv4NativeRoutingCIDR: 10.244.0.0/16 k8sServiceHost: 172.20.6.110 k8sServicePort: 6443 kubeProxyReplacement: true autoDirectNodeRoutes: true routingMode: native devices: ens18安裝 cilium
helm -n kube-system install cilium ./cilium-1.18.2.tgz -f custom-values.yaml確認 cilium 已安裝成功
cilium status執行結果：
/¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: OK \__/¯¯\__/ Hubble Relay: OK \__/ ClusterMesh: disabled DaemonSet cilium Desired: 6, Ready: 6/6, Available: 6/6 DaemonSet cilium-envoy Desired: 6, Ready: 6/6, Available: 6/6 Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2 Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 6 cilium-envoy Running: 6 cilium-operator Running: 2 clustermesh-apiserver hubble-relay Running: 1 hubble-ui Running: 1 Cluster Pods: 9/9 managed by Cilium Helm chart version: Image versions cilium harbor.example.com/library/cilium/cilium:v1.18.2: 6 cilium-envoy harbor.example.com/library/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324: 6 cilium-operator harbor.example.com/library/cilium/operator-generic:v1.18.2: 2 hubble-relay harbor.example.com/library/cilium/hubble-relay:v1.18.2: 1 hubble-ui harbor.example.com/library/cilium/hubble-ui-backend:v0.13.3: 1 hubble-ui harbor.example.com/library/cilium/hubble-ui:v0.13.3: 1確認 cilium 有啟用 eBPF host routing
cilium_pod=$(kubectl -n kube-system get pods --field-selector status.phase=Running -l k8s-app=cilium -o name | tail -n 1 | cut -d "/" -f 2) kubectl -n kube-system exec -it ${cilium_pod} -- cilium-dbg status執行結果：
KubeProxyReplacement: True [ens18 172.20.6.115 172.20.6.117 fe80::be24:11ff:fe2e:cc62 (Direct Routing)] Host firewall: Disabled SRv6: Disabled CNI Chaining: none CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist Cilium: Ok 1.18.2 (v1.18.2-5bd307a8) NodeMonitor: Listening for events on 2 CPUs with 64x4096 of shared memory Cilium health daemon: Ok IPAM: IPv4: 7/254 allocated from 10.244.5.0/24, IPv4 BIG TCP: Disabled IPv6 BIG TCP: Disabled BandwidthManager: Disabled Routing: Network: Native Host: BPF Attach Mode: TCX Device Mode: veth Masquerading: BPF [ens18] 10.244.0.0/16 [IPv4: Enabled, IPv6: Disabled] Controller Status: 47/47 healthy Proxy Status: OK, ip 10.244.5.177, 0 redirects active on ports 10000-20000, Envoy: external Global Identity Range: min 256, max 65535 Hubble: Ok Current/Max Flows: 2491/4095 (60.83%), Flows/s: 3.51 Metrics: Disabled Encryption: Disabled Cluster health: 6/6 reachable (2025-10-23T08:51:12Z) (Probe interval: 2m33.896961447s) Name IP Node Endpoints Modules Health: Stopped(12) Degraded(0) OK(78)
3.6. 新增其他 Control-plane 節點進 K8s 叢集 (單台 Control-plane 可跳過此步驟)
以可以執行 sudo 的 user 透過 ssh 連線至其他 Control-plane node
ssh sudo_user@ipaddderss將 kube-vip static pod 拷貝到
/etc/kubernetes/manifests/sudo cp "$HOME"/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml /etc/kubernetes/manifests/kube-vip.yaml設定 harbor fqdn
HARBOR_FQDN=harbor.example.com修改 kube-vip 的 image repo
sudo sed -i "s|image: ghcr.io|image: ${HARBOR_FQDN}/library|g" \ /etc/kubernetes/manifests/kube-vip.yaml獲得 Control-plane node Kubeadm Join Command 請在第一台 Control-plane node 執行以下命令
kubeadm_init_config_file="${HOME}/k8s/config/kubeadm-init.yaml" certificate_key=$(sudo kubeadm init phase upload-certs --upload-certs --config ${kubeadm_init_config_file} | tail -n 1) join_cmd=$(sudo kubeadm token create --print-join-command) join_cp_cmd="sudo $join_cmd --control-plane --certificate-key $certificate_key" echo $join_cp_cmd執行結果：
sudo kubeadm join 172.20.6.110:6443 --token rl5cdl.8552g1j3ezrvxk21 --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 --control-plane --certificate-key d52af404ed5765485f28e0c5e7e886ae244a8861a78dc3b1ce1a0487194e8a1a將 Control-plane 節點加入 K8s 叢集 請在要加入 K8s 的 Control-plane node 執行以下命令
sudo kubeadm join 172.20.6.110:6443 --token rl5cdl.8552g1j3ezrvxk21 --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 --control-plane --certificate-key d52af404ed5765485f28e0c5e7e886ae244a8861a78dc3b1ce1a0487194e8a1ajoin node 使用的 token 會在 24 小時後自動過期
重複以上 1. ~ 6. 步驟將其他 Control-plane node 加到 K8s Cluster
3.7. 新增 Worker 節點進 K8s 叢集
在第一台 Controlplane 將 join worker node 指令 print 出來
sudo kubeadm token create --print-join-command執行結果：
kubeadm join 172.20.6.110:6443 --token inyh77.qp4dqe6wto9uhk8m --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50以可以執行 sudo 的 user 透過 ssh 連線至 worker node
ssh sudo_user@ipaddderss將 worker node 加入 K8s 叢集
sudo kubeadm join 172.20.6.110:6443 \ --token inyh77.qp4dqe6wto9uhk8m \ --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50ssh 連線到其他 worker node，重複執行步驟 3，將其他 worker node 也加入 K8s 叢集
回到第一台 Control-plane 執行以下命令，將所有 worker nodes 都貼 label
kubectl label node topgun-w1.kubeantony.com node-role.kubernetes.io/worker= kubectl label node topgun-w2.kubeantony.com node-role.kubernetes.io/worker= kubectl label node topgun-w3.kubeantony.com node-role.kubernetes.io/worker=
3.8. 檢視 K8s 狀態是否正常
檢視 K8s node 狀態
kubectl get nodes執行結果：
NAME STATUS ROLES AGE VERSION topgun-c1.kubeantony.com Ready control-plane 3d15h v1.34.1 topgun-c2.kubeantony.com Ready control-plane 26h v1.34.1 topgun-c3.kubeantony.com Ready control-plane 3h24m v1.34.1 topgun-w1.kubeantony.com Ready worker 3d6h v1.34.1 topgun-w2.kubeantony.com Ready worker 17m v1.34.1 topgun-w3.kubeantony.com Ready worker 16m v1.34.1檢視 kube-system namespace 底下所有 pods 狀態
kubectl -n kube-system get pods執行結果：
NAME READY STATUS RESTARTS AGE cilium-b74sh 1/1 Running 0 26h cilium-btg2h 1/1 Running 0 26h cilium-envoy-24skx 1/1 Running 0 26h cilium-envoy-llwv7 1/1 Running 0 17m cilium-envoy-nb5l2 1/1 Running 0 3h24m cilium-envoy-nvkz9 1/1 Running 0 26h cilium-envoy-vz6kj 1/1 Running 0 16m cilium-envoy-wm4xw 1/1 Running 0 26h cilium-gp9v9 1/1 Running 0 17m cilium-operator-794b864946-8j8j5 1/1 Running 1 (10h ago) 26h cilium-operator-794b864946-9whct 1/1 Running 0 26h cilium-vqp5v 1/1 Running 0 26h cilium-xgcch 1/1 Running 0 16m cilium-zj2xj 1/1 Running 0 3h24m coredns-f74588c75-86h95 1/1 Running 0 26h coredns-f74588c75-9992t 1/1 Running 0 26h etcd-topgun-c1.kubeantony.com 1/1 Running 0 3d15h etcd-topgun-c2.kubeantony.com 1/1 Running 0 26h etcd-topgun-c3.kubeantony.com 1/1 Running 0 3h24m hubble-relay-7d6865bf7b-7wvhl 1/1 Running 0 26h hubble-ui-c6998f8ff-6m4cn 2/2 Running 0 26h kube-apiserver-topgun-c1.kubeantony.com 1/1 Running 0 3d15h kube-apiserver-topgun-c2.kubeantony.com 1/1 Running 0 26h kube-apiserver-topgun-c3.kubeantony.com 1/1 Running 0 3h24m kube-controller-manager-topgun-c1.kubeantony.com 1/1 Running 1 (10h ago) 3d15h kube-controller-manager-topgun-c2.kubeantony.com 1/1 Running 0 26h kube-controller-manager-topgun-c3.kubeantony.com 1/1 Running 0 3h24m kube-scheduler-topgun-c1.kubeantony.com 1/1 Running 1 (10h ago) 3d15h kube-scheduler-topgun-c2.kubeantony.com 1/1 Running 0 26h kube-scheduler-topgun-c3.kubeantony.com 1/1 Running 0 3h24m kube-vip-topgun-c1.kubeantony.com 1/1 Running 1 (10h ago) 3d15h kube-vip-topgun-c3.kubeantony.com 1/1 Running 0 3h24m
3.9. 檢測 K8s 網路延遲和吞吐量
主要測試以下項目
- 吞吐量 - TCP Throughput (TCP_STREAM)：有助於了解在特定配置下可達到的最大吞吐量。
- 此測試代表大量資料傳輸的應用程式，例如串流服務或執行資料上傳/下載的服務。
- 請求/回應速率 - Request/Response Rate (TCP_RR)：主要測量處理單一網路封包來回轉發的延遲與效率。這項基準測試將在線路上產生每秒可能的最大封包數（packets per second），並著重測試處理單一網路封包的成本（資源消耗）。
- 此測試代表了那些維持持續性連線（persistent connections）、並與其他服務交換請求/回應類型互動的服務。這對於使用 REST 或 gRPC API 的服務來說很常見。
- 連線速率 - Connection Rate (TCP_CRR)：它類似於 Request/Response Rate 測試，但會為每一次來回建立一個新的 TCP 連線。
- 此測試代表暴露於公網的服務，它會接收來自許多客戶端的連線。L4 代理或對外部端點開啟許多連線的服務，都是很好的例子。
- 吞吐量 - TCP Throughput (TCP_STREAM)：有助於了解在特定配置下可達到的最大吞吐量。
透過 cilium 指令檢測 K8s 網路頻寬
cilium connectivity perf \ --crr \ --host-to-pod \ --pod-to-host \ --performance-image quay.io/cilium/network-perf:1751527436-c2462ae \ --node-selector-client kubernetes.io/hostname=topgun-w1.kubeantony.com \ --node-selector-server kubernetes.io/hostname=topgun-w2.kubeantony.com執行結果如下：
🏃[cilium-test-1] Running 1 tests ... [=] [cilium-test-1] Test [network-perf] [1/1] ................................ 🔥 Network Performance Test Summary [cilium-test-1]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 📋 pod-to-pod | same-node | TCP_CRR | 10s | 50µs | 84.48µs | 11.99ms | 77µs | 96µs | 144µs | 11787.95 📋 pod-to-pod | same-node | TCP_RR | 10s | 11µs | 30.74µs | 4.659ms | 31µs | 35µs | 46µs | 32197.86 📋 pod-to-host | same-node | TCP_CRR | 10s | 61µs | 98.57µs | 21.094ms | 90µs | 109µs | 155µs | 10104.54 📋 pod-to-host | same-node | TCP_RR | 10s | 12µs | 31.86µs | 1.091ms | 32µs | 37µs | 49µs | 31071.90 📋 host-to-pod | same-node | TCP_CRR | 10s | 60µs | 100.33µs | 14.059ms | 90µs | 109µs | 160µs | 9930.36 📋 host-to-pod | same-node | TCP_RR | 10s | 12µs | 31.94µs | 892µs | 32µs | 36µs | 48µs | 30996.63 📋 host-to-host | same-node | TCP_CRR | 10s | 52µs | 76.41µs | 1.924ms | 73µs | 89µs | 127µs | 13016.02 📋 host-to-host | same-node | TCP_RR | 10s | 12µs | 31.89µs | 1.281ms | 32µs | 36µs | 48µs | 31043.05 📋 pod-to-pod | other-node | TCP_CRR | 10s | 242µs | 330.04µs | 13.257ms | 301µs | 342µs | 790µs | 3027.08 📋 pod-to-pod | other-node | TCP_RR | 10s | 87µs | 112.64µs | 4.085ms | 112µs | 124µs | 146µs | 8867.04 📋 pod-to-host | other-node | TCP_CRR | 10s | 243µs | 321.36µs | 15.197ms | 302µs | 340µs | 483µs | 3108.95 📋 pod-to-host | other-node | TCP_RR | 10s | 89µs | 113.52µs | 3.172ms | 113µs | 125µs | 147µs | 8797.96 📋 host-to-pod | other-node | TCP_CRR | 10s | 230µs | 311.48µs | 13.035ms | 287µs | 334µs | 635µs | 3207.10 📋 host-to-pod | other-node | TCP_RR | 10s | 88µs | 115.57µs | 4.126ms | 114µs | 128µs | 153µs | 8641.95 📋 host-to-host | other-node | TCP_CRR | 10s | 229µs | 303.13µs | 21.099ms | 285µs | 322µs | 417µs | 2739.19 📋 host-to-host | other-node | TCP_RR | 10s | 89µs | 114.78µs | 4.428ms | 114µs | 126µs | 148µs | 8702.07 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------- 📋 Scenario | Node | Test | Duration | Throughput Mb/s ---------------------------------------------------------------------------------------- 📋 pod-to-pod | same-node | TCP_STREAM | 10s | 18952.05 📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 45634.84 📋 pod-to-host | same-node | TCP_STREAM | 10s | 16338.34 📋 pod-to-host | same-node | TCP_STREAM_MULTI | 10s | 41785.81 📋 host-to-pod | same-node | TCP_STREAM | 10s | 15643.64 📋 host-to-pod | same-node | TCP_STREAM_MULTI | 10s | 40812.09 📋 host-to-host | same-node | TCP_STREAM | 10s | 36117.05 📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 82274.99 📋 pod-to-pod | other-node | TCP_STREAM | 10s | 9407.21 📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 9417.40 📋 pod-to-host | other-node | TCP_STREAM | 10s | 9411.27 📋 pod-to-host | other-node | TCP_STREAM_MULTI | 10s | 9411.89 📋 host-to-pod | other-node | TCP_STREAM | 10s | 9409.00 📋 host-to-pod | other-node | TCP_STREAM_MULTI | 10s | 9398.15 📋 host-to-host | other-node | TCP_STREAM | 10s | 9408.12 📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 9416.04 ---------------------------------------------------------------------------------------- ✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.未開 ebpf host routing 的數值：
... 🏃[cilium-test-1] Running 1 tests ... [=] [cilium-test-1] Test [network-perf] [1/1] ................................ 🔥 Network Performance Test Summary [cilium-test-1]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 📋 pod-to-pod | same-node | TCP_CRR | 10s | 50µs | 82.85µs | 10.656ms | 78µs | 94µs | 124µs | 12017.63 📋 pod-to-pod | same-node | TCP_RR | 10s | 10µs | 30.71µs | 828µs | 30µs | 35µs | 46µs | 32231.92 📋 pod-to-host | same-node | TCP_CRR | 10s | 59µs | 94.56µs | 12.326ms | 91µs | 108µs | 137µs | 10532.25 📋 pod-to-host | same-node | TCP_RR | 10s | 12µs | 31.94µs | 2.572ms | 32µs | 37µs | 48µs | 30996.44 📋 host-to-pod | same-node | TCP_CRR | 10s | 56µs | 93.13µs | 15.588ms | 90µs | 107µs | 136µs | 10694.60 📋 host-to-pod | same-node | TCP_RR | 10s | 12µs | 31.27µs | 1.842ms | 31µs | 36µs | 48µs | 31651.76 📋 host-to-host | same-node | TCP_CRR | 10s | 53µs | 75µs | 1.479ms | 72µs | 88µs | 123µs | 13261.02 📋 host-to-host | same-node | TCP_RR | 10s | 12µs | 31.87µs | 583µs | 32µs | 36µs | 47µs | 31059.78 📋 pod-to-pod | other-node | TCP_CRR | 10s | 278µs | 368.98µs | 9.809ms | 339µs | 382µs | 965µs | 2707.98 📋 pod-to-pod | other-node | TCP_RR | 10s | 95µs | 123.71µs | 4.119ms | 122µs | 137µs | 163µs | 8073.77 📋 pod-to-host | other-node | TCP_CRR | 10s | 248µs | 371.53µs | 1.052362s | 298µs | 335µs | 500µs | 2006.19 📋 pod-to-host | other-node | TCP_RR | 10s | 90µs | 114.49µs | 2.982ms | 114µs | 126µs | 147µs | 8723.87 📋 host-to-pod | other-node | TCP_CRR | 10s | 279µs | 366.76µs | 12.3ms | 338µs | 375µs | 850µs | 2724.34 📋 host-to-pod | other-node | TCP_RR | 10s | 105µs | 135.79µs | 15.723ms | 133µs | 148µs | 176µs | 7355.48 📋 host-to-host | other-node | TCP_CRR | 10s | 229µs | 301.42µs | 15.967ms | 284µs | 314µs | 415µs | 2739.19 📋 host-to-host | other-node | TCP_RR | 10s | 88µs | 114.62µs | 4.249ms | 114µs | 127µs | 148µs | 8713.25 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------- 📋 Scenario | Node | Test | Duration | Throughput Mb/s ---------------------------------------------------------------------------------------- 📋 pod-to-pod | same-node | TCP_STREAM | 10s | 19819.33 📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 44767.93 📋 pod-to-host | same-node | TCP_STREAM | 10s | 17519.15 📋 pod-to-host | same-node | TCP_STREAM_MULTI | 10s | 41288.42 📋 host-to-pod | same-node | TCP_STREAM | 10s | 15667.62 📋 host-to-pod | same-node | TCP_STREAM_MULTI | 10s | 40686.99 📋 host-to-host | same-node | TCP_STREAM | 10s | 36663.82 📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 83454.15 📋 pod-to-pod | other-node | TCP_STREAM | 10s | 5515.69 📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 5462.09 📋 pod-to-host | other-node | TCP_STREAM | 10s | 9393.23 📋 pod-to-host | other-node | TCP_STREAM_MULTI | 10s | 9399.02 📋 host-to-pod | other-node | TCP_STREAM | 10s | 5456.02 📋 host-to-pod | other-node | TCP_STREAM_MULTI | 10s | 5423.52 📋 host-to-host | other-node | TCP_STREAM | 10s | 9410.65 📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 9416.39 ---------------------------------------------------------------------------------------- ✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.
- 通用欄位
Scenario(情境): 描述流量的來源和目的地。pod-to-pod: 從一個 Pod 到另一個 Pod。pod-to-host: 從一個 Pod 到其運行的節點 (Host)。host-to-pod: 從一個節點 (Host) 到一個 Pod。host-to-host: 從一個節點 (Host) 到另一個節點 (Host)。
Node(節點): 描述來源和目的地的相對位置。same-node: 來源和目的地在同一個 Kubernetes 節點上。other-node: 來源和目的地在不同的 Kubernetes 節點上。
- 表 1
這張表格專注於延遲（完成一個操作需要多長時間）和事務處理速率（每秒可以完成多少操作），主要用於
TCP_CRR和TCP_RR測試。時間單位是微秒 (µs) 或毫秒 (ms)。Min(最小延遲): 在測試期間觀察到的最快的單次事務（請求/回應或連接/關閉）延遲。Mean(平均延遲): 所有事務延遲的算術平均值。Max(最大延遲): 在測試期間觀察到的最慢的單次事務延遲。高Max值可能表示網路抖動或偶發性的效能瓶頸。P50(50百分位延遲): 也稱為中位數。50% 的事務延遲低於此值。與Mean相比，P50受極端異常值的影響較小。P90(90百分位延遲): 90% 的事務延遲低於此值。這是一個常見的服務水平指標 (SLI)，用於衡量絕大多數用戶的體驗。P99(99百分位延遲): 99% 的事務延遲低於此值。這有助於識別「長尾延遲」(long-tail latency)，即影響少數請求的嚴重效能問題。Transaction rate OP/s(每秒事務處理速率): 每秒操作數 (Operations Per Second)。- 這個值越高越好，表示處理事務的效率越高。
- 表 2
這張表格專注於頻寬（每秒可以傳輸多少數據），主要用於
TCP_STREAM和TCP_STREAM_MULTI測試。Throughput Mb/s(吞吐量 Mb/s):- 每秒百萬位元 (Megabits per second)。這個值顯示了在測試期間，來源和目的地之間數據傳輸的平均速率。值越高，表示網路頻寬越大。
TCP_STREAM:- 使用單個 TCP 連接測量最大數據傳輸吞吐量（頻寬）。
TCP_STREAM_MULTI:- 使用 32 個並行 TCP 連接測量最大數據傳輸吞吐量。這通常能更好地佔滿網路鏈路，顯示系統的總頻寬上限。
4. 安裝 K8s Addon
4.1. 安裝 Metrics server
切換工作目錄
cd ${HOME}/k8s/addon/metrics/metrics-server編輯 Kustomization File
nano kustomization.yaml檔案內容：
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ./components.yaml images: - name: registry.k8s.io/metrics-server/metrics-server newName: harbor.example.com/library/metrics-server/metrics-server newTag: v0.8.0 patches: - target: kind: Deployment name: metrics-server patch: |- - op: add path: /spec/template/spec/containers/0/args/- value: --kubelet-insecure-tls驗證 yaml 設定檔是否有誤
kubectl apply -k . --dry-run=server正確執行結果：
serviceaccount/metrics-server created (server dry run) clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created (server dry run) clusterrole.rbac.authorization.k8s.io/system:metrics-server created (server dry run) rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created (server dry run) clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created (server dry run) clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created (server dry run) service/metrics-server created (server dry run) deployment.apps/metrics-server created (server dry run) apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created (server dry run)部署 Metrics Server
kubectl apply -k .檢視 Metrics Server pods 運作狀態
kubectl -n kube-system get pods -l k8s-app=metrics-server執行結果：
NAME READY STATUS RESTARTS AGE metrics-server-69779649f-frvqz 1/1 Running 0 43s檢視 K8s 所有 nodes 的 cpu 記憶體使用狀況
kubectl top nodes執行結果：
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) topgun-c1.kubeantony.com 41m 2% 1705Mi 37% topgun-c2.kubeantony.com 34m 1% 1175Mi 26% topgun-c3.kubeantony.com 28m 1% 1057Mi 23% topgun-w1.kubeantony.com 12m 0% 1066Mi 23% topgun-w2.kubeantony.com 10m 0% 947Mi 21% topgun-w3.kubeantony.com 9m 0% 778Mi 17%
4.2. 安裝 kube-vip On-Premises Cloud Controller
切換工作目錄
cd ${HOME}/k8s/addon/kube-vip/k8s-lb-service將可分配的 IP Range 寫入 configmap
kubectl -n kube-system create configmap kubevip \ --from-literal range-global=172.20.6.117-172.20.6.120 \ --dry-run=client -o yaml > kube-vip-cm.yaml編輯 Kustomization File
nano kustomization.yaml檔案內容：
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ./kube-vip-rbac.yaml - ./kube-vip-ds.yaml - ./kube-vip-cloud-controller.yaml - ./kube-vip-cm.yaml images: - name: ghcr.io/kube-vip/kube-vip newName: harbor.example.com/library/kube-vip/kube-vip newTag: v1.0.1 - name: ghcr.io/kube-vip/kube-vip-cloud-provider newName: harbor.example.com/library/kube-vip/kube-vip-cloud-provider newTag: v0.0.12 patches: - target: kind: DaemonSet name: kube-vip-ds namespace: kube-system patch: |- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-vip-ds spec: template: spec: serviceAccountName: kube-vip nodeSelector: node-role.kubernetes.io/worker: "" containers: - name: kube-vip env: - name: vip_interface value: "ens18" - name: vip_subnet value: "16" volumeMounts: - mountPath: /etc/kubernetes/admin.conf $patch: delete volumes: - name: kubeconfig $patch: delete - target: kind: ConfigMap name: kubevip namespace: kube-system patch: |- - op: add path: /data/range-global value: "172.20.6.117-172.20.6.120"驗證 yaml 設定檔是否有誤
kubectl apply -k . --dry-run=server正確執行結果：
serviceaccount/kube-vip created (server dry run) serviceaccount/kube-vip-cloud-controller created (server dry run) clusterrole.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-role created (server dry run) clusterrole.rbac.authorization.k8s.io/system:kube-vip-role created (server dry run) clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-binding created (server dry run) clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-binding created (server dry run) configmap/kubevip created (server dry run) Warning: spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[1].preference.matchExpressions[0].key: node-role.kubernetes.io/master is use "node-role.kubernetes.io/control-plane" instead deployment.apps/kube-vip-cloud-provider created (server dry run) daemonset.apps/kube-vip-ds created (server dry run)部署 kube-vip On-Premises Cloud Controller
kubectl apply -k .檢視 kube-vip On-Premises Cloud Controller pods 運作狀態
kubectl -n kube-system get pods -l component=kube-vip-cloud-provider執行結果：
NAME READY STATUS RESTARTS AGE kube-vip-cloud-provider-d9b9446b9-6ll6t 1/1 Running 0 45s檢視 kube-vip ds pods 運作狀態，並確認 pods 只跑在 worker nodes 上
kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip-ds -o wide執行結果：
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-vip-ds-kglmc 1/1 Running 0 59s 172.20.6.114 topgun-w1.kubeantony.com <none> <none> kube-vip-ds-krst5 1/1 Running 0 59s 172.20.6.115 topgun-w2.kubeantony.com <none> <none> kube-vip-ds-r8cmc 1/1 Running 0 59s 172.20.6.116 topgun-w3.kubeantony.com <none> <none>
4.3. 安裝 Ingress NGINX Controller
切換工作目錄
cd ${HOME}/k8s/addon/ingress/ingress-nginx編輯 Kustomization File
nano kustomization.yaml檔案內容如下：
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: #- ./deployment.yaml - ./daemonSet.yaml images: - name: registry.k8s.io/ingress-nginx/controller newName: harbor.example.com/library/ingress-nginx/controller newTag: v1.13.3 - name: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3 newName: harbor.example.com/library/ingress-nginx/kube-webhook-certgen newTag: v1.6.3 patches: - target: kind: DaemonSet name: ingress-nginx-controller patch: |- apiVersion: apps/v1 kind: DaemonSet metadata: name: ingress-nginx-controller spec: template: spec: nodeSelector: node-role.kubernetes.io/worker: ""驗證 yaml 設定檔是否有誤
kubectl apply -k . --dry-run=server正確執行結果：
namespace/ingress-nginx created (server dry run) clusterrole.rbac.authorization.k8s.io/ingress-nginx created (server dry run) clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created (server dry run) clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created (server dry run) clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created (server dry run) ingressclass.networking.k8s.io/nginx created (server dry run) validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created (server dry run) Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found部署 Ingress NGINX Controller
kubectl apply -k .檢視 Ingress NGINX Controller pods 運作狀態，並確認是否只在 woker nodes 上執行
kubectl -n ingress-nginx get pods -o wide執行結果：
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-controller-4tgxs 1/1 Running 0 90s 10.244.5.109 topgun-w2.kubeantony.com <none> <none> ingress-nginx-controller-n9t75 1/1 Running 0 67s 10.244.6.115 topgun-w3.kubeantony.com <none> <none> ingress-nginx-controller-thkd4 1/1 Running 0 44s 10.244.1.190 topgun-w1.kubeantony.com <none> <none>建立測試目錄
mkdir test編輯測試用 deployment
nano test/test-deployment檔案內容：
apiVersion: apps/v1 kind: Deployment metadata: labels: app: httpd name: httpd spec: replicas: 3 selector: matchLabels: app: httpd strategy: {} template: metadata: labels: app: httpd spec: containers: - image: harbor.example.com/library/library/busybox:stable name: busybox command: ["/bin/sh", "-c"] args: - echo haha > /tmp/index.html; httpd -p 888 -h /tmp -f編輯 K8s service yaml
nano test/test-svc.yaml檔案內容：
apiVersion: v1 kind: Service metadata: labels: app: httpd name: test-svc spec: ports: - port: 80 protocol: TCP targetPort: 888 selector: app: httpd編輯 K8s ingress yaml
nano test/test-ing.yaml檔案內容：
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress spec: ingressClassName: nginx rules: - host: test.k8s.io http: paths: - path: / pathType: Prefix backend: service: name: test-svc port: number: 80部署測試 deployment, service 和 ingress
kubectl apply -f test/確認 pods 狀態
kubectl get pods執行結果：
NAME READY STATUS RESTARTS AGE httpd-77c9cfcf7d-2tqks 1/1 Running 0 70m httpd-77c9cfcf7d-4g9fl 1/1 Running 0 70m httpd-77c9cfcf7d-6w2l2 1/1 Running 0 70m確認 ingress 有拿到節點的 ip
kubectl get ing執行結果：
NAME CLASS HOSTS ADDRESS PORTS AGE minimal-ingress nginx test.k8s.io 172.20.6.114,172.20.6.115,172.20.6.116 80 66m驗證可以存取網站服務
for i in {114..116}; do time curl -H "Host: test.k8s.io" http://172.20.6.${i};echo ;done正確執行結果：
haha real 0m0.006s user 0m0.003s sys 0m0.002s haha real 0m0.005s user 0m0.002s sys 0m0.002s haha real 0m0.005s user 0m0.002s sys 0m0.001s