Antony 筆記：技術原理與實務經驗 – Kubernetes

Install a kubeadm Kubernetes cluster with Cilium, disabling kube-proxy

Tue, 30 Jun 2026 09:58:18 +0800

Ubuntu 24.04

1. 每一台節點都要做環境準備

$ sudo nano /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
ens18:
dhcp4: false
addresses:
- 192.168.11.72/24
nameservers:
addresses:
- 192.168.11.11
routes:
- to: 0.0.0.0/0
via: 192.168.11.254
$ sudo nano /etc/systemd/timesyncd.conf
...
[Time]
NTP=192.168.11.11
...
$ sudo systemctl restart systemd-timesyncd
$ timedatectl show-timesync --all
$ sudo timedatectl set-timezone Asia/Taipei
$ timedatectl
Local time: Sun 2025-06-29 15:44:28 CST
Universal time: Sun 2025-06-29 07:44:28 UTC
RTC time: Sun 2025-06-29 07:44:28
Time zone: Asia/Taipei (CST, +0800)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
$ sudo swapoff -a
$ sudo sed -i '/swap/s/^/#/' /etc/fstab
$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF
$ sudo sysctl --system
$ sudo mkdir -p /opt/cni/bin && \
curl -sL "$(curl -sL https://api.github.com/repos/containernetworking/plugins/releases/latest | \
jq -r '.assets[].browser_download_url' | grep 'linux-amd64.*.tgz$')" -o ~/cni-plugins.tgz && \
sudo tar xf ~/cni-plugins.tgz -C /opt/cni/bin
$ wget https://github.com/opencontainers/runc/releases/download/v1.3.0/runc.amd64 && \
sudo install -m 755 runc.amd64 /usr/local/sbin/runc
$ wget https://github.com/containerd/containerd/releases/download/v2.1.3/containerd-2.1.3-linux-amd64.tar.gz && \
sudo tar Cxzvf /usr/local containerd-2.1.3-linux-amd64.tar.gz && \
wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service && \
sudo mkdir -p /usr/local/lib/systemd/system && \
sudo mv containerd.service /usr/local/lib/systemd/system/containerd.service
$ sudo mkdir /etc/containerd
$ sudo containerd config default | sudo tee /etc/containerd/config.toml
$ sudo nano /etc/containerd/config.toml
......
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true #add
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now containerd
$ sudo systemctl is-active containerd.service
active
$ sudo apt-get update; sudo apt-get upgrade -y
$ sudo apt-get install -y apt-transport-https ca-certificates curl gpg
$ K8S_VERSION='v1.33'
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/${K8S_VERSION}/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg && \
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/${K8S_VERSION}/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
# Worker node 可不用裝 kubectl
$ sudo apt-get update; sudo apt-get install -y kubelet kubeadm kubectl
$ sudo apt-mark hold kubelet kubeadm kubectl

裝 K8s

mkdir ~/k8s; cd ~/k8s
nano ~/k8s/init-config.yaml

檔案內容 :

apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.11.71 # change from Master node IP
bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock # change from containerd Unix Socket
imagePullPolicy: IfNotPresent
name: c1 # change from Master node hsotname
taints: null
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.33.2
controlPlaneEndpoint: 192.168.11.71:6443
apiServer:
timeoutForControlPlane: 4m0s
#certSANs:
# - 10.10.7.33 # 加入 kube-vip 的 VIP 作為 TLS SAN
certificatesDir: /etc/kubernetes/pki
clusterName: topgun # set your clusterName
controllerManager:
extraArgs:
bind-address: "0.0.0.0"
secure-port: "10257"
scheduler:
extraArgs:
bind-address: "0.0.0.0"
secure-port: "10259"
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
listen-metrics-urls: "http://0.0.0.0:2381"
dns: {}
imageRepository: registry.k8s.io
networking:
dnsDomain: cluster.local # DNS domain used by Kubernetes Services.
podSubnet: 10.244.0.0/16 # the subnet used by Pods.
serviceSubnet: 10.96.0.0/16 # subnet used by Kubernetes Services.
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
systemReserved:
memory: "1Gi"
kubeReserved:
memory: "2Gi"

初始化 K8s

sudo kubeadm init --skip-phases=addon/kube-proxy --upload-certs --config=k8s/init-config.yaml

Join worker node

sudo kubeadm join 192.168.11.71:6443 --token 7pr0b4.79tdi8egs81l4i9g --discovery-token-ca-cert-hash sha256:c6bed66cc0c7f8616a4f920ccc62121b782c37475888a6cc9831ced22fde474f

設定 kubeconfig

mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config

裝 Cilium

# Install the Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
API_SERVER_IP=192.168.11.71
API_SERVER_PORT=6443
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=${API_SERVER_IP} \
--set k8sServicePort=${API_SERVER_PORT} \
--set bpf.masquerade=true \
--set ipam.mode=kubernetes \
--set ipv4NativeRoutingCIDR=10.244.0.0/16 \
--set bpf.datapathMode=netkit
helm upgrade cilium cilium/cilium \
--namespace=kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=${API_SERVER_IP} \
--set k8sServicePort=${API_SERVER_PORT} \
--set hubble.ui.enabled=true \
--set hubble.relay.enabled=true \
--set hubble.ui.frontend.server.ipv6.enabled=false \
--set hubble.ui.service.type=NodePort \
--set bpf.datapathMode=netkit
helm upgrade cilium cilium/cilium --version 1.17.6 \
--namespace kube-system \
--reuse-values \
--set ingressController.enabled=true \
--set ingressController.loadbalancerMode=dedicated
kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart ds/cilium

Validate the Installation

$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
cilium-28kw8 1/1 Running 1 (117m ago) 124m
cilium-envoy-c5nxh 1/1 Running 1 (117m ago) 124m
cilium-envoy-phwpp 1/1 Running 0 117m
cilium-envoy-sb7c2 1/1 Running 0 117m
cilium-m5mvp 1/1 Running 0 116m
cilium-operator-c64699c6b-2kgr5 1/1 Running 0 117m
cilium-operator-c64699c6b-4sg5r 0/1 Completed 0 124m
cilium-operator-c64699c6b-ft9x8 0/1 Completed 0 124m
cilium-operator-c64699c6b-zflrv 1/1 Running 0 117m
cilium-r6x55 1/1 Running 0 117m
coredns-674b8bbfcf-bzk8x 1/1 Running 1 (117m ago) 131m
coredns-674b8bbfcf-tf5fh 1/1 Running 1 (117m ago) 131m
etcd-c1 1/1 Running 1 (117m ago) 131m
hubble-relay-67cfb6bcbd-hx9t8 1/1 Running 0 117m
hubble-ui-85bc4bfc56-57g8f 2/2 Running 0 96m
kube-apiserver-c1 1/1 Running 1 (117m ago) 131m
kube-controller-manager-c1 1/1 Running 1 (117m ago) 131m
kube-scheduler-c1 1/1 Running 1 (117m ago) 131m

$ cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet cilium-envoy Desired: 3, Ready: 3/3, Available: 3/3
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 3
cilium-envoy Running: 3
cilium-operator Running: 2, Succeeded: 2
clustermesh-apiserver
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 8/8 managed by Cilium
Helm chart version: 1.18.0-dev
Image versions cilium quay.io/cilium/cilium-ci:latest: 3
cilium-envoy quay.io/cilium/cilium-envoy:v1.34.1-1750869463-42c7e8cf0f93ea19c9cb7e4d0ad2a339b3f81ad2@sha256:892cab92ffaf8499be90bd227bf07181e4b460ecd97750032ea7e91710a8acce: 3
cilium-operator quay.io/cilium/operator-generic-ci:latest: 4
hubble-relay quay.io/cilium/hubble-relay-ci:latest: 1
hubble-ui quay.io/cilium/hubble-ui-backend:v0.13.2@sha256:a034b7e98e6ea796ed26df8f4e71f83fc16465a19d166eff67a03b822c0bfa15: 1
hubble-ui quay.io/cilium/hubble-ui:v0.13.2@sha256:9e37c1296b802830834cc87342a9182ccbb71ffebb711971e849221bd9d59392: 1

洞悉 Cilium 如何傳遞 K8s Pod 網路封包

Fri, 08 May 2026 06:59:56 +0800

1. Preface

本篇文章會在 CNI 使用 Cilium 的 Kubernetes 環境驗證，pod 跟 pod 之間互相溝通時，在同節點或在不同節點，網路的封包會如何傳遞，並搭配 tcpdump + wireshark 分析封包。

2. 環境資訊

K8s 節點及版本資訊

kubectl get nodes -o wide

執行結果：

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubeadm-c1.bbg.com Ready control-plane 27d v1.34.1 172.20.6.11 <none> Ubuntu 24.04.3 LTS 6.8.0-85-generic containerd://2.1.4
kubeadm-c2.bbg.com Ready control-plane 27d v1.34.1 172.20.6.12 <none> Ubuntu 24.04.3 LTS 6.8.0-85-generic containerd://2.1.4
kubeadm-c3.bbg.com Ready control-plane 26d v1.34.1 172.20.6.13 <none> Ubuntu 24.04.3 LTS 6.8.0-85-generic containerd://2.1.4
kubeadm-w1.bbg.com Ready worker 26d v1.34.1 172.20.6.14 <none> Ubuntu 24.04.3 LTS 6.8.0-85-generic containerd://2.1.4
kubeadm-w2.bbg.com Ready worker 26d v1.34.1 172.20.6.15 <none> Ubuntu 24.04.3 LTS 6.8.0-85-generic containerd://2.1.4
kubeadm-w3.bbg.com Ready worker 26d v1.34.1 172.20.6.16 <none> Ubuntu 24.04.3 LTS 6.8.0-85-generic containerd://2.1.4

cilium 設定資訊

helm -n kube-system get values cilium

執行結果：

USER-SUPPLIED VALUES: # 忽略此行
bpf:
masquerade: true
hubble:
relay:
enabled: true
ui:
enabled: true
service:
type: NodePort
ipam:
mode: kubernetes
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 127.0.0.1
k8sServicePort: 6443
kubeProxyReplacement: true

確認 cilium 版本、routing 模式和 device mode

kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep -iE "^cilium:|^routing|^device mode"

執行結果：

Cilium: Ok 1.18.2 (v1.18.2-5bd307a8)
Routing: Network: Tunnel [vxlan] Host: BPF
Device Mode: veth

本篇文章測試環境 routing-mode 為 tunnel，tunnel-protocol 使用 vxlan，沒有特別 tunning。

建立 NGINX pods

kubectl create deploy nginx --image=docker.io/library/nginx:latest --replicas=2

建立 Client pods，replicas 設成 2，並指定跑在 worker node

echo 'apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: client
name: client
spec:
replicas: 2
selector:
matchLabels:
app: client
template:
metadata:
labels:
app: client
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""
containers:
- image: quay.io/cooloo9871/debug.alp
name: debug-alp
securityContext:
privileged: true' | kubectl apply -f -

檢視 pods 各至在哪一個節點

kubectl get pods -o wide

執行結果類似以下螢幕輸出：

NAME STATUS NODE
client-65766844b7-72zg2 Running kubeadm-w1.bbg.com
client-65766844b7-c4w29 Running kubeadm-w2.bbg.com
nginx-66686b6766-6vd9x Running kubeadm-w2.bbg.com
nginx-66686b6766-s25sh Running kubeadm-w3.bbg.com

3. 檢視單一 Pod 的網路架構

查看 client pod 的 ip address 和 subnet mask

kubectl exec -it client-65766844b7-c4w29 -- ip a

執行結果：

...以上省略
34: eth0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether d6:c4:d2:00:8c:2e brd ff:ff:ff:ff:ff:ff
inet 10.244.7.250/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::d4c4:d2ff:fe00:8c2e/64 scope link
valid_lft forever preferred_lft forever

得知 subnet mask 為 /32，所以 pod 要上網一定會走 Default Gateway

查看 client Pod 的 Default Gateway

kubectl exec -it client-65766844b7-c4w29 -- ip r

執行結果：

default via 10.244.7.247 dev eth0
10.244.7.247 dev eth0 scope link

確認 Default Gateway 的值為 10.244.7.247

查看 client Pod 裡的 Neighbour Table
```
kubectl exec -it client-65766844b7-c4w29 -- ip n
```
執行結果：
```
10.244.7.247 dev eth0 lladdr 1e:49:e0:67:ec:e7 used 0/0/0 probes 4 STALE
```
得知 Default Gateway 10.244.7.247 對應的 Mac Address 為 1e:49:e0:67:ec:e7

到 pod 所在的節點查找 pod 內看到 Default Gateway 對應在主機是哪張網卡

ip a s | grep -A 3 -B 2 10.244.7.247

執行結果：

4: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ba:af:34:f7:09:12 brd ff:ff:ff:ff:ff:ff
inet 10.244.7.247/32 scope global cilium_host
valid_lft forever preferred_lft forever
inet6 fe80::b8af:34ff:fef7:912/64 scope link
valid_lft forever preferred_lft forever

發現 10.244.7.247 這張網卡對應的 Mac Address 竟然是 ba:af:34:f7:09:12 而不是在 client pod 裡面看到的 1e:49:e0:67:ec:e7

繼續在 pod 所在的節點查找 1e:49:e0:67:ec:e7 是哪張網卡

ip a s | grep -A 3 -B 1 '1e:49:e0:67:ec:e7'

執行結果：

35: lxc7904b406cec6@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 1e:49:e0:67:ec:e7 brd ff:ff:ff:ff:ff:ff link-netns cni-a59f663a-3aed-32da-7743-287142c3ccda
inet6 fe80::1c49:e0ff:fe67:ece7/64 scope link
valid_lft forever preferred_lft forever

由網卡名稱可以得知是 veth pair，並且另一端連接在 cni-a59f663a-3aed-32da-7743-287142c3ccda Linux Network Namespace

進入這個 Linux Network Namespace 查看網卡資訊

sudo ip netns exec cni-a59f663a-3aed-32da-7743-287142c3ccda ip a

執行結果：

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
34: eth0@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether d6:c4:d2:00:8c:2e brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.7.250/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::d4c4:d2ff:fe00:8c2e/64 scope link
valid_lft forever preferred_lft forever

確認 client pod eth0 在主機側的網卡名稱是 lxc7904b406cec6

確認初始狀態下 pod 內部 arp cache
```
kubectl exec -it client-65766844b7-c4w29 -- arp -a
```
執行結果
得知 pod 在初始狀態下 arp cache 應為空

小結論

Client Pod eth0 網卡對應在 host 主機這側的網卡是 lxc7904b406cec6
在 Client Pod 內部 Neighbour Table 紀錄 Default Gateway 的 IP 是在 host 主機 cilium_host 網卡的 ip，但是 Mac Address 卻是 pod 在 host 主機側網卡的 mac address
pod 在初始狀態下無任何 arp cache
網路架構如下圖

4. 檢視同一台節點內 Pods 之間網路封包如何傳遞

查詢 nginx pod 網卡資訊，在 pod 所在的節點執行以下命令

pod_name="nginx-66686b6766-6vd9x"
sudo ip netns exec $(sudo ip netns identify $(sudo crictl inspect $(sudo crictl ps -a | grep ${pod_name} | cut -d " " -f 1) | jq -r '.info.pid // .status.pid')) ip a s eth0

執行結果：

32: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 9a:08:1c:60:8e:77 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.7.203/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::9808:1cff:fe60:8e77/64 scope link
valid_lft forever preferred_lft forever

得知 nginx pod 網卡資訊：

名稱為 eth0

ip address 為 10.244.7.203/32

mac address 為 9a:08:1c:60:8e:77

找到 nginx pod 在主機側的網卡，在 pod 所在的節點執行以下命令

pod_name="nginx-66686b6766-6vd9x"
ip a s | grep -A 3 ^$(sudo ip netns exec $(sudo ip netns identify $(sudo crictl inspect $(sudo crictl ps -a | grep ${pod_name} | cut -d " " -f 1) | jq -r '.info.pid // .status.pid')) ip a s eth0 | head -n 1 | cut -d ":" -f 2 | tail -c 3)

執行結果：

33: lxcdad8b6341b55@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 2a:e0:31:24:98:d1 brd ff:ff:ff:ff:ff:ff link-netns cni-90146bc6-03b3-1ff1-72c8-61b61819bb95
inet6 fe80::28e0:31ff:fe24:98d1/64 scope link
valid_lft forever preferred_lft forever

得知 nginx pod 在主機側網卡資訊：

名稱為 lxcdad8b6341b55

mac address 為 2a:e0:31:24:98:d1

額外開啟 4 個終端機，分別都 ssh 到 client 和 nginx pod 所在的同一台節點上，並依序執行以下 4 個收集封包的動作
1. 在 client pod 主機側的網卡上，透過 tcpdump 命令收集 ARP 或 ICMP 封包
```
sudo tcpdump -n -i lxc7904b406cec6 arp or icmp -vvv -w client-pod-host.pcap
```
2. 在 nginx pod 主機側的網卡上，透過 tcpdump 命令收集 ARP 或 ICMP 封包
```
sudo tcpdump -n -i lxcdad8b6341b55 arp or icmp -vvv -w nginx-pod-host.pcap
```
3. 在 client pod Container 側的網卡上，透過 tcpdump 命令收集 ARP 或 ICMP 封包
```
sudo ip netns exec cni-a59f663a-3aed-32da-7743-287142c3ccda tcpdump -n -i eth0 arp or icmp -vvv -w client-pod.pcap
```
4. 在 nginx pod Container 側的網卡上，透過 tcpdump 命令收集 ARP 或 ICMP 封包
```
sudo ip netns exec cni-90146bc6-03b3-1ff1-72c8-61b61819bb95 tcpdump -n -i eth0 arp or icmp -vvv -w nginx-pod.pcap
```

切換到可執行 kubectl 主機的終端機，進到 client pod 去 ping nginx pod

kubectl exec -it client-65766844b7-c4w29 -- ping -c 1 10.244.7.203

執行結果：

PING 10.244.7.203 (10.244.7.203): 56 data bytes
64 bytes from 10.244.7.203: seq=0 ttl=63 time=0.198 ms
--- 10.244.7.203 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.198/0.198/0.198 ms

小總結

client Pod 去 ping 同一節點的 nginx pod，因 subnet mask 為 /32，網路封包一定往 Default Gateway 10.244.7.247 送，由於初始化 pod 內部 arp cache 為空，故觸發 arp 通訊協定來去得到 Default Gateway 的 mac address。封包詳細資訊：
由 arp 通訊協定得知 Default Gateway 的 mac address 為 1e:49:e0:67:ec:e7，這 mac address 對應的網卡是 client pod 在主機側的網卡 lxc7904b406cec6。封包詳細資訊：
更詳細的運作流程：
1. client Pod 以廣播的方式發送 arp request 封包
2. arp request 被在主機側的網卡收到，此時 cilium bpf cil_from_container 程式會立刻接手
3. 經過一連串判斷，會用自己的 MAC address 假裝自己是 gateway 來回復這個 arp request，詳情請看 shiun 大佬的源碼分析
  
  cil_from_container 在做什麼？簡單一句話來解釋就是：「cil_from_container 是 application container 網路的出口警衛，專門檢查所有從 app container 送出的網路封包，並決定它們的下一步該怎麼走。」
4. client pod 收到被偽造 Mac address 的 arp reply
此時 icmp echo request 封包從 pod 內發送到在主機側的 lxc7904b406cec6 網卡封包詳細資訊：
立刻又被 Cilium 掛在該網卡上的 BPF 程式 cil_from_container 接住，並作出以下判斷
- 如果目標 pod 是同一個 Node → 找到目標 pod 在主機側的網卡 → 然後將封包送進 Container 側的網卡
  - 更詳細運作流程：
    1. 拿到封包後，把目的 MAC Address 和來源的 MAC Address 的換掉，確保能送到正確的 Pod
    2. 直接送達：它不把包裹交給大樓的收發室 (Linux Network Stack)，而是直接搭電梯（tail_call_policy）衝到收件 Pod 的門口（在主機側的網卡），把包裹交給門口的警衛 (Ingress BPF 程式) 做安全檢查 (Policy Enforcement)
  - 以下是從 client pod 在主機側網卡收到的 icmp echo request 封包資訊 (為更換 mac address 前) 封包詳細資訊：遺憾的是沒收到更換完 mac address 的 icmp echo request 封包
- 如果目標 pod 是跨 Node → 會轉交 cilium_vxlan / ens33 網卡
很明顯此次的目標 pod 跟來源 pod 是在同一個 node，經過 cilium bpf 程式一番查找得知要傳給目標 pod 在主機側的網卡，所以 icmp echo request 封包會發送到 nginx pod 在主機側的網卡 lxcdad8b6341b55
Nginx pod 在主機側的網卡收到 icmp echo request 封包後，cilium bpf 程式 cil_from_container 立馬接手，執行 nginx Pod 的 Ingress Policy 檢查。
檢查沒問題後，BPF 對 kernel 說：「我檢查完了，這封包沒問題，你接手處理吧！」。
此時封包會送進 nginx pod 的 Linux Network Namspace，由於之前已經修改了目的地和來源的 mac address，封包會被正確路由到 nginx Pod 在 Container 側的網卡。封包詳細資訊：
nginx pod 收到 icmp echo request 封包後，會需要回傳 icmp echo reply 的網路封包，同樣會因為 subnet mask 為 /32，網路封包一定往 Default Gateway 10.244.7.247 送，由於初始化 pod 內部 arp cache 為空，故又觸發 arp 通訊協定來去得到 Default Gateway 的 mac address(實際上會得到的是 nginx pod 在主機側網卡的 mac address) (arp 詳細運作流程不再重複描述)
透過 arp 通訊協定得知 nginx 在主機側網卡 mac 位址後，icmp echo reply 封包會發送至 nginx pod 在主機側的網卡封包詳細資訊：
icmp echo reply 封包傳送到在 nginx pod 主機側的網卡後，又又又被 cilium bpf 程式 cil_from_container 接手，並作出以下判斷 :
1. 目標 pod 是同一個 Node
2. 把目的 MAC Address 和來源的 MAC Address 的換掉，確保能送回 client pod
3. 直接將網路封包送達 client pod 在主機側的網卡
client pod 在主機側的網卡收到 icmp echo reply 封包後，cilium bpf 程式 cil_from_container 接手，執行 client Pod 的 Ingress Policy 檢查，確認沒問題後最後傳回給 client pod 在 container 側的網卡。封包詳細資訊：

補充說明

安裝 cilium 後，會在主機的 os 裡面，看到 cilium_host, cilium_net, 和 cilium_vxlan 這 3 個網路介面，目前根據上面的驗證結果可得知在同節點內 pod 跟 pod 之間互相溝通，並不會經過這 3 張網卡。

如果想驗證，可在 pods 之間互 ping 之前，執行以下收集封包的命令

收集 cilium_host 網路介面的封包

sudo tcpdump -n -i cilium_host arp or icmp -vvv -w cilium_host.pcap

收集 cilium_net 網路介面的封包

sudo tcpdump -n -i cilium_net arp or icmp -vvv -w cilium_net.pcap

收集 cilium_vxlan 網路介面的封包

sudo tcpdump -n -i cilium_net arp or icmp -vvv -w cilium_vxlan

收集結果會看到 icmp 封包，但 icmp 封包內要傳遞的 data 一定會跟 pod 之間互 ping 的 data 不一樣

所以可以確定在同節點內 pod 跟 pod 之間互相溝通，並不會經過 cilium_host, cilium_net, 和 cilium_vxlan 這 3 張網卡。

5. 參考文章

CNI 運作原理：理解 Container Network Interface 如何讓跨主機的 Pod 通訊

Fri, 08 May 2026 05:29:18 +0800

TL;DR

當 worker-1 上的 Pod A ping worker-2 上的 Pod B：

Pod 內 eth0（/32 IP、169.254.1.1 link-local default gateway）→ 透過 Calico CNI 建的 veth pair 出到 host netns
Host netns 的 routing 表把對端 podCIDR 指向 flannel.1（VXLAN device，VNI 1）
Linux kernel 用 ip neigh + bridge fdb 查到對端 underlay IP，做 VXLAN encap（UDP port 8472）
封包經 node container 的 eth0 → host bridge podman1 → 對端 node 的 eth0 → 對端 flannel.1 做 VXLAN decap → veth → Pod B eth0

Calico 負責建 veth、設 proxy_arp；Flannel 負責跨 node 的 host route 與 VXLAN fdb；kernel 負責真正的 encap/decap。

不熟悉 veth pair / encap / overlay / fdb / VXLAN 等術語的讀者，建議先翻 Appendix B 術語表一次再讀正文。

TL;DR
0. 前提條件
1. Kubernetes networking model 三條公約
2. Canal 架構快速理解
3. 環境探勘（Podman 特有步驟）
4. 部署 demo workload
5. 第一層觀察：Pod 內看到什麼
6. 第二層觀察：node container 內看到什麼
7. 第三層觀察：抓真正的 VXLAN 封包
8. Packet flow 圖
9. 常見驗證 checklist
10. 收尾
Appendix A. 指令速查表
Appendix B. 術語表
References

目標

在一個 1 master + 2 workers 的 Kubernetes cluster（用 Podman 建立、運作方式類似 kind），CNI 採用 Canal 的情境下，回答一個具體問題：

當 worker-1 上的 Pod A ping worker-2 上的 Pod B，封包到底走了哪些 interface、被誰 encap、又被誰 decap？

讀完本文後，應能在自己的 cluster 上用 kubectl、podman exec、ip、tcpdump 等工具，把整條 datapath 親眼看一遍。

適合對象：已熟悉 Pod / Deployment / Service 的使用者，但對 overlay network、VXLAN、CNI plugin chain 仍是黑盒子。

驗證環境：cluster tk8s（v1.35.0、container runtime containerd 2.2.0、node image quay.io/cloudwalker/taroko:v1.35.0、3 node：tk8s-control-plane / tk8s-worker1 / tk8s-worker2）。其他環境若 IP/CIDR 不同，照樣換值即可。

0. 前提條件

這節重點：確認 cluster 是 1m2w、INTERNAL-IP 落在 Podman bridge 上、Canal 已安裝（每 node 一個 canal pod，內含 calico-node + kube-flannel 兩個 container）。

(1) 看 cluster 拓樸：

kubectl get node -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP ...
tk8s-control-plane Ready control-plane 2d v1.35.0 172.22.0.1 ...
tk8s-worker1 Ready taroko-worker 2d v1.35.0 172.22.0.2 ... ← 等等 demo Pod A 落腳點
tk8s-worker2 Ready taroko-gateway 2d v1.35.0 172.22.0.3 ... ← Pod B 落腳點

(2) 看 Canal 是否安裝、每個 canal pod 由哪些 container 組成：

kubectl get pods -n kube-system -l k8s-app=canal -o wide
kubectl get pods -n kube-system -l k8s-app=canal \
 -o jsonpath='{.items[0].spec.containers[*].name}'; echo

NAME READY STATUS RESTARTS AGE IP NODE
canal-4wvhj 2/2 Running 0 2d 172.22.0.1 tk8s-control-plane
canal-qsnqh 2/2 Running 0 2d 172.22.0.3 tk8s-worker2
canal-vjcm9 2/2 Running 0 2d 172.22.0.2 tk8s-worker1
calico-node kube-flannel ← 兩個 container：calico-node 跑 Felix、kube-flannel 跑 flanneld

找不到 canal pod 時，改用 kubectl get pods -n kube-system | grep -E 'canal|calico|flannel' 比對；不同 manifest 的 label 命名可能不同。

1. Kubernetes networking model 三條公約

這節重點：K8s 不規定怎麼做，只規定三件事必須成立；CNI plugin 的工作就是把它變成現實。

依 kubernetes.io 的規範：

#	公約	通俗說法
1	每個 Pod 拿到一個 cluster 內 routable 的 IP	不用 port forwarding、不會 port 衝突
2	Pod-to-Pod 通訊不經 NAT	Pod A 看到自己的 IP，跟 Pod B 看到 Pod A 的 IP，必須一致
3	Node agent（kubelet 等）能直連該 node 上的 Pod	host 和 Pod 之間沒有牆

CNI plugin 的職責：

IPAM：給每個 Pod 配 IP
Plumbing：在 host 與 Pod 之間建立 veth
Forwarding：設好 routing / encap，讓跨 node 的 Pod 互通

2. Canal 架構快速理解

這節重點：Canal = Flannel（負責 networking）+ Calico（負責 NetworkPolicy）。Calico 在這個組合下跑 policy-only mode、不跑 BGP。

角色	實作者	跑在哪裡
CNI plugin（建立 veth、配 IP）	Calico CNI	node 上的 `/opt/cni/bin/calico` binary
Overlay backend（跨 node 封裝）	Flannel VXLAN	canal pod 的 `kube-flannel` container
NetworkPolicy 執行	Calico Felix	canal pod 的 `calico-node` container
Routing protocol（BGP）	不啟用	`CALICO_NETWORKING_BACKEND=none`

本文只示範 datapath 連通；NetworkPolicy 不在範圍內。

3. 環境探勘（Podman 特有步驟）

這節重點：node 是 host 上的 Podman container；node 的 eth0 = host 的 podman1 bridge 上的 veth。之後抓 underlay 封包就要看這座 bridge。

本 cluster 的 Podman 跑在 rootful 模式，所以指令要加 sudo。

(1) 列出 node container：

sudo podman ps --format '{{.Names}}\t{{.Image}}\t{{.Status}}'

tk8s-control-plane quay.io/cloudwalker/taroko:v1.35.0 Up 2 days
tk8s-worker1 quay.io/cloudwalker/taroko:v1.35.0 Up 2 days
tk8s-worker2 quay.io/cloudwalker/taroko:v1.35.0 Up 2 days

(2) 找 nodes 共用的 Podman network：

sudo podman network ls
sudo podman network inspect tk8s

NETWORK ID NAME DRIVER
2f259bab93aa podman bridge ← 預設、未使用
2543f305ded9 tk8s bridge ← cluster 真正用的

[{
 "name": "tk8s",
 "driver": "bridge",
 "network_interface": "podman1", ← 對應 host 上的 bridge 名稱
 "subnets": [{
 "subnet": "172.22.0.0/24",
 "gateway": "172.22.0.254"
 }]
 // ... 其他欄位省略
}]

結論（記住這三件事，後面都會用到）：

Podman network tk8s ↔ host bridge podman1
三個 node 都掛在 podman1 上、IP 落在 172.22.0.0/24
node 內 eth0 = 接到 podman1 的 veth；node 的 INTERNAL-IP 就是它在 bridge 上的 IP

4. 部署 demo workload

這節重點：用 podAntiAffinity 強制兩個 nicolaka/netshoot Pod 各落到一台 worker；確認每 node 的 podCIDR 是 /25 切片；設好 $POD_A / $POD_B 等變數方便後續所有指令引用。

(1) 部署並等 Ready：

kubectl apply -f docs/manifests/cni-demo.yaml
kubectl wait --for=condition=Ready pod -l app=cni-demo --timeout=60s
kubectl get pod -l app=cni-demo -o wide

deployment.apps/cni-demo created
pod/cni-demo-548d4c6cbc-bgjqk condition met
pod/cni-demo-548d4c6cbc-lmk44 condition met
NAME READY STATUS IP NODE
cni-demo-548d4c6cbc-bgjqk 1/1 Running 10.244.0.153 tk8s-worker1 ← Pod A
cni-demo-548d4c6cbc-lmk44 1/1 Running 10.244.1.22 tk8s-worker2 ← Pod B

Pod 名稱與 IP 每次部署都會變動；只要結構一致（兩個 replica、各落在不同 worker、IP 在各自 podCIDR 內）即正確。

(2) 看 podCIDR 切片（cluster pod CIDR 10.244.0.0/16 被切成 /25 分給每個 node）：

kubectl get node -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'

tk8s-control-plane 10.244.0.0/25
tk8s-worker1 10.244.0.128/25 ← Pod A 來自這個區段
tk8s-worker2 10.244.1.0/25 ← Pod B 來自這個區段

(3) 設定 shell 變數（後面所有節都會用到）：

POD_A=$(kubectl get pod -l app=cni-demo --field-selector spec.nodeName=tk8s-worker1 -o jsonpath='{.items[0].metadata.name}')
POD_B=$(kubectl get pod -l app=cni-demo --field-selector spec.nodeName=tk8s-worker2 -o jsonpath='{.items[0].metadata.name}')
POD_A_IP=$(kubectl get pod $POD_A -o jsonpath='{.status.podIP}')
POD_B_IP=$(kubectl get pod $POD_B -o jsonpath='{.status.podIP}')
echo "POD_A=$POD_A ($POD_A_IP @ tk8s-worker1)"
echo "POD_B=$POD_B ($POD_B_IP @ tk8s-worker2)"

POD_A=cni-demo-548d4c6cbc-bgjqk (10.244.0.153 @ tk8s-worker1)
POD_B=cni-demo-548d4c6cbc-lmk44 (10.244.1.22 @ tk8s-worker2)

5. 第一層觀察：Pod 內看到什麼

這節重點：Pod 內只有 eth0、IP 是 /32、default gateway 是 dummy 的 169.254.1.1。Pod 不知道 overlay、不知道對端 node。

kubectl exec $POD_A -- ip a show eth0
kubectl exec $POD_A -- ip route
kubectl exec $POD_A -- ping -c 3 $POD_B_IP

2: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ... ← MTU 1500（Calico 預設）
link/ether 32:90:8e:80:e6:7a ...
inet 10.244.0.153/32 scope global eth0 ← 注意是 /32
default via 169.254.1.1 dev eth0 ← link-local dummy gateway
169.254.1.1 dev eth0 scope link
PING 10.244.1.22 (10.244.1.22) 56(84) bytes of data.
64 bytes from 10.244.1.22: icmp_seq=1 ttl=62 time=0.614 ms ← ttl 從 64 掉到 62
64 bytes from 10.244.1.22: icmp_seq=2 ttl=62 time=0.370 ms
64 bytes from 10.244.1.22: icmp_seq=3 ttl=62 time=0.399 ms
--- 10.244.1.22 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2084ms

為什麼會這樣？

/32 mask：Pod 把 host 視為 point-to-point 對端，沒有 subnet 概念
169.254.1.1 是 Calico CNI 寫死的 dummy next hop（Calico CNI 原始碼：gw := net.IPv4(169, 254, 1, 1)）；host-side veth 上設定 proxy_arp=1，所以 ARP 永遠回應 host MAC，然後封包就到 host netns 了
ttl 從 64 → 62 = 兩端 flannel.1 各 decrement 一次（VXLAN device 跨 layer-3 forwarding）
MTU 1500 vs 後面會看到的 flannel.1 MTU 1450 → § 8 MTU mismatch 小提醒

6. 第二層觀察：node container 內看到什麼

這節重點：node netns 上有三類 interface（eth0、flannel.1、caliXXXX veth）；routing / fdb / neigh 三張表加起來決定封包要怎麼走。

本節指令都用 sudo podman exec tk8s-worker1 <cmd> 從 host 跑；想拿 shell 互動的話，改用 sudo podman exec -it tk8s-worker1 bash。

6.1 Interface 全貌

sudo podman exec tk8s-worker1 ip -br a

lo UNKNOWN 127.0.0.1/8 ::1/128
eth0@if5 UP 172.22.0.2/24 ... ← 接到 host podman1 的 veth
flannel.1 UNKNOWN 10.244.0.128/32 ... ← VXLAN device，IP 為 podCIDR 首 IP
cali96e9f6c0c5f@if2 UP ... ┐
cali431d010c6bb@if2 UP ... │
cali6c6a1f0801c@if2 UP ... ├ Calico CNI 為各本機 Pod 建的 veth
cali55122502246@if2 UP ... │
caliaf66b54b075@if2 UP ... │
cali58f769715fe@if2 UP ... ┘ ← 這條對應到 demo Pod A

確認 VXLAN 參數：

sudo podman exec tk8s-worker1 ip -d link show flannel.1

5: flannel.1: ... mtu 1450 ...
link/ether e6:16:30:64:14:d3 ...
vxlan id 1 ← VNI 1
local 172.22.0.2 ← encap 後外層 src IP
dev eth0 ← 從 eth0 寄出
dstport 8472 ← Flannel VXLAN UDP port
nolearning ...

定位 Pod A 對應的 host-side veth（用 ifindex 對照）：

POD_A_IFLINK=$(kubectl exec $POD_A -- cat /sys/class/net/eth0/iflink)
echo "POD_A_IFLINK=$POD_A_IFLINK"
sudo podman exec tk8s-worker1 sh -c "ip link | grep -A1 \"^${POD_A_IFLINK}:\""

POD_A_IFLINK=29
29: cali58f769715fe@if2: ... mtu 1500 ...
link/ether ee:ee:ee:ee:ee:ee ...
link-netns cni-d3b2a847-5b76-c640-8ffa-0e3952643e1b ← 對端 netns（Pod 的 netns）

ifindex 與 caliXXXX hash 每個 Pod 都不同；29 / cali58f769715fe 只是這次的值。三類 interface 角色一覽：

Interface 角色

eth0 underlay 進出口；連到 host bridge podman1

flannel.1 VXLAN device（VNI 1、dstport 8472、IP 為 podCIDR 首 IP）

caliXXXX 每個本機 Pod 一條 veth（host 端）

Interface	角色
`eth0`	underlay 進出口；連到 host bridge `podman1`
`flannel.1`	VXLAN device（VNI 1、dstport 8472、IP 為 podCIDR 首 IP）
`caliXXXX`	每個本機 Pod 一條 veth（host 端）

6.2 Routing 表

sudo podman exec tk8s-worker1 ip route

default via 172.22.0.254 dev eth0 proto static metric 100 ← (a) 對外 default
10.244.0.0/25 via 10.244.0.0 dev flannel.1 onlink ← (c) 跨 node：到 control-plane podCIDR
10.244.0.130 dev cali96e9f6c0c5f scope link ┐
10.244.0.131 dev cali431d010c6bb scope link │
10.244.0.132 dev cali6c6a1f0801c scope link │
10.244.0.136 dev cali55122502246 scope link ├ (b) 本機 Pod：每個 /32 走對應 cali veth
10.244.0.138 dev caliaf66b54b075 scope link │
10.244.0.153 dev cali58f769715fe scope link ┘ ← demo Pod A
10.244.1.0/25 via 10.244.1.0 dev flannel.1 onlink ← (c) 跨 node：到 worker2 podCIDR ★
172.22.0.0/24 dev eth0 proto kernel scope link src 172.22.0.2 ← (d) host underlay subnet

封包送往 Pod B（10.244.1.22）時，命中第 (c) ★ 條規則：

next-hop 是 10.244.1.0 —— 正好是對端 worker2 上 flannel.1 的 IP
onlink 告訴 kernel：「直接視為同一鏈路」，不必再對 next-hop 做 ARP
出 device 是本機 flannel.1 —— 等於告訴 kernel「這條要做 VXLAN encap」

類型	範例	由誰寫的
(a) 對外 default	`default via 172.22.0.254 dev eth0`	node 啟動時 DHCP/Podman
(b) 本機 Pod /32	`10.244.0.153 dev cali58f769715fe`	Calico CNI 每建一個 Pod 寫一條
(c) 跨 node Pod CIDR	`10.244.1.0/25 via 10.244.1.0 dev flannel.1 onlink`	Flannel daemon
(d) host underlay	`172.22.0.0/24 dev eth0`	kernel 自動

6.3 VXLAN forwarding database 與 neighbor

sudo podman exec tk8s-worker1 bridge fdb show dev flannel.1
sudo podman exec tk8s-worker1 ip neigh show dev flannel.1

# bridge fdb show dev flannel.1 ← 對端 flannel.1 MAC → 對端 underlay IP
d2:41:35:4c:5e:90 dst 172.22.0.3 self permanent ← worker2
d6:e5:aa:e1:8a:0e dst 172.22.0.1 self permanent ← control-plane
# ip neigh show dev flannel.1 ← 對端 flannel.1 IP → 對端 flannel.1 MAC
10.244.1.0 lladdr d2:41:35:4c:5e:90 PERMANENT ← worker2
10.244.0.0 lladdr d6:e5:aa:e1:8a:0e PERMANENT ← control-plane

kernel 對 Pod B 做 VXLAN encap 的四步查表：

Step 1 ip route 10.244.1.22 命中 10.244.1.0/25 via 10.244.1.0 dev flannel.1 onlink
Step 2 ip neigh 10.244.1.0 → MAC d2:41:35:4c:5e:90
Step 3 bridge fdb MAC d2:41:35:4c:5e:90 → underlay dst 172.22.0.3
Step 4 build VXLAN packet outer = UDP 172.22.0.2:?? → 172.22.0.3:8472, VNI 1
inner = original ICMP (10.244.0.153 → 10.244.1.22)
送出 device = eth0

這兩張表是 Flannel daemon 透過 Kubernetes API 蒐集各 node 的 flannel.alpha.coreos.com/* annotation 後，主動寫到 kernel 的（PERMANENT 條目）—— 完全不需要 multicast、不需要 BGP。

6.4 Calico Felix 的痕跡

即使沒有 NetworkPolicy，Felix 也會建立基礎 chain 結構。本文不展開、只示範它存在。

sudo podman exec tk8s-worker1 iptables -t filter -L | grep -i cali | head

cali-INPUT ... /* cali:Cz_u1IQiXIMmKD4c */ ┐
cali-FORWARD ... /* cali:wUHhoiAYhphO9Mso */ ├ Felix 的三個 hook chain
cali-OUTPUT ... /* cali:tVnHkvAo15HuiPy0 */ ┘
ACCEPT ... /* Policy explicitly accepted packet. */ mark match 0x10000/0x10000
MARK ... MARK or 0x10000
MARK ... MARK and 0xffe5ffff
cali-from-hep-forward ... ← host endpoint forward dispatcher
cali-from-wl-dispatch ... ← workload (Pod) ingress dispatcher
cali-to-wl-dispatch ... ← workload (Pod) egress dispatcher

7. 第三層觀察：抓真正的 VXLAN 封包

這節重點：在 host 對 podman1 bridge 抓 UDP 8472，能同時看到外層 underlay 與內層 Pod-to-Pod ICMP。

本 cluster 的 node image 沒裝 tcpdump（sudo podman exec tk8s-worker1 which tcpdump 回 missing），所以直接在 host 抓最快。

開兩個 terminal：

# Terminal 1：host 上對 podman1 抓 UDP 8472（用 host filter 收緊噪音）
sudo tcpdump -i podman1 -nn -c 8 \
 'udp port 8472 and host 172.22.0.2 and host 172.22.0.3'

# Terminal 2：發 ping
kubectl exec $POD_A -- ping -c 4 $POD_B_IP

listening on podman1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
11:40:13.754197 IP 172.22.0.2.60047 > 172.22.0.3.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.153 > 10.244.1.22: ICMP echo request, id 9, seq 1, length 64
11:40:13.754358 IP 172.22.0.3.60047 > 172.22.0.2.8472: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.1.22 > 10.244.0.153: ICMP echo reply, id 9, seq 1, length 64
... (其餘略)

逐欄對照：

欄位	意義
`IP 172.22.0.2.60047 > 172.22.0.3.8472`	外層：兩個 worker 在 podman1 上的 IP；dst port 8472 = Flannel VXLAN
`OTV, flags [I] (0x08)`	tcpdump 預設把 UDP 8472 顯示為 OTV（同 port、格式相容）；要強制顯示 VXLAN 加 `-T vxlan`
`instance 1`	即 VNI 1（與 `ip -d link show flannel.1` 的 `vxlan id 1` 一致）
內層 `IP 10.244.0.153 > 10.244.1.22: ICMP echo request`	原始 Pod-to-Pod ICMP，被 VXLAN 包在裡面

8. Packet flow 圖

這節重點：把前面三節學到的 interface / route / fdb / encap 串成一張圖。

worker-1 (Podman container, 172.22.0.2) worker-2 (Podman container, 172.22.0.3)
+-------------------------------+ +-------------------------------+
| Pod A netns | | Pod B netns |
| +------+ | | +------+ |
| | eth0 | 10.244.0.153/32 | | 10.244.1.22/32| eth0 | |
| | mtu | 1500 | | 1500| mtu | |
| +--+---+ | | +---+--+ |
| | (veth pair) | | (veth pair) | |
| +--+--------------+ | | +--------------+--+ |
| | cali58f769715fe | ← Calico | | Calico → | caliYYYYYYYYY | |
| | proxy_arp=1 | CNI | | CNI | proxy_arp=1 | |
| +-------+---------+ 建立 | | 建立 +---------+------+ |
| | | | | |
| [host route, 在 node netns] | | [host route, 在 node netns] |
| 10.244.1.0/25 via 10.244.1.0 | | 10.244.0.128/25 via ... |
| dev flannel.1 onlink | | dev flannel.1 onlink |
| | | | | |
| +-------v-----+ | | +-----v-------+ |
| | flannel.1 | VXLAN encap | | VXLAN decap | flannel.1 | |
| | id 1 | (in-kernel) | | (in-kernel) | id 1 | |
| | 10.244.0.128| MTU 1450 | | MTU 1450 | 10.244.1.0 | |
| +-------+-----+ | | +-----+-------+ |
| | | | | |
| +-------v-----+ UDP 8472 | | UDP 8472 +--v-------+ |
| | eth0 | src 172.22.0.2 | dst 172.22.0.3 | eth0 | |
| | (veth) | | | | (veth) | |
| +-------+-----+ | | +----+-----+ |
+----------|--------------------+ +----------------------|-------+
| |
| +------------------------------+ |
+--------->| host bridge: podman1 |<---------------+
| (Podman network "tk8s") |
| subnet 172.22.0.0/24 |
+------------------------------+
(L2 forwarding；這裡看到的是 underlay UDP 封包)

各步驟責任歸屬：

Calico CNI：建立 Pod 的 veth pair、設定 host-side caliXXXX、加上 proxy_arp=1、寫 <pod-ip>/32 dev caliXXXX scope link host route
Flannel daemon（canal/kube-flannel container）：管理 flannel.1 的存在、寫入跨 node 的 host route 與 VXLAN fdb / neigh
Linux kernel：實際做 VXLAN encap/decap、ARP 與 routing；fdb / neigh 查表
host 的 podman1 bridge：純 L2 forwarding，把 underlay UDP 在 nodes 之間搬

MTU mismatch 小提醒

Pod eth0 MTU = 1500（Calico CNI 預設）
flannel.1 MTU = 1450（VXLAN header 50 bytes）
podman1 bridge MTU = 1500

意思是：Pod 直接送一個 1500 byte 的 packet，到了 flannel.1 因 MTU 較小，要嘛被分片、要嘛觸發 ICMP「Fragmentation Needed」回去做 PMTUD。本 demo 用 ping（小 packet）看不到問題；做大量 TCP 流量時會用 PMTUD 收斂到 1450。如要徹底避免這個 mismatch，可以在 Calico install spec 顯式把 mtu 設成 1450。

9. 常見驗證 checklist

這節重點：ping 不通時按以下順序排查（依 Flannel troubleshooting 文件）。

#	檢查項	指令	健康時應看到
1	node 互通	`sudo podman exec tk8s-worker1 ping -c 2 172.22.0.3`	`0% packet loss`
2	podCIDR 已設	`kubectl get node tk8s-worker1 -o jsonpath='{.spec.podCIDR}'`	`10.244.0.128/25`
3	flannel subnet	見 9.3	`FLANNEL_SUBNET` 與 podCIDR 一致
4	VXLAN checksum offload bug	見 9.4	（只在懷疑時才下）

9.1 node 互通

sudo podman exec tk8s-worker1 ping -c 2 172.22.0.3

PING 172.22.0.3 (172.22.0.3) 56(84) bytes of data.
64 bytes from 172.22.0.3: icmp_seq=1 ttl=64 time=0.308 ms
64 bytes from 172.22.0.3: icmp_seq=2 ttl=64 time=0.236 ms
--- 172.22.0.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1032ms

9.2 podCIDR 是否設置

kubectl get node tk8s-worker1 -o jsonpath='{.spec.podCIDR}'; echo

10.244.0.128/25

9.3 flannel 寫好的 subnet

/run/flannel/subnet.env 在 canal pod 的 kube-flannel container 內，不在 node container 的 host 檔案系統。

CANAL_POD=$(kubectl get pod -n kube-system -l k8s-app=canal \
 --field-selector spec.nodeName=tk8s-worker1 \
 -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n kube-system $CANAL_POD -c kube-flannel -- cat /run/flannel/subnet.env

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.129/25 ← 與 podCIDR 10.244.0.128/25 對應（差別在 .128 vs .129：podCIDR 用 network address，flannel 用第一個 host IP）
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

9.4 VXLAN checksum offload（只在懷疑時才下）

抓得到 underlay UDP 8472 卻看不到 inner packet 時，可能是 VXLAN checksum offload bug：

sudo podman exec tk8s-worker1 ethtool -K flannel.1 tx-checksum-ip-generic off

Actual changes:
tx-checksum-ip-generic: off
tx-tcp-segmentation: off [not requested]
tx-tcp-ecn-segmentation: off [not requested]
tx-tcp-mangleid-segmentation: off [not requested]
tx-tcp6-segmentation: off [not requested]

本 cluster 無此問題；上面只是示範指令在這個環境跑得起來。如不需要，請用 ... on 改回。

10. 收尾

kubectl delete -f docs/manifests/cni-demo.yaml

deployment.apps "cni-demo" deleted from default namespace

Appendix A. 指令速查表

依「想看什麼」反查：

想看什麼	在哪裡跑	指令
Pod IP / MTU / MAC	`kubectl exec`	`ip a show eth0`
Pod default route	`kubectl exec`	`ip route`
Pod-to-Pod 連通	`kubectl exec`	`ping <pod B IP>`
node 上所有 interface	node container	`ip -br a`
flannel.1 的 VXLAN 參數	node container	`ip -d link show flannel.1`
Pod A 對應的 host-side veth	node container	`ip link \| grep -A1 ^<iflink>:`
node 的 routing 表	node container	`ip route`
對端 flannel.1 IP→MAC	node container	`ip neigh show dev flannel.1`
對端 MAC→underlay IP	node container	`bridge fdb show dev flannel.1`
Felix iptables 痕跡	node container	`iptables -t filter -L \| grep cali`
underlay VXLAN 封包	host	`tcpdump -i podman1 -nn 'udp port 8472'`
flannel 的 subnet.env	canal pod (`kube-flannel` container)	`cat /run/flannel/subnet.env`
Podman bridge / network	host	`sudo podman network inspect tk8s`
node container 列表	host	`sudo podman ps`

Appendix B. 術語表

B.1 Linux kernel networking

network namespace（netns） kernel 提供的隔離機制；每個 netns 有自己獨立的一套 interface、routing table、iptables、socket。每個 Pod 跑在自己的 netns 裡，看不到 host 與其他 Pod 的網路狀態。Node 本身（不論是真機還是 Podman container）也是一個 netns。

veth pair（virtual Ethernet pair） kernel 提供的成對 virtual interface，行為像「一條虛擬網線」：從一端送進去的封包會從另一端原樣出現。常用法是把一端放在 Pod netns、另一端放在 host netns，當作 Pod 與 host 之間的連線。Calico CNI 對每個 Pod 建一條：host 端 = caliXXXXXXXXXX（host netns）、pod 端 = eth0（Pod netns）。

bridge（Linux bridge） kernel 內建的 software switch，把多個 interface 聚成一個 L2 廣播域、做 MAC learning 與 forwarding。本文裡 host 上的 podman1 就是一個 bridge，連著三個 node container 的 veth。

MAC address L2 layer 的 6-byte 位址；同一個 L2 廣播域內用 ARP 由 IP 查到 MAC。

ARP（Address Resolution Protocol）/ neighbor table IPv4 用 ARP 廣播「誰有這個 IP？」來解析 IP→MAC；kernel 把結果快取在 neighbor table，用 ip neigh 看。

proxy ARP Host 收到 ARP request 時，雖然該 IP 不是自己也回應「我來」，引導對方把封包送過來、再由自己 forward。Calico 在每條 host-side veth 上設 /proc/sys/net/ipv4/conf/<veth>/proxy_arp = 1，所以 Pod 對 dummy gateway 169.254.1.1 發 ARP 永遠收到 host MAC，封包就被導進 host netns 處理。

routing table / next-hop kernel 決定「封包該從哪個 interface 出、下一跳給誰」的查表結構；用 ip route 看。Next-hop 是命中該條後送往的下一個 IP：對端在同一鏈路時 = 目的 IP；跨 layer-3 時 = gateway。

onlink（route flag） 強制 kernel 把 next-hop 視為「同一鏈路」、不再去 ARP 解析它。Flannel 寫的跨 node 路由 10.244.1.0/25 via 10.244.1.0 dev flannel.1 onlink 用這個 flag——對端 flannel.1 的 IP 雖然不在自己 subnet 內，但因為走 VXLAN，下一跳由 fdb / neigh 表處理而非 ARP。

TTL（Time To Live） IPv4 header 的 8-bit 計數器，每經過一個 router 減 1，到 0 就丟掉並回 ICMP Time Exceeded。送出端通常從 64 起算；本文 ping 看到 ttl=62 表示中間經過兩跳（兩端 flannel.1 各一次）。

MTU（Maximum Transmission Unit）/ PMTUD Interface 一次能送的最大 L3 封包大小。乙太網典型 1500 byte；VXLAN 因要加 50 byte header（IP 20 + UDP 8 + VXLAN 8 + 內層 Ethernet 14），inner MTU 通常 = underlay MTU − 50。當 sender 送了超過 path 上某段 MTU 的封包，DF flag 為 1 時 router 會丟封包並回 ICMP “Fragmentation Needed”，sender 據此調整——這個過程叫 Path MTU Discovery (PMTUD)。

B.2 Encapsulation 與 overlay 網路

encapsulation（encap）/ decapsulation（decap） encap 就是「把一個完整的封包當成 payload，包進另一個封包裡」，外層 header 的 src/dst 是傳輸路徑用的位址，內層才是原始封包；decap 就是反向取出內層。VXLAN 是其中一種 encap：把整個 L2 frame 包進 UDP。

underlay network / overlay network Underlay 是底下「真的能 routable / switchable 的網路」；overlay 是邏輯上跑在 underlay 上的虛擬網路。本文中：

underlay = host bridge podman1 + 三個 node 的 172.22.0.0/24
overlay = 10.244.0.0/16 的 Pod IP 空間，靠 VXLAN 在 underlay 上重建

Pod 不知道 overlay 是「假的」；underlay 不知道 Pod 的存在。中間靠 flannel.1 做翻譯。

VXLAN（Virtual Extensible LAN） RFC 7348 定義的 overlay 協定，把 L2 frame 包進 UDP（標準 port 4789；Flannel 為了相容歷史用 8472）。Linux kernel 支援，建一條 flannel.1 這種 VXLAN device 即可：對它送的封包會自動 encap、收到的會自動 decap。

VNI（VXLAN Network Identifier） 24-bit ID，類似 VLAN tag 但容量大得多；不同 VNI 的封包互相不可見。Flannel 預設 VNI = 1。

fdb（forwarding database） Bridge / VXLAN device 的 MAC→出口對應表。一般 bridge 是 MAC→bridge port；VXLAN device 是 MAC→underlay IP（kernel 知道要把這個 MAC 的封包 encap 後送去哪個 underlay endpoint）。本文 bridge fdb show dev flannel.1 看到的 dst 172.22.0.3 就是「對端 flannel.1 的 MAC 對應到對端 node 的 underlay IP」。

B.3 Kubernetes / CNI 元件

CNI（Container Network Interface） CNI spec：kubelet 每次起 / 停一個 Pod 時，按 spec 呼叫指定的 binary（如 /opt/cni/bin/calico），由它負責建 veth、配 IP、寫 host route。

IPAM（IP Address Management） 配 Pod IP 的子系統。CNI plugin 可以自己做 IPAM，也可以呼叫獨立 IPAM plugin。Calico 在 Canal 模式中自己做 IPAM。

podCIDR Node 物件 .spec.podCIDR，給每個 node 預先切好的 Pod IP 段。本 cluster 是 cluster pod CIDR 10.244.0.0/16 → 每 node 切 /25。

NetworkPolicy K8s API 物件，宣告「哪些 Pod 可以連到哪些 Pod / port」。執行靠 CNI 端的 controller，例如 Calico Felix 把它翻成 iptables / eBPF rules。本文不示範。

Calico Felix Calico 的 per-node agent，負責 program iptables / eBPF rules（執行 NetworkPolicy）以及維護 endpoint state。

Flannel daemon（flanneld） Per-node agent，watch K8s API 收集各 node 的 podCIDR、flannel.alpha.coreos.com/* annotation，然後在本機建 flannel.1 VXLAN device、寫 host route、寫 VXLAN fdb / neigh entry。

Calico CNI plugin 被 kubelet 呼叫的 binary（不是常駐 daemon），負責 Pod 啟動時建 veth pair、配 Pod IP、寫 host-side <pod-ip>/32 dev caliXXXX 路由、設 proxy_arp。

Canal 把 Calico（policy）+ Flannel（networking）兩個 project 的 manifest 包成一份 install。Calico 在這個組合下跑 policy-only mode（CALICO_NETWORKING_BACKEND=none），所以 Calico 的 BIRD（BGP daemon）不啟用、改由 Flannel 的 VXLAN 負責跨 node 連線。

BGP（Border Gateway Protocol） Internet 的 routing protocol，Calico 預設用它讓各 node 互相 announce 自己的 podCIDR、不需要 overlay encap。Canal 模式下 BGP 停用、改用 Flannel VXLAN，所以本文看不到 BGP 相關痕跡。

References

技術細節皆引自下列官方來源（依 CLAUDE.md 規範要求）：

Kubernetes networking model：https://kubernetes.io/docs/concepts/cluster-administration/networking/
Canal（Calico + Flannel）：https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/install-for-flannel
Calico architecture（Felix 等元件職責）：https://docs.tigera.io/calico/latest/reference/architecture/overview
Flannel backends（VXLAN VNI、UDP 8472）：https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md
Flannel troubleshooting：https://github.com/flannel-io/flannel/blob/master/Documentation/troubleshooting.md
Calico CNI dataplane source（169.254.1.1 gateway、proxy_arp）：https://github.com/projectcalico/calico/blob/master/cni-plugin/pkg/dataplane/linux/dataplane_linux.go

Kubernetes kube-controller-manager 設定參數深度解析

Thu, 23 Apr 2026 13:43:54 +0800

從 node-cidr-mask-size 與 terminated-pod-gc-threshold 出發,了解 PodGC Controller 的運作原理,並透過 Kind 本機環境實作驗證。

文件資訊

項目	內容
文件主題	kube-controller-manager `extraArgs` 參數解析與 PodGC 實作驗證
建立日期	2026-04-23
適用版本	Kubernetes(以 master 分支原始碼為準)
驗證環境	Kind 本機叢集
主要參考	kubernetes/pkg/controller/podgc

一、背景:設定檔的用途

本文件所討論的設定檔片段:

controllerManager:
 extraArgs:
 - name: node-cidr-mask-size  # set pod cidr per node
 value: "25"
 # Garbage Collection
 # Max number of terminated pods to keep before cleaning up
 - name: terminated-pod-gc-threshold
 value: "500"

這是 Kubernetes kubeadm ClusterConfiguration 中,用來傳遞額外參數給 kube-controller-manager 的設定片段。extraArgs 裡面的每一項,最終會被轉換成 kube-controller-manager 啟動時的命令列參數(例如 --node-cidr-mask-size=25)。

kube-controller-manager 是 Kubernetes 控制平面的核心元件之一,內部包含了許多 controller(如 Node Controller、Replication Controller、Garbage Collector 等),負責維持叢集的「期望狀態」。

二、參數解析

2.1 `node-cidr-mask-size: "25"`

字面意義

設定「每個 Node 分配到的 Pod CIDR 子網路遮罩大小」為 /25。

運作原理

當您在叢集建立時設定了整個叢集的 Pod 網段(--cluster-cidr,例如 10.244.0.0/16),kube-controller-manager 會從這個大網段中,切出一小塊子網段分配給每個加入叢集的 Node,這塊子網段就是該 Node 上所有 Pod 的 IP 來源。

node-cidr-mask-size=25 代表每個 Node 會拿到一個 /25 的子網段。

實際影響(以 `/25` 為例)

每個 Node 可分配的 IP 數量:2^(32-25) = 128 個 IP
扣除網路位址、廣播位址等保留 IP,每個 Node 實際可跑約 126 個 Pod
若叢集的 cluster-cidr 是 /16(共 65,536 個 IP),理論上可切出 2^(25-16) = 512 個 Node 的配額

與預設值的差異

Kubernetes 預設值是 /24(每個 Node 可跑約 254 個 Pod)。設成 /25 代表:

✅ 節省 IP 位址:在有限的 cluster-cidr 下可容納更多 Node
⚠️ 限制單一 Node 的 Pod 密度:每個 Node 最多只能跑約 126 個 Pod(仍高於 kubelet 預設的 max-pods=110,所以實務上通常夠用)

什麼情境會這樣設定?

Pod IP 網段資源有限(例如 VPC 內可用的 CIDR 不大)
叢集 Node 數量多、但每個 Node 的 Pod 密度不高
希望在有限網段內擴充 Node 規模

2.2 `terminated-pod-gc-threshold: "500"`

字面意義

當叢集中「已終止(terminated)狀態的 Pod」數量超過 500 個時,Garbage Collector 會開始自動清理這些 Pod。

運作原理

當 Pod 執行完畢(Succeeded 或 Failed),它們不會立刻從 etcd 中刪除,而是會保留下來供使用者查詢記錄(例如 kubectl get pods 時看到 Completed 狀態的 Pod)。這些 Pod 雖然不再消耗運算資源,但仍會佔用:

etcd 儲存空間
API Server 的查詢效能

這個參數控制「保留多少個已終止 Pod 才開始清理」。

實際影響(設為 500)

當已終止 Pod 數量 ≤ 500:不會觸發清理,全部保留
當已終止 Pod 數量 > 500:從最舊的開始刪除,直到數量回到 500 以下

與預設值的差異

Kubernetes 預設值是 12500,設成 500 代表:

✅ 更積極地清理,減輕 etcd 負擔、加快 API Server 回應速度
✅ 適合有大量 Job / CronJob 產生短命 Pod 的叢集
⚠️ 可保留的歷史記錄變少,若需要排查較早的失敗 Pod 可能已被刪除(建議搭配 Log 收集系統,如 Loki、ELK)

什麼情境會這樣設定?

叢集會跑大量 Batch Job / CronJob
etcd 效能是瓶頸、或想降低 etcd 儲存壓力
已有完善的 log / event 外部保存機制,不需依賴 API Server 保留 Pod 記錄

2.3 兩個參數的整體設計意圖

關注點	對應參數	設計思路
IP 資源效率	`node-cidr-mask-size=25`	用比較小的 Pod CIDR 讓叢集能容納更多 Node
控制平面效能	`terminated-pod-gc-threshold=500`	積極清理終止 Pod,減輕 etcd / API Server 壓力

兩者都指向同一個方向:這是一個偏向中大型、但希望控制資源使用效率的叢集。

2.4 設定時的注意事項

node-cidr-mask-size 必須與 cluster-cidr、kubelet --max-pods 三者一起通盤考量,避免出現 Node 配額超過 IP 可用量,或 Pod 配額超過網段容量的狀況。
terminated-pod-gc-threshold 不能設太低(例如幾十個),否則剛跑完的 Job 馬上被清掉,會讓 debug 非常困難。500 是一個相對平衡的數字。
修改 controllerManager.extraArgs 後,需要重啟 kube-controller-manager Pod 才會生效(若是 kubeadm 叢集,修改 /etc/kubernetes/manifests/kube-controller-manager.yaml 會自動重啟)。

三、原理:PodGC Controller 如何認定「terminated Pod」

3.1 原始碼位置

Kubernetes 的 PodGC Controller 位於:

kubernetes/pkg/controller/podgc/gc_controller.go

3.2 原始碼層面的 terminated 定義

根據 Kubernetes 官方 pkg/controller/podgc 套件的 doc.go 描述:

PodGCController… If the number of pods in terminated phases (right now either Failed or Succeeded) surpasses a configurable threshold, the controller will delete pods in terminated state until the system reaches the allowed threshold again. The PodGCController prioritizes pods to delete by sorting by creation timestamp and deleting the oldest objects first.

這段話給了我們三個關鍵定義:

判定依據	說明
1. Pod 的 `status.phase`	必須是 `Failed` 或 `Succeeded`
2. 判斷時機	當 terminated Pod 總數 > threshold 才觸發清理
3. 刪除順序	依照 `creationTimestamp` 排序,最舊的先刪

在 gc_controller.go 中也能看到這個判定邏輯的核心程式碼片段:

if phase := pod.Status.Phase; phase != v1.PodPending &&
 phase != v1.PodRunning &&
 phase != v1.PodUnknown {
 // 這個 Pod 屬於 terminated,納入 GC 候選名單
}

這段程式碼使用反向列舉的方式定義 terminated:只要不是 Pending / Running / Unknown,就是 terminated。在 Kubernetes 的 Pod phase 定義中,剩下的就是 Succeeded 和 Failed 兩種。

3.3 用 kubectl 判斷 Pod 是否為 terminated

對應到 kubectl,可以用 status.phase 做 field selector 過濾:

# 查詢所有 Succeeded 的 Pod
kubectl get pods --all-namespaces --field-selector=status.phase=Succeeded

# 查詢所有 Failed 的 Pod
kubectl get pods --all-namespaces --field-selector=status.phase=Failed

# 同時查詢兩者(field-selector 不支援 OR,需要用 jq 輔助)
kubectl get pods --all-namespaces -o json | \
 jq '.items[] | select(.status.phase=="Succeeded" or .status.phase=="Failed") | .metadata.name'

# 快速統計 terminated Pod 數量
kubectl get pods --all-namespaces -o json | \
 jq '[.items[] | select(.status.phase=="Succeeded" or .status.phase=="Failed")] | length'

⚠️ 常見誤解澄清 kubectl get pods 看到的 Completed 不是 phase 值,而是 STATUS 欄位的顯示字串。真正的 phase 是 Succeeded。同理 Error 對應的是 Failed。驗證時一定要用 status.phase 當條件,不要用 STATUS 欄位的字串。

3.4 值得注意的原始碼細節

細節 1:GC 只會把超出的部分刪掉,不會歸零

根據 GitHub Issue #71421 的討論,PodGC 的設計行為是:

The current behavior seems to allow the terminated Pod amount fluctuate above and under the threshold, instead of clean up completely when threshold is met.

實例:假設 threshold = 500,當 terminated Pod 達到 600 時,GC 會刪掉 100 個(600 - 500),讓數量回到 500,而不是全部清光。這個設計是刻意的。

細節 2:threshold ≤ 0 代表停用 GC

when set terminated-pod-gc-threshold <= 0, the terminated pod garbage collector is disabled.

細節 3:GC 檢查週期

原始碼中有個常數 gcCheckPeriod,預設是 20 秒。也就是 PodGC Controller 每 20 秒會檢查一次是否需要清理。

四、驗證實驗:用 Kind 驗證 `terminated-pod-gc-threshold`

4.1 實驗設計

項目	設定值	設計目的
叢集	Kind 本機叢集	快速建置、可改 controller-manager 參數
threshold	500(接近真實情境)	驗證實際設定檔的值
Job 數量	600 個(超過 threshold 100 個)	驗證超量時的清理行為
觀察點 1	Job 全部完成瞬間的 terminated Pod 數量	應該是 600
觀察點 2	GC 週期後(等 30~60 秒)	應該被砍到 500 左右
觀察點 3	被保留的 Pod creationTimestamp	應該是較新的 500 個(舊的被刪)

4.2 完整實驗步驟

Step 1: 建立 Kind 叢集並掛入自訂的 controller-manager 參數

建立 kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: podgc-test
nodes:
 - role: control-plane
 kubeadmConfigPatches:
 - |
 kind: ClusterConfiguration
 controllerManager:
 extraArgs:
 terminated-pod-gc-threshold: "500"
 - role: worker
 - role: worker

建立叢集:

kind create cluster --config kind-config.yaml

Step 2: 驗證參數有正確套用到 kube-controller-manager

# 進入 control-plane 容器查看 manifest
docker exec podgc-test-control-plane \
 cat /etc/kubernetes/manifests/kube-controller-manager.yaml \
 | grep terminated-pod-gc-threshold

# 應該看到: --terminated-pod-gc-threshold=500

另一個驗證方式是看 controller-manager Pod 的 command:

kubectl -n kube-system get pod -l component=kube-controller-manager \
 -o jsonpath='{.items[0].spec.containers[0].command}' | tr ',' '\n' | grep terminated

Step 3: 建立 600 個會立即完成的 Job

準備腳本 create-jobs.sh:

#!/bin/bash
set -e

NAMESPACE="podgc-test"
kubectl create namespace $NAMESPACE --dry-run=client -o yaml | kubectl apply -f -

for i in $(seq 1 600); do
 cat <<EOF | kubectl apply -f - >/dev/null
apiVersion: batch/v1
kind: Job
metadata:
 name: quickjob-$(printf "%03d" $i)
 namespace: $NAMESPACE
spec:
 ttlSecondsAfterFinished: 99999 # 避免 Job controller 自己先清掉 Pod,干擾實驗
 backoffLimit: 0
 template:
 spec:
 restartPolicy: Never
 containers:
 - name: c
 image: busybox:1.36
 command: ["sh", "-c", "echo done"]
EOF
 # 每 50 個稍微喘一下,避免 API Server 過載
 if (( i % 50 == 0 )); then
 echo "Created $i jobs..."
 sleep 2
 fi
done

echo "All 600 jobs submitted."

⚠️ 關鍵設計:ttlSecondsAfterFinished 設得很大,是為了排除 Job Controller 的 TTL 機制(那是另一個清理機制),確保清理行為只由 PodGC 觸發,這樣實驗才乾淨。

執行:

chmod +x create-jobs.sh
./create-jobs.sh

Step 4: 持續觀察 terminated Pod 數量變化

開啟另一個終端機,跑一個監控腳本 watch-gc.sh:

#!/bin/bash
NAMESPACE="podgc-test"

while true; do
 TS=$(date '+%H:%M:%S')
 TOTAL=$(kubectl get pods -n $NAMESPACE -o json 2>/dev/null | \
 jq '[.items[] | select(.status.phase=="Succeeded" or .status.phase=="Failed")] | length')
 RUNNING=$(kubectl get pods -n $NAMESPACE -o json 2>/dev/null | \
 jq '[.items[] | select(.status.phase=="Running" or .status.phase=="Pending")] | length')
 echo "[$TS] terminated=$TOTAL active=$RUNNING"
 sleep 5
done

執行:

chmod +x watch-gc.sh
./watch-gc.sh

Step 5: 預期觀察結果

您會看到類似以下的輸出序列:

[14:30:05] terminated=0 active=120 # Job 開始跑
[14:30:10] terminated=45 active=200
[14:30:20] terminated=180 active=300
[14:30:30] terminated=420 active=180
[14:30:40] terminated=590 active=10 # 接近高峰
[14:30:45] terminated=600 active=0 # 全部完成
[14:30:50] terminated=600 active=0
[14:30:55] terminated=600 active=0
[14:31:05] terminated=500 active=0 # ← GC 觸發!砍掉 100 個
[14:31:10] terminated=500 active=0 # 穩定在 threshold

驗證點:

✅ 高峰值達到 600 → 證明 Pod 確實累積
✅ GC 觸發後剩 500(而非 0) → 驗證「只砍超出部分」的設計
✅ GC 觸發延遲 15~20 秒 → 對應原始碼中的 gcCheckPeriod = 20s

Step 6: 驗證「刪除順序是從最舊開始」

# 看留下來的 Pod 中,最早的 creationTimestamp 是什麼時候
kubectl get pods -n podgc-test \
 --field-selector=status.phase=Succeeded \
 --sort-by=.metadata.creationTimestamp \
 -o custom-columns=NAME:.metadata.name,CREATED:.metadata.creationTimestamp \
 | head -5

# 對照實驗:最早的 100 個 Pod(quickjob-001 ~ quickjob-100)應該都不見了
kubectl get pod quickjob-001 -n podgc-test 2>&1 | grep -i "not found"
kubectl get pod quickjob-600 -n podgc-test # 應該還在

如果 quickjob-001 找不到、quickjob-600 還在,就印證了原始碼中的排序刪除邏輯:最舊的優先刪除。

Step 7: 查看 controller-manager 的 log 做最後驗證

kubectl -n kube-system logs -l component=kube-controller-manager \
 --tail=200 | grep -i "podgc\|garbage"

會看到類似:

I0423 14:31:05.123456 podgc_controller.go:xxx] "PodGC is force deleting Pod" pod="podgc-test/quickjob-001"
I0423 14:31:05.234567 podgc_controller.go:xxx] "PodGC is force deleting Pod" pod="podgc-test/quickjob-002"
...

這直接證明 PodGC Controller 確實在依照您設定的 threshold 工作。

4.3 實驗清理

kind delete cluster --name podgc-test

五、整體驗證流程總結

原始碼層面定義 terminated
└─ status.phase ∈ {Succeeded, Failed}
│
▼
kubectl 指令層面驗證
└─ --field-selector=status.phase=Succeeded/Failed
│
▼
實驗層面驗證(Kind + 600 Jobs + threshold=500)
├─ 驗證點 1: 達到 600 → 證明 Pod 有累積
├─ 驗證點 2: 砍到剩 500 → 證明 threshold 精準生效
├─ 驗證點 3: 留下的是新的 → 證明 oldest-first 排序
└─ 驗證點 4: controller-manager log 有 GC 紀錄 → 直接證據

六、參考資料

類型	連結
PodGC 原始碼	https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podgc/gc_controller.go
PodGC 套件說明	https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/podgc/doc.go
PodGC 行為討論 Issue	https://github.com/kubernetes/kubernetes/issues/71421
Enhancement Proposal	https://github.com/kubernetes/enhancements/issues/2890
Pod Lifecycle 官方文件	https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/
kubeadm ClusterConfiguration	https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/
Kind 官方文件	https://kind.sigs.k8s.io/

本文件由 Claude 整理自對話紀錄,供學習與參考使用。

Cilium Gateway API

Mon, 20 Apr 2026 02:44:58 +0800

適用環境： RKE2 bare-metal / VM + 外部 F5 / Nginx L4 LB Envoy 模式： Standalone DaemonSet（Cilium 1.16+ 預設） 目標： 讓 Cilium Gateway API 透過 hostNetwork 直接在 worker node 上監聽 80/443，由外部 LB 分流

架構總覽

流量路徑說明：

外部 Client 發出請求
F5 / Nginx (L4 LB) 將流量分發到 3 台標記為 role=gateway 的 worker node IP
每台 worker node 上的 Cilium Envoy 以 hostNetwork 模式監聽 80/443
Cilium 的 eBPF 透過 TPROXY 機制將封包透明轉發給 Envoy
Gateway API 的 HTTPRoute 根據 host / path 規則路由到對應的後端 K8s Service

前置條件

Cilium 版本 >= 1.16（hostNetwork 模式自 1.16 起支援）
Gateway API CRDs 已安裝
kubeProxyReplacement 已啟用
至少 3 台 worker node 可供對外使用

Step 0：事前確認與備份

確認 Cilium 版本：

cilium version
helm list -n kube-system | grep cilium

匯出目前的 Helm values 備份（重要，萬一需要 rollback）：

helm get values cilium -n kube-system -o yaml > cilium-backup-values.yaml

Step 1：為 Worker Node 打上 Label

選擇要對外的 3 台 worker node，為它們打上 role=gateway label：

kubectl label node <worker-node-1> role=gateway
kubectl label node <worker-node-2> role=gateway
kubectl label node <worker-node-3> role=gateway

驗證 label 是否正確：

kubectl get nodes -l role=gateway

預期輸出：

NAME STATUS ROLES AGE VERSION
worker-node-1 Ready <none> 30d v1.xx.x
worker-node-2 Ready <none> 30d v1.xx.x
worker-node-3 Ready <none> 30d v1.xx.x

Step 2：準備 Helm Values 檔案

建立 cilium-gateway-hostnetwork-values.yaml：

# ── Gateway API 設定 ──────────────────────────────────────────
gatewayAPI:
 enabled: true
 hostNetwork:
 enabled: true
 # 只在有 role=gateway label 的 node 上暴露 listener
 # 如果要所有 node 都暴露，移除 nodes 區塊即可
 nodes:
 matchLabels:
 role: gateway

# ── Envoy (Standalone DaemonSet 模式) ────────────────────────
envoy:
 enabled: true
 securityContext:
 capabilities:
 # 允許 Envoy bind 80/443 等 privileged port
 keepCapNetBindService: true
 envoy:
 - NET_ADMIN
 - PERFMON
 - BPF
 - NET_BIND_SERVICE

# ── 基礎設定（通常已在現有 values 中，僅供確認）──────────────
kubeProxyReplacement: true

重點說明：

gatewayAPI.hostNetwork.enabled: true：啟用後會自動停用 LoadBalancer type Service，兩者互斥（Cilium 不再為 Gateway 建立 LoadBalancer type Service，改為直接在 host network 上監聽。）
gatewayAPI.hostNetwork.nodes.matchLabels：控制只在指定 node 上暴露 listener
NET_BIND_SERVICE capability：允許 Envoy bind 80/443 等 1024 以下的 privileged port

Step 3：執行 Helm Upgrade

helm upgrade cilium cilium/cilium \
 --namespace kube-system \
 --reuse-values \
 -f cilium-gateway-hostnetwork-values.yaml

注意： --reuse-values 會保留現有的所有設定，只覆蓋 values 檔案中指定的項目。

Step 4：重啟 Cilium 元件

Helm upgrade 後需重啟相關元件使設定生效：

kubectl -n kube-system rollout restart deployment/cilium-operator
kubectl -n kube-system rollout restart daemonset/cilium
kubectl -n kube-system rollout restart daemonset/cilium-envoy

等待所有 Pod 就緒：

kubectl -n kube-system rollout status deployment/cilium-operator --timeout=120s
kubectl -n kube-system rollout status daemonset/cilium --timeout=120s
kubectl -n kube-system rollout status daemonset/cilium-envoy --timeout=120s

確認 Cilium 整體狀態正常：

cilium status

Step 5：安裝 Gateway API CRDs

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_gatewayclasses.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_gateways.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_httproutes.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_referencegrants.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/v1.4.1/config/crd/standard/gateway.networking.k8s.io_grpcroutes.yaml

確認 GatewayClass 已存在：

kubectl get gatewayclass

執行結果:

NAME CONTROLLER ACCEPTED AGE
cilium io.cilium/gateway-controller True 3m52s

Step 6：部署 Demo App

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.11/samples/bookinfo/platform/kube/bookinfo.yaml

檢視 Pods

$ kubectl get pods

執行結果:

NAME READY STATUS RESTARTS AGE
details-v1-75c85494bb-b76bs 1/1 Running 0 36s
productpage-v1-7c9bf9c988-f7cpf 1/1 Running 0 36s
ratings-v1-6456c55c66-7fczr 1/1 Running 0 36s
reviews-v1-545b8ff7c8-4fdxk 1/1 Running 0 36s
reviews-v2-5f6c87ccdb-68ws8 1/1 Running 0 36s
reviews-v3-74fb78c454-2s6ng 1/1 Running 0 36s

Step 7：Deploy the Cilium Gateway

⚠️ 關於 spec.addresses 欄位： 由於 Issue #42786 的緣故，hostNetwork 模式下此欄位不會生效也不會被填入 Gateway status。以下 YAML 中保留此欄位僅作為「此 Gateway 實際對外的 node IP」的文件化註記，Cilium 並不會使用它來分配位址。

cat <<'EOF' | kubectl apply -f -
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: my-gateway
spec:
gatewayClassName: cilium
addresses:
- type: IPAddress
value: "192.168.11.173"
- type: IPAddress
value: "192.168.11.174"
- type: IPAddress
value: "192.168.11.175"
listeners:
- protocol: HTTP
port: 80
name: web-gw
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-app-1
spec:
parentRefs:
- name: my-gateway
namespace: default
rules:
- matches:
- path:
type: PathPrefix
value: /details
backendRefs:
- name: details
port: 9080
- matches:
- headers:
- type: Exact
name: magic
value: foo
queryParams:
- type: Exact
name: great
value: example
path:
type: PathPrefix
value: /
method: GET
backendRefs:
- name: productpage
port: 9080
EOF

Step 8：驗證端到端連通

⚠️ 驗證方法變更： 由於 Issue #42786，Gateway 層的 Programmed 條件會永遠是 False，因此不能用它來判定部署成功。請改用以下方法驗證。

8.1 檢查 Gateway 與 Listener 狀態（預期輸出已修正）

kubectl get gateway my-gateway
kubectl describe gateway my-gateway

預期輸出（受 Issue #42786 影響）：

NAME CLASS ADDRESS PROGRAMMED AGE
my-gateway cilium False 1m

在 kubectl describe gateway 的輸出中，請確認以下條件皆為 True（這些才是部署成功的實際指標）：

層級	Condition Type	預期 Status	預期 Reason
Gateway	`Accepted`	True	`Accepted`
Listener	`Accepted`	True	`Accepted`
Listener	`ResolvedRefs`	True	`ResolvedRefs`
Gateway	`Programmed`	False（Known Issue，忽略）	`AddressNotAssigned`
Listener	`Programmed`	False（Known Issue，忽略）	`Pending`

只要上表中 Accepted 與 ResolvedRefs 為 True，即代表 Gateway 設定被成功接受與解析，可以進入下一步實際連通驗證。

8.2 確認 Envoy 已在 Gateway node 上監聽

SSH 到任一 gateway node，確認 Envoy 有在監聽 80 port：

ss -tlnp | grep ':80'

應該可以看到 cilium-envoy 程序正在監聽 0.0.0.0:80。

8.3 從外部實測 HTTP 連通性（最重要的驗證）

從外部用 3 台 gateway node 的 IP 分別測試：

curl -H "Host: app.example.com" http://<worker-node-1-ip>/details/1
curl -H "Host: app.example.com" http://<worker-node-2-ip>/details/1
curl -H "Host: app.example.com" http://<worker-node-3-ip>/details/1

只要能成功取得 HTTP 回應，代表整個路徑是通的 —— Gateway status 顯示 Programmed: False 純粹是 Issue #42786 的顯示問題，不影響流量。

8.4 建議作法：用 listener 條件代替 gateway 條件作為自動化等待點

若有 CI/CD 需要「等待 Gateway 就緒」，請使用以下替代方式之一：

方式 A：等待 Listener Accepted（不受 Issue #42786 影響）

kubectl wait --for=jsonpath='{.status.listeners[0].conditions[?(@.type=="Accepted")].status}'=True \
 gateway/my-gateway --timeout=60s

方式 B：直接 curl 實測

until curl -sf -H "Host: app.example.com" http://<worker-node-1-ip>/details/1 > /dev/null; do
 sleep 2
done
echo "Gateway is serving traffic"

⚠️ 請勿使用：

# 這個會永遠 timeout，因為 Programmed 條件受 Issue #42786 影響
kubectl wait --for=condition=Programmed gateway/my-gateway --timeout=60s

F5 / Nginx LB 設定

在外部 L4 LB 上設定 pool members，指向 3 台 gateway node：

Protocol	Pool Members
HTTP	`<worker-node-1-ip>:80`, `<worker-node-2-ip>:80`, `<worker-node-3-ip>:80`
HTTPS	`<worker-node-1-ip>:443`, `<worker-node-2-ip>:443`, `<worker-node-3-ip>:443`

Health Check 建議：

TCP monitor（最簡單）：對 port 80 / 443 做 TCP 連線檢查
HTTP monitor（更精確）：HTTP GET /，回傳 404 也算健康（Envoy 在沒有匹配 route 時的預設行為）

💡 提示： F5/Nginx 的 pool member 設定不會從 Gateway status.addresses 讀取。所以 Issue #42786 不影響此設定流程，你仍然是手動指定 3 台 gateway node 的 IP。

注意事項

hostNetwork 與 LoadBalancer 互斥： 啟用 hostNetwork 後，Cilium 不會再為 Gateway 建立 LoadBalancer type Service
Port 衝突風險： 確保 80/443 在 gateway node 上沒有被其他程式佔用（例如原本的 Ingress-Nginx）
移除舊 Ingress-Nginx： 如果要完全切換，記得先停用 RKE2 內建的 Ingress-Nginx，避免 port 衝突
Gateway listener port 唯一性： 每個 Gateway 資源的 listener port 在所有 node 上必須唯一

Rollback 方式： 如需還原，使用備份的 values 重新 upgrade 即可：

helm upgrade cilium cilium/cilium \
 --namespace kube-system \
 -f cilium-backup-values.yaml

Gateway status 顯示問題（Issue #42786）： 在 hostNetwork 模式下 Gateway 永遠顯示 Programmed: False / AddressNotAssigned，此為已知的 upstream issue（截至 2026-04-20 仍為 Open 狀態，尚無關聯修復 PR）。此現象僅影響 status 顯示，不影響實際流量。驗證部署是否成功請以 Listener 的 Accepted/ResolvedRefs 條件與實際 curl 測試為準，詳見 Step 8。

⚠️ 已知問題（Known Issue）— 必讀

Upstream Issue： cilium/cilium#42786 — Gateway API with Host Network: Gateway stays PROGRAMMED: False 最後確認時間： 2026-04-20

症狀

kubectl get gateway 的 ADDRESS 欄空白、PROGRAMMED 欄顯示 False（Reason: AddressNotAssigned）。Listener 層的 Programmed 同樣為 False（Reason: Pending）。

Issue 現狀（截至 2026-04-20）

GitHub Issue 仍為 Open 狀態，無關聯修復 PR，Cilium v1.19.x 亦未修復。詳細追蹤資訊請見文末附錄。

影響範圍與根因

回報者 @dke 在 issue 中明確指出：

“Not a regression, but wanted to state that I can get a working gateway-api if I don’t use host network mode, but rather a setup with, for example, cilium and bgp based load balancers.”

也就是說：

這不是 regression（不是新版本把舊版本弄壞的問題）
這是 hostNetwork 模式特有的行為——一旦改用 Cilium + BGP LoadBalancer（有真正的 Service 可供 LB-IPAM 指派 IP）就會恢復正常
根因是 Cilium 的 spec.addresses 實作與 LB-IPAM 強耦合，而 hostNetwork 模式會停用 LoadBalancer Service，導致 status.addresses 沒有 IP 來源可以填入

此問題不影響的部分 ✅

實際流量 routing 完全正常：Envoy 仍會在 host 的 80/443 port 監聽，F5 → node IP → Envoy → backend 的路徑運作如預期
Listener 的 Accepted 與 ResolvedRefs 都會是 True：代表 Gateway / HTTPRoute 的設定本身是有效、可接受的
HTTPRoute 功能正常：path 比對、header 比對、backend 轉發都會生效

此問題會影響的部分 ❌

kubectl get gateway 的 ADDRESS 欄永遠是空白
Gateway 層的 Programmed 條件永遠是 False
不可使用 kubectl wait --for=condition=programmed gateway/<name> 作為 CI/CD 或部署自動化的等待條件

etcd 寫入一筆資料後，Leader 和 Follower 做了哪些事？

Mon, 13 Apr 2026 08:58:06 +0800

前置準備

1. 建立 tkha HA 叢集

本實驗需要一個 3 control-plane 的 etcd HA 叢集。如果還沒建立，先用 tk 工具建立：

~/tk/bin/kto tkha

kto 會建立叢集並初始化第一個 control-plane（tkha-control-plane），但不會自動加入另外兩個 control-plane。

2. 加入另外兩個 Control-plane 節點

確認三個 control-plane 是否都已加入叢集：

sudo podman exec tkha-control-plane etcdctl member list -w table

如果只有 1 個 member，需要手動加入另外兩個：

# 產生 certificate key 和 join 指令
CERT_KEY=$(sudo podman exec tkha-control-plane bash -c \
 'kubeadm init phase upload-certs --upload-certs' 2>/dev/null | tail -1 | tr -d '\r')

JOIN_CMD=$(sudo podman exec tkha-control-plane bash -c \
 'kubeadm token create --print-join-command' | tr -d '\r')

IGNORE='--ignore-preflight-errors=SystemVerification'

# 加入第二個 control-plane
sudo podman exec tkha-control-plane1 bash -c \
 "$JOIN_CMD --control-plane --certificate-key $CERT_KEY $IGNORE"

# 等待加入完成
sleep 30

# 加入第三個 control-plane
sudo podman exec tkha-control-plane2 bash -c \
 "$JOIN_CMD --control-plane --certificate-key $CERT_KEY $IGNORE"

# 等待加入完成
sleep 30

加入後再次確認，應該看到 3 個 member：

sudo podman exec tkha-control-plane etcdctl member list -w table

3. 安裝 host 上的工具

# tcpdump — 抓取網路封包
which tcpdump || sudo apk add tcpdump

# etcd-dump-logs — 解析 etcd WAL 檔案（需要 Go 環境）
which etcd-dump-logs || {
 go install go.etcd.io/etcd/tools/etcd-dump-logs@latest
 sudo cp $(go env GOPATH)/bin/etcd-dump-logs /usr/local/bin/
}

容器內的 etcdctl、crictl、sed 是 tkha 叢集自帶的，不需要額外安裝。

實驗環境

項目	值
叢集	tkha（3 control-plane HA）
etcd 版本	3.6.5
Leader	172.22.64.1（tkha-control-plane）
Follower	172.22.64.2、172.22.64.3
Heartbeat 間隔	250ms
Election Timeout	2500ms

先搞懂這些術語

Leader / Follower
- 原文：At any given time, one server is the leader and the others are followers. The leader handles all client requests.
- 白話：老大和跟班。叢集中只有一個老大（Leader），其他都是跟班（Follower）。所有寫入都必須先經過老大，由老大負責分發給跟班。
AppendEntries
- 原文：AppendEntries RPCs are initiated by the leader to replicate log entries and to provide a form of heartbeat.
- 白話：老大把「要做的事」抄送給跟班的動作。像是老大寫了一張便條紙，影印後發給每個跟班說：「照著做，做完回報。」
WAL（Write Ahead Log）
- 原文：The write ahead log is a persistent ordered record of every change made to etcd’s data.
- 白話：「先寫日記再做事」的日記本。不管是老大還是跟班，收到一筆操作後都要先寫進這本日記。這樣即使突然當機，重開機後看日記就能恢復。
fsync
- 原文：fsync() transfers all modified in-core data of the file to the disk device, ensuring that all changed data is physically written to disk.
- 白話：把資料真正刷到磁碟上。一般寫檔案只是寫到記憶體的快取，斷電就沒了。fsync 強制把快取刷到磁碟，確保斷電也不丟。這是整個流程中最花時間的地方之一。
ACK（Acknowledgement）
- 原文：A response from a follower indicating that it has successfully written the entry to its log.
- 白話：跟班回報「收到，我已經記在日記裡了」。老大要等到夠多跟班回報 ACK 才算寫入成功。
Quorum
- 原文：The minimum number of members that must agree for a change to be committed. For a cluster of N members, quorum is (N/2)+1.
- 白話：多數決。超過半數的節點都寫好日記了，才算數。3 個節點需要 2 個同意（老大自己算 1 個，只要再有 1 個跟班 ACK 就夠了）。
Commit
- 原文：A log entry is committed once the leader that created the entry has replicated it on a majority of the servers.
- 白話：老大確認多數決通過後，把這筆操作正式生效——寫入真正的資料庫（BoltDB），而不只是日記。
RAFT INDEX
- 原文：Each log entry stores a state machine command along with the term number and log index. The log index is an integer that identifies its position in the log.
- 白話：每筆操作的全域流水號。老大和跟班共用同一組編號，同一個 index 在所有節點上代表的是同一筆操作。
RAFT TERM
- 原文：Each term begins with an election. Terms act as a logical clock in Raft and allow servers to detect obsolete information.
- 白話：老大的「任期編號」。每次選出新老大，term 就加 1。可以把它想成「第幾屆」。

結果：完整寫入流程

當 Client 執行 etcdctl put /hello world 時，etcd 內部會經過以下 7 個步驟，Client 才會收到 “OK”：

步驟	誰做的	做了什麼	影響因素
(1)	Client	發送 put 請求給 Leader（port 2379）	—
(2)	Leader	寫入自己的 WAL，fsync 到磁碟	:floppy_disk: 磁碟延遲
(3)	Leader	發送 AppendEntries 給 Follower（port 2380）	:globe_with_meridians: 網路延遲
(4)	Follower	寫入自己的 WAL，fsync 到磁碟	:floppy_disk: 磁碟延遲
(5)	Follower	回覆 ACK 給 Leader	:globe_with_meridians: 網路延遲
(6)	Leader	確認 Quorum（2/3），commit，回覆 Client “OK”	:floppy_disk: 磁碟延遲
(7)	Leader	非同步通知 Follower commit（Client 已收到 OK）	—

時序圖：

 Client Leader (172.22.64.1) Follower (172.22.64.2)
| | |
| (1) put /hello world | |
| --------------------------> | |
| (port 2379) | |
| | |
| (2) Leader 寫入自己的 WAL |
| (fsync 到磁碟) |
| | |
| | (3) AppendEntries |
| | ----------------------------> |
| | (port 2380, 639 bytes) |
| | |
| | (4) Follower 寫入 WAL
| | (fsync 到磁碟)
| | |
| | (5) ACK |
| | <---------------------------- |
| | (port 2380, 75 bytes) |
| | (間隔 2.040ms) |
| | |
| (6) Quorum 達成 (2/3) |
| Leader commit |
| | |
| "OK" | |
| <-------------------------- | |
| | |
| | (7) 通知 commit（非同步） |
| | ----------------------------> |
| | Follower commit
| | |

Client 收到 “OK” 的時間點是在步驟 (6)，不是 (7)。Follower commit 是非同步的，不影響 Client 感受到的延遲。

以下用 5 個實驗，一步一步驗證上述流程是對的。

驗證 1：辨識誰是 Leader

對應步驟 (1)：Client 要把 put 請求送給 Leader，所以我們先確認誰是 Leader。

執行命令：

sudo podman exec tkha-control-plane etcdctl \
 --endpoints=https://172.22.64.1:2379,https://172.22.64.2:2379,https://172.22.64.3:2379 \
 endpoint status -w table

執行結果（節錄關鍵欄位）：

+--------------------------+------------------+---------+-----------+-----------+
| ENDPOINT | ID | VERSION | IS LEADER | RAFT TERM |
+--------------------------+------------------+---------+-----------+-----------+
| https://172.22.64.1:2379 | f49dafb8a84b142e | 3.6.5 | true | 2 |
| https://172.22.64.2:2379 | bc789b7376e525b5 | 3.6.5 | false | 2 |
| https://172.22.64.3:2379 | b8399ab2942e1696 | 3.6.5 | false | 2 |
+--------------------------+------------------+---------+-----------+-----------+

實際輸出會包含更多欄位（DB SIZE、QUOTA 等），以上只列出關鍵欄位。

這證明了什麼？

172.22.64.1 是 Leader（IS LEADER = true）
三個節點的 RAFT TERM 都是 2，代表目前是「第 2 屆」Leader

驗證 2：開啟 Follower 的 debug log

為了在後面的驗證中看到 Follower 收到 AppendEntries 的日誌，需要先開啟 debug log。這個步驟必須在寫入資料之前完成。

執行命令：

sudo podman exec tkha-control-plane1 bash -c '
 sed -i "/- etcd$/a\\ - --log-level=debug" /etc/kubernetes/manifests/etcd.yaml
'

kubelet 偵測到 manifest 變更，會自動重啟 Follower 的 etcd pod。等約 1~2 分鐘後確認叢集恢復健康（如果還是 unhealthy，再等 30 秒重試）：

sudo podman exec tkha-control-plane etcdctl \
 --endpoints=https://172.22.64.1:2379,https://172.22.64.2:2379,https://172.22.64.3:2379 \
 endpoint health -w table

執行結果（三個 endpoint 都要是 healthy 才能繼續）：

+--------------------------+--------+
| ENDPOINT | HEALTH |
+--------------------------+--------+
| https://172.22.64.1:2379 | true |
| https://172.22.64.2:2379 | true |
| https://172.22.64.3:2379 | true |
+--------------------------+--------+

如果 172.22.64.2 還是 unhealthy，每隔 30 秒重試一次，直到三個都是 true。

驗證 3：tcpdump 觀察 Leader 發封包給 Follower

對應步驟 (3) 和 (5)：Leader 透過 port 2380 發送 AppendEntries 給 Follower，Follower 回覆 ACK。

準備：在 host 上啟動 tcpdump

執行命令（Terminal 1）：

sudo tcpdump -i any host 172.22.64.1 and host 172.22.64.2 and port 2380 -nn -ttt -q -c 100

觸發：寫入一筆資料

執行命令（Terminal 2）：

sudo podman exec tkha-control-plane etcdctl put /hello "world"

執行結果：

OK

tcpdump 抓到的封包

執行結果（Terminal 1，節錄重點）：

Heartbeat 封包（背景流量，大小固定 ~84-85 bytes）：

00:00:00.000000 IP 172.22.64.1.2380 > 172.22.64.2.43206: tcp 85
00:00:00.000034 IP 172.22.64.2.43206 > 172.22.64.1.2380: tcp 0
00:00:00.000157 IP 172.22.64.2.2380 > 172.22.64.1.58336: tcp 84
00:00:00.000027 IP 172.22.64.1.58336 > 172.22.64.2.2380: tcp 0

資料寫入觸發的封包：

00:00:00.005596 IP 172.22.64.1.2380 > 172.22.64.2.43192: tcp 639 <-- AppendEntries
00:00:00.000034 IP 172.22.64.2.43192 > 172.22.64.1.2380: tcp 0 <-- TCP ACK
00:00:00.002040 IP 172.22.64.2.2380 > 172.22.64.1.58336: tcp 75 <-- Follower ACK

實際的 port 號和封包大小會略有不同，重點是觀察 Leader→Follower 出現一個比 heartbeat 大很多的封包，以及 Follower→Leader 的回覆。

這證明了什麼？

步驟 (3) 已驗證 — Leader (172.22.64.1) 發了一個 639 bytes 的封包給 Follower (172.22.64.2)，比 heartbeat 的 85 bytes 大很多。這就是包含 /hello = world 的 AppendEntries。

步驟 (5) 已驗證 — Follower 在 2.040ms 後回覆了一個 75 bytes 的小封包（ACK），比 request 小很多，只是在說「我記好了」。2.040ms 的間隔就是 Follower 處理 + 寫入 WAL (fsync) 的時間。

驗證 4：debug log 觀察 Follower 收到資料

對應步驟 (4)：Follower 收到 AppendEntries 後，寫入自己的 WAL。

執行命令：

sudo podman exec tkha-control-plane1 bash -c '
 crictl logs $(crictl ps -q --name etcd) 2>&1 | grep "hello"
'

執行結果：

{"level":"debug","ts":"2026-04-13T08:17:16.219400Z",
 "caller":"etcdserver/server.go:1971",
 "msg":"applyEntryNormal",
 "raftReq":"header:<ID:1454272925698278610 username:\"kube-apiserver-etcd-client\" auth_revision:1 > put:<key:\"/hello\" value:\"world\" > "}

實際的 timestamp 和 ID 會不同，重點是看到 applyEntryNormal 和 put:<key:"/hello" value:"world">。

這證明了什麼？

步驟 (4) 已驗證 — Follower 的 applyEntryNormal 日誌明確顯示收到了 put:<key:"/hello" value:"world">。這就是 Leader 透過 AppendEntries 送過來的資料，Follower 寫入 WAL 後 apply 到狀態機。

驗證 5：WAL dump 比對 Leader 和 Follower

對應步驟 (2) 和 (4)：Leader 和 Follower 都會把操作寫入 WAL，兩邊的 WAL 應該包含相同的 entry。

寫入測試資料

執行命令：

sudo podman exec tkha-control-plane etcdctl put /wal-test "raft-replicated-data"

複製 WAL 到 host 並用 etcd-dump-logs 解析

執行命令：

# 複製 WAL
sudo podman cp tkha-control-plane:/var/lib/etcd/member /tmp/leader-member
sudo podman cp tkha-control-plane1:/var/lib/etcd/member /tmp/follower-member

# 建立 etcd-dump-logs 需要的目錄結構
sudo mkdir -p /tmp/leader-data/member /tmp/follower-data/member
sudo ln -sf /tmp/leader-member/wal /tmp/leader-data/member/wal
sudo ln -sf /tmp/leader-member/snap /tmp/leader-data/member/snap
sudo ln -sf /tmp/follower-member/wal /tmp/follower-data/member/wal
sudo ln -sf /tmp/follower-member/snap /tmp/follower-data/member/snap

# Dump Leader WAL
sudo etcd-dump-logs /tmp/leader-data | grep "wal-test"

# Dump Follower WAL
sudo etcd-dump-logs /tmp/follower-data | grep "wal-test"

執行結果：

Leader WAL：

 2 41857 norm header:<ID:1454272925698278720 username:"kube-apiserver-etcd-client" auth_revision:1 > put:<key:"/wal-test" value:"raft-replicated-data" >

Follower WAL：

 2 41857 norm header:<ID:1454272925698278720 username:"kube-apiserver-etcd-client" auth_revision:1 > put:<key:"/wal-test" value:"raft-replicated-data" >

實際的 index 和 ID 會不同，重點是兩邊的 term、index、data 完全一致。

這證明了什麼？

步驟 (2) 和 (4) 已驗證 — Leader 和 Follower 的 WAL 中都有完全一致的 entry。term、index、data 一模一樣，證明 Leader 複製給 Follower 的是原封不動的 Raft entry。

驗證 6：從 Follower 讀取確認 commit

對應步驟 (7)：Leader 非同步通知 Follower commit 後，Follower 應該也能讀到這筆資料。

執行命令：

sudo podman exec tkha-control-plane etcdctl --endpoints=https://172.22.64.2:2379 get /hello

執行結果：

/hello
world

這證明了什麼？

步驟 (7) 已驗證 — 從 Follower（172.22.64.2）的 endpoint 直接讀取，確認 /hello = world 已經 commit 並 apply 到 Follower 的 BoltDB。整個 Raft 寫入流程走完了。

清理

# 刪除測試 key
sudo podman exec tkha-control-plane etcdctl del /hello
sudo podman exec tkha-control-plane etcdctl del /wal-test

# 還原 Follower 的 log level
sudo podman exec tkha-control-plane1 bash -c \
 'sed -i "/--log-level=debug/d" /etc/kubernetes/manifests/etcd.yaml'

# 清理 host 上的暫存檔
sudo rm -rf /tmp/leader-member /tmp/follower-member /tmp/leader-data /tmp/follower-data

驗證結果總表

步驟	理論描述	驗證編號	驗證方式	結果
(1)	Client 發送 put 請求	驗證 3	etcdctl put	:white_check_mark:
(2)	Leader 寫入 WAL	驗證 5	WAL dump	:white_check_mark:
(3)	Leader 發送 AppendEntries	驗證 3	tcpdump: 639B 封包	:white_check_mark:
(4)	Follower 寫入 WAL	驗證 4+5	debug log + WAL dump	:white_check_mark:
(5)	Follower 回覆 ACK	驗證 3	tcpdump: 75B 封包, 2.040ms	:white_check_mark:
(6)	Leader commit	驗證 3	Client 收到 “OK”	:white_check_mark:
(7)	Follower commit	驗證 6	從 Follower 讀到 `world`	:white_check_mark:

寫入延遲的組成

Client 感受到的延遲 = (2) Leader fsync + (3) 網路傳輸 + (4) Follower fsync + (5) 網路回覆 + (6) Leader commit
實測：(3) -> (5) 的時間差 = 2.040ms（包含網路 + Follower fsync）

因素	影響的步驟	說明
:floppy_disk: 磁碟延遲（fsync）	(2)、(4)、(6)	每個步驟各需要一次 fsync，把資料刷到磁碟
:globe_with_meridians: 網路延遲（peer 通訊）	(3)、(5)	Leader 和 Follower 之間各需要一次單程傳輸

Logging Operator

Tue, 24 Mar 2026 10:12:54 +0800

Preface

本篇文章會先介紹什麼是 Logging Operator，並透過 Rancher Logging 實作將 Pod Log 導入 OpenSearch

簡介

Logging Operator 是一套專為 Kubernetes 環境設計的日誌管理工具。它能夠自動部署與設定完整的日誌收集流程（logging pipeline），讓你不必手動處理繁瑣的日誌基礎設施配置。

運作流程

Logging Operator 的日誌處理分為三個階段：

1. 日誌收集（Log Collector）

Operator 會在 Kubernetes 叢集中的每個節點上部署一個 Fluent Bit DaemonSet，負責從節點的檔案系統中收集 App Containers 和應用程式的日誌。

2. 中繼資料 enrichment（Metadata Enrichment）

Fluent Bit 會查詢 Kubernetes API，為日誌附加 Pod 的相關中繼資料（metadata），例如 Pod 名稱、namespace、標籤等。接著，將日誌連同這些中繼資料一併傳送給日誌轉發器（Log Forwarder）。

3. 日誌轉發（Log Forwarding）

日誌轉發器負責接收、過濾與轉換傳入的日誌，並將處理完的結果傳送到一個或多個目的地輸出（destination outputs）。

目前支援的日誌轉發器有兩種：

轉發器	說明
Fluentd	廣泛使用的開源日誌轉發工具
syslog-ng（透過 AxoSyslog 發行版）	高效能的系統日誌處理引擎

安全性

所有日誌的傳輸皆透過經過驗證（authenticated）且加密（encrypted）的通道進行，確保資料在傳輸過程中的安全性。

核心優勢

Logging Operator 最大的特色在於：你可以將日誌的行為描述直接定義在應用程式的 Helm Chart 中，Operator 會自動完成其餘的部署與配置工作。

換句話說，日誌設定可以隨著應用程式一起版本控制與部署，實現基礎設施即程式碼（Infrastructure as Code）的理念。

功能特色

Logging Operator 提供以下核心功能：

Namespace 隔離：日誌流（flow）與輸出（output）可以限定在特定的 namespace 中運作，實現多租戶隔離。
原生 Kubernetes Label Selector：直接使用 Kubernetes 原生的標籤選擇器來篩選要收集的日誌來源。
安全通訊（TLS）：所有日誌傳輸皆支援 TLS 加密，確保資料安全。
設定檔驗證：自動驗證設定是否正確，降低配置錯誤的風險。
多重 Flow 支援：同一份日誌可以經由多條 flow 進行不同的過濾與轉換處理。
多重 Output 支援：同一份日誌可以同時傳送到多個儲存目的地，例如 S3、GCS、Elasticsearch、Loki 等。
多重 Logging 系統支援：在同一個叢集中可以部署多套 Fluentd 與 Fluent Bit 實例，滿足不同的日誌需求。
雙轉發器引擎：同時支援 syslog-ng 與 Fluentd 作為核心日誌路由元件。

架構概覽

Logging Operator 管理日誌基礎設施中的三個核心角色：日誌收集器（Log Collector）、日誌轉發器（Log Forwarder），以及定義日誌該送往何處的路由規則。

日誌收集器（Log Collector）

部署在每個 Kubernetes 節點上的端點代理程式（endpoint agent）。
負責收集該節點上所有 App Container 的日誌，並傳送給日誌轉發器。
目前使用 Fluent Bit 作為日誌收集器。

日誌轉發器（Log Forwarder / Log Aggregator）

接收來自收集器的日誌後，進行過濾（filter）與轉換（transform）。
將處理後的日誌傳送到一個或多個目的地輸出。
支援兩種轉發器引擎：

轉發器	適用場景
Fluentd	生態系成熟、外掛豐富，適合大多數使用情境
syslog-ng	高效能、低資源消耗，適合需要進階日誌處理邏輯的場景

如何選擇適合的轉發器，可參考官方文件 Which log forwarder to use。

路由規則（Flow & Output）

你可以透過 Flow 自訂資源來過濾、處理日誌訊息，並將它們路由到適當的 Output（目的地），例如 Elasticsearch 或 Amazon S3。

除了 namespace 層級的 flow 與 output，你也可以定義叢集層級的 ClusterFlow 與 ClusterOutput，讓所有 namespace 的使用者都能引用同一組集中式輸出，但無法修改其設定。

注意： Flow 與 Output 資源是依據你選用的轉發器類型（Fluentd 或 syslog-ng）而不同的，兩者的 CRD 不能混用。

Custom Resource Definitions（CRD）一覽

共用資源

CRD	說明	作用範圍
logging	定義叢集的日誌基礎設施，包含日誌收集器與轉發器的設定。也可包含 Fluent Bit、Fluentd、syslog-ng 的配置。自 4.5 版起，這些元件也可作為獨立資源分開設定。	叢集層級

Fluentd 專用 CRD

CRD	說明	作用範圍
output	定義 Fluentd 的日誌輸出目的地。	Namespace
flow	定義 Fluentd 的日誌路由規則，使用 filter 與 output 將選定的日誌導向指定目的地。	Namespace
clusteroutput	定義可供所有 flow 與 clusterflow 使用的 Fluentd 輸出。預設僅在 `controlNamespace` 中生效，除非設定 `allowClusterResourcesFromAllNamespaces: true`。	叢集層級
clusterflow	定義可從所有 namespace 收集日誌的 Fluentd 路由規則。預設僅在 `controlNamespace` 中生效，除非設定 `allowClusterResourcesFromAllNamespaces: true`。	叢集層級

syslog-ng 專用 CRD

CRD	說明	作用範圍
SyslogNGOutput	定義 syslog-ng 的日誌輸出目的地。	Namespace
SyslogNGFlow	定義 syslog-ng 的日誌路由規則，使用 filter 與 output 將選定的日誌導向指定目的地。	Namespace
SyslogNGClusterOutput	定義可供所有 flow 與 clusterflow 使用的 syslog-ng 輸出。預設僅在 `controlNamespace` 中生效，除非設定 `allowClusterResourcesFromAllNamespaces: true`。	叢集層級
SyslogNGClusterFlow	定義可從所有 namespace 收集日誌的 syslog-ng 路由規則。預設僅在 `controlNamespace` 中生效，除非設定 `allowClusterResourcesFromAllNamespaces: true`。	叢集層級

Fluentd 與 syslog-ng 的 CRD 結構相似，但各自針對自身引擎的特性進行了調整。選用哪一組 CRD 取決於你在 logging 資源中選擇的轉發器類型。

Rancher Logging — 將 Pod Log 導入 OpenSearch 實作指南

環境概述

Rancher 的日誌管理由 Logging operator 提供支持。以下實作將透過 Rancher logging，將指定 Kubernetes Pod 的 stdout log 經由 Fluentd 轉送至外部 OpenSearch。

環境準備

Rancher 安裝 DownStream RKE2 Cluster
DownStream RKE2 Cluster 已安裝 Rancher Logging
在 DownStream RKE2 Cluster 外部已安裝好 opensearch

步驟一：建立 Namespace

kubectl create ns app

步驟二：部署測試用 Pod

建立一個每秒輸出一行遞增計數 log 的 Pod，用以驗證 log pipeline 是否正常運作。

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
 name: test-logging-pod
 namespace: app
 labels:
 logging-test: "true"
spec:
 automountServiceAccountToken: false
 securityContext:
 runAsNonRoot: true
 runAsUser: 1000
 runAsGroup: 1000
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: sak
 image: rancherlabs/swiss-army-knife:latest
 command: ["/bin/sh"]
 args:
 - -c
 - |
 i=0; while true; do echo "$(date) INFO $i"; i=$((i+1)); sleep 1; done
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop:
 - ALL
EOF

確認 Pod 已正常啟動：

kubectl -n app get pods -l logging-test=true

預期輸出：

NAME READY STATUS RESTARTS AGE
test-logging-pod 1/1 Running 0 5s

確認 log 輸出正常：

kubectl -n app logs test-logging-pod

預期輸出（每秒遞增一行）：

Mon Mar 16 07:43:57 UTC 2026 INFO 0
Mon Mar 16 07:43:58 UTC 2026 INFO 1
Mon Mar 16 07:43:59 UTC 2026 INFO 2
...

步驟三：建立 OpenSearch 認證 Secret

將 OpenSearch 的帳號密碼以 kubernetes.io/basic-auth 類型的 Secret 儲存於叢集中，供後續 Output CR 引用。

PW='Test12345@'

kubectl create secret generic opensearch-user-auth \
 --namespace=app \
 --type=kubernetes.io/basic-auth \
 --from-literal=username=admin \
 --from-literal=password=$PW

步驟四：建立 Logging Output CR

定義 Fluentd 將 log 發送至 OpenSearch 的目標端點與 buffer 策略。

cat <<'EOF' | kubectl apply -f -
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
 name: opensearch-admin
 namespace: app
spec:
 opensearch:
 host: opensearch-node.kubeantony.com
 port: 9200
 scheme: https
 ssl_verify: false
 ssl_version: TLSv1_2
 suppress_type_name: true
 user: admin
 password:
 valueFrom:
 secretKeyRef:
 key: password
 name: opensearch-user-auth
 buffer:
 flush_interval: 10s
 flush_mode: interval
 retry_type: exponential_backoff
 retry_forever: true
 chunk_limit_size: 8MB
 total_limit_size: 512MB
EOF

Buffer 參數說明

參數	值	說明
`flush_mode`	`interval`	以固定時間間隔觸發 flush，在即時性與效能間取得平衡
`flush_interval`	`10s`	每 10 秒將累積的 log 批次送出
`chunk_limit_size`	`8MB`	單一 chunk 上限；達到後即使未到 flush 時間也會送出，防止記憶體暴漲
`total_limit_size`	`512MB`	所有 buffer 佔用的磁碟空間上限；超過後新 log 將依 `overflow_action` 處理
`retry_type`	`exponential_backoff`	發送失敗時以指數退避策略重試（1s → 2s → 4s → …），避免對 OpenSearch 造成衝擊
`retry_forever`	`true`	無限重試直到成功，確保 log 不遺失（代價是 buffer 可能在長時間故障時被填滿）

確認 Output 資源已建立（此時 ACTIVE 為 false 是正常的，表示尚未被任何 Flow 引用）：

kubectl -n app get outputs.logging.banzaicloud.io

預期輸出：

NAME ACTIVE PROBLEMS
opensearch-admin false

步驟五：建立 Logging Flow CR

建立 Flow 將帶有 logging-test: "true" label 的 Pod log 路由至上一步定義的 Output。

cat <<'EOF' | kubectl apply -f -
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
 name: app-to-opensearch
 namespace: app
spec:
 match:
 - select:
 labels:
 logging-test: "true"
 localOutputRefs:
 - opensearch-admin
EOF

確認 Flow 已啟用：

kubectl -n app get flows.logging.banzaicloud.io

預期輸出：

NAME ACTIVE PROBLEMS
app-to-opensearch true

步驟六：於 OpenSearch 驗證 Log 寫入

確認索引已建立

於 OpenSearch Dashboards 左上角選單進入 Index Management → Indexes，搜尋 fluentd，應可看到對應索引已自動建立。

查詢最新 Log

於 Dev Tools 執行以下查詢，取得最近 3 筆 log：

GET fluentd/_search
{
 "size": 3,
 "sort": [{ "time": { "order": "desc" }}],
 "query": {
 "match_all": {}
 }
}

回傳結果中，每筆文件的 _source 應包含 message（原始 log 訊息）、kubernetes（Pod metadata）、以及 time（時間戳記）等欄位，代表 log pipeline 已完整串通。

以下為其中一筆 log

{
 "took": 1000,
 "timed_out": false,
 "_shards": {
 "total": 1,
 "successful": 1,
 "skipped": 0,
 "failed": 0
 },
 "hits": {
 "total": {
 "value": 10000,
 "relation": "gte"
 },
 "max_score": null,
 "hits": [
 {
 "_index": "fluentd",
 "_id": "q0239pwBR50SWBx4toac",
 "_score": null,
 "_source": {
 "time": "2026-03-16T20:55:59.312256038+08:00",
 "stream": "stdout",
 "logtag": "F",
 "message": "Mon Mar 16 12:55:59 UTC 2026 INFO 9961",
 "kubernetes": {
 "pod_name": "test-logging-pod",
 "namespace_name": "app",
 "pod_id": "50148584-185c-475c-b203-4120526c755d",
 "labels": {
 "logging-test": "true"
 },
 "annotations": {
 "kubectl.kubernetes.io/last-applied-configuration": """{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"logging-test":"true"},"name":"test-logging-pod","namespace":"app"},"spec":{"automountServiceAccountToken":false,"containers":[{"args":["-c","i=0; while true; do echo \"$(date) INFO $i\"; i=$((i+1)); sleep 1; done\n"],"command":["/bin/sh"],"image":"rancherlabs/swiss-army-knife:latest","name":"sak","securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true}}],"securityContext":{"runAsGroup":1000,"runAsNonRoot":true,"runAsUser":1000,"seccompProfile":{"type":"RuntimeDefault"}}}}
"""
 },
 "host": "rke2-w2",
 "container_name": "sak",
 "docker_id": "d820db7b9416cde0b968f2e2fa314e74d656a064def3ecc183b7768bef65102a",
 "container_hash": "docker.io/rancherlabs/swiss-army-knife@sha256:af25a3ace6269adb9e494b693644bc2f897ec872076d78f78bc5ded69f2ee222",
 "container_image": "docker.io/rancherlabs/swiss-army-knife:latest"
 },
 "kubernetes_namespace": {
 "name": "app",
 "labels": {
 "field.cattle.io/projectId": "p-pzhk7",
 "kubernetes.io/metadata.name": "app"
 }
 }
 },
 "sort": [
 1773665759312
 ]
 }
 ]
 }
}

log 架構說明

以下逐層說明這個 OpenSearch 查詢回應的結構：

第一層：查詢執行資訊

took 表示 OpenSearch 花了 1000 毫秒完成這次查詢。timed_out: false 代表查詢在設定的超時限制內完成。_shards 記錄了參與查詢的分片狀態，這裡只有 1 個分片且執行成功、無失敗。

第二層：結果集合 hits

hits.total.value 為 10000，搭配 relation: "gte" 表示實際命中數量「大於或等於」10,000 筆，這是 OpenSearch 預設的計數上限（track_total_hits 未啟用精確計數時的行為）。max_score 為 null 是因為查詢使用了 sort 排序，此時相關性分數不會被計算。

第三層：單筆文件 hits.hits[]

每筆文件包含 _index（寫入的索引名稱，這裡是 fluentd）、_id（OpenSearch 自動產生的文件 ID）、以及 sort（排序欄位的原始值，此處為 time 的毫秒級 epoch）。

第四層：_source — Fluentd 寫入的原始資料

這是最核心的部分，由 Fluentd 的 Kubernetes metadata filter 自動附加：

time：log 產生的時間戳記（ISO 8601 格式，含時區）
message：容器的原始 stdout 輸出內容
stream：輸出流類型，stdout 或 stderr
logtag：CRI log 的完整性標記，F 代表完整行（Full），P 代表部分行（Partial，尚有後續）
kubernetes：Pod 層級的元資料，包含 pod 名稱、namespace、pod ID、labels、annotations、所在節點（host）、容器名稱與 image 資訊
kubernetes_namespace：Namespace 層級的元資料，包含名稱及其 labels（如 Rancher 的 projectId）

Rancher Logging — 將 Pod Log 導入 OpenSearch 實作指南 2

環境概述

以下實作將透過 Rancher logging，將指定 Kubernetes namespace 底下所有 Pods 的 log 經由 Fluentd 轉送至外部 logstash 再轉送到 opensearch。

環境準備

Rancher 安裝 DownStream RKE2 Cluster
DownStream RKE2 Cluster 已安裝 Rancher Logging
在 DownStream RKE2 Cluster 外部已安裝好 logstash
在 DownStream RKE2 Cluster 外部已安裝好 opensearch

開始實作

# Secret: CA 憑證
kubectl -n kube-system create secret generic logstash-ca \
--from-file=ca.crt=/etc/logstash/certs/ca.crt
# Secret: basic auth 帳密
kubectl -n kube-system create secret generic logstash-auth \
--from-literal=username=logstash \
--from-literal=password=logstash
cat <<'EOF' | kubectl apply -f -
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
name: logstash-http
namespace: kube-system
spec:
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
name: logstash-http
namespace: kube-system
spec:
http:
endpoint: "https://logstash.kubeantony.com:8080"
content_type: "application/json"
json_array: true
auth:
username:
valueFrom:
secretKeyRef:
name: logstash-auth
key: username
password:
valueFrom:
secretKeyRef:
name: logstash-auth
key: password
tls_ca_cert_path:
mountFrom:
secretKeyRef:
name: logstash-ca
key: ca.crt
buffer:
type: file
flush_interval: 5s
flush_mode: interval
timekey: 1m
timekey_wait: 10s
total_limit_size: 1GB
EOF
cat <<'EOF' | kubectl apply -f -
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: kube-logs
namespace: kube-system
spec:
match:
- select: {}
localOutputRefs:
- logstash-http
EOF

確認索引已建立

於 OpenSearch Dashboards 左上角選單進入 Index Management → Indexes，搜尋 fluentd，應可看到對應索引已自動建立。

查詢最新 Log

於 Dev Tools 執行以下查詢，對日誌索引執行無條件檢索，並依時間戳（@timestamp）降序排列以提取最新的一筆數據紀錄：

GET opensearch-logstash-docker-2026.03.24/_search
{
 "size": 1,
 "query": {
 "match_all": {}
 },
 "sort": [{"@timestamp": "desc"}]
}

以下為其中一筆 log

{
 "took": 563,
 "timed_out": false,
 "_shards": {
 "total": 1,
 "successful": 1,
 "skipped": 0,
 "failed": 0
 },
 "hits": {
 "total": {
 "value": 10000,
 "relation": "gte"
 },
 "max_score": null,
 "hits": [
 {
 "_index": "opensearch-logstash-docker-2026.03.24",
 "_id": "ielEH50BXMB4tDz9Y0CX",
 "_score": null,
 "_source": {
 "logtag": "F",
 "message": "W0324 09:54:23.735867 1 logging.go:55] [core] [Channel #4240 SubChannel #4241]grpc: addrConn.createTransport failed to connect to {Addr: \"127.0.0.1:2379\", ServerName: \"127.0.0.1:2379\", BalancerAttributes: {\"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" }}. Err: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled\"",
 "url": {
 "domain": "logstash.kubeantony.com",
 "path": "/",
 "port": 8080
 },
 "kubernetes": {
 "container_hash": "registry.rancher.com/rancher/hardened-kubernetes@sha256:4fab2ebc142aa1bb04a9641daebd6197404040806758cb3f3e78a6acfde47853",
 "annotations": {
 "kubernetes.io/config.mirror": "88a0909b75e7783678932b6535c63316",
 "kubernetes.io/config.hash": "88a0909b75e7783678932b6535c63316",
 "kubernetes.io/config.source": "file",
 "kubernetes.io/config.seen": "2026-03-04T13:33:01.249775809+08:00"
 },
 "pod_name": "kube-apiserver-rke2-c2",
 "container_image": "registry.rancher.com/rancher/hardened-kubernetes:v1.34.4-rke2r1-build20260210",
 "labels": {
 "tier": "control-plane",
 "component": "kube-apiserver"
 },
 "host": "rke2-c2",
 "container_name": "kube-apiserver",
 "pod_id": "80dc734a-bbbc-48bc-a95f-ba88100ec24b",
 "docker_id": "61c3121ba35fa4fa01f25037d1f27181d0b455cb350d196de37b55f13edb2444",
 "namespace_name": "kube-system"
 },
 "@timestamp": "2026-03-24T09:54:28.785431836Z",
 "user_agent": {
 "original": "Ruby"
 },
 "http": {
 "version": "HTTP/1.1",
 "request": {
 "body": {
 "bytes": "1448"
 },
 "mime_type": "application/json"
 },
 "method": "POST"
 },
 "event": {
 "original": """[{"time":"2026-03-24T17:54:23.73604561+08:00","stream":"stderr","logtag":"F","message":"W0324 09:54:23.735867 1 logging.go:55] [core] [Channel #4240 SubChannel #4241]grpc: addrConn.createTransport failed to connect to {Addr: \"127.0.0.1:2379\", ServerName: \"127.0.0.1:2379\", BalancerAttributes: {\"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" }}. Err: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: operation was canceled\"","kubernetes":{"pod_name":"kube-apiserver-rke2-c2","namespace_name":"kube-system","pod_id":"80dc734a-bbbc-48bc-a95f-ba88100ec24b","labels":{"component":"kube-apiserver","tier":"control-plane"},"annotations":{"kubernetes.io/config.hash":"88a0909b75e7783678932b6535c63316","kubernetes.io/config.mirror":"88a0909b75e7783678932b6535c63316","kubernetes.io/config.seen":"2026-03-04T13:33:01.249775809+08:00","kubernetes.io/config.source":"file"},"host":"rke2-c2","container_name":"kube-apiserver","docker_id":"61c3121ba35fa4fa01f25037d1f27181d0b455cb350d196de37b55f13edb2444","container_hash":"registry.rancher.com/rancher/hardened-kubernetes@sha256:4fab2ebc142aa1bb04a9641daebd6197404040806758cb3f3e78a6acfde47853","container_image":"registry.rancher.com/rancher/hardened-kubernetes:v1.34.4-rke2r1-build20260210"},"kubernetes_namespace":{"name":"kube-system","labels":{"field.cattle.io/projectId":"p-xnskv","kubernetes.io/metadata.name":"kube-system"}}}
]"""
 },
 "stream": "stderr",
 "time": "2026-03-24T17:54:23.73604561+08:00",
 "@version": "1",
 "host": {
 "ip": "192.168.11.156"
 },
 "kubernetes_namespace": {
 "name": "kube-system",
 "labels": {
 "field.cattle.io/projectId": "p-xnskv",
 "kubernetes.io/metadata.name": "kube-system"
 }
 }
 },
 "sort": [
 1774346068785
 ]
 }
 ]
 }
}

沒問題！我將根據您提供的架構，結合這份日誌的實際內容進行邏輯拆解與解釋。這份日誌記錄了 2026-03-24 某個 Kubernetes 核心組件發生連線異常的瞬間。

Log 架構邏輯說明

以下逐層說明這個 OpenSearch 查詢回應的結構：

第一層：查詢執行資訊 (Query Execution Info)

took: 表示 OpenSearch 花了 563 毫秒 完成這次查詢。
timed_out: false: 代表查詢在設定的超時限制內順利完成。
_shards: 記錄了參與查詢的分片狀態，這裡只有 1 個分片 且執行成功、無失敗，顯示索引結構較為簡單或資料量尚在單一分片承載範圍內。

第二層：結果集合 hits (Result Metadata)

hits.total.value: 10000 搭配 relation: "gte": 表示實際命中數量「大於或等於」10,000 筆。這是 OpenSearch 為了查詢效能所做的預設優化（未開啟 track_total_hits），告訴運維人員：符合條件的日誌非常多，這只是冰山一角。
max_score: null: 因為我們在命令中使用了 sort（依照時間排序），OpenSearch 會跳過相關性分數（Relevance Score）的計算，這在日誌分析中是標準做法。

第三層：單筆文件 hits.hits[] (Document Identity)

_index: 寫入的索引名稱為 opensearch-logstash-docker-2026.03.24。
_id: 該筆日誌的唯一標記符 ielEH50BXMB4tDz9Y0CX。
sort: 排序欄位的原始值 1774346068785（這是一個毫秒級的 Epoch 時間戳記，對應到 @timestamp）。

第四層：_source — 採集器寫入的原始資料 (Enriched Data)

這是運維診斷的核心，由日誌採集器（如 Logstash/Fluentd）自動附加的元資料：

時間戳記與時區邏輯：
- message 內: 09:54:23（容器原始輸出的 UTC 時間）。
- time: 17:54:23...+08:00（採集器轉換後的本地時間，兩者相差 8 小時）。
- @timestamp: 09:54:28Z（OpenSearch 儲存用的標準 UTC 格式，比 log 產出晚 5 秒，代表處理延遲）。
message: 核心錯誤內容，顯示 kube-apiserver 無法連線至本地的 etcd (127.0.0.1:2379)。
stream: stderr: 標記這是一條錯誤流日誌，並非正常的標準輸出。
logtag: F: 代表這是一行完整的（Full）Log，沒有因為長度限制而被截斷。
kubernetes (Pod 元資料):
- 定位：發生在命名空間 kube-system 中的 Pod kube-apiserver-rke2-c2。
- 環境：運行於節點 rke2-c2，映像檔版本為 v1.34.4。
- 身份識別：包含 pod_id 與 docker_id，方便進一步去主機端（Node）下 docker/crictl inspect 指令查核。
kubernetes_namespace: 顯示該 Namespace 屬於 Rancher 專案 p-xnskv，這有助於在多租戶環境下快速判斷受影響的專案範圍。

Quick Start Kube-ovn on Kubeadm K8s

Mon, 09 Feb 2026 01:50:20 +0800

Quick Start Kube-ovn on HA Kubeadm K8s

1. 建置 K8s 前置作業

K8s 叢集中的每台節點需做以下設定

對於其他 Linux 發行版，請確保 geneve 、 openvswitch 、 ip_tables 、 iptable_nat 、 overlay 和 br_netfilter kernel module 存在。

檢查 OS Kernel 是否有 geneve, openvswitch, ip_tables, iptable_nat 模組

# 定義需要檢查的模組
MODULES="geneve|openvswitch|ip_tables|iptable_nat|overlay|br_netfilter"
# 執行檢查
sudo lsmod | grep -E "$MODULES"

執行結果：

ip_tables 32768 0
x_tables 65536 1 ip_tables
bridge 421888 1 br_netfilter
overlay 212992 0

永久添加

cat <<EOF | sudo tee /etc/modules-load.d/kube-ovn.conf
geneve
openvswitch
ip_tables
iptable_nat
EOF
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

現在立刻添加

sudo modprobe geneve
sudo modprobe openvswitch
sudo modprobe ip_tables
sudo modprobe iptable_nat
sudo modprobe overlay
sudo modprobe br_netfilter

檢查

sudo lsmod | grep -E "$MODULES"

執行結果：

iptable_nat 12288 0
openvswitch 217088 0
nsh 12288 1 openvswitch
nf_conncount 24576 1 openvswitch
nf_nat 61440 2 openvswitch,iptable_nat
nf_conntrack 196608 3 nf_nat,openvswitch,nf_conncount
geneve 49152 0
ip6_udp_tunnel 16384 1 geneve
udp_tunnel 32768 1 geneve
br_netfilter 32768 0
bridge 421888 1 br_netfilter
overlay 212992 0
ip_tables 32768 1 iptable_nat
x_tables 65536 2 ip_tables,iptable_nat
libcrc32c 12288 6 nf_conntrack,nf_nat,openvswitch,btrfs,nf_tables,raid456

核心啟動需要開啟 IPv6，如果核心啟動參數包含 ipv6.disable=1 需要設定為 0。

檢查當前 kernel 參數

cat /proc/cmdline

執行結果：

BOOT_IMAGE=/vmlinuz-6.8.0-60-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro ipv6.disable=1

設定 ipv6.disable=0

sudo nano /etc/default/grub

需修改的檔案內容如下：

...
GRUB_CMDLINE_LINUX="ipv6.disable=0"

更新 Grub
```
sudo update-grub
```
重新開機
```
sudo reboot
```

確認值為 0

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

執行結果：

kube-proxy 正常運作，Kube-OVN 可以透過 Service ClusterIP 存取 kube-apiserver。
確認 kubelet 設定參數開啟了 CNI，且配置在標準路徑下, kubelet 啟動時應包含下列參數：
- --network-plugin=cni
- --cni-bin-dir=/opt/cni/bin
- --cni-conf-dir=/etc/cni/net.d
確認未安裝其他網路插件，或其他網路插件已被清除，檢查 /etc/cni/net.d/ 路徑下無其他網路插件設定檔。如果之前安裝過其他網路插件，建議刪除後重新啟動機器清理殘留網路資源。

確認 swap 是否關閉

cat /proc/swaps

執行結果如下

Filename Type Size Used Priority

永久添加必要 Kernel Parameters

cat <<EOF | sudo tee /etc/sysctl.d/99-k8s-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

立即載入 Kernel Parameters
```
sudo sysctl --system
```

驗證 net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables, 和 net.ipv4.ip_forward 的值都設成 1

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

執行結果：

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

定義要安裝 K8s 軟體套件版本

KUBERNETES_VERSION=v1.35
CRIO_VERSION=v1.35

安裝添加儲存庫的依賴項

sudo apt-get update
sudo apt-get install -y software-properties-common curl

添加 Kubernetes 和 CRI-O 儲存庫

curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key |
gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" |
tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key |
gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/ /" |
tee /etc/apt/sources.list.d/cri-o.list

安裝套件

sudo apt-get update
sudo apt-get install -y cri-o kubelet kubeadm kubectl crun runc

關閉 ufw Systemd Service
```
sudo systemctl disable --now ufw
```

設定 kubelet service node ip

INTERFACE=ens18
IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)
sudo sed -i \
-e '/^KUBELET_EXTRA_ARGS=$/s/$/\"--node-ip='"$IPV4_IP"'\"/' \
/etc/default/kubelet

檢查是否正確修改

cat /etc/default/kubelet

執行結果：

KUBELET_EXTRA_ARGS="--node-ip=172.20.6.61"

啟動 kubelet 服務

sudo systemctl enable --now kubelet.service

安裝 chrony
```
sudo apt-get install -y chrony
```
設定 chronyd 服務開機自動啟動
```
sudo systemctl enable chrony
```

設定跟 Google 校時

sudo nano /etc/chrony/chrony.conf

修改內容如下：

server time.google.com prefer

啟用 chronyd 服務
```
sudo systemctl start chrony
```

手動確定已校時

sudo chronyc sources

執行結果如下：

MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* time4.google.com 1 6 377 22 +271us[ +332us] +/- 3178us

SSH 連到其他節點做步驟 1 ~ 22 的設定

2. 透過 kubeadm 開始建立 HA K8s

ssh 連到第一台 Control-plane node
```
ssh sudo_user@ipaddderss
```

設定 kube-vip

export VIP="172.20.6.60"
export VIPSUBSET="16"
export INTERFACE="ens18"
KVVERSION="v1.0.1"
alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"
sudo mkdir -p /etc/kubernetes/manifests/
kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--vipSubnet $VIPSUBSET \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip-static-pod.yaml

拷貝到另外兩台 Control plane nodes

ssh sudo_user@control_plane_node_2_node_hostname 'sudo mkdir -p /etc/kubernetes/manifests/'
ssh sudo_user@control_plane_node_3_node_hostname 'sudo mkdir -p /etc/kubernetes/manifests/'
scp /etc/kubernetes/manifests/kube-vip-static-pod.yaml sudo_user@control_plane_node_2_hostname:
scp /etc/kubernetes/manifests/kube-vip-static-pod.yaml sudo_user@control_plane_node_3_node_hostname:
ssh sudo_user@control_plane_node_3_node_hostname 'sudo mv kube-vip-static-pod.yaml /etc/kubernetes/manifests/'
ssh sudo_user@control_plane_node_3_node_hostname 'sudo mv kube-vip-static-pod.yaml /etc/kubernetes/manifests/'

修改 kube-vip 的 Service Account，只需在第一台 Control-plane 需要修改

sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' \
/etc/kubernetes/manifests/kube-vip-static-pod.yaml

編輯 kubeadm 初始化設定檔

mkdir -p ${HOME}/k8s/config; \
nano "$HOME"/k8s/config/kubeadm-init.yaml

檔案內容如下：

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.20.6.61
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/crio/crio.sock
imagePullPolicy: IfNotPresent
imagePullSerial: true
name: taroko-m1.kubeantony.com
taints: []
timeouts:
controlPlaneComponentHealthCheck: 4m0s
discovery: 5m0s
etcdAPICall: 2m0s
kubeletHealthCheck: 4m0s
kubernetesAPICall: 1m0s
tlsBootstrap: 5m0s
upgradeManifests: 5m0s
---
apiServer:
certSANs:
- 127.0.0.1
- api-taroko.kubeantony.com
extraVolumes:
- name: tz-config
hostPath: /etc/localtime
mountPath: /etc/localtime
readOnly: true
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: taroko-ovn
controllerManager:
extraArgs:
- name: bind-address
value: "0.0.0.0"
- name: secure-port
value: "10257"
extraVolumes:
- name: tz-config
hostPath: /etc/localtime
mountPath: /etc/localtime
readOnly: true
#dns:
# imageRepository: harbor.example.com/library/coredns
encryptionAlgorithm: RSA-2048
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
- name: listen-metrics-urls
value: http://0.0.0.0:2381
#imageRepository: harbor.example.com/library
kind: ClusterConfiguration
kubernetesVersion: 1.35.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/16
podSubnet: 10.244.0.0/16
#proxy:
# disabled: true
scheduler:
extraArgs:
- name: bind-address
value: "0.0.0.0"
- name: secure-port
value: "10259"
extraVolumes:
- name: tz-config
hostPath: /etc/localtime
mountPath: /etc/localtime
readOnly: true
#controlPlaneEndpoint: 172.20.6.60:6443
controlPlaneEndpoint: api-taroko.kubeantony.com:6443
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
crashLoopBackOff: {}
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: "168h"
imageMinimumGCAge: "2m0s"
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
text:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
maxPods: 110
systemReserved:
memory: "1Gi"
kubeReserved:
memory: "2Gi"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
metricsBindAddress: "0.0.0.0:10249"
mode: "nftables"
nftables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 1s
syncPeriod: 30s

Dry run set up the Kubernetes control plane，用以檢視是否設定檔有誤

sudo kubeadm init \
--dry-run \
--upload-certs \
--config="$HOME"/k8s/config/kubeadm-init.yaml

正確執行結果：

...
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/tmp/kubeadm-init-dryrun1288532023/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes running the following command on each as root:
kubeadm join api-taroko.kubeantony.com:6443 --token tvxdmf.8yjcy5j5wgi0tkk8 \
--discovery-token-ca-cert-hash sha256:7c7ff5930966bcb4e5ef89f4008fbc5a1578fceff088cd0dcf5b67ecb6ae4c07 \
--control-plane --certificate-key 962bae62c93cbd3f706b07014d837be78b7f120e0e75adb6f82ad942f18a092a
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join api-taroko.kubeantony.com:6443 --token tvxdmf.8yjcy5j5wgi0tkk8 \
--discovery-token-ca-cert-hash sha256:7c7ff5930966bcb4e5ef89f4008fbc5a1578fceff088cd0dcf5b67ecb6ae4c07

Set up the Kubernetes control plane

sudo kubeadm init \
--upload-certs \
--config="$HOME"/k8s/config/kubeadm-init.yaml

正確執行結果 :

[init] Using Kubernetes version: v1.35.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [api-taroko.kubeantony.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local taroko-m1.kubeantony.com] and IPs [10.96.0.1 172.20.6.61 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost taroko-m1.kubeantony.com] and IPs [172.20.6.61 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost taroko-m1.kubeantony.com] and IPs [172.20.6.61 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 501.521647ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://172.20.6.61:6443/livez
[control-plane-check] Checking kube-controller-manager at https://0.0.0.0:10257/healthz
[control-plane-check] Checking kube-scheduler at https://0.0.0.0:10259/livez
[control-plane-check] kube-controller-manager is healthy after 505.527348ms
[control-plane-check] kube-scheduler is healthy after 1.008943332s
[control-plane-check] kube-apiserver is healthy after 3.001586313s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
064ae9995157a471491e92d05c1498092c59b389d99878b87ad08eb5c5a1231f
[mark-control-plane] Marking the node taroko-m1.kubeantony.com as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[bootstrap-token] Using token: 2mjfj9.wluywejrou2isf9n
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes running the following command on each as root:
kubeadm join api-taroko.kubeantony.com:6443 --token 2mjfj9.wluywejrou2isf9n \
--discovery-token-ca-cert-hash sha256:a7b161d19a125ccc6efc5511a81ea5dfb4563667a8d5fd922aa5f31093609094 \
--control-plane --certificate-key 064ae9995157a471491e92d05c1498092c59b389d99878b87ad08eb5c5a1231f
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join api-taroko.kubeantony.com:6443 --token 2mjfj9.wluywejrou2isf9n \
--discovery-token-ca-cert-hash sha256:a7b161d19a125ccc6efc5511a81ea5dfb4563667a8d5fd922aa5f31093609094

設定 kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
printf "\nsource <(kubectl completion bash)\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\n" >> ~/.bashrc
source ~/.bashrc

第一台 control plan 安裝好後再將 kube-vip 權限調整回來

sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip-static-pod.yaml
sudo systemctl daemon-reload
sudo systemctl restart kubelet

確認 kube-system pods 運作正常

k -n kube-system get pods

執行結果：

NAME READY STATUS RESTARTS AGE
coredns-7d764666f9-5mnc5 1/1 Running 0 58s
coredns-7d764666f9-62cqv 1/1 Running 0 58s
etcd-taroko-m1.kubeantony.com 1/1 Running 0 65s
kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 65s
kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 67s
kube-proxy-xww8m 1/1 Running 0 59s
kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 66s
kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 21s

下載 kube-ovn CNI 安裝腳本

mkdir -p ${HOME}/k8s/addon/cni/kube-ovn
wget -P ${HOME}/k8s/addon/cni/kube-ovn \
https://raw.githubusercontent.com/kubeovn/kube-ovn/refs/tags/v1.15.2/dist/images/install.sh

修改設定參數

nano ${HOME}/k8s/addon/cni/kube-ovn/install.sh

檔案內容：

IFACE=ens18
POD_CIDR="10.244.0.0/16" # Do NOT overlap with NODE/SVC/JOIN CIDR
POD_GATEWAY="10.244.0.1"
SVC_CIDR="10.96.0.0/16"

執行安裝腳本

cd ${HOME}/k8s/addon/cni/kube-ovn; \
sudo KUBECONFIG="$HOME"/.kube/config bash install.sh

執行結果：

-------------------------------
Kube-OVN Version: v1.15.2
Default Network Mode: geneve
Default Subnet CIDR: 10.244.0.0/16
Join Subnet CIDR: 100.64.0.0/16
Enable SVC LB: true
Enable Networkpolicy: true
Enable EIP and SNAT: true
Enable Mirror: false
-------------------------------
[Step 1/6] Label kube-ovn-master node and label datapath type
node/taroko-m1.kubeantony.com labeled
-------------------------------
[Step 2/6] Install OVN components
Install OVN DB in 172.20.6.61
customresourcedefinition.apiextensions.k8s.io/vpc-dnses.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/switch-lb-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpc-nat-gateways.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpc-egress-gateways.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-eips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-fip-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-dnat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/iptables-snat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-eips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-fips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-snat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ovn-dnat-rules.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vpcs.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vips.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/subnets.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/ippools.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/vlans.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/provider-networks.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/security-groups.kubeovn.io created
customresourcedefinition.apiextensions.k8s.io/qos-policies.kubeovn.io created
serviceaccount/ovn-ovs created
clusterrole.rbac.authorization.k8s.io/system:ovn-ovs created
clusterrolebinding.rbac.authorization.k8s.io/ovn-ovs created
serviceaccount/ovn created
clusterrole.rbac.authorization.k8s.io/system:ovn created
clusterrolebinding.rbac.authorization.k8s.io/ovn created
rolebinding.rbac.authorization.k8s.io/ovn created
serviceaccount/kube-ovn-cni created
clusterrole.rbac.authorization.k8s.io/system:kube-ovn-cni created
role.rbac.authorization.k8s.io/secret-reader-ovn-ipsec created
clusterrolebinding.rbac.authorization.k8s.io/kube-ovn-cni created
rolebinding.rbac.authorization.k8s.io/kube-ovn-cni created
rolebinding.rbac.authorization.k8s.io/kube-ovn-cni-secret-reader created
serviceaccount/kube-ovn-app created
clusterrole.rbac.authorization.k8s.io/system:kube-ovn-app created
clusterrolebinding.rbac.authorization.k8s.io/kube-ovn-app created
rolebinding.rbac.authorization.k8s.io/kube-ovn-app created
service/ovn-nb created
service/ovn-sb created
service/ovn-northd created
deployment.apps/ovn-central created
daemonset.apps/ovs-ovn created
Waiting for deployment "ovn-central" rollout to finish: 0 of 1 updated replicas are available...
deployment "ovn-central" successfully rolled out
Waiting for daemon set "ovs-ovn" rollout to finish: 0 of 1 updated pods are available...
daemon set "ovs-ovn" successfully rolled out
-------------------------------
[Step 3/6] Install Kube-OVN
configmap/ovn-vpc-nat-config created
configmap/ovn-vpc-nat-gw-config created
deployment.apps/kube-ovn-controller created
daemonset.apps/kube-ovn-cni created
deployment.apps/kube-ovn-monitor created
service/kube-ovn-monitor created
service/kube-ovn-controller created
service/kube-ovn-cni created
Waiting for deployment "kube-ovn-controller" rollout to finish: 0 of 1 updated replicas are available...
deployment "kube-ovn-controller" successfully rolled out
Waiting for daemon set "kube-ovn-cni" rollout to finish: 0 of 1 updated pods are available...
daemon set "kube-ovn-cni" successfully rolled out
-------------------------------
Check to delete multus pods to reload CNI config
[Step 4/6] Delete pod that not in host network mode
pod "coredns-7d764666f9-bdk5d" deleted from kube-system namespace
pod "coredns-7d764666f9-r94v9" deleted from kube-system namespace
Waiting for deployment "coredns" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "coredns" rollout to finish: 1 out of 2 new replicas have been updated...
Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available...
Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available...
deployment "coredns" successfully rolled out
Install Kube-ovn-pinger
service/kube-ovn-pinger created
daemonset.apps/kube-ovn-pinger created
Waiting for daemon set "kube-ovn-pinger" rollout to finish: 0 of 1 updated pods are available...
daemon set "kube-ovn-pinger" successfully rolled out
pod/kube-ovn-pinger-lgml7 condition met
-------------------------------
[Step 5/6] Add kubectl plugin PATH
[Step 6/6] Run network diagnose
Defaulted container "cni-server" out of: cni-server, hostpath-init (init), install-cni (init)
tar: Removing leading `/' from member names
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-7d764666f9-c66hz 1/1 Running 0 34s 10.244.0.5 taroko-m1.kubeantony.com <none> <none>
coredns-7d764666f9-nj725 1/1 Running 0 35s 10.244.0.4 taroko-m1.kubeantony.com <none> <none>
etcd-taroko-m1.kubeantony.com 1/1 Running 0 4m6s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 4m4s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 4m4s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-ovn-cni-v8nl5 1/1 Running 0 46s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-ovn-controller-6bd89b7549-xj2l6 1/1 Running 0 46s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-ovn-monitor-5d9bf9858c-sw8lq 1/1 Running 0 46s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-ovn-pinger-lgml7 1/1 Running 0 19s 10.244.0.6 taroko-m1.kubeantony.com <none> <none>
kube-proxy-8qvlw 1/1 Running 0 3m59s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 4m4s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 3m22s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
ovn-central-55d8dfb46c-wqktr 1/1 Running 0 77s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
ovs-ovn-tctvd 1/1 Running 0 77s 172.20.6.61 taroko-m1.kubeantony.com <none> <none>
NAME CREATED AT
vpcs.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
vpc-nat-gateways.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
vpc-egress-gateways.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
subnets.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
ips.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
vlans.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
provider-networks.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
security-groups.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
vips.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
vpc-dnses.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
switch-lb-rules.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
ippools.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
ovn-eips.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
ovn-fips.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
ovn-dnat-rules.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
ovn-snat-rules.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
iptables-eips.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
iptables-fip-rules.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
iptables-snat-rules.kubeovn.io 2026-02-05T10:23:16Z
NAME CREATED AT
iptables-dnat-rules.kubeovn.io 2026-02-05T10:23:16Z
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 4m5s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4m6s
NAME AGE
ovn 78s
NAME CREATED AT
system:ovn 2026-02-05T10:23:17Z
NAME ROLE AGE
ovn ClusterRole/system:ovn 78s
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
taroko-m1.kubeantony.com Ready control-plane 4m8s v1.35.0 172.20.6.61 <none> Ubuntu 24.04.3 LTS 6.8.0-90-generic cri-o://1.35.0
switch 7aeda674-777a-48ca-8f85-657035f13057 (ovn-default)
port coredns-7d764666f9-nj725.kube-system
addresses: ["52:93:cd:e8:0d:18 10.244.0.4"]
port coredns-7d764666f9-c66hz.kube-system
addresses: ["3a:e4:49:26:c4:51 10.244.0.5"]
port kube-ovn-pinger-lgml7.kube-system
addresses: ["ca:41:f2:66:e8:0b 10.244.0.6"]
port ovn-default-ovn-cluster
type: router
router-port: ovn-cluster-ovn-default
switch 7736cdb4-13b5-40d8-aa59-2f7682578e08 (join)
port join-ovn-cluster
type: router
router-port: ovn-cluster-join
port node-taroko-m1.kubeantony.com
addresses: ["ca:d5:9f:ca:6e:3e 100.64.0.2"]
router 0484e79c-fb16-403a-8feb-f9e9c1ca7578 (ovn-cluster)
port ovn-cluster-join
mac: "92:a1:79:05:14:a6"
ipv6-lla: "fe80::90a1:79ff:fe05:14a6"
networks: ["100.64.0.1/16"]
port ovn-cluster-ovn-default
mac: "96:27:4d:84:37:3b"
ipv6-lla: "fe80::9427:4dff:fe84:373b"
networks: ["10.244.0.1/16"]
Routing Policies
31000 ip4.dst == 10.244.0.0/16 allow
31000 ip4.dst == 100.64.0.0/16 allow
30000 ip4.dst == 172.20.6.61 reroute 100.64.0.2
29000 ip4.src == $ovn.default.taroko.m1.kubeantony.com_ip4 reroute 100.64.0.2
IPv4 Routes
Route Table <main>:
0.0.0.0/0 100.64.0.1 dst-ip
UUID LB PROTO VIP IPs
40a8ae8f-4e37-4ce6-aba2-2ad07cf5e062 cluster-tcp-load tcp 10.96.0.10:53 10.244.0.4:53,10.244.0.5:53
tcp 10.96.0.10:9153 10.244.0.4:9153,10.244.0.5:9153
tcp 10.96.0.1:443 172.20.6.61:6443
tcp 10.96.100.143:6641 172.20.6.61:6641
tcp 10.96.146.40:6643 172.20.6.61:6643
tcp 10.96.210.107:10665 172.20.6.61:10665
tcp 10.96.220.97:8080 10.244.0.6:8080
tcp 10.96.222.234:10661 172.20.6.61:10661
tcp 10.96.247.174:10660 172.20.6.61:10660
tcp 10.96.68.175:6642 172.20.6.61:6642
b298161a-2218-4447-a410-1c86548f3ba4 cluster-udp-load udp 10.96.0.10:53 10.244.0.4:53,10.244.0.5:53
_uuid : 26e09350-9f60-4dbd-a61b-b9448f3ab531
action : drop
direction : from-lport
external_ids : {parent=ovn.sg.kubeovn_deny_all, vendor=kube-ovn}
label : 0
log : false
match : "inport == @ovn.sg.kubeovn_deny_all && ip"
meter : []
name : []
options : {}
priority : 2003
sample_est : []
sample_new : []
severity : []
tier : 2
_uuid : 193e7062-2a24-459b-a696-5806a78aab34
action : drop
direction : to-lport
external_ids : {parent=ovn.sg.kubeovn_deny_all, vendor=kube-ovn}
label : 0
log : false
match : "outport == @ovn.sg.kubeovn_deny_all && ip"
meter : []
name : []
options : {}
priority : 2003
sample_est : []
sample_new : []
severity : []
tier : 2
Chassis "d4527da6-6748-420f-9839-646dbded1df7"
hostname: taroko-m1.kubeantony.com
Encap geneve
ip: "172.20.6.61"
options: {csum="true", is_default="true"}
Port_Binding kube-ovn-pinger-lgml7.kube-system
Port_Binding node-taroko-m1.kubeantony.com
Port_Binding coredns-7d764666f9-c66hz.kube-system
Port_Binding coredns-7d764666f9-nj725.kube-system
54cf
Name: OVN_Southbound
Cluster ID: 218b (218b1cca-afbe-4670-94c0-18f16abb94e3)
Server ID: 54cf (54cf652c-99c9-431b-9bb3-d0dcdffa4bf3)
Address: tcp:[172.20.6.61]:6644
Status: cluster member
Role: leader
Term: 1
Leader: self
Vote: self
Last Election won: 64441 ms ago
Election timer: 5000
Log: [2, 84]
Entries not yet committed: 0
Entries not yet applied: 0
Connections:
Disconnections: 0
Servers:
54cf (54cf at tcp:[172.20.6.61]:6644) (self) next_index=2 match_index=83
status: ok
8f1c
Name: OVN_Northbound
Cluster ID: e5d5 (e5d5a2ef-3b2d-493b-8bff-96396e4d6461)
Server ID: 8f1c (8f1c7a0c-ef84-4d5f-b157-96c2f1ffd614)
Address: tcp:[172.20.6.61]:6643
Status: cluster member
Role: leader
Term: 1
Leader: self
Vote: self
Last Election won: 64879 ms ago
Election timer: 5000
Log: [2, 101]
Entries not yet committed: 0
Entries not yet applied: 0
Connections:
Disconnections: 12
Servers:
8f1c (8f1c at tcp:[172.20.6.61]:6643) (self) next_index=2 match_index=100
status: ok
ds kube-proxy ready
kube-proxy ready
deployment ovn-central ready
deployment kube-ovn-controller ready
ds kube-ovn-cni ready
ds ovs-ovn ready
deployment coredns ready
ovn-nb leader check ok
ovn-sb leader check ok
ovn-northd leader check ok
service/test-node-port created
### kube-ovn-controller recent log
Defaulted container "kube-ovn-controller" out of: kube-ovn-controller, hostpath-init (init)
### start to diagnose node taroko-m1.kubeantony.com
#### ovn-controller log:
Defaulted container "pinger" out of: pinger, hostpath-init (init)
2026-02-05T10:24:09.776Z|00073|binding|INFO|Claiming lport coredns-7d764666f9-c66hz.kube-system for this chassis.
2026-02-05T10:24:09.776Z|00074|binding|INFO|coredns-7d764666f9-c66hz.kube-system: Claiming 3a:e4:49:26:c4:51 10.244.0.5
2026-02-05T10:24:09.777Z|00075|binding|INFO|Setting lport coredns-7d764666f9-c66hz.kube-system ovn-installed in OVS
2026-02-05T10:24:09.784Z|00076|binding|INFO|Setting lport coredns-7d764666f9-c66hz.kube-system up in Southbound
2026-02-05T10:24:10.345Z|00077|binding|INFO|Releasing lport coredns-7d764666f9-r94v9.kube-system from this chassis (sb_readonly=0)
2026-02-05T10:24:10.345Z|00078|binding|INFO|Setting lport coredns-7d764666f9-r94v9.kube-system down in Southbound
2026-02-05T10:24:15.878Z|00079|binding|INFO|Claiming lport kube-ovn-pinger-lgml7.kube-system for this chassis.
2026-02-05T10:24:15.878Z|00080|binding|INFO|kube-ovn-pinger-lgml7.kube-system: Claiming ca:41:f2:66:e8:0b 10.244.0.6
2026-02-05T10:24:15.882Z|00081|binding|INFO|Setting lport kube-ovn-pinger-lgml7.kube-system ovn-installed in OVS
2026-02-05T10:24:15.889Z|00082|binding|INFO|Setting lport kube-ovn-pinger-lgml7.kube-system up in Southbound
#### ovs-vswitchd log:
Defaulted container "pinger" out of: pinger, hostpath-init (init)
2026-02-05T10:23:56.412Z|00041|bridge|INFO|bridge br-int: added interface ovn0 on port 2
2026-02-05T10:23:57.868Z|00042|bridge|INFO|bridge br-int: added interface 7a4d9583419a_h on port 3
2026-02-05T10:23:57.983Z|00043|bridge|INFO|bridge br-int: added interface 99fecbc31137_h on port 4
2026-02-05T10:24:00.369Z|00044|connmgr|INFO|br-int<->unix#3: 657 flow_mods in the 9 s starting 10 s ago (634 adds, 23 deletes)
2026-02-05T10:24:00.463Z|00045|bridge|INFO|bridge br-int: added interface d10dda60b2bd_h on port 5
2026-02-05T10:24:09.337Z|00046|bridge|INFO|bridge br-int: deleted interface 7a4d9583419a_h on port 3
2026-02-05T10:24:09.775Z|00047|bridge|INFO|bridge br-int: added interface fc4fb4fc28e6_h on port 6
2026-02-05T10:24:10.345Z|00048|bridge|INFO|bridge br-int: deleted interface 99fecbc31137_h on port 4
2026-02-05T10:24:15.876Z|00049|bridge|INFO|bridge br-int: added interface 8ed870f01d82_h on port 7
2026-02-05T10:24:18.249Z|00002|ofproto_dpif_xlate(handler1)|WARN|dropping packet received on port mirror0, which is reserved exclusively for mirroring on bridge br-int while processing icmp6,in_port=1,vlan_tci=0x0000,dl_src=d6:f9:e3:fe:29:40,dl_dst=33:33:00:00:00:02,ipv6_src=fe80::d4f9:e3ff:fefe:2940,ipv6_dst=ff02::2,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=255,nw_frag=no,icmp_type=133,icmp_code=0
#### ovs-vsctl show results:
Defaulted container "pinger" out of: pinger, hostpath-init (init)
4dcba15a-02ed-4933-860e-d52da853027b
Bridge br-int
fail_mode: secure
datapath_type: system
Port d10dda60b2bd_h
Interface d10dda60b2bd_h
Port ovn0
Interface ovn0
type: internal
Port br-int
Interface br-int
type: internal
Port mirror0
Interface mirror0
type: internal
Port fc4fb4fc28e6_h
Interface fc4fb4fc28e6_h
Port "8ed870f01d82_h"
Interface "8ed870f01d82_h"
ovs_version: "3.5.3"
#### pinger diagnose results:
Defaulted container "pinger" out of: pinger, hostpath-init (init)
I0205 18:24:40.105197 53 pinger.go:21]
-------------------------------------------------------------------------------
Kube-OVN:
Version: v1.15.2
Build: 2026-01-29_08:13:34
Commit: git-0622723
Go Version: go1.25.6
Arch: amd64
-------------------------------------------------------------------------------
I0205 18:24:40.105233 53 pinger.go:24] current capabilities: cap_net_bind_service,cap_net_raw=ep
I0205 18:24:40.153647 53 config.go:202] pinger config is &{KubeConfigFile: KubeClient:0xc000637a40 Port:8080 Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:taroko-m1.kubeantony.com HostIP:172.20.6.61 PodName:kube-ovn-pinger-lgml7 PodNamespace:kube-system PodIP:10.244.0.6 PodProtocols:[IPv4] LabelSelector:app=kube-ovn-pinger ExternalAddress:1.1.1.1,2606:4700:4700::1111 NetworkMode:kube-ovn EnableMetrics:true PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid EnableVerboseConnCheck:false TCPConnCheckPort:8100 UDPConnCheckPort:8101 TargetIPPorts:tcp-172.20.6.61-31185 LogPerm:640}
I0205 18:24:40.157925 53 ovn.go:36] ovsdb-server and ovs-vswitchd are up
I0205 18:24:40.159364 53 ovn.go:52] ovn-controller is up
I0205 18:24:40.159374 53 ovn.go:60] start to check port binding
I0205 18:24:40.161788 53 ovn.go:66] ports in ovs: coredns-7d764666f9-c66hz.kube-system, coredns-7d764666f9-nj725.kube-system, kube-ovn-pinger-lgml7.kube-system, node-taroko-m1.kubeantony.com
I0205 18:24:40.165161 53 ovn.go:73] ports in sb: coredns-7d764666f9-c66hz.kube-system, coredns-7d764666f9-nj725.kube-system, kube-ovn-pinger-lgml7.kube-system, node-taroko-m1.kubeantony.com
I0205 18:24:40.165170 53 ovn.go:84] ovs and ovn-sb port binding check passed
I0205 18:24:40.165173 53 ping.go:383] start to check apiserver connectivity
I0205 18:24:40.165690 53 ping.go:394] connect to apiserver success in 0.51ms
I0205 18:24:40.165698 53 ping.go:174] start to check pod connectivity
I0205 18:24:40.489218 53 ping.go:220] ping pod: kube-ovn-pinger-lgml7 10.244.0.6, count: 3, loss count 0, average rtt 0.11ms
I0205 18:24:40.489239 53 ping.go:106] start to check node connectivity
I0205 18:24:40.752376 53 ping.go:151] ping node: taroko-m1.kubeantony.com 172.20.6.61, count: 3, loss count 0, average rtt 0.09ms
I0205 18:24:40.752395 53 ping.go:339] start to check dns connectivity
I0205 18:24:40.753552 53 ping.go:356] resolve dns kubernetes.default to [10.96.0.1] in 1.15ms
I0205 18:24:40.753563 53 ping.go:291] start to check Service or externalIPPort connectivity
I0205 18:24:40.753575 53 ping.go:298] checking targetIPPort tcp-172.20.6.61-31185
I0205 18:24:40.754269 53 ping.go:321] TCP connectivity to targetIPPort 172.20.6.61:31185 success
I0205 18:24:40.754281 53 ping.go:255] start to check ping external to 1.1.1.1
I0205 18:24:41.025593 53 ping.go:271] ping external address: 1.1.1.1, total count: 3, loss count 0, average rtt 3.43ms
### finish diagnose node taroko-m1.kubeantony.com
kubectl delete svc -l app=conn-check -n kube-system
service "test-node-port" deleted from kube-system namespace
-------------------------------
,,,,
,::,
,,::,,,,
,,,,,::::::::::::,,,,,
,,,::::::::::::::::::::::,,,
,,::::::::::::::::::::::::::::,,
,,::::::::::::::::::::::::::::::::,,
,::::::::::::::::::::::::::::::::::::,
,:::::::::::::,, ,,:::::,,,::::::::::,
,,:::::::::::::, ,::, ,:::::::::,
,:::::::::::::, :x, ,:: :, ,:::::::::,
,:::::::::::::::, ,,, ,::, ,, ,::::::::::,
,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x
,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x
,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x
,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x
,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx
,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx
,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,::
,::::, ,::::, ,::::, ,::::, ,::::,
,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,,
,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::,
,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::,
,::::, ,::::, ,::::, ,::::::::::::,,,,,
,,,, ,::::, ,,,, ,,,::::,,,,
,::::,
,,::,
Thanks for choosing Kube-OVN!
For more advanced features, please read https://kubeovn.github.io/docs/stable/en/
If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose

確認 kube-ovn 安裝成功

k -n kube-system get pods

執行結果：

NAME READY STATUS RESTARTS AGE
coredns-7d764666f9-c66hz 1/1 Running 0 2m47s
coredns-7d764666f9-nj725 1/1 Running 0 2m48s
etcd-taroko-m1.kubeantony.com 1/1 Running 0 6m19s
kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 6m17s
kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 6m17s
kube-ovn-cni-v8nl5 1/1 Running 0 2m59s
kube-ovn-controller-6bd89b7549-xj2l6 1/1 Running 0 2m59s
kube-ovn-monitor-5d9bf9858c-sw8lq 1/1 Running 0 2m59s
kube-ovn-pinger-lgml7 1/1 Running 0 2m32s
kube-proxy-8qvlw 1/1 Running 0 6m12s
kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 6m17s
kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 5m35s
ovn-central-55d8dfb46c-wqktr 1/1 Running 0 3m30s
ovs-ovn-tctvd 1/1 Running 0 3m30s

新增其他 Control-plane 節點進 K8s 叢集

獲得 Control-plane node Kubeadm Join Command

kubeadm_init_config_file="${HOME}/k8s/config/kubeadm-init.yaml"
certificate_key=$(sudo kubeadm init phase upload-certs --upload-certs --config ${kubeadm_init_config_file} | tail -n 1)
join_cmd=$(sudo kubeadm token create --print-join-command)
join_cp_cmd="sudo $join_cmd --control-plane --certificate-key $certificate_key"
echo $join_cp_cmd

執行結果：

sudo kubeadm join api-taroko.kubeantony.com:6443 --token j1d0fh.3se96b07m4h4gst6 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6 --control-plane --certificate-key 1e2dd91966c34bbc34602c07415bc30cf58348e96275b72f475b6c0001986e47

以可以執行 sudo 的 user 透過 ssh 連線至其他 Control-plane node
```
ssh sudo_user@ipaddderss
```

將 Control-plane 2 節點加入 K8s 叢集

sudo kubeadm join api-taroko.kubeantony.com:6443 --token j1d0fh.3se96b07m4h4gst6 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6 --control-plane --certificate-key 1e2dd91966c34bbc34602c07415bc30cf58348e96275b72f475b6c0001986e47

ssh 連到第三台 control plane node
```
ssh sudo_user@ipaddderss
```

將 Control-plane 3 節點加入 K8s 叢集

sudo kubeadm join api-taroko.kubeantony.com:6443 --token j1d0fh.3se96b07m4h4gst6 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6 --control-plane --certificate-key 1e2dd91966c34bbc34602c07415bc30cf58348e96275b72f475b6c0001986e47

新增 Worker 節點進 K8s 叢集

在第一台 Controlplane 將 join worker node 指令 print 出來

sudo kubeadm token create --print-join-command

執行結果：

kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6

以可以執行 sudo 的 user 透過 ssh 連線至 worker node 1
```
ssh sudo_user@ipaddderss
```

將 worker node 1 加入 K8s 叢集

sudo kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6

以可以執行 sudo 的 user 透過 ssh 連線至 worker node 2
```
ssh sudo_user@ipaddderss
```

將 worker node 2 加入 K8s 叢集

sudo kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6

以可以執行 sudo 的 user 透過 ssh 連線至 worker node 3
```
ssh sudo_user@ipaddderss
```

將 worker node 3 加入 K8s 叢集

sudo kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6

回到第一台 Control-plane 執行以下命令，將所有 worker nodes 都貼 label

kubectl label node taroko-w1.kubeantony.com node-role.kubernetes.io/worker=
kubectl label node taroko-w2.kubeantony.com node-role.kubernetes.io/worker=
kubectl label node taroko-w3.kubeantony.com node-role.kubernetes.io/worker=

檢視 K8s 狀態是否正常

檢視 K8s node 狀態

kubectl get nodes

執行結果：

NAME STATUS ROLES AGE VERSION
taroko-m1.kubeantony.com Ready control-plane 20m v1.35.0
taroko-m2.kubeantony.com Ready control-plane 7m35s v1.35.0
taroko-m3.kubeantony.com Ready control-plane 7m1s v1.35.0
taroko-w1.kubeantony.com Ready worker 105s v1.35.0
taroko-w2.kubeantony.com Ready worker 99s v1.35.0
taroko-w3.kubeantony.com Ready worker 94s v1.35.0

檢視 kube-system namespace 底下所有 pods 狀態

kubectl -n kube-system get pods

執行結果：

NAME READY STATUS RESTARTS AGE
coredns-7d764666f9-c66hz 1/1 Running 0 18m
coredns-7d764666f9-nj725 1/1 Running 0 18m
etcd-taroko-m1.kubeantony.com 1/1 Running 0 21m
etcd-taroko-m2.kubeantony.com 1/1 Running 0 8m13s
etcd-taroko-m3.kubeantony.com 1/1 Running 0 7m40s
kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 21m
kube-apiserver-taroko-m2.kubeantony.com 1/1 Running 0 8m13s
kube-apiserver-taroko-m3.kubeantony.com 1/1 Running 0 7m40s
kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 21m
kube-controller-manager-taroko-m2.kubeantony.com 1/1 Running 0 8m13s
kube-controller-manager-taroko-m3.kubeantony.com 1/1 Running 0 7m40s
kube-ovn-cni-qrx5m 1/1 Running 0 2m13s
kube-ovn-cni-r2bjz 1/1 Running 0 7m40s
kube-ovn-cni-v8nl5 1/1 Running 0 18m
kube-ovn-cni-wcqdd 1/1 Running 0 2m18s
kube-ovn-cni-xfgqv 1/1 Running 0 2m24s
kube-ovn-cni-zkwxj 1/1 Running 0 8m14s
kube-ovn-controller-6bd89b7549-xj2l6 1/1 Running 0 18m
kube-ovn-monitor-5d9bf9858c-sw8lq 1/1 Running 0 18m
kube-ovn-pinger-dtm8w 1/1 Running 0 112s
kube-ovn-pinger-h5sqd 1/1 Running 0 104s
kube-ovn-pinger-lbd48 1/1 Running 0 113s
kube-ovn-pinger-lgml7 1/1 Running 0 17m
kube-proxy-5kf45 1/1 Running 0 7m40s
kube-proxy-8qvlw 1/1 Running 0 21m
kube-proxy-c2vbq 1/1 Running 0 2m13s
kube-proxy-ct7jn 1/1 Running 0 8m14s
kube-proxy-nfb9w 1/1 Running 0 2m18s
kube-proxy-xnkwj 1/1 Running 0 2m24s
kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 21m
kube-scheduler-taroko-m2.kubeantony.com 1/1 Running 0 8m13s
kube-scheduler-taroko-m3.kubeantony.com 1/1 Running 0 7m40s
kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 20m
kube-vip-taroko-m2.kubeantony.com 1/1 Running 0 8m13s
kube-vip-taroko-m3.kubeantony.com 1/1 Running 0 7m40s
ovn-central-55d8dfb46c-wqktr 1/1 Running 0 18m
ovs-ovn-8rg8h 1/1 Running 0 2m18s
ovs-ovn-b2zmj 1/1 Running 0 2m24s
ovs-ovn-ch9jw 1/1 Running 0 7m40s
ovs-ovn-ft9m8 1/1 Running 0 2m13s
ovs-ovn-nq67b 1/1 Running 0 8m14s
ovs-ovn-tctvd 1/1 Running 0 18m

檢測 K8s 網路延遲和吞吐量

安裝 cilium 指令

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

透過 cilium 指令檢測 K8s 網路頻寬

cilium connectivity perf \
--crr \
--host-to-pod \
--pod-to-host \
--node-selector-client kubernetes.io/hostname=taroko-w1.kubeantony.com \
--node-selector-server kubernetes.io/hostname=taroko-w2.kubeantony.com

執行結果：

🔥 [taroko-ovn] Deleting connectivity check deployments...
✨ [taroko-ovn] Creating namespace cilium-test-1 for connectivity check...
ℹ️ Nodes used for performance testing:
ℹ️ * Node name: taroko-w2.kubeantony.com, zone:
ℹ️ * Node name: taroko-w1.kubeantony.com, zone:
✨ [taroko-ovn] Deploying perf-client deployment...
✨ [taroko-ovn] Deploying perf-client-other-node deployment...
✨ [taroko-ovn] Deploying perf-server deployment...
✨ [taroko-ovn] Deploying perf-client-host-net deployment...
✨ [taroko-ovn] Deploying perf-client-other-node-host-net deployment...
✨ [taroko-ovn] Deploying perf-server-host-net deployment...
⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client to become ready...
⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client-other-node to become ready...
⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-server to become ready...
⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client-host-net to become ready...
⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client-other-node-host-net to become ready...
⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-server-host-net to become ready...
ℹ️ Cilium version: 0.0.0
🏃[cilium-test-1] Running 1 tests ...
[=] [cilium-test-1] Test [network-perf] [1/1]
................................
🔥 Network Performance Test Summary [cilium-test-1]:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 60µs | 85.91µs | 677µs | 82µs | 98µs | 125µs | 11615.84
📋 pod-to-pod | same-node | TCP_RR | 10s | 14µs | 29.22µs | 695µs | 28µs | 33µs | 48µs | 34070.14
📋 pod-to-host | same-node | TCP_CRR | 10s | 74µs | 88.54µs | 179µs | 86µs | 96µs | 120µs | 474.50
📋 pod-to-host | same-node | TCP_RR | 10s | 18µs | 29.87µs | 1.801ms | 29µs | 33µs | 49µs | 33340.49
📋 host-to-pod | same-node | TCP_CRR | 10s | 82µs | 96.1µs | 189µs | 92µs | 113µs | 137µs | 11.60
📋 host-to-pod | same-node | TCP_RR | 10s | 16µs | 29.97µs | 716µs | 29µs | 33µs | 48µs | 33217.61
📋 host-to-host | same-node | TCP_CRR | 10s | 59µs | 76.15µs | 683µs | 73µs | 84µs | 118µs | 13101.67
📋 host-to-host | same-node | TCP_RR | 10s | 15µs | 28.35µs | 700µs | 28µs | 31µs | 46µs | 35113.98
📋 pod-to-pod | other-node | TCP_CRR | 10s | 242µs | 293.67µs | 1.817ms | 290µs | 323µs | 381µs | 2641.99
📋 pod-to-pod | other-node | TCP_RR | 10s | 96µs | 121.1µs | 1.509ms | 118µs | 137µs | 167µs | 8245.27
📋 pod-to-host | other-node | TCP_CRR | 10s | 246µs | 1.07221ms | 3.071576s | 293µs | 329µs | 400µs | 926.89
📋 pod-to-host | other-node | TCP_RR | 10s | 93µs | 135.28µs | 1.03699s | 118µs | 137µs | 169µs | 7382.95
📋 host-to-pod | other-node | TCP_CRR | 10s | 246µs | 299.94µs | 2.051ms | 296µs | 325µs | 386µs | 3331.08
📋 host-to-pod | other-node | TCP_RR | 10s | 98µs | 121.49µs | 1.927ms | 118µs | 137µs | 168µs | 8218.90
📋 host-to-host | other-node | TCP_CRR | 10s | 190µs | 230.35µs | 1.95ms | 227µs | 255µs | 305µs | 4336.79
📋 host-to-host | other-node | TCP_RR | 10s | 85µs | 107.45µs | 3.69ms | 106µs | 120µs | 148µs | 9292.06
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 11263.46
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 41560.29
📋 pod-to-host | same-node | TCP_STREAM | 10s | 10561.07
📋 pod-to-host | same-node | TCP_STREAM_MULTI | 10s | 40874.81
📋 host-to-pod | same-node | TCP_STREAM | 10s | 11463.51
📋 host-to-pod | same-node | TCP_STREAM_MULTI | 10s | 42231.56
📋 host-to-host | same-node | TCP_STREAM | 10s | 26166.77
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 96386.74
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 2823.65
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 4573.82
📋 pod-to-host | other-node | TCP_STREAM | 10s | 2886.42
📋 pod-to-host | other-node | TCP_STREAM_MULTI | 10s | 4405.81
📋 host-to-pod | other-node | TCP_STREAM | 10s | 2828.99
📋 host-to-pod | other-node | TCP_STREAM_MULTI | 10s | 5137.81
📋 host-to-host | other-node | TCP_STREAM | 10s | 9404.16
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 9407.35
----------------------------------------------------------------------------------------
✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.

Step by step install HA Kubernetes with Kubeadm in Disconnected environment

Fri, 23 Jan 2026 12:22:46 +0800

0. Preface

本篇文章會介紹，如何在全離線的環境下，在多台 VM 上使用 kubeadm 安裝 3 台 Control-plane Node 和 3 台 Worker Node 架構的 Kubernetes Cluster

0.1. 環境架構規劃

1. 準備 K8s 全離線安裝包

1.1. 準備 K8s 軟體套件

連線到一台 Disk 空間 70G 或以上, 可以透過網路正常連線到公有網路的 RHEL 9 電腦

註冊 RHEL 9

sudo rhc connect --activation-key $ACTIVATION-KEY --organization $ORGANIZATION

設定 Kubernetes 和 CRI-O 版本

KUBERNETES_VERSION=v1.34
CRIO_VERSION=v1.34

添加 Kubernetes yum repository

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/rpm/repodata/repomd.xml.key
EOF

添加 CRI-O yum repository

cat <<EOF | sudo tee /etc/yum.repos.d/cri-o.repo
[cri-o]
name=CRI-O
baseurl=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/
enabled=1
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/rpm/repodata/repomd.xml.key
EOF

建立工作目錄

mkdir -p "$HOME"/k8s/{addon/{cni/cilium,csi/nfs-provisioner,ingress/ingress-nginx,metrics/metrics-server,logging/{fluentd,logging-operator},kube-vip/{ha-control-plane,k8s-lb-service}},img,pkg/{kubernetes,podman,skopeo},config}

目錄結構如下

tree "$HOME"/k8s

執行結果：

/home/bigred/k8s
├── addon
│   ├── cni
│   │   └── cilium
│   ├── csi
│   │   └── nfs-provisioner
│   ├── ingress
│   │   └── ingress-nginx
│   ├── kube-vip
│   │   ├── ha-control-plane
│   │   └── k8s-lb-service
│   ├── logging
│   │   ├── fluentd
│   │   └── logging-operator
│   └── metrics
│   └── metrics-server
├── config
├── img
└── pkg
├── kubernetes
├── podman
└── skopeo
21 directories, 0 files

將必要套件載入指定目錄

sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/kubernetes -y \
crun \
runc \
cri-o \
kubeadm \
kubectl \
kubelet

檢視已載入的套件

ls -lh "$HOME"/k8s/pkg/kubernetes

執行結果：

total 78M
-rw-r--r--. 1 root root 240K Oct 16 16:28 conntrack-tools-1.4.7-4.el9_5.x86_64.rpm
-rw-r--r--. 1 root root 66K Oct 16 16:28 container-selinux-2.235.0-1.el9_6.noarch.rpm
-rw-r--r--. 1 root root 21M Oct 16 16:28 cri-o-1.34.1-150500.1.1.x86_64.rpm
-rw-r--r--. 1 root root 7.3M Oct 16 16:28 cri-tools-1.34.0-150500.1.1.x86_64.rpm
-rw-r--r--. 1 root root 563K Oct 16 16:28 criu-3.19-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 33K Oct 16 16:28 criu-libs-3.19-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 244K Oct 16 16:28 crun-1.23.1-2.el9_6.x86_64.rpm
-rw-r--r--. 1 root root 13M Oct 16 16:28 kubeadm-1.34.1-150500.1.1.x86_64.rpm
-rw-r--r--. 1 root root 12M Oct 16 16:28 kubectl-1.34.1-150500.1.1.x86_64.rpm
-rw-r--r--. 1 root root 13M Oct 16 16:28 kubelet-1.34.1-150500.1.1.x86_64.rpm
-rw-r--r--. 1 root root 8.6M Oct 16 16:28 kubernetes-cni-1.7.1-150500.1.1.x86_64.rpm
-rw-r--r--. 1 root root 60K Oct 16 16:28 libnet-1.2-7.el9.x86_64.rpm
-rw-r--r--. 1 root root 26K Oct 16 16:28 libnetfilter_cthelper-1.0.0-22.el9.x86_64.rpm
-rw-r--r--. 1 root root 26K Oct 16 16:28 libnetfilter_cttimeout-1.0.0-19.el9.x86_64.rpm
-rw-r--r--. 1 root root 31K Oct 16 16:28 libnetfilter_queue-1.0.5-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 38K Oct 16 16:28 protobuf-c-1.3.3-13.el9.x86_64.rpm
-rw-r--r--. 1 root root 3.6M Oct 16 16:28 runc-1.2.4-2.el9.x86_64.rpm
-rw-r--r--. 1 root root 42K Oct 16 16:28 yajl-2.1.0-25.el9.x86_64.rpm

下載並準備 podman pkg

sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/podman -y \
podman

檢視已載入的套件

ls -l "$HOME"/k8s/pkg/podman

執行結果：

total 24048
-rw-r--r--. 1 bigred bigred 902510 Oct 16 16:59 aardvark-dns-1.14.0-1.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 55519 Oct 16 16:59 conmon-2.1.12-1.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 158417 Oct 16 16:59 containers-common-1-117.el9_6.x86_64.rpm
-rw-r--r--. 1 bigred bigred 67105 Oct 16 16:59 container-selinux-2.235.0-1.el9_6.noarch.rpm
-rw-r--r--. 1 bigred bigred 576009 Oct 16 16:59 criu-3.19-1.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 33643 Oct 16 16:59 criu-libs-3.19-1.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 249255 Oct 16 16:59 crun-1.23.1-2.el9_6.x86_64.rpm
-rw-r--r--. 1 bigred bigred 58706 Oct 16 16:59 fuse3-3.10.2-9.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 95573 Oct 16 16:59 fuse3-libs-3.10.2-9.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 8750 Oct 16 16:59 fuse-common-3.10.2-9.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 71022 Oct 16 16:59 fuse-overlayfs-1.14-1.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 61278 Oct 16 16:59 libnet-1.2-7.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 71992 Oct 16 16:59 libslirp-4.4.0-8.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 3846970 Oct 16 16:59 netavark-1.14.1-1.el9_6.x86_64.rpm
-rw-r--r--. 1 bigred bigred 268131 Oct 16 16:59 'passt-0^20250217.ga1e48a0-10.el9_6.x86_64.rpm'
-rw-r--r--. 1 bigred bigred 27658 Oct 16 16:59 'passt-selinux-0^20250217.ga1e48a0-10.el9_6.noarch.rpm'
-rw-r--r--. 1 bigred bigred 17803252 Oct 16 16:59 podman-5.4.0-13.el9_6.x86_64.rpm
-rw-r--r--. 1 bigred bigred 38224 Oct 16 16:59 protobuf-c-1.3.3-13.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 90389 Oct 16 16:59 shadow-utils-subid-4.9-12.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 50387 Oct 16 16:59 slirp4netns-1.3.2-1.el9.x86_64.rpm
-rw-r--r--. 1 bigred bigred 42487 Oct 16 16:59 yajl-2.1.0-25.el9.x86_64.rpm

下載並準備 skopeo pkg

sudo yum install --downloadonly --downloaddir="$HOME"/k8s/pkg/skopeo -y \
skopeo

檢視已載入的套件

ls -l "$HOME"/k8s/pkg/skopeo

執行結果：

total 9308
-rw-r--r--. 1 root root 9530108 Oct 17 10:19 skopeo-1.18.1-2.el9_6.x86_64.rpm

1.2. 準備 kubeadm 要安裝 K8s 所需的 images

列出安裝 kubeadm 要安裝 K8s 所需的 images

kubeadm config images list | tee ${HOME}/k8s/img/img-list.txt

執行結果：

registry.k8s.io/kube-apiserver:v1.34.1
registry.k8s.io/kube-controller-manager:v1.34.1
registry.k8s.io/kube-scheduler:v1.34.1
registry.k8s.io/kube-proxy:v1.34.1
registry.k8s.io/coredns/coredns:v1.12.1
registry.k8s.io/pause:3.10.1
registry.k8s.io/etcd:3.6.4-0

將 K8s images 備份到指定目錄中

for i in $(cat ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
podman save -m $(cat ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/all-K8s-Components-v1.34.1.tar

拷貝 K8s 所有 components 的 images 到 Harbor 的 library project 中

for i in $(cat ${HOME}/k8s/img/img-list.txt); do skopeo copy docker://${i} docker://${i/registry.k8s.io/${HARBOR_FQDN}/library}; done

1.2.3. 準備 ingress-nginx `v1.13.3` 所需之 images 和安裝 yaml

找到 ingress-nginx v1.13.3 所需之 images

INGRESS_NGINX_VERSION="v1.13.3"
curl -s https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX_VERSION}/deploy/static/provider/cloud/deploy.yaml | grep image: | sort -u | tr -s " " | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39

將 Ingress-nginx Controller image 備份到指定目錄

for i in $(grep ingress-nginx ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*};done
podman save -m $(grep ingress-nginx ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/ingress-nginx-controller-v1.13.3.tar

將安裝 yaml 下載至指定目錄

curl -so ${HOME}/k8s/addon/ingress/ingress-nginx/deploy.yaml https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX_VERSION}/deploy/static/provider/cloud/deploy.yaml

1.3. 準備 kube-vip `v1.0.1` 所需之 images

設定環境資訊

export VIP="172.20.6.110"
export VIPSUBSET="16"
export INTERFACE="ens18"
KVVERSION="v1.0.1"
alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"

產生 kube-vip static pod yaml

kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--vipSubnet $VIPSUBSET \
--controlplane \
--arp \
--leaderElection | tee ${HOME}/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml

上述參數沒有 --service 是因為 control-plane 的 static pod 不需要管理 loadbalancer type service

產生 kube-vip daemonset yaml

kube-vip manifest daemonset \
--interface $INTERFACE \
--vipSubnet $VIPSUBSET \
--services \
--arp \
--leaderElection \
--servicesElection | tee ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-ds.yaml

下載 kube-vip Cloud Provider yaml 檔

curl -so ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-cloud-controller.yaml https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml

下載 kube-vip daemonset 所需之 rbac.yaml

curl -so ${HOME}/k8s/addon/kube-vip/k8s-lb-service/kube-vip-rbac.yaml https://kube-vip.io/manifests/rbac.yaml

檢視 Kubevip 執行所需之 images

find ${HOME}/k8s/addon/kube-vip -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

ghcr.io/kube-vip/kube-vip:v1.0.1
ghcr.io/kube-vip/kube-vip-cloud-provider:v0.0.12

將 kube-vip images 備份到指定目錄

for i in $(tail -n 2 ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
podman save -m $(tail -n 2 ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/kube-vip-v1.0.1.tar

1.4. 準備 Metrics Server `v0.8.0` 所需之 images

定義 Metrics Server 版本變數
```
METRICS_SERVER_VERSION="v0.8.0"
```

下載安裝 yaml 到指定路徑

curl -sLo ${HOME}/k8s/addon/metrics/metrics-server/components.yaml https://github.com/kubernetes-sigs/metrics-server/releases/download/${METRICS_SERVER_VERSION}/components.yaml

檢視 Metrics Server 執行所需之 images

find ${HOME}/k8s/addon/metrics/metrics-server -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

registry.k8s.io/metrics-server/metrics-server:v0.8.0

將 Metrics Server image 備份到指定目錄

for i in $(grep metrics-server ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
podman save -m $(grep metrics-server ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/metrics-server-${METRICS_SERVER_VERSION}.tar

1.5. 準備 Fluentd `v1.19.0` 所需之 images

定義 Fluentd 版本變數
```
FLUENTED_VERSION="v1.19.0-1.1"
```

下載安裝 yaml 到指定路徑

curl -sLo ${HOME}/k8s/addon/logging/fluentd/fluentd-daemonset-syslog.yaml https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/refs/tags/${FLUENTED_VERSION}/fluentd-daemonset-syslog.yaml

修改 yaml

sed -i "s|version: v1|version: ${FLUENTED_VERSION}|g" fluentd-daemonset-syslog.yaml
sed -i "s|v1-debian-syslog|v1.19.0-debian-syslog-amd64-1.1|g" fluentd-daemonset-syslog.yaml
sed -i 's|fluent/fluentd-kubernetes-daemonset|docker.io/fluent/fluentd-kubernetes-daemonset|g' fluentd-daemonset-syslog.yaml

檢視 fluentd 執行所需之 images

find ${HOME}/k8s/addon/logging/fluentd -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

docker.io/fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-amd64-1.1

將 fluentd image 備份到指定目錄

for i in $(grep fluentd ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
podman save -m $(grep fluentd ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/fluentd-kubernetes-daemonset-v1.19.0-debian-syslog-amd64-1.1.tar

1.6. 準備 cilium `1.18.2` 所需之 images

定義 helm 和 cilium 版本變數

HELM_VERSION="v3.19.0"
CILIUM_VERSION="1.18.2"

下載 helm Binary

curl -sLo ${HOME}/k8s/addon/cni/cilium/helm-${HELM_VERSION}-linux-amd64.tar.gz https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz

添加 Cilium Helm repository

helm repo add cilium https://helm.cilium.io/

下載 cilium chart file

helm fetch cilium/cilium --version=${CILIUM_VERSION} -d ${HOME}/k8s/addon/cni/cilium/

設定 helm values yaml

echo 'bpf:
masquerade: true
hubble:
relay:
enabled: true
ui:
enabled: true
service:
type: NodePort
ipam:
mode: kubernetes
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 172.20.6.10
k8sServicePort: 6443
kubeProxyReplacement: true' > ${HOME}/k8s/addon/cni/cilium/custom-values.yaml

檢視 cilium 執行所需之 images

helm template ${HOME}/k8s/addon/cni/cilium/cilium-${CILIUM_VERSION}.tgz -f ${HOME}/k8s/addon/cni/cilium/custom-values.yaml | awk '$1 ~ /image:/ {print $2}' | sed -e 's/\"//g' | sort -u | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0
quay.io/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667
quay.io/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac
quay.io/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953
quay.io/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d
quay.io/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805

將 cilium images 備份到指定目錄

for i in $(grep "quay.io/cilium" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done
podman save -m $(grep "quay.io/cilium" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/cilium-${CILIUM_VERSION}.tar

下載 Cilium CLI

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum

下載 Hubble Client

HUBBLE_VERSION="$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)"
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum

準備 cilium 測試網路效能 image

sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
cilium connectivity perf --print-image-artifacts | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

quay.io/cilium/network-perf:1751527436-c2462ae@sha256:0c491ed7ca63e6c526593b3a2d478f856410a50fbbce7fe2b64283c3015d752f

將 cilium 測試網路效能 images 備份到指定目錄

for i in $(grep "network-perf" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done
podman save $(grep "network-perf" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/cilium-network-perf.tar

1.7. 準備 NFS subdir external provisioner `v4.0.2` 所需之 images

定義 NFS provisioner 版本變數
```
NFS_PROVISIONER_VERSION="v4.0.2"
```

下載安裝 yaml 到指定路徑

git clone --no-checkout --depth=1 --filter=tree:0 https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.git ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner
cd ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner
git sparse-checkout set --no-cone /deploy
git checkout
mv ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner/deploy/ ${HOME}/k8s/addon/csi/nfs-provisioner/
sudo rm -r ${HOME}/k8s/addon/csi/nfs-provisioner/nfs-subdir-external-provisioner
sed -i "s|busybox:stable|docker.io/library/busybox:stable|g" ${HOME}/k8s/addon/csi/nfs-provisioner/deploy/test-pod.yaml

建立 kustomization.yaml

echo 'namespace: nfs-provisioner
bases:
- ./deploy
resources:
- namespace.yaml
patchesStrategicMerge:
- patch_nfs_details.yaml' > ${HOME}/k8s/addon/csi/nfs-provisioner/kustomization.yaml

建立 namespace.yaml

echo '# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: nfs-provisioner' > ${HOME}/k8s/addon/csi/nfs-provisioner/namespace.yaml

建立 patch_nfs_details.yaml

echo 'apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nfs-client-provisioner
name: nfs-client-provisioner
spec:
template:
spec:
containers:
- name: nfs-client-provisioner
env:
- name: NFS_SERVER
value: 172.20.6.9
- name: NFS_PATH
value: /k8s
volumes:
- name: nfs-client-root
nfs:
server: 172.20.6.9
path: /k8s' > ${HOME}/k8s/addon/csi/nfs-provisioner/patch_nfs_details.yaml

檢視 nfs-provisioner 執行所需之 images

find ${HOME}/k8s/addon/csi/nfs-provisioner -name "*.yaml" -type f -exec cat {} \; | grep image: | tr -s " " | uniq | cut -d " " -f 3 | tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
docker.io/library/busybox:stable

將 nfs-provisioner images 備份到指定目錄

for i in $(grep -E "nfs-subdir-external-provisioner|busybox" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
podman save -m $(grep -E "nfs-subdir-external-provisioner|busybox" ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/nfs-provisioner-${NFS_PROVISIONER_VERSION}.tar

1.8. 準備 Ingress NGINX `v1.13.3` 所需之 images 和 yaml 檔

定義 Ingress NGINX 版本變數
```
INGRESS_NGINX="v1.13.3"
```

下載部署 Ingress NGINX YAML

curl -sLo ${HOME}/k8s/addon/ingress/ingress-nginx/deployment.yaml \
https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-${INGRESS_NGINX}/deploy/static/provider/baremetal/deploy.yaml

將 deployment 轉成 daemonset

nano daemonSet.yaml

apiVersion: v1
kind: Namespace
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
name: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- coordination.k8s.io
resourceNames:
- ingress-nginx-leader
resources:
- leases
verbs:
- get
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission
namespace: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission
namespace: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: ingress-nginx
---
apiVersion: v1
data: null
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-controller
namespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
externalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-controller-admission
namespace: ingress-nginx
spec:
ports:
- appProtocol: https
name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: ClusterIP
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
spec:
automountServiceAccountToken: true
containers:
- args:
- /nginx-ingress-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --enable-metrics=false
- --report-node-internal-ip-address=true
- --watch-ingress-without-class=true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
image: registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: controller
ports:
- containerPort: 80
name: http
protocol: TCP
hostPort: 80
- containerPort: 443
name: https
protocol: TCP
hostPort: 443
- containerPort: 8443
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 90Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
readOnlyRootFilesystem: false
runAsGroup: 82
runAsNonRoot: true
runAsUser: 101
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /usr/local/certificates/
name: webhook-cert
readOnly: true
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission-create
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission-create
spec:
automountServiceAccountToken: true
containers:
- args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39
imagePullPolicy: IfNotPresent
name: create
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
seccompProfile:
type: RuntimeDefault
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
ttlSecondsAfterFinished: 0
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission-patch
namespace: ingress-nginx
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission-patch
spec:
automountServiceAccountToken: true
containers:
- args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39
imagePullPolicy: IfNotPresent
name: patch
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
seccompProfile:
type: RuntimeDefault
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
serviceAccountName: ingress-nginx-admission
ttlSecondsAfterFinished: 0
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: nginx
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.13.3
name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
service:
name: ingress-nginx-controller-admission
namespace: ingress-nginx
path: /networking/v1/ingresses
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.nginx.ingress.kubernetes.io
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None

檢視 ingress nginx 執行所需之 images

find ${HOME}/k8s/addon/ingress/ingress-nginx \
-name "daemonSet.yaml" \
-type f \
-exec grep image: {} + | \
tr -s " " | \
cut -d " " -f 3 | \
sort -u | \
uniq | \
tee -a ${HOME}/k8s/img/img-list.txt

執行結果：

registry.k8s.io/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39

將 ingress nginx images 備份到指定目錄

for i in $(grep "ingress-nginx" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; podman tag ${i} ${i%@*}; done
podman save -m $(grep "ingress-nginx" ${HOME}/k8s/img/img-list.txt | cut -d "@" -f 1) > ${HOME}/k8s/img/ingress-nginx-controller-${INGRESS_NGINX}.tar

1.9. 準備 logging operator `6.1.0` 所需之 images

定義 logging operator 版本變數
```
LOGGING_OPERATOR="6.1.0"
```

下載 logging operator chart file

helm pull oci://ghcr.io/kube-logging/helm-charts/logging-operator \
--version ${LOGGING_OPERATOR} \
-d ${HOME}/k8s/addon/logging/logging-operator/

檢視 logging operator 執行所需之 images

echo 'ghcr.io/kube-logging/logging-operator:6.1.0
ghcr.io/axoflow/axosyslog:4.18.1
ghcr.io/kube-logging/logging-operator/fluentd:6.1.0-full
ghcr.io/kube-logging/logging-operator/syslog-ng-reloader:6.1.0
ghcr.io/kube-logging/logging-operator/config-reloader:6.1.0
ghcr.io/kube-logging/logging-operator/fluentd-drain-watch:6.1.0
ghcr.io/kube-logging/logging-operator/node-exporter:6.1.0
ghcr.io/axoflow/axosyslog-metrics-exporter:0.0.13' >> ${HOME}/k8s/img/img-list.txt

將 logging operator images 備份到指定目錄

for i in $(grep -E "kube-logging|axoflow" ${HOME}/k8s/img/img-list.txt); do podman pull ${i}; done
podman save -m $(grep -E "kube-logging|axoflow" ${HOME}/k8s/img/img-list.txt) > ${HOME}/k8s/img/kube-logging-${LOGGING_OPERATOR}.tar

1.10. 將整個工作目錄打包並匯入全離線的主機

壓縮出全離線安裝包
```
cd ~; tar -zcvf k8s-v1.34.1.tar.gz k8s/
```

檢視全離線安裝包

ls -lh ${HOME}/k8s-v1.34.1.tar.gz

執行結果：

-rw-r--r--. 1 bigred bigred 1.5G Oct 17 17:25 /home/bigred/k8s-v1.34.1.tar.gz

將檔案上傳至所有全離線的 K8s 叢集主機，包含可上傳 image 至 Harbor 的主機
```
scp ${HOME}/k8s-v1.34.1.tar.gz user_name@vm_ipaddress:.
```

1.11. 將離線安裝所需之 Container Images 匯入 Harbor

先決條件：已安裝好 Harbor

連線到可將 image tar 檔上傳至 Harbor 的主機
```
ssh user@ipaddress
```
定義 Harbor FQDN
```
HARBOR_FQDN=harbor.example.com
```

下載 Harbor CA 憑證

sudo curl -k \
-o /etc/pki/ca-trust/source/anchors/harbor-ca.crt \
https://${HARBOR_FQDN}/api/v2.0/systeminfo/getcert

更新系統的 CA 信任清單，讓 HTTPS 連線時能信任此憑證
```
sudo update-ca-trust
```

設定 podman 信任 Harbor 自簽 CA 憑證

sudo mkdir /etc/containers/certs.d/${HARBOR_FQDN}
sudo curl -k \
-o /etc/containers/certs.d/${HARBOR_FQDN}/ca.crt \
https://${HARBOR_FQDN}/api/v2.0/systeminfo/getcert

登入 harbor

podman login ${HARBOR_FQDN}

執行結果：

Username: admin
Password:
Login Succeeded!

上傳所有 container images 至 Harbor

cd ~/k8s/img; ./imgctl.sh

正確執行結果：

Load All Images Successfully
Currnt: 1/29 harbor.example.com/library/kube-apiserver:v1.34.1 push ok
Currnt: 2/29 harbor.example.com/library/kube-controller-manager:v1.34.1 push ok
Currnt: 3/29 harbor.example.com/library/kube-scheduler:v1.34.1 push ok
Currnt: 4/29 harbor.example.com/library/kube-proxy:v1.34.1 push ok
Currnt: 5/29 harbor.example.com/library/coredns/coredns:v1.12.1 push ok
Currnt: 6/29 harbor.example.com/library/pause:3.10.1 push ok
Currnt: 7/29 harbor.example.com/library/etcd:3.6.4-0 push ok
Currnt: 8/29 harbor.example.com/library/ingress-nginx/controller:v1.13.3@sha256:1b044f6dcac3afbb59e05d98463f1dec6f3d3fb99940bc12ca5d80270358e3bd push ok
Currnt: 9/29 harbor.example.com/library/ingress-nginx/kube-webhook-certgen:v1.6.3@sha256:3d671cf20a35cd94efc5dcd484970779eb21e7938c98fbc3673693b8a117cf39 push ok
Currnt: 10/29 harbor.example.com/library/kube-vip/kube-vip:v1.0.1 push ok
Currnt: 11/29 harbor.example.com/library/kube-vip/kube-vip-cloud-provider:v0.0.12 push ok
Currnt: 12/29 harbor.example.com/library/metrics-server/metrics-server:v0.8.0 push ok
Currnt: 13/29 harbor.example.com/library/fluent/fluentd-kubernetes-daemonset:v1.19.0-debian-syslog-amd64-1.1 push ok
Currnt: 14/29 harbor.example.com/library/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0 push ok
Currnt: 15/29 harbor.example.com/library/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667 push ok
Currnt: 16/29 harbor.example.com/library/cilium/hubble-relay:v1.18.2@sha256:6079308ee15e44dff476fb522612732f7c5c4407a1017bc3470916242b0405ac push ok
Currnt: 17/29 harbor.example.com/library/cilium/hubble-ui-backend:v0.13.3@sha256:db1454e45dc39ca41fbf7cad31eec95d99e5b9949c39daaad0fa81ef29d56953 push ok
Currnt: 18/29 harbor.example.com/library/cilium/hubble-ui:v0.13.3@sha256:661d5de7050182d495c6497ff0b007a7a1e379648e60830dd68c4d78ae21761d push ok
Currnt: 19/29 harbor.example.com/library/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805 push ok
Currnt: 20/29 harbor.example.com/library/sig-storage/nfs-subdir-external-provisioner:v4.0.2 push ok
Currnt: 21/29 harbor.example.com/library/library/busybox:stable push ok
Currnt: 22/29 harbor.example.com/library/kube-logging/logging-operator:6.1.0 push ok
Currnt: 23/29 harbor.example.com/library/axoflow/axosyslog:4.18.1 push ok
Currnt: 24/29 harbor.example.com/library/kube-logging/logging-operator/fluentd:6.1.0-full push ok
Currnt: 25/29 harbor.example.com/library/kube-logging/logging-operator/syslog-ng-reloader:6.1.0 push ok
Currnt: 26/29 harbor.example.com/library/kube-logging/logging-operator/config-reloader:6.1.0 push ok
Currnt: 27/29 harbor.example.com/library/kube-logging/logging-operator/fluentd-drain-watch:6.1.0 push ok
Currnt: 28/29 harbor.example.com/library/kube-logging/logging-operator/node-exporter:6.1.0 push ok
Currnt: 29/29 harbor.example.com/library/axoflow/axosyslog-metrics-exporter:0.0.13 push ok

2. 建置 HA K8s 前置作業

2.1. 先決條件

已將 K8s 全離線安裝包(檔名:k8s-v1.34.1.tar.gz)匯入 K8s 叢集中的每一台主機
已將 Harbor 自簽 root CA 憑證匯入匯入 K8s 叢集中的每一台 (Harbor 如果不是自簽 root CA 則不用做到此步驟)

2.2. 連線至 K8s 的第一台 Control-plane 節點

ssh 連線至第一台 Control-plane 節點
```
ssh user@ipaddress
```

2.3. 關閉 Firewalld Systemd Service

關閉 Firewalld Systemd Service

sudo systemctl disable --now firewalld.service

2.4. 調整 SELinux 為 `permissive` mode

調整 SELinux 為 permissive mode

# Set SELinux in permissive mode (effectively disabling it)
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

2.5. 確認 SWAP 已關閉

確認 SWAP 已關閉

free -mh

執行結果：

 total used free shared buff/cache available
Mem: 7.5Gi 530Mi 6.2Gi 8.0Mi 1.0Gi 7.0Gi
Swap: 0B 0B 0B

確認 swap 欄位的值都是 0 B

2.6. 確認 Kernel 版本 >= 5.10

確認 Kernel 版本 >= 5.10
```
uname -r
```
執行結果：
```
5.14.0-570.52.1.el9_6.x86_64
```

2.7. 添加必要 Kernel Modules

永久添加 br_netfilter 和 overlay Kernel Modules

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

立即啟用 br_netfilter 和 overlay Kernel Modules

sudo modprobe overlay
sudo modprobe br_netfilter

驗證已載入 br_netfilter 和 overlay Kernel Modules

sudo lsmod | grep -E "overlay|br_netfilter"

執行結果：

br_netfilter 36864 0
bridge 417792 1 br_netfilter
overlay 229376 0

2.8. 添加必要 Kernel Parameters

永久添加必要 Kernel Parameters

cat <<EOF | sudo tee /etc/sysctl.d/99-k8s-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

立即載入 Kernel Parameters
```
sudo sysctl --system
```

驗證 net.bridge.bridge-nf-call-iptables, net.bridge.bridge-nf-call-ip6tables, 和 net.ipv4.ip_forward 的值都設成 1

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

執行結果：

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

2.9. 解壓縮 K8s 全離線安裝包

設定 K8s 全離線安裝包所在的目錄變數
```
PREFIX="$HOME"
```

解壓縮 K8s 全離線安裝包

tar -zxvf "$PREFIX"/k8s-v1.34.1.tar.gz

執行結果：

...
tar: k8s/img/ingress-nginx-controller-v1.13.3.tar: time stamp 2025-10-20 14:18:13 is 166476.992186816 s in the future
k8s/config/
tar: k8s/img: time stamp 2025-10-20 14:34:01 is 167424.992012128 s in the future

2.10. 安裝所有必要軟體套件

安裝所有必要軟體套件

sudo yum localinstall "$HOME"/k8s/pkg/kubernetes/*.rpm

2.11. 確認節點可以透過 DNS 解析到 harbor

定義 Harbor FQDN 變數

HARBOR_FQDN=harbor.example.com
DOMAIN_NAME_SYSTEM_SERVER=172.20.6.11

確認節點可以透過 DNS 解析到 harbor

dig @$DOMAIN_NAME_SYSTEM_SERVER $HARBOR_FQDN +short

執行結果：

172.20.7.32

2.12. 啟動 CRI-O Service 並設為開機自動啟動

啟動 CRI-O Service 並設為開機自動啟動
```
sudo systemctl enable --now crio.service
```

確認 CRI-O Service 狀態

sudo systemctl status crio.service

執行結果：

● crio.service - Container Runtime Interface for OCI (CRI-O)
Loaded: loaded (/usr/lib/systemd/system/crio.service; enabled; preset: disabled)
Active: active (running) since Sat 2025-10-18 16:48:59 CST; 1min 18s ago
Docs: https://github.com/cri-o/cri-o
Main PID: 5734 (crio)
Tasks: 7
Memory: 20.0M
CPU: 126ms
CGroup: /system.slice/crio.service
└─5734 /usr/bin/crio
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314737723+08:00" le…cher"
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314774402+08:00" le…face"
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314808872+08:00" le…bled"
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314814672+08:00" le…ated"
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314820552+08:00" le… NRI"
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314823302+08:00" le…p..."
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314825973+08:00" le…s..."
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314833213+08:00" le…tate"
Oct 18 16:48:59 topgun-c1.kubeantony.com crio[5734]: time="2025-10-18T16:48:59.314882773+08:00" le…bled"
Oct 18 16:48:59 topgun-c1.kubeantony.com systemd[1]: Started Container Runtime Interface for OCI (…I-O).
Hint: Some lines were ellipsized, use -l to show in full.

2.13. 設定 private registry 驗證

將 Harbor 自簽 ca 憑證上傳至主機家目錄
設定 Harbor 自簽 ca 憑證所在的目錄和 CA 憑證檔名為變數
```
CA_PREFIX="$HOME"
CA_FILE_NAME="ca.pem"
```

讓 RHEL 信任 Harbor 自簽 ca 憑證

sudo cp "$CA_PREFIX"/"$CA_FILE_NAME" /etc/pki/ca-trust/source/anchors/harbor-ca.crt
sudo update-ca-trust

編輯 CRI-O 設定檔，告訴它去哪裡讀取全域的認證資訊
```
sudo nano /etc/crio/crio.conf.d/10-crio.conf
```
檔案內容：

新增 global_auth_file 那一行：指定 CRI-O 讀取 Docker 格式的全域認證檔案路徑還有 pause_image 這行，如果沒指定這行預設還是會到 K8s 的 image repo 去拉還有 conmon_cgroup 和 cgroup_manager 也要指定在 crio.runtime 區塊底下
```
[crio.image]
signature_policy = "/etc/crio/policy.json"
global_auth_file = "/etc/crio/registries.d/auth.json"
pause_image="harbor.example.com/library/pause:3.10.1"
[crio.runtime]
default_runtime = "crun"
conmon_cgroup = "pod"
cgroup_manager = "systemd"
```
建立存放認證檔案的目錄
```
sudo mkdir -p /etc/crio/registries.d/
```
設定 Harbor Registry 的 FQDN (網域名稱) 變數，方便後續使用
```
HARBOR_FQDN=harbor.example.com
```
使用 podman 登入 Harbor，並將產生的認證 token 儲存到 CRI-O 指定的 auth.json 檔案中
```
sudo podman login $HARBOR_FQDN --authfile /etc/crio/registries.d/auth.json
```

驗證認證檔案是否已成功建立

sudo cat /etc/crio/registries.d/auth.json

執行結果如下：

{
"auths": {
"harbor.example.com": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
}
}

編輯 system-wide 的 registry 設定檔，供 Podman, CRI-O, Skopeo 等工具共用
```
sudo nano /etc/containers/registries.conf
```
檔案內容如下：

注意，Harbor FQDN 每個環境會不同。
```
[[registry]]
location = "harbor.example.com"
insecure = false 
```
重新啟動 CRI-O 服務，使其載入所有新的設定 (10-crio.conf 和 registries.conf)
```
sudo systemctl restart crio
```

使用 crictl (CRI 的客戶端工具) 測試是否能成功從私有 Harbor 拉取映像檔

sudo crictl pull ${HARBOR_FQDN}/library/library/busybox:stable

成功執行結果：

Image is up to date for harbor.example.com/library/library/busybox@sha256:f0d4f96113b0ebc3bc741b49cc158254ddd91bbba8dba20281b2dacd2da4a45f

清理測試，移除剛剛拉取的映像檔

sudo crictl rmi ${HARBOR_FQDN}/library/library/busybox:stable

2.14. 設定 kubelet service node ip

指定節點 internal ip

INTERFACE=ens18
IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)
sudo sed -i \
-e '/^KUBELET_EXTRA_ARGS=$/s/$/\"--node-ip='"$IPV4_IP"'\"/' \
/etc/sysconfig/kubelet

檢查是否正確修改

cat /etc/sysconfig/kubelet

執行結果:

KUBELET_EXTRA_ARGS="--node-ip=172.20.6.111"

2.15. 在執行 kubeadm 之前啟用 kubelet 服務

啟動 kubelet 服務

sudo systemctl enable --now kubelet.service

2.16. 設定 NTP 校時

編輯 chrony 設定檔

sudo nano /etc/chrony.conf

檔案內容如下：

#pool 2.rhel.pool.ntp.org iburst
server 172.20.6.249 iburst

重啟 chronyd 服務
```
sudo systemctl restart chronyd
```

確認是否成功校時

sudo chronyc sources

正確執行結果：

MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 172.20.6.249 3 6 377 20 -64us[ -237us] +/- 19ms

2.16. 在其他所有 K8s 節點重複執行所有步驟 2.2. ~ 2.16.

3. 透過 kubeadm 開始建立 HA K8s

3.1. 設定 kube-vip (單台 Control-plane 可跳過此步驟)

以可以執行 sudo 的 user 透過 ssh 連線至第一台 Control-plane node
```
ssh sudo_user@ipaddderss
```

將 kube-vip static pod 拷貝到 /etc/kubernetes/manifests/

sudo cp "$HOME"/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml /etc/kubernetes/manifests/kube-vip.yaml

修改 kube-vip 的 Service Account，只需在第一台 Control-plane 需要修改

sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' \
/etc/kubernetes/manifests/kube-vip.yaml

設定 harbor fqdn
```
HARBOR_FQDN=harbor.example.com
```

修改 kube-vip 的 image repo

sudo sed -i "s|image: ghcr.io|image: ${HARBOR_FQDN}/library|g" \
/etc/kubernetes/manifests/kube-vip.yaml

$ sudo nano /etc/kubernetes/manifests/kube-vip.yaml
......
spec:
containers:
......
- name: vip_interface
value: ens18
- name: vip_subnet
value: "24"
......
- name: address
value: 10.10.7.31

3.2. 編輯 kubeadm 初始化設定檔

設定 kubeadm 初始化設定檔

nano "$HOME"/k8s/config/kubeadm-init.yaml

注意每個環境需設定以下項目:

localAPIEndpoint.advertiseAddress: API Server 宣告(Advertise)給叢集其他成員的 IP 位址。

nodeRegistration.criSocket: Kubelet 應使用的 CRI socket 路徑。

nodeRegistration.name: 此節點在 Kubernetes 叢集中的唯一識別名稱 (通常使用主機名稱)。

clusterName: K8s 叢集的名稱。

controllerManager.extraVolumes: 掛載額外的磁碟區 (Volume) 到 kube-controller-manager Pod 中。

imageRepository: 用於拉取 Kubernetes 核心組件 (如 API Server, etcd) images 的 Registry 位址。

kubernetesVersion: 要部署的 Kubernetes 確切版本。

networking.serviceSubnet: 分配給 Kubernetes Services 使用的虛擬 IP 位址範圍 (CIDR)。

networking.podSubnet: 分配給叢集中所有 Pods 使用的 IP 位址範圍 (CIDR)。

proxy.disabled: 停用節點上的 kube-proxy 組件。

controlPlaneEndpoint: HA 架構的 K8s LoadBalance IP，供所有節點和使用者存取 Kubernetes API Server 的 IP 或 FQDN。

imageMaximumGCAge: 節點上未使用 images 在被垃圾回收 (GC) 前可保留的最長存留時間。

imageMinimumGCAge: 節點上未使用 images 在被垃圾回收 (GC) 前必須保留的最短存留時間。

shutdownGracePeriod: Kubelet 執行正常關機程序 (Node Shutdown) 的總寬限時間。

shutdownGracePeriodCriticalPods: 在正常關機期間，Kubelet 等待「關鍵 Pods (Critical Pods)」優雅終止的專用寬限時間。

systemReserved.memory: 為 OS 系統守護進程 (如 systemd, sshd) 保留的記憶體量，Kubelet 不會使用這部分。

kubeReserved.memory: 為 Kubernetes 系統組件 (如 kubelet, 容器執行階段) 保留的記憶體量，Pod 無法使用這部分。

檔案內容如下 :

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.20.6.111
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/crio/crio.sock
imagePullPolicy: IfNotPresent
imagePullSerial: true
name: topgun-c1.kubeantony.com
taints: null
timeouts:
controlPlaneComponentHealthCheck: 4m0s
discovery: 5m0s
etcdAPICall: 2m0s
kubeletHealthCheck: 4m0s
kubernetesAPICall: 1m0s
tlsBootstrap: 5m0s
upgradeManifests: 5m0s
---
apiServer:
certSANs:
- 127.0.0.1
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: topgun
controllerManager:
extraArgs:
- name: bind-address
value: "0.0.0.0"
- name: secure-port
value: "10257"
extraVolumes:
- name: tz-config
hostPath: /etc/localtime
mountPath: /etc/localtime
readOnly: true
dns:
imageRepository: harbor.example.com/library/coredns
encryptionAlgorithm: RSA-2048
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
- name: listen-metrics-urls
value: http://0.0.0.0:2381
imageRepository: harbor.example.com/library
kind: ClusterConfiguration
kubernetesVersion: 1.34.1
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/16
podSubnet: 10.244.0.0/16
proxy:
disabled: true
scheduler:
extraArgs:
- name: bind-address
value: "0.0.0.0"
- name: secure-port
value: "10259"
controlPlaneEndpoint: 172.20.6.110:6443
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
crashLoopBackOff: {}
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: "168h"
imageMinimumGCAge: "2m0s"
kind: KubeletConfiguration
logging:
flushFrequency: 0
options:
json:
infoBufferSize: "0"
text:
infoBufferSize: "0"
verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
maxPods: 110
systemReserved:
memory: "1Gi"
kubeReserved:
memory: "2Gi"

3.3. 初始化第一台 Kubernetes control plane

Dry run set up the Kubernetes control plane，用以檢視是否設定檔有誤

sudo kubeadm init \
--dry-run \
--upload-certs \
--config="$HOME"/k8s/config/kubeadm-init.yaml

正確執行結果 :

...
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.20.6.110:6443 --token 0n2iwr.ugdsni61mrtuqv4l \
--discovery-token-ca-cert-hash sha256:0e9dd65a4b3ba662a1a9b9f448cc1a632b808f863dd892d46fa4005b731da5bb

Set up the Kubernetes control plane

sudo kubeadm init \
--upload-certs \
--config="$HOME"/k8s/config/kubeadm-init.yaml

正確執行結果 :

W1018 22:31:59.203382 8373 initconfiguration.go:332] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta4", Kind:"ClusterConfiguration"}: strict decoding error: unknown field "apiServer.timeoutForControlPlane"
[init] Using Kubernetes version: v1.34.1
[preflight] Running pre-flight checks
[WARNING SystemVerification]: kernel release 5.14.0-570.52.1.el9_6.x86_64 is unsupported. Supported LTS versions from the 5.x series are 5.4, 5.10 and 5.15. Any 6.x version is also supported. For cgroups v2 support, the recommended version is 5.10 or newer
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local topgun-c1.kubeantony.com] and IPs [10.96.0.1 172.20.6.111 172.20.6.110 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost topgun-c1.kubeantony.com] and IPs [172.20.6.111 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost topgun-c1.kubeantony.com] and IPs [172.20.6.111 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 501.238442ms
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://172.20.6.111:6443/livez
[control-plane-check] Checking kube-controller-manager at https://0.0.0.0:10257/healthz
[control-plane-check] Checking kube-scheduler at https://0.0.0.0:10259/livez
[control-plane-check] kube-controller-manager is healthy after 1.860780055s
[control-plane-check] kube-scheduler is healthy after 2.364803775s
[control-plane-check] kube-apiserver is healthy after 4.00165287s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
681423a365cee2a5ff5b612831e42a97ff941c0c6ad0595298a6ff603b0fae5d
[mark-control-plane] Marking the node topgun-c1.kubeantony.com as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node topgun-c1.kubeantony.com as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: z6kmo1.qp9d9u4xcb85xmjl
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes running the following command on each as root:
kubeadm join 172.20.6.110:6443 --token z6kmo1.qp9d9u4xcb85xmjl \
--discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 \
--control-plane --certificate-key 681423a365cee2a5ff5b612831e42a97ff941c0c6ad0595298a6ff603b0fae5d
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.20.6.110:6443 --token z6kmo1.qp9d9u4xcb85xmjl \
--discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50

3.4. 設定 kubeConfig

設定 kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
printf "\nsource <(kubectl completion bash)\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\n" >> ~/.bashrc
source ~/.bashrc

第一台 control plan 安裝好後再將 kube-vip 權限調整回來

$ sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

3.5. 安裝 Cilium CNI

安裝 helm 指令

cd ${HOME}/k8s/addon/cni/cilium/
tar -zxvf helm-v3.19.0-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/

安裝 cilium 與 hubble client 指令

tar -zxvf cilium-linux-amd64.tar.gz -C /tmp
tar -zxvf hubble-linux-amd64.tar.gz -C /tmp
sudo mv /tmp/cilium /usr/bin/
sudo mv /tmp/hubble /usr/bin/

編輯 cilium 設定檔

nano custom-values.yaml

檔案內容：

operator:
image:
repository: harbor.example.com/library/cilium/operator
tag: v1.18.2
useDigest: false
image:
repository: harbor.example.com/library/cilium/cilium
tag: v1.18.2
useDigest: false
envoy:
image:
repository: harbor.example.com/library/cilium/cilium-envoy
tag: v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324
useDigest: false
bpf:
masquerade: true
hubble:
relay:
enabled: true
image:
repository: harbor.example.com/library/cilium/hubble-relay
tag: v1.18.2
useDigest: false
ui:
enabled: true
service:
type: NodePort
frontend:
image:
repository: harbor.example.com/library/cilium/hubble-ui
tag: v0.13.3
useDigest: false
backend:
image:
repository: harbor.example.com/library/cilium/hubble-ui-backend
tag: v0.13.3
useDigest: false
ipam:
mode: kubernetes
ipv4NativeRoutingCIDR: 10.244.0.0/16
k8sServiceHost: 172.20.6.110
k8sServicePort: 6443
kubeProxyReplacement: true
autoDirectNodeRoutes: true
routingMode: native
devices: ens18

安裝 cilium

helm -n kube-system install cilium ./cilium-1.18.2.tgz -f custom-values.yaml

確認 cilium 已安裝成功

cilium status

執行結果：

 /¯¯\
/¯¯\__/¯¯\ Cilium: OK
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Envoy DaemonSet: OK
\__/¯¯\__/ Hubble Relay: OK
\__/ ClusterMesh: disabled
DaemonSet cilium Desired: 6, Ready: 6/6, Available: 6/6
DaemonSet cilium-envoy Desired: 6, Ready: 6/6, Available: 6/6
Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2
Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1
Deployment hubble-ui Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 6
cilium-envoy Running: 6
cilium-operator Running: 2
clustermesh-apiserver
hubble-relay Running: 1
hubble-ui Running: 1
Cluster Pods: 9/9 managed by Cilium
Helm chart version:
Image versions cilium harbor.example.com/library/cilium/cilium:v1.18.2: 6
cilium-envoy harbor.example.com/library/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324: 6
cilium-operator harbor.example.com/library/cilium/operator-generic:v1.18.2: 2
hubble-relay harbor.example.com/library/cilium/hubble-relay:v1.18.2: 1
hubble-ui harbor.example.com/library/cilium/hubble-ui-backend:v0.13.3: 1
hubble-ui harbor.example.com/library/cilium/hubble-ui:v0.13.3: 1

確認 cilium 有啟用 eBPF host routing

cilium_pod=$(kubectl -n kube-system get pods --field-selector status.phase=Running -l k8s-app=cilium -o name | tail -n 1 | cut -d "/" -f 2)
kubectl -n kube-system exec -it ${cilium_pod} -- cilium-dbg status

執行結果：

KubeProxyReplacement: True [ens18 172.20.6.115 172.20.6.117 fe80::be24:11ff:fe2e:cc62 (Direct Routing)]
Host firewall: Disabled
SRv6: Disabled
CNI Chaining: none
CNI Config file: successfully wrote CNI configuration file to /host/etc/cni/net.d/05-cilium.conflist
Cilium: Ok 1.18.2 (v1.18.2-5bd307a8)
NodeMonitor: Listening for events on 2 CPUs with 64x4096 of shared memory
Cilium health daemon: Ok
IPAM: IPv4: 7/254 allocated from 10.244.5.0/24,
IPv4 BIG TCP: Disabled
IPv6 BIG TCP: Disabled
BandwidthManager: Disabled
Routing: Network: Native Host: BPF
Attach Mode: TCX
Device Mode: veth
Masquerading: BPF [ens18] 10.244.0.0/16 [IPv4: Enabled, IPv6: Disabled]
Controller Status: 47/47 healthy
Proxy Status: OK, ip 10.244.5.177, 0 redirects active on ports 10000-20000, Envoy: external
Global Identity Range: min 256, max 65535
Hubble: Ok Current/Max Flows: 2491/4095 (60.83%), Flows/s: 3.51 Metrics: Disabled
Encryption: Disabled
Cluster health: 6/6 reachable (2025-10-23T08:51:12Z) (Probe interval: 2m33.896961447s)
Name IP Node Endpoints
Modules Health: Stopped(12) Degraded(0) OK(78)

3.6. 新增其他 Control-plane 節點進 K8s 叢集 (單台 Control-plane 可跳過此步驟)

以可以執行 sudo 的 user 透過 ssh 連線至其他 Control-plane node
```
ssh sudo_user@ipaddderss
```

將 kube-vip static pod 拷貝到 /etc/kubernetes/manifests/

sudo cp "$HOME"/k8s/addon/kube-vip/ha-control-plane/kube-vip-static-pod.yaml /etc/kubernetes/manifests/kube-vip.yaml

設定 harbor fqdn
```
HARBOR_FQDN=harbor.example.com
```

修改 kube-vip 的 image repo

sudo sed -i "s|image: ghcr.io|image: ${HARBOR_FQDN}/library|g" \
/etc/kubernetes/manifests/kube-vip.yaml

獲得 Control-plane node Kubeadm Join Command 請在第一台 Control-plane node 執行以下命令

kubeadm_init_config_file="${HOME}/k8s/config/kubeadm-init.yaml"
certificate_key=$(sudo kubeadm init phase upload-certs --upload-certs --config ${kubeadm_init_config_file} | tail -n 1)
join_cmd=$(sudo kubeadm token create --print-join-command)
join_cp_cmd="sudo $join_cmd --control-plane --certificate-key $certificate_key"
echo $join_cp_cmd

執行結果：

sudo kubeadm join 172.20.6.110:6443 --token rl5cdl.8552g1j3ezrvxk21 --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 --control-plane --certificate-key d52af404ed5765485f28e0c5e7e886ae244a8861a78dc3b1ce1a0487194e8a1a

將 Control-plane 節點加入 K8s 叢集請在要加入 K8s 的 Control-plane node 執行以下命令

sudo kubeadm join 172.20.6.110:6443 --token rl5cdl.8552g1j3ezrvxk21 --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50 --control-plane --certificate-key d52af404ed5765485f28e0c5e7e886ae244a8861a78dc3b1ce1a0487194e8a1a

join node 使用的 token 會在 24 小時後自動過期

重複以上 1. ~ 6. 步驟將其他 Control-plane node 加到 K8s Cluster

3.7. 新增 Worker 節點進 K8s 叢集

在第一台 Controlplane 將 join worker node 指令 print 出來

sudo kubeadm token create --print-join-command

執行結果：

kubeadm join 172.20.6.110:6443 --token inyh77.qp4dqe6wto9uhk8m --discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50

以可以執行 sudo 的 user 透過 ssh 連線至 worker node
```
ssh sudo_user@ipaddderss
```

將 worker node 加入 K8s 叢集

sudo kubeadm join 172.20.6.110:6443 \
--token inyh77.qp4dqe6wto9uhk8m \
--discovery-token-ca-cert-hash sha256:4ea08e749828d599c6e3aefb69be7bbfbfe47bdab702907464fe033f22547e50

ssh 連線到其他 worker node，重複執行步驟 3，將其他 worker node 也加入 K8s 叢集

回到第一台 Control-plane 執行以下命令，將所有 worker nodes 都貼 label

kubectl label node topgun-w1.kubeantony.com node-role.kubernetes.io/worker=
kubectl label node topgun-w2.kubeantony.com node-role.kubernetes.io/worker=
kubectl label node topgun-w3.kubeantony.com node-role.kubernetes.io/worker=

3.8. 檢視 K8s 狀態是否正常

檢視 K8s node 狀態

kubectl get nodes

執行結果：

NAME STATUS ROLES AGE VERSION
topgun-c1.kubeantony.com Ready control-plane 3d15h v1.34.1
topgun-c2.kubeantony.com Ready control-plane 26h v1.34.1
topgun-c3.kubeantony.com Ready control-plane 3h24m v1.34.1
topgun-w1.kubeantony.com Ready worker 3d6h v1.34.1
topgun-w2.kubeantony.com Ready worker 17m v1.34.1
topgun-w3.kubeantony.com Ready worker 16m v1.34.1

檢視 kube-system namespace 底下所有 pods 狀態

kubectl -n kube-system get pods

執行結果：

NAME READY STATUS RESTARTS AGE
cilium-b74sh 1/1 Running 0 26h
cilium-btg2h 1/1 Running 0 26h
cilium-envoy-24skx 1/1 Running 0 26h
cilium-envoy-llwv7 1/1 Running 0 17m
cilium-envoy-nb5l2 1/1 Running 0 3h24m
cilium-envoy-nvkz9 1/1 Running 0 26h
cilium-envoy-vz6kj 1/1 Running 0 16m
cilium-envoy-wm4xw 1/1 Running 0 26h
cilium-gp9v9 1/1 Running 0 17m
cilium-operator-794b864946-8j8j5 1/1 Running 1 (10h ago) 26h
cilium-operator-794b864946-9whct 1/1 Running 0 26h
cilium-vqp5v 1/1 Running 0 26h
cilium-xgcch 1/1 Running 0 16m
cilium-zj2xj 1/1 Running 0 3h24m
coredns-f74588c75-86h95 1/1 Running 0 26h
coredns-f74588c75-9992t 1/1 Running 0 26h
etcd-topgun-c1.kubeantony.com 1/1 Running 0 3d15h
etcd-topgun-c2.kubeantony.com 1/1 Running 0 26h
etcd-topgun-c3.kubeantony.com 1/1 Running 0 3h24m
hubble-relay-7d6865bf7b-7wvhl 1/1 Running 0 26h
hubble-ui-c6998f8ff-6m4cn 2/2 Running 0 26h
kube-apiserver-topgun-c1.kubeantony.com 1/1 Running 0 3d15h
kube-apiserver-topgun-c2.kubeantony.com 1/1 Running 0 26h
kube-apiserver-topgun-c3.kubeantony.com 1/1 Running 0 3h24m
kube-controller-manager-topgun-c1.kubeantony.com 1/1 Running 1 (10h ago) 3d15h
kube-controller-manager-topgun-c2.kubeantony.com 1/1 Running 0 26h
kube-controller-manager-topgun-c3.kubeantony.com 1/1 Running 0 3h24m
kube-scheduler-topgun-c1.kubeantony.com 1/1 Running 1 (10h ago) 3d15h
kube-scheduler-topgun-c2.kubeantony.com 1/1 Running 0 26h
kube-scheduler-topgun-c3.kubeantony.com 1/1 Running 0 3h24m
kube-vip-topgun-c1.kubeantony.com 1/1 Running 1 (10h ago) 3d15h
kube-vip-topgun-c3.kubeantony.com 1/1 Running 0 3h24m

3.9. 檢測 K8s 網路延遲和吞吐量

主要測試以下項目
- 吞吐量 - TCP Throughput (TCP_STREAM)：有助於了解在特定配置下可達到的最大吞吐量。
  - 此測試代表大量資料傳輸的應用程式，例如串流服務或執行資料上傳/下載的服務。
- 請求/回應速率 - Request/Response Rate (TCP_RR)：主要測量處理單一網路封包來回轉發的延遲與效率。這項基準測試將在線路上產生每秒可能的最大封包數（packets per second），並著重測試處理單一網路封包的成本（資源消耗）。
  - 此測試代表了那些維持持續性連線（persistent connections）、並與其他服務交換請求/回應類型互動的服務。這對於使用 REST 或 gRPC API 的服務來說很常見。
- 連線速率 - Connection Rate (TCP_CRR)：它類似於 Request/Response Rate 測試，但會為每一次來回建立一個新的 TCP 連線。
  - 此測試代表暴露於公網的服務，它會接收來自許多客戶端的連線。L4 代理或對外部端點開啟許多連線的服務，都是很好的例子。

透過 cilium 指令檢測 K8s 網路頻寬

cilium connectivity perf \
--crr \
--host-to-pod \
--pod-to-host \
--performance-image quay.io/cilium/network-perf:1751527436-c2462ae \
--node-selector-client kubernetes.io/hostname=topgun-w1.kubeantony.com \
--node-selector-server kubernetes.io/hostname=topgun-w2.kubeantony.com

執行結果如下：

🏃[cilium-test-1] Running 1 tests ...
[=] [cilium-test-1] Test [network-perf] [1/1]
................................
🔥 Network Performance Test Summary [cilium-test-1]:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 50µs | 84.48µs | 11.99ms | 77µs | 96µs | 144µs | 11787.95
📋 pod-to-pod | same-node | TCP_RR | 10s | 11µs | 30.74µs | 4.659ms | 31µs | 35µs | 46µs | 32197.86
📋 pod-to-host | same-node | TCP_CRR | 10s | 61µs | 98.57µs | 21.094ms | 90µs | 109µs | 155µs | 10104.54
📋 pod-to-host | same-node | TCP_RR | 10s | 12µs | 31.86µs | 1.091ms | 32µs | 37µs | 49µs | 31071.90
📋 host-to-pod | same-node | TCP_CRR | 10s | 60µs | 100.33µs | 14.059ms | 90µs | 109µs | 160µs | 9930.36
📋 host-to-pod | same-node | TCP_RR | 10s | 12µs | 31.94µs | 892µs | 32µs | 36µs | 48µs | 30996.63
📋 host-to-host | same-node | TCP_CRR | 10s | 52µs | 76.41µs | 1.924ms | 73µs | 89µs | 127µs | 13016.02
📋 host-to-host | same-node | TCP_RR | 10s | 12µs | 31.89µs | 1.281ms | 32µs | 36µs | 48µs | 31043.05
📋 pod-to-pod | other-node | TCP_CRR | 10s | 242µs | 330.04µs | 13.257ms | 301µs | 342µs | 790µs | 3027.08
📋 pod-to-pod | other-node | TCP_RR | 10s | 87µs | 112.64µs | 4.085ms | 112µs | 124µs | 146µs | 8867.04
📋 pod-to-host | other-node | TCP_CRR | 10s | 243µs | 321.36µs | 15.197ms | 302µs | 340µs | 483µs | 3108.95
📋 pod-to-host | other-node | TCP_RR | 10s | 89µs | 113.52µs | 3.172ms | 113µs | 125µs | 147µs | 8797.96
📋 host-to-pod | other-node | TCP_CRR | 10s | 230µs | 311.48µs | 13.035ms | 287µs | 334µs | 635µs | 3207.10
📋 host-to-pod | other-node | TCP_RR | 10s | 88µs | 115.57µs | 4.126ms | 114µs | 128µs | 153µs | 8641.95
📋 host-to-host | other-node | TCP_CRR | 10s | 229µs | 303.13µs | 21.099ms | 285µs | 322µs | 417µs | 2739.19
📋 host-to-host | other-node | TCP_RR | 10s | 89µs | 114.78µs | 4.428ms | 114µs | 126µs | 148µs | 8702.07
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 18952.05
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 45634.84
📋 pod-to-host | same-node | TCP_STREAM | 10s | 16338.34
📋 pod-to-host | same-node | TCP_STREAM_MULTI | 10s | 41785.81
📋 host-to-pod | same-node | TCP_STREAM | 10s | 15643.64
📋 host-to-pod | same-node | TCP_STREAM_MULTI | 10s | 40812.09
📋 host-to-host | same-node | TCP_STREAM | 10s | 36117.05
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 82274.99
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 9407.21
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 9417.40
📋 pod-to-host | other-node | TCP_STREAM | 10s | 9411.27
📋 pod-to-host | other-node | TCP_STREAM_MULTI | 10s | 9411.89
📋 host-to-pod | other-node | TCP_STREAM | 10s | 9409.00
📋 host-to-pod | other-node | TCP_STREAM_MULTI | 10s | 9398.15
📋 host-to-host | other-node | TCP_STREAM | 10s | 9408.12
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 9416.04
----------------------------------------------------------------------------------------
✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.

未開 ebpf host routing 的數值：

...
🏃[cilium-test-1] Running 1 tests ...
[=] [cilium-test-1] Test [network-perf] [1/1]
................................
🔥 Network Performance Test Summary [cilium-test-1]:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_CRR | 10s | 50µs | 82.85µs | 10.656ms | 78µs | 94µs | 124µs | 12017.63
📋 pod-to-pod | same-node | TCP_RR | 10s | 10µs | 30.71µs | 828µs | 30µs | 35µs | 46µs | 32231.92
📋 pod-to-host | same-node | TCP_CRR | 10s | 59µs | 94.56µs | 12.326ms | 91µs | 108µs | 137µs | 10532.25
📋 pod-to-host | same-node | TCP_RR | 10s | 12µs | 31.94µs | 2.572ms | 32µs | 37µs | 48µs | 30996.44
📋 host-to-pod | same-node | TCP_CRR | 10s | 56µs | 93.13µs | 15.588ms | 90µs | 107µs | 136µs | 10694.60
📋 host-to-pod | same-node | TCP_RR | 10s | 12µs | 31.27µs | 1.842ms | 31µs | 36µs | 48µs | 31651.76
📋 host-to-host | same-node | TCP_CRR | 10s | 53µs | 75µs | 1.479ms | 72µs | 88µs | 123µs | 13261.02
📋 host-to-host | same-node | TCP_RR | 10s | 12µs | 31.87µs | 583µs | 32µs | 36µs | 47µs | 31059.78
📋 pod-to-pod | other-node | TCP_CRR | 10s | 278µs | 368.98µs | 9.809ms | 339µs | 382µs | 965µs | 2707.98
📋 pod-to-pod | other-node | TCP_RR | 10s | 95µs | 123.71µs | 4.119ms | 122µs | 137µs | 163µs | 8073.77
📋 pod-to-host | other-node | TCP_CRR | 10s | 248µs | 371.53µs | 1.052362s | 298µs | 335µs | 500µs | 2006.19
📋 pod-to-host | other-node | TCP_RR | 10s | 90µs | 114.49µs | 2.982ms | 114µs | 126µs | 147µs | 8723.87
📋 host-to-pod | other-node | TCP_CRR | 10s | 279µs | 366.76µs | 12.3ms | 338µs | 375µs | 850µs | 2724.34
📋 host-to-pod | other-node | TCP_RR | 10s | 105µs | 135.79µs | 15.723ms | 133µs | 148µs | 176µs | 7355.48
📋 host-to-host | other-node | TCP_CRR | 10s | 229µs | 301.42µs | 15.967ms | 284µs | 314µs | 415µs | 2739.19
📋 host-to-host | other-node | TCP_RR | 10s | 88µs | 114.62µs | 4.249ms | 114µs | 127µs | 148µs | 8713.25
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
📋 Scenario | Node | Test | Duration | Throughput Mb/s
----------------------------------------------------------------------------------------
📋 pod-to-pod | same-node | TCP_STREAM | 10s | 19819.33
📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 44767.93
📋 pod-to-host | same-node | TCP_STREAM | 10s | 17519.15
📋 pod-to-host | same-node | TCP_STREAM_MULTI | 10s | 41288.42
📋 host-to-pod | same-node | TCP_STREAM | 10s | 15667.62
📋 host-to-pod | same-node | TCP_STREAM_MULTI | 10s | 40686.99
📋 host-to-host | same-node | TCP_STREAM | 10s | 36663.82
📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 83454.15
📋 pod-to-pod | other-node | TCP_STREAM | 10s | 5515.69
📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 5462.09
📋 pod-to-host | other-node | TCP_STREAM | 10s | 9393.23
📋 pod-to-host | other-node | TCP_STREAM_MULTI | 10s | 9399.02
📋 host-to-pod | other-node | TCP_STREAM | 10s | 5456.02
📋 host-to-pod | other-node | TCP_STREAM_MULTI | 10s | 5423.52
📋 host-to-host | other-node | TCP_STREAM | 10s | 9410.65
📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 9416.39
----------------------------------------------------------------------------------------
✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.

通用欄位
- Scenario (情境): 描述流量的來源和目的地。
  - pod-to-pod: 從一個 Pod 到另一個 Pod。
  - pod-to-host: 從一個 Pod 到其運行的節點 (Host)。
  - host-to-pod: 從一個節點 (Host) 到一個 Pod。
  - host-to-host: 從一個節點 (Host) 到另一個節點 (Host)。
- Node (節點): 描述來源和目的地的相對位置。
  - same-node: 來源和目的地在同一個 Kubernetes 節點上。
  - other-node: 來源和目的地在不同的 Kubernetes 節點上。
表 1 這張表格專注於延遲（完成一個操作需要多長時間）和事務處理速率（每秒可以完成多少操作），主要用於 TCP_CRR 和 TCP_RR 測試。時間單位是微秒 (µs) 或毫秒 (ms)。
- Min (最小延遲): 在測試期間觀察到的最快的單次事務（請求/回應或連接/關閉）延遲。
- Mean (平均延遲): 所有事務延遲的算術平均值。
- Max (最大延遲): 在測試期間觀察到的最慢的單次事務延遲。高 Max 值可能表示網路抖動或偶發性的效能瓶頸。
- P50 (50百分位延遲): 也稱為中位數。50% 的事務延遲低於此值。與 Mean 相比，P50 受極端異常值的影響較小。
- P90 (90百分位延遲): 90% 的事務延遲低於此值。這是一個常見的服務水平指標 (SLI)，用於衡量絕大多數用戶的體驗。
- P99 (99百分位延遲): 99% 的事務延遲低於此值。這有助於識別「長尾延遲」(long-tail latency)，即影響少數請求的嚴重效能問題。
- Transaction rate OP/s (每秒事務處理速率): 每秒操作數 (Operations Per Second)。
  - 這個值越高越好，表示處理事務的效率越高。
表 2 這張表格專注於頻寬（每秒可以傳輸多少數據），主要用於 TCP_STREAM 和 TCP_STREAM_MULTI 測試。
- Throughput Mb/s (吞吐量 Mb/s):
  - 每秒百萬位元 (Megabits per second)。這個值顯示了在測試期間，來源和目的地之間數據傳輸的平均速率。值越高，表示網路頻寬越大。
- TCP_STREAM:
  - 使用單個 TCP 連接測量最大數據傳輸吞吐量（頻寬）。
- TCP_STREAM_MULTI:
  - 使用 32 個並行 TCP 連接測量最大數據傳輸吞吐量。這通常能更好地佔滿網路鏈路，顯示系統的總頻寬上限。

4. 安裝 K8s Addon

4.1. 安裝 Metrics server

切換工作目錄

cd ${HOME}/k8s/addon/metrics/metrics-server

編輯 Kustomization File

nano kustomization.yaml

檔案內容：

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./components.yaml
images:
- name: registry.k8s.io/metrics-server/metrics-server
newName: harbor.example.com/library/metrics-server/metrics-server
newTag: v0.8.0
patches:
- target:
kind: Deployment
name: metrics-server
patch: |-
- op: add
path: /spec/template/spec/containers/0/args/-
value: --kubelet-insecure-tls

驗證 yaml 設定檔是否有誤

kubectl apply -k . --dry-run=server

正確執行結果：

serviceaccount/metrics-server created (server dry run)
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created (server dry run)
clusterrole.rbac.authorization.k8s.io/system:metrics-server created (server dry run)
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created (server dry run)
service/metrics-server created (server dry run)
deployment.apps/metrics-server created (server dry run)
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created (server dry run)

部署 Metrics Server
```
kubectl apply -k .
```

檢視 Metrics Server pods 運作狀態

kubectl -n kube-system get pods -l k8s-app=metrics-server

執行結果：

NAME READY STATUS RESTARTS AGE
metrics-server-69779649f-frvqz 1/1 Running 0 43s

檢視 K8s 所有 nodes 的 cpu 記憶體使用狀況

kubectl top nodes

執行結果：

NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
topgun-c1.kubeantony.com 41m 2% 1705Mi 37%
topgun-c2.kubeantony.com 34m 1% 1175Mi 26%
topgun-c3.kubeantony.com 28m 1% 1057Mi 23%
topgun-w1.kubeantony.com 12m 0% 1066Mi 23%
topgun-w2.kubeantony.com 10m 0% 947Mi 21%
topgun-w3.kubeantony.com 9m 0% 778Mi 17%

4.2. 安裝 kube-vip On-Premises Cloud Controller

切換工作目錄

cd ${HOME}/k8s/addon/kube-vip/k8s-lb-service

將可分配的 IP Range 寫入 configmap

kubectl -n kube-system create configmap kubevip \
--from-literal range-global=172.20.6.117-172.20.6.120 \
--dry-run=client -o yaml > kube-vip-cm.yaml

編輯 Kustomization File

nano kustomization.yaml

檔案內容：

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./kube-vip-rbac.yaml
- ./kube-vip-ds.yaml
- ./kube-vip-cloud-controller.yaml
- ./kube-vip-cm.yaml
images:
- name: ghcr.io/kube-vip/kube-vip
newName: harbor.example.com/library/kube-vip/kube-vip
newTag: v1.0.1
- name: ghcr.io/kube-vip/kube-vip-cloud-provider
newName: harbor.example.com/library/kube-vip/kube-vip-cloud-provider
newTag: v0.0.12
patches:
- target:
kind: DaemonSet
name: kube-vip-ds
namespace: kube-system
patch: |-
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-vip-ds
spec:
template:
spec:
serviceAccountName: kube-vip
nodeSelector:
node-role.kubernetes.io/worker: ""
containers:
- name: kube-vip
env:
- name: vip_interface
value: "ens18"
- name: vip_subnet
value: "16"
volumeMounts:
- mountPath: /etc/kubernetes/admin.conf
$patch: delete
volumes:
- name: kubeconfig
$patch: delete
- target:
kind: ConfigMap
name: kubevip
namespace: kube-system
patch: |-
- op: add
path: /data/range-global
value: "172.20.6.117-172.20.6.120"

驗證 yaml 設定檔是否有誤

kubectl apply -k . --dry-run=server

正確執行結果：

serviceaccount/kube-vip created (server dry run)
serviceaccount/kube-vip-cloud-controller created (server dry run)
clusterrole.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/system:kube-vip-role created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-binding created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/system:kube-vip-cloud-controller-binding created (server dry run)
configmap/kubevip created (server dry run)
Warning: spec.template.spec.affinity.nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution[1].preference.matchExpressions[0].key: node-role.kubernetes.io/master is use "node-role.kubernetes.io/control-plane" instead
deployment.apps/kube-vip-cloud-provider created (server dry run)
daemonset.apps/kube-vip-ds created (server dry run)

部署 kube-vip On-Premises Cloud Controller
```
kubectl apply -k .
```

檢視 kube-vip On-Premises Cloud Controller pods 運作狀態

kubectl -n kube-system get pods -l component=kube-vip-cloud-provider

執行結果：

NAME READY STATUS RESTARTS AGE
kube-vip-cloud-provider-d9b9446b9-6ll6t 1/1 Running 0 45s

檢視 kube-vip ds pods 運作狀態，並確認 pods 只跑在 worker nodes 上

kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip-ds -o wide

執行結果：

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-vip-ds-kglmc 1/1 Running 0 59s 172.20.6.114 topgun-w1.kubeantony.com <none> <none>
kube-vip-ds-krst5 1/1 Running 0 59s 172.20.6.115 topgun-w2.kubeantony.com <none> <none>
kube-vip-ds-r8cmc 1/1 Running 0 59s 172.20.6.116 topgun-w3.kubeantony.com <none> <none>

4.3. 安裝 Ingress NGINX Controller

切換工作目錄

cd ${HOME}/k8s/addon/ingress/ingress-nginx

編輯 Kustomization File

nano kustomization.yaml

檔案內容如下：

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
#- ./deployment.yaml
- ./daemonSet.yaml
images:
- name: registry.k8s.io/ingress-nginx/controller
newName: harbor.example.com/library/ingress-nginx/controller
newTag: v1.13.3
- name: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.3
newName: harbor.example.com/library/ingress-nginx/kube-webhook-certgen
newTag: v1.6.3
patches:
- target:
kind: DaemonSet
name: ingress-nginx-controller
patch: |-
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ingress-nginx-controller
spec:
template:
spec:
nodeSelector:
node-role.kubernetes.io/worker: ""

驗證 yaml 設定檔是否有誤

kubectl apply -k . --dry-run=server

正確執行結果：

namespace/ingress-nginx created (server dry run)
clusterrole.rbac.authorization.k8s.io/ingress-nginx created (server dry run)
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created (server dry run)
ingressclass.networking.k8s.io/nginx created (server dry run)
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created (server dry run)
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found
Error from server (NotFound): error when creating ".": namespaces "ingress-nginx" not found

部署 Ingress NGINX Controller
```
kubectl apply -k .
```

檢視 Ingress NGINX Controller pods 運作狀態，並確認是否只在 woker nodes 上執行

kubectl -n ingress-nginx get pods -o wide

執行結果：

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-controller-4tgxs 1/1 Running 0 90s 10.244.5.109 topgun-w2.kubeantony.com <none> <none>
ingress-nginx-controller-n9t75 1/1 Running 0 67s 10.244.6.115 topgun-w3.kubeantony.com <none> <none>
ingress-nginx-controller-thkd4 1/1 Running 0 44s 10.244.1.190 topgun-w1.kubeantony.com <none> <none>

建立測試目錄
```
mkdir test
```

編輯測試用 deployment

nano test/test-deployment

檔案內容：

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: httpd
name: httpd
spec:
replicas: 3
selector:
matchLabels:
app: httpd
strategy: {}
template:
metadata:
labels:
app: httpd
spec:
containers:
- image: harbor.example.com/library/library/busybox:stable
name: busybox
command: ["/bin/sh", "-c"]
args:
- echo haha > /tmp/index.html;
httpd -p 888 -h /tmp -f

編輯 K8s service yaml

nano test/test-svc.yaml

檔案內容：

apiVersion: v1
kind: Service
metadata:
labels:
app: httpd
name: test-svc
spec:
ports:
- port: 80
protocol: TCP
targetPort: 888
selector:
app: httpd

編輯 K8s ingress yaml

nano test/test-ing.yaml

檔案內容：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
spec:
ingressClassName: nginx
rules:
- host: test.k8s.io
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: test-svc
port:
number: 80

部署測試 deployment, service 和 ingress
```
kubectl apply -f test/
```

確認 pods 狀態

kubectl get pods

執行結果：

NAME READY STATUS RESTARTS AGE
httpd-77c9cfcf7d-2tqks 1/1 Running 0 70m
httpd-77c9cfcf7d-4g9fl 1/1 Running 0 70m
httpd-77c9cfcf7d-6w2l2 1/1 Running 0 70m

確認 ingress 有拿到節點的 ip

kubectl get ing

執行結果：

NAME CLASS HOSTS ADDRESS PORTS AGE
minimal-ingress nginx test.k8s.io 172.20.6.114,172.20.6.115,172.20.6.116 80 66m

驗證可以存取網站服務

for i in {114..116}; do time curl -H "Host: test.k8s.io" http://172.20.6.${i};echo ;done

正確執行結果：

haha
real 0m0.006s
user 0m0.003s
sys 0m0.002s
haha
real 0m0.005s
user 0m0.002s
sys 0m0.002s
haha
real 0m0.005s
user 0m0.002s
sys 0m0.001s

How to Authenticate Users in Kubernetes with Keycloak OIDC Provider for Secure Cluster Access

Sun, 11 Jan 2026 06:54:20 +0800

1. Preface

本篇文章會介紹如何用 keycloak 作為 OIDC provider 來保護 Kubernetes API Server，並搭配 Kubrenetes RBAC 控制來自 keycloak users 或 group 的訪問。

2. 痛點和解決方案

靜態 Kubeconfig 檔案不僅拖慢團隊效率，更會導致資安漏洞。

對於 AI 開發人員、研究人員和 Kubernetes 管理員而言，存取叢集往往意味著必須費心管理散落在多台機器與使用者之間的長期憑證。這不僅效率低落、難以稽核，更是反覆出現且令人頭痛的合規難題。

解決方案：利用 OpenID Connect (OIDC) 將 Kubernetes 連接至現有的身分提供者 (Identity Provider)。

2. 什麼是 OpenID Connect (OIDC)

OpenID Connect (OIDC) 是建構於 OAuth 2.0 之上的一種身分認證層 (Identity Layer)。它允許客戶端（例如 Kubernetes）透過外部身分提供者 (IdP) 來驗證使用者身分，並以標準化、符合 REST 風格的方式取得基本的使用者資料。

OAuth（Open Authorization）是一個關於授權（authorization）的開放網路標準，允許使用者授權第三方應用程式存取他們儲存在其他服務提供者上的信息，而不需要將使用者名稱和密碼提供給第三方應用。 OAuth 在全世界得到了廣泛的應用，目前的版本是 2.0 。

OpenID Connect (OIDC) 是一種身分驗證協議，基於 OAuth 2.0 系列規格。 OAuth2 提供了access_token 來解決授權第三方用戶端存取受保護資源的問題，OpenID Connect 在這個基礎上提供了 id_token 來解決第三方用戶端標識使用者身分的問題。

OpenID Connect 的核心在於，在 OAuth2 的授權流程中，同時提供使用者的身份資訊（id_token）給到第三方用戶端。id_token 使用 JWT（JSON Web Token）格式進行封裝，得益於 JWT 的自包含性，緊湊性以及防篡改機制等特點，使得 id_token 可以安全地傳遞給第三方客戶端程序並且易於驗證。

JSON Web Token（JWT）是一個開放的行業標準（RFC 7519），它定義了一種簡潔的、自包含的協議格式，用於在通信雙方間傳遞 JSON 對象，傳遞的信息經過數字簽名可以被驗證和信任。想要了解 JWT 的詳細內容請參考 JWT（JSON Web Token）。

在 Kubernetes 環境中，OIDC 取代了靜態且由本地管理的憑證，轉而信任組織現有的認證系統。這能實現：

強化安全性：以短效 Token (Short-lived tokens) 取代靜態的 kubeconfig 檔案。
加速人員異動流程：透過統一的身分生命週期管理，加快人員到職啟用 (Onboarding) 與離職停用 (Deprovisioning) 的速度。
內建合規性：具備集中式認證與可稽核性 (Auditability)。

3. 靜態 Kubeconfig 檔案的問題

許多 Kubernetes 叢集仍仰賴手動分發 Kubeconfig 檔案，其中包含的憑證有效期可能長達數月，甚至永久有效。這導致了以下問題：

資安風險：設定檔一旦遭竊，攻擊者即可在未被察覺的情況下持續存取叢集。
維運阻礙：原本在 kubeconfig 檔案中，user client 憑證要一年換一次，且憑證輪替 (Credential rotation) 的過程緩慢容易出錯。
擴展性瓶頸：隨著團隊規模擴大，管理個別使用者的設定檔將變得難以掌控。

4. OIDC 如何解決此問題

將 OIDC 與 Kubernetes 整合，讓您能透過以下方式，解決手動分發 kubeconfig 檔案的問題：

移除 user client 憑證：每一次的連線階段 (Session) 都必須透過 IdP 進行認證。
實施角色權限控制 (RBAC)：將 IdP 的群組對應 (Map) 至 Kubernetes 的 RBAC 角色。
集中身分管理：在 IdP 中統一管理所有的使用者和群組，而非讓設定散落在各處的設定檔中。

5. Keycloak 介紹

keycloak 是一個開源的、面向現代應用程式和服務的 IAM（身分認證和存取控制）解決方案。 Keycloak 提供了單一登入（SSO）功能，支援 OpenID Connect、OAuth 2.0、SAML 2.0 等協議，同時 Keycloak 也支援整合不同的身分認證服務，例如 LDAP、Active Directory、Github、Google 和 Facebook 等等。

開始部署和初始化設定 HA Keycloak

1. 先決條件

部署好一套 Kubernetes 叢集，我使用的叢集版本是 v1.34.2。
已在作業系統中安裝 kubectl 和 wget 工具。

具有 cluster-admin 的管理權限。

可執行 kubectl auth whoami 確認

預期執行結果：

ATTRIBUTE VALUE
Username kubernetes-admin
Groups [kubeadm:cluster-admins system:authenticated]
Extra: authentication.kubernetes.io/credential-id [X509SHA256=e0692cad816cb69cf9f8685f14343bd5f06618cb6ddbae57e70632ea595064dc]

已在 Kuebernetes 中安裝 ingress，在我的環境中是安裝 ingress-nginx
- 可執行 kubectl get ingressclasses 確認
- 預期執行結果：
```
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 37d
```

本篇文章測試的 K8s 節點資訊如下

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubeadm-c1.kubeantony.com Ready control-plane 37d v1.34.2 10.200.8.21 <none> Ubuntu 24.04.3 LTS 6.8.0-88-generic cri-o://1.34.3
kubeadm-c2.kubeantony.com Ready control-plane 37d v1.34.2 10.200.8.22 <none> Ubuntu 24.04.3 LTS 6.8.0-71-generic cri-o://1.34.3
kubeadm-c3.kubeantony.com Ready control-plane 37d v1.34.2 10.200.8.23 <none> Ubuntu 24.04.3 LTS 6.8.0-71-generic cri-o://1.34.3
kubeadm-w1.kubeantony.com Ready logging,worker 37d v1.34.2 10.200.8.24 <none> Ubuntu 24.04.3 LTS 6.8.0-71-generic cri-o://1.34.3
kubeadm-w2.kubeantony.com Ready logging,worker 37d v1.34.2 10.200.8.25 <none> Ubuntu 24.04.3 LTS 6.8.0-71-generic cri-o://1.34.3
kubeadm-w3.kubeantony.com Ready logging,worker 37d v1.34.2 10.200.8.26 <none> Ubuntu 24.04.3 LTS 6.8.0-71-generic cri-o://1.34.3

2. 安裝 Keycloak Operator

使用 ssh 連線進可以執行 kubectl 的 K8s 管理主機

建立並切換工作目錄

mkdir -p ${HOME}/k8s/kubeadm/addon/keycloak; cd ${HOME}/k8s/kubeadm/addon/keycloak

建立 Kustomize 目錄結構

mkdir -p {env/test,resource/base-v26.4.7/operator/crd}

檢視當前目錄結構

tree -d .

執行結果：

.
├── env
│   └── test ## 用來存放在測試環境中，keycloak 相關的 yaml 檔
└── resource
└── base-v26.4.7
└── operator ## 用來存放 keycloak operator 相關的 yaml 檔
└── crd
7 directories

下載 keycloak operator 的 crd yaml 檔

wget -P resource/base-v26.4.7/operator/crd \
https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.4.7/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
wget -P resource/base-v26.4.7/operator/crd \
https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.4.7/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml

撰寫 keycloak operator 的 namespace yaml 檔

cat <<EOF > resource/base-v26.4.7/operator/keycloak-operator-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: keycloak
EOF

下載 keycloak operator 的 deployment yaml 檔

wget -O resource/base-v26.4.7/operator/keycloak-operator-deployment.yaml \
https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.4.7/kubernetes/kubernetes.yml

撰寫 keycloak operator 的 kustomization yaml 檔

cat <<EOF > resource/kustomization.yaml
kind: Kustomization
resources:
- ./base-v26.4.7/operator/crd/keycloaks.k8s.keycloak.org-v1.yml
- ./base-v26.4.7/operator/crd/keycloakrealmimports.k8s.keycloak.org-v1.yml
- ./base-v26.4.7/operator/keycloak-operator-ns.yaml
- ./base-v26.4.7/operator/keycloak-operator-deployment.yaml
namespace: keycloak
EOF

測試部署 keycloak operator 是否有誤

kubectl apply -k resource/ --dry-run=server

執行結果：

Warning: resource namespaces/keycloak is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
namespace/keycloak configured (server dry run)
customresourcedefinition.apiextensions.k8s.io/keycloakrealmimports.k8s.keycloak.org created (server dry run)
customresourcedefinition.apiextensions.k8s.io/keycloaks.k8s.keycloak.org created (server dry run)
serviceaccount/keycloak-operator created (server dry run)
role.rbac.authorization.k8s.io/keycloak-operator-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/keycloak-operator-clusterrole created (server dry run)
clusterrole.rbac.authorization.k8s.io/keycloakcontroller-cluster-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/keycloakrealmimportcontroller-cluster-role created (server dry run)
rolebinding.rbac.authorization.k8s.io/keycloak-operator-role-binding created (server dry run)
rolebinding.rbac.authorization.k8s.io/keycloak-operator-view created (server dry run)
rolebinding.rbac.authorization.k8s.io/keycloakcontroller-role-binding created (server dry run)
rolebinding.rbac.authorization.k8s.io/keycloakrealmimportcontroller-role-binding created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/keycloak-operator-clusterrole-binding created (server dry run)
service/keycloak-operator created (server dry run)
deployment.apps/keycloak-operator created (server dry run)

確認沒問題後，正式部署
```
kubectl apply -k resource/ 
```

確認 keycloak operator pods 運作狀態

kubectl -n keycloak get pods

執行結果：

NAME READY STATUS RESTARTS AGE
keycloak-operator-59c9ff54-h9wvv 1/1 Running 0 2m13s

2. 安裝 local path provisioner (如果環境已有 StorageClass 可跳過此步驟)

部署 local path provisioner

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.34/deploy/local-path-storage.yaml

檢視 local path provisioner 的 pods 運作狀態

kubectl -n local-path-storage get pods

執行結果：

NAME READY STATUS RESTARTS AGE
local-path-provisioner-5cf8df5d54-wtl8k 1/1 Running 1 (12h ago) 12h

檢視 storageclass 名稱

kubectl get sc

執行結果：

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path rancher.io/local-path Delete WaitForFirstConsumer false 107d

3. 透過 CloudNativePG Operator 安裝 PostgreSQL cluster

建立並切換工作目錄

mkdir -p ${HOME}/k8s/kubeadm/addon/cloudNativePG; cd ${HOME}/k8s/kubeadm/addon/cloudNativePG

建立 Kustomize 目錄結構

mkdir -p {env/test,resource/base-v1.28.0/operator}

檢視當前目錄結構

tree -d .

執行結果：

.
├── env
│   └── test
└── resource
└── base-v1.28.0
└── operator
6 directories

下載 CloudNativePG Operator yaml 檔

wget -P resource/base-v1.28.0/operator \
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.28/releases/cnpg-1.28.0.yaml

撰寫 CloudNativePG Operator 的 kustomization yaml 檔

cat <<EOF > resource/kustomization.yaml
kind: Kustomization
resources:
- ./base-v1.28.0/operator/cnpg-1.28.0.yaml
EOF

測試部署 CloudNativePG Operator

kubectl apply -k resource/ --dry-run=server

執行結果：

namespace/cnpg-system created (server dry run)
customresourcedefinition.apiextensions.k8s.io/backups.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/clusterimagecatalogs.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/clusters.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/databases.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/failoverquorums.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/imagecatalogs.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/publications.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/scheduledbackups.postgresql.cnpg.io created (server dry run)
customresourcedefinition.apiextensions.k8s.io/subscriptions.postgresql.cnpg.io created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-database-editor-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-database-viewer-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-manager created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-publication-editor-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-publication-viewer-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-subscription-editor-role created (server dry run)
clusterrole.rbac.authorization.k8s.io/cnpg-subscription-viewer-role created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/cnpg-manager-rolebinding created (server dry run)
mutatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-mutating-webhook-configuration created (server dry run)
validatingwebhookconfiguration.admissionregistration.k8s.io/cnpg-validating-webhook-configuration created (server dry run)
Error from server (Invalid): error when creating "resource/": CustomResourceDefinition.apiextensions.k8s.io "poolers.postgresql.cnpg.io" is invalid: metadata.annotations: Too long: may not be more than 262144 bytes
Error from server (NotFound): error when creating "resource/": namespaces "cnpg-system" not found
Error from server (NotFound): error when creating "resource/": namespaces "cnpg-system" not found
Error from server (NotFound): error when creating "resource/": namespaces "cnpg-system" not found
Error from server (NotFound): error when creating "resource/": namespaces "cnpg-system" not found

部署 CloudNativePG Operator

kubectl apply -k resource/ --server-side

檢視 CloudNativePG Operator 的 pods 運作狀態

kubectl -n cnpg-system get pods

執行結果：

NAME READY STATUS RESTARTS AGE
cnpg-controller-manager-6b9f78f594-8g2qt 1/1 Running 0 40s

撰寫 PostgreSQL cluster 的 namespace yaml 檔

cat <<EOF > env/test/cnpg-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: cnpg
EOF

撰寫 PostgreSQL cluster 儲存初始化資料庫的使用者帳密的 secret yaml 檔

cat <<EOF > env/test/cnpg-user-secret.yaml
apiVersion: v1
data:
password: YmlncmVk
username: YmlncmVk
kind: Secret
metadata:
name: cnpg-bigred
namespace: cnpg
EOF

撰寫 PostgreSQL cluster 的 cluster yaml 檔

cat <<EOF > env/test/cnpg-cluster.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: psql
namespace: cnpg
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:17.5
bootstrap:
initdb:
database: keycloak
owner: bigred
secret:
name: cnpg-bigred
storage:
storageClass: local-path
size: 1Gi
EOF

撰寫 PostgreSQL cluster 的 pooler yaml 檔

cat <<EOF > env/test/cnpg-pooler.yaml
apiVersion: postgresql.cnpg.io/v1
kind: Pooler
metadata:
name: pooler-rw
namespace: cnpg
spec:
cluster:
name: psql
instances: 3
type: rw
pgbouncer:
poolMode: session
parameters:
max_client_conn: "1000"
default_pool_size: "10"
EOF

撰寫 PostgreSQL cluster 的 kustomizte yaml 檔

cat <<EOF > env/test/kustomization.yaml
kind: Kustomization
resources:
- ./cnpg-ns.yaml
- ./cnpg-user-secret.yaml
- ./cnpg-cluster.yaml
- ./cnpg-pooler.yaml
EOF

測試部署 PostgreSQL cluster

kubectl apply -k env/test/ --dry-run=server

執行結果：

namespace/cnpg created (server dry run)
Error from server (NotFound): error when creating "env/test/": namespaces "cnpg" not found
Error from server (NotFound): error when creating "env/test/": namespaces "cnpg" not found
Error from server (NotFound): error when creating "env/test/": namespaces "cnpg" not found

正式部署 PostgreSQL cluster

kubectl apply -k env/test/

執行結果：

namespace/cnpg created
secret/cnpg-bigred created
cluster.postgresql.cnpg.io/psql created
pooler.postgresql.cnpg.io/pooler-rw created

檢視 PostgreSQL cluster pods 運作狀態

kubectl -n cnpg get pods -o wide

執行結果：

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pooler-rw-8949649fd-9fw6z 1/1 Running 0 8m9s 10.244.4.204 kubeadm-w2 <none> <none>
pooler-rw-8949649fd-h47mt 1/1 Running 0 8m9s 10.244.3.106 kubeadm-w1 <none> <none>
pooler-rw-8949649fd-n862s 1/1 Running 0 8m9s 10.244.5.208 kubeadm-w3 <none> <none>
psql-1 1/1 Running 0 7m2s 10.244.3.1 kubeadm-w1 <none> <none>
psql-2 1/1 Running 0 6m5s 10.244.5.164 kubeadm-w3 <none> <none>
psql-3 1/1 Running 0 4m59s 10.244.4.229 kubeadm-w2 <none> <none>

檢視 PostgreSQL cluster service

kubectl -n cnpg get svc

執行結果：

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pooler-rw ClusterIP 10.96.28.116 <none> 5432/TCP 6m43s
psql-r ClusterIP 10.96.216.168 <none> 5432/TCP 6m44s
psql-ro ClusterIP 10.96.3.136 <none> 5432/TCP 6m44s
psql-rw ClusterIP 10.96.21.102 <none> 5432/TCP 6m44s

4. 安裝 cert-manager

建立並切換工作目錄

mkdir -p ${HOME}/k8s/kubeadm/addon/cert-manager; cd ${HOME}/k8s/kubeadm/addon/cert-manager

建立 Kustomize 目錄結構

mkdir -p {env/test,resource/base-v1.19.2}

檢視當前目錄結構

tree -d .

執行結果：

.
├── env
│   └── test
└── resource
└── base-v1.19.2
5 directories

下載 cert-manager yaml 檔

wget -P resource/base-v1.19.2 \
https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml

撰寫 cert-manager 基本湯底的 kustomize yaml

cat <<EOF > resource/kustomization.yaml
kind: Kustomization
resources:
- ./base-v1.19.2/cert-manager.yaml
EOF

撰寫測試區 cert-manager 的 kustomize yaml

cat <<EOF > env/test/kustomization.yaml
kind: Kustomization
resources:
- ../../resource/
EOF

測試部署 cert-manager

kubectl apply -k env/test/ --dry-run=server

正式部署 cert-manager
```
kubectl apply -k env/test/
```

確認 cert-mangaer pods 運作狀態

kubectl -n cert-manager get pods

執行結果：

NAME READY STATUS RESTARTS AGE
cert-manager-548f7cf98c-v47q9 1/1 Running 1 (14h ago) 14h
cert-manager-cainjector-8798f647f-8f2xl 1/1 Running 1 (14h ago) 14h
cert-manager-webhook-6c8678dc46-hw5r4 1/1 Running 1 (14h ago) 14h

5. 透過 Keycloak Operator 安裝 HA Keycloak

切換工作目錄
```
cd ${HOME}/k8s/kubeadm/addon/keycloak
```

撰寫 keycloak namespace yaml 檔

cat <<EOF > env/test/keycloak-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: keycloak
EOF

撰寫 keycloak 儲存初始化管理員的使用者帳密 secret yaml 檔

cat <<EOF > env/test/keycloak-bootstrapAdmin-user-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: keycloak-bootstrapadmin-user
namespace: keycloak
type: kubernetes.io/basic-auth
stringData:
username: admin
password: admin
EOF

撰寫 keycloak 要串接 postgresql 的帳號密碼的 secret yaml 檔

cat <<EOF > env/test/cnpg-user-secret.yaml
apiVersion: v1
data:
password: YmlncmVk
username: YmlncmVk
kind: Secret
metadata:
name: cnpg-bigred
namespace: keycloak
EOF

撰寫 keycloak yaml 檔

cat <<EOF > env/test/keycloak.yaml
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
name: keycloak
namespace: keycloak
spec:
instances: 3
env:
- name: KC_PROXY
value: 'edge'
bootstrapAdmin:
user:
secret: keycloak-bootstrapadmin-user
db:
vendor: postgres
host: pooler-rw.cnpg.svc.cluster.local
usernameSecret:
name: cnpg-bigred
key: username
passwordSecret:
name: cnpg-bigred
key: password
http:
httpEnabled: true
ingress:
enabled: false
hostname:
hostname: keycloak.kubeantony.com
proxy:
headers: xforwarded
EOF

撰寫 keycloak 憑證 yaml 檔

cat <<EOF > env/test/keycloak-cert-manager-cert.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-issuer
namespace: cert-manager
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: root-ca
namespace: cert-manager
spec:
isCA: true
duration: 87600h # 10 years
renewBefore: 720h # 30d
commonName: root-ca
secretName: root-ca-secret
issuerRef:
name: selfsigned-issuer
kind: Issuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: keycloak-ca-clusterissuer
spec:
ca:
secretName: root-ca-secret
EOF

撰寫 keycloak ingress yaml 檔

cat <<EOF > env/test/keycloak-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: keycloak
namespace: keycloak
annotations:
cert-manager.io/cluster-issuer: keycloak-ca-clusterissuer
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-redirect: 'true'
nginx.ingress.kubernetes.io/proxy-body-size: '0'
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "INGRESS_ROUTE"
nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"
nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
spec:
ingressClassName: nginx
tls:
- hosts:
- keycloak.kubeantony.com
secretName: keycloak-tls
rules:
- host: keycloak.kubeantony.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: keycloak-service
port:
number: 8080
EOF

撰寫 keycloak kustomize yaml 檔

cat <<EOF > env/test/kustomization.yaml
kind: Kustomization
resources:
- ./keycloak-ns.yaml
- ./cnpg-user-secret.yaml
- ./keycloak-bootstrapAdmin-user-secret.yaml
- ./keycloak-cert-manager-cert.yaml
- ./keycloak.yaml
- ./keycloak-ingress.yaml
EOF

測試部署 keycloak

kubectl apply -k env/test/ --dry-run=server

執行結果：

namespace/keycloak unchanged (server dry run)
secret/cnpg-bigred created (server dry run)
secret/keycloak-bootstrapadmin-user created (server dry run)
certificate.cert-manager.io/root-ca unchanged (server dry run)
clusterissuer.cert-manager.io/keycloak-ca-clusterissuer unchanged (server dry run)
issuer.cert-manager.io/selfsigned-issuer unchanged (server dry run)
keycloak.k8s.keycloak.org/keycloak created (server dry run)
ingress.networking.k8s.io/keycloak created (server dry run)

正式部署 keycloak

kubectl apply -k env/test/

執行結果：

namespace/keycloak unchanged
secret/cnpg-bigred created
secret/keycloak-bootstrapadmin-user created
certificate.cert-manager.io/root-ca created
clusterissuer.cert-manager.io/keycloak-ca-clusterissuer created
issuer.cert-manager.io/selfsigned-issuer created
keycloak.k8s.keycloak.org/keycloak created
ingress.networking.k8s.io/keycloak created

確認 keycloak 部署完成

kubectl -n keycloak rollout status statefulset keycloak

執行結果：

Waiting for 3 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 2 pods to be ready...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
partitioned roll out complete: 3 new pods have been updated...

確認 keycloak pods 運作狀態

kubectl -n keycloak get pods

執行結果：

NAME READY STATUS RESTARTS AGE
keycloak-0 1/1 Running 0 2m38s
keycloak-1 1/1 Running 0 117s
keycloak-2 1/1 Running 0 81s
keycloak-operator-59c9ff54-h9wvv 1/1 Running 0 3h

檢視 keycloak ingress

kubectl -n keycloak get ing

執行結果：

NAME CLASS HOSTS ADDRESS PORTS AGE
keycloak nginx keycloak.kubeantony.com 10.200.8.24,10.200.8.25,10.200.8.26 80, 443 4d3h

打開瀏覽器，連線至 keycloak 管理網站，URL 是 ingress 的 HOSTS 的值，我的文章是 keycloak.kubeantony.com

登入 keycloak 管理網站，執行以下命令獲取帳號密碼

kubectl -n keycloak get secrets keycloak-bootstrapadmin-user -o go-template='User: {{.data.username | base64decode}}{{"\n"}}Pass: {{.data.password | base64decode}}{{"\n"}}'

執行結果：

User: admin
Pass: admin

按 Sign In 按鈕後，登入畫面如下：

6. 初始化 Keycloak 設定

建立永久的 admin 帳號左側選單 Users -> Add User 在 Username 欄位輸入：administrator，然後點選 Create 按鈕
幫 admin 帳號設定永久有效的密碼點選 Credentials 按鈕 -> Set password 按鈕在 Password 欄位輸入：password 在 Password confimation 欄位再次輸入：password 點選 Save 按鈕點選 Save password 按鈕
設定 admin 權限給 administrator 使用者點選 Role mapping 按鈕 -> Assign role 按鈕 -> Realm roles 按鈕將所有權限都打勾，然後點選左下角 Assign 按鈕

設定 Keycloak 作為 Kubernetes API Server 的認證服務

1. Keycloak 先備知識

在 Keycloak 中有以下幾個主要概念：

領域（realms）：領域管理一批使用者、憑證、角色、群組等等，不同領域之間的資源是相互隔離的，實現了多租戶的效果。
客戶端（clients）：需接取 Keycloak 實現使用者認證的應用程式和服務，以下實作範例的 kube-apiserver 就會是 client。
使用者（users）：使用者是能夠登入應用程式系統的實體，擁有相關的屬性，例如電子郵件、使用者名稱、地址、電話號碼和生日等等。
群組（groups）：一組使用者的集合，你可以將一系列的角色賦予定義好的使用者群組，一旦某使用者屬於該使用者群組，那麼該使用者將獲得對應群組的所有角色權限。
證書（credential）：Keycloak 用於驗證使用者的憑證，例如密碼、一次性密碼、證書、指紋等等。

2. 建立 Keycloak Realm

點選左側選單 Manage realms 按鈕 -> Create realm 按鈕
Realm name 欄位輸入：kubernetes-project 點選左下角 Create 按鈕

3. 建立 keycloak User

點選左側選單 Users 按鈕 -> Create new user
Username 欄位輸入：bigred Email 欄位輸入：bigred@example.com First name 欄位輸入：bigred Last name 欄位輸入：bigred 點選下方 Create 按鈕
點選 Credentials 按鈕 -> Set password 按鈕
Password 欄位輸入: bigred Password confirmation: bigred Temporary 調整成 Off 點選 Save 按鈕
點選 Save password 按鈕
點選左側選單 Users 按鈕 -> Create new user
Username 欄位輸入：rbean Email 欄位輸入：rbean@example.com First name 欄位輸入：rbean Last name 欄位輸入：rbean 點選下方 Create 按鈕
點選 Credentials 按鈕 -> Set password 按鈕
Password 欄位輸入: rbean Password confirmation: rbean Temporary 調整成 Off 點選 Save 按鈕
點選 Save password 按鈕

4. 建立 keycloak Group

點選左側選單 Groups 按鈕 -> Create group 按鈕
Name 欄位輸入：admin Description 欄位輸入：Kubernetes cluster admin 點選 Create 按鈕
點選 admin
點選 Members 按鈕 -> Add member 按鈕
勾選 bigred user -> 點選下方 Add 按扭
點選左側選單 Groups 按鈕 -> Create group 按鈕
Name 欄位輸入：developers Description 欄位輸入：Kubernetes namespace admin 點選 Create 按鈕
點選 developers
點選 Members 按鈕 -> Add member 按鈕
勾選 rbean user -> 點選下方 Add 按扭

5. 建立 Keycloak Client

在 kubernetes-project Realm 建立 Keycloak Client 點選左側選單 Clients 按鈕 -> Create client 按鈕
Client ID 欄位輸入：kube-apiserver Name 欄位輸入：kube-apiserver 點選 Next 按鈕
勾選 Client authentication 點選 Next 按鈕
Valid redirect URIs 欄位輸入：urn:ietf:wg:oauth:2.0:oob 點選 Save 按鈕

什麼是 urn:ietf:wg:oauth:2.0:oob ? 當使用此重新導向 URI 時，Keycloak 會顯示一個頁面，授權碼 (code) 會同時出現在網頁標題 (title) 以及頁面上的文字方塊中。應用程式可以偵測瀏覽器標題的變更，或者由使用者手動複製該代碼並貼上至應用程式中。透過此重新導向 URI，使用者還可以使用不同的裝置來取得代碼，再將其貼回原本的應用程式。

6. 添加 Group Membership 類型的 mapper

在 Keycloak 的 Client 設定中，Mapper 的核心功能可以簡單概括為：「決定 Token 裡面要包含什麼資料，以及這些資料要叫什麼名字」。

當使用者登入成功後，Keycloak 會發發 Token（如 ID Token 或 Access Token）給應用程式 (Client)。預設情況下，Token 裡只會有基本資料（如 username, email, sub）。

Group Membership Mapper: 把使用者所屬的群組 (Groups) 放進 Token。在 Kubernetes OIDC 整合，這個非常重要，因為 K8s 通常是靠這個欄位來綁定 RBAC 權限的。

點選 Client scopes 按鈕 -> kube-apiserver-dedicated 按鈕
點選 Configure a new mapper 按鈕
點選 Group Membership 按鈕
Name 欄位輸入：kube-groups Token Claim Name 欄位輸入：kube-groups Full group path 設定為 Off Add to ID token 保持為 On Add to access token 設定為 Off Add to lightweight access token 保持為 Off Add to userinfo 設定為 Off Add to token introspection 設定為 Off

7. 延長 Token 時間 (可跳過)

Keycloak 中設定的 access_token 和 id_token 的有效期限預設是 5 分鐘，為了方便後續的實作，這裡將令牌的有效期延長至 30 分鐘。

點選左側選單 Realm settings 按鈕 -> 點選右側選單 Access tokens 按鈕 Access Token Lifespan 預設為 5 分鐘
將 Access Token Lifespan 調整成 30 分鐘點選下方 Save 按鈕

8. 查看 Keycloak Realm Endpoint 資訊

點選左側選單 Realm settings 按鈕 -> General 按鈕 -> Endpoints 欄位的 OpenID Endpoint Configuration 超連結按鈕
勾選美化排版，記住 issuer 欄位的 url，稍後會用到

9. 永久設定 Kube-apiserver 參數 (K8s 版本升級時設定不會被原始值覆蓋)

建立並切換工作目錄

mkdir -p ${HOME}/k8s/kubeadm/config/; cd ${HOME}/k8s/kubeadm/config/

匯出當前 K8s 叢集的設定檔 kubeadm-config

kubectl -n kube-system get cm kubeadm-config -o yaml > kubeadm-config-$(date '+%Y-%m-%d').yaml

備份 K8s 叢集的設定檔

cp kubeadm-config-$(date '+%Y-%m-%d').yaml kubeadm-config-$(date '+%Y-%m-%d').yaml.bk

修改 K8s 叢集的設定檔，將 kube-apiserver 啟用 OpenID Connect 認證
```
nano kubeadm-config-$(date '+%Y-%m-%d').yaml
```
需修改的檔案內容如下：
```
data:
ClusterConfiguration: |
apiServer:
extraArgs:
- name: oidc-issuer-url
value: "https://keycloak.kubeantony.com/realms/kubernetes-project"
- name: oidc-client-id
value: "kube-apiserver"
- name: oidc-username-claim
value: "preferred_username"
- name: oidc-groups-claim
value: "kube-groups"
- name: oidc-ca-file
value: "/etc/kubernetes/pki/keycloak-ca.crt"
```
需要在 API Server 的啟動參數中新增下列設定：
- --oidc-issuer-url：OpenID 發行者的 URL，僅接受 HTTPS 協定。若有設定，將用於驗證 OIDC JSON Web Token (JWT)。
  - 在步驟 8 查看 Keycloak Realm Endpoint 資訊有得到這個值
- --oidc-client-id：OpenID Connect 客戶端的 Client ID。若已設定 oidc-issuer-url，則必須設定此項。
  - 在步驟 5 建立 Keycloak Client 中有設定 client id 的值
- --oidc-username-claim：用來作為使用者名稱的 OpenID claim。請注意，除了預設值 (sub) 以外的宣告，無法保證其唯一性與不可變更性。
  - 請依照以下步驟查看
    1. 點選左側選單 Client scopes 按鈕 -> profile 按鈕
    2. 點選 Mappers 按鈕 -> username 按鈕
    3. 確認 Token Claim Name 欄位的值，本篇文章為 preferred_username
- --oidc-groups-claim: 若有提供，此為用來指定使用者群組的自訂 OpenID Connect claim 名稱。該宣告的值預期為字串或字串陣列。
  - 在步驟 6 添加 Group Membership 類型的 mapper 中有設定 Token Claim Name
- --oidc-ca-file：若有設定，OpenID 伺服器的憑證將由 oidc-ca-file 中的憑證授權單位 (CA) 進行驗證；否則，將使用主機本身的根憑證庫 (Root CA set)。

獲取由 cert-manager 幫 keycloak 簽發的 Root CA 憑證

kubectl -n cert-manager get secret root-ca-secret \
-o jsonpath='{.data.ca\.crt}' | base64 -d \
> ${HOME}/k8s/kubeadm/addon/keycloak/env/test/ca.pem

將 keycloak Root CA 憑證配發至 K8s 叢集中的每一台 Control-plane node

scp ${HOME}/k8s/kubeadm/addon/keycloak/env/test/ca.pem <user>@<control-plane node 1 ip>:./ca.pem
scp ${HOME}/k8s/kubeadm/addon/keycloak/env/test/ca.pem <user>@<control-plane node 2 ip>:./ca.pem
scp ${HOME}/k8s/kubeadm/addon/keycloak/env/test/ca.pem <user>@<control-plane node 3 ip>:./ca.pem

在每一台 Control-plane node 將 keycloak Root CA 憑證拷貝到先前 K8s 叢集設定檔中指定的目錄區

ssh <user>@<control-plane node 1 ip>
sudo cp ${HOME}/ca.pem /etc/kubernetes/pki/keycloak-ca.crt
ssh <user>@<control-plane node 2 ip>
sudo cp ${HOME}/ca.pem /etc/kubernetes/pki/keycloak-ca.crt
ssh <user>@<control-plane node 3 ip>
sudo cp ${HOME}/ca.pem /etc/kubernetes/pki/keycloak-ca.crt

回到 K8s 管理主機，模擬套用 Kubernetes 叢集設定檔，檢查設定檔語法是否正確以及是否能被叢集接受

kubectl apply -f kubeadm-config-$(date '+%Y-%m-%d').yaml --dry-run=server

執行結果：

Warning: resource configmaps/kubeadm-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
configmap/kubeadm-config configured (server dry run)

確認沒問題後，正式套用 Kubernetes 叢集設定檔

kubectl apply -f kubeadm-config-$(date '+%Y-%m-%d').yaml

執行結果：

Warning: resource configmaps/kubeadm-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
configmap/kubeadm-config configured

將叢集現有的 K8s 叢集設定檔提取出來，並儲存為本機檔案 config.yaml

kubectl -n kube-system get cm kubeadm-config \
-o jsonpath={.data.ClusterConfiguration} > /tmp/config.yaml

將 config.yaml 派送至每一台 Control-plane node 的 /tmp 目錄區底下

scp /tmp/config.yaml <user>@<control-plane node 1 ip>:/tmp/
scp /tmp/config.yaml <user>@<control-plane node 2 ip>:/tmp/
scp /tmp/config.yaml <user>@<control-plane node 3 ip>:/tmp/

在每一台 Control-plane node 修改 API Server 的啟動參數

ssh <user>@<control-plane node 1 ip>
sudo kubeadm init phase control-plane apiserver --config /tmp/config.yaml
ssh <user>@<control-plane node 2 ip>
sudo kubeadm init phase control-plane apiserver --config /tmp/config.yaml
ssh <user>@<control-plane node 3 ip>
sudo kubeadm init phase control-plane apiserver --config /tmp/config.yaml

會在每一台 Control-plane 看到以下執行結果：

[control-plane] Creating static Pod manifest for "kube-apiserver"

回到 K8s 管理主機，確認 kube-apiserver pods 是否有重啟，並且狀態正常

kubectl -n kube-system get pods -l "component=kube-apiserver"

執行結果：

NAME READY STATUS RESTARTS AGE
kube-apiserver-kubeadm-c1 1/1 Running 0 104s
kube-apiserver-kubeadm-c2 1/1 Running 0 58s
kube-apiserver-kubeadm-c3 1/1 Running 0 50s

10. 設定 K8s RBAC

建立 developers 群組專用的 K8s namespace
```
kubectl create ns project-1
```

設定 developers 群組為特定 namespace 的 admin

kubectl create rolebinding developers-group-ns-admin \
--clusterrole=admin \
--group=developers \
--namespace=project-1

設定 admin 群組為 K8s 叢集管理員

kubectl create clusterrolebinding admin-group-cluster-admin \
--clusterrole=cluster-admin \
--group=admin

11. 使用 `Kubelogin` 進行身份驗證 - 搭配瀏覽器模式

當執行 kubectl 指令時，kubelogin 會給一個超連結，當使用瀏覽器連線到該連結的網站，使用者需要輸入使用者名稱和密碼登入程序，認證通過後，kubelogin 會從認證伺服器取得一個 token，然後 kubectl 就可以使用該 token 和 K8s API Server 進行溝通，具體的流程圖如下：

安裝 krew kubectl plugin

(
set -x; cd "$(mktemp -d)" &&
OS="$(uname | tr '[:upper:]' '[:lower:]')" &&
ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" &&
KREW="krew-${OS}_${ARCH}" &&
curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" &&
tar zxvf "${KREW}.tar.gz" &&
./"${KREW}" install krew
)
echo 'export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

安裝 kubelogin kubectl plugin

kubectl krew install oidc-login

執行結果：

Updated the local copy of plugin index.
Installing plugin: oidc-login
Installed plugin: oidc-login
\
| Use this plugin:
| kubectl oidc-login
| Documentation:
| https://github.com/int128/kubelogin
| Caveats:
| \
| | You need to setup the OIDC provider, Kubernetes API server, role binding and kubeconfig.
| /
/
WARNING: You installed plugin "oidc-login" from the krew-index plugin repository.
These plugins are not audited for security by the Krew maintainers.
Run them at your own risk.

設定 kubectl 使用 oidc-test 帳號時，透過 OIDC 協議與 Keycloak 進行互動式認證以取得存取 Kubernetes 的 Token。
```
kubectl config set-credentials oidc-web \
--exec-api-version=client.authentication.k8s.io/v1 \
--exec-interactive-mode=IfAvailable \
--exec-command=kubectl \
--exec-arg=oidc-login \
--exec-arg=get-token \
--exec-arg="--oidc-issuer-url=https://keycloak.kubeantony.com/realms/kubernetes-project" \
--exec-arg="--oidc-client-id=kube-apiserver" \
--exec-arg="--oidc-client-secret=EM06cnVYdLFjRa46DDgoBRl4NL7zhMxJ" \
--exec-arg="--oidc-redirect-url=urn:ietf:wg:oauth:2.0:oob" \
--exec-arg="--certificate-authority=/etc/kubernetes/pki/keycloak-ca.crt" \
--exec-arg="--grant-type=authcode-keyboard"
```
- kubectl config set-credentials oidc-web：
  - 在 kubeconfig 檔案中建立或更新名為 oidc-web 的使用者條目。
- --exec-api-version & --exec-command：
  - 指定使用 Kubernetes 的 client.authentication.k8s.io/v1 API，並呼叫 kubectl 作為執行檔（實際是透過它來呼叫插件）。
- --exec-arg=oidc-login, get-token：
  - 指定實際執行的插件指令為 oidc-login (即 kubelogin)，動作為獲取 Token。
- --exec-arg="--oidc-issuer-url/client-id/secret"：
  - 提供連線到 OIDC Provider (此處為 Keycloak) 所需的端點、客戶端 ID 與 secret。
- --exec-arg="--grant-type=authcode-keyboard"：
  - 指定認證流程為「鍵盤輸入授權碼」。使用者在瀏覽器登入後，必須手動複製授權碼 (Auth Code) 並貼回終端機，而非透過本機 Port 自動回傳。
- --exec-arg="--oidc-redirect-url=urn:ietf:wg:oauth:2.0:oob"：
  - 配合上述模式，告訴 Keycloak 驗證完成後顯示 Out-of-Band (OOB) 頁面（通常是顯示一段代碼），而非跳轉到特定的 URL。
- 請執行以下步驟得到 oidc-client-secret 的值
  1. 點選左側選單 Clients 按鈕 -> kube-apiserver 按鈕
  2. 點選 Credentials 按鈕 -> Client Secret 欄位右側的複製按鈕
執行結果：
```
User "oidc-web" set.
```

驗證存取 K8s

kubectl --user=oidc-web auth whoami

執行結果：

Please visit the following URL in your browser: https://keycloak.kubeantony.com/realms/kubernetes-project/protocol/openid-connect/auth?access_type=offline&client_id=kube-apiserver&code_challenge=P4RQKK8ISp0lNuDjEdFh21JzyP-mXZvmw9NxPhgie8Y&code_challenge_method=S256&nonce=YXM_s1sD3u6mmsLQpe_Zg1Nsem-0OhBodCbORiJO5T4&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=openid&state=UYOI-OojjJRVk1rWr4m_BxVnHuu46ratNYmf2FZMWAA
Enter code:

記得先在 keycloak 登出 administrator 帳號連線到命令給的驗證連結，並輸入以下內容：

Username or email: bigred
Password: bigred 點選 Sign In 按鈕就會得到驗證碼，點擊右側的複製按鈕

將驗證碼貼到終端機上

Enter code: 229cea71-4d33-8bc1-4662-704dff97a9ac.172198d7-1020-1881-90f3-e75d092fb006.31751bd5-670a-4446-a183-6ea3386ea8e2

執行結果：

ATTRIBUTE VALUE
Username https://keycloak.kubeantony.com/realms/kubernetes-project#bigred
Groups [admin system:authenticated]
Extra: authentication.kubernetes.io/credential-id [JTI=89287e03-0da6-d421-ea72-7adfd388e894]

確認 bigred 使用者是否具備整個 Kubernetes 叢集的完整超級管理員（cluster-admin）權限
```
kubectl --user=oidc-web auth can-i '*' '*' -A
```
執行結果
```
yes
```

12. 使用 `Kubelogin` 進行身份驗證 - 純命令模式

設定 keycloak client 允許 Direct access grants 點選左側選單 Clients 按鈕 -> kube-apiserver 點選右側選單 Capability config 按鈕 -> 勾選 Direct access grants 點選下方 Save 按鈕
刪除 token cache
```
kubectl oidc-login clean
```

測試使用命令模式登入

kubectl config set-credentials oidc-cli \
--exec-api-version=client.authentication.k8s.io/v1 \
--exec-interactive-mode=IfAvailable \
--exec-command=kubectl \
--exec-arg=oidc-login \
--exec-arg=get-token \
--exec-arg="--oidc-issuer-url=https://keycloak.kubeantony.com/realms/kubernetes-project" \
--exec-arg="--oidc-client-id=kube-apiserver" \
--exec-arg="--oidc-client-secret=EM06cnVYdLFjRa46DDgoBRl4NL7zhMxJ" \
--exec-arg="--certificate-authority=/etc/kubernetes/pki/keycloak-ca.crt" \
--exec-arg="--grant-type=password"

--exec-arg="--grant-type=password"：指定認證流程為 ROPC 模式。 kubectl 在執行時，直接在終端機提示使用者輸入 Username 和 Password，並直接將這些憑據發送給 Keycloak 進行驗證

執行結果：

User "oidc-cli" set.

驗證存取 K8s

kubectl --user=oidc-cli auth whoami

執行結果：

Username: rbean
Password: rbean
ATTRIBUTE VALUE
Username https://keycloak.kubeantony.com/realms/kubernetes-project#rbean
Groups [developers system:authenticated]
Extra: authentication.kubernetes.io/credential-id [JTI=43a41723-b6d7-95fe-543a-d8bf5069e6cb]

驗證 rbean 使用者是否能在其他 K8s namespace 建立 deployment
```
kubectl --user=oidc-cli auth can-i create deployment -n default
```
執行結果：
```
no
```
驗證 rbean 使用者是否能在 project-1 namespace 建立 deployment
```
kubectl --user=oidc-cli auth can-i create deployment -n project-1
```
執行結果：
```
yes
```

確認 rbean 使用者在其他 K8s namespace 擁有的權限

kubectl --user=oidc-cli auth can-i --list -n default

執行結果：

Resources Non-Resource URLs Resource Names Verbs
selfsubjectreviews.authentication.k8s.io [] [] [create]
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version/] [] [get]
[/version] [] [get]
[/version] [] [get]

確認 rbean 使用者是否能獲得 K8s 叢集節點資訊

kubectl --user=oidc-cli auth can-i get nodes

執行結果：

Warning: resource 'nodes' is not namespace scoped
no

將 oidc-cli user 的 kubeconfig 獨立出來

kubectl config set-context oidc-context \
--cluster=topgun \
--user=oidc-cli \
--namespace=project-1
kubectl config view \
--context=oidc-context \
--minify \
--flatten \
> oidc-cli.kubeconfig

檢視 oidc-cli user 的 kubeconfig

kubectl --kubeconfig oidc-cli.kubeconfig config view

執行結果:

apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://192.168.11.176:6443
name: topgun
contexts:
- context:
cluster: topgun
namespace: project-1
user: oidc-cli
name: oidc-context
current-context: oidc-context
kind: Config
users:
- name: oidc-cli
user:
exec:
apiVersion: client.authentication.k8s.io/v1
args:
- oidc-login
- get-token
- --oidc-issuer-url=https://keycloak.kubeantony.com/realms/kubernetes-project
- --oidc-client-id=kube-apiserver
- --oidc-client-secret=EM06cnVYdLFjRa46DDgoBRl4NL7zhMxJ
- --certificate-authority=/etc/kubernetes/pki/keycloak-ca.crt
- --grant-type=password
command: kubectl
env: null
interactiveMode: IfAvailable
provideClusterInfo: false

可以看到沒有 user 的 client-certificate-data 跟 client-key-data，變成動態的到 keycloak 去取得 token

參考文章

HA K8s Control-plane components 之間如何運作？

Tue, 09 Dec 2025 11:50:46 +0800

1. Preface

本篇文章會深度探討在高可用的 Kubernetes 叢集中，Control-plane nodes 上的 Components (kube-apiserver, etcd, …等)之間會如何運作，並借由 kubectl create 建立一個 Deployment 來實踐理論。

2. 先備知識

由以上 K8s 官網文件中的叢集架構圖可以看到單一台 Control-plane 內部的元件有 4 個，讓我們先來認識他們：

kube-apiserver : 負責接受前端管理者透過 CLI ( Command Line Interface, 命令模式 ) 或是在 Web UI 上面點點選選的方式下達的命令，並且 kube-controller-manager 和 kube-scheduler 要正常運作也需要對它呼叫 API Request。
- 在接收到 API Request 後，還會對這個 request 進行身份的驗證，並確保該身份有足夠的權限可以執行 request 內要做的動作。
kube-controller-manager：負責維護 K8s 叢集的狀態，它會試圖將當前叢集狀態移動到更接近所需狀態的地方，例如偵測節點是否故障、偵測 Pod 是否毀損等，並作出回應，此時期望的狀態是節點跟 pod 是否處於健康，Controller-manager 會去確認這些狀態，並在故障時做出對應的處理。
- 在這個元件裡面內定會執行很多個 controller processes。從邏輯上講，每個 controller 都是一個單獨的程序，但是為了降低複雜性，它們都被編譯到同一個可執行檔案，並在一個程序中執行，所以它會被稱為 manager。
- controller process 可以想像成執行 bash script while true 迴圈的一個程序，裡面會去比對 node, pod, …等物件當前的狀態與期望的狀態是否一致。
kube-scheduler : 負責監控新建立的 Pod ，選擇出最適合的節點讓 Pod 在上面運作。排程決策會有兩個階段，
1. Predicates (過濾) 階段：會先根據 pod 的規格檢查有哪些節點有足夠的硬體資源適合執行這個 pod。
2. Priorities（評分) 階段：負責對通過 Predicates 的節點進行評分 (0-10 分），最終選擇分數最高的節點。
etcd：負責儲存 K8s 叢集的狀態，也就是 K8s 運作時產生的重要資料都會

Running Nginx with Kubernetes Restricted Pod Security Standards

Fri, 05 Dec 2025 14:11:54 +0800

1. 先決條件

K8s 已安裝完畢
請使用 kubernetes-admin 的 user 操作 K8s
作業系統已安裝 podman，並且可以上網

2. 建立 Nginx App Container Image

創建並切換工作目錄

mkdir -p "$HOME"/wulin/nginx; cd "$HOME"/wulin/nginx

撰寫 Dockerfile

nano Dockerfile

檔案內容：

# 使用輕量版 Alpine 且包含 OpenTelemetry 支援的 Nginx 基礎映像
FROM docker.io/library/nginx:1.29.3-alpine-otel
# 定義參數
ARG user=bigred
ARG uid=1001
ARG gid=1001
ARG port=8080
# 合併 RUN 指令以減少圖層 (Layers) 並縮小體積
# 使用 set -e 確保任何指令失敗時停止建置
RUN set -e && \
# 1. 新增使用者與群組
addgroup -g ${gid} -S ${user} && \
adduser -u ${uid} -S -G ${user} ${user} && \
# 2. 建立必要的暫存與 PID 目錄
mkdir -p /var/run/nginx \
/var/cache/nginx/client_temp \
/var/cache/nginx/proxy_temp \
/var/cache/nginx/fastcgi_temp \
/var/cache/nginx/scgi_temp \
/var/cache/nginx/uwsgi_temp && \
# 3. 修改設定檔
sed -i -E "s/listen\s+80;/listen ${port};/g" /etc/nginx/conf.d/default.conf && \
sed -i "s|/var/run/nginx.pid|/var/run/nginx/nginx.pid|g" /etc/nginx/nginx.conf && \
# 4. 權限調整
chown -R ${uid}:${gid} \
/var/cache/nginx \
/var/run/nginx \
/var/log/nginx \
/run && \
# 5. 清理不必要的暫存檔 (Alpine 特性，若有安裝套件時使用)
rm -rf /tmp/*
# 定義 run 成 app container 後內定執行的使用者
USER ${user}
# 宣告對外開的 tcp port number
EXPOSE ${port}
# 定義 run 成 app container 後內定執行的啟動命令
CMD ["nginx", "-g", "daemon off;"]

建立 App Container Image

podman build --build-arg user=bigred \
--build-arg uid=1001 \
--build-arg gid=1001 \
--build-arg port=8080 \
--squash \
--tag nginx:1.29.3-alpine-otel-nonroot .

執行 Nginx App Container

podman run --name nginx \
--detach \
--publish 8080:8080 \
localhost/nginx:1.29.3-alpine-otel-nonroot

檢視 App Container 運作狀態

podman ps -a

執行結果：

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
28bde4893f9b localhost/nginx:1.29.3-alpine-otel-nonroot nginx -g daemon o... 1 second ago Up 2 seconds 0.0.0.0:8080->8080/tcp nginx

驗證 Nginx 網站是否可以正常連接
```
curl -sI localhost:8080 | head -n 1
```
執行結果：
```
HTTP/1.1 200 OK
```
登入 quay.io
```
podman login quay.io
```

修改 Nginx Image tag

podman tag localhost/nginx:1.29.3-alpine-otel-nonroot quay.io/hahappyman/nginx:1.29.3-alpine-otel-nonroot

將 Nginx Image 上傳到 quay.io

podman push quay.io/hahappyman/nginx:1.29.3-alpine-otel-nonroot

3. 設定 Namespace 強制執行 `Restricted` Pod Security Standards (PSS)

建立 namespace
```
kubectl create ns mytest
```

設定 Namespace 強制執行 Restricted Pod Security Standards

kubectl label --overwrite ns mytest \
pod-security.kubernetes.io/enforce=restricted \
pod-security.kubernetes.io/enforce-version=v1.34

檢視是否成功

kubectl describe ns mytest

執行結果：

Name: mytest
Labels: kubernetes.io/metadata.name=mytest
pod-security.kubernetes.io/enforce=restricted
pod-security.kubernetes.io/enforce-version=v1.34
Annotations: <none>
Status: Active
No resource quota.
No LimitRange resource.

4. 將 Nginx 部署至強制執行 `Restricted` PSS 的 namespace

撰寫 nginx pod yaml

nano nginx-pod.yaml

檔案內容如下：

apiVersion: v1
kind: Pod
metadata:
name: restricted-nginx-pod
namespace: mytest
labels:
app: restricted-nginx
spec:
containers:
- name: restricted-nginx-container
image: quay.io/hahappyman/nginx:1.29.3-alpine-otel-nonroot
securityContext:
privileged: false
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
runAsNonRoot: true
runAsUser: 1001
seccompProfile:
type: "RuntimeDefault"

注意，runAsNonRoot 會用 user id 判斷 App Container 是否有透過非 root 的帳號去執行，所以當 App Container image 內定執行的使用者指定的是 user name 不是 user id，就必須要再額外設定 runAsUser 指定 uid 去執行，否則你會遇到以下錯誤訊息：

Error: container has runAsNonRoot and image has non-numeric user (bigred), cannot verify user is non-root (pod: "restricted-nginx-pod_mytest(d40b3343-1027-48c2-9b7d-e9acac1c7524)", container: restricted-nginx-container)

部署 Nginx Pod

kubectl apply -f nginx-pod.yaml

執行結果：

pod/restricted-nginx-pod created

確認 Nginx Pod 運作狀態是否正常

kubectl -n mytest get pods -o wide

執行結果：

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
restricted-nginx-pod 1/1 Running 0 22s 10.244.3.113 kubeadm-w1 <none> <none>

驗證網站是否能正常連線

curl -sI http://10.244.3.113:8080 | head -n 1

執行結果：

HTTP/1.1 200 OK

在 nginx 所在的 worker node 上查詢 container user uid

sudo crictl inspect --name restricted-nginx-container | jq '.info.runtimeSpec.process.user.uid'

執行結果：

實作 Cilium BGP

Sat, 29 Nov 2025 13:19:52 +0800

1. Create Kind Cluster

nano cluster.yaml

檔案內容如下：

透過 VMware Aria Operations 監控 Kubernetes

Fri, 14 Nov 2025 08:01:36 +0800

1. 先決條件

在使用 VMware Aria Operations Management Pack for Kubernetes 來監控 Kubernetes 叢集之前，您必須先準備好您的 VMware Aria Operations 環境。您必須確保 VMware Aria Operations 符合下列一般需求。

確保您已安裝 VMware Aria Operations (SaaS) 或 VMware Aria Operations 8.1 或更高版本。
確認您已安裝 VMware Aria Operations Management Pack for Kubernetes。
確保您的環境中已部署一個 Kubernetes 叢集。
確保您已建立一個服務帳戶，該帳戶具有讀取權限，可用於存取叢集並取得詳細目錄。
決定您的 Kubernetes 叢集的身份驗證模式。
決定您要使用 cAdvisor 還是 Prometheus 來監控您的 Kubernetes 環境。
- 如果您選擇 cAdvisor，請確保 cAdvisor 是以 Kubelet 或 DaemonSet 的形式執行。
- 如果您選擇 Prometheus，請確保您已在叢集內部署 Prometheus，或是在叢集外部署一個獨立的 Prometheus 執行個體。

2. 環境準備

2.1. 在 Kubernetes 中安裝 VMware Aria Operations Management Pack

2.1.1. 先決條件

已安裝 Kubernetes
已安裝 VMware Aria Operations
確認擁有最新版本的管理套件，且該管理套件與您的 VMware Aria Operations 版本相容。
確保擁有安裝管理套件的權限。

2.1.2. 開始安裝

從 VMware Marketplace 下載該管理套件的 PAK 檔案。
使用管理員權限登入 VMware Aria Operations。
從左側選單中，點擊 Data Sources > Integrations。前往右側窗格中的 Repository 索引標籤。
在 Repository 索引標籤上，點擊 Add。
此時會出現「Add Solutions Wizard Options」(新增解決方案精靈選項)。該精靈包含三個頁面，您可以在此尋找並上傳 PAK 檔案、接受 EULA (使用者授權合約) 並進行安裝，以及檢視安裝過程。點擊 Browse (瀏覽) 以找到您的暫存資料夾並選取 PAK 檔案。
- 在升級至最新版本時，您可以勾選 Install the PAK file even if it is already installed (即使已安裝也要安裝 PAK 檔案) 和 Reset Default Content (重設預設內容) 這兩個選項。
點擊 Upload (上傳)。上傳過程可能需要幾分鐘。
閱讀並接受 EULA，然後點擊 Next (下一步)。
當 VMware Aria Operations Management Pack for Kubernetes 安裝完成後，點擊 Finish (完成)。
從 Data Sources -> Integrations -> ADD → Kubernetes 索引標籤設定此項整合。
先在 K8s 建立一個 read only 的 serviceAccount

cat <<EOF | kubectl apply -f -
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: '*'
name: vrops-read-only
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["get", "list", "watch"]
- nonResourceURLs:
- /
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: vrops-mp-read-only-user
namespace: default
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: vrops-mp-read-only-user-token
namespace: default
annotations:
kubernetes.io/service-account.name: "vrops-mp-read-only-user"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vrops-mp-read-only-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: vrops-read-only
subjects:
- kind: ServiceAccount
name: vrops-mp-read-only-user
namespace: default
EOF

獲取 sa 的 token

echo $(kubectl -n default get secret vrops-mp-read-only-user-token -o jsonpath='{.data.token}' | base64 --decode )

執行結果：

eyJhbGciOiJSUzI1NiIsImtpZCI6Ijk5Vi1DZlc5M1Y0eUh5UEtBYTA4bG9OaC1jMFFTT0s1cElCa0lOZ3NuaTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZyb3BzLW1wLXJlYWQtb25seS11c2VyLXRva2VuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZyb3BzLW1wLXJlYWQtb25seS11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDI2MmJmNTctMzA5OC00ZjYxLWFlMDAtNDE3M2UzMmYzZGJjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dnJvcHMtbXAtcmVhZC1vbmx5LXVzZXIifQ.lpM7IEWiiB4sbrCLYB-Re_R4d7-g3sELisU4npjJyp_pwzJB0LehOo-pwnlwWH33FtTOqrsD26hYlYO5POmEpGjAUt49DoL9-SkoOfCjcQy82lGKbj_F6kMVqyddVEcHBIUWIuHAckvekSerLxH1M4dIztBnTFXulKJ_GIY72VA6aQOt0kGX35-dCH8GK-nHUJIu5cKcbrWV1NAfkzLBpFqlkVnjcOgHpf2jXJ3-3ozGQ4wMvCeZlG9YA86rY8Xn1XrlbztQx2klURLuy4b2RgEt7sfZuilrZScrFoPI68a7Av5prb8EQeRnzEZSsagNP5RfH5PG1qmRW64R_BrAlg

依序輸入以下欄位內容：

Cloud Account Information
- Name: topgun
Connection Information
- Control Plane URL: https://172.20.6.10:6443
- Collector Service: Prometheus Server
- Credential: <點選右邊的 + 按鈕>
  - Credential Kind: Token Auth
  - Credential name: kubeadm-prometheus
  - Bearer Token: 將步驟 11 得到的 token 複製貼上
  - Prometheus Server: http://prometheus.bbg.com:80
  - Prometheus endpoint username: bigred
  - Prometheus endpoint password: bigred
- Collector / Group: 選 CloudProxy-xx-xx-xx-xx-xxx
Advanced Settings
- Java Process Monitoring: Activatied
- Delete nonexistent object older than: 2 hours
- cAdvisor Install Check Deactivated
點選左下角 SAVE 按鈕

確認連接成功

ETCD Performance

Tue, 11 Nov 2025 08:36:04 +0800

理解效能

etcd 提供穩定、持續的高效能。有兩個因素定義了效能：延遲 (latency) 和吞吐量 (throughput) :

延遲是完成一個操作所需的時間。
吞吐量是在某個時間段內完成的總操作數。

通常，當 etcd 接受並發的客戶端請求時，平均延遲會隨著整體吞吐量的增加而增加。在常見的雲端環境中，例如 Google Compute Engine (GCE) 上的標準 n-4 機器或 AWS 上同等級的機器類型，一個三成員的 etcd 叢集在輕度負載下，能於一毫秒內完成請求，並可在重度負載下每秒完成超過 30,000 個請求。

etcd 使用 Raft 共識演算法在成員之間複製請求並達成共識。共識效能，尤其是提交延遲 (commit latency)，受到兩個物理限制：網路 IO 延遲和磁碟 IO 延遲。完成一個 etcd 請求的最短時間是成員之間的網路來回時間 (RTT)，加上 fdatasync 將資料提交到永久儲存所需的時間。

資料中心內的 RTT 可能長達數百微秒。在美國境內的典型 RTT 約為 50 毫秒，而在跨洲之間可能慢至 400 毫秒。旋轉磁碟 (spinning disk) 的典型 fdatasync 延遲約為 10 毫秒。對於 SSD (固態硬碟)，延遲通常低於 1 毫秒。為了提高吞吐量，etcd 將多個請求批次處理 (batching) 在一起，並將它們提交給 Raft。這種批次處理策略使 etcd 即使在重度負載下也能達到高吞吐量。

還有其他子系統會影響 etcd 的整體效能。每個序列化的 etcd 請求都必須通過 etcd 的由 boltdb 支援的 MVCC 儲存引擎，這通常需要數十微秒才能完成。etcd 會定期對其最近套用的請求進行增量快照 (snapshot)，將它們與先前的磁碟快照合併。這個過程可能會導致延遲突波 (latency spike)。儘管這在 SSD 上通常不是問題，但在 HDD (傳統硬碟) 上可能會使觀察到的延遲加倍。同樣地，執行中的壓縮 (compaction) 也可能影響 etcd 的效能。幸運的是，這種影響通常微不足道，因為壓縮是交錯執行的，因此不會與常規請求競爭資源。RPC 系統 (gRPC) 為 etcd 提供了一個定義良好、可擴展的 API，但它也引入了額外的延遲，特別是對於本地讀取。

2025 香港 KubeCon 參加心得：Cilium 如何達到高性能網路

Wed, 05 Nov 2025 13:19:02 +0800

聽了很多個議程，挑了以下議程跟大家分享： Simplifying the Networking and Security Stack With Cilium, Hubble, and Tetragon

以下文章皆來自講者分享的內容。

Cilium 如何達到高性能網路

1. 什麼是 eBPF

eBPF 是 Linux Kernel 的技術，你可以編寫一些程序放在 Kernel 當中，當特定的事件發生的時候，Kernel 會自動調用你的程序來實現你想要的功能。

Cilium 其實就是很多 eBPF 的程序來實現網路的轉發、安全性和可觀測。

除了這些之外，Cilium 其實還使用到了 envoy 來實現 Service Mesh 和 7 層相關的功能，根據上圖，大家可以得知，Cilium 已經可以部署到大多數的雲上了，而且大多數的 Kubernetes 的發行版都已經支持了。

2. Cilium features

以上是 Cilium 網路部分支持的功能，今天主要關注的是最左上角高性能網路的部分。

3. Standard Container Networking

Letʼs look at a standard container networking packet path:

The pod to ClusterIP service that backend pod is on the same node.
The pod access the Internet.

提到高性能網路，我們就必須看一下標準的網路轉發路徑，我們在這兩個場景下，就考慮 Kubernetes 的一個節點，然後一個節點就 2 個 pods，左邊的 Pod 就是 Pod A，IP 地址是 1.1.1.1，右邊的 Pod B IP 地址就是 1.1.1.2。

第一個場景是，在同一個 node 中 Pod A 透過 Service IP 存取 Pod B，第二個場景是 Pod A 直接訪問外部互聯網大概是一個什麼樣的場景，網路封包會怎麼流轉。

從 Pod A 假設要訪問 Service IP 是 10.96.0.1 然後網路封包就會從 Pod A 的網卡發送到主機側，主機側會開始進入主機的網路內核站，這個時候它一般會流向有一個表叫 prerouting nat 的表，然後這個地方會有 kubeproxy 生成 Iptable 的規則，把 IP 的目的地址改成 Pod B 的，然後這個封包會繼續流轉，然後在查詢到路油表後，就會發現是去 Pod B 的，然後會轉發到 Pod B 主機側的網卡，主機側的網卡會把封包發給 Pod B，這樣 Pod B 就收到了。

主機側網卡指的是 veth pair 中連接在 Host 主機上那一端的網卡，從技術的角度看，veth pair 是指有一對網卡會跨 Linux Network Namespace 互相連接，一端在 Pod 的 Linux Network Namespace，另一端就是在 Host 主機裡面 root 的 Linux network namespace 中的網卡。

第二個場景是 Pod A 直接連線出外部的互聯網時，Pod A 查路油表時，會發現是出去的流量，一般情況下會有 SNAT 的行為發生，Pod A 的 IP 會變成主機的實體網卡。

4. Iptables and IPVS

通常提到網路優化，很多時候會提到 IPVS 對 Iptables 的優化，

大家都知道 iptables 的查詢複雜程度是隨著服務數量或後端 Pod 數量的增加，查詢時間會線性增長，IPVS 則是在服務數量或後端 Pod 數量增加後，查詢時間並不會受到影響，也就是說大叢集的網路選 ipvs 的效率更高，但是具體情況發生什麼還是稍微有點複雜的。

5. IPVS

下圖就可以看到 IPVS 到底做了哪些事情，其實 IPVS 在這個地方只是替換了 prerouting nat 的表，當叢集很大時 IPVS 確實是有優勢的，問題是它只替換了這一部份，那你的封包還是在主機側的網路內核協議站走了整個內核協議站，這是其一，其二是當叢集沒那麼大時，IPVS 對 Iptables 的優化是未知的，IPVS 在運作上其實也有自己的開銷。

6. Kube-proxy replacement

從 cilium 的角度來看，cilium 有提供一個角度是 kube-proxy replacement，其實就是代替了 Kube-proxy，會有一個 eBPF 的程序掛載到 veth-pair 在主機側的網卡上，當 Pod A 到 Pod B 的流量在轉發時，Pod A 發給 Service IP，然後主機側的網卡收到封包後，會直接查詢的動作，然後把 IP 地址改到 Pod B，改到 Pod B 之後就不再通過整個內核協議站去查詢這些了，因為它就已經知道，要發到哪一張網卡上了，就直接透過 eBPF 程序把封包發給 Pod B 主機側的網卡，然後主機側的網卡就會再把封包發給 Pod B。

大家可以看到這已經跳過了很多內核協議站的網路部分，但因為是 Kube-proxy replacement，它只管理 Service 相關的，但這個時候 Pod A 出向的流量還是會查路由表，有的時候還是會通過 iptables 做 SNAT，在這種情況下。

7. eBPF host routing

Cilium 另一個優化的點其實是叫 eBPF host routing，這個地方一般情況下我們會說它有兩個優化 :

Pod A 出向的流量，會直接在 Pod A 主機側的網卡收到封包後，透過 eBPF 程序查詢，根據路由表，就直接發到出向的實體網卡。
如同剛剛的場景，eBPF 程序收到了 Pod A 的封包後，他不解釋發到 Pod B 主機側的網卡，它其實是直接發到 Pod B 裡面，這樣的話又節省了一次從主機側將封包發給 Pod B 網卡的動作，這樣的話又減少了一點傳輸時間。

以上是兩個常見的優化。

8. Netkit

第三個是叫 Netkit，這是 Cilium 1.16 引入的，它其實是根據 Linux 核心中新的網路類型叫 Netkit 網路類型，這時候 eBPF 程序就掛載到了 pod 側，這種情況下就非常有意思，當 Pod A 發出一個流量到外部時，eBPF 程序就在 Pod 側就已經抓到了程序，可以直接發到物理網卡側，這樣 Pod A 就不需要把封包傳到主機側，這樣的話是一個非常大的優化，但比較可惜的是 Pod A 到 Pod B 的流量，他沒辦法做到 Pod A 到 Pod B 的直接轉發，雖然他在 Pod A 的容器側收到了，還是要發到 Pod B 主機側，還是沒有特別完美的轉發到 Pod B 這側。

在 Kubernetes 中啟用 Audit Log

Wed, 29 Oct 2025 03:09:04 +0800

1. 什麼是 Kubernetes Audit?

Kubernetes Auditing 功能提供了與安全相關的、按時間順序排列的記錄集，記錄每個使用者、使用 Kubernetes API 的應用程式以及控制面自身引發的活動（所有存取 kube-apiserver 服務的客戶端）。

Audit 功能使得 K8s 叢集管理員能夠回答以下問題：

發生了什麼事？
什麼時候發生的？
誰觸發的？
活動發生在哪一個（些）物件上？
在哪觀察到的？
它從哪觸發的？
活動的後續處理行為是什麼？

這對平台管理者來說十分重要，能夠回答一些在故障時候出現的問題。哪個用戶，從哪個 ip 上，發起了什麼請求。如刪除了一個命名空間，導致這個命名空間下的所有 pod 被回收，對故障進行複盤。

Kubernetes Audit 功能由 kube-apiserver 服務提供。每個請求在不同執行階段都會產生 Audit Events；這些 Audit Events 會根據特定 Policy 被預處理並寫入後端。Policy 決定要記錄的內容，目前的後端支援 logs 檔案和 webhook。

每個請求都可被記錄其相關的階段（stage）。已定義的階段有：

RequestReceived - 此階段對應 kube-apiserver audit handler 接收到請求後，且在將請求向下傳遞給處理鏈之前，立即把產生的事件給記錄下來。
ResponseStarted - 在回應訊息的頭部發送後，回應訊息體發送前產生的事件。只有長時間運行的請求（例如 watch）才會產生這個階段。
ResponseComplete - 當回應訊息體完成且沒有更多資料需要傳輸的時候。（白話來說就是 kube-apiserver 已經完全處理完了該客戶端 (client) 提交的請求，在這個階段把 audit log 給記錄下來。）
Panic - 當 panic 發生時產生。

設定 K8s Audit logs 注意事項：

audit logs 功能會增加 API Server 的記憶體消耗量，因為 audit 所需的部分上下文資訊 (context) 會針對每個請求儲存起來。記憶體消耗量取決於 audit logs 的設定。
開啟 K8s 原生 Audit 功能必須設定 Audit Stage 和 Audit backend 否則不會執行 Audit Filter 邏輯。
根據 Audit Policy 設定檔中的 omitStages 的定義值，不設定 omitStages 的話預設 RequestReceived、ResponseStarted、ResponseComplete 這三個階段都會產生 audit events，在保留的 Audit Stage 產生 audit events 並輸出到對應後端。

Cilium Tetragon Overview

Thu, 18 Sep 2025 16:09:42 +0800

Cilium Tetragon 能力

Cilium 的 Tetragon 組件以 eBPF 技術為基礎，提供了強大的即時安全可觀測性與 Runtime Enforcement 能力。

Tetragon 能偵測並對具安全意義的事件做出反應，例如：

程式執行事件
System call 活動
I/O 活動，包括網路與檔案存取

在 Kubernetes 環境中使用時，Tetragon 具備 Kubernetes 感知能力──也就是它能識別 namespace、Pods 等 Kubernetes 身分資訊──從而可以針對個別 Workload 來設定安全事件的偵測規則。

Deploy Tetragon

helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon ${EXTRA_HELM_FLAGS[@]} cilium/tetragon -n kube-system

確認 tetragon 部屬成功

kubectl -n kube-system get pods -l "app.kubernetes.io/instance=tetragon"

執行結果 :

NAME READY STATUS RESTARTS AGE
tetragon-2fp24 2/2 Running 0 5m11s
tetragon-7z89d 2/2 Running 0 5m11s
tetragon-cntz4 2/2 Running 0 5m11s
tetragon-kj669 2/2 Running 0 5m11s
tetragon-mktc9 2/2 Running 0 5m11s
tetragon-operator-6767dc8c8d-cqhcq 1/1 Running 0 5m11s
tetragon-sw7mz 2/2 Running 0 5m11s

Deploy demo application

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.15.3/examples/minikube/http-sw-app.yaml

檢查 pod 狀態

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
deathstar-8c4c77fb7-4tsl8 1/1 Running 0 25s
deathstar-8c4c77fb7-g55dt 1/1 Running 0 25s
tiefighter 1/1 Running 0 25s
xwing 1/1 Running 0 25s

實作 Tetragon 核心功能

1. Execution Monitoring

Tetragon 的核心是追蹤 Kubernetes 叢集、虛擬機器和裸機系統中所有執行情況。這為 Tetragon 將所有系統行為歸因於特定二進位檔案及其相關元資料（Container、Pod、節點和叢集）奠定了基礎。

1.1. 觀察 Tetragon 執行事件

要開兩個終端機在第一個終端機執行

# 使用此指令來取得與「xwing」Pod 位於同一 Kubernetes 節點上的 Tetragon Pod 的名稱：
POD=$(kubectl -n kube-system get pods -l 'app.kubernetes.io/name=tetragon' -o name --field-selector spec.nodeName=$(kubectl get pod xwing -o jsonpath='{.spec.nodeName}'))
# 使用 kubectl exec 定位它來執行 tetra getevents 命令
kubectl exec -ti -n kube-system $POD -c tetragon -- tetra getevents -o compact --pods xwing

在第二個終端機執行以下命令，讓 xwing pod 去存取一個網站

kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'

在第一個終端機會看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 0

2. File Access Monitoring

可以透過 YAML 設定檔將追蹤政策加入 Tetragon，以擴充其基本的執行追蹤功能。這些政策會在核心層進行過濾，確保只有「有興趣的事件」會從執行於核心中的 BPF 程式發佈到使用者空間。這樣的設計可確保即使在繁忙的系統中也能維持低負載。

以下的說明是延伸自「執行監控」的範例，加入了監控 Linux 中敏感檔案的政策。所使用的政策為 file_monitoring.yaml，你可以視需求檢閱並擴充。該政策中所監控的檔案是一組良好的基礎集合。

2.1. Apply the tracing policy

kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml

2.2. 觀察 Tetragon 文件訪問事件

在第二個終端機執行以下命令，產生一個讀取敏感檔案的事件

kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'

會在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "cat /etc/shadow"
🚀 process default/xwing /usr/bin/cat /etc/shadow
📚 read default/xwing /usr/bin/cat /etc/shadow
📚 read default/xwing /usr/bin/cat /etc/shadow
📚 read default/xwing /usr/bin/cat /etc/shadow
📚 read default/xwing /usr/bin/cat /etc/shadow
💥 exit default/xwing /usr/bin/cat /etc/shadow 0

根據追蹤政策，Tetragon 會在嘗試寫入敏感目錄（例如嘗試寫入 /etc 目錄）時產生寫入事件。

在第二個終端機執行以下命令

kubectl exec -ti xwing -- bash -c 'echo foo >> /etc/bar'

會在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "echo foo >> /etc/bar"
📝 write default/xwing /usr/bin/bash /etc/bar
📝 write default/xwing /usr/bin/bash /etc/bar
💥 exit default/xwing /usr/bin/bash -c "echo foo >> /etc/bar" 0

3. Network Monitoring

首先，需要找到 K8s 叢集正在使用的 Pod CIDR 和 Service CIDR。

export PODCIDR=$(kubectl get cm kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' | grep -oP 'podSubnet:\s*\K\S+')
export SERVICECIDR=$(kubectl get cm kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' | grep -oP 'serviceSubnet:\s*\K\S+')

取得這些資訊後，可以自訂策略，以排除流向 PODCIDR 和 SERVICECIDR 環境變數中儲存的網路的網路流量。使用 envsubst 執行此操作，然後使用 kubectl apply 將策略套用至您的 Kubernetes 叢集：

curl -s https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/network_egress_cluster.yaml | envsubst | kubectl apply -f -

一旦應用了追蹤策略，您就可以然後在 xwing Pod 中執行 curl 命令來訪問我們最喜歡的網站之一。

在第二個終端機執行以下命令

kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'

會在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect default/xwing /usr/bin/curl tcp 10.0.5.114:44810 -> 172.67.71.235:443
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon 0

看到在 tetra getevents 命令的輸出中報告了一個 connect 事件

可以透過向我們的服務發送 curl 請求來確認內核 BPF 過濾器是否未對叢集內部流量產生事件，若無 connect 事件產生，即表示未產生事件。

在第二個終端機執行以下命令

kubectl exec -ti xwing -- bash -c 'curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing'

會在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing"
🚀 process default/xwing /usr/bin/curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
💥 exit default/xwing /usr/bin/curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing 0

4. Policy Enforcement

Tetragon 的追蹤策略支援監控內核函式以報告事件，例如檔案存取事件或網路連線事件，並對這些相同的內核函式強制執行限制。在 Tetragon 中使用內核過濾提供了關鍵的效能提升，通過限制從內核到使用者空間的事件傳遞。內核過濾還使 Tetragon 能夠在內核層級強制執行策略限制。例如，當檢測到策略違規時，通過向進程發送 SIGKILL，該進程將無法繼續執行。如果策略強制執行是通過系統呼叫觸發的，則意味著應用程式將無法從該系統呼叫返回，並將被終止。

限制 Kubernetes 上的網路流量

export PODCIDR=$(kubectl get cm kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' | grep -oP 'podSubnet:\s*\K\S+')
export SERVICECIDR=$(kubectl get cm kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' | grep -oP 'serviceSubnet:\s*\K\S+')

得到 PodCIDR 和 SERVICECIDR 後，即可自訂並套用強制執行策略。

curl -s https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/network_egress_cluster_enforce.yaml | envsubst | kubectl apply -n default -f -

若要產生 Tetragon 將報告的事件，請使用 curl 連接到 Kubernetes 叢集外部的網站：

kubectl exec -ti xwing -- bash -c 'curl https://ebpf.io/applications/#tetragon'

你會得到以下錯誤訊息

command terminated with exit code 137

並在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "curl https://ebpf.io/applications/#tetragon"
🚀 process default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon
🔌 connect default/xwing /usr/bin/curl tcp 10.0.5.114:38040 -> 104.26.5.27:443
🔌 connect default/xwing /usr/bin/curl tcp 10.0.5.114:38040 -> 104.26.5.27:443
💥 exit default/xwing /usr/bin/curl https://ebpf.io/applications/#tetragon SIGKILL

與叢集內部目標建立網路連線將如預期般運作：

kubectl exec -ti xwing -- bash -c 'curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing'

並在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing"
🚀 process default/xwing /usr/bin/curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing
💥 exit default/xwing /usr/bin/curl -s -XPOST deathstar.default.svc.cluster.local/v1/request-landing 0

強制執行檔案存取限制

以下內容擴展了「檔案存取監控」的範例，以確保敏感檔案不被讀取。所使用的策略是 file_monitoring_enforce.yaml，您可以根據需要進行審查和擴展。觀察策略與強制執行策略之間的唯一區別是，強制執行策略新增了一個 action 區塊，該區塊會在檢測到操作違規時，發送 SIGKILL 信號終止應用程式，並使該操作返回錯誤。

kubectl delete -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring.yaml
kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/quickstart/file_monitoring_enforce.yaml

接下來，嘗試讀取敏感文件（定義的策略中包含的文件之一）：

kubectl exec -ti xwing -- bash -c 'cat /etc/shadow'

得到以下訊息

E0715 21:25:27.582529 11112 websocket.go:297] Unknown stream id 1, discarding message
command terminated with exit code 137

並在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "cat /etc/shadow"
🚀 process default/xwing /usr/bin/cat /etc/shadow
📚 read default/xwing /usr/bin/cat /etc/shadow
📚 read default/xwing /usr/bin/cat /etc/shadow
💥 exit default/xwing /usr/bin/cat /etc/shadow SIGKILL

嘗試讀取或寫入不屬於強製檔案策略的檔案不會受到影響 :

kubectl exec -ti xwing -- bash -c 'echo foo >> bar; cat bar'

得到以下訊息

foo

並在第一個終端機看到以下訊息

🚀 process default/xwing /usr/bin/bash -c "echo foo >> bar; cat bar"
🚀 process default/xwing /usr/bin/cat bar
💥 exit default/xwing /usr/bin/cat bar 0

Backstage Quick Start

Wed, 03 Sep 2025 08:08:28 +0800

Run Local

先決條件

Ubuntu 24.04.2 LTS

Node Version 20 (It is recommended to use volta or nvm to manage node versions)

# Install nvm (Node Version Manager)
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
# Load nvm in your current shell (also opens new shells with nvm)
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
# Install and use Node 20 (latest 20.x)
nvm install 20
nvm use 20
nvm alias default 20
# Verify
node -v
npm -v

Yarn to manage dependencies

# Install Corepack, an intermediary tool that will let you configure your package manager version on a per-project basis
npm install -g corepack
# update Yarn to the 4.4.1 version
yarn set version 4.4.1

Git for source control (OS 預設就有裝了)

docker

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# Install the Docker packages
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Creating an Instance

首先透過 npx 執行 Backstage 的 create-app 腳本：

npx @backstage/create-app

系統將提示輸入 Backstage 實例的名稱；該名稱將用於命名儲存它的目錄。在本例中，我們選擇了 terroir-portal 這個名稱。

Need to install the following packages:
@backstage/create-app@0.7.3
Ok to proceed? (y) y
? Enter a name for the app [required] terroir-portal

腳本將複製 Backstage 的範本及其目錄和檔案並安裝其所有相依性。

現在會看到以下目錄結構：

─ packages
├─ app
└─ package.json
└─ backend
└─ package.json
─ app-config.yaml
─ catalog-info.yaml
─ package.json

請注意，這裡有多個 package.json 文件，這是因為你的 Backstage 實例是以 monorepo 的方式實現的。因此，前端 ( app ) 和後端 ( backend ) 都有各自的依賴項，可以獨立部署。
也要注意 catalog-info.yaml 檔案。 Backstage 使用這個元資料檔案將自身加入軟體目錄。沒錯，Backstage 想要追蹤組織中的所有軟體資產，包括它自己！
最後，請查看 app-config.yaml 。這是主配置文件，您可以在其中定義實例的名稱並設定後端、身份驗證和其他整合的選項。

現在，讓我們看看 Backstage 是什麼樣子的。在終端機中導航到實例的目錄並執行：

cd terroir-portal; yarn start

現在，讓我們開始透過更改 Backstage 實例的名稱來讓它更具您的特色。轉到您的 app-config.yaml 文件並將應用程式的標題變更為您自己的標題。在我們的例子中，它看起來像這樣：

nano app-config.yaml

修改內容如下：

把預設 localhost 改成本機網卡的 ip

app:
title: Terroir Portal
baseUrl: http://192.168.11.133:3000
organization:
name: Terroir Technologies
backend:
baseUrl: http://192.168.11.133:7007
cors:
origin: http://192.168.11.133:3000

開發後台伺服器將會取得更改，您將在瀏覽器中看到名稱！

Installing PostgreSQL

get PostgreSQL through apt-get:

sudo apt-get install -y postgresql

在本課程中，您將使用預設的 postgres 使用者。因此，讓我們為該用戶設定一個密碼。透過以下方式看到相同的提示符號：

sudo -u postgres psql

使用此命令將使用者 postgres 密碼設定為您想要的任何密碼（ secret 是密碼範例）：

ALTER USER postgres PASSWORD 'secret'

Setting up Postgres in Backstage

Backstage 可讓您針對不同的環境使用不同的配置，這非常方便。您已經處理了主設定檔 app-config.yaml 來設定您的組織名稱。此外，您還會發現 app-config.local.yaml 和 app-config.production.yaml ，它們分別用於本機環境和生產環境。

目前，您將使用 app-config.local.yaml 檔案。 Backstage 預設將此檔案包含在 .gitignore 檔案中，這表示您在此處寫入的任何內容都不會被提交或推送到上游。這可以讓您更安心地將本地開發的機密文件存放在此。

若要在您的機器上使用 Postgres 設定 Backstage，請將以下內容新增至您的 app-config.local.yaml ：

nano app-config.local.yaml

檔案內容如下：

backend:
database:
connection:
host: 192.168.11.133
# Default postgresql port, change accordingly to your case
port: 5432
user: postgres
# The password you set earlier
password: secret

停止實例，然後使用 yarn start 再次運行。瀏覽器應該會再次開啟 http://192.168.11.133:3000/ 並顯示相同的 Catalog 畫面。

Setting up Authentication

您很可能需要對開發者入口網站進行身份驗證。 Backstage 支援 11 個開箱即用的身份驗證提供程序，包括 Okta、Google 和自訂 OAuth 2 代理程式。在本課程中，您將開始使用 GitHub 進行身份驗證，因為在接下來的章節中您將繼續使用 GitHub。

若要啟用 GitHub 身份驗證，您必須：

設定 GitHub OAuth 應用
新增 GitHub 身份驗證配置
在前端啟用 GitHub 作為身份驗證方法
新增 GitHub 後端提供者
將您的 GitHub 使用者名稱新增至使用者實體

nano app-config.local.yaml

新增內容如下

auth:
environment: development
providers:
github:
development:
clientId: Ov23liI1teDPwZFZR7Mi
clientSecret: 64e2b72bf62215b6e841ac78d061c52e7fe5d95c
signIn:
resolvers:
- resolver: usernameMatchingUserEntityName

nano packages/app/src/App.tsx

檔案內容：

// auth
backend.add(import('@backstage/plugin-auth-backend-module-github-provider'));

Introduction to Backstage: Developer Portals Made Easy (LFS142)

Mon, 01 Sep 2025 01:40:04 +0800

1. 認識 Developer Portals

1.1. 開發者自主權：一把雙面刃

雲原生組織不僅受益於其服務的擴展性與彈性，也得益於敏捷的團隊動態。由於服務是獨立部署的，團隊可以全權負責他們所開發的軟體元件。這是一大優勢，因為十多年來的研究已指出，能夠選擇工具並享受其工作環境的開發者，通常表現更佳。然而，開發者自主權可能導致組織碎片化，從而造成重複性工作、孤兒專案與資訊不透明等問題。

開發者自主權之所以至關重要，有兩個原因：更快的交付速度和人才留存。正如 Google 的 DORA DevOps 報告所強調：「沒有人比第一線的實踐者更清楚他們需要什麼才能有效工作。」這份業界報告觀察到，被賦予選擇自由及執行決策能力的團隊，其產出高於其他團隊。

此外，Microsoft 的開發者生產力 SPACE 框架指出，除了為任務選擇合適的工具，軟體所有權還包含心理層面的因素。當開發者能掌控自己的工作時，他們會感到被賦予權力。團隊也需要對其服務的品質、成本和其他面向負責，並會努力進行優化。

然而，自主不代表無政府狀態。組織必須提供一個有用的基準，讓開發者能在此之上建構解決方案。在限制與自由之間找到平衡點需要下功夫。透過結合政策、自助式基礎設施，以及能引導工程師走向最佳路徑的管線 (pipelines)，可以打造出保障自主權的更佳開發體驗 (Development Experience)。

但即使有良好的實踐方法，碎片化在雲原生組織中仍是一個常見問題。組織規模越大，其需要管理的複雜性就越高。隨著每個團隊交付自己的服務——而這些服務往往相互關聯且依賴第三方雲端服務——要得知有哪些服務可用以及如何使用它們，會變得很困難。

碎片化對一般貢獻者、新進人員和領導者都構成了挑戰。一般貢獻者在各自的孤島中工作，必須盲目地四處摸索才能存取組織資源。新進人員被拋入一個充滿不確定性且資訊不透明的環境，使得他們的到職流程 (onboarding) 更加困難與漫長。領導者則無法掌握其團隊正在開發的軟體，也無從得知它是否符合成熟度和安全標準。

1.2. 可發現性 (Discoverability)：唯有自動化才有效

「可發現性」取決於您能多快找到關於服務、函式庫、工具、團隊或儲存庫 (repository) 的資訊。在現今環境下，數十甚至數百個開發團隊為多個程式碼庫 (codebase) 貢獻、使用數十種開發工具並每日交付服務，因此一開始幾乎不可能知道要尋找什麼。由於雲原生團隊需處理不同的技術堆疊 (stack) 和情境，他們必須設計出一種溝通方式，讓他人了解他們的工作內容與方法，並藉此向其他團隊學習。通常，這意味著由某人手動將各種資源彙整在一份文件中。問題不在於撰寫文件，而在於如何保持其內容更新，並讓每個人都知道它的存在。

根據我在大型組織進行開發體驗 (DevExp) 研究的經驗，我發現人們會採用幾種策略來探索其內部生態系。出乎意料的是，最有效的方法似乎是「口耳相傳」：你在一個 Slack 頻道提問，他們會指引你到另一個頻道，在那裡你又會被轉介給幾個可能幫得上忙的人。接著你得想辦法和他們談談——如果他們的團隊在另一個大洲，那就祝你好運了。

但為什麼這種臨時性的策略最有效？儘管這個過程可能耗時數天或數週，但考量到其他現有文件的局限性，開發者往往視其為唯一的選擇。另一個選項是去尋找某人去年在 Atlassian、Google Docs 或 Notion 上寫的文件。但不僅跨平台搜尋不易，裡面的手繪架構圖、API 參考資料和其他資訊，在被找到時很可能也早已過時。

此外，假設你為你的服務 API 及相關函式庫寫了一份終極指南，並承諾定期維護。但你要如何告訴一位你根本不認識的同事，你可能已經建立好一個他正好能用的測試框架 (test harness)？你辦不到，因為你不知道他們需要，而他們此刻也正在從零開始打造自己的。除非有一個統一且可靠的方式來探索組織內的可用資源，否則「可發現性」很難實現。

大規模、統一且可靠的資訊，唯有透過自動化才可能實現。不少公司已為不同需求建立了資訊聚合器，這卻導致人們需要同時使用多個聚合器來搜尋資訊。對於那些投資發展內部開源 (Inner Source) 文化的組織而言，提供自動化的「可發現性」已成為改善協作的一大關鍵推手。

1.3. 什麼是開發者入口網站 (Developer Portal)？

開發者入口網站 (Developer Portal) 是開發者需要了解並貢獻於其內部生態系時的首選平台。Developer Portal 的存在是為了在「開發者自主權 (Developer Autonomy)」與「可發現性 (Discoverability)」之間取得平衡：它允許團隊獨立工作、選擇偏好的工具並進行每日部署，同時也將其工作成果展示出來，讓所有人都能探索並受益於他們的微服務、API、函式庫、網站、管線 (pipeline) 和演算法。Developer Portal 有三大主要目的：

協助開發者探索其內部生態系。
透過自助式服務賦予工程師能力。
為領導者和團隊提供關於技術健康度與成熟度的洞見。

Developer Portal 中的「可發現性」是自動化的，並集中呈現在一個軟體目錄 (software catalog) 中。Developer Portal 可透過多種方式收集組織內服務與其他軟體元件的資訊。它會聚合每個元件的相關資訊，包括負責團隊、文件、可用性、部署情況、Snyk 問題、PagerDuty 事件，以及任何其他可能對團隊有用的資訊。

確實，Developer Portal 一個廣受歡迎的功能，就是能夠追蹤服務之間的「所有權」與「相互依賴性」。每個軟體元件都有所屬的負責團隊或個人貢獻者。此外，軟體目錄會將元件之間的依賴關係圖形化。

此外，Developer Portal 還能為您的系統建立模型，將所有軟體元件依據有意義的分類法 (taxonomy) 進行整理。如此一來，開發者便能透過功能類別來探索生態系，更輕易地找到有用資訊。

另一方面，Developer Portal 不一定是唯讀的。DevOps 團隊可以提供樣板 (template)，讓開發者在 Portal 中一鍵完成新服務的基礎設施配置 (provision)，而無須再開立工單 (ticket) 請求支援。這樣做有雙重好處：開發者因近乎即時的可用性而更具生產力；DevOps 團隊則能在不強迫的情況下，引導開發者走上「黃金路徑」(golden path)。

最後，由於 Developer Portal 聚合了來自不同來源、關於組織大部分技術資產的資訊，因此有大量機會從中萃取洞見。技術洞見對領導層有益，因為他們可以全面審視技術成熟度與遷移計畫，並找出需要額外支援之處。團隊和開發者同樣能從這些洞見與透明度中受益，因為他們可以依據標準進行優化，並為自身的進步感到自豪。

1.4. 開發者入口網站 (Developer Portal) 與開發平台 (Development Platform) 有何關聯？

根據雲原生生態系中最全面的調查《DORA’s State of DevOps 2024》，約有 89% 的組織正在使用開發平台 (Development Platform)。但什麼是「平台 (Platform)」？

CNCF 平台工作小組 (CNCF Platforms Working Group) 將「平台」定義為：

一系列能力、文件和工具的集合，用以支援產品和服務交付過程中的開發、部署、營運和／或管理。一個平台可能包含 Web 入口網站、API、CLI、協定定義、文件、標準和／或黃金路徑樣板 (golden path templates)。若建置得當，平台能讓組織更快、更可靠地交付其應用程式和服務。

如您所見，這個定義為詮釋留下了空間。這是因為不同組織的平台樣貌大相逕庭。平台的設計與建置，必須與組織的目標、基礎設施和團隊相契合。

為了讓定義更清晰，CNCF 平台工作小組提供了一張圖表，說明平台能力的樣貌：

請看圖中的「平台介面 (Platform Interfaces)」部分。這裡列出的功能——文件、透過 Web 入口網站提供的 GUI，以及專案樣板——正好對應到 Developer Portal 的功能。

因此，Developer Portal 是「平台介面 (Platform Interfaces)」的一種。然而，平台也可以有不同類型的介面，例如基於 API 或 CLI 的介面。

1.5. 知識確認

Which of the following are the benefits of Developer Autonomy? Select all answers that apply.

A. Developers feel more motivated when empowered with ownership B. The organization promotes creativity by having several ways to solve a problem C. Developers are more productive when they can choose their tools D. Easier onboarding for new hires

正確答案：A、C

A. 當被賦予所有權時，開發者會感到更有動力
- ✔️ 正確
  - 原因： 賦予所有權能提升開發者的責任感與工作投入度。當團隊能完全掌控自己的工作成果時，會更有動力去追求卓越的品質與效能。
B. 組織透過擁有多種解決問題的方法來促進創造力
- ❌ 錯誤
  - 原因： 雖然表面上聽起來像是促進創意，但在文章的脈絡中，這種情況更多被描述為一個負面的副作用。自主權導致的「多種解決方案」往往等同於「重複性工作 (redundant work)」和「組織碎片化 (organizational fragmentation)」。
  - 當不同團隊在資訊不互通的情況下各自為政，他們可能會花費大量時間和資源去重複打造功能相似的輪子。從組織整體效率的角度來看，這並非值得鼓勵的「創造力」，而是一種需要透過開發者平台 (Development Platform) 來管理的資源浪費。
C. 當開發者可以選擇自己的工具時，他們會更有生產力
- ✔️ 正確
  - 原因： 開發者使用自己熟悉且偏好的工具時，能大幅減少學習成本與操作摩擦，直接提升開發效率與產出。
D. 新進人員的到職流程更容易
- ❌ 錯誤
  - 原因： 這恰好是自主權的主要缺點。各團隊獨立運作會導致技術與資訊的「碎片化」，使組織環境變得複雜不透明，反而讓新進人員的到職流程 (onboarding) 變得更加困難。
Why do developers often discover resources through a word-of-mouth approach? Select all answers that apply.

A. Developers don’t like reading documentation B. Developers don’t always know the name of what they’re looking for C. Developers want to break silos by talking with more teams D. Developers find it difficult to find reliable documentation

正確答案：B、D

A. Developers don’t like reading documentation (開發者不喜歡閱讀文件)
- ❌ 錯誤
  - 原因： 問題的核心不在於開發者「不喜歡」閱讀，而在於現存的文件常常是不可靠或過時的。如果有一份集中管理、內容準確的最新文件，開發者通常會很樂意使用它。
B. Developers don’t always know the name of what they’re looking for (開發者不一定知道他們要找的東西叫什麼名字)
- ✔️ 正確
  - 原因： 在複雜的組織中，開發者可能只知道自己要解決的「問題」（例如：我需要一個處理用戶支付的 API），但並不知道解決這個問題的內部服務或工具的「確切名稱」。既然無法用關鍵字搜尋，最直接的方式就是向人描述問題。
C. Developers want to break silos by talking with more teams (開發者想透過與更多團隊交談來打破孤島效應)
- ❌ 錯誤
  - 原因： 這種跨團隊溝通通常是為了解決問題而不得不採取的低效率手段，而非開發者主動想要社交或打破壁壘。文章將這個過程描述為耗時數天甚至數週，是「別無選擇」下的方法，而不是出於主動的社交意願。
D. Developers find it difficult to find reliable documentation (開發者覺得很難找到可靠的文件)
- ✔️ 正確
  - 原因： 這是最主要的原因。文章明確指出，文件通常散落在不同平台（Google Docs, Notion 等），導致搜尋困難；更重要的是，手動維護的文件很容易過時，資訊不可靠。因此，開發者寧願去問「人」，因為人通常能提供比過期文件更即時、更準確的資訊。
How do Developer Portals help reduce software fragmentation?

A. Through a comprehensive software catalog B. By nudging developers to a golden path through self-service options C. By aggregating information from different sources for each component D. By providing insights on maturity and security

正確答案：B

A. 透過一個全面的軟體目錄 (Through a comprehensive software catalog)
- ❌ 錯誤
  - 原因：軟體目錄主要解決的是「可發現性 (Discoverability)」問題，它讓開發者能「看見」已經存在的碎片化情況。雖然找到並重複使用現有元件能間接減少碎片化，但這是一個被動的功能。它幫助你了解現狀，但沒有主動引導你如何標準化地「建立新東西」。
B. 透過自助式選項，引導開發者走上黃金路徑 (By nudging developers to a golden path through self-service options)
- ✔️ 正確
  - 原因：這是最直接且主動減少碎片化的方法。「黃金路徑 (golden path)」代表了組織推薦的最佳實踐與標準化建置方式。開發者入口網站透過提供「自助式」的樣板（template）或一鍵生成功能，讓「遵循標準」成為最簡單、最快速的選項。這從源頭上確保了新建立的服務或元件都具備一致性，從而有效抑制了碎片化的產生。
C. 透過為每個元件聚合來自不同來源的資訊 (By aggregating information from different sources for each component)
- ❌ 錯誤
  - 原因：這個選項描述的是如何建構軟體目錄 (選項 A)的技術細節。它解釋了入口網站如何收集資訊，但這個「資訊聚合」的動作本身並不能減少碎片化，它只是更完整地呈現了現有的軟體樣貌。
D. 透過提供關於成熟度和安全性的洞見 (By providing insights on maturity and security)
- ❌ 錯誤
  - 原因：這個功能主要是用於「衡量與監控」。它幫助領導者和團隊「發現」因碎片化而導致的問題（例如：安全標準不一、技術債過高）。這是在問題發生後進行觀察與分析，而不是像選項 B 那樣在開發初期就主動預防問題的發生。
What is the relationship between a Developer Portal and a Developer Platform?

A. Developer Portal and Developer Platform are just different names for the same thing B. A Developer Portal can contain a Developer Platform as part of its features C. A Developer Portal can be an interface for a Developer Platform D. There is no relationship between a Developer Portal and a Developer Platform

正確答案：C

A. 開發者入口網站和開發平台只是同一樣東西的不同名稱
- ❌ 錯誤
  - 原因：這兩個是不同但相關的概念。平台是底層的、更廣泛的「能力集合」（包含工具、API、基礎設施等）；而入口網站是上層的、用來與平台互動的「介面」。
B. 開發者入口網站可以包含一個開發平台作為其功能的一部分
- ❌ 錯誤
  - 原因：這個選項將兩者的關係說反了。開發平台是更大、更基礎的實體。應該是平台「包含」了入口網站作為其介面層，而不是入口網站包含平台。
C. 開發者入口網站可以是開發平台的一個介面
- ✔️ 正確
  - 原因：Portal 可作為 Platform 的使用介面（例如服務目錄、模板、流水線啟動、文件與治理護欄）。
D. 開發者入口網站和開發平台之間沒有關係
- ❌ 錯誤
  - 原因：它們之間有非常緊密的關係。入口網站是現代開發平台策略中，面向開發者的關鍵使用者體驗層，是平台能否成功被採用的重要一環。

Quick Start Istio with ambient mode

Fri, 29 Aug 2025 15:16:16 +0800

Step 1: 下載 Istio CLI

curl -L https://istio.io/downloadIstio | sh -
cd istio-1.26.1
export PATH=$PWD/bin:$PATH

檢查是否能執行 `istioctl`

istioctl version

命令執行結果 :

Istio is not present in the cluster: no running Istio pods in namespace "istio-system"
client version: 1.26.1

在此階段，Istio 尚未安裝到您的叢集中，因此您會看到沒有任何 Pod 處於 Ready 狀態。

Step2: Install Istio on to your cluster

istioctl 支援多種設定範本，這些範本包含不同的預設選項，並可依照您的生產需求進行自訂。ambient 模式的支援已包含在 ambient 範本中。使用以下指令安裝 Istio：

istioctl install --set profile=ambient --skip-confirmation

命令執行結果 :

 |\
| \
| \
| \
/|| \
/ || \
/ || \
/ || \
/ || \
/ || \
/______||__________\
____________________
\__ _____/
\_____/
✔ Istio core installed ⛵️
✔ Istiod installed 🧠
✔ CNI installed 🪢
✔ Ztunnel installed 🔒
✔ Installation complete
The ambient profile has been installed successfully, enjoy Istio without sidecars!

檢查 Istio pods 狀態是否正常

kubectl -n istio-system get pods

命令執行結果 :

NAME READY STATUS RESTARTS AGE
istio-cni-node-bc4fx 1/1 Running 0 8m34s
istio-cni-node-kww8q 1/1 Running 0 8m34s
istio-cni-node-qhvtt 1/1 Running 0 8m34s
istiod-799b8c5498-gpqs6 1/1 Running 0 8m44s
ztunnel-2jr6q 1/1 Running 0 8m22s
ztunnel-bnvps 1/1 Running 0 8m22s
ztunnel-hjlch 1/1 Running 0 8m22s

Step3: Install the Kubernetes Gateway API CRDs

接下來將使用 Kubernetes Gateway API 來設定流量路由。

請注意，大多數 Kubernetes 叢集預設並未安裝 Kubernetes Gateway API 的 CRD，因此在使用 Gateway API 之前，請先確保這些 CRD 已安裝：

kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml

檢查 CRD 是否安裝成功

kubectl get -f https://github.com/kubernetes-sigs/gateway-api/rel
eases/download/v1.3.0/standard-install.yaml

命令執行結果 :

NAME CREATED AT
gatewayclasses.gateway.networking.k8s.io 2025-06-02T06:02:07Z
gateways.gateway.networking.k8s.io 2025-06-02T06:02:07Z
grpcroutes.gateway.networking.k8s.io 2025-06-02T06:02:07Z
httproutes.gateway.networking.k8s.io 2025-06-02T06:02:07Z
referencegrants.gateway.networking.k8s.io 2025-06-02T06:02:07Z

實作 Bookinfo Application

Before you begin

此範例部署了一個由四個獨立的微服務組成的範例應用程序，用於演示各種 Istio 功能。

該應用程式會顯示有關一本書的資訊，類似於線上書店的單一目錄條目。頁面上會顯示書籍的描述、書籍細節（ISBN、頁數等等），以及一些書評。

Bookinfo 應用程式拆分為四個獨立的微服務：

productpage：productpage 微服務會呼叫 details 和 reviews 微服務，來填充頁面內容。
details：details 微服務包含書籍的基本資訊。
reviews：reviews 微服務包含書評，並且會呼叫 ratings 微服務。
ratings：ratings 微服務則包含書評所搭配的書籍評分資訊。

其中，reviews 微服務有三個版本：

版本 v1 不會呼叫 ratings 服務。
版本 v2 會呼叫 ratings 服務，並將每個評分顯示為 1 到 5 顆黑色星星。
版本 v3 會呼叫 ratings 服務，並將每個評分顯示為 1 到 5 顆紅色星星。

應用程式的端到端架構如下所示。

該應用程式是多語言（polyglot）實作，也就是說，微服務是用不同程式語言編寫的。值得注意的是，這些服務本身並不依賴 Istio，但由於服務數量眾多、所使用的語言多樣，且 reviews 服務還有多個版本，因此成為一個頗具吸引力的服務網格示例。

Deploying the application

首先部署應用程式：

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/bookinfo/platform/kube/bookinfo-versions.yaml

若要驗證應用程式是否正在執行，請檢查 pod 的狀態：

kubectl get pods

執行結果 :

NAME READY STATUS RESTARTS AGE
details-v1-766844796b-2d7qd 1/1 Running 0 96s
productpage-v1-54bb874995-g8d4b 1/1 Running 0 95s
ratings-v1-5dc79b6bcd-zb28h 1/1 Running 0 95s
reviews-v1-598b896c9d-4hmrj 1/1 Running 0 95s
reviews-v2-556d6457d-r6q75 1/1 Running 0 95s
reviews-v3-564544b4d6-zmvzj 1/1 Running 0 95s

Deploy and configure the ingress gateway

您將使用 Kubernetes Gateway API 來部署一個名為 bookinfo-gateway 的 Gateway：

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/bookinfo/gateway-api/bookinfo-gateway.yaml

預設情況下，Istio 會為 Gateway 建立一個 LoadBalancer 類型的 Service。由於您會透過隧道訪問此 Gateway，因此不需要 LoadBalancer，請將 Service 類型改為 ClusterIP，方法是為 Gateway 加上註解：

kubectl annotate gateway bookinfo-gateway networking.istio.io/service-type=ClusterIP --namespace=default

要檢查 Gateway 的狀態，請執行：

kubectl get gateway

執行結果 :

NAME CLASS ADDRESS PROGRAMMED AGE
bookinfo-gateway istio bookinfo-gateway-istio.default.svc.cluster.local True 42s

請等到 Gateway 顯示為已「PROGRAMMED」後再繼續。

動手實作 veeam kasten

Thu, 07 Aug 2025 08:20:48 +0800

先決條件

已安裝 StorageClass ( 需要有 support VolumeSnapshot APIs )
已安裝 Minio (非必要)

支援平台

作業系統	架構	是否支援 FIPS	Veeam Repository Exports	vSphere Block Mode Exports
Linux	x86_64 (amd64)	是	是	是
Linux	Arm (arm64/v8)	否	否	否
Linux	Power (ppc64le)	否	否	不適用

快速開始安裝

adding the Kasten repository to your Helm client:

helm repo add kasten https://charts.kasten.io/
helm repo update

Run the following to initiate the install into the kasten-io namespace:

helm install k10 kasten/k10 \
--namespace=kasten-io \
--create-namespace \
--version=8.0.2

After the previous helm command returns, monitor the Pod status until all Pods have reached a fully “READY” state (i.e. 1/1, 2/2, etc.), as shown below (~2 minutes):

watch "kubectl get pods -n kasten-io"

執行結果 :

Every 2.0s: kubectl get pods -n kasten-io c1: Thu Aug 7 15:39:01 2025
NAME READY STATUS RESTARTS AGE
aggregatedapis-svc-7bb54699bf-r2v52 1/1 Running 1 (85m ago) 85m
auth-svc-6478997bd4-s2d84 1/1 Running 0 85m
catalog-svc-6b86c49dd4-7dgn7 2/2 Running 0 85m
controllermanager-svc-867bbb788b-s4h5b 1/1 Running 0 85m
crypto-svc-6496ffd558-xh54x 4/4 Running 0 85m
dashboardbff-svc-5d9c4494d5-84z2g 2/2 Running 0 85m
executor-svc-5895f744f5-6clhv 1/1 Running 0 85m
executor-svc-5895f744f5-p8lsj 1/1 Running 0 85m
executor-svc-5895f744f5-qm4bw 1/1 Running 0 85m
frontend-svc-869cd877f8-msprr 1/1 Running 0 85m
gateway-55fdbb977c-k8wbj 1/1 Running 0 85m
jobs-svc-78988bc957-7c89r 1/1 Running 0 85m
kanister-svc-795b4b9947-rnrwn 1/1 Running 0 85m
logging-svc-fbd98b769-7gq6r 1/1 Running 0 85m
metering-svc-6988c4c6c8-rmhnx 1/1 Running 0 85m
prometheus-server-54dcdf4b9d-pqvxh 2/2 Running 0 85m
state-svc-6dc8f8c55d-4dcq8 2/2 Running 0 85m

連接到 Web UI

# 取得 ens18 網卡的 ip
IP=$(ip -4 -o addr show dev ens18 \
| awk '{print $4}' \
| cut -d/ -f1)
kubectl -n kasten-io port-forward --address ${IP} service/gateway 8080:80 &

執行結果 :

[1] 5784
Forwarding from 192.168.11.71:8080 -> 8000

Veeam Kasten dashboard 會在 http://192.168.11.71:8080/k10/#/

設定 Primary Storage

在我們開始保護應用程式之前，Kasten 必須能夠對叢集所使用的儲存體執行快照功能。 Kasten 可以與任何已實作 VolumeSnapshot API 支援的 CSI (Container Storage Interface) 儲存供應程式 (storage provisioner) 搭配使用，同時也提供直接的 API 整合，以支援非 CSI 的供應程式或啟用非 CSI 的功能。

列出叢集中的 VolumeSnapshotClasses

kubectl describe volumesnapshotclasses

執行結果 :

Name: longhorn
Namespace:
Labels: <none>
Annotations: <none>
API Version: snapshot.storage.k8s.io/v1
Deletion Policy: Delete
Driver: driver.longhorn.io
Kind: VolumeSnapshotClass
Metadata:
Creation Timestamp: 2025-08-07T07:37:39Z
Generation: 1
Resource Version: 692621
UID: e2b6a677-0a38-4f2f-86e8-8cb356d636ea
Events: <none>

添加必要的 Kasten annotation 給 VolumeSnapshotClass:
```
kubectl annotate volumesnapshotclass longhorn \
k10.kasten.io/is-snapshot-class=true
```
這麼做的目的，是為了讓 Kasten 在備份期間向 Kubernetes API server 提出請求以建立 PersistentVolumeClaims 的快照時，可以識別要參照哪一個 VolumeSnapshotClass。

Kasten 內建一個工具，可以在部署過程中執行，以快速索引並驗證叢集的儲存設定。這包括識別可用的 StorageClasses，以及其關聯的 VolumeSnapshotClass 是否已套用 Kasten annotation。您可以選擇透過執行以下指令，在叢集上進行測試：

curl https://docs.kasten.io/downloads/latest/tools/k10_primer.sh | bash

執行結果：

Namespace option not provided, using default namespace
Checking for tools
--> Found kubectl
--> Found helm
--> Found jq
--> Found cat
--> Found base64
--> Found tr
Checking if the Kasten Helm repo is present
--> The Kasten Helm repo was found
Checking for required Helm version (>= v3.11.0)
--> Helm binary version meet the requirements
K10Primer image
--> Using Image (gcr.io/kasten-images/k10tools:8.0.5) to run test
Checking access to the Kubernetes context kubernetes-admin@topgun
--> Able to access the default Kubernetes namespace
K10 Kanister tools image
--> Using Kanister tools image (gcr.io/kasten-images/k10tools:8.0.5) to run test
Running K10Primer Job in cluster with command
./k10tools primer
serviceaccount/k10-primer created
clusterrolebinding.rbac.authorization.k8s.io/k10-primer created
job.batch/k10primer created
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod k10primer-wqmb7 is in Pending phase
Pod Ready!
================================================================
Kubernetes Version Check:
current kubernetes version (v1.33.2) is not supported. Supported versions: [1.28 1.29 1.30 1.31 1.32] - Error
RBAC Check:
Kubernetes RBAC is enabled - OK
Aggregated Layer Check:
The Kubernetes Aggregated Layer is enabled - OK
CSI Capabilities Check:
Using CSI GroupVersion snapshot.storage.k8s.io/v1 - OK
Validating Provisioners:
driver.longhorn.io:
Is a CSI Provisioner - OK
Storage Classes:
longhorn
Valid Storage Class - OK
longhorn-static
Valid Storage Class - OK
Volume Snapshot Classes:
longhorn
Has k10.kasten.io/is-snapshot-class annotation set to true - OK
Has deletionPolicy 'Delete' - OK
Validate Generic Volume Snapshot:
Pod created successfully - OK
GVS Backup command executed successfully - OK
Pod deleted successfully - OK
================================================================
serviceaccount "k10-primer" deleted
clusterrolebinding.rbac.authorization.k8s.io "k10-primer" deleted
job.batch "k10primer" deleted

設定 Location Profile

您現在已設定好使用 Kasten 對 Kubernetes 應用程式執行本機快照所需的一切 — 但快照不等於備份！ 為了在本地叢集或主要儲存空間受損時能夠還原，應將一份資料副本匯出到其他位置。

這些備份目標的設定被稱為 Location Profiles。Kasten 支援多種選項，包括：

AWS S3
Azure Blob
Google Cloud Storage
S3-Compatible
NFS
SMB
Veeam Backup & Replication

提醒您，您的實驗室環境中已經預先部署了一台 MinIO 物件儲存伺服器。

實作 K8s 實體機節點硬碟備份與還原

Wed, 09 Jul 2025 10:04:30 +0800

0. Preface

本篇文章將實作透過 Clonezilla 再生龍來備份硬碟到其他裝置，並還原到新的硬碟上。 (實作環境透過 Proxmox 進行模擬)

實作兩個情境：

備份 Control-plane node (同時扮演 Etcd member) 的硬碟，並還原在新的一顆乾淨的硬碟上。

實作環境並沒用切割系統碟和資料碟，通通儲存在同一顆硬碟。
備份 Control-plane node (同時扮演 Etcd leader) 的硬碟，在備份完後，破壞一些 K8s 核心服務的檔案及目錄，再透過備份好的硬碟還原到新的硬碟上，最後檢查 K8s 是否能正常運作。

1. 下載 Clonezilla live CD

到以下 Clonezilla 官方網站下載 Clonezilla live CD： https://clonezilla.nchc.org.tw/clonezilla-live/download/
選擇 “穩定-發行版”
CPU 架構選擇：amd64, 檔案類別選擇：iso，檔案儲存庫：auto，確認好後點選 下載 按鈕
可透過Ｒufus 將 ISO 做成 USB 可開機裝置; 如過是虛擬環境則將 ISO 匯入虛擬化平台

2. 將第一顆硬碟備份，存為印象檔(disk image)，放置在第二顆硬碟上

Step1: 由再生龍 live CD 開機

新增一顆硬碟 (稍後會作為儲放備份 disk image 的硬碟)
設定 Size 為 50 Gi (須視環境做調整)，確認好後點選 Add 按鈕
將 /dev/sdb 格式化為 XFS 檔案系統
```
sudo mkfs.xfs /dev/sdb
```
將 VM 關機
點選 Hardware -> 點選 Add -> 點選 CD/DVD Driver
選擇 Storage 和 ISO image: clonezilla-live-3.2.2-15-amd64.iso，確認好後點選 Add 按鈕
新增一顆硬碟 (稍後會作為儲放備份 disk image 的硬碟)
設定可開機裝置的順序，點選 options -> 點選 Boot Order -> 點選 Edit
將再生龍 live CD 設為第一個可開機裝置，確認好後點選 OK 按鈕
將 VM 開機

Step2: Clonezilla live 開機選單

選擇 “Clonezilla live (Default settings, VGA 800x600)"：第1個選項是 Clonezilla Live 的預設模式：使用 framebuffer 並設定螢幕解析度為 800x600.

Step3: 選擇語言

選擇 zh_TW.UFT-8 Chinese (Traditional) | 正體中文 - 臺灣

Step4: 選擇鍵盤

使用預設的鍵盤，直接按 Enter 鍵

Step5: 選擇 “使用再生龍”

使用預設的選項，直接按 Enter 鍵

Step6: 選擇 “device-image”

選擇預設的選項：“device-image”

Step7: 選擇 “local_dev” 來指定 sdb 作為 Disk image 家目錄

選擇 “local_dev”, 就可以將印象檔存到本機其他硬碟上或 USB 裝置. 如果使用 USB 儲存裝置,請等待一下後在將 USB 儲存裝置插上
將 USB 接上電腦，過 5 秒後按 Enter 鍵（前面步驟已先額外新增一顆硬碟，故此步驟可直接按 Enter 鍵）
每過幾秒就會顯示裝置掃描的結果，如下面的提示
確認裝置都認到後，按 Ctrl + C

Step8: 選擇 sdb 作為 Disk image 的目錄

選擇 sdb 硬碟裡的目錄作為印象檔目錄.
選擇 “fsck” 檢查來源硬碟的檔案系統
選擇要哪個目錄給再生龍存 disk image，確定後按 Tab 鍵，選擇 <Done>
顯示硬碟的使用狀況
選擇 “Beginner” (初學模式)
選擇 “savedisk”

Step9: 輸入 Disk image 檔名和選擇來源硬碟

輸入備份之後 disk image 的檔名，確認後按 Enter 鍵
選擇要備份哪一顆硬碟，本篇範例為 sda
選擇印象檔壓縮演算法 -z9p (zstd):
選擇 “fsck” 檢查來源硬碟的檔案系統
選擇檢查備份後的 Disk image 是否完整
選擇不要對 Disk image 加密
選擇將 log 檔也儲存在 Disk image 中
選擇 -p choose，等程式跑完後再選擇關機或重開或進入命令模式
Clonezilla 出現此次工作的完整命令提示. 此命令可在製作客制化Clonezilla live時使用，按 Enter 鍵繼續
開始備份之前，再次詢問是否進行，輸入 y，按 Enter 鍵確定開始備份

Step 10: 完成將硬碟 (sda) 備份到第二顆硬碟 (sdb) 中

完成時，Clonezilla 提示若要再次使用再生龍選單的方式，按 Enter 鍵繼續
選擇關機
按 Enter 繼續關機

3. 將 Disk 還原到第一顆硬碟上

Step1: 由再生龍 live CD 開機

新增一顆硬碟 (稍後會作為儲放備份 disk image 的硬碟)
設定 Size 為 100 Gi (須視環境做調整)，確認好後點選 Add 按鈕
點選 Hardware -> 點選 Add -> 點選 CD/DVD Driver
選擇 Storage 和 ISO image: clonezilla-live-3.2.2-15-amd64.iso，確認好後點選 Add 按鈕
設定可開機裝置的順序，點選 options -> 點選 Boot Order -> 點選 Edit
將再生龍 live CD 設為第一個可開機裝置，確認好後點選 OK 按鈕
將 VM 開機

Step2: Clonezilla live 開機選單

選擇 “Clonezilla live (Default settings, VGA 800x600)"：第1個選項是 Clonezilla Live 的預設模式：使用 framebuffer 並設定螢幕解析度為 800x600.

Step3: 選擇語言

選擇 zh_TW.UFT-8 Chinese (Traditional) | 正體中文 - 臺灣

Step4: 選擇鍵盤

使用預設的鍵盤，直接按 Enter 鍵

Step5: 選擇 “使用再生龍”

使用預設的選項，直接按 Enter 鍵

Step6: 選擇 “device-image”

選擇預設的選項：“device-image”

Step7: 選擇 “local_dev” 來指定 sdb 作為 Disk image 家目錄

選擇 “local_dev”, 就可以將印象檔存到本機其他硬碟上或 USB 裝置. 如果使用 USB 儲存裝置,請等待一下後在將 USB 儲存裝置插上
將 USB 接上電腦，過 5 秒後按 Enter 鍵（前面步驟已先額外新增一顆硬碟，故此步驟可直接按 Enter 鍵）
每過幾秒就會顯示裝置掃描的結果，如下面的提示
確認裝置都認到後，按 Ctrl + C

Step8: 選擇 sdb 作為 Disk image 的目錄

選擇 sdb 硬碟裡的目錄作為印象檔目錄.
選擇 “fsck” 檢查來源硬碟的檔案系統
選擇要哪個目錄給再生龍存 disk image，確定後按 Tab 鍵，選擇 <Done>
顯示硬碟的使用狀況
選擇 “Beginner” (初學模式)
選擇 “restoredisk”

Step9: 選擇印象檔與目標碟

選擇要透過哪個 Disk image 還原，確認後按 Enter 鍵
選擇要還原到哪一顆硬碟上，本篇範例為 sdc
選擇印象檔中的硬碟分割表
選擇 “是”，在還原前檢查來源硬硬碟的檔案完整性
選擇將 log 檔也儲存在 Disk image 中
選擇 -p choose，等程式跑完後再選擇關機或重開或進入命令模式
Clonezilla 出現此次工作的完整命令提示. 此命令可在製作客制化Clonezilla live時使用，按 Enter 鍵繼續
因為剛剛有選擇要檢查印象檔所以在還原之前，就會進行印象檔的檢查
開始備份之前，詢問是否進行，按 Enter 鍵確定開始備份
開始備份之前，再次詢問是否進行，按y，再按 Enter 鍵確定開始備份
開始備份之前，第三次詢問是否進行，按y，再按 Enter 鍵確定開始備份

Step 10: 完成將硬碟 (sda) 備份到第二顆硬碟 (sdb) 中

完成時，Clonezilla 提示若要再次使用再生龍選單的方式，按 Enter 鍵繼續
選擇關機
按 Enter 繼續關機

Step 11: 驗證是否備份成功

設定可開機裝置的順序，點選 options -> 點選 Boot Order -> 點選 Edit
將第三顆硬碟 sdc 設為第一個可開機裝置，確認好後點選 OK 按鈕
將 VM 開機

檢視 K8s 節點是否健康

$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane,master 7d5h v1.21.14
m2 Ready control-plane,master 7d4h v1.21.14
m3 Ready control-plane,master 7d3h v1.21.14
w1 Ready <none> 7d3h v1.21.14
w2 Ready <none> 7d3h v1.21.14

確認 K8s 核心服務是否正常

$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6477b97f9f-zk24r 1/1 Running 6 5d3h
calico-node-558x6 1/1 Running 0 7d3h
calico-node-cnzrx 1/1 Running 2 5d3h
calico-node-gpcbl 1/1 Running 6 7d3h
calico-node-mh4k6 1/1 Running 7 7d3h
calico-node-rsvmr 1/1 Running 1 7d3h
coredns-558bd4d5db-nvbxc 1/1 Running 0 24h
coredns-558bd4d5db-sg5nb 1/1 Running 0 24h
etcd-m1 1/1 Running 3 26h
etcd-m2 1/1 Running 0 7d4h
etcd-m3 1/1 Running 2 7d3h
kube-apiserver-m1 1/1 Running 0 26h
kube-apiserver-m2 1/1 Running 0 7d4h
kube-apiserver-m3 1/1 Running 2 7d3h
kube-controller-manager-m1 1/1 Running 1 26h
kube-controller-manager-m2 1/1 Running 2 7d3h
kube-controller-manager-m3 1/1 Running 2 7d3h
kube-proxy-6w529 1/1 Running 0 7d3h
kube-proxy-75c4k 1/1 Running 0 7d3h
kube-proxy-dpr4d 1/1 Running 2 5d4h
kube-proxy-lzxml 1/1 Running 0 5d4h
kube-proxy-vlfwv 1/1 Running 0 7d3h
kube-scheduler-m1 1/1 Running 1 26h
kube-scheduler-m2 1/1 Running 2 7d3h
kube-scheduler-m3 1/1 Running 2 7d3h
kube-vip-m1 1/1 Running 2 7d3h
kube-vip-m2 1/1 Running 2 7d4h
kube-vip-m3 1/1 Running 2 7d3h

檢查該節點上的 Pod 是否正常

$ kubectl get pods --field-selector spec.nodeName=m3 -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-cnzrx 1/1 Running 2 5d3h
kube-system etcd-m3 1/1 Running 2 7d3h
kube-system kube-apiserver-m3 1/1 Running 2 7d3h
kube-system kube-controller-manager-m3 1/1 Running 2 7d3h
kube-system kube-proxy-dpr4d 1/1 Running 2 5d4h
kube-system kube-scheduler-m3 1/1 Running 2 7d3h
kube-system kube-vip-m3 1/1 Running 2 7d3h

確認 etcd Member 是不是都還健在

$ alias etcdctl="ETCDCTL_API=3 sudo /usr/local/bin/etcdctl \
--endpoints=127.0.0.1:2379,172.20.7.81:2379,172.20.7.82:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key"
$ etcdctl member list
768bf4f42edfb9b3, started, m1, https://172.20.7.80:2380, https://172.20.7.80:2379, false
e044dfaaca4a3cf4, started, m2, https://172.20.7.81:2380, https://172.20.7.81:2379, false
ff78f71ad246a1cd, started, m3, https://172.20.7.82:2380, https://172.20.7.82:2379, false

確認 etcd 狀態

$ etcdctl endpoint status -w table
+------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| 127.0.0.1:2379 | 768bf4f42edfb9b3 | 3.4.13 | | 9.0 MB | 4.3 MB | 52% | 0 B | false | false | 243 | 1963834 | 1963834 | | |
false |
| 172.20.7.81:2379 | e044dfaaca4a3cf4 | 3.4.13 | | 8.9 MB | 4.3 MB | 52% | 0 B | true | false | 243 | 1963834 | 1963834 | | |
false |
| 172.20.7.82:2379 | ff78f71ad246a1cd | 3.4.13 | | 8.9 MB | 4.3 MB | 52% | 0 B | false | false | 243 | 1963834 | 1963834 | | |
false |
+------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

透過 IS LEADER 欄位確認 m2 主機是 etcd leader

4. 挑戰破壞 etcd leader vm 並透過再生龍備份的硬碟 restore 回去

開始備份與還原

確認當前哪一台節點是 etcd leader

$ etcdctl endpoint status --cluster -w table
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://172.20.7.81:2379 | 31954c2b28bfea6a | 3.4.13 | | 9.0 MB | 3.9 MB | 57% | 0 B | false | false | 244 | 2226833 | 2226833 | | | false |
| https://172.20.7.80:2379 | 768bf4f42edfb9b3 | 3.4.13 | | 9.0 MB | 3.9 MB | 57% | 0 B | true | false | 244 | 2226833 | 2226833 | | | false |
| https://172.20.7.82:2379 | ff78f71ad246a1cd | 3.4.13 | | 8.9 MB | 3.9 MB | 57% | 0 B | false | false | 244 | 2226833 | 2226833 | | | false |
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

由以上執行結果得知 leader 是 m1 節點

先透過再生龍備份 m1 主機的硬碟 sda (備份動作請參考前面的步驟，這裡不再贅述)

透過原來的硬碟 sda 將 VM 開機，並破壞 K8s

$ ssh m1
$ sudo systemctl disable --now containerd.service
$ sudo systemctl disable --now kubelet
$ sudo rm -r /etc/containerd/
$ sudo rm -r /etc/kubernetes/
$ sudo poweroff

透過再生龍將備份好的 disk image 還原到新的硬碟 sdc 上 (還原動作請參考前面的步驟，這裡不再贅述)
透過還原好的硬碟 sdc 將 VM 開機

檢查在 m1 節點的 pod 是否有異常

$ kubectl get pods --field-selector spec.nodeName=m1 -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-558x6 1/1 Running 1 7d21h
kube-system etcd-m1 0/1 CrashLoopBackOff 7 44h
kube-system kube-apiserver-m1 0/1 Error 3 44h
kube-system kube-controller-manager-m1 1/1 Running 2 44h
kube-system kube-proxy-6w529 1/1 Running 1 7d21h
kube-system kube-scheduler-m1 1/1 Running 2 44h
kube-system kube-vip-m1 1/1 Running 3 7d21h

發現 etcd 跟 API Server 掛了

查找 etcd 死亡原因

$ kubectl -n kube-system logs etcd-m1

執行結果如下：

... 以上省略
2025-07-09 03:45:22.108428 I | rafthttp: started streaming with peer ff78f71ad246a1cd (stream MsgApp v2 reader)
raft2025/07/09 03:45:22 INFO: 768bf4f42edfb9b3 [term: 244] received a MsgHeartbeat message with higher term from 31954c2b28bfea6a [term: 245]
raft2025/07/09 03:45:22 INFO: 768bf4f42edfb9b3 became follower at term 245
raft2025/07/09 03:45:22 tocommit(2238877) is out of range [lastIndex(2235883)]. Was the raft log corrupted, truncated, or lost?
panic: tocommit(2238877) is out of range [lastIndex(2235883)]. Was the raft log corrupted, truncated, or lost?
goroutine 75 [running]:
log.(*Logger).Panicf(0xc0001e6140, 0x10db3a0, 0x5d, 0xc000fa6380, 0x2, 0x2)
/usr/local/go/src/log/log.go:219 +0xc1
go.etcd.io/etcd/raft.(*DefaultLogger).Panicf(0x1ac1310, 0x10db3a0, 0x5d, 0xc000fa6380, 0x2, 0x2)
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/logger.go:127 +0x60
go.etcd.io/etcd/raft.(*raftLog).commitTo(0xc00015e150, 0x22299d)
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/log.go:203 +0x131
go.etcd.io/etcd/raft.(*raft).handleHeartbeat(0xc0000f9a40, 0x8, 0x768bf4f42edfb9b3, 0x31954c2b28bfea6a, 0xf5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/raft.go:1396 +0x54
go.etcd.io/etcd/raft.stepFollower(0xc0000f9a40, 0x8, 0x768bf4f42edfb9b3, 0x31954c2b28bfea6a, 0xf5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/raft.go:1341 +0x480
go.etcd.io/etcd/raft.(*raft).Step(0xc0000f9a40, 0x8, 0x768bf4f42edfb9b3, 0x31954c2b28bfea6a, 0xf5, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/raft.go:984 +0x1113
go.etcd.io/etcd/raft.(*node).run(0xc0003bd020)
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/node.go:352 +0xab6
created by go.etcd.io/etcd/raft.RestartNode
/tmp/etcd-release-3.4.13/etcd/release/etcd/raft/node.go:240 +0x33c

🔍 分析 etcd 毀損原因

收到 heartbeat，變成 follower
```
 INFO: 768bf4f42edfb9b3 [term: 244] received a MsgHeartbeat message with higher term from 31954c2b28bfea6a [term: 245]
INFO: 768bf4f42edfb9b3 became follower at term 245
```
表示本節點（768bf4f42edfb9b3）接受到領導者心跳，發現自己在舊 term（244），於是以新 term（245）降階為 follower。這是正常 Raft 流程。
提交索引 tocommit 超出日誌範圍
```
tocommit(2238877) is out of range [lastIndex(2235883)]
```
這裡 leader 傳來了一筆要 commit 的索引 2238877，但本地的最後日誌索引只有 2235883。缺少 2994 筆日誌紀錄，即 index gap。Raft 本意要求 follower 在 commit 前先擁有所有日誌條目；若 commit index 超出本地 lastIndex，代表日誌是不連續、不完整的。
導致 panic 崩潰 Raft 為了保障一致性，若遇到不合法的 commit 請求（tocommit 超界），會直接 panic 停止，以避免資料錯誤或分裂。這是設計上的安全機制。

💥 Root Cause

Raft 在 etcd 中負責維持複本間的一致性（consensus），當一個成員嘗試啓動並提交過舊的資料時，領導者會嘗試將該成員更新到目前的提交進度。如果領導者無法更新該成員，就會觸發 raft 錯誤，以避免集群內資料損壞，並導致該成員因 raft panic 而啓動失敗。

這個問題可能發生於某個 etcd 成員長時間與其他成員隔離，導致 etcd 領導者無法保留所有的提交歷史來正確更新該成員。

✅ 解決辦法

檢查當前 etcd member

$ etcdctl member list
31954c2b28bfea6a, started, m2, https://172.20.7.81:2380, https://172.20.7.81:2379, false
768bf4f42edfb9b3, started, m1, https://172.20.7.80:2380, https://172.20.7.80:2379, false
ff78f71ad246a1cd, started, m3, https://172.20.7.82:2380, https://172.20.7.82:2379, false

檢查每個 etcd 的狀態

$ etcdctl endpoint status --cluster -w table
{"level":"warn","ts":"2025-07-09T13:10:30.247283+0800","logger":"etcd-client","caller":"v3@v3.6.1/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000480000/172.20.7.80:2379","method":"/etcdserverpb.Maintenance/Status","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing: dial tcp 172.20.7.80:2379: connect: connection refused\""}
Failed to get the status of endpoint https://172.20.7.80:2379 (context deadline exceeded)
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://172.20.7.81:2379 | 31954c2b28bfea6a | 3.4.13 | | 9.0 MB | 4.0 MB | 56% | 0 B | true | false | 245 | 2264113 | 2264113 | | | false |
| https://172.20.7.82:2379 | ff78f71ad246a1cd | 3.4.13 | | 8.9 MB | 4.0 MB | 56% | 0 B | false | false | 245 | 2264130 | 2264130 | | | false |
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

確認在 m1 node 的 etcd 已故障

在 m1 主機停止 etcd container

sudo mv /etc/kubernetes/manifests/etcd.yaml .

將 m1 移出 etcd cluster，注意此命令須在健康的 etcd 節點執行

etcdctl member remove 768bf4f42edfb9b3

執行結果：

Member 768bf4f42edfb9b3 removed from cluster 4bfe2fbd656adb1a

清理在 m1 主機上 etcd 的資料

ssh m1
sudo mv /var/lib/etcd/member /var/lib/etcd/member-old
sudo mkdir -p /var/lib/etcd/member
sudo chmod 700 /var/lib/etcd/member

修改 etcd 參數

sudo nano etcd.yaml

修改內容如下：

spec:
containers:
- command:
- etcd
- --advertise-client-urls=https://172.20.7.80:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --initial-advertise-peer-urls=https://172.20.7.80:2380
#- --initial-cluster=m1=https://172.20.7.80:2380
## 將上一行註解，並添加以下這行，如果原本有以下兩個參數，就修改成跟以下一樣
- --initial-cluster=m1=https://172.20.7.80:2380,m2=https://172.20.7.81:2380,m3=https://172.20.7.82:2380
- --initial-cluster-state=existing

將 m1 etcd 以 learner 的身份加回 etcd cluster，注意此命令須在健康的 etcd 節點執行

etcdctl member add m1 --peer-urls="https://172.20.7.80:2380" --learner

執行結果：

Member 3bf15e520c24448c added as learner to cluster 4bfe2fbd656adb1a
ETCD_NAME="m1"
ETCD_INITIAL_CLUSTER="m2=https://172.20.7.81:2380,m1=https://172.20.7.80:2380,m3=https://172.20.7.82:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.20.7.80:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

在 m1 主機啟動 etcd container

sudo mv etcd.yaml /etc/kubernetes/manifests/

確認 m1 主機的 etcd 是否成功加入叢集，並啟動

etcdctl member list -w table

執行結果：

+------------------+---------+------+--------------------------+--------------------------+------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
+------------------+---------+------+--------------------------+--------------------------+------------+
| 31954c2b28bfea6a | started | m2 | https://172.20.7.81:2380 | https://172.20.7.81:2379 | false |
| fbc0ce2210efe023 | started | m1 | https://172.20.7.80:2380 | https://172.20.7.80:2379 | true |
| ff78f71ad246a1cd | started | m3 | https://172.20.7.82:2380 | https://172.20.7.82:2379 | false |
+------------------+---------+------+--------------------------+--------------------------+------------+

確認 m1 主機 etcd 同步狀態

etcdctl endpoint status --cluster -w table

執行結果：

+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| https://172.20.7.81:2379 | 31954c2b28bfea6a | 3.4.13 | | 9.0 MB | 4.3 MB | 53% | 0 B | true | false | 379 | 2314290 | 2314290 | | | false |
| https://172.20.7.80:2379 | fbc0ce2210efe023 | 3.4.13 | | 9.0 MB | 4.3 MB | 53% | 0 B | false | true | 379 | 2314290 | 2314290 | | | false |
| https://172.20.7.82:2379 | ff78f71ad246a1cd | 3.4.13 | | 8.9 MB | 4.3 MB | 53% | 0 B | false | false | 379 | 2314290 | 2314290 | | | false |
+--------------------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+

觀察 RAFT_INDEX 與 RAFT_APPLIED_INDEX，若兩者相同或非常接近，代表該 learner 已完全跟上 leader 的日誌，catch‑up 已完成

確認 etcd learner 已同步資料後，將它升級為 member

etcdctl member promote fbc0ce2210efe023

執行結果：

Member fbc0ce2210efe023 promoted in cluster 4bfe2fbd656adb1a

確認 API Server Contaienr 運作正常

$ sudo crictl ps --name kube-apiserver

執行結果：

CONTAINER IMAGE CREATED STATE NAME ATTEMPT
POD ID POD NAMESPACE
8a13e70428b12 e58b890e4ab44 About a minute ago Running kube-apiserver 7
010418cc69c62 kube-apiserver-m1 kube-system

確認 m1 主機的 Pods 運作正常

kubectl get pods --field-selector spec.nodeName=m1 -A

執行結果：

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-558x6 1/1 Running 1 8d
kube-system etcd-m1 1/1 Running 0 25m
kube-system kube-apiserver-m1 1/1 Running 7 36m
kube-system kube-controller-manager-m1 1/1 Running 2 2d1h
kube-system kube-proxy-6w529 1/1 Running 1 8d
kube-system kube-scheduler-m1 1/1 Running 2 2d1h
kube-system kube-vip-m1 1/1 Running 3 8d

5. 參考資料

Clonezilla live 文件集

尋找 K8s pod 在 node 上對應的網卡

Sat, 28 Jun 2025 04:26:34 +0800

# 1. 在 Kubernetes 建立一個 Pod，命名為 n1，使用官方 nginx 映像檔
$ kubectl run n1 --image=docker.io/library/nginx
pod/n1 created
# 2. 查詢 Pod 的 IP、所屬節點等詳細資訊
$ kubectl get pod n1 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
n1 1/1 Running 0 6m46s 10.244.9.12 tkbp-worker2 <none> <none>
# 進到 tkbp-worker2 節點執行以下命令
# 3. 列出所有 PID 命名空間（lsns 為列出 Linux namespace 狀態的工具）
$ lsns -t pid
NS TYPE NPROCS PID USER COMMAND
...以上省略
4026533346 pid 5 61422 root nginx: master process nginx -g daemon off;
# 4. 查詢 nginx 程序的 pid 所屬的 linux network namespace
$ ip netns identify 61422
cni-343bc6af-c579-f885-ad00-5dd56292fb5b
# 5. 進入該 linux network namespace，查詢 app container 內的網路介面與 IP 設定
$ ip netns exec cni-343bc6af-c579-f885-ad00-5dd56292fb5b ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether e6:88:62:99:ad:f4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.9.12/32 scope global eth0
valid_lft forever preferred_lft forever
# 從以上結果來看 pod 的 IP 位址 10.244.9.12 綁定在網路介面 eth0 上，而 eth0 被連接到 9 號介面上。
# 6. 在節點主機上，查看 9 號網路介面資訊
$ ip a s
...以上省略
9: calif87b22aca35@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-343bc6af-c579-f885-ad00-5dd56292fb5b

實作 K8s 網站應用 Graceful shutdown

Thu, 19 Jun 2025 16:27:14 +0800

1. 前情提要

可以試想現在有一台電腦直接被按下關機鍵，上面的程式會無法正確跑完、使用者也無法正確的完成執行到一半的動作。

而關掉 HTTP Server 也是同理，處理到一半的 HTTP 請求會直接失敗，處理不好的話會讓使用者有不好的體驗或是甚至在資料庫留下處理一半的資料。

那該如何優雅地關閉服務?

2. 先備知識

2.1. 什麼是 Graceful Shutdown？

優雅的關閉就是要確保網站服務在完成現有操作的同時安全終止，而不是突然停止並可能導致任務未完成。

在任何應用程式中，所謂「優雅關閉」（Graceful Shutdown）通常要滿足三個基本條件：

關閉進入口：先停止接收新的請求或訊息來源，例如來自 HTTP、訊息佇列（pub/sub）等；但對於連到資料庫或快取系統的「對外連線」，可以先保持不關閉。
等待目前正在處理的請求完成：讓已經在處理中的請求有機會執行完；如果某些請求拖太久，也要回應一個適當的錯誤訊息（讓使用者知道中斷原因）。
釋放重要資源：像是釋放資料庫連線、解除檔案鎖定、關閉網路監聽器等。這個階段也會做一些最後的清理工作，避免資源洩漏或程式異常退出。
- 釋放的意思是把原本占用的資源歸還給系統，讓其他程式可以使用。
- 「Database connections（資料庫連線）」是應用程式和資料庫之間的一條持續通訊通道，讓你的程式可以讀寫資料庫裡的資料（像是查詢、寫入、更新等）。
- 「File lock（檔案鎖）」是一種防止多個程式同時存取同一個檔案而導致資料錯亂或損壞的機制。
- 「Network listeners（網路監聽器）」是指應用程式中負責等待外部連線請求的元件。簡單來說，它就像一個「守門員」，在某個 IP 和 Port 上「聽」有沒有人要來連線。

2.2. Kubernetes 如何終止 Pod？

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination

2.3. 認識 Kubernetes 的 Container Lifecycle Hooks：PreStop

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

3. 開始動手實作

3.1. 使用 Go 語言實作支援 Graceful Shutdown 的 HTTP 伺服器

3.2. 將應用程式 Container 化（Containerize the App）

3.3. 設定 K8s Pod 啟用 PreStop hook

參考文章 :

K8s 實作應用程式零停機部署更新

Sat, 24 May 2025 18:41:22 +0800

1. 目標

在 Kubernetes 中 Pod 數量動態縮減（scale down）時，若該 Pod 仍有連線，則延遲其終止，直到所有連線結束後再移除該 Pod。

2. 實現方法

Readiness Probe：定期檢查應用是否準備好接收流量。當探針失敗時，Kubernetes 會將該 Pod 從 Service 的 endpoints 中移除，停止將新流量導向此 Pod。
PreStop Hook：在容器接收到終止信號（SIGTERM）前執行的命令。可用於執行清理工作，例如等待現有連線結束。
terminationGracePeriodSeconds：定義 Kubernetes 在強制終止容器前等待的時間。這段時間包括 PreStop Hook 的執行時間和容器關閉所需的時間。

3. 關鍵概念解釋

PreStop Hook 的執行時機
- 當 Kubernetes 決定終止一個 app container 時，會先執行該 app container 定義的 PreStop hook。
- 這個 hook 不是非同步 執行的；也就是說，在 PreStop hook 完成之前，Kubernetes 不會發送 SIGTERM 信號給 app container 的主進程。
terminationGracePeriodSeconds 的作用
- terminationGracePeriodSeconds 定義了 Kubernetes 在強制終止 app container 之前，給予 app container 的總寬限時間。
- 這個時間包括了 PreStop hook 的執行時間以及 app container 在接收到 SIGTERM 信號後的關閉時間
PreStop Hook 執行時間過長
- 如果 PreStop hook 在執行時卡住或耗時過長，導致超過了 terminationGracePeriodSeconds 所設定的時間，Kubernetes 會強制終止 app container 。
- 此時，Pod 的狀態會保持在 Terminating，直到寬限時間結束，Kubernetes 發送 SIGKILL 信號強制終止 app container 。

3.1. 舉例說明

假設你設定了：

terminationGracePeriodSeconds: 60（即 pod 要被強制刪除的寬限時間為 60 秒）
PreStop hook 執行時間為 55 秒
app container 在接收到 SIGTERM 信號後，需要 10 秒才能正常關閉

在這種情況下，總共需要 55（PreStop）+ 10（ app container 關閉）= 65 秒，但 terminationGracePeriodSeconds 只有 60 秒。

因此，Kubernetes 會在 60 秒時強制終止 app container ，導致 app container 無法正常完成關閉流程。

3.2. 實務建議

合理設定 terminationGracePeriodSeconds：確保這個值大於等於 PreStop hook 的執行時間加上 app container 關閉所需的時間。

4. 撰寫 Pod YAML

實測 API Server 會 cache K8s metadata

Sat, 17 May 2025 10:08:42 +0800

API Server data flow

API Server 可以視為在 etcd 前面的一个代理（proxy）

 +--------+ +---------------+ +------------+
| Client | -----------> | Proxy (cache) | --------------> | Data store |
+--------+ +---------------+ +------------+
infra services apiserver etcd

絕大部分情況下，Api Server 直接從本地快取提供服務（因為它快取了叢集全量資料）；

某些特殊情況，例如如下比較常見的 2 種:

客戶端明確要求從 etcd 讀取資料（追求最高的資料準確性），
apiserver 本地快取還沒建好 apiserver 就只能將請求轉寄給 etcd, 這裡就要特別注意了客戶端 LIST 參數設定不當也可能會走到這個邏輯

常見的 List 請求可歸類為兩種：

List 全量資料：開銷主要花在資料傳輸；
List 指定用 label 或欄位（field）過濾，只需要匹配的資料。

這裡要特別說明的是第二種情況，也就是 list 請求帶了過濾條件，不要以為 list 帶了過濾條件就不會全量查詢 etcd ,也分兩種情況:

大部分情況下，apiserver 會用自己的快取做過濾，走快取的操作很快，直接從 apiserver 的記憶體就可以返回，因此耗時主要花在資料網路傳輸
需要將請求轉給 etcd 的情況

注意，etcd 只是 Key/Value(KV) 存儲，並不理解 label/field 訊息，因此在 etcd 層面無法處理過濾請求。實際的過程是：apiserver 從 etcd 拉全量數據，然後在記憶體做過濾，再回傳給客戶端。因此除了資料傳輸開銷（網路頻寬），這種情況下還會佔用大量 apiserver CPU 和記憶體

以幾個常見的 LIST 請求為例：

LIST api/v1/pods?limit=500&timeout=30s&resourceVersion=0

這個請求是獲取 K8s 上的所有 pods

這裡同時傳了兩個參數，但 resourceVersion=0 會導致 apiserver 忽略 limit=500，所以客戶端拿到的是全量 pods 的資料。

但由於指定了 resourceVersion=0, 所以雖然是全量數據，但是會直接從 apiserver 的快取中返回。

LIST api/v1/pods?fieldSelector=spec.nodeName%3Dtesting

這個請求是獲取 testing node 上的所有 pods（%3D 是 =的轉義）。

根據 nodename 做過濾，給人的感覺可能是資料量不太大，但其實背後要比看起來複雜：

首先，這裡沒有指定 resourceVersion=0，導致 apiserver 跳過緩存，直接去 etcd 讀資料；其次，etcd 只是 KV 存儲，沒有按 label/field 過濾功能（只處理limit/continue），所以 apiserver 是從 etcd 拉全量的 pods 數據，然後在記憶體做 fieldselector 過濾，開銷也是很大的，這種行為是要避免的，除非對資料準確度有極高要求，特意要繞過 apiserver 快取。

LIST api/v1/pods?filedSelector=spec.nodeName%3Dnode1&resourceVersion=0

跟 2 的差別是加上了 resourceVersion=0，因此 apiserver 會從快取讀取數據，效能會有量級的提升。

但要注意，雖然實際上回傳給客戶端的可能只有幾百 KB 到上百 MB , 但 apiserver 需要處理的資料量可能是幾個 GB。

以上可以看到，不同的 LIST 操作產生的影響是不一樣的，而客戶端看到資料還有可能只是 apiserver/etcd 處理資料的一小部分。如果基礎服務大規模啟動或重啟，就極有可能把控制平面打爆。

總結一下

resourceVersion 作用: 保證客戶端資料一致性與順序性，樂觀鎖，實現並發控制

設定 ListOptions 時，resourceVersion 有三種設定方法：

不設定(不傳遞 ListOptions 或不設定 resourceVersion 欄位)，此時會直接從 etcd 讀取，此時資料是最新的
設定為0，此時會從 API Server Cache 中取得數據
設定為指定的 resourceVersion，取得 resourceVersion 大於指定版本的所有資源對象

開始實作

# 進到 Control-plane node
$ ssh <contol-plane node>
# 檢視原本 default namespace 有的 pods
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-5c9848d864-t7mj7 1/1 Running 2 (7h43m ago) 8d
synergy-leverager-5469477c9d-flrst 0/1 Error 0 8d
synergy-leverager-5469477c9d-gs6sv 1/1 Running 2 (7h43m ago) 8d
# 將 etcd manifests 移出 static pod 目錄區
$ mv /etc/kubernetes/manifests/etcd.yaml ~
# 確認 etcd 去世
$ crictl ps --name etcd -a
ONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
56e144e375402 a9e7e6b294baf 5 minutes ago Exited etcd 0 0353cbdc58e9a etcd-cka kube-system
# 確認 api/v1/namespaces/kube-system/pods?limit=500 這 API Call 要跟 ETCD 拿資料的話，會得不到資料
$ kubectl get pods --field-selector=spec.nodeName=testing --request-timeout=3s -v 6
I0517 17:01:27.466943 686541 loader.go:402] Config loaded from file: /root/.kube/config
I0517 17:01:27.467316 686541 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0517 17:01:27.467357 686541 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0517 17:01:27.467363 686541 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0517 17:01:27.467367 686541 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0517 17:01:30.474745 686541 round_trippers.go:560] GET https://172.16.8.121:6443/api/v1/namespaces/default/pods?fieldSelector=spec.nodeName%3Dtesting&limit=500&timeout=3s in 3002 milliseconds
I0517 17:01:30.474847 686541 helpers.go:264] Connection error: Get https://172.16.8.121:6443/api/v1/namespaces/default/pods?fieldSelector=spec.nodeName%3Dtesting&limit=500&timeout=3s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Unable to connect to the server: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
# 確認添加 resourceVersion=0 參數一定會跟 API Server 本地的 cache 拿資料
$ kubectl get --raw '/api/v1/namespaces/default/pods?resourceVersion=0' | jq
...
"containers": [
{
"name": "synergy-leverager",
"image": "quay.io/flysangel/library/busybox",
"command": [
"/bin/sh"
],
"args": [
"-c",
"i=0;\nmkdir -p /var/log;\nwhile true;\ndo\n echo \"$(date) INFO $i\" | tee -a /var/log/synergy-leverager.log;\n i=$((i+1));\n sleep 1;\ndone\n"
],
"resources": {},
"volumeMounts": [
{
"name": "kube-api-access-tm6cz",
"readOnly": true,
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
}
],
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "Always"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 30,
"dnsPolicy": "ClusterFirst",
"serviceAccountName": "default",
"serviceAccount": "default",
"nodeName": "cka",
"securityContext": {},
"schedulerName": "default-scheduler",
"tolerations": [
{
"key": "node.kubernetes.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
},
{
"key": "node.kubernetes.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 300
}
],
"priority": 0,
"enableServiceLinks": true,
"preemptionPolicy": "PreemptLowerPriority"
},
"status": {
"phase": "Running",
"conditions": [
{
"type": "PodReadyToStartContainers",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2025-05-17T01:08:12Z"
},
{
"type": "Initialized",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2025-05-08T17:22:59Z"
},
{
"type": "Ready",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2025-05-17T01:08:12Z"
},
{
"type": "ContainersReady",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2025-05-17T01:08:12Z"
},
{
"type": "PodScheduled",
"status": "True",
"lastProbeTime": null,
"lastTransitionTime": "2025-05-08T17:22:59Z"
}
],
"hostIP": "172.16.8.121",
"hostIPs": [
{
"ip": "172.16.8.121"
}
],
"podIP": "10.244.16.78",
"podIPs": [
{
"ip": "10.244.16.78"
}
],
"startTime": "2025-05-08T17:22:59Z",
"containerStatuses": [
{
"name": "synergy-leverager",
"state": {
"running": {
"startedAt": "2025-05-17T01:08:11Z"
}
},
"lastState": {
"terminated": {
"exitCode": 255,
"reason": "Unknown",
"startedAt": "2025-05-10T01:14:26Z",
"finishedAt": "2025-05-17T01:07:21Z",
"containerID": "containerd://01341ba2dccb4cee04f8429bf91840e2c7d39dc28f2eaa01ac20c0df86ae8aef"
}
},
"ready": true,
"restartCount": 2,
"image": "quay.io/flysangel/library/busybox:latest",
"imageID": "quay.io/flysangel/library/busybox@sha256:ad9fa4d07136a83e69a54ef00102f579d04eba431932de3b0f098cc5d5948f9f",
"containerID": "containerd://8e48c93e8098ce2e77e6a48ae1fe218ee18be6397b5198bf7b7bbe0939d72308",
"started": true,
"volumeMounts": [
{
"name": "kube-api-access-tm6cz",
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"readOnly": true,
"recursiveReadOnly": "Disabled"
}
]
}
],
"qosClass": "BestEffort"
}
}
]
}

REf

https://izsk.me/2023/12/14/Kubernetes-listOption-1/

K3s 系統性崩潰分析報告

Tue, 13 May 2025 16:48:38 +0800

K3s log 分析

etcd 要將資料寫入磁碟時超過 46 秒 (一般應在數毫秒～1 秒內完成)

{
"level": "warn",
"ts": "2025-05-11T00:22:46.781+0800",
"caller": "wal/wal.go:808",
"msg": "slow fdatasync",
"took": "46.889101479s",
"expected-duration": "1s"
}

etcd 處理一個單純 Read 請求耗時 45.89 秒，遠遠超出預期的 100 毫秒，而且最終還是失敗了（context deadline exceeded）。

{
"level": "warn",
"ts": "2025-05-11T00:22:46.791+0800",
"caller": "etcdserver/util.go:163",
"msg": "apply request took too long",
"took": "45.8901115s",
"expected-duration": "100ms",
"prefix": "read-only range ",
"request": "key:\"/registry/mutatingwebhookconfigurations/vault-agent-injector-cfg\" ",
"response": "",
"error": "context deadline exceeded"
}

etcd 為了完成高一致性讀取（linearized read），花了 45.89 秒等待 raft 演算法取得共識

{
"level": "info",
"ts": "2025-05-11T00:22:46.791+0800",
"caller": "traceutil/trace.go:145",
"msg": "trace[381326135] range",
"detail": "{range_begin:/registry/mutatingwebhookconfigurations/vault-agent-injector-cfg; range_end:; }",
"duration": "45.890191566s",
"start": "2025-05-11T00:22:00.901+0800",
"end": "2025-05-11T00:22:46.791+0800",
"steps": [
"trace[381326135] agreement among raft nodes before linearized reading (duration: 45.890156006s)"
]
}

檢查本地 etcd 執行狀態失敗，因為檢查超過最大等候時間，仍未完成回應

{
"time": "2025-05-11T00:22:46.791838023+08:00",
"level": "error",
"msg": "Failed to check local etcd status for learner management: context deadline exceeded"
}

K3s 再次嘗試從 etcd 讀取 PVC（PersistentVolumeClaim）資源時，為了保證讀取一致性（linearized read），觸發了 etcd 的 Raft 共識程序。不幸的是，這個操作被卡住了整整 46.9 秒（遠超出預期的 100 毫秒），最終因逾時（request timed out）失敗。

[
{
"log_time": "May 11 00:22:48",
"host": "fetci",
"process": "k3s",
"pid": 1321,
"entry": {
"level": "warn",
"ts": "2025-05-11T00:22:46.808+0800",
"caller": "etcdserver/util.go:163",
"msg": "apply request took too long",
"took": "46.914240959s",
"expected-duration": "100ms",
"prefix": "read-only range ",
"request": "key:\"/registry/persistentvolumeclaims/minio/data-2-minio-1\" ",
"response": "",
"error": "etcdserver: request timed out"
}
},
{
"log_time": "May 11 00:22:48",
"host": "fetci",
"process": "k3s",
"pid": 1321,
"entry": {
"level": "info",
"ts": "2025-05-11T00:22:46.808+0800",
"caller": "traceutil/trace.go:145",
"msg": "trace[656268623] range",
"detail": "{range_begin:/registry/persistentvolumeclaims/minio/data-2-minio-1; range_end:; }",
"duration": "46.914291657s",
"start": "2025-05-11T00:21:59.894+0800",
"end": "2025-05-11T00:22:46.808+0800",
"steps": [
"trace[656268623] 'agreement among raft nodes before linearized reading' (duration: 46.914239825s)"
]
}
}
]

K3s 在 2025-05-11 00:22:46 時，因 etcd 效能嚴重惡化，導致 Kubernetes 內部的 leader election 無法維持，當前節點放棄領導地位（stopped leading）。隨後，關鍵元件（如 HPA、PVC、DaemonSet、Volume 管理等 controller）陸續進入 shutdown，Kubelet 因 container runtime 不可用而停止同步 Pod，API server 請求大量 timeout，整個 control plane 逐步失能。這是一場由 etcd timeout 引發的 control plane node 系統性崩潰。

May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.308855 1321 horizontal.go:215] horizontal pod autoscaler controller worker shutting down
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.365686 1321 event.go:291] "Event occurred" object="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="fetci_29748a93-d583-4cd5-bb55-a9c43db0a7ab stopped leading"
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.365719 1321 pvc_protection_controller.go:122] "Shutting down PVC protection controller"
May 11 00:22:48 fetci k3s[1321]: E0511 00:22:46.423085 1321 leaderelection.go:325] error retrieving resource lock kube-system/cloud-controller-manager: Get "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s": context deadline exceeded
May 11 00:22:48 fetci k3s[1321]: E0511 00:22:46.423232 1321 kubelet.go:1870] "Skipping pod synchronization" err="container runtime is down"
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.480179 1321 pv_controller_base.go:461] volume worker queue shutting down
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.480226 1321 daemon_controller.go:299] Shutting down daemon sets controller
May 11 00:22:48 fetci k3s[1321]: E0511 00:22:46.480295 1321 wrap.go:54] timeout or abort while handling: GET "/apis/coordination.k8s.io/v1/namespaces/tigera-operator/leases/operator-lock"
May 11 00:22:48 fetci k3s[1321]: E0511 00:22:46.480634 1321 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.480859 1321 endpointslicemirroring_controller.go:224] Shutting down EndpointSliceMirroring controller
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.481235 1321 pv_protection_controller.go:95] Shutting down PV protection controller
May 11 00:22:48 fetci k3s[1321]: I0511 00:22:46.768134 1321 trace.go:205] Trace[1208467430]: "GuaranteedUpdate etcd3" type:*coordination.Lease (11-May-2025 00:22:06.292) (total time: 40475ms):
May 11 00:22:48 fetci k3s[1321]: Trace[1208467430]: [40.475369945s] [40.475369945s] END

systemd 偵測到 K3s 無法繼續正常運作，觸發服務重啟機制。

May 11 00:22:53 fetci systemd[1]: k3s.service: Service hold-off time over, scheduling restart.
May 11 00:22:53 fetci systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1.
May 11 00:22:53 fetci systemd[1]: Stopped Lightweight Kubernetes.

原始 log 資訊

May 11 00:22:48 rancher k3s[1321]: {"level":"warn","ts":"2025-05-11T00:22:46.781+0800","caller":"wal/wal.go:808","msg":"slow fdatasync","took":"46.889101479s","expected-duration":"1s"}
May 11 00:22:48 rancher k3s[1321]: {"level":"warn","ts":"2025-05-11T00:22:46.782+0800","caller":"etcdserver/server.go:1079","msg":"failed to revoke lease","lease-id":"51ab96b8f865613c","error":"etcdserver: request timed out"}
May 11 00:22:48 rancher k3s[1321]: time="2025-05-11T00:22:46.788366607+08:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
May 11 00:22:48 rancher k3s[1321]: {"level":"warn","ts":"2025-05-11T00:22:46.791+0800","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"45.8901115s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/mutatingwebhookconfigurations/vault-agent-injector-cfg\" ","response":"","error":"context deadline exceeded"}
May 11 00:22:48 rancher k3s[1321]: {"level":"info","ts":"2025-05-11T00:22:46.791+0800","caller":"traceutil/trace.go:145","msg":"trace[381326135] range","detail":"{range_begin:/registry/mutatingwebhookconfigurations/vault-agent-injector-cfg; range_end:; }","duration":"45.890191566s","start":"2025-05-11T00:22:00.901+0800","end":"2025-05-11T00:22:46.791+0800","steps":["trace[381326135] 'agreement among raft nodes before linearized reading' (duration: 45.890156006s)"]}
May 11 00:22:48 rancher k3s[1321]: {"level":"warn","ts":"2025-05-11T00:22:46.791+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
May 11 00:22:48 rancher k3s[1321]: time="2025-05-11T00:22:46.791838023+08:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
May 11 00:22:48 rancher k3s[1321]: {"level":"warn","ts":"2025-05-11T00:22:46.808+0800","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"46.914240959s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/persistentvolumeclaims/minio/data-2-minio-1\" ","response":"","error":"etcdserver: request timed out"}
May 11 00:22:48 rancher k3s[1321]: {"level":"info","ts":"2025-05-11T00:22:46.808+0800","caller":"traceutil/trace.go:145","msg":"trace[656268623] range","detail":"{range_begin:/registry/persistentvolumeclaims/minio/data-2-minio-1; range_end:; }","duration":"46.914291657s","start":"2025-05-11T00:21:59.894+0800","end":"2025-05-11T00:22:46.808+0800","steps":["trace[656268623] 'agreement among raft nodes before linearized reading' (duration: 46.914239825s)"]}
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.308855 1321 horizontal.go:215] horizontal pod autoscaler controller worker shutting down
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.365686 1321 event.go:291] "Event occurred" object="" kind="Lease" apiVersion="coordination.k8s.io/v1" type="Normal" reason="LeaderElection" message="rancher_29748a93-d583-4cd5-bb55-a9c43db0a7ab stopped leading"
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.365719 1321 pvc_protection_controller.go:122] "Shutting down PVC protection controller"
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.423085 1321 leaderelection.go:325] error retrieving resource lock kube-system/cloud-controller-manager: Get "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeout=5s": context deadline exceeded
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.423232 1321 kubelet.go:1870] "Skipping pod synchronization" err="container runtime is down"
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.480179 1321 pv_controller_base.go:461] volume worker queue shutting down
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.480226 1321 daemon_controller.go:299] Shutting down daemon sets controller
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.480295 1321 wrap.go:54] timeout or abort while handling: GET "/apis/coordination.k8s.io/v1/namespaces/tigera-operator/leases/operator-lock"
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.480634 1321 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.480859 1321 endpointslicemirroring_controller.go:224] Shutting down EndpointSliceMirroring controller
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.481235 1321 pv_protection_controller.go:95] Shutting down PV protection controller
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.768134 1321 trace.go:205] Trace[1208467430]: "GuaranteedUpdate etcd3" type:*coordination.Lease (11-May-2025 00:22:06.292) (total time: 40475ms):
May 11 00:22:48 rancher k3s[1321]: Trace[1208467430]: [40.475369945s] [40.475369945s] END
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.779070 1321 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, context canceled]"
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.791360 1321 trace.go:205] Trace[800629099]: "GuaranteedUpdate etcd3" type:*admissionregistration.MutatingWebhookConfiguration (11-May-2025 00:22:00.896) (total time: 45894ms):
May 11 00:22:48 rancher k3s[1321]: Trace[800629099]: [45.894632804s] [45.894632804s] END
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.792371 1321 trace.go:205] Trace[1819075853]: "Patch" url:/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/vault-agent-injector-cfg,user-agent:vault-k8s/v0.0.0 (linux/amd64) kubernetes/$Format,client:172.99.56.6,accept:application/json, */*,protocol:HTTP/1.1 (11-May-2025 00:22:00.896) (total time: 45895ms):
May 11 00:22:48 rancher k3s[1321]: Trace[1819075853]: [45.895797558s] [45.895797558s] END
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.792578 1321 trace.go:205] Trace[1934991911]: "Create" url:/api/v1/namespaces/default/events,user-agent:k3s/v1.21.11+k3s1 (linux/amd64) kubernetes/4eb3a48/kube-controller-manager,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (11-May-2025 00:22:08.573) (total time: 38218ms):
May 11 00:22:48 rancher k3s[1321]: Trace[1934991911]: [38.218578079s] [38.218578079s] END
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.797679 1321 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
May 11 00:22:48 rancher k3s[1321]: I0511 00:22:46.799129 1321 trace.go:205] Trace[1793283042]: "GuaranteedUpdate etcd3" type:*core.Endpoints (11-May-2025 00:22:05.680) (total time: 41118ms):
May 11 00:22:48 rancher k3s[1321]: Trace[1793283042]: [41.118330989s] [41.118330989s] END
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.808172 1321 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
May 11 00:22:48 rancher k3s[1321]: E0511 00:22:46.815927 1321 server.go:273] "Unable to authenticate the request due to an error" err="context canceled"
May 11 00:22:53 rancher systemd[1]: k3s.service: Service hold-off time over, scheduling restart.
May 11 00:22:53 rancher systemd[1]: k3s.service: Scheduled restart job, restart counter is at 1.
May 11 00:22:53 rancher
systemd[1]: Stopped Lightweight Kubernetes.

Run Kube-vip as Cloud Provider on HA K3s

Sat, 10 May 2025 08:53:04 +0800

1. 先備知識

1.1. kube-vip 專案主要提供兩大功能領域：

透過 control-plane 的 VIP 實現高可用（HA）的 Kubernetes 叢集
提供 Kubernetes 中的 Service 類型： LoadBalancer

1.2. HA Kubernetes Control Plane

重新實作負載平衡功能：由於先前的請求，HTTP 負載平衡功能被移除，只保留了控制平面的高可用（HA）功能。此功能將重新實作，可能會採用原本的 round-robin HTTP 請求方式，或是改用 IPVS。
利用 Kubernetes API 判斷其他控制平面節點：一旦單節點叢集運行後，kube-vip 可以透過 Kubernetes API 來判斷其他控制平面成員。目前的做法是每個控制平面節點都必須由 Cluster-API provider 投放一份 static manifest。
重新評估 Raft：kube-vip 最初的設計是作為獨立於 Kubernetes 的 raft 叢集，但因為像 CAPV（Cluster API Provider vSphere）這類元件在升級過程中存在的一些限制，使得改為在 Kubernetes 內部使用 leaderElection 成為更好的選擇。

1.3. Kubernetes service type:LoadBalancer

每個 LoadBalancer 使用 ARP 的 LeaderElection：目前僅有被選為 leader 的單一 Pod 會處理某個 VIP 的所有流量。若能為每個 Service 產生一個獨立的 leaderElection token，將可讓服務在整個叢集中的所有 Pod 上分散部署與運作。

2. Install kube-vip as static pods

2.0. 本篇文章測試環境

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k3s-kubevip-a1 Ready control-plane,etcd,master,worker 13h v1.31.3+k3s1 172.20.6.61 <none> SUSE Linux Micro 6.0 6.4.0-18-default containerd://1.7.23-k3s2
k3s-kubevip-a2 Ready control-plane,etcd,master,worker 13h v1.31.3+k3s1 172.20.6.62 <none> SUSE Linux Micro 6.0 6.4.0-18-default containerd://1.7.23-k3s2
k3s-kubevip-a3 Ready control-plane,etcd,master,worker 13h v1.31.3+k3s1 172.20.6.63 <none> SUSE Linux Micro 6.0 6.4.0-18-default containerd://1.7.23-k3s2

底層是一台 VM 分別在各自的實體機上

實體網路速度 10 G

kubeproxy mode 為 ipvs

可透過在 K3s 任一 node 執行以下指令檢查 : curl http://localhost:10249/proxyMode

2.1. Set configuration details

設定用於 control plane node 的 VIP 位址：

export VIP=172.20.6.60

將 INTERFACE 名稱設定為將宣布 VIP 的 control plane node 上的介面名稱。在許多 Linux 發行版中，可以使用 ip a 指令找到它。

export INTERFACE=eth0

透過解析 GitHub API 取得 kube-vip 版本的最新版本。此步驟需要安裝 jq 和 curl。

KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")

2.2. Creating the manifest

現在已經設定好輸入值，我們可以拉取並執行 kube-vip image，並提供所需的 flags 與參數。一旦依照你選擇的方法（ARP 或 BGP）產生出靜態 Pod 的 manifest，如果你是運行多個 control plane 節點，請確保將該 manifest 放入每個 control plane 節點的 static manifest 目錄中（預設為 /etc/kubernetes/manifests）。

根據使用的 container runtime，請使用兩個別名命令中的其中一個來建立 kube-vip 命令，以 app container 的方式執行 kube-vip 映像檔。

若使用的是 containerd，請執行以下命令：

alias kube-vip="ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION; ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"

2.2.1. ARP

在設定好輸入參數與別名命令後，我們可以執行 kube-vip container 產生一個 Static Pod 的 manifest，並將其寫入 /var/lib/rancher/k3s/agent/pod-manifests/kube-vip.yaml 檔案中。因此，這個動作預期是在第一個 control plane 節點上執行。

這個設定會建立一個 manifest，啟動 kube-vip，以使用 leaderElection 方法與 ARP 提供 control plane 的虛擬 IP（VIP）以及 Kubernetes Service 的管理功能。當該實例被選為 leader 時，它會將 VIP 綁定到指定的網路介面上。這個行為與 Kubernetes 中的 LoadBalancer 類型 Service 相同。

kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--controlplane \
--services \
--arp \
--leaderElection \
--vipSubnet 16 \
--servicesElection \
--enableLoadBalancer \
--k8sConfigPath /etc/rancher/k3s/k3s.yaml | tee /var/lib/rancher/k3s/agent/pod-manifests/kube-vip.yaml

🔍 參數解析

--interface $INTERFACE：指定 kube-vip 綁定的網路介面，例如 eth0。
--address $VIP：設定虛擬 IP（VIP），作為控制平面和服務的入口點。
--controlplane：啟用控制平面 VIP 功能，提供高可用的 Kubernetes 控制平面。
--services：啟用對 Kubernetes 中 LoadBalancer 類型服務的支援。
--arp：使用 ARP 模式廣播 VIP，適用於 Layer 2 網路環境。
--leaderElection：啟用領導者選舉機制，確保在多個節點中只有一個節點持有 VIP。
--vipSubnet 16：設定 VIP 的子網掩碼為 /16。
--servicesElection：為每個服務啟用獨立的領導者選舉，允許服務在不同節點上分佈(也就是說這允許 kube-vip 在多個節點之間分配 loadBalancer 位址)。
--enableLoadBalancer：啟用 IPVS 負載平衡器，將流量分發到後端 Pod。
--k8sConfigPath /etc/rancher/k3s/k3s.yaml：指定 kube-vip 使用的 Kubernetes 配置檔案路徑。

📌 注意事項

確保 $INTERFACE 和 $VIP 變數已正確設定，並符合您的網路環境。
--vipSubnet 的設定應根據您的網路規劃進行調整，常見的值有 /32、/24 等。
在使用 ARP 模式時，確保網路設備允許 ARP 廣播。
--servicesElection 功能需要 kube-vip 版本 ≥ 0.5.0。
--serviceInterface 可以讓 loadbalancer service 的 ip 跑在這參數指定的網卡上。

kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--controlplane \
--services \
--arp \
--leaderElection \
--vipSubnet 16 \
--k8sConfigPath /etc/rancher/k3s/k3s.yaml | tee /var/lib/rancher/k3s/agent/pod-manifests/kube-vip.yaml

2.3. 檢視 kube-vip pod 狀態

kubectl -n kube-system get pods -o wide

執行結果 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-6cb54596b5-fp52d 1/1 Running 2 (116m ago) 13h 10.42.0.4 k3s-kubevip-a1 <none> <none>
kube-vip-cloud-provider-6ffc794f6-x9zr8 1/1 Running 0 36m 10.42.1.5 k3s-kubevip-a3 <none> <none>
kube-vip-k3s-kubevip-a1 1/1 Running 0 5m42s 172.20.6.61 k3s-kubevip-a1 <none> <none>
local-path-provisioner-598c97774d-dbfhd 1/1 Running 2 (116m ago) 13h 10.42.0.5 k3s-kubevip-a1 <none> <none>
metrics-server-66f4d95f59-r9kxn 1/1 Running 2 (116m ago) 13h 10.42.0.3 k3s-kubevip-a1 <none> <none>

2.4. 檢視網卡

連線到 kube-vip pod 所在的 node

ssh <user>@<node ip>

檢視 eth0 網卡資訊

ip a s eth0

執行結果 :

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether bc:24:11:0b:f6:e5 brd ff:ff:ff:ff:ff:ff
altname enp0s18
altname ens18
inet 172.20.6.61/16 brd 172.20.255.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.20.6.60/32 scope global eth0
valid_lft forever preferred_lft forever

可以看到這張網卡多了一個 IP 位址

2.5. 到其他 Control-plane node 重複執行 2.1 ~ 2.3 的操作步驟

2.6. 再次檢視 kube-vip pod 狀態

kubectl -n kube-system get pods -o wide

執行結果 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-6cb54596b5-fp52d 1/1 Running 2 (116m ago) 13h 10.42.0.4 k3s-kubevip-a1 <none> <none>
kube-vip-cloud-provider-6ffc794f6-x9zr8 1/1 Running 0 36m 10.42.1.5 k3s-kubevip-a3 <none> <none>
kube-vip-k3s-kubevip-a1 1/1 Running 0 5m42s 172.20.6.61 k3s-kubevip-a1 <none> <none>
kube-vip-k3s-kubevip-a2 1/1 Running 0 2m8s 172.20.6.62 k3s-kubevip-a2 <none> <none>
kube-vip-k3s-kubevip-a3 1/1 Running 0 35s 172.20.6.63 k3s-kubevip-a3 <none> <none>
local-path-provisioner-598c97774d-dbfhd 1/1 Running 2 (116m ago) 13h 10.42.0.5 k3s-kubevip-a1 <none> <none>
metrics-server-66f4d95f59-r9kxn 1/1 Running 2 (116m ago) 13h 10.42.0.3 k3s-kubevip-a1 <none> <none>

3. Install the kube-vip Cloud Provider

kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml

檢查

kubectl -n kube-system get pods -l "app=kube-vip"

執行結果如下 :

NAME READY STATUS RESTARTS AGE
kube-vip-cloud-provider-88689889-62msw 1/1 Running 0 8m40s

4. Create a global CIDR or IP Range

為了讓 kube-vip 能夠為類型為 LoadBalancer 的 Service 設定 IP 位址，必須先提供可供分配的 IP 位址範圍。這些資訊儲存在 Kubernetes 的 ConfigMap 中，kube-vip 可以存取該 ConfigMap。您可以透過 ConfigMap 中的 key 來控制 IP 分配的範圍。可以指定 CIDR 區段或 IP 範圍，並將其設定為全域（整個叢集）或特定 Namespace 使用。

若要允許 kube-vip 在任何 Namespace 中為類型為 LoadBalancer 的 Service 分配 IP 位址，您可以建立一個名為 kubevip 的 ConfigMap，並設定 key 為 cidr-global，其值為您環境中可用的 CIDR 區段。例如，以下指令建立了一個全域的 CIDR 區段 172.20.6.65/32，kube-vip 將從中分配 IP 位址。

kubectl create configmap -n kube-system kubevip --from-literal cidr-global=172.20.6.65/32

其他指定 IP 的方式，請參考以下連結 : The kube-vip Cloud Provider ConfigMap - Kube-vip Docs

5. 建立測試的應用程式

kube-vip 的「平均」故障轉移時間約為 3 秒，但這可能會因底層基礎設施而有很大差異，例如虛擬和實體交換器可能會阻止或限制 kube-vip 更新網路。要開始評估您的基礎設施效能，可以進行以下最簡單的測試：

kubectl create deployment source-ip-app \
--image=registry.k8s.io/echoserver:1.10 \
--replicas 3

5.1. 檢視 pod 狀態

kubectl get pods -o wide

執行結果 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
source-ip-app-78477d56f4-cqmhh 1/1 Running 0 78s 10.42.0.10 k3s-kubevip-a1 <none> <none>
source-ip-app-78477d56f4-fw6d7 1/1 Running 0 78s 10.42.2.2 k3s-kubevip-a2 <none> <none>
source-ip-app-78477d56f4-hsgrv 1/1 Running 0 78s 10.42.1.8 k3s-kubevip-a3 <none> <none>

5.2. 建一個 loadbalancer service

kubectl expose deployment source-ip-app \
--name=loadbalancer \
--port=80 \
--target-port=8080 \
--type=LoadBalancer

5.3. 檢視 SVC 的 `EXTERNAL-IP`

kubectl get svc

執行結果 :

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 13h
loadbalancer LoadBalancer 10.43.123.204 172.20.6.65 80:32303/TCP 2m11s

5.4. 查詢 kube-vip 為服務選舉的 leader 節點

kubectl get lease -n kube-system plndr-svcs-lock

執行結果 :

NAME HOLDER AGE
plndr-svcs-lock k3s-kubevip-a3 5h17m

5.5. 檢視網卡

連線到 kube-vip pod 所在的 node

ssh <user>@<node ip>

檢視 eth0 網卡資訊

ip a s eth0

執行結果 :

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether bc:24:11:70:32:5c brd ff:ff:ff:ff:ff:ff
altname enp0s18
altname ens18
inet 172.20.6.63/16 brd 172.20.255.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.20.6.65/16 brd 172.20.255.255 scope global secondary eth0
valid_lft forever preferred_lft forever

可以看到這張網卡多了一個 IP 位址

5.6 . 測試 Failover

在同網路的其他機器執行以下命令

while true; do start=$(date +%s%3N); wget --timeout=1 -qO - 172.20.6.65 | grep Hostname || echo "wget error"; end=$(date +%s%3N); echo "耗時: $((end - start)) 毫秒"; sleep 1; done

執行結果 :

Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 7 毫秒
Hostname: source-ip-app-78477d56f4-sgm2x
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-sgm2x
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-sgm2x
耗時: 5 毫秒
...

將 kube-vip leader pod 所在的節點關機用以模擬故障

ssh <user>@<NodeIP>
sudo poweroff

執行結果 :

Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-sgm2x
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5007 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2006 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2007 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2006 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2006 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2006 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2006 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 2007 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒
Hostname: source-ip-app-78477d56f4-jjmhj
耗時: 6 毫秒
Hostname: source-ip-app-78477d56f4-wf8kw
耗時: 5 毫秒

實測下來 vip failover 的時間約 5 秒如果該節點同時是 cp 和 service 的 leader，failover 的時間為 9 秒左右

5.7 再次查詢 kube-vip leader

kubectl get lease -n kube-system plndr-svcs-lock

執行結果 :

NAME HOLDER AGE
plndr-svcs-lock k3s-kubevip-a3 5h17m

有發現 kube-vip 無法更新這個 lease，但 VIP 實際上已移轉至其他節點

5.8. 驗證網卡資訊

連線到 kube-vip pod 所在的 node

ssh <user>@<node ip>

檢視 eth0 網卡資訊

ip a s eth0

執行結果 :

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether bc:24:11:d7:a9:4d brd ff:ff:ff:ff:ff:ff
altname enp0s18
altname ens18
inet 172.20.6.62/16 brd 172.20.255.255 scope global noprefixroute eth0
valid_lft forever preferred_lft forever
inet 172.20.6.65/16 brd 172.20.255.255 scope global secondary eth0
valid_lft forever preferred_lft forever

6. 清除環境

# 1. 移除測試應用
kubectl delete deploy source-ip-app; kubectl delete svc loadbalancer
# 2. 移除 kube-vip cloud provider
kubectl apply -f https://raw.githubusercontent.com/kube-vip/kube-vip-cloud-provider/main/manifest/kube-vip-cloud-controller.yaml
# 3. 到每台 Contol-plane 移除 kube-vip static pod yaml
rm /var/lib/rancher/k3s/agent/pod-manifests/kube-vip.yaml

7. 參考文件

雲原生世代最佳舵手 Kubernetes

Sat, 03 May 2025 06:16:10 +0800

Kubernetes-Oerview

Kubernetes 新一代的雲端作業系統

管理實體電腦或虛擬電腦裡面的多台 Container
由 Google 公司設計

單純使用 Podman/Docker 會面臨的挑戰

單點毀損 Podman/Docker 啟動的 App Container 只會在一個機器上，如果該主機掛了，這個 Application 怎麼辦? 我們希望 Application 擁有容錯率，他能夠在另外一台 Podman/Docker 主機啟動一樣的 App Contianer，但 Podman/Docker 沒辦法自動做到這件事，只能靠工程師手動在另外一台機器上去做部署。
高可用性(High Availability) High Availability 的意思是，同一個 Applcation 在不同機器上同時都在運作，可以想成本尊跟分身的概念，如果一台電腦故障了，另外一台電腦上的 App 還是能持續提供服務。
自動橫向擴充假設當前的 Application 只有一個本尊跟分身，但萬一今天下午發生了一些事件，譬如應用系統是一個售票網站，在下午三點開賣，一堆流量灌進來，這時候我們就會希望有一個系統來會自動幫我們做到橫向擴充，英文專有名詞叫 Auto-scaling，也就是自動多長幾個分身出來一起提供服務。
Rolling updates 為了要讓企業的 Application 能達到長長久久穩穩當當，就還會需要 Rolling updates，因為任何一個企業開發的應用系統，一定會有第一代、第二代、第三代，會新增功能或刪除功能，也就會產生不同的版本，這時候就會需要系統要可以自動幫我們做進版跟退版。
資料儲存(Storage) Application 在運作的時候，一定會有資料的產生，所以還要幫 app 設計資料永存的解決方案，要讓 app 的資料能夠存的穩穩當當。

Kubernetes 最高戰略目標

幫企業 run Application ( 應用系統 ) run 的長長久久、穩穩當當

Kubernetes 的能力介紹

Containerisation (貨櫃化) has brought a lot of flexibility for developers in terms of managing the deployment of the applications. However, the more granular the application is, the more components it consists of and hence requires some sort of management for those.

One still needs to take care of scheduling the deployment of a certain number of containers to a specific node, managing networking between the containers, following the resource allocation, moving them around as they grow and much more.

Nearly all applications nowadays need to have answers for things like

Replication of components (Pod)
- 同樣的 pod 可以複製很多台出來 (本尊與分身的概念，分身也會做事)
- 叢集，同一個服務會有好幾個 pod 來提供服務，避免單點損毀的情況，提高可用性 (HA) 與容錯
Auto-scaling
- 自動橫向擴充，可以依照暴漲的流量擴充 pod
- 一開始只做一個本尊和一個分身的 pod (兩個相同的應用系統再跑)，當偵測到同一時間有大量連線時，會自動長出一樣的應用系統出來一起作業，連線恢復正常時，多的分身會消失
Load balancing
- 平衡負載，讓每個 pod 都能平均分工
Replication of components + Auto-scaling + Load balancing 能達到隨需擴增
Rolling updates
- 滾動式更新，專門給程式設計師使用，因增加或修改功能會更新程式版本，也就是進版或退版
- 當 Application 部屬到測試環境的 K8S Cluster 時，可檢查程式碼有無死邏輯，當服務發現有異常時，程式會退版，確認都無問題後，再部屬到 Production 環境
- Example: 藍綠部屬、金絲雀部屬
Logging across components
- 每台 pod 運作的 log 檔都會記錄下來
- grafana ( 發音 : 寡發哪 ) ：透過儀錶板來監控所有的 Container
Monitoring and health checking
- 透過 log 來監控 pod 的狀態和對 pod 進行健康檢查
由於在整個 K8S Cluster 上面會跑多個應用系統，所以需要 Logging across components + Monitoring and health checking 來幫助我們維運
Service discovery (DNS Server)
- DNS Server 的功能 : 將人看得懂的網站網址 (URL)，變成 IP 位址
- 由 CoreDNS System 得知，你的 Service 會對到哪一個 pod，會將 Service 的 Domain name 轉成綁在 Service 上的 pod 的 IP 位址。
- Domain name wiki link
Authentication
- 身分認證，透過憑證 ( 公鑰 and 私鑰 ) 作業
- 在 K8s 中沒人在打帳號密碼，全靠憑證作業。

Google has given a combined solution for that which is Kubernetes, or how it’s shortly called – K8s.

甚麼時候會需要用到隨需擴充?

網站伺服器，遇到多人大量同時連線時會需要，例如 : 演唱會訂票系統, 台鐵在節慶時的訂票系統
擴充的方式 : 金絲雀 ( 之後會學的 )…等

雲原生作業系統 - Kubernetes 大架構

這張圖中，有3台虛擬電腦

kubernetes master 的 hostname : m1
Worker node 的 hostname : w1 和 w2
- 可以視 Application 的複雜程度新增 Worker node 主機
kubernetes 最小的運作單位為 pod ( 豆莢 )
- pod 由一台或以上的 Container 組成
- 同一個 pod 裡的多台 Container 會共用同一張網路卡，網路卡對內透過記憶體溝通（localhost），對外有同一個 Pod 的 IP 位址。
- 一個 pod 裡面 3 台 Container 跑的 Application（舉例，Container 1 跑 nginx 的網站; Container 2 跑 MYSQL 資料庫; Container 3 跑 log Server ）= > 可以理解為 3 個 Container 跑企業用的 Application 在一個 pod 裡面。

Master node

Master node 裡一定會有 4 個 pod 分別是 API Server 、Secheduler 、Controller-Manager 和 etcd
API Server
- 負責接受前端管理者透過 CLI ( Command Line Interface, 命令模式 ) 下達的命令，例如 : kubectl 、kubeadm…等
- 也接受 Web UI ，在瀏覽器的網頁上點點選選，對 API Server 下達命令，例如 : Rancher
  
  Rancher 這家公司還有推出 K3s ( 無肥肉、身材好、精實型的 K8s )
- Authentication 身分認證，你是誰
- Authorization 授權，你有什麼權限，可以做什麼事情
Secheduler
- 負責安排 pod ( 企業的 application ) 要在哪一台工作主機上 run
- 安排的方式不只看記憶體，會有更複雜的演算法，溝通的對象為該工作主機的 kubelet
Controller-Manager
- 會負責照顧 node 裡面的 pod ，如果一台 node 裡面的 pod 掛了，它會去請 Scheduler 幫我們把 pod 建立回來，但是不一定會在原來的 node 把 pod 建立回來。 ( Kubernetes 內建的容錯功能 )
- 會監控 pod 的狀態，如果發現很多 connection 連到 pod ，它就會請 Sechedular，幫忙把 pod 複製出來，也就是隨需擴增。
etcd
- 一個資料庫。
- 存放整個 K8S 系統運作的重要資料，e.g., node 裡面會存/跑什麼 application, pod 叫什麼名字 …等。

Worker node

kubelet
- 在背景執行的程序
- 把 Container-Runtime-Engine (Podman, Docker, CRI-O) 做出來的一個或多個 Container 包成一個 pod
- 實際上 Podman, Docker, CRI-O 會透過 runc/crun (有人根據 OCI 組織制定的 Runtime spec 寫出來的程式) 來將 Container 建立出來。
- Docker 用的是 runc 來建立 Container，Podman 用的是 crun ，CRI-O 預設是用 runc
- runc/crun 在建立 Container 時，一定會需要 bundle ( runtime 設定檔 + rootfs 檔案系統目錄 )
Kube-proxy ( 哭ㄅㄆㄨㄚ seeˇ )
- 一個 pod
- 讓不同 node 中 pod 的網路能互相連接
- 在我們的環境中，kube-proxy 會透過 Flannel 網路套件，來讓 Pods 跨主機互通有無

m1 只要執行 kubelet ，其實也可以扮演 worker 的角色

情境敘述

我們 ( 前端管理者 ) 打 kubectl 這個命令告訴 API Server，我有一個 pod 要 run， API Server 接收到後，會去找 Scheduler，請他安排，Secheduler 會去找看哪一台 node 最閒，然後會去通知那台 node 裡面的 kubelet ，叫他去請 Container-Runtime-Engine 幫我們產生 Container 。
K8s 的 pod 在外面工作，我們稱它為 Solution ( 解決方案 ) ，注意 ! 它不能稱為一台電腦，因為 pod 是由一台或多台 Container 組成， Container 是一台電腦，pod 就是很多台電腦堆疊起來的一整個機櫃，舉例 : 應用系統由一個或多個 process 組成，而 pod 就是一個企業所需的應用系統，pod 裡面的 Container 就是 process 和 process 的相依檔。

Kubernetes CRI 的架構演進圖

最上面是第一代的流程，因為 K8S 不紅，所以自己寫了一個 Dockershim 來跟 Docker Engine 來溝通，建立 Container ，透過 Docker Engine 來做到內部網路，所以速度慢，效率差。
第二代：把 Docker 給砍掉了，並且 K8S 自己寫了 CRI-Cotainerd 來跟 Containerd 做溝通，不依賴 Docker Engine
第三代：Containerd 討好 K8S ，直接提供 CRI plugin ( Container Runtime Interface, 內建一個介面，根據 CRI 標準寫的程式 ) 給 K8S，讓 K8S 可以直接使用 Containerd，但因為 Containerd 本身還有給 Docker 平台使用，所以制定 image 的標準也有 Docker 的，跟下面最新的架構比，效率還是差了點。
目前：使用 CRI-O 這個專案，特別為 K8S 量身訂做，直接翻譯 kubelet 的話，效率最優。
- 而且直接對 runc 溝通，不用再經過好多流程才產生 container
- CRI-O 是 open source 的專案，老師目前使用最下面的架構，kubernetes 會直接透過 CRI-O 來跟 rnc 溝通產生 Container

結論
- Docker 並沒有遵循 CRI 的標準

選擇 Kubernetes Cluster

Cluster 叢集
kubernetes Cluster 就是 kubernetes 會用到的電腦，代表 K8S 是由多台電腦組成的作業系統

Managed Kubernetes Services

意思是有人已經幫你設好了 Kubernetes

雲端商的 Kubernetes (資料安全上會有問題)
- Amazon Elastic Kubernetes Service | Amazon 做出來的 K8S ，簡稱 : EKS/Amazon EKS
- Google Kubernetes Engine | Google 做出來的 K8S ，簡稱 : GKE
- Azure Kubernetes Service | Microsoft 做出來的 K8S ，簡稱 : AKS
- 服務供應商也會有像是 quay.io 的 image 管理中心，供使用者 Pull、Push image，還會有 CI/CD ，讓程式設計師有健身房可以用，達到敏捷開發。
- 不同雲端供應商都追加了不同的系統功能，但其核心都是 Kubernetes Cluster
K8S 商軟（distribution (“distro”) of Kubernetes）
- 由大公司幫我們做好 K8S Cluster ，我們只需要付錢錢，就可已買回來自己管理
- RedHat OpenShift | RedHat 做的 Kubernetes
- VMware Tanzu ｜ VMware 做的 Kubernetes
- Rancher | K8S (前端用的是 Web UI，價格較親民) + K3S（比 K8S 更輕量，可用於物聯網。）
- 還有很多 K8S 商軟 …等
Self-Hosting Kubernetes
- 自己建置 Kubernetes。

Kubernetes Cluster 核心運作

Open Interfaces in Kubernetes

CRI Container Runtime Interface ， Kubernetes 建置 Container 的標準
- 選用 CRI-O 的原因是因為，CRI-O 為了 K8S 而活，效能最好，支援多種 OCI Runtime , 不只有 runC/Crun ，還可以選用其他的，相容度很高
CNI Container Network Interface ，網路的標準
- 讓不同 node 中的 pod 網路可以互通有無，還有讓別人可以透過網路連接到我們建立對外的 Service （ Routing Table and NAT and Iptables ）
- 例如 : Flannel 網路套件
CSI Container Storage Interface ，儲存系統的標準，
- 其實講的就是 Volume ， bypass overlay2，讓 pod 的資料可以達到永存。e.g., Rook、ceph、NFS (Net Work Filesystem)
陳老師建置出來的 K8S CRI 用的是 CRI-O 這個專案，CNI 用的是 Flannel 網路套件，CSI 用的是 NFS 以上3種標準會衍生出很多不同的專案，但這些專案都是根據這3種標準做出來的 Kuberentes 可以根據不同標準做出不同架構的 Kubernetes

k8s 的 scale out ，自動擴充出分身的 pod ，硬體設備要夠強，不然會有災難。

Kubernetes Pods (CRI 的規矩)

pod 不會存在兩個 node 或以上，只會在一個 node 上
pods 的總記憶體用量為所在的 node 記憶體的總量乘上 80%，保留 20% 給 node 主機作業要使用。

Kubernetes Pods (CNI)

不同 pod 裡的 Container 之間的溝通無法使用 localhost
一個 pod 會有一個 IP 位址和一片網路卡，裡面的多個 Container 會共用 pod 的網路卡和 IP 位址
pod 對外有 IP 位址，對內多個 Container 是用 localhost 這個名稱來互通有無 (裡面是記憶體作業)

Kubernetes Pods (CSI)

上圖為 NFS 專案的架構，CSI 還有很多種專案
Container 使用的 overlay2 檔案系統，如果大量的新增、修改和刪除檔案，檔案系統會掛點，所以他不適合儲存大量的資料。
pod 可以透過 bypass 跳過 overlay2 檔案系統，使用 K8S 的 volume 來做儲存，這個 volume 會透過網路連到 NFS 。
NFS（共享資料夾）是一台機器(網路儲存設備，e.g., qnap)，把一堆資料夾，透過網路分享給不同的 pod 來使用，如果 NFS 使用的是 linux 系統，那麼 pod 就會用 ext4 檔案系統做儲存
pod 的 volume 可以連到我們的 big data 的平台 – > ==ROOK==，
- ROOK ，類似 Hadoop ，他的底層運作和 Hadoop 很像

Kubernetes Pods (Scheduling)

Scheduler 專門安排 pod 在哪一個 node run，靠的是 Labels (標籤)
我們的 pod 可以貼 Label ( 標籤 ) ，標籤儲存在資料庫裡面
Pod 可指定如何佈署到帶有特定 Labels 的 Node

Kubernetes Pods (High Availability)

pod 可以有很多本尊和分身，當他們同時在運作時，只要資料的儲存系統做的好（像是透過 ROOK ），多個 pod 可以並行運作，可以解決突然在短時間大量連線，造成系統當機的狀況。
結論：Pod 可輕易做到高可用性 ( HA )

Kubernetes Master 容錯

在 Kubernetes Master 裡面的 etcd 儲存整個 K8S 的 meta data ，只要 Master 這台主機掛了，整個 K8S 再見，所以要做容錯
多台 Master 主機的 etcd 會自動同步
根據 Raft 演算法可以算出如何做到 Master 主機的容錯。
- etcd: 計算故障容許節點數為 (N-1)/2，其中 N 為 K8S Master 數量, 而由 (N/2)+1 計算出 Majority, 只要滿足這數量, 代表 K8S 可以新增 Service, Deployment 等功能。
當一個 K8S Cluster 只存在 1 台或 2 台 Master 無法做到容錯。
3 台 Master 可以允許壞一台 Master 主機，如果再壞一台，會進入鎖死狀態，代表 Application 可以繼續 run ，但無法新增刪除修改資料。
如果要壞 2 台 Matser ，必須要有 5 台 Master 主機
如果只有 3 台實體主機，要如何做到 K8S 的容錯 ? 將 3 台實體主機都做成 Master 主機，透過以下命令

# 設定 K8S Master 可以執行 Pod
$ kubectl taint node m1 node-role.kubernetes.io/control-plane:NoSchedule-
output :
node/m1 untainted

讓 master 主機也能 run pod ，但如果要這樣做，一開始在規劃主機的硬體規格時，要盡量規劃強一點。

{%hackmd 6-wDy9uiQNKd6G23w78uog %} {%hackmd ISud-e3CSsO6Uw5UL8adSQ %} {%hackmd Yfx_m5QMQWezA08KN2_BPw %}

How to shutdown node in K8s

Mon, 28 Apr 2025 07:06:02 +0800

Graceful node shutdown

Kubernetes CPU throttling

Wed, 23 Apr 2025 17:43:36 +0800

你必須小心設定 K8s Workload 的 CPU limit，太小的值會為你的應用程式帶來額外的、無意義的延遲，太大的值會削弱叢集的整體穩定性。

1. 先備知識

Linux 上的 CPU 是如何運作的？

就像我們的大腦，CPU 每次只能處理一件事。然而，與人腦不同的是，CPU 能以極高速度在不同任務間切換，因此給人一種「同時處理很多事情」的錯覺。

在 Linux 作業系統中，CFS（Completely Fair Scheduler）就是負責控制哪些程式可以被排入 CPU 執行、哪些必須等待的調度機制。

CFS 的工作原理是：將 CPU 的處理時間分割成極短的「時間片」（time slice），並記錄每個程式實際佔用了多少時間。它會根據這些紀錄，公平地讓每個程式輪流使用 CPU，避免某些程式佔用過多、而其他程式被忽視。

為了幫助理解這個概念，想像三個小孩睡前搶著跟爸媽說話：爸媽一次只能聽一個孩子講話，這就像 CPU 只能一次處理一項任務。此時爸媽就像 CFS，會小心且公平地輪流聆聽每個孩子，讓大家都有說話的機會，還會記住誰講得多、誰講得少，防止某個孩子講太久而忽略其他人。

如果其中兩個小孩先睡著了，爸媽就能把所有注意力放在剩下清醒的那一位身上。同樣的，如果當前只有一個程式需要 CPU，那麼它就可以獨佔所有的處理資源。

核心觀念總結：

CPU 雖快，但實際上一次只能做一件事
CFS 是 Linux 核心中控制「哪一個程式可以用 CPU」的公平調度機制
它透過公平分配時間片，確保所有程式有機會執行

背景脈絡與核心概念

在伺服器或 containerized 環境中，單純用「公平分配」CPU 時間的方式（例如桌面系統所偏好的模式）已經不敷使用。原因是：

某些服務（如資料庫、監控或網頁服務）比其他服務更關鍵，需要優先取得資源。
containerized 環境常同時執行多個應用程式，若沒有資源隔離與優先級控管，會造成性能不可預期。

為了解決這個問題，Google 工程師在 2006 年開發了 Cgroups（Control Groups），並於 2008 年併入 Linux 主線核心，成為現今資源控制的關鍵基礎。

Cgroups 是什麼？

Cgroups 是 Linux 核心提供的一個機制，可用來限制、追蹤和隔離一個或多個程序的資源使用情況（如 CPU、記憶體、磁碟 I/O、網路等）。

Cgroups 的 CPU 模組允許你設定每個 container 的「share」值，CFS 在管理 CPU 時間時會考慮該值

單位通常為 milliCPU(或是 millicores)，1000m = 1 顆實體 CPU。
若某 container 設定為 200m，代表它在資源競爭情況下會被保證至少拿到 20% 的 CPU 時間。
如果沒有其他 container 需要 CPU，則可獲得高達 100% 的 CPU 時間。

2. K8s 如何套用 CPU 資源的限制?

當 kubelet 啟動 Pod 中的 container 時，它會將該 container 的記憶體與 CPU 的需求（requests）和限制（limits）資訊傳遞給 container runtime (Containerd/CRI-O…)。

在 Linux 系統中，container runtime 通常會設定在 kernel 中的 cgroups ，以應用並強制執行定義的資源限制。

以下引用來自 K8s 官網的定義與描述 :

The CPU limit defines a hard ceiling on how much CPU time the container can use. During each scheduling interval (time slice), the Linux kernel checks to see if this limit is exceeded; if so, the kernel waits before allowing that cgroup to resume execution.

container 設定 CPU limit 時，這個限制會被視為 container 可使用 CPU 時間的「硬上限」。在每個 time slice（排程週期）中，Linux 核心會檢查該 container 是否已超過其限制；如果超過，核心會暫停該 container，一直等待到下一個排程週期開始，重新分配 CPU 時間後，container 才能去執行。

Kubelet 還會使用 Linux Kernel CFS（Completely Fair Scheduler，完全公平調度）來限制 Workload 的 CPU time 的使用率，要點如下：

CPU 使用量的計量週期為 100ms
CPU limit 決定每計量週期（100ms）內 container 可以使用的 CPU 時間的上限
本週期若 container 的 CPU 時間用量達到上限，CPU throttling (限流) 開始，container 只能在下個週期繼續執行
1 CPU = 100 ms CPU 時間每計量週期，以此類推，0.2 CPU = 20 ms CPU 時間每計量週期，2.5 CPU = 250 ms CPU 時間每計量週期
如果程式用了多個核，CPU 時間會累積統計。

cgroups 提供了資源限制與隔離的框架，而 CFS 則在此框架下，實施具體的 CPU 時間分配與調度。

CKS

Wed, 09 Apr 2025 13:34:16 +0800

Preface

本篇文章為 2024 更新後的 CKS 真題，包含題目模擬與題目解答

可以透過點擊以下目錄，選擇想看的內容，跳轉至特定章節

Question #1: kube-bench 修復不安全項目

Context:

You must resolve issues that a CIS Benchmark tool found for the kubeadm provisioned cluster.

Task

Fix all issues via configuration and restart the affected components to ensure the new settings take effect.

Fix all of the following violations that were found against the kubelet:

2.1.2 Ensure that the anonymous-auth argument is set to false (FAIL)
2.1.3 Ensure that the --authorization-mode argument is not set to AlwaysAllow (FAIL)

Use Webhook authentication/authorization where possible.

Fix all of the following violations that were found against etcd:

2.2 Ensure that the --client-cert-auth argument is set to true (FAIL)

Prepare Env

# 1. 安裝必要套件
$ apt update; apt install -y nano \
yq \
dnsutils
# 2. 安裝 kube-bench
$ curl -sL $(curl -sL https://api.github.com/repos/aquasecurity/kube-bench/releases/latest | jq -r .assets[].browser_download_url | grep 'linux_amd64.deb') -o kube-bench.deb
$ dpkg -i kube-bench.deb
$ rm -r kube-bench.deb
# 3. etcd
$ cp /etc/kubernetes/manifests/etcd.yaml .
$ sed -i 's|--client-cert-auth=true|--client-cert-auth=false|g' /etc/kubernetes/manifests/etcd.yaml
# 4. kubelet
$ cp /var/lib/kubelet/config.yaml .
$ yq -iy '.authentication.anonymous.enabled = true' /var/lib/kubelet/config.yaml
$ yq -iy '.authentication.webhook.enabled = false' /var/lib/kubelet/config.yaml
$ yq -iy '.authorization.mode = "AlwaysAllow"' /var/lib/kubelet/config.yaml
$ systemctl daemon-reload
$ systemctl restart kubelet.service

Answer

# 確認 kubelet 的參數需要修改
$ kube-bench run -s node
...
[FAIL] 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[FAIL] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
...
# 備份 kubelet 設定檔
$ cp /var/lib/kubelet/config.yaml ./kubelet-config.yaml.bk
# 修改 kubelet 設定檔
$ nano /var/lib/kubelet/config.yaml
...
authentication:
anonymous:
enabled: false <--- 這裡改 false
webhook:
cacheTTL: 0s
enabled: true <--- 這裡改 true
...
authorization:
mode: Webhook <--- 這裡改 Webhook
webhook:
cacheAuthorizedTTL: 0s
cacheUnauthorizedTTL: 0s
...
# 重新加载 systemd 的設定文件，使得 systemd 能够讀取最新的設定資訊。
$ systemctl daemon-reload
# 重啟 kubelet 服務
$ systemctl restart kubelet
# 確認 ETCD 的參數需要修改
$ kube-bench run -s etcd
...
[FAIL] 2.2 Ensure that the --client-cert-auth argument is set to true (Automated)
...
# 備份 etcd 設定檔
$ cp /etc/kubernetes/manifests/etcd.yaml ./etcd.yaml.bk
# 修改 etcd 設定檔
$ nano /etc/kubernetes/manifests/etcd.yaml
...
spec:
containers:
- command:
- etcd
...
- --client-cert-auth=true <--- 這裡改 true
# 檢查 etcd container 的狀態，確認建立時間是新的，且狀態是 Running
$ crictl ps --name etcd
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
ce981115becf6 3861cfcd7c04c 2 minutes ago Running etcd 0 747918759d292 etcd-c30-control-plane
# 檢查 ETCD Pods 狀態
$ kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
etcd-c30-control-plane 1/1 Running 0 2m17s
...
# 檢查 kubelet 是否修改正確
$ kube-bench run -s node
[PASS] 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
...
# 檢查 ETCD 是否修改正確
$ kube-bench run -s etcd
[PASS] 2.2 Ensure that the --client-cert-auth argument is set to true (Automated)
...

Question #2: 啟用 API server 認證

Context

For testing purposes, the kubeadm provisioned cluster’s API server was configured to allow unauthenticated and unauthorized access.

Task

First, secure the cluster’s API server by configuring it as follows:

Forbid anonymous authentication
Use authorization mode Node,RBAC
Use admission controller NodeRestriction

kubectl is configured to use unauthenticated and unauthorized access. You do not have to change it, but be aware that kubectl will stop working once you have secured the cluster.

You can use the cluster’s original kubectl configuration file located at /etc/kubernetes/admin.conf to access the secured cluster.

Next, to clean up, remove the ClusterRoleBinding system:anonymous.

Prepare Env

# 建立 anonymous ClusterRoleBinding
$ echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:anonymous
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:anonymous' | kubectl apply -f -
# 修改 kube-apiserver
$ cp /etc/kubernetes/manifests/kube-apiserver.yaml .
$ sed -i '/- --enable-admission-plugins=NodeRestriction/d' /etc/kubernetes/manifests/kube-apiserver.yaml
$ sed -i 's|--authorization-mode=Node,RBAC|--authorization-mode=AlwaysAllow\n - --anonymous-auth=true|g' /etc/kubernetes/manifests/kube-apiserver.yaml

Answer

官網文件收尋關鍵字 : kube-apiserver 找到這篇文章 : kube-apiserver | Kubernetes 按 Ctrl + F 搜尋 : -admission

# 備份 kube-apiserver 設定檔
$ cp /etc/kubernetes/manifests/kube-apiserver.yaml ./kube-apiserver.yaml.bk
# 修改 kube-apiserver 設定檔
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
containers:
- command:
...
- --authorization-mode=Node,RBAC <-- 這裡改 Node,RBAC (逗號後面不能空格)
- --anonymous-auth=false <-- 這裡改 false (實際考試這個參數會藏在所有參數的最底下)
- --enable-admission-plugins=NodeRestriction <-- 到官網文件複製此參數，並新增此行設定 (如果題目原本就有此參數，但值不是題目要求的，就直接刪除該值，換成跟題目一模一樣的 NodeRestriction)
# 重啟 kubelet service
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet.service
# 修改後，確認 kube-apiserver Container 的狀態
$ crictl ps --name kube-apiserver
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
a6b389f7f3c6f 7f6c51674d5ef About a minute ago Running kube-apiserver 0 f09b8f9c8a33f kube-apiserver-c30-control-plane
# 確認 kube-apiserver Pods 的狀態
$ kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system get pods
NAME READY STATUS RESTARTS AGE
kube-apiserver-c30-control-plane 1/1 Running 0 3m11s
# 刪除 system:anonymous ClusterRoleBinding
$ kubectl --kubeconfig=/etc/kubernetes/admin.conf delete clusterrolebinding system:anonymous
clusterrolebinding.rbac.authorization.k8s.io "system:anonymous" deleted

Question #3: ImagePolicyWebhook Container images 掃描

Context

You must fully integrate a container image scanner into the kubeadm provisioned cluster.

Task

Given an incomplete configuration located at /etc/kubernetes/bouncer and a functional container image scanner with an HTTPS endpoint at https://smooth-yak.local/image_policy, perform the following tasks to implement a validating admission controller:

First, re-configure the API server to enable all admission plugin(s) to support the provided AdmissionConfiguration.

Next, re-configure the ImagePolicyWebhook configuration to deny images on backend failure.

Next, Complete the backend configuration to point to the container image scanner’s endpoint at https://smooth-yak.local/image_policy.

Finally, to test the configuration, deploy the test resource defined in ~/vulnerable.yaml, which is using an image that should be denied.

You may delete and re-create the resource as often as needed.

The container image scanner’s log file is located at /var/log/nginx/access_log.

Prepare ENV

# Create a Service
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
creationTimestamp: null
labels:
app: imagepolicywebhook
name: image-bouncer-webhook
spec:
ports:
- name: "443"
port: 443
protocol: TCP
targetPort: 443
selector:
app: imagepolicywebhook
type: ClusterIP
EOF
# We create an environment variable for the IP to use it later.
$ export SERVICE_IP=$(kubectl get svc -o jsonpath='{.items[0].spec.clusterIP}')
# Create Server Certificates
$ mkdir ~/ssl; cd ~/ssl
$ openssl genrsa -out webhook-server.key 2048
$ openssl req -new -key webhook-server.key -subj "/CN=system:node:imagepolicywebhook/O=system:nodes" -addext "subjectAltName = DNS:image-bouncer-webhook.default.svc.cluster.local,DNS:image-bouncer-webhook.default.svc,DNS:image-bouncer-webhook.default.pod.cluster.local,IP:$SERVICE_IP" -out webhook-server.csr
# To use the csr in a Kubernetes manifest, we need to change the encoding to base64 and export it as an environment variable.
$ export SIGNING_REQUEST=$(cat webhook-server.csr | base64 | tr -d "\n")
# We then send a CertificateSigningRequest manifest to the kube-apiserver to sign our certificate. The certificate usages in this manifest are the usages that the signer accept to sign.
$ cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: webhook-server
spec:
request: $SIGNING_REQUEST
signerName: kubernetes.io/kubelet-serving
expirationSeconds: 864000 # ten days
usages:
- digital signature
- key encipherment
- server auth
EOF
# This signing request is now ready to be approved. After approving it, we can write the signed certificate to a file.
$ kubectl get csr
NAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITION
csr-4jpkv 3d kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:abcdef <none> Approved,Issued
webhook-server 56s kubernetes.io/kubelet-serving kubernetes-admin 10d Pending
$ kubectl certificate approve webhook-server
certificatesigningrequest.certificates.k8s.io/webhook-server approved
$ kubectl get csr webhook-server -o=jsonpath={.status.certificate} | base64 --decode > webhook-server.crt
# Create a secret with the certificates for the server
$ kubectl create secret tls webhook-server --cert=webhook-server.crt --key=webhook-server.key
secret/webhook-server created
# Deploy the imagePolicyWebhook server
$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: imagepolicywebhook
name: imagepolicywebhook
spec:
replicas: 1
selector:
matchLabels:
app: imagepolicywebhook
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: imagepolicywebhook
spec:
volumes:
- name: cert
secret:
secretName: webhook-server
items:
- key: tls.crt
path: webhook-server.crt
- name: key
secret:
secretName: webhook-server
items:
- key: tls.key
path: webhook-server.key
containers:
- image: stephang/imagepolicywebhook:latest
name: imagepolicywebhook
ports:
- containerPort: 443
volumeMounts:
- name: cert
readOnly: true
mountPath: /etc/ssl/certs/
- name: key
readOnly: true
mountPath: /etc/ssl/private/
EOF
# Prepare the kube-apiserver configuration
## To tell the kube-apiserver to use our webhook, it needs the server certificate, an admissionConfiguration and a kubeconfig to connect to the server. We provide the needed Data in the /etc/kubernetes/epconfig webhook directory.
$ mkdir -p /etc/kubernetes/admission-control && \
cp webhook-server.crt /etc/kubernetes/admission-control
# The AdmissionConfiguration points to a kubeConfigfile that will authenticate the webhook. This configuration will allow every image in our cluster when the webhook is not reachable.
$ cat <<EOF >/etc/kubernetes/admission-control/imagepolicy.conf
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ImagePolicyWebhook
configuration:
imagePolicy:
kubeConfigFile: /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
allowTTL: 50
denyTTL: 50
retryBackoff: 500
defaultAllow: true
EOF
# The kube-apiserver uses only external DNS server and will not find the internal DNS service adresses. To let the apiserver use the internal service, the server in the kubeconfig must be set to the IP of the service. For standard admission controller, we could use a special notation to encode the dns name in the kubeconfig. As the client certificate, we use the existing apiserver certificate.
$ cat <<EOF >/etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /etc/kubernetes/admission-control/webhook-server.crt
server: https://$SERVICE_IP
name: webhook
contexts:
- context:
cluster: webhook
user: image-bouncer-webhook.default.svc
name: webhook
current-context: webhook
kind: Config
users:
- name: image-bouncer-webhook.default.svc
user:
client-certificate: /etc/kubernetes/pki/apiserver.crt
client-key: /etc/kubernetes/pki/apiserver.key
EOF
# Prepare the kube-apiserver
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
## 新增以下內容
metadata:
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
dnsPolicy: ClusterFirstWithHostNet
# 準備測試 YAML 檔
$ cat <<EOF > /root/vulns-pod.yml
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: alpine
name: alpine
spec:
containers:
- args:
- sleep
- 1d
image: alpine
name: alpine
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
EOF

Answer

官網文件收尋關鍵字 : ImagePolicyWebhook 找到這篇文章 : Admission Controllers Reference | Kubernetes 章節 : ImagePolicyWebhook

# 修改 AdmissionConfiguration 設定檔
$ nano /etc/kubernetes/admission-control/imagepolicy
...
plugins:
- name: ImagePolicyWebhook
configuration:
imagePolicy:
...
defaultAllow: false <- 這裡改成 false
# 編輯 werbhook 要用的 kubeconfig，把 server 改成正確的位置
$ nano /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
clusters:
- cluster:
certificate-authority: /etc/kubernetes/epconfig/webhook-server.crt
server: https://image-bouncer-webhook.default.svc <- 這裡改成題目要的
# 啟用 imagepolicywebhook
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
## 新增以下內容
containers:
- command:
- --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
- --admission-control-config-file=/etc/kubernetes/admission-control/imagepolicy
...
volumeMounts:
- mountPath: /etc/kubernetes/admission-control
name: webhook
readOnly: true
...
- hostPath:
path: /etc/kubernetes/admission-control
type: DirectoryOrCreate
name: webhook
# 確認 API Server 正常
$ crictl ps
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-apiserver-c30-control-plane 1/1 Running 0 84s
$ kubectl -n kube-system get pods
kube-apiserver-c30-control-plane 1/1 Running 0 91s
# 測試建立 Pod 看 ImagePolicyWebhook 是否正常工作
$ kubectl apply -f /root/vulns-pod.yml
Error from server (Forbidden): error when creating "/root/vulns-pod.yml": pods "alpine" is forbidden: image policy webhook backend denied one or more images: Only nginx images are allowed
## 考試的時候要用 kubectl get event 去找錯誤

Question #4: Dockerfile 和 yaml 檔案安全性偵測

Task

Check the Dockerfile located at /root/Dockerfile on the CLI server. This Dockerfile is based upon alpine:3.13.5.
Correct the two security issues within the file.

Note: If you need an OS user, you can use the user nobody with id 65535.

Check the YAML file located at /root/scooby-gang-deploy.yml on the CLI server.
Correct the two security issues within the file.

Prepare ENV

$ cat <<'EOF' > /root/Dockerfile
FROM alpine:latest
COPY sunnydale.sh .
USER root
CMD ["./sunnydale.sh"]
EOF
$ cat <<'EOF' > /root/scooby-gang-deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
securityContext:
{'capabilities': {'add': ['NET_BIND_SERVICE'], 'drop': ['all']}, 'privileged': true, 'readOnlyRootFilesystem': false, 'runAsUser': 0}
EOF

Answer

# 修改 Dockerfile
$ nano /root/Dockerfile
FROM ubuntu:22.04
ENTRYPOINT ["/sunnydale.sh"]
USER nobody
CMD ["./sunnydale.sh"]
# 修改 Deployment
$ nano /root/scooby-gang-deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
securityContext:
# privileged 設成 False
# readOnlyRootFilesystem 設成 True
# runAsUser 設成 65535
{'capabilities': {'add': ['NET_BIND_SERVICE'], 'drop': ['all']}, 'privileged': false, 'readOnlyRootFilesystem': true, 'runAsUser': 65535}

Question #5: log audit

Context

You must implement auditing for the kubeadm provisioned cluster.

Task

First, reconfigure the cluster’s API server, so that:

The basic audit policy located at /etc/kubernetes/logpolicy/audit-policy.yaml is used.
Logs are stored at /var/log/kubernetes/audit-logs.txt.
A maximum of 2 logs are retained for 10 days.

The basic policy only specifies what not to log.

Next, edit and extend the basic policy to log:

namespaces interactions at RequestResponse level.
The request body of deployments interactions in the namespace webapps.
ConfigMap and Secret interactions in all namespaces at the Metadata level.
All other requests at the Metadata level.

Make sure the API server uses the extended policy. Failure to do so may result in a reduced score.

Prepare ENV

$ sudo mkdir -p /etc/kubernetes/logpolicy
$ cat <<EOF > /etc/kubernetes/logpolicy/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core API group
resources: ["endpoints", "services"]
- level: None
userGroups: ["system:authenticated"]
nonResourceURLs:
- "/api*" # Wildcard matching.
- "/version"
EOF

Answer

官網文件收尋關鍵字 : audit 找到這篇文章 : Auditing | Kubernetes 章節 : Audit policy 和 Log backend

# 確認 deployment 的 API Group
$ kubectl api-resources | grep deploy
deployments deploy apps/v1 true Deployment
# 備份 audit-policy.yaml
$ cp /etc/kubernetes/logpolicy/audit-policy.yaml ./audit-policy.yaml.bk
# 修改 audit-policy.yaml
## 不要刪除原有規則, 可以在下面繼續追加題目要求的規則
$ nano /etc/kubernetes/logpolicy/audit-policy.yaml
…
- level: RequestResponse
resources:
- group: ""
resources: ["namespaces"]
- level: Request
resources:
- group: "apps"
resources: ["deployments"]
namespaces: ["webapps"]
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps"]
- level: Metadata
# 啟用 Audit Log
# 備份 kube-apiserver 的 YAML 檔
$ cp /etc/kubernetes/manifests/kube-apiserver.yaml ./kube-apiserver.yaml
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
containers:
- command:
...
## 新增以下內容 (實際考試的時候，一樣要仔細確認每個參數在實際環境是否已經存在，如果存在就把值改得跟題目要求一樣，如果沒有參數再手動加，volumeMounts 和 volumes 這兩個項目的內容，在實際考試環境可能已經存在)
- --audit-policy-file=/etc/kubernetes/logpolicy/audit-policy.yaml
- --audit-log-path=/var/log/kubernetes/audit-logs.txt
- --audit-log-maxage=10
- --audit-log-maxbackup=2
volumeMounts:
- mountPath: /etc/kubernetes/logpolicy/audit-policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/
name: audit-log
readOnly: false
volumes:
- name: audit
hostPath:
path: /etc/kubernetes/logpolicy/audit-policy.yaml
type: File
- name: audit-log
hostPath:
path: /var/log/kubernetes/
type: DirectoryOrCreate
# 重啟 kubelet service
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet
## 等待 kube-apiserver 自動重起，且恢復正常
$ crictl ps
$ kubectl get nodes
## 檢查 Audit log 已啟用，看到 audit log 一直噴就代表這題成功!
$ tail -f /var/log/kubernetes/audit-logs.txt
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"0969e8fc-8dbd-4cfa-b1bd-44ed9d915344","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/events","verb":"create","user":{"username":"system:node:c30-worker","groups":["system:nodes","system:authenticated"]},"sourceIPs":["10.89.0.10"],"userAgent":"kubelet/v1.30.0 (linux/amd64) kubernetes/7c48c2b","objectRef":{"resource":"events","namespace":"default","name":"nginx-bf5d5cf98-gk9br.180afb34a972b0aa","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-11-24T18:33:30.277486Z","stageTimestamp":"2024-11-24T18:33:30.279054Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
...

Question #6: Container Security Context

Context

Container Security Context 應在特定 namespace 中修改 Deployment。

Task

依照下列要求修改 sec-ns namespace 裡的 Deployment secdep

使用 ID 為 30000 的使用者啟動 Container（設定使用者 ID 為: 30000）
不允許程序獲得超出其父程序的特權（禁止 allowPrivilegeEscalation）
以 readOnly 方式載入 Container 的根檔案系統（對根檔案的唯讀權限）

Prepare ENV

$ kubectl create ns sec-ns
$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: secdep
labels:
app: nginx
namespace: sec-ns
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
containers:
- name: sec-ctx-demo-1
image: registry.k8s.io/pause:3.9
ports:
- name: sec-ctx-demo-2
image: registry.k8s.io/pause:3.9
ports:
EOF

Answer

官網文件收尋關鍵字 : Security Context 找到這篇文章 : Configure a Security Context for a Pod or Container | Kubernetes 章節 : Set the security context for a Pod 和 Log backend

# 備份 Deployment 的 yaml 檔
$ kubectl -n sec-ns get deployment secdep -o yaml > secdep.yaml
# 修改 Deployment
$ kubectl -n sec-ns edit deployment secdep
spec:
template:
spec:
containers:
- image: registry.k8s.io/pause:3.9
imagePullPolicy: IfNotPresent
name: sec-ctx-demo-1
resources: {}
securityContext: ## 新增此行
allowPrivilegeEscalation: false ## 新增此行
readOnlyRootFilesystem: true ## 新增此行
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
- image: registry.k8s.io/pause:3.9
imagePullPolicy: IfNotPresent
name: sec-ctx-demo-2
resources: {}
securityContext: ## 新增此行
allowPrivilegeEscalation: false ## 新增此行
readOnlyRootFilesystem: true ## 新增此行
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: ## 新增此行
runAsUser: 30000 ## 新增此行
# 檢查修改後 pod 是否正常
$ kubectl -n sec-ns get pods
NAME READY STATUS RESTARTS AGE
secdep-56c5959674-n5wb5 2/2 Running 0 2m13s

Question #7: Networkpolicy

Task

A default deny network policy is a policy that blocks all traffic in a namespace by default.

Create a network policy called default-deny that blocks all incoming traffic to all Pods in the mordor namespace.

Create a network policy called mtdoom-np that allows specific traffic on port 80 to reach the mtdoom Pod in the mordor namespace.

The policy should allow incoming traffic:

From all Pods in the frodo namespace.
From all Pods with the label app=sam, regardless of which namespace they are in.

考試時有測試的 pod 可以用 ping 命令

Prepare ENV

$ kubectl create ns mordor
$ kubectl create ns frodo --dry-run=client -o yaml | kubectl label --local -f - app=frodo --dry-run=client -o yaml | kubectl apply -f -
$ kubectl -n mordor run mtdoom --image=nginx
$ kubectl -n frodo run haha --image=nginx
$ kubectl run hello --image=nginx -l app=sam

Answer

$ nano default-denyall.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: mordor
spec:
podSelector: {}
policyTypes:
- Ingress
$ kubectl apply -f default-denyall.yaml
$ kubectl -n mordor get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
mtdoom 1/1 Running 0 13m run=mtdoom
$ kubectl get ns frodo --show-labels
NAME STATUS AGE LABELS
frodo Active 19m app=frodo,kubernetes.io/metadata.name=frodo
$ nano ingress-netpol.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: mtdoom-np
namespace: mordor
spec:
podSelector:
matchLabels:
run: mtdoom
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
app: frodo
- podSelector:
matchLabels:
app: sam
ports:
- protocol: TCP
port: 80
$ kubectl apply -f ingress-netpol.yaml
$ ip=$(kubectl -n mordor get pod mtdoom --template={{.status.podIP}})
$ kubectl -n frodo exec -it haha -- curl -s http://${ip} | grep nginx
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
<a href="http://nginx.org/">nginx.org</a>.<br/>
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
$ kubectl exec -it hello -- curl -s http://${ip} | grep nginx
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
<a href="http://nginx.org/">nginx.org</a>.<br/>
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
$ kubectl run qqq --image=nginx
$ kubectl exec qqq -- curl -m 1 -s http://${ip} | grep nginx
command terminated with exit code 28

Question #8: Ingress TLS

Context

您必須使用儲存在 TLS Secret 中的 SSL 文件，來保護 Web Server 的安全存取。

Task

在 clever-cactus namespace中為名為 clever-cactus 的現有Deployment 建立名為 clever-cactus 的 TLS Secret 。

使用以下 SSL 檔案：

證書: /ca-cert/web.k8s.local.crt
密鑰: /ca-cert/web.k8s.local.key

Deployment 已設定為使用 TLS Secret。請勿修改現有的 Deployment。

service、deployment 和 secret 都好了，只要建有 TLS 的 Ingress，並且要啟用 ssl-redirect。

Answer

第一篇官網文章 :

官網文件收尋關鍵字 : ssl-redirect
找到這篇文章 : NGINX Ingress Controller + front F5 Load Balancer - General ...
不要點進去文章!!!

第二篇官網文章 :

官網文件收尋關鍵字 : ingress 找到這篇文章 : Ingress | Kubernetes 章節 : TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tls-example-ingress
spec:
tls:
- hosts:
- https-example.foo.com
secretName: testsecret-tls
rules:
- host: https-example.foo.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: service1
port:
number: 80

Question #9: ServiceAccount

Task

要關自動產生的 token 檔案，但是 pod 要把 token mount 回來，並且 volumemount 要設 readOnly

Prepare ENV

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: build-robot
EOF

Answer

官網文件收尋關鍵字 : serviceaccount 找到這篇文章 : Configure Service Accounts for Pods | Kubernetes 章節 : Opt out of API credential automounting 和 Launch a Pod using service account token projection

# 設定 serviceaccount 取消 API 憑證自動掛載
$ kubectl edit sa build-robot
apiVersion: v1
automountServiceAccountToken: false ## 新增此行
kind: ServiceAccount
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"build-robot","namespace":"default"}}
creationTimestamp: "2024-11-24T19:35:28Z"
name: build-robot
namespace: default
resourceVersion: "16129"
uid: 1e4d3c5a-8cb8-4612-a690-5d0960f2034e
# 設定 pod
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: nginx
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
containers:
- image: nginx
name: nginx
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: vault-token
readOnly: true
serviceAccountName: build-robot
volumes:
- name: vault-token
projected:
sources:
- serviceAccountToken:
path: vault-token
expirationSeconds: 7200
audience: vault
EOF
$ kubectl exec nginx -- ls -l /var/run/secrets/tokens
total 0
lrwxrwxrwx 1 root root 18 Nov 24 19:43 vault-token -> ..data/vault-token

Question #10: Upgrade Worker node

Context

The kubeadm provisioned cluster was recently upgraded, leaving one node on a slightly older version due to workload compatibility concerns.

Task

Upgrade the cluster node compute-0 to match the version of the control plane node. Use a command like the following to connect to the compute node:

[candidate@cks000034] $ ssh compute-0

Prepare ENV

$ apt update
$ apt install -y gpg-agent software-properties-common
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list
$ kubelet --version
Kubernetes v1.31.2
[註] 把 kubeadm kubelet kubectl 套件安裝回來
$ apt update
$ apt install -y --allow-change-held-packages kubeadm=1.31.2-1.1 kubelet=1.31.2-1.1
*** kubelet (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/default/kubelet ...
$ apt-mark hold kubeadm kubelet

Answer

官網文件收尋關鍵字 : upgrade linux 找到這篇文章 : Upgrading Linux nodes | Kubernetes 章節 : Upgrading worker nodes

# 確認 cluster 狀態
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
c30-up-control-plane Ready control-plane 4m29s v1.30.3
c30-up-worker Ready <none> 4m10s v1.30.2
# Drain the worker node
$ kubectl drain c30-up-worker --ignore-daemonsets
# 連線到 worker node
$ ssh compute-0
# 確認要升級的套件名稱
$ apt-cache madison kubeadm | grep '1.31'
kubeadm | 1.31.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb Packages
kubeadm | 1.31.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb Packages
kubeadm | 1.31.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb Packages
kubeadm | 1.31.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb Packages
# Upgrade kubeadm
$ apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm='1.30.3-1.1' && \
apt-mark hold kubeadm
# Call "kubeadm upgrade"
$ kubeadm upgrade node
# Upgrade kubelet
$ apt-mark unhold kubelet && \
apt-get update && apt-get install -y kubelet='1.30.3-1.1' && \
apt-mark hold kubelet
# Restart the kubelet
$ systemctl daemon-reload && \
systemctl restart kubelet
# Uncordon the node
$ kubectl uncordon c30-up-worker
# Check node status
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
c30-up-control-plane Ready control-plane 32m v1.30.3
c30-up-worker Ready <none> 32m v1.30.3

Question #11: BOM

Task

有一個 pod 裡面有 3 個 Container，要找出特定版本的特定套件在哪一個 Container images，並透過 bom 命令，將 container images 的套件相依的清單資訊倒到一個檔案。

Prepare ENV

$ curl -sL $(curl -sL https://api.github.com/repos/kubernetes-sigs/bom/releases/latest | jq -r .assets[].browser_download_url | grep 'bom-amd64-linux$') -o bom
install -o root -g root -m 0755 bom /usr/local/bin/bom
rm -r bom*
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: alpine
name: alpine
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
containers:
- args:
- sleep
- 1d
image: alpine
name: alpine
- args:
- sleep
- 1d
image: busybox:1.28
name: busybox
resources: {}
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
dnsPolicy: ClusterFirst
restartPolicy: Always
EOF

Answer

## 先得到 image 的名稱
$ kubectl get pods alpine -o yaml | grep image: | tail -n 3
image: docker.io/library/alpine:latest
image: docker.io/library/busybox:1.28
image: docker.io/library/nginx:1.14.2
$ bom generate --scan-images=false --image=docker.io/library/alpine:latest -o /root/alp.spdx
$ bom document outline --purl /root/alp.spdx

Question #12: Docker Daemon Troubleshooting

Task

要把 docker 群組裡面的其中一個 user 移除，sock 的權限是錯的，另一個送他(把題目背出來)

Answer

$ gpasswd -d developer docker
$ vim /usr/lib/systemd/system/docker.socket
## 將 SocketGroup 欄位設為 root
$ 修改docker.service，將[Service]下面ExecStart裡面的 -H tcp://0.0.0.0:2375 刪掉

Question #13: Cilium NetworkPolicy、MTLS

Context

使用 Cilium 執行以下任務，以保護現有應用程式的內部和外部網路流量。

Task

您可以使用瀏覽器存取 Cilium 的文件。

首先，在 nodebb namespace 中建立一個名為 nodebb 的 L4 CiliumNetworkPolicy，並以以下方式設定它：

允許在 ingress-nginx namespace 中運行的所有 Pod 存取 nodebb Deployment 的 Pod
要求相互驗證

然後，將前一步建立的網路策略擴展如下：

允許主機存取 nodebb Deployment 的 Pod
不要使用相互身份驗證

Answer

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: nodebb
namespace: nodebb
spec:
endpointSelector:
matchLabels:
app: nodebb
ingress:
- fromEndpoints:
- matchLabels:
kubernetes.io/metadata.name: ingress-nginx
authentication:
mode: "required"
fromEntities:
- "host"

Question #14: Pod security standard

Task

題目會給你一個 YAML 檔去 apply，但會有很多錯誤訊息，慢慢改

Prepare ENV

$ kubectl create ns mytest
$ kubectl patch namespace mytest -p '{"metadata": {"labels": {"pod-security.kubernetes.io/enforce": "restricted"}}}'
$ kubectl get ns mytest --show-labels
NAME STATUS AGE LABELS
mytest Active 10s kubernetes.io/metadata.name=mytest,pod-security.kubernetes.io/enforce=restricted
$ mkdir -p /cks/pss/ && echo 'apiVersion: v1
kind: Pod
metadata:
name: restricted-pod
namespace: mytest
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
containers:
- name: restricted-container
image: registry.k8s.io/pause:3.9
securityContext:
privileged: true' > /cks/pss/pod.yaml

Answer

# 查看具體遇到那些錯誤
$ kubectl apply -f /cks/pss/pod.yaml
Error from server (Forbidden): error when creating "/cks/pss/pod.yaml": pods "baseline-pod" is forbidden: violates PodSecurity "restricted:latest": privileged (container "baseline-container" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "baseline-container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "baseline-container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "baseline-container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "baseline-container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
# 修改 YAML 檔
$ nano /cks/pss/pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: restricted-pod
namespace: mytest
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
containers:
- name: restricted-container
image: registry.k8s.io/pause:3.9
securityContext:
### 新增以下設定
privileged: false
allowPrivilegeEscalation: false
capabilities:
drop:
- "ALL"
runAsNonRoot: true
seccompProfile:
type: "RuntimeDefault"
# 部署修改好後的 Pod YAML
$ kubectl apply -f /cks/pss/pod.yaml
# 檢查可成功部署
$ kubectl get -f /cks/pss/pod.yaml
NAME READY STATUS RESTARTS AGE
restricted-pod 1/1 Running 0 4s

Question #15: TLS Secrets 創建與掛載

Context

您必須使用儲存在 TLS Secret 中的 SSL 文件，來保護 Web Server 的安全存取。

Task

在 clever-cactus namespace中為名為 clever-cactus 的現有Deployment 建立名為 clever-cactus 的 TLS Secret 。

使用以下 SSL 檔案：

證書: /root/nginx/nginx.crt
密鑰: /root/nginx/nginx.key

Deployment 已設定為使用 TLS Secret。請勿修改現有的 Deployment。

Prepare ENV

$ kubectl create ns clever-cactus
$ mkdir -p nginx
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ./nginx/nginx.key -out ./nginx/nginx.crt
$ cat <<EOF > default.conf
server {
listen 443 ssl;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
server_name localhost;
ssl_certificate /etc/nginx/ssl/tls.crt;
ssl_certificate_key /etc/nginx/ssl/tls.key;
}
EOF
$ kubectl -n clever-cactus create configmap nginx-conf --from-file=default.conf
$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: clever-cactus
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
volumes:
- name: tls-volume
secret:
secretName: clever-cactus
- name: config-volume
configMap:
name: nginx-conf
containers:
- name: nginx
image: nginx
ports:
- containerPort: 443
volumeMounts:
- mountPath: /etc/nginx/ssl
name: tls-volume
- mountPath: /etc/nginx/conf.d
name: config-volume
EOF

Answer

官網文件收尋關鍵字 : secret 找到這篇文章 : Secrets | Kubernetes 章節 : TLS Secrets

$ kubectl -n clever-cactus get pods
NAME READY STATUS RESTARTS AGE
nginx-b9b7995fc-hfhm4 0/1 ContainerCreating 0 90s
$ kubectl -n clever-cactus describe pod nginx-b9b7995fc-hfhm4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 46s default-scheduler Successfully assigned clever-cactus/nginx-b9b7995fc-hfhm4 to c30-worker
Warning FailedMount 14s (x7 over 46s) kubelet MountVolume.SetUp failed for volume "clever-cactus" : secret "nginx-tls" not found
$ kubectl create secret tls clever-cactus -n clever-cactus \
--cert=/root/nginx/nginx.crt \
--key=/root/nginx/nginx.key
$ kubectl -n clever-cactus get pods
NAME READY STATUS RESTARTS AGE
nginx-5b4dcf49bd-ksl8z 1/1 Running 0 17s

Question #16: 考 Falco 送他

REF

KCNA 真題

Tue, 08 Apr 2025 17:08:58 +0800

KCNA 的考題最後更新於 2024 年 11 月 16 日。

Question #1

What native runtime is Open Container Initiative (OCI) compliant?

A. runc B. runV C. kata-containers D. gvisor

Here’s why:

runc is a CLI tool for spawning and running containers on Linux according to the OCI specification.

REF : runc Introduction

Question #2

Which API object is the recommended way to run a scalable, stateless application on your cluster?

A. ReplicaSet B. Deployment C. DaemonSet D. Pod

Here’s why:

ReplicaSet for statefull applications

Deployment is for stateless applications

DaemonSet is for application that has to be on every node.

Pod is created as part of replicaset, deployment, daemonset

Question #3

A CronJob is scheduled to run by a user every one hour. What happens in the cluster when it’s time for this CronJob to run?

A. Kubelet watches API Server for CronJob objects. When it’s time for a Job to run, it runs the Pod directly. B. Kube-scheduler watches API Server for CronJob objects, and this is why it’s called kube-scheduler. C. CronJob controller component creates a Pod and waits until it finishes to run. D. CronJob controller component creates a Job. Then the Job controller creates a Pod and waits until it finishes to run.

Here’s why:

CronJob Controller: The CronJob controller is responsible for managing CronJobs. It schedules Jobs based on the CronJob’s schedule (e.g., every hour).

Job Creation: When it’s time for a CronJob to run, the CronJob controller creates a Job resource. This Job represents the workload that needs to be executed.

Job Controller: Once the Job is created, the Job controller is responsible for managing this Job. It ensures that the Pods specified by the Job are created and completed successfully.

Pod Execution: The Job controller creates one or more Pods based on the Job specification. These Pods execute the tasks defined by the Job. The Job controller waits until the Pod(s) complete their execution.

Question #4

What is the purpose of the kubelet component within a Kubernetes cluster?

A. dashboard for Kubernetes Clusters that allows management and troubleshooting of applications. B. A network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept. C. A component that watches for newly created Pods with no assigned node, and selects a node for them to run on. D. An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

Here’s why:

kubelet An agent that runs on each node in the cluster. It makes sure that containers are running in a Pod.

The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn’t manage containers which were not created by Kubernetes.

REF : Cluster Architecture - kubelet - K8s Docs

Question #5

What is the default value for authorization-mode in Kubernetes API server?

A. --authorization-mode=RBAC B. --authorization-mode=AlwaysAllow C. --authorization-mode=AlwaysDeny D. --authorization-mode=ABAC

Here’s why:

--authorization-mode strings Ordered list of plug-ins to do authorization on secure port. Defaults to AlwaysAllow if --authorization-config is not used. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node.

REF : kube-apiserver - K8s Docs

Question #6

Let’s assume that an organization needs to process large amounts of data in bursts, on a cloud-based Kubernetes cluster. For instance: each Monday morning, they need to run a batch of 1000 compute jobs of 1 hour each, and these jobs must be completed by Monday night. What’s going to be the most cost-effective method?

A. Run a group of nodes with the exact required size to complete the batch on time, and use a combination of taints, tolerations, and nodeSelectors to reserve these nodes to the batch jobs. B. Leverage the Kubernetes Cluster Autoscaler to automatically start and stop nodes as they’re needed. C. Commit to a specific level of spending to get discounted prices (with e.g. “reserved instances” or similar mechanisms). D. Use PriorityСlasses so that the weekly batch job gets priority over other workloads running on the cluster, and can be completed on time.

Here’s why:

Burst processing workloads, like the one described, benefit from the elasticity provided by cloud-based Kubernetes clusters. With Kubernetes Cluster Autoscaler, you can scale your cluster up when there’s a demand for more resources (e.g., Monday mornings when the batch jobs need to run) and scale it down during periods of low demand (e.g., after the batch jobs are completed). This ensures that you’re only paying for the resources you actually need, avoiding over-provisioning and reducing costs.

Question #7

What is a Kubernetes service with no cluster IP address called?

A. Headless Service B. Nodeless Service C. IPLess Service D. Specless Service

Here’s why:

Headless Service: This type of service doesn’t have a cluster IP address assigned. Instead, it maps directly to the pods it manages, allowing communication with individual pods using their unique IP addresses.

REF : Headless Services - K8s Docs

Question #8

CI/CD stands for:

A. Continuous Information / Continuous Development B. Continuous Integration / Continuous Development C. Cloud Integration / Cloud Development D. Continuous Integration / Continuous Deployment

Here’s why:

In software engineering, CI/CD or CICD is the combined practices of continuous integration (CI) and continuous delivery (CD) or, less often, continuous deployment.

REF : CI/CD - Wiki

Question #9

What default level of protection is applied to the data in Secrets in the Kubernetes API?

A. The values use AES Symmetric Encryption B. The values are stored in plain text C. The values are encoded with SHA256 hashes D. The values are base64 encoded

為何 Kubernetes Secret 的內容被編碼為 base64 的字串? Kubernetes 最初採用這種編碼方法的決策，是為了讓 Secret 能夠容納二進位資料。

參考文章 :

Yet another rant in favor of Kubernetes Secrets https://dev.to/canelasevero/yet-another-rant-in-favor-of-kubernetes-secrets-432k

Question #10

What function does kube-proxy provide to a cluster?

A. Implementing the Ingress resource type for application traffic. B. Forwarding data to the correct endpoints for Services. C. Managing data egress from the cluster nodes to the network. D. Managing access to the Kubernetes API.

kube-proxy 的主要職責就是為叢集中的每個 Service 管理網路流量，讓外部或內部的流量能正確導向到對應的 Pod。

Question #11

How long should a stable API element in Kubernetes be supported (at minimum) after deprecation?

A. 9 months B. 24 months C. 12 months D. 6 months

Here’s why:

https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-a-feature-or-behavior

Question #12

What is the name of the lightweight Kubernetes distribution built for IoT and edge computing?

A. OpenShift B. k3s C. RKE D. k1s

Here’s why:

https://docs.k3s.io/

Question #13

Kubernetes ___ allows you to automatically manage the number of nodes in your cluster to meet demand.

A. Node Autoscaler B. Cluster Autoscaler C. Horizontal Pod Autoscaler D. Vertical Pod Autoscaler

Here’s why:

What is Cluster Autoscaler? Cluster Autoscaler is a standalone program that adjusts the size of a Kubernetes cluster to meet the current needs.

When does Cluster Autoscaler change the size of a cluster? Cluster Autoscaler increases the size of the cluster when:

there are pods that failed to schedule on any of the current nodes due to insufficient resources.

adding a node similar to the nodes currently present in the cluster would help.

Cluster Autoscaler decreases the size of the cluster when some nodes are consistently unneeded for a significant amount of time. A node is unneeded when it has low utilization and all of its important pods can be moved elsewhere.

Question #14

Which of the following statements is correct concerning Open Policy Agent (OPA)?

A. The policies must be written in Python language. B. Kubernetes can use it to validate requests and apply policies. C. Policies can only be tested when published. D. It cannot be used outside Kubernetes.

Here’s why:

The Open Policy Agent (OPA, pronounced “oh-pa”) is an open source, general-purpose policy engine that unifies policy enforcement across the stack. OPA provides a high-level declarative language that lets you specify policy as code and simple APIs to offload policy decision-making from your software. You can use OPA to enforce policies in microservices, Kubernetes, CI/CD pipelines, API gateways, and more.

Question #15

In a cloud native world, what does the IaC abbreviation stands for?

A. Infrastructure and Code B. Infrastructure as Code C. Infrastructure above Code D. Infrastructure across Code

Here’s why:

Infrastructure as Code，中文翻為 “基礎架構程式碼” ，提供了一種替代方法，將基礎架構的建立、設定與管理方式自動化。執行它的好處不僅可以降低 Human Error 、縮短建置系統的時間，還可以確保每次都建立出一模一樣的基礎架構。

Question #16

In which framework do the developers no longer have to deal with capacity, deployments, scaling and fault tolerance, and OS?

A. Docker Swam B. Kubernetes C. Mesos D. Serverless

Question #17

Which of the following characteristics is associated with container orchestration?

A. Application message distribution B. Dynamic scheduling C. Deploying application JAR files D. Virtual Machine distribution

Question #18

Which of the following workload require a headless service while deploying into the namespace?

A. StatefulSet B. CronJob C. Deployment D. DaemonSet

Here’s why:

StatefulSets currently require a Headless Service to be responsible for the network identity of the Pods. You are responsible for creating this Service.

REF : StatefulSets Limitations - K8s Docs

Question #19

What is Helm?

A. An open source dashboard for Kubernetes. B. A package manager for Kubernetes applications. C. A custom scheduler for Kubernetes. D. An end to end testing project for Kubernetes applications.

Here’s why:

Helm helps you manage Kubernetes applications — Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.

REF : What is Helm? - Helm Docs

Question #20

Which is the correct kubectl command to display logs in real time?

A. kubectl logs -p test-container-1 B. kubectl logs -c test-container-1 C. kubectl logs -l test-container-1 D. kubectl logs -f test-container-1

Here’s why:

The -f flag in the kubectl logs command stands for “follow,” which allows you to continuously stream the logs in real-time as new log entries are added. This is useful for monitoring applications or troubleshooting issues as they occur.

Question #21

How to load and generate data required before the Pod startup?

A. Use an init container with shared file storage. B. Use a PVC volume. C. Use a sidecar container with shared volume. D. Use another pod with a PVC.

Here’s why:

Init containers are executed before the main containers in a Pod are started. They are designed to perform initialization tasks, such as loading and generating data required before the main containers start. Init containers can share file storage with the main containers in the Pod, allowing them to perform tasks like data loading or generation and then making that data available to the main containers.

Question #22

What is the core functionality of GitOps tools like Argo CD and Flux?

A. They track production changes made by a human in a Git repository and generate a human-readable audit trail. B. They replace human operations with an agent that tracks Git commands. C. They automatically create pull requests when dependencies are outdated. D. They continuously compare the desired state in Git with the actual production state and notify or act upon differences.

Here’s why:

GitOps tools like Argo CD and Flux are designed to manage Kubernetes clusters by using Git repositories as the source of truth for cluster configuration. The core functionality of these tools involves continuously comparing the desired state of the cluster, as defined in Git, with the actual state of the cluster running in production. If there are differences between the desired and actual states, these tools will automatically reconcile those differences by either updating the production state to match the desired state or notifying operators of any discrepancies.

Question #23

Which Kubernetes workload resource ensures that all (or some) nodes run a copy of a Pod?

A. ReplicaSet B. StatefulSet C. DaemonSet D. Deployment

Here’s why:

A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.

REF : DaemonSet - K8s Docs

Question #24

We can extend the Kubernetes API with Kubernetes API Aggregation Layer and CRDs. What is CRD?

A. Custom Resource Definition B. Custom Restricted Definition C. Customized RUST Definition D. Custom RUST Definition

Here’s why:

CRD stands for Custom Resource Definition. It’s a Kubernetes extension mechanism that allows users to define their custom resources and their schema, effectively extending the Kubernetes API. Once defined, these custom resources can be managed and interacted with using standard Kubernetes API operations, just like built-in resources like Pods or Deployments.

Question #25

The Kubernetes project work is carried primarily by SIGs. What does SIG stand for?

A. Special Interest Group B. Software Installation Guide C. Support and Information Group D. Strategy Implementation Group

Here’s why:

SIG stands for Special Interest Group. In the context of Kubernetes, SIGs are groups of contributors focused on specific areas or domains within the project. Each SIG is responsible for overseeing and contributing to the development, maintenance, and improvement of its designated area, such as networking, storage, scalability, or documentation.

Question #26

What is the order of 4C’s in Cloud Native Security, starting with the layer that a user has the most control over?

A. Cloud -> Container -> Cluster -> Code B. Container -> Cluster -> Code -> Cloud C. Cluster -> Container -> Code -> Cloud D. Code -> Container -> Cluster -> Cloud

REF : The 4C’s of Cloud Native Kubernetes security

Question #27

Which group of container runtimes provides additional sandboxed isolation and elevated security?

A. rune, cgroups B. docker, containerd C. runsc, kata D. crun, cri-o

Here’s why:

runsc (gVisor): It’s a lightweight container runtime that runs containers inside a sandboxed environment, providing an additional layer of isolation using user-space kernel emulation. This allows containers to have their own isolated kernel without the overhead of full virtualization.

Kata Containers: It’s an open-source project that combines the security of virtual machines with the speed and manageability of containers. It uses lightweight VMs to run each container, providing strong isolation between containers without the performance overhead of traditional virtual machines.

Question #28

What is the common standard for Service Meshes?

A. Service Mesh Specification (SMS) B. Service Mesh Technology (SMT) C. Service Mesh Interface (SMI) D. Service Mesh Function (SMF)

Here’s why:

The Service Mesh Interface (SMI) is a specification for implementing service mesh functionality across different service mesh implementations. It provides a standard set of APIs for controlling and observing service mesh behavior, allowing for interoperability between various service mesh solutions.

An analogy for SMI could be a universal remote control standard. Just as different electronic devices from different manufacturers can be controlled using a universal remote that adheres to a common standard, service mesh implementations can communicate and interoperate effectively using the Service Mesh Interface (SMI) specification.

Question #29

Which statement about Ingress is correct?

A. Ingress provides a simple way to track network endpoints within a cluster. B. Ingress is a Service type like NodePort and ClusterIP. C. Ingress is a construct that allows you to specify how a Pod is allowed to communicate. D. Ingress exposes routes from outside the cluster to services in the cluster.

Here’s why:

Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster. Traffic routing is controlled by rules defined on the Ingress resource.

What is Ingress? - K8s Docs

Question #30

What best describes cloud native service discovery?

A. It’s a mechanism for applications and microservices to locate each other on a network. B. It’s a procedure for discovering a MAC address, associated with a given IP address. C. It’s used for automatically assigning IP addresses to devices connected to the network. D. It’s a protocol that turns human-readable domain names into IP addresses on the Internet.

Here’s why:

Cloud native service discovery is a crucial aspect of distributed systems architecture, particularly in microservices environments. It enables applications and microservices to dynamically discover and communicate with each other without hardcoding IP addresses or relying on static configurations. This mechanism allows services to be scalable, resilient, and loosely coupled, facilitating efficient communication and interaction within the cloud environment.

An analogy for service discovery could be a directory in a large office building. Just as a directory helps individuals find the locations of various offices or departments within a building, service discovery enables components within a distributed system to locate and communicate with each other dynamically, regardless of their specific network locations or configurations.

Question #31

What components are common in a service mesh?

A. tracing and log storage B. circuit breaking and Pod scheduling C. data plane and runtime plane D. service proxy and control plane

Here’s why:

In a service mesh architecture, service proxies (such as Envoy, Linkerd proxy, or Istio sidecar) are deployed alongside each service instance to handle communication between services. The control plane consists of various components responsible for configuring, managing, and monitoring the behavior of the service proxies, including features like traffic routing, load balancing, encryption, authentication, and observability. Together, these components enable advanced traffic management, security, and observability capabilities in a distributed microservices environment.

Question #32

Which storage operator in Kubernetes can help the system to self-scale, self-heal, etc?

A. Rook B. Kubernetes C. Helm D. Container Storage Interface (CSI)

Here’s why:

Rook provides features like self-scaling, self-healing, monitoring, and automation of storage management tasks. It abstracts the complexities of managing distributed storage systems and integrates them seamlessly with Kubernetes, enabling operators to deploy and manage storage clusters as easily as deploying other Kubernetes resources.

Question #33

What fields must exist in any Kubernetes object (e.g. YAML) file?

A. apiVersion, kind, metadata B. kind, namespace, data C. apiVersion, metadata, namespace D. kind, metadata, data

Here’s why:

In any Kubernetes object definition file, there are three mandatory fields that must exist:

apiVersion: Specifies the version of the Kubernetes API that the object uses. This field ensures compatibility and defines the structure of the object.

kind: Specifies the type of Kubernetes object being defined, such as Pod, Service, Deployment, etc. This field determines the behavior and functionality of the object.

metadata: Contains metadata about the object, such as its name, namespace, labels, and annotations. This metadata is used by Kubernetes to identify and manage the object within the cluster.

These fields are essential for Kubernetes to interpret and process the object correctly. The other options do not include all three mandatory fields or contain fields that are not universally required in every Kubernetes object.

Question #34

Which of the following would fall under the responsibilities of an SRE?

A. Developing a new application feature. B. Creating a monitoring baseline for an application. C. Submitting a budget for running an application in a cloud. D. Writing policy on how to submit a code change.

Here’s why:

Designing strategies for disaster recovery and fault tolerance: Just as the maintenance engineer creates contingency plans for breakdowns or accidents, the SRE devises plans for system failures or outages, ensuring that services can quickly recover and continue functioning.

Analyzing system performance metrics: Like reviewing maintenance logs and performance data for each truck, the SRE analyzes metrics from the application’s monitoring systems to identify areas for improvement and optimize performance.

By considering the SRE’s responsibilities in the context of managing a fleet of delivery trucks, we can draw parallels that illustrate the focus on reliability, efficiency, and proactive maintenance inherent in the role.

Question #35

What are the initial namespaces that Kubernetes starts with?

A. default, kube-system, kube-public, kube-node-lease B. default, system, kube-public C. kube-default, kube-system, kube-main, kube-node-lease D. kube-default, system, kube-main, kube-primary

Here’s why:

Kubernetes starts with four initial namespaces:

default Kubernetes includes this namespace so that you can start using your new cluster without first creating a namespace.

kube-node-lease This namespace holds Lease objects associated with each node. Node leases allow the kubelet to send heartbeats so that the control plane can detect node failure.

kube-public This namespace is readable by all clients (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.

kube-system The namespace for objects created by the Kubernetes system.

REF : Initial namespaces - K8s Docs

$ kubectl -n kube-node-lease get lease
NAME HOLDER AGE
tkllm-control-plane tkllm-control-plane 35d
tkllm-worker1 tkllm-worker1 35d
tkllm-worker2 tkllm-worker2 35d

Question #36

What is a probe within Kubernetes?

A. A monitoring mechanism of the Kubernetes API. B. A pre-operational scope issued by the kubectl agent. C. A diagnostic performed periodically by the kubelet on a container. D. A logging mechanism of the Kubernetes API.

Here’s why:

Within Kubernetes, a “probe” refers to a diagnostic mechanism used by the kubelet to check the health of containers running within pods.Probes are configured within the PodSpec of Kubernetes pods and are used to determine if the containers within the pod are healthy and ready to serve traffic. There are three types of probes:

Liveness Probe: Determines if the container is still running and healthy. If the liveness probe fails, Kubernetes restarts the container.

Readiness Probe: Determines if the container is ready to serve traffic. If the readiness probe fails, the pod is removed from load balancers, and no traffic is routed to it.

Startup Probe: Similar to the liveness probe, but only runs during the initial startup of a container. It helps delay the liveness and readiness probes until the application inside the container has started.

REF : Container probes - K8s Docs

Question #37

Which Kubernetes feature would you use to guard against split brain scenarios with your distributed application?

A. Replication controllers B. Consensus protocols C. Rolling updates D. StatefulSet

Here’s why:

Think of StatefulSet in Kubernetes as a traffic control system at an intersection with multiple lanes. Just as the traffic control system manages the flow of vehicles through the intersection, ensuring that only one lane can proceed at a time to prevent collisions or gridlock, StatefulSet manages the deployment of stateful applications, ensuring that only one instance can be active or “in charge” at any given time to prevent conflicts or inconsistencies in distributed systems.

REF : StatefulSet considerations - K8s Docs

Question #38

What feature must a CNI support to control specific traffic flows for workloads running in Kubernetes?

A. Border Gateway Protocol B. IP Address Management C. Pod Security Policy D. Network Policies

Here’s why:

If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), NetworkPolicies allow you to specify rules for traffic flow within your cluster, and also between Pods and the outside world. Your cluster must use a network plugin that supports NetworkPolicy enforcement.

Think of Network Policies in Kubernetes as security checkpoints at different entrances to a building. Just as security personnel at each entrance check IDs and verify permissions before allowing individuals to enter specific areas of the building, Network Policies control the flow of network traffic within the Kubernetes cluster, ensuring that only authorized Pods can communicate with each other and that access to sensitive services is restricted according to defined rules.

REF : Network Policies - K8s Docs

Question #39

What is the main role of the Kubernetes DNS within a cluster?

A. Acts as a DNS server for virtual machines that are running outside the cluster. B. Provides a DNS as a Service, allowing users to create zones and registries for domains that they own. C. Allows Pods running in dual stack to convert IPv6 calls into IPv4 calls. D. Provides consistent DNS Names for Pods and Services for workloads that need to communicate with each other.

Here’s why:

Think of Kubernetes DNS as the address book in a large office building. Just as the address book provides a consistent reference for finding the location of different departments or individuals within the building, Kubernetes DNS provides consistent DNS names for Pods and Services, allowing workloads within the cluster to discover and communicate with each other efficiently, regardless of their dynamic IP addresses or locations.

Question #40

Scenario: You have a Kubernetes cluster hosted in a public cloud provider. When trying to create a Service of type LoadBalancer, the external-ip is stuck in the “Pending” state. Which Kubernetes component is failing in this scenario?

A. Cloud Controller Manager B. Load Balancer Manager C. Cloud Architecture Manager D. Cloud Load Balancer Manager

Here’s why:

Cloud controller manager functions The controllers inside the cloud controller manager include:

Service controller Services integrate with cloud infrastructure components such as managed load balancers, IP addresses, network packet filtering, and target health checking. The service controller interacts with your cloud provider’s APIs to set up load balancers and other infrastructure components when you declare a Service resource that requires them.

REF : Service controller - K8s Docs

Question #41

What are the characteristics for building every cloud-native application?

A. Resiliency, Operability, Observability, Availability B. Resiliency, Containerd, Observability, Agility C. Kubernetes, Operability, Observability, Availability D. Resiliency, Agility, Operability, Observability

Here’s why:

A cloud native application is engineered to run on a platform and is designed for resiliency, agility, operability, and observability.

Resiliency embraces failures instead of trying to prevent them; it takes advantage of the dynamic nature of running on a platform.

Agility allows for fast deployments and quick iterations.

Operability adds control of application life cycles from inside the application instead of relying on external processes and monitors.

Observability provides information to answer questions about application state.

Question #42

What does CNCF stand for?

A. Cloud Native Community Foundation B. Cloud Native Computing Foundation C. Cloud Neutral Computing Foundation D. Cloud Neutral Community Foundation

Question #43

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called:

A. namespaces B. containers C. hypervisors D. cgroups

Here’s why:

https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ “Namespaces are intended for use in environments with many users spread across multiple teams, or projects. For clusters with a few to tens of users, you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide.”

Question #44

What component enables end users, different parts of the Kubernetes cluster, and external components to communicate with one another?

A. kubectl B. AWS Management Console C. Kubernetes API D. Google Cloud SDK

Here’s why:

The Kubernetes API server is the central component that enables communication between end users, different parts of the Kubernetes cluster, and external components.

Question #45

Which command will list the resource types that exist within a cluster?

A. kubectl api-resources B. kubectl get namespaces C. kubectl api-versions D. curl https://kubectrl/namespaces

Question #46

Which of these components is part of the Kubernetes Control Plane?

A. coredns B. cloud-controller-manager C. kube-proxy D. kubelet

Here’s why:

https://kubernetes.io/docs/concepts/architecture/cloud-controller/

Question #47

Which of the following systems is NOT compatible with the CRI runtime interface standard?

A. CRI-0 B. dockershim C. systemd D. containerd

Here’s why:

CRI - Container Runtime Interface. systemd is not container runtime interface.

Question #48

What is a key feature of a container network?

A. Proxying REST requests across a set of containers. B. Allowing containers running on separate hosts to communicate. C. Allowing containers on the same host to communicate. D. Caching remote disk access.

Here’s why:

B - Allowing containers running on separate hosts to communicate

Question #49

How can you monitor the progress for an updated Deployment/DaemonSets/StatefulSets?

A. kubectl rollout watch B. kubectl rollout progress C. kubectl rollout state D. kubectl rollout status

Here’s why:

The correct answer to the question “How can you monitor the progress for an updated Deployment/DaemonSets/StatefulSets?” is D. kubectl rollout status.

Explanation: The kubectl rollout status command is used to monitor the progress of a rollout for Kubernetes resources such as Deployments, DaemonSets, or StatefulSets. This command provides real-time updates on whether the new Pods are being created correctly and whether the old ones are being terminated. It ensures that the rollout is proceeding as expected and alerts if any issues occur during the update.

Other options like kubectl rollout watch, kubectl rollout progress, and kubectl rollout state are not valid Kubernetes commands for monitoring rollout progress. The proper command to use for tracking progress is kubectl rollout status

Question #50

What is the goal of load balancing?

A. Automatically measure request performance across instances of an application. B. Automatically distribute requests across different versions of an application. C. Automatically distribute instances of an application across the cluster. D. Automatically distribute requests across instances of an application.

Question #51

How are ReplicaSets and Deployments related?

A. Deployments manage ReplicaSets and provide declarative updates to Pods. B. ReplicaSets manage stateful applications, Deployments manage stateless applications. C. Deployments are runtime instances of ReplicaSets. D. ReplicaSets are subsets of Jobs and CronJobs which use imperative Deployments.

Here’s why:

Deployments are higher-level abstractions in Kubernetes that manage ReplicaSets.ReplicaSets, on the other hand, are lower-level controllers responsible for maintaining a specified number of identical Pods to ensure high availability and fault tolerance. ReplicaSets are used by Deployments to manage the lifecycle of Pods, including scaling, rolling updates, and maintaining a desired number of replicas.

Question #52

What factors influence the Kubernetes scheduler when it places Pods on nodes?

A. Pod memory requests, node taints, and Pod affinity. B. Pod labels, node labels, and request labels. C. Node taints, node level, and Pod priority. D. Pod priority, container command, and node labels.

Here’s why:

The Kubernetes scheduler takes into account factors like memory and CPU requests/limits, node taints and tolerations, and Pod affinity and anti-affinity rules to decide where to place Pods.

Question #53

What is the core metric type in Prometheus used to represent a single numerical value that can go up and down?

A. Summary B. Counter C. Histogram D. Gauge

Here’s why:

The Prometheus data model provides four core metrics:

Counter: A value that increases, like a request or error count Gauge: Values that increase or decrease, like memory size Histogram: A sample of observations, like request duration or response size Summary: Similar to a histogram, but also provides the total count of observations. Seems like “up and down” is the most exact match for “increase and decrease”, hence D

Question #54

What is the primary mechanism to identify grouped objects in a Kubernetes cluster?

A. Custom Resources B. Labels C. Label Selector D. Pod

Here’s why:

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#:~:text=Via%20a%20label%20selector%2C%20the%20client/user%20can%20identify%20a%20set%20of%20objects.%20The%20label%20selector%20is%20the%20core%20grouping%20primitive%20in%20Kubernetes.

Question #55

What is the name of the Kubernetes resource used to expose an application?

A. Port B. Service C. DNS D. Deployment

Here’s why:

https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/#:~:text=Expose%20Your%20App-,Using%20a%20Service%20to%20Expose%20Your%20App,-Objectives

Question #56

What is a DaemonSet?

A. It’s a type of workload that ensures a specific set of nodes run a copy of a Pod. B. It’s a type of workload responsible for maintaining a stable set of replica Pods running in any node. C. It’s a type of workload that needs to be run periodically on a given schedule. D. It’s a type of workload that provides guarantees about ordering, uniqueness, and identity of a set of Pods.

Here’s why:

A DaemonSet ensures that all (or some) nodes run a copy of a specific Pod. When a new node is added to the cluster, a Pod from the DaemonSet is automatically added to that node. When a node is removed from the cluster, the Pod is also removed.

Question #57

What is the telemetry component that represents a series of related distributed events that encode the end-to-end request flow through a distributed system?

A. Metrics B. Logs C. Spans D. Traces

Here’s why:

In a distributed system, traces represent a series of related events that show the end-to-end request flow. Each trace is composed of one or more spans, which represent individual operations within the trace.

Question #58

In the Kubernetes platform, which component is responsible for running containers?

A. etcd B. CRI-O C. cloud-controller-manager D. kube-controller-manager

Here’s why:

CRI-O is a lightweight container runtime specifically designed for Kubernetes. It implements the Kubernetes Container Runtime Interface (CRI) to enable the use of containers within Kubernetes clusters. It is responsible for running the containers specified by Kubernetes

Question #59

Services and Pods in Kubernetes are ______ objects.

A. JSON B. YAML C. Java D. REST

Here’s why:

The Kubernetes API is the most important component of a Kubernetes cluster. Without it, communication with the cluster is not possible, every user and every component of the cluster itself needs the api-server. Like many other APIs, the Kubernetes API is implemented as a RESTful interface that is exposed over HTTPS. Through the API, a user or service can create, modify, delete or retrieve resources that reside in Kubernetes.

https://docs.openshift.com/container-platform/3.11/architecture/core_concepts/pods_and_services.html#:~:text=Like%20pods%2C%20services%20are%20REST%20objects.

Question #60

What Kubernetes component handles network communications inside and outside of a cluster, using operating system packet filtering if available?

A. kube-proxy B. kubelet C. etcd D. kube-controller-manager

Here’s why:

https://kubernetes.io/docs/concepts/overview/components/#:~:text=kube%2Dproxy%20maintains,and%20it%27s%20available.

Question #61

What Kubernetes control plane component exposes the programmatic interface used to create, manage and interact with the Kubernetes objects?

A. kube-controller-manager B. kube-proxy C. kube-apiserver D. etcd

Here’s why:

The kube-apiserver is the Kubernetes control plane component that exposes the Kubernetes API. It provides the programmatic interface used to create, manage, and interact with Kubernetes objects.

Question #62

Which type of Service requires manual creation of Endpoints?

A. LoadBalancer B. Services without selectors C. NodePort D. ClusterIP with selectors

Here’s why:

When you create a Kubernetes Service without specifying selectors, Kubernetes does not automatically create Endpoints for the Service. In such cases, you need to manually create the Endpoints to map the Service to the set of Pods or IP addresses you want it to target

Question #63

Which of these commands is used to retrieve the documentation and field definitions for a Kubernetes resource?

A. kubectl explain B. kubectl api-resources C. kubectl get –help D. kubectl show

Here’s why:

The kubectl explain command is used to retrieve documentation and field definitions for a Kubernetes resource. It provides detailed information about the fields and their descriptions for a specific resource type.

Question #64

Which of the following is a lightweight tool that manages traffic flows between services, enforces access policies, and aggregates telemetry data, all without requiring changes to application code?

A. NetworkPolicy B. Linkerd C. kube-proxy D. Nginx

Question #65

Which Kubernetes resource uses immutable: true boolean field?

A. Deployment B. Pod C. ConfigMap D. ReplicaSet

Here’s why:

In Kubernetes, you can create ConfigMaps with an immutable: true boolean field. This feature ensures that the data within the ConfigMap cannot be modified after creation, which can help improve the cluster’s security and stability.

https://kubernetes.io/docs/concepts/configuration/configmap/#configmap-immutable

Question #66

Which statement about the Kubernetes network model is correct?

A. Pods can only communicate with Pods exposed via a Service. B. Pods can communicate with all Pods without NAT. C. The Pod IP is only visible inside a Pod. D. The Service IP is used for the communication between Services.

Here’s why:

In the Kubernetes network model, each Pod has a unique IP address, and Pods can communicate with each other directly without needing Network Address Translation (NAT).

https://kubernetes.io/docs/concepts/services-networking/#:~:text=pods%20can%20communicate%20with%20all%20other%20pods%20on%20any%20other%20node%20without%20NAT

Question #67

What is the resource type used to package sets of containers for scheduling in a cluster?

A. Pod B. ContainerSet C. ReplicaSet D. Deployment

Question #68

Can a Kubernetes Service expose multiple ports?

A. No, you can only expose one port per each Service. B. Yes, but you must specify an unambiguous name for each port. C. Yes, the only requirement is to use different port numbers. D. No, because the only port you can expose is port number 443.

Here’s why:

In Kubernetes, a Service can expose multiple ports, but you need to assign a unique name to each port to avoid ambiguity.

https://kubernetes.io/docs/concepts/services-networking/service/#multi-port-services

Question #69

Which persona is normally responsible for defining, testing, and running an incident management process?

A. Site Reliability Engineers B. Project Managers C. Application Developers D. Quality Engineers

Here’s why:

Site Reliability Engineers (SREs) are typically responsible for defining, testing, and running incident management processes. They focus on ensuring the reliability and performance of systems, as well as handling operational issues and incidents.

Question #70

What is the default deployment strategy in Kubernetes?

A. Rolling update B. Blue/Green deployment C. Canary deployment D. Recreate deployment

Question #71

Which command provides information about the field replicas within the spec resource of a deployment object?

A. kubectl get deployment.spec.replicas B. kubectl explain deployment.spec.replicas C. kubectl describe deployment.spec.replicas D. kubectl explain deployment –spec.replicas

Here’s why:

The command kubectl explain is used to provide detailed information about the fields of Kubernetes objects, including their specifications. The command kubectl explain deployment.spec.replicas specifically provides information about the replicas field within the spec of a deployment object. This is the most appropriate command for getting details about that particular field.

Question #72

Which of the following is a responsibility of the governance board of an open source project?

A. Decide about the marketing strategy of the project. B. Review the pull requests in the main branch. C. Outline the project’s “terms of engagement”. D. Define the license to be used in the project.

Here’s why:

The governance board of an open source project is responsible for setting the project’s “terms of engagement,” which includes defining the rules, guidelines, and processes for contributing to and interacting with the project. This ensures that the project operates smoothly and transparently.

Question #73

What is the role of a NetworkPolicy in Kubernetes?

A. The ability to cryptic and obscure all traffic. B. The ability to classify the Pods as isolated and non isolated. C. The ability to prevent loopback or incoming host traffic. D. The ability to log network security events.

Here’s why:

In Kubernetes, a NetworkPolicy is used to control the communication between Pods within the cluster. It allows you to specify rules for which Pods can communicate with each other, effectively classifying Pods as isolated or non-isolated based on the defined rules

https://kubernetes.io/docs/concepts/services-networking/network-policies/#the-two-sorts-of-pod-isolation:~:text=There%20are%20two,pod%20to%20another.

Question #74

What are the most important resources to guarantee the performance of an etcd cluster?

A. CPU and disk capacity. B. Network throughput and disk I/O. C. CPU and RAM memory. D. Network throughput and CPU.

Here’s why:

Consensus performance, especially commit latency, is limited by two physical constraints: network IO latency and disk IO latency.

https://etcd.io/docs/v3.4/op-guide/performance/

Question #75

How do you deploy a workload to Kubernetes without additional tools?

A. Create a Bash script and run it on a worker node. B. Create a Helm Chart and install it with helm. C. Create a manifest and apply it with kubectl. D. Create a Python script and run it with kubectl.

Question #76

How do you perform a command in a running container of a Pod?

A. kubectl exec -- B. docker exec C. kubectl run -- D. kubectl attach -i

Question #77

How to create a headless service?

A. By specifying .spec.ClusterIP: headless B. By specifying .spec.clusterIP: None C. By specifying .spec.ClusterIP: 0.0.0.0 D. By specifying .spec.ClusterIP: localhost

Question #78

How does dynamic storage provisioning work?

A. A user requests dynamically provisioned storage by including an existing storage class in their PersistentVolumeClaim. B. An administrator creates a storage class and includes it in their pod YAML definition file without creating a PersistentVolumeClaim. C. A pod requests dynamically provisioned storage by including a storage class and the pod name in their PersistentVolumeClaim. D. An administrator creates a PersistentVolume and includes the name of the PersistentVolume in their pod YAML definition file.

Here’s why:

https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/#:~:text=The%20dynamic%20provisioning%20feature%20eliminates,it%20is%20requested%20by%20users.

Question #79

Which of the following are tasks performed by a container orchestration tool?

A. Schedule, scale, and manage the health of containers. B. Create images, scale, and manage the health of containers. C. Debug applications, and manage the health of containers. D. Store images, scale, and manage the health of containers.

Here’s why:

https://cloud.google.com/discover/what-is-container-orchestration?hl=en

Question #80

Which of the following is a definition of Hybrid Cloud?

A. An architecture that uses a combination of services running in public and private data centers, only including data centers from the same cloud provider. B. A cloud native architecture that uses a combination of services running in public clouds, excluding data centers in different availability zones. C. A cloud native architecture that uses a combination of services running in different public and private clouds, including on-premises data centers. D. An architecture that uses a combination of services running in public and private data centers, excluding serverless functions.

Here’s why:

A cloud native architecture that uses a combination of services running in different public and private clouds, including on-premises data centers.

Question #81

What is a Kubernetes Service Endpoint?

A. It is the API Endpoint of our Kubernetes cluster. B. It is a name of special Pod in kube-system namespace. C. It is an IP address that we can access from the Internet. D. It is an object that gets IP addresses of individual Pods assigned to it.

Question #82

Why is Cloud-Native Architecture important?

A. Cloud Native Architecture revolves around containers, microservices and pipelines. B. Cloud Native Architecture removes constraints to rapid innovation. C. Cloud Native Architecture is modern for application deployment and pipelines. D. Cloud Native Architecture is a bleeding edge technology and service.

Question #83

Which kubernetes component is the smallest deployable unit of computing?

A. StatefulSet B. Deployment C. Pod D. Container

Here’s why:

In Kubernetes, the Pod is the smallest deployable unit of computing. A Pod can contain one or more containers, but it is the fundamental unit that Kubernetes deploys, manages, and scales. Containers within a Pod share the same network and storage resources, making the Pod the atomic unit for scheduling and running containers in Kubernetes.

Question #84

What kubectl command is used to retrieve the resource consumption (CPU and memory) for nodes or Pods?

A. kubectl cluster-info B. kubectl version C. kubectl top D. kubectl api-resources

Question #85

Which are the two primary modes for Service discovery within a Kubernetes cluster?

A. Environment variables and DNS B. API Calls and LDAP C. Labels and Radius D. Selectors and DHCP

Here’s why:

https://www.densify.com/kubernetes-autoscaling/kubernetes-service-discovery/

Question #86

Which of the following capabilities are you allowed to add to a container using the Restricted policy?

A. CHOWN B. SYS_CHROOT C. SETUID D. NET_BIND_SERVICE

Here’s why:

Containers must drop ALL capabilities, and are only permitted to add back the NET_BIND_SERVICE capability.

https://kubernetes.io/docs/concepts/security/pod-security-standards/#:~:text=add%20back%20the-,NET_BIND_SERVICE,-capability.%20This

Question #87

What methods can you use to scale a deployment?

A. With kubectl edit deployment exclusively. B. With kubectl scale-up deployment exclusively. C. With kubectl scale deployment and kubectl edit deployment. D. With kubectl scale deployment exclusively.

Question #88

What is a sidecar container?

A. A Pod that runs next to another container within the same Pod. B. A container that runs next to another Pod within the same namespace. C. A container that runs next to another container within the same Pod. D. A Pod that runs next to another Pod within the same namespace.

Question #89

Which is an industry-standard container runtime with an “emphasis” on simplicity, robustness, and portability?

A. cri-o B. lxd C. containerd D. kata-runtime

Question #90

What does vertical scaling an application deployment describe best?

A. The act of adding/removing applications to meet demand. B. The act of adding/removing node instances to the cluster to meet demand. C. The act of adding/removing resources to applications to meet demand. D. The act of adding/removing application instances of the same application to meet demand.

Question #91

Which Prometheus metric represents a single value that can go up and down?

A. Counter B. Gauge C. Summary D. Histogram

Question #92

What is Serverless computing?

A. A computing method of providing backend services on an as-used basis. B. A computing method of providing services for AI and ML operating systems. C. A computing method of providing services for quantum computing operating systems. D. A computing method of providing services for cloud computing operating systems.

Here’s why:

Serverless computing is a cloud computing model in which the cloud provider automatically manages the infrastructure and dynamically allocates resources as needed. Developers can focus on writing code without worrying about server management, and they are only charged for the actual usage of resources rather than pre-provisioned capacity.

Question #93

What is the purpose of the CRI?

A. To provide runtime integration control when multiple runtimes are used. B. Support container replication and scaling on nodes. C. Provide an interface allowing Kubernetes to support pluggable container runtimes. D. Allow the definition of dynamic resource criteria across containers.

Question #94

Imagine there is a requirement to run a database backup every day. Which Kubernetes resource could be used to achieve that?

A. Kube-scheduler B. CronJob C. Task D. Job

Question #95

In CNCF, who develops specifications for industry standards around container formats and runtimes?

A. Open Container Initiative (OCI) B. Linux Foundation Certification Group (LFCG) C. Container Network Interface (CNI) D. Container Runtime Interface (CRI)

Question #96

Which of the following options includes valid API versions?

A. alpha1v1, beta3v3, v2 B. alpha1, beta3, v2 C. v1alpha1, v2beta3, v2 D. v1alpha1, v2beta3, 2.0

Here’s why:

Kubernetes follows a specific convention for API versioning, which includes versions like v1alpha1, v2beta3, and stable versions like v1, v2, etc. The format typically consists of a version (vX), followed by an optional alpha or beta designation with a version number (e.g., alpha1, beta3). This structure is used to represent different stages of API stability, from alpha (experimental) to beta (more stable) to full release (e.g., v1, v2)

https://kubernetes.io/docs/reference/using-api/

Question #97

Which of the following will view the snapshot of previously terminated ruby container logs from Pod web-1?

A. kubectl logs -p -c ruby web-1 B. kubectl logs -c ruby web-1 C. kubectl logs -p ruby web-1 D. kubectl logs -p -c web-1 ruby

Here’s why:

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs

Question #98

A Kubernetes _____ is an abstraction that defines a logical set of Pods and a policy by which to access them.

A. Selector B. Controller C. Service D. Job

Question #99

How many hosts are required to set up a highly available Kubernetes cluster when using an external etcd topology?

A. Four hosts. Two for control plane nodes and two for etcd nodes. B. Four hosts. One for a control plane node and three for etcd nodes. C. Three hosts. The control plane nodes and etcd nodes share the same host. D. Six hosts. Three for control plane nodes and three for etcd nodes.

Here’s why:

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/ha-topology/#:~:text=A%20minimum%20of%20three%20hosts,HA%20cluster%20with%20this%20topology.

Question #100

Which of these events will cause the kube-scheduler to assign a Pod to a node?

A. When the Pod crashes because of an error. B. When a new node is added to the Kubernetes cluster. C. When the CPU load on the node becomes too high. D. When a new Pod is created and has no assigned node.

Question #101

What helps an organization to deliver software more securely at a higher velocity?

A. Kubernetes B. apt-get C. Docker Images D. CI/CD Pipeline

Question #102

Which resource do you use to attach a volume in a Pod?

A. StorageVolume B. PersistentVolume C. StorageClass D. PersistentVolumeClaim

Question #103

Which key-value store is used to persist Kubernetes cluster data?

A. etcd B. ZooKeeper C. ControlPlaneStore D. Redis

Question #104

What Linux namespace is shared by default by containers running within a Kubernetes Pod?

A. Host Network B. Network C. Process ID D. Process Name

Here’s why:

By default, containers running within a Kubernetes Pod share the same network namespace. This means that they can communicate with each other using localhost and can also share the same IP address and port space. Sharing the network namespace allows containers within the same Pod to efficiently collaborate and communicate.

Question #105

What is a Dockerfile?

A. A bash script that is used to automatically build a docker image. B. A config file that defines which image registry a container should be pushed to. C. A text file that contains all the commands a user could call on the command line to assemble an image. D. An image layer created by a running container stored on the host.

Question #106

What does the “nodeSelector” within a PodSpec use to place Pods on the target nodes?

A. Annotations B. IP Addresses C. Hostnames D. Labels

Question #107

What do Deployments and StatefulSets have in common?

A. They manage Pods that are based on an identical container spec. B. They support the OnDelete update strategy. C. They support an ordered, graceful deployment and scaling. D. They maintain a sticky identity for each of their Pods.

Question #108

What is the practice of bringing financial accountability to the variable spend model of cloud resources?

A. FaaS B. DevOps C. CloudCost D. FinOps

Question #109

What is a best practice to minimize the container image size?

A. Use a DockerFile. B. Use multistage builds. C. Build images with different tags. D. Add a build.sh script.

Question #110

Which tools enable Kubernetes HorizontalPodAutoscalers to use custom, application-generated metrics to trigger scaling events?

A. Prometheus and the prometheus-adapter. B. Graylog and graylog-autoscaler metrics. C. Graylog and the kubernetes-adapter. D. Grafana and Prometheus.

Here’s why:

Prometheus is widely used for monitoring and collecting metrics in Kubernetes environments. The prometheus-adapter allows Kubernetes HorizontalPodAutoscalers (HPAs) to use custom, application-generated metrics from Prometheus to trigger scaling events.

Question #111

Which of the following is a valid PromQL query?

A. SELECT * from http_requests_total WHERE job=apiserver B. http_requests_total WHERE (job=“apiserver”) C. SELECT * from http_requests_total D. http_requests_total(job=“apiserver”)

Here’s why:

In PromQL (Prometheus Query Language), the syntax for filtering metrics based on label values uses curly braces {}. Option D correctly filters the http_requests_total metric to include only the metrics where the label job equals “apiserver”.

Question #112

Which of the following best describes horizontally scaling an application deployment?

A. The act of adding/removing node instances to the cluster to meet demand. B. The act of adding/removing applications to meet demand. C. The act of adding/removing application instances of the same application to meet demand. D. The act of adding/removing resources to application instances to meet demand.

Question #113

How many different Kubernetes service types can you define?

A. 2 B. 3 C. 4 D. 5

Here’s why:

https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

Question #114

What is the difference between a Deployment and a ReplicaSet?

A. With a Deployment, you can’t control the number of pod replicas. B. A ReplicaSet does not guarantee a stable set of replica pods running. C. A Deployment is basically the same as a ReplicaSet with annotations. D. A Deployment is a higher-level concept that manages ReplicaSets.

Question #115

The Container Runtime Interface (CRI) defines the protocol for the communication between:

A. The kubelet and the container runtime. B. The container runtime and etcd. C. The kube-apiserver and the kubelet. D. The container runtime and the image registry.

Question #116

Which authorization-mode allows granular control over the operations that different entities can perform on different objects in a Kubernetes cluster?

A. Webhook Mode Authorization Control B. Role Based Access Control C. Node Authorization Access Control D. Attribute Based Access Control

Here’s why:

Role-Based Access Control (RBAC) is an authorization mode in Kubernetes that provides fine-grained control over the actions that users, groups, and service accounts can perform on various resources within the cluster. RBAC allows administrators to define roles and role bindings to specify permissions for different entities.

Question #117

Which of the following is a correct definition of a Helm chart?

A. A Helm chart is a collection of YAML files bundled in a tar.gz file and can be applied without decompressing it. B. A Helm chart is a collection of JSON files and contains all the resource definitions to run an application on Kubernetes. C. A Helm chart is a collection of YAML files that can be applied on Kubernetes by using the kubectl tool. D. A Helm chart is similar to a package and contains all the resource definitions to run an application on Kubernetes.

Question #118

Which of the following sentences is true about namespaces in Kubernetes?

A. You can create a namespace within another namespace in Kubernetes. B. You can create two resources of the same kind and name in a namespace. C. The default namespace exists when a new cluster is created. D. All the objects in the cluster are namespaced by default.

Question #119

How does Horizontal Pod autoscaling work in Kubernetes?

A. The Horizontal Pod Autoscaler controller adds more CPU or memory to the pods when the load is above the configured threshold, and reduces CPU or memory when the load is below. B. The Horizontal Pod Autoscaler controller adds more pods when the load is above the configured threshold, but does not reduce the number of pods when the load is below. C. The Horizontal Pod Autoscaler controller adds more pods to the specified DaemonSet when the load is above the configured threshold, and reduces the number of pods when the load is below. D. The Horizontal Pod Autoscaler controller adds more pods when the load is above the configured threshold, and reduces the number of pods when the load is below.

Here’s why:

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/

Question #120

What is a Pod?

A. A networked application within Kubernetes. B. A storage volume within Kubernetes. C. A single container within Kubernetes. D. A group of one or more containers within Kubernetes.

Question #121

What element allows Kubernetes to run Pods across the fleet of nodes?

A. The node server. B. The etcd static pods. C. The API server. D. The kubelet.

Question #122

What is the Kubernetes object used for running a recurring workload?

A. Job B. Batch C. DaemonSet D. CronJob

Question #123

In the DevOps framework and culture, who builds, automates, and offers continuous delivery tools for developer teams?

A. Application Users B. Application Developers C. Platform Engineers D. Cluster Operators

Question #124

Which kubectl command is useful for collecting information about any type of resource that is active in a Kubernetes cluster?

A. describe B. list C. expose D. explain

Question #125

The cloud native architecture centered around microservices provides a strong system that ensures ______________.

A. fallback B. resiliency C. failover D. high reachability

Question #126

Which of the following is the correct command to run a nginx deployment with 2 replicas?

A. kubectl run deploy nginx --image=nginx --replicas=2 B. kubectl create deploy nginx --image=nginx --replicas=2 C. kubectl create nginx deployment --image=nginx –replicas=2 D. kubectl create deploy nginx --image=nginx --count=2

Question #127

What does “Continuous Integration” mean?

A. The continuous integration and testing of code changes from multiple sources manually. B. The continuous integration and testing of code changes from multiple sources via automation. C. The continuous integration of changes from one environment to another. D. The continuous integration of new tools to support developers in a project.

Question #128

Which of the following options is true about considerations for large Kubernetes clusters?

A. Kubernetes supports up to 1000 nodes and recommends no more than 1000 containers per node. B. Kubernetes supports up to 5000 nodes and recommends no more than 500 pods per node. C. Kubernetes supports up to 5000 nodes and recommends no more than 110 pods per node. D. Kubernetes supports up to 50 nodes and recommends no more than 1000 containers per node.

Here’s why:

https://kubernetes.io/docs/setup/best-practices/cluster-large/

Question #129

Which component of the node is responsible to run workloads?

A. The kubelet. B. The kubeproxy. C. The kube-apiserver. D. The container runtime.

Here’s why:

The component of the node responsible for running workloads is the kubelet. While the container runtime (like Docker, containerd, or CRI-O) is crucial for actually running the containers, the kubelet is the orchestrator that ensures the containers are running as specified by the Kubernetes control plane.

Question #130

The IPv4/IPv6 dual stack in Kubernetes:

A. Translates an IPv4 request from a service to an IPv6 service. B. Allows you to access the IPv4 address by using the IPv6 address. C. Requires NetworkPolicies to prevent services from mixing requests. D. Allows you to create IPv4 and IPv6 dual stack services.

Here’s why:

Kubernetes 支援 IPv4/IPv6 dual stack，意思是：叢集中的 Pod、Service 可以同時擁有 IPv4 與 IPv6 位址，並且可以透過任一協定進行通訊。 https://kubernetes.io/docs/concepts/services-networking/dual-stack/

Question #131

What does “continuous” mean in the context of CI/CD?

A. Frequent releases, Manual processes, Repeatable, Fast processing B. Periodic releases, Manual processes, Repeatable, Automated Processing C. Frequent releases, Automated processes, Repeatable, Fast processing D. Periodic releases, Automated processes, Repeatable, Automated processing

Here’s why:

https://spot.io/resources/ci-cd/what-is-ci-cd-continuous-integration-continuous-deployment/

Question #132

What is ephemeral storage?

A. Storage space that need not persist across restarts. B. Storage that may grow dynamically. C. Storage used by multiple consumers (e.g. multiple Pods). D. Storage that is always provisioned locally

Here’s why:

https://godleon.github.io/blog/Kubernetes/k8s-Scheduling-Manage-Compute-Resource-for-Container/#%E4%BB%80%E9%BA%BC%E6%98%AF-ephemeral-storage

Question #133

What is the reference implementation of the OCI runtime specification?

A. lxc B. cri-o C. runc D. docker

Here’s why:

runC is the reference implementation of the OCI runtime specification. It is a lightweight, portable container runtime that provides a standardized way to run containers. runC was originally extracted from Docker and has since become the reference implementation for the OCI runtime specification, ensuring compatibility and interoperability across different container runtimes.

Question #134

What is a Service?

A. A static network mapping from a Pod to a port. B. A way to expose an application running on a set of Pods. C. The network configuration for a group of Pods. D. An NGINX load balancer that gets deployed for an application.

Here’s why:

https://kubernetes.io/docs/concepts/services-networking/service/

Question #135

What’s the difference between a security profile and a security context?

A. Security Contexts configure Clusters and Namespaces at runtime. Security profiles are control plane mechanisms to enforce specific settings in the Security Context. B. Security Contexts configure Pods and Containers at runtime. Security profiles are control plane mechanisms to enforce specific settings in the Security Context. C. Security Profiles configure Pods and Containers at runtime. Security Contexts are control plane mechanisms to enforce specific settings in the Security Profile. D. Security Profiles configure Clusters and Namespaces at runtime. Security Contexts are control plane mechanisms to enforce specific settings in the Security Profile.

Here’s why:

https://kubernetes.io/docs/concepts/security/pod-security-standards/#what-s-the-difference-between-a-security-profile-and-a-security-context

Question #136

At which layer would distributed tracing be implemented in a cloud native deployment?

A. Network B. Application C. Database D. Infrastructure

Here’s why:

https://www.cncf.io/blog/2023/03/29/distributed-tracing-in-kubernetes-apps-what-you-need-to-know/#:~:text=How%20distributed%20tracing%20works

Question #137

What framework does Kubernetes use to authenticate users with JSON Web Tokens?

A. OpenID Connect B. OpenID Container C. OpenID Cluster D. OpenID CNCF

Here’s why:

Kubernetes 使用 OpenID Connect (OIDC) 來處理身份驗證，並利用 JSON Web Tokens (JWTs) 進行用戶身份驗證。 https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

Question #138

Which of the following is a feature Kubernetes provides by default as a container orchestration tool?

A. A portable operating system. B. File system redundancy. C. A container image registry. D. Automated rollouts and rollbacks.

Here’s why:

https://kubernetes.io/docs/concepts/overview/#:~:text=Automated%20rollouts%20and%20rollbacks

Question #139

Which of the following sentences is true about container runtimes in Kubernetes?

A. If you let iptables see bridged traffic, you don’t need a container runtime. B. If you enable IPv4 forwarding, you don’t need a container runtime. C. Container runtimes are deprecated, you must install CRI on each node D. You must install a container runtime on each node to run pods on it.

Here’s why:

每個 Node 上都必須安裝一個 Container Runtime，才能讓該節點執行 Pod。Pod 裡的貨櫃由 Container Runtime 負責實際建立、管理與執行。

Question #140

If a Pod was waiting for container images to download on the scheduled node, what state would it be in?

A. Failed B. Succeeded C. Unknown D. Pending

Here’s why:

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase

Question #141

What is CloudEvents?

A. It is a specification for describing event data in common formats for Kubernetes network traffic management and cloud providers. B. It is a specification for describing event data in common formats in all cloud providers including major cloud providers C. It is a specification for describing event data in common formats to provide interoperability across services, platforms and systems D. It is a Kubernetes specification for describing events data in common formats for iCloud services, iOS platforms and iMac.

Here’s why:

CloudEvents 是由 CNCF 推出的開放規範，目的是讓來自不同來源的事件（Events）能用一致的格式進行描述，實現跨系統、平台與服務的互通性 https://cloudevents.io/

Question #142

What is the main purpose of etcd in Kubernetes?

A. etcd stores all cluster data in a key value store. B. etcd stores the containers running in the cluster for disaster recovery. C. etcd stores copies of the Kubernetes config files that live /etc/. D. etcd stores the YAML definitions for all the cluster components.

Here’s why:

https://www.armosec.io/glossary/etcd-kubernetes/#:~:text=key%2Dvalue%20data%20store.

Question #143

Imagine you’re releasing open-source software for the first time. Which of the following is a valid semantic version?

A. 1.0 B. 2021-10-11 C. 0.1.0-rc D. v1beta1

Here’s why:

https://stackoverflow.com/questions/52236193/what-is-the-proper-first-version-0-1-0-or-0-0-1-or-anything-else

Question #144

Which cloud native tool keeps Kubernetes clusters in sync with sources of configuration (like Git repositories), and automates updates to configuration when there is new code to deploy?

A. Flux and ArgoCD B. GitOps Toolkit C. Linkerd and Istio D. Helm and Kustomize

Here’s why:

https://www.pomerium.com/blog/best-kubernetes-management-tools

https://earthly.dev/blog/flux-vs-argo-cd/

Question #145

Which tool is used to streamline installing and managing Kubernetes applications?

A. apt B. helm C. service D. brew

Here’s why:

https://helm.sh/

Question #146

What are the two steps performed by the kube-scheduler to select a node to schedule a pod?

A. Grouping and placing B. Filtering and selecting C. Filtering and scoring D. Scoring and creating

kube-scheduler selects a node for the pod in a 2-step operation:

Filtering Scoring

The filtering step finds the set of Nodes where it’s feasible to schedule the Pod. For example, the PodFitsResources filter checks whether a candidate Node has enough available resources to meet a Pod’s specific resource requests. After this step, the node list contains any suitable Nodes; often, there will be more than one. If the list is empty, that Pod isn’t (yet) schedulable.

In the scoring step, the scheduler ranks the remaining nodes to choose the most suitable Pod placement. The scheduler assigns a score to each Node that survived filtering, basing this score on the active scoring rules.

Finally, kube-scheduler assigns the Pod to the Node with the highest ranking. If there is more than one node with equal scores, kube-scheduler selects one of these at random.

參考文章 : https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/#kube-scheduler-implementation

Question #147

To visualize data from Prometheus you can use expression browser or console templates. What is the other data visualization tool commonly used together with Prometheus?

A. Grafana B. Graphite C. Nirvana D. GraphQL

Here’s why:

https://k21academy.com/prometheus/an-introduction-to-prometheus-dashboard-its-options/#4:~:text=what%20is%20important.-,Grafana,-Grafana%20is%20a

Question #148

Which item is a Kubernetes node component?

A. kube-scheduler B. kubectl C. kube-proxy D. etcd

Here’s why:

https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/#synopsis

Question #149

In a serverless computing architecture:

A. Users of the cloud provider are charged based on the number of requests to a function. B. Serverless functions are incompatible with containerized functions. C. Users should make a reservation to the cloud provider based on an estimation of usage. D. Containers serving requests are running in the background in idle status.

Here’s why:

https://cloud.google.com/discover/what-is-serverless-architecture?hl=zh_tw

Question #150

How is application data maintained in containers?

A. Store data into data folders. B. Store data in separate folders. C. Store data into sidecar containers. D. Store data into volumes.

Question #151

Which of the following scenarios would benefit the most from a service mesh architecture?

A. A few applications with hundreds of pod replicas running in multiple clusters, each one providing multiple services. B. Thousands of distributed applications running in a single cluster, each one providing multiple services. C. Tens of distributed applications running in multiple clusters, each one providing multiple services. D. Thousands of distributed applications running in multiple clusters, each one providing multiple services.

Here’s why:

Service Mesh 最適用情境：

應用數量多

分散於多叢集（multi-cluster）

微服務頻繁互動、需要安全通信與觀察能力

Question #152

Kubernetes___ protect you against voluntary interruptions (such as draining nodes) to run applications in a highly available manner.

A. Pod Topology Spread Constraints B. Pod Disruption Budgets C. Taints and Tolerances D. Resource Limits and Requests

Here’s why:

PodDisruptionBudget（PDB）是 Kubernetes 提供的一種機制，用來確保應用程式在節點維護(drain) Pod 時，仍然維持一定數量的 Pod，以避免服務中斷，從而維持應用的可用性。 https://github.com/mercari/production-readiness-checklist/blob/master/docs/concepts/pod-disruption-budget.md

Question #153

What sentence is true about CronJobs in Kubernetes?

A. A CronJob creates one or multiple Jobs on a repeating schedule. B. A CronJob creates one container on a repeating schedule. C. CronJobs are useful on Linux but are obsolete In Kubernetes. D. The CronJob schedule format Is different in Kubernetes and Linux.

Question #154

What is the purpose of the kube-proxy?

A. The kube-proxy balances network requests to pods. B. The kube-proxy maintains network rules on nodes. C. The kube-proxy ensures the cluster connectivity with internet. D. The kube-proxy maintains the DNS rules of the cluster.

Here’s why:

https://isovalent.com/blog/post/why-replace-iptables-with-ebpf/#:~:text=Kube%2Dproxy%20is,the%2Dbox%20default.

Question #155

Manual reclamation policy of a PV resource is known as:

A. claimRef B. Delete C. Retain D. Recycle

Here’s why:

https://spot.io/resources/kubernetes-architecture/7-stages-in-the-life-of-a-kubernetes-persistent-volume-pv/#:~:text=The%20retain%20reclaim,up%20the%20data.

Question #156

Which component of the Kubernetes architecture is responsible for integration with the CRI container runtime?

A. kubeadm B. kubelet C. kube-aplserver D. kubectl

Here’s why:

https://kubernetes.io/docs/concepts/architecture/cri/

Question #157

Which one of the following is an open source runtime security tool?

A. lxd B. containerd C. falco D. gvisor

Here’s why:

https://falco.org/

Question #158

What are the advantages of adopting a GitOps approach for your deployments?

A. Reduce failed deployments, operational costs, and fragile release processes. B. Reduce failed deployments, configuration drift, and fragile release processes. C. Reduce failed deployments, operational costs, and learn git. D. Reduce failed deployments, configuration drift and improve your reputation.

Question #159

Which command lists the running containers in the current Kubernetes namespace?

A. kubectl get pods B. kubectl ls C. kubectl ps D. kubectl show pods

Question #160

Which of the following is a good habit for cloud native cost efficiency?

A. Follow an automated approach to cost optimization, including visibility and forecasting. B. Follow manual processes for cost analysis, including visibility and forecasting. C. Use only one cloud provider to simplify the cost analysis. D. Keep your legacy workloads unchanged, to avoid cloud costs.

Question #161

Which of these is a valid container restart policy?

A. On login B. On update C. On start D. On failure

Here’s why:

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

2024/11/18 考試心得

多了差不多 10 題不一樣，以下是有印象的幾題

edge and service proxy tool 是誰，答案是 envoy
node 在 false 或 notready 預設幾分鐘後會驅逐 pod，答案是 5 分鐘
metrics server 的 API 是甚麼，答案是 metrics.k8s.io
v2beta3 代表什麼意思，答案是 The software is well tested. Enabling a feature is considered safe.。
What Linux namespace is shared by default by containers running within a Kubernetes Pod? 答案變成 storage and netwrok resource
cloud native application 的定義，答案待確認。
flux ….? 題目忘了，但答案應該是 Gitops。
Container Orchestration Tools 相關，答案待確認。
Ephemeral Containers 的定義，答案是 a special type of container that runs temporarily in an existing Pod。有選項是 init contianer 的定義，不要搞混。

Create a normal user with Client certificate in Kubernetes

Thu, 27 Feb 2025 07:39:54 +0800

Step1: 生成私鑰

openssl genrsa -out user2.key 2048

功能：使用 OpenSSL 工具生成一個 2048 位元的 RSA 私鑰，並將其保存到文件 user2.key 中。
參數說明：
- genrsa：OpenSSL 的子命令，用於生成 RSA 私鑰。
- -out user2.key：指定輸出文件名稱為 user2.key。
- 2048：指定私鑰的位元長度為 2048 位，這是當前推薦的安全標準，提供足夠的加密強度。
作用：私鑰是用戶身份認證的基礎，用於後續生成證書簽名請求（CSR）和與簽署的證書配對進行身份驗證。

2. 生成證書簽名請求（CSR）

openssl req -new -key user2.key -out user2.csr -subj "/CN=user2/O=group1/O=group2"

功能：使用私鑰 user2.key 生成一個新的證書簽名請求（CSR），並將其保存到文件 user2.csr 中。
參數說明：
- req -new：OpenSSL 的子命令，表示生成新的證書簽名請求。
- -key user2.key：指定用於生成 CSR 的私鑰文件。
- -out user2.csr：指定輸出 CSR 文件名稱為 user2.csr。
- -subj "/CN=user2/O=group1/O=group2"：指定 CSR 的主題（Subject），其中：
  - CN=user2：Common Name（通用名稱），在 Kubernetes 中用作用戶名稱。
  - O=group1 和 O=group2：Organization（組織），在 Kubernetes 中表示用戶所屬的組，用於基於角色的訪問控制（RBAC）。
作用：CSR 包含用戶的公鑰和身份信息（如用戶名和組），後續將提交給 Kubernetes 集群的證書頒發機構（CA）進行簽署。

3. 創建並提交 CSR 到 Kubernetes 集群

kubectl apply -f - <<EOF
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
 name: user2
spec:
 request: $(cat user2.csr | base64 | tr -d '\n')
 usages: ['digital signature', 'key encipherment', 'client auth']
 signerName: kubernetes.io/kube-apiserver-client
EOF

功能：通過 kubectl apply 命令創建一個 CertificateSigningRequest (CSR) 資源，並將其提交到 Kubernetes 集群。
參數說明：
- kubectl apply -f -：從標準輸入（這裡使用 heredoc <<EOF）應用資源定義。
- apiVersion: certificates.k8s.io/v1：指定 Kubernetes API 版本為 v1，適用於 Kubernetes v1.19 及以上版本。
- kind: CertificateSigningRequest：指定資源類型為 CSR。
- metadata.name: user2：CSR 的名稱為 user2。
- spec.request：包含 Base64 編碼的 CSR 內容，通過命令 (cat user2.csr | base64 | tr -d '\n') 將 CSR 文件轉換為一行 Base64 字符串。
- spec.usages：指定證書的用途，包括：
  - 'digital signature'：用於數據簽名。
  - 'key encipherment'：用於加密密鑰。
  - 'client auth'：用於客戶端身份驗證，這是 Kubernetes 用戶認證的主要用途。
- spec.signerName: kubernetes.io/kube-apiserver-client：指定簽署者為 Kubernetes 內置的 kube-apiserver-client，用於生成客戶端證書。
作用：將 CSR 提交到 Kubernetes 集群，等待管理員批准並由集群的 CA 簽署。

4. 批准 CSR

kubectl certificate approve user2

功能：批准名為 user2 的 CSR，使其可以被集群的證書簽署者簽署。
參數說明：
- kubectl certificate approve：Kubernetes 命令，用於批准指定的 CSR。
- user2：要批准的 CSR 名稱。
作用：批准後，Kubernetes 集群的 CA 會自動簽署該 CSR，生成證書並將其存儲在 CSR 資源的 status.certificate 字段中。

5. 獲取並保存簽署的證書

kubectl get csr user2 -o jsonpath='{.status.certificate}' | base64 --decode > user2.crt

功能：從 CSR 資源中提取經 CA 簽署的證書，解碼後保存到文件 user2.crt 中。
參數說明：
- kubectl get csr user2 -o jsonpath='{.status.certificate}'：從名為 user2 的 CSR 中獲取 status.certificate 字段，該字段包含 Base64 編碼的證書。
- base64 --decode：將 Base64 編碼的證書解碼為 PEM 格式。
- > user2.crt：將解碼後的證書保存到文件 user2.crt 中。
作用：獲取由 Kubernetes CA 簽署的證書，該證書與 user2.key 配對，用於認證用戶身份。

6. 設定 `kubectl` 的用戶憑證

kubectl config set-credentials user2 --client-key user2.key --client-certificate user2.crt --embed-certs

功能：在 kubectl 的配置文件（默認為 ~/.kube/config）中添加名為 user2 的用戶憑證。
參數說明：
- kubectl config set-credentials user2：創建或更新名為 user2 的用戶憑證。
- --client-key user2.key：指定私鑰文件 user2.key。
- --client-certificate user2.crt：指定證書文件 user2.crt。
- --embed-certs：將私鑰和證書的內容內嵌到配置文件中，而不是僅引用文件路徑，便於集中管理。
作用：配置 kubectl 以使用 user2 的私鑰和證書進行身份驗證。

7. 設定 `kubectl` 的上下文

kubectl config set-context user2 --cluster single --user user2

功能：創建一個名為 user2 的上下文，關聯到集群 single 和用戶 user2。
參數說明：
- kubectl config set-context user2：創建或更新名為 user2 的上下文。
- --cluster single：指定集群名稱為 single（需確保該集群已在 ~/.kube/config 中定義）。
- --user user2：指定用戶為 user2。
作用：上下文定義了如何連接特定集群並使用哪個用戶的憑證。創建後，可通過 kubectl config use-context user2 切換到此上下文。

8. 獲取節點列表

kubectl get nodes

功能：列出 Kubernetes 集群中的所有節點。
參數說明：
- kubectl get nodes：顯示集群中節點的名稱、狀態、角色等信息。
作用：這是一個測試命令，用於驗證 user2 的憑證是否正確配置並有權訪問集群。如果成功執行，表示認證和上下文配置有效；否則，可能需要檢查用戶的 RBAC 權限。

總結

上述命令完成了一個完整的用戶認證配置流程：

生成私鑰：創建用戶 user2 的 RSA 私鑰。
生成 CSR：使用私鑰生成證書簽名請求，包含用戶身份信息。
提交 CSR：將 CSR 提交到 Kubernetes 集群。
批准 CSR：管理員批准 CSR，獲取 CA 簽署的證書。
獲取證書：提取並保存簽署的證書。
配置憑證：將私鑰和證書添加到 kubectl 配置中。
配置上下文：關聯用戶和集群，創建上下文。
測試訪問：驗證認證是否成功並檢查權限。

這個過程展示了如何為 Kubernetes 集群添加新用戶並配置其認證，確保安全性（通過證書和私鑰）和靈活性（通過 RBAC 控制權限）。如果您有其他問題，歡迎隨時提出！

Install Taroko K8s on Ubuntu 20.04

Sun, 26 Jan 2025 16:45:18 +0800

1. 環境設定

# 1. 安裝與設定 podman v5.2.5
$ curl -fsSL -o podman-linux-amd64.tar.gz https://github.com/mgoltzsche/podman-static/releases/download/v5.2.5/podman-linux-amd64.tar.gz && \
tar -xzf podman-linux-amd64.tar.gz && \
sudo cp -r podman-linux-amd64/usr podman-linux-amd64/etc /
# 2. 設定 Cgroup v2
$ sudo nano /etc/default/grub
## 在 GRUB_CMDLINE_LINUX 新增參數
systemd.unified_cgroup_hierarchy=1
$ sudo update-grub
# 3. 設定使用者登入環境
$ sudo nano /etc/profile
## 在檔案最下方新增以下內容
gw=$(route -n | grep -e "^0.0.0.0 ")
export GWIF=${gw##* }
ips=$(ifconfig $GWIF | grep 'inet ')
export IP=$(echo $ips | cut -d' ' -f2 | cut -d':' -f2)
export NETID=${IP%.*}
export GW=$(route -n | grep -e '^0.0.0.0' | tr -s \ - | cut -d ' ' -f2)
export PATH="/home/bigred/tk/bin:/home/bigred/wulin/bin:/home/bigred/kind/bin:$PATH"
clear && sleep 2
echo "Welcome to Ubuntu Linux : `cat /etc/os-release | grep -w VERSION | cut -d '\"' -f 2`"
[ "$IP" != "" ] && echo "IP : $IP"
echo ""
export PS1="[\${STY#*.}]\u@\h:\w$ "
export KUBE_EDITOR="nano"
export NOW="--force --grace-period 0"
alias ping='ping -c 4 '
alias pingdup='sudo arping -D -I eth0 -c 2 '
alias dir='ls -alh '
alias poweroff='sudo poweroff; sleep 5'
alias reboot='sudo reboot; sleep 5'
alias kg='kubectl get'
alias ka='kubectl apply'
alias kd='kubectl delete'
alias kc='kubectl create'
alias ks='kubectl get pods -n kube-system'
alias docker='sudo podman'
alias pc='sudo podman system prune -a -f'
alias vms='sudo /usr/bin/vmware-toolbox-cmd disk shrink /'
alias kp='kubectl get pods -o wide -A | sed '\''s/(.*)//'\'' | tr -s '\'' '\'' | cut -d '\'' '\'' -f 1-4,7,8 | column -t'
alias kt='kubectl top'
alias dsize='sudo du -sh -- /* 2>/dev/null | sort -rh'
alias kgip="kubectl get pod --template '{{.status.podIP}}'"
alias dkimg='curl -X GET -s -u bigred:bigred http://dkreg.kube-system:5000/v2/_catalog | jq ".repositories[]"'
export KIND_EXPERIMENTAL_PROVIDER='podman kind create cluster'
[ $USER == "bigred" ] && sudo mount --make-rshared /
# 4. 設定 Rootless Podman
$ echo 'bigred:500000:65535' | sudo tee /etc/subuid
$ echo 'bigred:500000:65535' | sudo tee /etc/subgid
$ echo 'rbean:100000:65535' | sudo tee -a /etc/subuid
$ echo 'rbean:100000:65535' | sudo tee -a /etc/subgid
# 5. 關閉 swap (註解掉 /swapfile 那行)
$ sudo nano /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda5 during installation
UUID=69b47d93-2f82-40c5-864c-392f4b609b7a / ext4 errors=remount-ro 0 1
# /boot/efi was on /dev/sda1 during installation
UUID=C0A9-83BA /boot/efi vfat umask=0077 0 1
#/swapfile none swap sw 0 0
# 6. 啟用 IP Forward
$ sudo nano /etc/sysctl.conf
將 net.ipv4.ip_forward=1 取消註解
# 7. 重新開機
$ sudo reboot
# 8. 確認 OS 是否有啟用 cgroupVersion v2
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
# 9. 確認 podman cgroupVersion 版本
$ podman info | grep 'cgroupVersion'
cgroupVersion: v2

2. 建置 Taroko K8S

# 1. 下載 Taroko CDC 套件
$ wget http://www.oc99.org/zip/tk2025v1.0.zip; unzip tk2025v1.0.zip
# 2. 設定 tkllm K8S 叢集的 Gateway IP
## 修改 GWIP 變數的值，需設定與 VM 同網段的一個可用 IP
$ nano tk/conf/tkllm.conf
export TKIND="K8S"
export NTP="Internal"
# Container Configuration
export NID="172.22.72.0/24"
export NGW="172.22.72.254"
export IMG="docker.io/kindest/node:v1.32.0"
export LCTN="172.22.72.1:tkllm-control-plane:6G:2 \
172.22.72.2:tkllm-worker1:6G:2 \
172.22.72.3:tkllm-worker2:8G:2"
# K8S Configuration
export KVER="1.32.0"
export SVRIP="172.22.72.1"
export PNID="10.244.72.0/21"
export SNID="10.98.72.0/24"
# k8S Gateway Configuration
export TKGW="true"
export GWIF=""
export GWIP="10.103.105.32/24"
# 3. 建立 tkllm K8S 叢集
$ kto tkllm
[tkllm Cluster Infrastructure]
tkllm Network (Local:172.22.72.0/24) created
tkllm-control-plane container created
tkllm-worker1 container created
tkllm-worker2 container created
[tkllm K8S Cluster]
tkllm-control-plane ok
pod/kube-proxy-bmnw4 condition met
pod/canal-4ksn9 condition met
tkllm-worker1 join ok
tkllm-worker2 join ok
gVisor & crun ok
add label 'app=taroko' to tkllm-worker2
pod/metrics-server-8467fcc7b7-5q2rn condition met
Local Path Provisioner ok
pod/taroko-tkadm-56f4959c4-jvw8m condition met
tkadd-dnat.sh tkdel-dnat.sh created
tkllm-control-plane /etc/hosts ok
tkllm-worker1 /etc/hosts ok
tkllm-worker2 /etc/hosts ok
dkreg secret ok
tkllm: take office

Install AI on Taroko K8s

Sun, 26 Jan 2025 14:43:22 +0800

# 1. 查看安裝環境
$ kls tkllm
tkllm:Up (Internal)
---------------------------------------------------------
[tkllm Infrastructure]
tkllm-control-plane : 172.22.72.1
tkllm-worker1 : 172.22.72.2
tkllm-worker2 : 172.22.72.3
[tkllm Cluster Network]
Service Subnet : 10.98.72.0/24
Pod Subnet : 10.244.72.0/21
TKGW : true
Gateway Interface : ens160
Gateway IP : 10.103.25.32/24
NODE_NAME POD_CIDR
tkllm-control-plane 10.244.72.0/25
tkllm-worker1 10.244.72.128/25
tkllm-worker2 10.244.73.0/25
# 2. 安裝 Local-Path
$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.30/deploy/local-path-storage.yaml
$ kubectl get storageclass
$ kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
# 3. 安裝 helm
$ curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
# 4. 安裝 Milvus
$ kubectl create namespace ollama
$ helm repo add milvus https://zilliztech.github.io/milvus-helm/
$ helm repo update
$ helm upgrade --install milvus -n ollama \
--set cluster.enabled=false \
--set etcd.replicaCount=1 \
--set pulsar.enabled=false \
--set etcd.clusterDomain=tkllm.k8s \
--set minio.mode=standalone milvus/milvus
# 5. 檢查 Milvus 是否運作正常
$ kubectl -n ollama get pods
NAME READY STATUS RESTARTS AGE
milvus-etcd-0 1/1 Running 0 83m
milvus-minio-cd798dd6f-n8fl7 1/1 Running 0 83m
milvus-pulsarv3-bookie-0 1/1 Running 0 83m
milvus-pulsarv3-bookie-1 1/1 Running 0 83m
milvus-pulsarv3-bookie-2 1/1 Running 0 83m
milvus-pulsarv3-broker-0 1/1 Running 0 83m
milvus-pulsarv3-broker-1 1/1 Running 0 83m
milvus-pulsarv3-proxy-0 1/1 Running 0 83m
milvus-pulsarv3-proxy-1 1/1 Running 0 83m
milvus-pulsarv3-recovery-0 1/1 Running 0 83m
milvus-pulsarv3-zookeeper-0 1/1 Running 0 83m
milvus-pulsarv3-zookeeper-1 1/1 Running 0 83m
milvus-pulsarv3-zookeeper-2 1/1 Running 0 83m
milvus-standalone-848fbd9c57-mwzcm 1/1 Running 0 83m
# 6. 安裝 Open Webui 並使用 llama3 model
$ helm repo add open-webui https://helm.openwebui.com/
$ helm repo update
$ nano ollama-values.yaml
ollama:
enabled: true
fullnameOverride: open-webui-ollama
ollama:
gpu:
enabled: false
number: 1
type: nvidia
models:
pull:
- llama3.2:1b
run:
- llama3.2:1b
persistentVolume:
enabled: true
ingress:
annotations: {}
class: nginx
enabled: true
existingSecret: ''
host: ollama.example.com
tls: false
openaiBaseApiUrl: open-webui-ollama.ollama.svc.tkllm.k8s
persistence:
accessModes:
- ReadWriteOnce
annotations: {}
enabled: true
existingClaim: ''
selector: ''
size: 2Gi
storageClass: local-path
nodeSelector:
app: taroko
extraEnvVars:
- name: OPENAI_API_KEY
value: "0p3n-w3bu!"
- name: WEBUI_AUTH
value: "False"
- name: VECTOR_DB
value: "milvus"
- name: MILVUS_URI
value: http://milvus.ollama.svc.tkllm.k8s:19530
clusterDomain: tkllm.k8s
$ helm upgrade --install open-webui open-webui/open-webui -n ollama -f ollama-values.yaml
# 7. 檢查 Open-Web UI 是否運作正常
$ kubectl -n ollama get pods
NAME READY STATUS RESTARTS AGE
milvus-etcd-0 1/1 Running 0 83m
milvus-minio-cd798dd6f-n8fl7 1/1 Running 0 83m
milvus-pulsarv3-bookie-0 1/1 Running 0 83m
milvus-pulsarv3-bookie-1 1/1 Running 0 83m
milvus-pulsarv3-bookie-2 1/1 Running 0 83m
milvus-pulsarv3-broker-0 1/1 Running 0 83m
milvus-pulsarv3-broker-1 1/1 Running 0 83m
milvus-pulsarv3-proxy-0 1/1 Running 0 83m
milvus-pulsarv3-proxy-1 1/1 Running 0 83m
milvus-pulsarv3-recovery-0 1/1 Running 0 83m
milvus-pulsarv3-zookeeper-0 1/1 Running 0 83m
milvus-pulsarv3-zookeeper-1 1/1 Running 0 83m
milvus-pulsarv3-zookeeper-2 1/1 Running 0 83m
milvus-standalone-848fbd9c57-mwzcm 1/1 Running 0 83m
open-webui-0 1/1 Running 0 16m
open-webui-ollama-7ccd54cf7c-5sp74 1/1 Running 0 18m
open-webui-pipelines-6445445d74-n8l9j 1/1 Running 0 18m
# 8. 將 Open-webui 對外開在 Taroko Gateway 的 30888 Port 上
$ kubectl -n ollama edit sts open-webui
...
spec:
containers:
name: open-webui
ports:
- containerPort: 8080
hostPort: 30888 # 新增此行
name: http
protocol: TCP
# 9. 再次檢查 Open-webui Pod 是否運作正常
$ kubectl -n ollama get pods
NAME READY STATUS RESTARTS AGE
milvus-etcd-0 1/1 Running 0 87m
milvus-minio-cd798dd6f-n8fl7 1/1 Running 0 87m
milvus-pulsarv3-bookie-0 1/1 Running 0 87m
milvus-pulsarv3-bookie-1 1/1 Running 0 87m
milvus-pulsarv3-bookie-2 1/1 Running 0 87m
milvus-pulsarv3-broker-0 1/1 Running 0 87m
milvus-pulsarv3-broker-1 1/1 Running 0 87m
milvus-pulsarv3-proxy-0 1/1 Running 0 87m
milvus-pulsarv3-proxy-1 1/1 Running 0 87m
milvus-pulsarv3-recovery-0 1/1 Running 0 87m
milvus-pulsarv3-zookeeper-0 1/1 Running 0 87m
milvus-pulsarv3-zookeeper-1 1/1 Running 0 87m
milvus-pulsarv3-zookeeper-2 1/1 Running 0 87m
milvus-standalone-848fbd9c57-mwzcm 1/1 Running 0 87m
open-webui-0 1/1 Running 0 20m
open-webui-ollama-7ccd54cf7c-5sp74 1/1 Running 0 22m
open-webui-pipelines-6445445d74-n8l9j 1/1 Running 0 22m
# 10. 在瀏覽器存取 Open WebUI
http://192.168.38.231:30888
# 11. 查看 service
# 12. 測試用 curl API 跟 Ollama 交流
$ curl http://10.98.72.164:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.2:1b",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "1+1=?"
}
]
}'
# 13. 清除環境
$ helm -n ollama uninstall open-webui
$ helm -n ollama uninstall milvus
$ for i in $(kubectl -n ollama get pvc -o name); do kubectl -n ollama delete ${i}; done
$ kubectl delete ns ollama

Pod Priority and Preemption

Sun, 26 Jan 2025 04:15:36 +0800

Pod 可以有 priority (優先順序)。優先順序表示 Pod 相對於其他 Pod 的重要性。如果 Pod 無法排程，scheduler 會嘗試先佔用 (驅逐) 優先順序較低的 Pod，使待排程 Pod 的排程成為可能。

在並非所有使用者都受到信任的群集中，惡意使用者可能會以最高的優先順序建立 Pod，導致其他 Pod 被驅逐/無法排程。管理員可以使用 ResourceQuota 防止使用者以高優先順序建立 Pod。

How to use priority and preemption

要使用 priority 和 preemption：

新增一個或多個 PriorityClasses。
建立 Pod，並將 PriorityClassName 設定為新增的 PriorityClasses 其中的一個。當然，不需要直接建立 Pod；通常會將 priorityClassName 加入集合物件（如 Deployment）的 Pod 模版。

PriorityClass

PriorityClass 是一個 non-namespaced 的 Object，定義了從優先級名稱到優先級整數值的映射。名稱在 PriorityClass 物件元資料的 name 欄位中指定。值在必填 value 欄位中指定。值越大，優先順序越高。 PriorityClass 物件的名稱必須是有效的 DNS subdomain name，且不能以 system- 為前綴。

PriorityClass 物件可以有任何小於或等於 10 億的 32 位元整數值。這表示 PriorityClass 物件的 values 範圍為 -2147483648 至 1000000000（含 1000000000）。更大的數字保留給代表關鍵系統 Pod 的內建 PriorityClass。叢集管理員應該為他們想要的每個此類映射建立一個 PriorityClass 物件。

PriorityClass 也有兩個可選欄位：globalDefault 和 description。 globalDefault 欄位表示此 PriorityClass 的值應該用於沒有 priorityClassName 的 Pod。系統中只能存在一個 globalDefault 設為 true 的 PriorityClass。如果沒有設定 globalDefault 的 PriorityClass，則沒有 priorityClassName 的 Pod 的優先順序為零。

description 欄位是任意字串。它是用來告訴叢集使用者何時應該使用此 PriorityClass。

PodPriority 與現有叢集的注意事項

如果您升級現有的叢集，但沒有此功能，現有 Pod 的優先順序實際上為零。
增加 GlobalDefault 設為 true 的 PriorityClass 不會改變現有 Pod 的優先順序。
如果刪除 PriorityClass，使用已刪除 PriorityClass 名稱的現有 Pod 將維持不變，但無法建立更多使用已刪除 PriorityClass 名稱的 Pod。

Example PriorityClass

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for XYZ service pods only."

Non-preempting PriorityClass

具有 preemptionPolicy： Never 的 Pod 會排在較低優先順序的 Pod 之前進入排程佇列，但它們無法優先排程其他 Pod。等待排程的非優先 Pod 會留在排程佇列中，直到有足夠的可用資源，可以進行排程為止。非優先排程的 Pod 和其他 Pod 一樣，都會受到排程器後關的限制。這表示如果排程器嘗試這些 Pod 但無法排程，它們會以較低的頻率重新嘗試，讓其他優先順序較低的 Pod 在它們之前排程。

preemptionPolicy 預設為 PreemptLowerPriority，這會允許該 PriorityClass 的 Pod 搶先佔用較低優先順序的 Pod (現有的預設行為)。如果 preemptionPolicy 設定為 Never，則該 PriorityClass 中的 Pod 將不會優先處理。

資料科學工作負載的使用範例。使用者可能會提交一項工作，希望其優先順序高於其他 workloads，但不希望因為搶先執行 Pod 而丟棄現有工作。具有 preemptionPolicy： Never 的高優先順序工作，只要有足夠的叢集資源「自然地」空出來，就會排在其他已排隊的 Pod 之前。

Example Non-preempting PriorityClass

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
 name: high-priority-nonpreempting
value: 1000000
preemptionPolicy: Never
globalDefault: false
description: "This priority class will not cause other pods to be preempted."

Pod priority

有了一個或多個 PriorityClass 之後，就可以建立 Pod，在規格中指定其中一個 PriorityClass 名稱。 priority admission controller 會使用 priorityClassName 欄位，並填入優先順序的整數值。如果找不到 PriorityClass，Pod 就會被拒絕。以下 YAML 是 Pod 設定的範例，使用了前面範例中建立的 PriorityClass。優先順序允許控制器會檢查規格，並將 Pod 的優先順序解析為 1000000。

apiVersion: v1
kind: Pod
metadata:
 name: nginx
 labels:
 env: test
spec:
 containers:
 - name: nginx
 image: nginx
 imagePullPolicy: IfNotPresent
 priorityClassName: high-priority

OPA Gatekeeper for Kubernetes Admission Control

Tue, 21 Jan 2025 09:39:42 +0800

甚麼是 Open Policy Agent (OPA)？和 OPA Gatekeeper 有什麼差別?

在開發服務的時候，我們會需要注意到服務的驗證 (Authentication) 和授權 (Authorization) 這兩套機制要怎麼實作，區別如下：

Authentication（驗證）：確認使用者是否存在於系統當中，比方你在登入公司系統，系統會先確認你是否是這家公司的員工。
Authorization（授權）：根據使用者的角色來授予應有的權限，成功登入公司系統後，系統會根據你的職等，決定你可以看什麼資料、做什麼事情。

而 OPA 的概念就是要把授權(Authorization) 的概念從服務中獨立出來，讓不同的服務都能夠呼叫同一個 OPA Server 做授權 (Authorization) 這件事，概念圖如下：

比方說 API Server 接收到了一個 Request，在已經確認這個 Request 是由合法的使用者發出的請求後，API Server 會再發出一個請求到 OPA Server，確認這個 Request 要做的事情和不合乎規範，若合乎系統規範，回傳成功，反之則回傳失敗。

那 OPA Gatekeeper 又是什麼？ Gatekeeper 是專為 OPA 的 Kubernetes Admission Control 使用個案所建立。OPA Gatekeeper 封裝了 OPA 的程式碼，讓 OPA 可以以 Webhook Server 的方式運作在 Kubernetes 裡面，解決了 Kubernetes 各種部署的審核問題。

開始實作

目標 : 限制 Deployment 的 Replicas 數量

# 1. 安裝 OPA Gatekeeper
$ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.18.2/deploy/gatekeeper.yaml
# 2. 建立 replicalimits 的 ConstraintTemplate
## ConstraintTemplate 定義了用於強制執行約束的政策，以及約束的一般模式。
$ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper-library/master/library/general/replicalimits/template.yaml
# 3. 設定 Replica 數量只能在 3 到 5 之間
$ cat <<EOF | kubectl apply -f -
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sReplicaLimits
metadata:
name: replica-limits
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
# 以下為限制 kubectl scale 的時候也會阻擋，如果不設定，就只有 Create 的時候會阻擋
- apiGroups: ["autoscaling"] # add
kinds: ["Scale"] # add
parameters:
ranges:
- min_replicas: 3
max_replicas: 5
EOF
# 4. 建立 test namespace
$ kubectl create ns test
namespace/test created
# 5. 建立 6 個 replicas 的 deployment
$ kubectl -n test create deployment nginx --image=nginx --replicas=6
error: failed to create deployment: admission webhook "validation.gatekeeper.sh" denied the request: [replica-limits] The provided number of replicas is not allowed for Deployment: nginx. Allowed ranges: {"ranges": [{"max_replicas": 5, "min_replicas": 3}]}
# 6. 建立符合規則的 deployment
$ kubectl -n test create deployment nginx --image=nginx --replicas=5
# 7. 檢視 pod 運作狀態
$ kubectl -n test get pod
NAME READY STATUS RESTARTS AGE
nginx-676b6c5bbc-crdj5 1/1 Running 0 5m50s
nginx-676b6c5bbc-j6dlq 1/1 Running 0 5m50s
nginx-676b6c5bbc-r42nr 1/1 Running 0 5m50s
nginx-676b6c5bbc-wkbtw 1/1 Running 0 5m50s
nginx-676b6c5bbc-z54w5 1/1 Running 0 5m50s
# 8. 測試將 nginx deployment 的 replica scale 到 999
$ kubectl -n test scale deployment nginx --replicas 999 --dry-run=server
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [replica-limits] The provided number of replicas is not allowed for Scale: nginx. Allowed ranges: {"ranges": [{"max_replicas": 5, "min_replicas": 3}]}
# 成功阻擋，收工!

參考文件

Adding Windows worker nodes

Mon, 20 Jan 2025 10:01:36 +0800

1. 開始之前

確認 Windows Server 2022 或更高版本。

打開 cmd 命令提示字元執行以下命令

systeminfo | findstr /N "." | findstr /R "^[3]:"

執行結果 :

3:作業系統名稱: Microsoft Windows Server 2025 Standard

由 kubeadm init 建立的 kubeadm 群集。

再開一個 cmd 命令提示字元，SSH 連到 Taroko 管理主機後，執行以下命令

kubectl -n kube-system get cm kubeadm-config

執行結果 :

NAME DATA AGE
kubeadm-config 1 23h

確認 Containerd 版本。

kubectl -n kube-system get nodes \
-o custom-columns="NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion"

執行結果 :

NAME VERSION RUNTIME
tkdip-control-plane v1.31.4 containerd://1.7.24
tkdip-worker1 v1.31.4 containerd://1.7.24
tkdip-worker2 v1.31.4 containerd://1.7.24

2. 開始加入 Windows worker nodes

針對每台機器執行下列步驟：

在機器上開啟 PowerShell。
確認使用管理員的身分執行

(New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent())).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)

執行結果 :

true

然後進行下列步驟。

2.1. 安裝 containerd

要安裝 containerd，首先執行以下命令：

curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/Install-Containerd.ps1

然後執行以下命令，但首先要將 CONTAINERD_VERSION 替換為 containerd 倉庫中的最新版本。版本必須沒有 v 前綴。例如，使用 1.7.22 而不是 v1.7.22

.\Install-Containerd.ps1 -ContainerDVersion 1.7.24

根據需要調整 Install-Containerd.ps1 的其他參數，例如 netAdapterName。

如果機器不支援 Hyper-V，無法承載 Hyper-V 隔離容器，請設定 skipHypervisorSupportCheck。

如果變更 Install-Containerd.ps1 可選參數 CNIBinPath 和/或 CNIConfigPath，需要使用匹配的值設定已安裝的 Windows CNI plugin。

執行結果 :

警告: Windows feature 'Containers' is not installed. Installing it now...
Success Restart Needed Exit Code Feature Result
------- -------------- --------- --------------
True Yes SuccessRest... {容器}
警告: 您必須重新啟動此伺服器才能完成安裝程序。
警告: Windows feature 'Hyper-V' is not installed. Installing it now...
True Yes SuccessRest... {Hyper-V}
警告: 您必須重新啟動此伺服器才能完成安裝程序。
警告: Windows feature 'Hyper-V-PowerShell' is not installed. Installing it now...
True Yes SuccessRest... {適用於 Windows PowerShell 的 Hyper-V 模組...
警告: 您必須重新啟動此伺服器才能完成安裝程序。
Please reboot and re-run this script.

重新開機

shutdown /r /t 1

再次執行安裝腳本

.\Install-Containerd.ps1 -ContainerDVersion 1.7.24

執行結果 :

Windows feature 'Containers' is already installed.
Windows feature 'Hyper-V' is already installed.
Windows feature 'Hyper-V-PowerShell' is already installed.
Getting ContainerD binaries
Downloading https://github.com/containerd/containerd/releases/download/v1.7.24/containerd-1.7.24-windows-amd64.tar.gz to C:\Program Files\containerd\containerd.tar.gz
x containerd.exe
x containerd-shim-runhcs-v1.exe
x containerd-stress.exe
x ctr.exe
Registering ContainerD as a service
Starting ContainerD service
Downloading https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.27.0/crictl-v1.27.0-windows-amd64.tar.gz to C:\Program Files\containerd\crictl.tar.gz
x crictl.exe
目錄: C:\Users\bigred
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2025/1/20 下午 04:17 .crictl
Done - please remember to add '--cri-socket "npipe:////./pipe/containerd-containerd"' to your kubeadm join command if your kubernetes version is below 1.25!

2.2. 安裝 kubeadm and kubelet

執行下列指令以安裝 kubeadm 和 kubelet：

curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/PrepareNode.ps1
.\PrepareNode.ps1 -KubernetesVersion v1.31.4

執行結果 :

PS C:\Users\bigred> curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/PrepareNode.ps1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 7626 100 7626 0 0 12349 0 --:--:-- --:--:-- --:--:-- 12359
PS C:\Users\bigred> .\PrepareNode.ps1 -KubernetesVersion v1.31.4
Using Kubernetes version: v1.31.4
目錄: C:\
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2025/1/20 下午 04:27 k
Downloading https://dl.k8s.io/v1.31.4/bin/windows/amd64/kubelet.exe to C:\k\kubelet.exe
Downloading https://dl.k8s.io/v1.31.4/bin/windows/amd64/kubeadm.exe to C:\k\kubeadm.exe
目錄: C:\var\log
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2025/1/20 下午 04:28 kubelet
目錄: C:\var\lib\kubelet\etc
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2025/1/20 下午 04:28 kubernetes
目錄: C:\etc\kubernetes
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2025/1/20 下午 04:28 pki
d----- 2025/1/20 下午 04:28 manifests
目錄: C:\var\lib\kubelet\etc\kubernetes
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----l 2025/1/20 下午 04:28 pki
Installing nssm
目錄: C:\Program Files
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2025/1/20 下午 04:28 nssm
Downloading https://k8stestinfrabinaries.blob.core.windows.net/nssm-mirror/nssm-2.24.zip to nssm.zip
x nssm.exe
Registering kubelet service
Service "kubelet" installed successfully!
Set parameter "AppStdout" for service "kubelet".
Set parameter "AppStderr" for service "kubelet".
Set parameter "AppRotateFiles" for service "kubelet".
Set parameter "AppRotateOnline" for service "kubelet".
Set parameter "AppRotateSeconds" for service "kubelet".
Set parameter "AppRotateBytes" for service "kubelet".
Set parameter "DependOnService" for service "kubelet".
Caption :
Description :
ElementName : kubelet
InstanceID : kubelet
CommonName :
PolicyKeywords :
Enabled : True
PolicyDecisionStrategy : 2
PolicyRoles :
ConditionListType : 3
CreationClassName : MSFT|FW|FirewallRule|kubelet
ExecutionStrategy : 2
Mandatory :
PolicyRuleName :
Priority :
RuleUsage :
SequencedActions : 3
SystemCreationClassName :
SystemName :
Action : Allow
Direction : Inbound
DisplayGroup :
DisplayName : kubelet
EdgeTraversalPolicy : Block
EnforcementStatus : NotApplicable
LocalOnlyMapping : False
LooseSourceMapping : False
Owner :
PackageFamilyName :
Platforms : {}
PolicyAppId :
PolicyStoreSource : PersistentStore
PolicyStoreSourceType : Local
PrimaryStatus : OK
Profiles : 0
RemoteDynamicKeywordAddresses : {}
RuleGroup :
Status : 已從存放區成功剖析規則。 (65536)
StatusCode : 65536
PSComputerName :
Name : kubelet
ID : kubelet
Group :
Profile : Any
Platform : {}
LSM : False
Please remember that after you have joined the node to the cluster, that you have to apply the cni daemonset/service and the kube-proxy
Also remember that for kube-proxy you have to change the its version from the name of the image in the kube-proxy.yml to that of your kubernetes version
In case you use rancher, use the following commands:
For Windows you can use the following command:
curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/kube-proxy/kube-proxy.yml
(Get-Content "kube-proxy.yml") -Replace 'VERSION', 'v1.31.4' | Set-Content "kube-proxy.yml"
For Linux, you can use the following command:
curl -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/kubeadm/kube-proxy/kube-proxy.yml
sed -i 's/VERSION/v1.31.4/g' kube-proxy.yml
In case you use flannel, use the following commands:
For Windows you can use the following command:
curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/flannel/kube-proxy/kube-proxy.yml
(Get-Content "kube-proxy.yml") -Replace 'image: (.*):(.*)-(.*)-(.*)$', 'image: $1:v1.31.4-$3-$4' | Set-Content "kube-proxy.yml"
For Linux, you can use the following command:
curl -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/flannel/kube-proxy/kube-proxy.yml
sed -i -E 's/image: (.*):(.*)-(.*)-(.*)$/image: \1:v1.31.4-\3-\4/g' kube-proxy.yml
In case you use calico, use the following commands:
For Windows you can use the following command:
curl.exe -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml
(Get-Content "kube-proxy.yml") -Replace 'image: (.*):(.*)-(.*)-(.*)$', 'image: $1:v1.31.4-$3-$4' | Set-Content "kube-proxy.yml"
For Linux, you can use the following command:
curl -LO https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml
sed -i -E 's/image: (.*):(.*)-(.*)-(.*)$/image: \1:v1.31.4-\3-\4/g' kube-proxy.yml

產生 kubeadm join 指令

# 透過 cn 變數定義叢集名稱 (ClusterName)
cn=tkdip
# 透過 join 變數定義 worker node 加入叢集所需之命令
join=$(sudo podman exec -it "${cn}"-control-plane bash -c 'kubeadm token create --print-join-command' | tr -d '\r')
# 將字串轉換為陣列（以空格分割）
arr=($join)
# 替換第三個元素 (陣列以 0 為基礎，因此索引 2 是第三個欄位)
arr[2]="${IP}:$(sudo podman port ${cn}-control-plane | grep 6443 | cut -d ":" -f 2 )"
# 從陣列重建字串
join="${arr[@]}"
# 在螢幕印出 worker node 加入叢集所需之命令
echo $join

執行結果 :

kubeadm join 192.168.11.31:32240 --token bn7c36.hwi82i6yqvvlfs7w --discovery-token-ca-cert-hash sha256:c221fb40cf07b77620115fd5544506aad1e27055c16731d9ca52fdc7bca145db

在 Windows 執行 kubeadm join 指令

kubeadm join 192.168.11.31:32240 --token bn7c36.hwi82i6yqvvlfs7w --discovery-token-ca-cert-hash sha256:c221fb40cf07b77620115fd5544506aad1e27055c16731d9ca52fdc7bca145db

執行結果 :

Setting up K3s on bonding network device

Tue, 14 Jan 2025 01:21:26 +0800

1. PreRequest

準備好一台 2 張網卡的機器

2. Setting up bonding devices with active-backup mode

Understanding the Cilium

Fri, 03 Jan 2025 11:45:10 +0800

Cilium 不僅僅是一個網路解決方案；它是一個強大、現代且極具巧思的網路和安全專案，專為 Container 時代而設計。Cilium 的核心功能是為基於微服務的應用程式提供透明、高性能的網路和安全性，幫助您克服傳統網路方法的限制。

在本章中，我們將深入淺出地向您解釋 Cilium 的基本原理，目的是讓您了解並掌握這些概念。此外，我們還將更新上一章建立的叢集，使其使用 Cilium 作為其 CNI。

Cilium 元件

Cilium Operator
- 能夠簡化在 Kubernetes 叢集上部署、更新與維護 Cilium 的流程，包含自動執行安裝、升級以及管理特定設定等工作。它透過使用 Custom Resource Definitions（CRDs）來定義 Cilium 在叢集中的各項設定，並根據這些 CRDs 提供所需的安裝、更新及設定管理能力，大幅減少人工操作的複雜度與重複性。
Cilium Agent
- 在叢集中的每個節點上執行。從高層次來看，它會透過 Kubernetes 或其他 API 接收各種設定，包括網路設定、service load-balancing、network policies，以及 visibility (可視性) 與監控需求。
- 同時會監聽 Kubernetes 的事件，以偵測 container 或 workloads 的啟動與停止，並管理作業系統核心中的 eBPF 程式，用以控制 containers 之間或 containers 對外的所有網路存取。

Cilium 與 Cilium Operator 相互協作，從叢集層級接收設定與策略，並在每個節點上進行適當的應用。

eBPF (Extended Berkeley Packet Filter) : 它使用 eBPF 這項劃時代技術，在不需修改或重新編譯作業系統核心的情況下，即可實現核心層級的程式化能力，讓開發者能更加有效地強化 security policies (安全政策)、deep packet inspection (深度檢測封包) 並實作自訂的 network functions (網路功能)。
第七層可視性方面： Cilium 在應用程式層運作，Cilium 能深入瞭解應用程式之間如何透過 API 互動，讓您可以根據應用程式層的屬性強制執行安全政策。
HTTP 和 gRPC 感知： Cilium 並不僅止於 IP 和 port；它還能感知 HTTP 和 gRPC 通訊協定，為現代 API-driven 的應用程式實現精細的安全政策。
身份感知連接性： Cilium 可確保只有經過認證與授權的服務才能進行通訊，為您的微服務架構增加額外的安全層級。
Load Balancing 與 Service Discovery：將 load balancing 與 service discovery 無縫整合至 Kubernetes 叢集，簡化應用程式的部署與擴充。
原生 Kubernetes 支援： Cilium 是 Kubernetes-native，可與您的叢集無縫整合。這表示可輕鬆部署、自動擴充，並與 Kubernetes Network Policies 原生相容。
Global Network Policies： Cilium 為多叢集和多租戶環境提供統一的 policy framework (政策框架)，讓您可以定義跨越叢集和租戶的 policies (政策)。

eBPF: Cilium 成功的關鍵要素

Cilium 的強大功能主要建立在名為 eBPF 的技術之上。Berkeley Packet Filter（簡稱 BPF，沒有，我們沒有忘記那個「e」）是一種用於觀察 Linux 作業系統的工具，已經存在好幾年了。透過在作業系統中快速且安全地執行一小段程式碼，BPF 為使用者提供了存取核心的能力。雖然 BPF 最初是為了封包過濾而設計，但它之後被改進，用於動態追蹤 Linux 核心。舉例來說，使用者現在可以撰寫一個與 cgroups 相關的程式，用來允許或禁止對 CPU、記憶體、網路頻寬以及其他程序群組等系統資源的存取。

因此，eBPF（extended BPF）是一種新的 Linux 核心技術，它允許在 Linux 中動態插入強大的安全可視性與控制邏輯。由於 eBPF 在 Linux 核心內運行，Cilium 的 security policies 可以在不修改應用程式程式碼或 container 設定的情況下套用並更新。考量到核心能監控並控制整個系統，歷來作業系統一直是實現安全性、可觀察性與網路功能的理想平台。然而，作業系統核心在系統中扮演關鍵角色，且對安全與穩定性的要求嚴格，導致其在演進上相對保守。也因此，相較於在外部實作的功能，作業系統層級的創新往往較為遲緩。

eBPF 徹底改變了這樣的情況，允許沙盒化的程式在作業系統中運行。對應用程式開發團隊而言，eBPF 程式是一種在 runtime (執行期間) 動態新增功能的實用工具。類似於原生編譯，作業系統透過及時 (JIT) 編譯器與驗證引擎，確保這些程式能在安全且高效的環境下執行。

認識與實作 Kubernetes Gateway API

Wed, 25 Dec 2024 09:17:16 +0800

Gateway API is an official Kubernetes project focused on L4 and L7 routing in Kubernetes. This project represents the next generation of Kubernetes Ingress, Load Balancing, and Service Mesh APIs. From the outset, it has been designed to be generic, expressive, and role-oriented.

關鍵角色與人物

背景

在 Kubernetes 最初的設計中，Ingress 與 Service 這兩種資源是基於一種使用模式：建立這些 Services 和 Ingresses 的 developers，擁有對定義和公開其應用程式給使用者的所有控制權。

然而，實際上叢集和基礎設施往往是共用的，這點在原本的 Ingress 模型中並沒有被很好地呈現。關鍵在於：當基礎設施被多個使用者共用時，每個使用者的關注點並不相同。要讓基礎設施專案成功，就必須同時滿足所有使用者的需求。

這就引出了核心挑戰：如何在滿足基礎設施使用者的彈性需求的同時，仍然保有基礎設施擁有者的控制權？

為了在可用性、彈性以及控制之間取得平衡，Gateway API 透過定義數個不同的角色和對應的人物，作為討論不同使用者需求的工具。Gateway API 的設計工作特別以這些人物為基礎。

需要注意的是，實際情況中（依照環境而異），同一個人可能同時扮演多個角色，下文將有詳細討論。

主要角色與人物

Gateway API 定義了三種角色及其對應的人物：

Ian (he/him)
- 是基礎設施提供者 ( Infrastructure Provider )，負責維護能讓多個隔離的叢集服務多個租戶的基礎設施。
- 他不受制於任何單一租戶，而是關心所有租戶的整體需求。
- Ian 通常在雲端提供商（AWS、Azure、GCP …）或平台即服務（PaaS）提供商工作。
Chihiro (they/them)
- 是叢集營運者（cluster operator），負責管理叢集並確保能滿足各類使用者的需求。
- 通常會關注政策（policy）、網路存取、應用程式權限等問題。
- 同樣地，Chihiro 不替叢集的任何單一用戶服務，而是確保叢集能符合所有用戶的需要。
Ana (she/her)
- 是應用程式開發者 ( Application developer ) ，負責在叢集內建立並管理應用程式。
- 就 Gateway API 而言，Ana 需要設定諸如逾時（timeout）、請求匹配/過濾，以及如何將服務組合（例如設定路徑將流量導向後端）等。
- Ana 與其他人物不同，她主要關注的是應用程式本身要達成的業務目標，而非 Kubernetes 或 Gateway API。事實上，她可能覺得 Gateway API 和 Kubernetes 只是阻礙她完成工作的「摩擦」。

根據實際環境，不同角色可能由同一個人擔任：

若將所有角色都賦予同一位使用者，這就複刻了所謂「自行服務（self-service）」的模式，小型新創公司在裸機（bare metal）上跑 Kubernetes 時，可能就會這麼做。
比較常見的狀況是小型新創公司使用雲端提供商的叢集。此時，Ana 與 Chihiro 的角色可能由同一個人擔任，而 Ian 則是雲端提供商的員工（或是一個自動化流程）。
在更大的組織裡，我們預期上述三個人物會分別由不同的人擔任（而且可能隸屬於不同部門，互相之間直接接觸並不多）。

Resource model

在資源模型中，有三種主要物件：

GatewayClass：定義了擁有共同設定與行為的一組 Gateways。
Gateway：提出一個「流量切入點」的需求，能將外部流量轉譯至叢集中的 Service。
Routes：描述從 Gateway 進來的流量如何對應（映射）到各個 Service。

GatewayClass

GatewayClass 定義了一組具備共同設定與行為的 Gateway。每個 GatewayClass 都會由單一 controller 來處理，儘管一個 controller 也可以同時管理多個 GatewayClass。

GatewayClass 是一種叢集層級（cluster-scoped）的資源。要讓 Gateway 正常運作，系統中至少必須定義一個 GatewayClass。任何實作了 Gateway API 的 controller，都是透過提供對應的 GatewayClass 資源，讓使用者在其 Gateway 中引用。

這與 Ingress 的 IngressClass、以及 PersistentVolume 的 StorageClass 類似。在 Ingress v1beta1 中，與 GatewayClass 最相近的是 ingress-class annotation；而在 Ingress v1 中，最接近的則是 IngressClass 這個物件。

Gateway

Gateway 用來描述叢集中流量如何被轉譯到 Service。換句話說，它定義了一種要求，能把「外面不知道 Kubernetes 的地方」發過來的流量，轉譯到「能理解 Kubernetes 的地方」。例如，從雲端負載平衡器（cloud load balancer）、叢集內的代理（in-cluster proxy），或是外部硬體負載平衡器（external hardware load balancer）傳送到 Kubernetes Service 的流量。雖然很多情境中，客戶端的流量可能來自叢集「外部」，但這並不是必要條件。

Gateway 定義了一個對「特定負載平衡器設定」的請求，該設定實作了 GatewayClass 的設定與行為約定。這個資源可以由操作人員（operator）直接建立，或者由管理 GatewayClass 的 controller 來建立。

由於 Gateway 規範（spec）是為了捕捉使用者的意圖，因此它可能不包含所有屬性的完整規範。例如，使用者可能略過像是位址（addresses）、TLS 設定這類的欄位。如此一來，負責管理 GatewayClass 的控制器就能為使用者填入這些設定，讓整個規範更具可攜性。這項行為會透過 GatewayClass 的狀態（Status）物件來說明。

一個 Gateway 可以附加到一個或多個 Route references ，這些 Route 用於將部分流量導向至特定服務。

可以說成 GatewayClass 的具體實現，聲明後由 GatewayClass 的 Infrastructure Provider 提供一個具體存在的 Pod，充當了進入 Kubernetes 集群的流量的入口，負責流量接入以及往後轉發，同時還可以起到一個初步過濾的效果。

Route Resources

Route resources define protocol-specific rules for mapping requests from a Gateway to Kubernetes Services.

HTTPRoute

定義了與 HTTP 有關的規則，用於將來自 Gateway listener 的流量，對應到後端網路端點的表示方式。這些端點通常以 Service 的形式呈現。

Route summary table

Object	OSI Layer	Routing Discriminator	TLS Support	Purpose
HTTPRoute	Layer 7	Anything in the HTTP Protocol	Terminated only	HTTP and HTTPS Routing
TLSRoute	Somewhere between layer 4 and 7	SNI or other TLS properties	Passthrough or Terminated	Routing of TLS protocols including HTTPS where inspection of the HTTP stream is not required.
TCPRoute	Layer 4	destination port	Passthrough or Terminated	Allows for forwarding of a TCP stream from the Listener to the Backends
UDPRoute	Layer 4	destination port	None	Allows for forwarding of a UDP stream from the Listener to the Backends.
GRPCRoute	Layer 7	Anything in the gRPC Protocol	Terminated only	gRPC Routing over HTTP/2 and HTTP/2 cleartext

Gateway API Quick Start

Step1: Install the Gateway API CRDs and Envoy Gateway

$ kubectl apply --server-side -f https://github.com/envoyproxy/gateway/releases/download/v1.2.4/install.yaml

查看安装的 CRDs 資源

$ kubectl get crd | grep networking.k8s.io

執行結果 :

backendlbpolicies.gateway.networking.k8s.io 2024-12-22T11:11:31Z
backendtlspolicies.gateway.networking.k8s.io 2024-12-22T11:11:31Z
gatewayclasses.gateway.networking.k8s.io 2024-12-22T11:11:31Z
gateways.gateway.networking.k8s.io 2024-12-22T11:11:31Z
grpcroutes.gateway.networking.k8s.io 2024-12-22T11:11:31Z
httproutes.gateway.networking.k8s.io 2024-12-22T11:11:31Z
referencegrants.gateway.networking.k8s.io 2024-12-22T11:11:31Z
tcproutes.gateway.networking.k8s.io 2024-12-22T11:11:31Z
tlsroutes.gateway.networking.k8s.io 2024-12-22T11:11:31Z
udproutes.gateway.networking.k8s.io 2024-12-22T11:11:31Z

查看安装的 envoy controller 運作狀態是否正常

$ kubectl get pod -n envoy-gateway-system

執行結果 :

NAME READY STATUS RESTARTS AGE
envoy-gateway-67469b6f59-n4js6 1/1 Running 0 73s

Step2: Install the GatewayClass, Gateway, HTTPRoute and example app

1. 安裝 GatewayClass

$ cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
EOF

GatewayClass 是 Cluster level 的資源, GatewayClass 規格定義如下:

spec : Spec 定義了 GatewayClass 期望的狀態
- controllerName : ControllerName is the name of the controller that is managing Gateways of this class. The value of this field MUST be a domain prefixed path. 此欄位不可變更，且不能為空。
更多有關 GatewayClass 的定義請參考此連結

controllername 被定義在 envoy-gateway-system namespace 底下的 envoy-gateway-config configmap

kubectl -n envoy-gateway-system get cm envoy-gateway-config -o yaml

執行結果 :

apiVersion: v1
data:
 envoy-gateway.yaml: |
 apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: EnvoyGateway
 gateway:
 controllerName: gateway.envoyproxy.io/gatewayclass-controller
 logging:
 level:
 default: info
 provider:
 kubernetes:
 rateLimitDeployment:
 container:
 image: docker.io/envoyproxy/ratelimit:49af5cca
 patch:
 type: StrategicMerge
 value:
 spec:
 template:
 spec:
 containers:
 - imagePullPolicy: IfNotPresent
 name: envoy-ratelimit
 shutdownManager:
 image: envoyproxy/gateway:v1.2.4
... 以下省略

檢視 GatewayClass

$ kubectl get gatewayclass

執行結果 :

NAME CONTROLLER ACCEPTED AGE
eg gateway.envoyproxy.io/gatewayclass-controller True 13m

在本範例中，已實作 Gateway API 的 controller 被設定為管理 GatewayClasses，controller 名稱為 gateway.envoyproxy.io/gatewayclass-controller。此類別的 Gateway 將由實作的 controller 管理。

2. 安裝範例 Application

依序建立：

backend ServiceAccount：供權限管理使用。
backend Service：監聽 3000 tcp/port 並依據標籤 app: backend 對應到後端 Pod。
backend Deployment：啟動一個網站，並自動設定 POD_NAME、NAMESPACE 環境變數。

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
---
apiVersion: v1
kind: Service
metadata:
name: backend
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
EOF

檢視 Application 部署狀態

$ kubectl get pods,svc -l app=backend

執行結果 :

NAME READY STATUS RESTARTS AGE
pod/backend-765694d47f-4rcr6 1/1 Running 0 28s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/backend ClusterIP 10.98.32.77 <none> 3000/TCP 28s

3. 安裝 Gateway

$ cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
EOF

Gateway 規格定義如下:

Gateway represents an instance of a service-traffic handling infrastructure by binding Listeners to a set of IP addresses.
GatewayClassName : 定義此 Gateway 使用的 GatewayClass 物件的名稱。
Listeners : 定義 listener 的 hostnames、ports、protocol、termination、TLS settings，以及哪些 routes 可以附加到 listener。
- Listener 體現了 Gateway 接受網路連線的邏輯端點概念。
Addresses : 定義此 Gateway 所要求的 IP 位址，If no Addresses are specified, the implementation MAY schedule the Gateway in an implementation-specific manner, assigning an appropriate set of Addresses.

檢視 Gateway

$ kubectl get gateway

執行結果 :

NAME CLASS ADDRESS PROGRAMMED AGE
eg eg False 16m

檢查流量入口的 Pod

$ kubectl -n envoy-gateway-system get pods \
-o wide \
-l "app.kubernetes.io/component=proxy"

執行結果 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
envoy-default-eg-e41e7b31-75c864f477-4drs5 2/2 Running 0 23s 10.244.33.16 tkdev-worker1

檢查流量入口 Pod 的 Service

$ kubectl -n envoy-gateway-system get svc,endpointslices \
-l "app.kubernetes.io/component=proxy"

執行結果 :

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/envoy-default-eg-e41e7b31 LoadBalancer 10.98.32.24  80:31375/TCP 65s
NAME ADDRESSTYPE PORTS ENDPOINTS AGE
endpointslice.discovery.k8s.io/envoy-default-eg-e41e7b31-qswvj IPv4 10080 10.244.33.16 65s

可以看到 Envoy 在使用者建立 Gateway 後，會自動建立接收外部流量的 pod 跟 Service

4. 設定與安裝 HttpRoute

HTTPRoute 提供一種路由 HTTP 請求的方式。這包括根據 hostname、path、header 或 query param 來匹配請求的功能。Filters 可用於指定額外的處理步驟。Backends 指定匹配的請求應被路由到哪裡。

$ cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF

spec : 定義 HTTPRoute 的期望狀態。
- parentRefs : 用來指定一個路由（Route）想要附加的資源（通常是 Gateways）。
- hostnames : Hostnames 是用來定義一組主機名稱，用來匹配 HTTP 請求中的 Host Header，以選擇適合的 HTTPRoute 處理該請求。
  - 在進行匹配時，實作（例如 Gateway 或控制器）必須忽略 HTTP Host 標頭中包含的埠號（Port Value）。
  - 如果沒有進行任何與標頭修改相關的配置，實作必須原樣將 Host 標頭轉發給後端（Backend），不進行任何更改。
  - 如果 Listener 和 HTTPRoute 都指定了主機名稱，則 HTTPRoute 必須至少有一個相交的主機名稱才能附加到 Listener。
- rules : HTTPRouteRule 是用來定義如何處理 HTTP 請求的規則，主要分為以下三個部分：
  1. 匹配條件 (matches)：
    根據請求的特定條件（例如 URL 路徑、方法、標頭等）來判斷是否符合規則。
    - path：根據 URL 路徑做為的匹配條件。
    - type：匹配類型，這裡是 PathPrefix，表示路徑前綴匹配。
    - value：路徑值，這裡是 /，表示所有以 / 開頭的路徑都會匹配此規則。
  2. 處理方式（filters）：
    指定如何處理這些匹配的請求，比如添加、修改或刪除標頭，執行重新導向等操作。
  3. 轉發目標（backendRefs）：
    將處理後的請求轉發給指定的 API 物件，例如後端服務或其他資源。
    - group：API 群組，這裡是空字串，表示 Core API 群組。
    - kind：資源類型，這裡是 Service，表示 Kubernetes Service。
    - name：Service 的名稱，這裡是 backend。
    - port：服務的 port number，這裡是 3000。
    - weight：權重，這裡是 1，在多個後端之間進行流量分配時使用。

查看安装的 httproute 資源

$ kubectl get httproute

執行結果 :

NAME HOSTNAMES AGE
backend ["www.example.com"] 9s

5. 測試連線並檢視請求流程

如果 K8s 有安裝 MetalLB，則可跳過 5.1 和 5.2 步驟，直接執行以下命令

取得 Envoy Service 的 external IP，請執行：

$ export GATEWAY_HOST=$(kubectl get gateway/eg \
-o jsonpath='{.status.addresses[0].value}')

透過 Envoy proxy 連線範例應用 Application

$ curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get

5.1. 取得範例 Gateway 所建立的 Envoy Service 的名稱：

就是 Envoy 在使用者建立 Gateway 後，會自動建立接收外部流量的 pod 跟 Service，先把這個 Service 的名稱設為變數

$ export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')

5.2. 將 Envoy Service Port forward 出去：

$ kubectl -n envoy-gateway-system port-forward \
--address 192.168.61.128 \
service/${ENVOY_SERVICE} 8888:80 &

--address， Addresses to listen on (comma separated)，192.168.61.128 為 tkcdc 主機的 IP

執行結果 :

[1] 32937
Forwarding from 192.168.61.128:8888 -> 10080
Handling connection for 8888

5.3. 透過 Envoy proxy 連線範例應用 App：

在 Taroko K8s 叢集外部執行以下命令 :

$ curl --verbose --header "Host: www.example.com" http://192.168.61.128:8888/get

執行結果 :

* Trying 192.168.61.128:8888...
* connect to 192.168.61.128 port 8888 from 0.0.0.0 port 9362 failed: Connection refused
* Failed to connect to 192.168.61.128 port 8888 after 2037 ms: Could not connect to server
* closing connection #0
curl: (7) Failed to connect to 192.168.61.128 port 8888 after 2037 ms: Could not connect to server
C:\Users\antony>curl --verbose --header "Host: www.example.com" http://192.168.61.128:8888/get
* Trying 192.168.61.128:8888...
* Connected to 192.168.61.128 (192.168.61.128) port 8888
* using HTTP/1.x
> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/8.10.1
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Wed, 25 Dec 2024 02:17:01 GMT
< content-length: 456
<
{
"path": "/get",
"host": "www.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.10.1"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"10.244.33.9"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"1788f941-4043-42b7-b7ea-893bf3f4a449"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-765694d47f-tv9ft"
}* Connection #0 to host 192.168.61.128 left intact

5.4. 檢視請求流程

在此範例中，當 Gateway 作為反向代理時，請求流程如下：

客戶端開始準備一個指向 http://www.example.com 的 HTTP 請求。
客戶端的 DNS 解析器查詢目標名稱，並獲取與 Gateway 關聯的一個或多個 IP 位址。
客戶端向 Gateway 的 IP 位址發送請求；反向代理接收該 HTTP 請求，並使用 Host: 標頭來匹配從 Gateway 和附加的 HTTPRoute 衍生的配置。
（可選）反向代理可以根據 HTTPRoute 的匹配規則，對請求的標頭和/或路徑進行匹配。
（可選）反向代理可以修改該請求；例如，根據 HTTPRoute 的過濾規則添加或刪除標頭。
最後，反向代理將該請求轉發給一個或多個後端服務。

6. 清除環境

移除 Qucik Start 中的所有內容 :

$ kubectl delete -f \
https://github.com/envoyproxy/gateway/releases/download/v1.2.4/quickstart.yaml \
--ignore-not-found=true

刪除 Gateway API CRD 和 Envoy Gateway：

$ kubectl delete -f \
https://github.com/envoyproxy/gateway/releases/download/v1.2.4/install.yaml \
--ignore-not-found=true

參考文件

理解 K8s 如何處理不同類型 service 的封包來源 IP

Wed, 04 Dec 2024 07:08:12 +0800

在 Kubernetes 叢集中執行的應用程式會透過服務抽象（Service abstraction）來尋找彼此並與外界溝通。本文件將說明傳送至不同類型服務的封包的來源 IP 會如何處理，以及您如何根據需求切換此行為。

在 Kubernetes 叢集中運行的應用程式，透過 Service（服務）的抽象概念來找到彼此並進行溝通，也可以和外部世界連接。這份文件將解釋，當封包發送到不同類型的 Service 時，來源 IP（Source IP）會發生什麼變化，以及如何根據需求調整這種行為。

術語

NAT 網路位址轉換

Source NAT 取代封包上的來源 IP；在本頁中，通常是指取代為 node 的 IP 位址。

Destination NAT 取代封包上的目的地 IP；在本頁中，這通常是指取代為 Pod 的 IP 位址

VIP 虛擬 IP 位址，例如指定給 Kubernetes 中每個 Service 的虛擬 IP 位址

kube-proxy

一個網路背景程序，負責在每個 node 上協調管理 Service VIP（虛擬 IP）。

目標

透過各種不同類型的 Service 將簡單應用程式對外公開
瞭解每種 Service 類型如何處理來源 IP 的 NAT（網路地址轉換）
瞭解在保留來源 IP 時涉及的權衡取捨

建立測試的應用程式

$ kubectl create deployment source-ip-app --image=registry.k8s.io/echoserver:1.10

執行結果 :　

deployment.apps/source-ip-app created

Source IP for Services with Type=ClusterIP

如果您在叢集中運行 kube-proxy 並使用預設的 iptables mode，從叢集內部發送到 ClusterIP 的封包，不會進行來源 NAT（Source NAT）。您可以透過查詢 kube-proxy 所在節點上的網址 http://localhost:10249/proxyMode，來確認 kube-proxy 的運行模式。

$ curl http://localhost:10249/proxyMode

執行結果 :

ipvs

如果使用 Cilium 取代 kube-proxy，則此命令將不適用。

您可以透過為來源 IP 測試應用程式建立一個 Service，來測試來源 IP 的保留功能 :

$ kubectl expose deployment source-ip-app --name=clusterip --port=80 --target-port=8080

執行結果 :

service/clusterip exposed

查看 Service 資訊

$ kubectl get svc clusterip

執行結果 :

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
clusterip ClusterIP 172.100.53.4 <none> 80/TCP 28s

在同一個 K8s 叢集中，從某個 Pod 發送請求到 ClusterIP 的 Service

$ kubectl run busybox -it --image=busybox:1.28 --restart=Never --rm

輸出與此類似：

If you don't see a command prompt, try pressing enter.
/ #

然後，您可以在 Pod 內執行指令：

# Run this inside the terminal from "kubectl run"
ip addr

執行結果 :

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
3: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1480 qdisc noqueue qlen 1000
link/ether 2e:2b:01:66:08:1b brd ff:ff:ff:ff:ff:ff
inet 172.99.236.23/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::2c2b:1ff:fe66:81b/64 scope link
valid_lft forever preferred_lft forever

…然後使用 wget 查詢本機 webserver

# Replace "172.100.53.4" with the IPv4 address of the Service named "clusterip"
wget -qO - 172.100.53.4

執行結果 :

...
Request Information:
client_address=172.99.236.23
method=GET
...

無論客戶端 Pod 和 server 端 Pod 是否位於同一個節點，client_address（客戶端位址）始終會顯示為客戶端 Pod 的 IP 地址。這表示，當您從叢集中的一個 Pod 向 ClusterIP 發送請求時，來源 IP 並未被修改。

在 node 使用 wget 查詢本機 webserver

curl 172.100.53.4

執行結果 :

...
Request Information:
client_address=172.20.1.11
method=GET
...

Source IP for Services with Type=NodePort

傳送至 Type=NodePort 服務的封包預設為來源 NAT。您可以建立一個 NodePort 服務來測試：

$ kubectl expose deployment source-ip-app --name=nodeport --port=80 --target-port=8080 --type=NodePort

執行結果 :

service/nodeport exposed

現在您可以嘗試從群集外部透過上述分配的節點連接埠連線到服務。

$ NODEPORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services nodeport)
$ NODES=$(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }')
$ for node in $NODES; do curl -s $node:$NODEPORT | grep -i client_address; done

輸出類似於：

client_address=172.20.22.11

請注意，這些不是正確的用戶端 IP，而是群集內部 IP。這就是所發生的事：

用戶端傳送封包到 node2:nodePort
節點 2 以自己的 IP 位址取代封包中的來源 IP 位址 (SNAT)
節點 2 以 pod IP 取代封包上的目的地 IP
封包會傳送到節點 1，然後傳送到端點
pod 的回覆會傳回到節點 2
pod 的回覆會傳回到用戶端

為了避免這種情況，Kubernetes 有一個保留 Client 端 source IP 的功能。如果您將 service.spec.externalTrafficPolicy 設定為 Local 值，kube-proxy 只會 proxies proxy requests to local endpoints，而不會將流量轉送至其他節點。此方法可保留原始來源 IP 位址。如果沒有 local endpoints，傳送到節點的封包會被丟棄，因此您可以在任何封包處理規則中依賴正確的原始 IP 位址，這些規則可能會應用於傳送到端點的封包。

設定 service.spec.externalTrafficPolicy 欄位如下：

$ kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'

輸出類似於：

service/nodeport patched

現在，重新執行測試：

$ for node in $NODES; do curl --connect-timeout 1 -s $node:$NODEPORT | grep -i client_address; done

輸出類似於：

client_address=172.20.22.11

請注意，您只從端點 Pod 執行的節點得到一個回覆，並且是正確的 Client IP。

這就是發生的事情：

客戶端傳送封包到 node2:nodePort，而 node2:nodePort 沒有任何 endpoints
封包被丟棄
用戶端傳送封包到 node1:nodePort，而 node1:nodePort 確實有 endpoints
node1 路由封包到具有正確來源 IP 的端點

Source IP for Services with Type=LoadBalancer

傳送至 Type=LoadBalancer 的服務的封包，預設會進行來源 NAT，因為所有處於 Ready 狀態的可排程 Kubernetes 節點，都符合負載平衡流量的資格。因此，如果封包到達沒有端點的節點，系統會將其代理到有端點的節點，並以節點的 IP 取代封包上的來源 IP (如上一節所述)。

您可以透過負載平衡器暴露來源 IP 應用程式來測試：

$ kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer

輸出為 :

service/loadbalancer exposed

列印服務的 IP 位址：

$ kubectl get svc loadbalancer

輸出與此類似：

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
loadbalancer LoadBalancer 10.43.205.52 172.20.22.199 80:30729/TCP 8m24s

接下來，傳送一個要求到此服務的 external-ip：

curl 172.20.22.199

輸出與此類似：

client_address=172.20.22.11

但是，如果您在 Google Kubernetes Engine/GCE 上執行，將相同的 service.spec.externalTrafficPolicy 欄位設定為 Local，就會強制沒有 Service 端點的節點，透過故意使健康檢查失敗的方式，將自己從符合負載平衡流量資格的節點清單中移除。

您可以透過設定註解來測試：

$ kubectl patch svc loadbalancer -p '{"spec":{"externalTrafficPolicy":"Local"}}'

您應該會立即看到 Kubernetes 分配的 service.spec.healthCheckNodePort 欄位：

$ kubectl get svc loadbalancer -o yaml | grep -i healthCheckNodePort

輸出與此類似：

 healthCheckNodePort: 30865

service.spec.healthCheckNodePort 欄位會指向每個節點上為 /healthz 提供健康檢查服務的連接埠。您可以測試這一點：

$ kubectl get pod -o wide -l app=source-ip-app

輸出與此類似：

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
source-ip-app-75c58c94c6-m57pb 1/1 Running 0 99m 10.42.0.132 antony-rancher <none> <none>

使用 curl 在不同節點上取得 /healthz endpoint ：

# Run this locally on a node you choose
$ curl localhost:30865/healthz

輸出與此類似：

{
"service": {
"namespace": "default",
"name": "loadbalancer"
},
"localEndpoints": 1,
"serviceProxyHealthy": true
}

在不同的節點上，您可能會得到不同的結果：

# Run this locally on a node you choose
$ curl localhost:30865/healthz

輸出與此類似：

No Service Endpoints Found

Cleaning up

# Delete the Services:
$ kubectl delete svc -l app=source-ip-app
# Delete the Deployment, ReplicaSet and Pod:
$ kubectl delete deployment source-ip-app

local-path-provisioner sharedFileSystemPath example

Sun, 01 Dec 2024 10:41:14 +0800

sharedFileSystemPath 定義

sharedFileSystemPath 參數允許 provisioner 使用一個同時掛載在所有節點上的共享文件系統。這意味著所有節點都可以同時存取該文件系統。在這種情況下，所有的存取模式（Access Modes）都被支援，包括：

ReadWriteOnce：單個節點可讀寫
ReadOnlyMany：多個節點可讀取
ReadWriteMany：多個節點可讀寫

這些模式可用於存儲請求（Storage Claims），滿足不同的應用需求。

需要特別注意的是，nodePathMap 和 sharedFileSystemPath 是互斥的，不能同時使用。如果你使用了 sharedFileSystemPath，那麼必須將 nodePathMap 設置為空陣列 []。預設情況下，sharedFileSystemPath 的值為空字串，如下所示：

sharedFileSystemPath: ""

總結來說，使用 sharedFileSystemPath 可以方便地在多個節點間共享存儲，但需要確保配置正確，避免與 nodePathMap 衝突。

設定 sharedFileSystemPath

K3s single node cluster 新增 node-externel-ip

Wed, 13 Nov 2024 02:56:52 +0800

Step1: 設定 node 的 external ip

$ sudo nano /etc/rancher/k3s/config.yaml

要新增的內容如下 :

node-external-ip: "172.20.50.150"

注意 !!! 一定要確認沒有設定 disable-cloud-controller: "true"，如果有設定，請將 true 改成 false。因為 K3s 內建提供的 Cloud Controller Manager (CCM) 會根據 --node-ip 和 --node-external-ip 標籤設定節點 InternalIP 及 ExternalIP address 欄位。

Step2: 重啟 k3s 服務

$ sudo systemctl restart k3s.service

Step3: 檢查是否符合預期

$ kubectl get nodes -o wide

執行結果 :

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE
KERNEL-VERSION CONTAINER-RUNTIME
k3s Ready control-plane,etcd,master 24h v1.31.2+k3s1 172.20.1.12 172.20.50.150 SUSE Linux Enterprise Micro 5.5 5.14.21-150500.55.19-default containerd://1.7.22-k3s1

Step4: 檢查所有 Pods 運作都正常

$ kubectl get pods -A

執行結果 :

NAMESPACE NAME READY STATUS RESTARTS AGE
default web 1/1 Running 1 (13h ago) 24h
kube-system calico-kube-controllers-7c4bfcd748-zlzrg 1/1 Running 188 (44m ago) 24h
kube-system calico-node-z6xkr 1/1 Running 1 (13h ago) 24h
kube-system coredns-7975c6f78-bgmcr 1/1 Running 1 (13h ago) 24h
kube-system local-path-provisioner-7f796c67dc-bwnz8 1/1 Running 1 (13h ago) 24h
kube-system metrics-server-cdc9988b4-c4p2m 1/1 Running 1 (13h ago) 24h

REF

Deploying an External Cloud Controller Manager - K3s Docs

k8s ingress

Sun, 27 Oct 2024 04:58:10 +0800

The Ingress resource

一個最小的 Ingress 資源範例：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: minimal-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx-example
rules:
- http:
paths:
- path: /testpath
pathType: Prefix
backend:
service:
name: test
port:
number: 80

一個 Ingress 需要有 apiVersion、kind、metadata 和 spec 這些欄位。Ingress 物件的名稱必須是有效的 DNS Subdomain Names。Ingress 通常會使用 annotations 來設定一些選項，這取決於所使用的 Ingress Controller，例如 rewrite-target annotation。不同的 Ingress controllers 支持不同的 annotations，建議查閱你選擇的 Ingress controller 文件來了解支援的 annotations 有哪些。

Ingress spec 包含了設定 load balancer 或 proxy server 所需的所有資訊。最重要的是，它包含了一組規則，用來匹配所有進入的請求。Ingress 資源只支援 HTTP(S) 流量的規則。

如果沒有指定 ingressClassName，應該定義一個 default Ingress class。

有一些 Ingress controllers 在沒有定義預設 IngressClass 的情況下也能工作，例如 Ingress-NGINX controller 可以通過 --watch-ingress-without-class 這個標誌來設定。不過，建議還是像下面這樣指定預設的 IngressClass。

Kubernetes Pods Disruptions

Sat, 26 Oct 2024 08:54:50 +0800

此頁面介紹如何限制您的應用程式所經歷的同時中斷次數，從而在允許叢集管理員管理叢集節點的同時提高應用程式的可用性。

Voluntary and involuntary disruptions

Pod 不會自行消失，除非有人（例如某人或控制器）將其銷毀，或者發生了無法避免的硬體或系統軟體錯誤。

我們稱這些無法避免的情況為應用程式的非自願中斷，以下是一些例子：

支援節點的實體機器發生硬體故障
叢集管理員誤刪了虛擬機 (instance)
雲提供商或虛擬化管理程式故障使虛擬機消失
核心崩潰（kernel panic）
因叢集網路分區，節點從叢集中消失
因節點資源耗盡而逐出 Pod

除了資源耗盡的情況外，這些情況對大多數用戶來說都應該是熟悉的，它們並不是 Kubernetes 特有的。

我們將其他情況稱為自願中斷，這包括由應用程式所有者或叢集管理員發起的動作。典型的應用程式所有者行為包括：

刪除管理 Pod 的 deployment 或其他 controller
更新 deployment’s Pod template 導致重啟
直接刪除 Pod（例如誤操作）

叢集管理員的行為包括：

排空節點以進行維修或升級
從叢集中排空節點以縮小叢集規模（了解 Cluster Autoscaling）
將 Pod 從節點上移除以便讓其他 workload 適配該節點

這些動作可能由叢集管理員直接執行，或由叢集管理員運行的自動化工具執行，或由您的叢集託管提供商執行。

詢問您的叢集管理員，或查閱雲提供商或發行版的文檔，確定您的叢集是否啟用了任何自願中斷的來源。如果未啟用，您可以跳過創建 Pod 中斷預算（Pod Disruption Budgets）。

Note:

並非所有的自願中斷都受 Pod Disruption Budgets 的限制。例如，刪除 deployments 或 Pod 將繞過 Pod 中斷預算的限制。

Dealing with disruptions

https://kubernetes.io/docs/concepts/workloads/pods/disruptions/

Specifying a Disruption Budget for your Application

Sat, 26 Oct 2024 08:38:36 +0800

Debugging using a copy of the Pod

Sat, 26 Oct 2024 06:52:54 +0800

有時候，Pod 的設定選項會使某些情況下的排查變得困難。例如，如果 container image 不包含 shell，或者應用程式在啟動時崩潰，就無法使用 kubectl exec 來排查 container。在這些情況下，可以使用 kubectl debug 創建一個 Pod 的副本，並更改設定值以便進行除錯。

Copying a Pod while adding a new container

當您的應用程式正在運行但行為不符合預期時，添加一個新 Container 可能會很有用，因為這樣可以向 Pod 添加額外的除錯工具。

例如，您的應用程式 container images 可能基於 busybox 構建，但您需要一些 busybox 中未包含的除錯工具。您可以使用 kubectl run 模擬這種情況：

$ kubectl run myapp --image=busybox:1.28 --restart=Never -- sleep 1d

運行此命令以創建一個名為 myapp-debug 的 myapp 副本，並添加一個新的 Ubuntu container 進行除錯：

$ kubectl debug myapp -it --image=ubuntu --share-processes --copy-to=myapp-debug

執行結果 :

Defaulting debug container name to debugger-bzgnv.
If you don't see a command prompt, try pressing enter.
root@myapp-debug:/#

如果沒有使用 --container 標誌選擇 container 名稱，kubectl debug 會自動生成一個 container 名稱。
myapp，要排除故障的 Pod 名稱。
-i 標誌使 kubectl debug 默認連接到新 container。您可以通過指定 --attach=false 來防止這種情況發生。如果會話被斷開連接，可以使用 kubectl attach 重新連接。
-t，為 container 分配一個虛擬終端（TTY），使您能夠與 container交互。
--image=''， debug container 使用的 Container image.
--share-processes 允許這個 Pod 中的 container 查看其他 container 中的 processes。要了解更多有關其工作原理的信息，請參閱 Share Process Namespace between Containers in a Pod。
--copy-to=''，將原始 Pod myapp 複製為一個新的 Pod，名稱為 myapp-debug。

使用完畢後，別忘了清理 debugging Pod：

$ kubectl delete pod myapp myapp-debug

Copying a Pod while changing its command

有時候更改 container 的命令是很有用的，例如添加 debugging flag 或應用程式崩潰時。

要模擬應用程式崩潰，使用 kubectl run 創建一個立即退出的 container ：

$ kubectl run --image=busybox:1.28 myapp -- false

您可以使用 kubectl describe pod myapp 看到這個 container 正在崩潰：

Containers:
myapp:
Image: busybox:1.28
...
Args:
false
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1

您可以使用 kubectl debug 創建這個 Pod 的副本，並將命令更改為交互式 shell：

$ kubectl debug myapp -it --copy-to=myapp-debug --container=myapp -- sh

執行結果 :

If you don't see a command prompt, try pressing enter.
/ #

現在您有一個交互式的 shell，可以用來執行檢查文件系統路徑或手動執行 container command 等任務。

若要更改特定 container 預設執行的命令，您必須使用 --container 指定其名稱，否則 kubectl debug 會創建一個新的 container 來執行您指定的命令。
-i 標誌使 kubectl debug 默認連接到 container。您可以通過指定 --attach=false 來防止這種情況。如果會話被斷開連接，可以使用 kubectl attach 重新連接。

使用完畢後，別忘了清理 debugging Pod：

$ kubectl delete pod myapp myapp-debug

Copying a Pod while changing container images

在某些情況下，您可能需要將出現異常行為的 Pod 從其正常的 production container images 更改為包含除錯版本或附加工具的映像。

例如，使用 kubectl run 創建一個 Pod：

$ kubectl run myapp --image=busybox:1.28 --restart=Never -- sleep 1d

現在使用 kubectl debug 創建副本並將其 container image 更改為 Ubuntu：

$ kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu

--set-image 的語法使用與 kubectl set image 相同的 container_name=image 語法。*=ubuntu 表示將所有 containers 的 image 更改為 Ubuntu。

完成後，別忘了清理除錯用的 Pod：

$ kubectl delete pod myapp myapp-debug

Debugging Profiles

當使用 kubectl debug 透過除錯 Pod 來對節點進行除錯，或通過 ephemeral container 對 Pod 進行除錯，或者使用 Pod 副本時，可以使用 --profile 標誌對它們套用除錯設定檔。通過套用設定檔，可以設置特定的屬性，如 securityContext，以適應不同的情境。

可用的設定檔如下：

Profile	描述
`legacy`	一組與 1.22 行為向後相容的屬性
`general`	為每次除錯過程提供合理的一組通用屬性
`baseline`	一組與 PodSecurityStandard baseline policy 相容的屬性
`restricted`	一組與 PodSecurityStandard restricted policy 相容的屬性
`netadmin`	包含一組網路管理員權限的屬性
`sysadmin`	包含一組系統管理員（root）權限的屬性集

如果您未指定 --profile，則會默認使用 legacy 配置檔，但計劃在不久的將來棄用它。因此，建議使用其他配置檔，例如 general。

假設您創建一個 Pod 並對其進行除錯。首先，創建一個名為 myapp 的 Pod 作為示例：

$ kubectl run myapp --image=busybox:1.28 --restart=Never -- sleep 1d

接著，使用 ephemeral container 對該 Pod 進行除錯。如果 ephemeral container 需要具有權限，您可以使用 sysadmin 設定檔：

$ kubectl debug -it myapp --image=busybox:1.28 --target=myapp --profile=sysadmin

執行結果 :

Targeting container "myapp". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-zdxhp.
If you don't see a command prompt, try pressing enter.
/ #

在 container 內運行以下命令來檢查 ephemeral container process 的 capabilities ：

$ grep Cap /proc/$$/status

執行結果 :

CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000

這意味著通過應用 sysadmin 設定檔，container process 被授予作為 privileged container 的完整能力。查看有關 capabilities 的更多細節。

您也可以檢查該 ephemeral container 是否被創建為 privileged container：

$ kubectl get pod myapp -o jsonpath='{.spec.ephemeralContainers[0].securityContext}'

執行結果 :

{"privileged":true}

完成後，清理該 Pod：

kubectl delete pod myapp

Ref

Debugging using a copy of the Pod - Kubernetes Docs

Antony 筆記：技術原理與實務經驗 – Kubernetes

Install a kubeadm Kubernetes cluster with Cilium, disabling kube-proxy

Ubuntu 24.04

1. 每一台節點都要做環境準備

裝 K8s

設定 kubeconfig

裝 Cilium

Validate the Installation

洞悉 Cilium 如何傳遞 K8s Pod 網路封包

1. Preface

2. 環境資訊

3. 檢視單一 Pod 的網路架構

小結論

4. 檢視同一台節點內 Pods 之間網路封包如何傳遞

小總結

補充說明

5. 參考文章

CNI 運作原理：理解 Container Network Interface 如何讓跨主機的 Pod 通訊

TL;DR

目錄

目標

0. 前提條件

1. Kubernetes networking model 三條公約

2. Canal 架構快速理解

3. 環境探勘（Podman 特有步驟）

4. 部署 demo workload

5. 第一層觀察：Pod 內看到什麼

6. 第二層觀察：node container 內看到什麼

6.1 Interface 全貌

6.2 Routing 表

6.3 VXLAN forwarding database 與 neighbor

6.4 Calico Felix 的痕跡

7. 第三層觀察：抓真正的 VXLAN 封包

8. Packet flow 圖

MTU mismatch 小提醒

9. 常見驗證 checklist

9.1 node 互通

9.2 podCIDR 是否設置

9.3 flannel 寫好的 subnet

9.4 VXLAN checksum offload（只在懷疑時才下）

10. 收尾

Appendix A. 指令速查表

Appendix B. 術語表

B.1 Linux kernel networking

B.2 Encapsulation 與 overlay 網路

B.3 Kubernetes / CNI 元件

References

Kubernetes kube-controller-manager 設定參數深度解析

文件資訊

目錄

一、背景:設定檔的用途

二、參數解析

2.1 node-cidr-mask-size: "25"

字面意義

運作原理

實際影響(以 /25 為例)

與預設值的差異

什麼情境會這樣設定?

2.2 terminated-pod-gc-threshold: "500"

字面意義

運作原理

實際影響(設為 500)

與預設值的差異

什麼情境會這樣設定?

2.3 兩個參數的整體設計意圖

2.4 設定時的注意事項

三、原理:PodGC Controller 如何認定「terminated Pod」

3.1 原始碼位置

3.2 原始碼層面的 terminated 定義

3.3 用 kubectl 判斷 Pod 是否為 terminated

3.4 值得注意的原始碼細節

細節 1:GC 只會把超出的部分刪掉,不會歸零

細節 2:threshold ≤ 0 代表停用 GC

細節 3:GC 檢查週期

四、驗證實驗:用 Kind 驗證 terminated-pod-gc-threshold

4.1 實驗設計

4.2 完整實驗步驟

Step 1: 建立 Kind 叢集並掛入自訂的 controller-manager 參數

Step 2: 驗證參數有正確套用到 kube-controller-manager

Step 3: 建立 600 個會立即完成的 Job

2.1 `node-cidr-mask-size: "25"`

實際影響(以 `/25` 為例)

2.2 `terminated-pod-gc-threshold: "500"`

四、驗證實驗:用 Kind 驗證 `terminated-pod-gc-threshold`