Skip to content

antony@notes:~/misc$ cat "驗證-kubelet-serializeImagePulls-行為差異.md"

驗證 kubelet `serializeImagePulls` 行為差異

2026-04-29· misc

驗證 kubelet serializeImagePulls 行為差異

透過實際部署多個 Pod,觀察 kubelet 在 serializeImagePulls=true 與 false 兩種設定下的 image 拉取行為差異。

目錄

背景知識

兩個層級的序列化

Kubernetes 的 image pull 序列化行為其實有兩個層級:

  1. 跨 Pod 層級:由 serializeImagePulls 控制,決定不同 Pod 之間的 image 拉取是序列還是平行
  2. Pod 內部層級:kubelet 永遠會逐一處理同一個 Pod 內的 container,這個行為不受 serializeImagePulls 影響

⚠️ 重要:測試時若用「單一 Pod 多 container」,無論 serializeImagePulls 設成什麼,都會看到序列行為。必須用「多 Pod 同時部署」才能觀察到差異。

參數版本支援

  • Kubernetes 1.27+:maxParallelImagePulls 為 Alpha
  • Kubernetes 1.28+:升為 Beta
  • Kubernetes 1.32+:加入 e2e 測試

前置準備

1. 選定測試用的 worker node 

kubectl get nodes

本次測試使用節點:tk8s-worker2

2. 備份 kubelet 設定檔

SSH 到目標 worker node:

sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak

3. 準備測試用 YAML

建立檔案 multi-pod-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: pull-test-1
spec:
  nodeName: tk8s-worker2
  containers:
  - name: c
    image: nginx:1.25
    imagePullPolicy: Always
    command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: pull-test-2
spec:
  nodeName: tk8s-worker2
  containers:
  - name: c
    image: redis:7.2
    imagePullPolicy: Always
    command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: pull-test-3
spec:
  nodeName: tk8s-worker2
  containers:
  - name: c
    image: mysql:8.0
    imagePullPolicy: Always
    command: ["sleep", "3600"]
---
apiVersion: v1
kind: Pod
metadata:
  name: pull-test-4
spec:
  nodeName: tk8s-worker2
  containers:
  - name: c
    image: postgres:16
    imagePullPolicy: Always
    command: ["sleep", "3600"]

第一階段:測試 serializeImagePulls=false(平行拉取)

步驟 1:修改 kubelet 設定為 false

在 worker node 上編輯設定檔:

sudo vi /var/lib/kubelet/config.yaml

加入或修改以下欄位:

serializeImagePulls: false
maxParallelImagePulls: 5

步驟 2:重啟 kubelet 

sudo systemctl restart kubelet
sudo systemctl status kubelet

確認狀態為 active (running)。

步驟 3:驗證設定生效 

kubectl get --raw "/api/v1/nodes/tk8s-worker2/proxy/configz" \
  | jq '.kubeletconfig | {serializeImagePulls, maxParallelImagePulls}'

預期輸出:

{
  "serializeImagePulls": false,
  "maxParallelImagePulls": 5
}

步驟 4:清除節點上的舊 image

在 worker node 上執行:

sudo crictl rmi nginx:1.25 redis:7.2 mysql:8.0 postgres:16

步驟 5:開啟 events 監看視窗

另開一個 terminal:

kubectl get events --watch

步驟 6:部署 Pod 進行測試 

kubectl apply -f multi-pod-test.yaml

步驟 7:實際執行結果 

0s   Normal   Pulling   pod/pull-test-1   Pulling image "nginx:1.25"
0s   Normal   Pulling   pod/pull-test-2   Pulling image "redis:7.2"
0s   Normal   Pulling   pod/pull-test-3   Pulling image "mysql:8.0"
0s   Normal   Pulling   pod/pull-test-4   Pulling image "postgres:16"
0s   Normal   Pulled    pod/pull-test-2   Successfully pulled image "redis:7.2"     in 3.559s (3.559s including waiting). Image size: 44645040 bytes.
0s   Normal   Created   pod/pull-test-2   Container created
0s   Normal   Started   pod/pull-test-2   Container started
0s   Normal   Pulled    pod/pull-test-3   Successfully pulled image "mysql:8.0"     in 3.792s (3.792s including waiting). Image size: 233592821 bytes.
0s   Normal   Created   pod/pull-test-3   Container created
0s   Normal   Started   pod/pull-test-3   Container started
0s   Normal   Pulled    pod/pull-test-4   Successfully pulled image "postgres:16"   in 4.077s (4.077s including waiting). Image size: 160157788 bytes.
0s   Normal   Created   pod/pull-test-4   Container created
0s   Normal   Started   pod/pull-test-4   Container started
0s   Normal   Pulled    pod/pull-test-1   Successfully pulled image "nginx:1.25"    in 5.568s (5.568s including waiting). Image size: 71005258 bytes.
0s   Normal   Created   pod/pull-test-1   Container created
0s   Normal   Started   pod/pull-test-1   Container started

步驟 8:結果判讀(平行拉取的證據)

Imagepull timeincluding waiting等待時間
redis:7.23.559s3.559s0s ✅
mysql:8.03.792s3.792s0s ✅
postgres:164.077s4.077s0s ✅
nginx:1.255.568s5.568s0s ✅

判讀重點:

  • ✅ 四個 Pulling 事件同時發生(時間戳同為 0s)
  • ✅ 每個 image 的 pull time = including waiting,代表沒有任何排隊等待
  • ✅ 完成順序與啟動順序不同(redis 第一個拉完,但 nginx 第一個啟動)
  • ✅ 總時間 ≈ 最慢 image 的時間(~5.5 秒)

步驟 9:清理測試環境 

kubectl delete -f multi-pod-test.yaml

第二階段:測試 serializeImagePulls=true(序列拉取)

步驟 10:修改 kubelet 設定為 true

SSH 到 worker node,編輯設定檔:

sudo vi /var/lib/kubelet/config.yaml

修改為:

serializeImagePulls: true

⚠️ 注意:設為 true 時必須移除或註解 maxParallelImagePulls 那一行,否則 kubelet 會啟動失敗。

步驟 11:重啟 kubelet 

sudo systemctl restart kubelet
sudo systemctl status kubelet

步驟 12:驗證設定生效 

kubectl get --raw "/api/v1/nodes/tk8s-worker2/proxy/configz" \
  | jq '.kubeletconfig.serializeImagePulls'

預期輸出:

true

步驟 13:清除節點上的 image

在 worker node 上執行:

sudo crictl rmi nginx:1.25 redis:7.2 mysql:8.0 postgres:16

步驟 14:重新部署 Pod 並監看 events 

kubectl get events --watch

另開 terminal:

kubectl apply -f multi-pod-test.yaml

步驟 15:實際執行結果 

0s   Normal   Pulling   pod/pull-test-3   Pulling image "mysql:8.0"
0s   Normal   Pulling   pod/pull-test-1   Pulling image "nginx:1.25"
0s   Normal   Pulling   pod/pull-test-2   Pulling image "redis:7.2"
0s   Normal   Pulling   pod/pull-test-4   Pulling image "postgres:16"
0s   Normal   Pulled    pod/pull-test-3   Successfully pulled image "mysql:8.0"     in 3.858s (3.858s  including waiting). Image size: 233592821 bytes.
0s   Normal   Created   pod/pull-test-3   Container created
0s   Normal   Started   pod/pull-test-3   Container started
0s   Normal   Pulled    pod/pull-test-1   Successfully pulled image "nginx:1.25"    in 4.783s (8.629s  including waiting). Image size: 71005258 bytes.
0s   Normal   Created   pod/pull-test-1   Container created
0s   Normal   Started   pod/pull-test-1   Container started
0s   Normal   Pulled    pod/pull-test-2   Successfully pulled image "redis:7.2"     in 3.565s (12.188s including waiting). Image size: 44645040 bytes.
0s   Normal   Created   pod/pull-test-2   Container created
0s   Normal   Started   pod/pull-test-2   Container started
0s   Normal   Pulled    pod/pull-test-4   Successfully pulled image "postgres:16"   in 3.585s (15.751s including waiting). Image size: 160157788 bytes.
0s   Normal   Created   pod/pull-test-4   Container created
0s   Normal   Started   pod/pull-test-4   Container started

步驟 16:結果判讀(序列拉取的證據)

順序Imagepull timeincluding waiting等待時間
1mysql:8.03.858s3.858s0s
2nginx:1.254.783s8.629s3.846s ⏳
3redis:7.23.565s12.188s8.623s ⏳
4postgres:163.585s15.751s12.166s ⏳

判讀重點:

  • ⚠️ 雖然四個 Pulling 事件看起來也是同一秒(這是事件精度只到秒的觀察陷阱),但底層只有一個真的在下載
  • ✅ including waiting 時間遞增(0s → 3.8s → 8.6s → 12.2s),代表後面的 image 都在排隊
  • ✅ 等待時間 ≈ 前面所有 image 的 pull time 累加
  • ✅ 總時間 ≈ 所有 image 拉取時間總和(~15.7 秒)

步驟 17:清理測試環境 

kubectl delete -f multi-pod-test.yaml

第三階段:對照分析

兩種模式並列比較

觀察項目serializeImagePulls=falseserializeImagePulls=true
各 image 等待時間0s, 0s, 0s, 0s0s, 3.8s, 8.6s, 12.2s
pull time vs including waiting兩者相同including waiting 大於 pull time
完成順序取決於 image 大小與網速接近啟動順序
最後一個完成時間~5.5 秒~15.7 秒
總耗時特徵≈ 最慢 image 時間≈ 所有 image 時間總和
效能差異基準慢約 2.85 倍

視覺化對照

serializeImagePulls=false(平行):

時間軸: 0s ─────────── 5.5s
mysql    [████████]              ← 同時下載
nginx    [██████████]
redis    [████████]
postgres [██████████]

serializeImagePulls=true(序列):

時間軸: 0s ─────────────────────── 15.7s
mysql    [████]                              ← 先下載
nginx        [█████]                         ← 等 mysql 完
redis             [████]                     ← 再等
postgres              [████]                 ← 最後

🎯 最關鍵的單一判斷依據

看 Pulled 事件訊息中的 including waiting 數字:

  • in X (X including waiting) → 平行拉取(沒排隊)
  • in X (Y including waiting) 且 Y > X → 序列拉取(有排隊)

這個欄位就是 Kubernetes 為了幫使用者判斷序列化行為而特別設計的觀察點。

還原步驟

步驟 18:還原 kubelet 設定 

sudo cp /var/lib/kubelet/config.yaml.bak /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
sudo systemctl status kubelet

步驟 19:確認還原成功 

kubectl get --raw "/api/v1/nodes/tk8s-worker2/proxy/configz" \
  | jq '.kubeletconfig.serializeImagePulls'

注意事項

1. 測試節點選擇

選一台非生產關鍵的 worker node。重啟 kubelet 期間該節點上的 Pod 會短暫斷線。

2. kubeadm 叢集警告

本機修改的 /var/lib/kubelet/config.yaml 在執行 kubeadm upgrade 時可能會被覆蓋。正式長期套用建議改 kube-system 命名空間下的 kubelet-config ConfigMap(但會影響全部節點)。

3. Image 必須不存在於節點

每次測試前都要 crictl rmi,否則 kubelet 會跳過 pull 階段直接顯示 Container image already present,觀察不到差異。

4. maxParallelImagePulls 限制規則

  • 需要 Kubernetes 1.27+ 才支援
  • 設為 ≥ 2 時,必須搭配 serializeImagePulls: false
  • 若 serializeImagePulls: true,則 maxParallelImagePulls 必須設為 1 或不設定
  • 違反規則 kubelet 會啟動失敗

5. Pod 內 container 不受影響

kubelet 永遠逐一處理同一個 Pod 內的多個 container 的 image 拉取,這是固定設計,與 serializeImagePulls 設定無關。所以驗證時務必使用「多 Pod 同時部署」的方式。

參考資料

部署 VKS ClustersUbuntu XFCE 安裝繁體中文輸入法 & 設定左 Shift 切換中英文