antony@notes:~/security$ cat "Pods-status-UnexpectedAdmissionError.md"
Pods status UnexpectedAdmissionError
Pods status UnexpectedAdmissionError
Environment
- K3s:
v1.31.3+k3s1 - 單節點，此節點同時扮演 control-plane, etcd, worker 三種角色
Issue
當 deployment pod resources 中有宣告使用 gpu，在節點重新開機後，原來的 Pod 狀態會變成 UnexpectedAdmissionError 或 Unknown 或 ContainerStatusUnknown，並且會看在 Pod Event 中看到 kubelet 噴出以下訊息
Allocate failed due to no healthy devices present; cannot allocate unhealthy devices nvidia.com/gpu, which is unexpected但是過幾分鐘後，又有正常 Running 的 Pod
複現問題
- 建立測試應用
echo 'apiVersion: apps/v1
kind: Deployment
metadata:
name: gpu-deployment-sample
spec:
replicas: 1
selector:
matchLabels:
app: gpu-sample
template:
metadata:
labels:
app: gpu-sample
spec:
containers:
- name: cuda-container
image: quay.io/cloudwalker/alpine
command: ["sh", "-c", "sleep infinity"]
resources:
limits:
nvidia.com/gpu: 1' | kubectl apply -f -- 檢視 pod 運作狀態
kubectl get pods -l app=gpu-sample執行結果
NAME READY STATUS RESTARTS AGE
gpu-deployment-sample-7dd75997f5-qv9l8 1/1 Running 0 22s- 重開機
reboot- 再次檢視運作狀態
kubectl get pods -l app=gpu-sample執行結果 :
NAME READY STATUS RESTARTS AGE
gpu-deployment-sample-7dd75997f5-qv9l8 0/1 Unknown 0 46m
gpu-deployment-sample-7dd75997f5-stbjz 1/1 Running 0 42m- 檢視 Event
kubectl get events \
--field-selector involvedObject.name=gpu-deployment-sample-7dd75997f5-qv9l8 \
--sort-by='.lastTimestamp'執行結果 :
LAST SEEN TYPE REASON OBJECT MESSAGE
49m Normal Scheduled pod/gpu-deployment-sample-7dd75997f5-qv9l8 Successfully assigned default/gpu-deployment-sample-7dd75997f5-qv9l8 to testing
49m Normal Pulling pod/gpu-deployment-sample-7dd75997f5-qv9l8 Pulling image "quay.io/cloudwalker/alpine"
49m Normal Pulled pod/gpu-deployment-sample-7dd75997f5-qv9l8 Successfully pulled image "quay.io/cloudwalker/alpine" in 705ms (705ms including waiting). Image size: 3751656 bytes.
49m Normal Created pod/gpu-deployment-sample-7dd75997f5-qv9l8 Created container cuda-container
49m Normal Started pod/gpu-deployment-sample-7dd75997f5-qv9l8 Started container cuda-container
44m Warning UnexpectedAdmissionError pod/gpu-deployment-sample-7dd75997f5-qv9l8 Allocate failed due to no healthy devices present; cannot allocate unhealthy devices nvidia.com/gpu, which is unexpected
44m Warning FailedMount pod/gpu-deployment-sample-7dd75997f5-qv9l8 MountVolume.SetUp failed for volume "kube-api-access-ns48f" : object "default"/"kube-root-ca.crt" not registered- 查看 rs 狀態
kubectl get rs -l app=gpu-sample -o wide執行結果 :
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
gpu-deployment-sample-7dd75997f5 1 1 1 73m cuda-container quay.io/cloudwalker/alpine app=gpu-sample,pod-template-hash=7dd75997f5