antony@notes:~/openshift$ cat "修復-ocp-叢集憑證過期.md"
修復 ocp 叢集憑證過期
修復 ocp 叢集憑證過期
1. 環境資訊
- Red Hat OpenShift Container Platform (RHOCP) 4.19.10
2. 問題描述
ocp 叢集的美台節點都關機一個月以上，再次重新開機時，遇到 Control-plane 上的核心元件 (kube-apiserver, kube-scheduler…等)的憑證過期。
無法透過 oc 指令跟 kube-apiserver 溝通
oc get nodes執行結果：
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-11-21T11:47:17+08:00 is after 2025-10-17T04:33:31Z3. 問題根原因
Control Plane 憑證已經過期超過一個月，由於叢集被關機超過 2 個月以上，導致憑證自動輪替 (Rotation) 機制失效時。
kube-apiserver 的 kubelet-client 憑證已過期，API Server 無法與 Kubelet 進行 TLS 驗證。
4. 診斷步驟
列出系統當前時間
TZ=GMT date執行結果：
Fri Nov 21 03:52:02 AM GMT 2025列出在 openshift 叢集中所有過期的憑證
echo -e "NAMESPACE\tNAME\tEXPIRY_DATE" && \ oc get secrets --insecure-skip-tls-verify -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | \ while read namespace name cert; do if ! echo "$cert" | base64 -d | openssl x509 -noout -checkend 0 > /dev/null; then enddate=$(echo "$cert" | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//') echo -e "$namespace\t$name\t$enddate" fi done | column -t執行結果：
NAMESPACE NAME EXPIRY_DATE openshift-config-managed kube-controller-manager-client-cert-key Oct 17 04:33:39 2025 GMT openshift-config-managed kube-scheduler-client-cert-key Oct 17 04:33:39 2025 GMT openshift-kube-apiserver-operator aggregator-client-signer Sep 18 04:10:54 2025 GMT openshift-kube-apiserver aggregator-client Sep 18 04:10:54 2025 GMT openshift-kube-apiserver check-endpoints-client-cert-key Oct 17 04:33:39 2025 GMT openshift-kube-apiserver control-plane-node-admin-client-cert-key Oct 17 04:33:35 2025 GMT openshift-kube-apiserver external-loadbalancer-serving-certkey Oct 17 04:33:31 2025 GMT openshift-kube-apiserver internal-loadbalancer-serving-certkey Oct 17 04:33:34 2025 GMT openshift-kube-apiserver kubelet-client Oct 17 04:33:35 2025 GMT openshift-kube-apiserver localhost-serving-cert-certkey Oct 17 04:33:31 2025 GMT openshift-kube-apiserver service-network-serving-certkey Oct 17 04:33:31 2025 GMT openshift-kube-controller-manager-operator csr-signer Sep 18 04:10:55 2025 GMT openshift-kube-controller-manager-operator csr-signer-signer Sep 18 04:10:55 2025 GMT openshift-kube-controller-manager csr-signer Sep 18 04:10:55 2025 GMT openshift-kube-controller-manager kube-controller-manager-client-cert-key Oct 17 04:33:39 2025 GMT openshift-kube-scheduler kube-scheduler-client-cert-key Oct 17 04:33:39 2025 GMT openshift-operator-lifecycle-manager pprof-cert Sep 18 04:45:10 2025 GMT
4. 解決問題
觸發 OpenShift 叢集內所有 Master 和 Worker 節點的滾動式重新開機
oc adm reboot-machine-config-pool mcp/worker mcp/master \ --insecure-skip-tls-verify執行結果：
machineconfig.machineconfiguration.openshift.io/95-oc-initiated-reboot-worker rolling reboot initiated machineconfigpool.machineconfiguration.openshift.io/worker rolling reboot initiated machineconfig.machineconfiguration.openshift.io/95-oc-initiated-reboot-master rolling reboot initiated machineconfigpool.machineconfiguration.openshift.io/master rolling reboot initiated檢視被 drian 的節點
oc get nodes --insecure-skip-tls-verify執行結果：
NAME STATUS ROLES AGE VERSION master-1 Ready control-plane,master 65d v1.32.7 master-2 NotReady,SchedulingDisabled control-plane,master 65d v1.32.7 master-3 Ready control-plane,master 65d v1.32.7 worker-1 Ready worker 65d v1.32.7 worker-2 Ready worker 65d v1.32.7 worker-3 NotReady,SchedulingDisabled worker 65d v1.32.7如果節點很久都沒自動重新開機，請執行以下步驟：
檢視該節點上是否還有 pod 沒被驅逐
oc get pods --all-namespaces \ --insecure-skip-tls-verify \ --field-selector spec.nodeName=master-2執行結果：
NAMESPACE NAME READY STATUS RESTARTS AGE openshift-cluster-node-tuning-operator tuned-wq2dg 1/1 Running 0 65d openshift-dns dns-default-rk5nw 2/2 Running 0 65d openshift-dns node-resolver-6b5xt 1/1 Running 0 65d openshift-etcd etcd-master-2 5/5 Running 0 65d openshift-etcd revision-pruner-10-master-2 0/1 Pending 0 64d openshift-image-registry node-ca-hshgn 1/1 Running 0 65d openshift-kube-apiserver kube-apiserver-master-2 5/5 Running 0 65d openshift-kube-controller-manager kube-controller-manager-master-2 4/4 Running 2 (64d ago) 65d openshift-kube-scheduler openshift-kube-scheduler-master-2 3/3 Running 0 65d openshift-kube-scheduler revision-pruner-6-master-2 0/1 Pending 0 64d openshift-machine-config-operator kube-rbac-proxy-crio-master-2 1/1 Running 12 (65d ago) 65d openshift-machine-config-operator machine-config-daemon-vtgwl 2/2 Running 0 65d openshift-machine-config-operator machine-config-server-tx5vx 1/1 Running 0 65d openshift-monitoring node-exporter-vcfsd 2/2 Running 0 65d openshift-multus multus-47z8l 1/1 Running 0 65d openshift-multus multus-additional-cni-plugins-zc8vb 1/1 Running 0 65d openshift-multus network-metrics-daemon-6h8k7 2/2 Running 0 65d openshift-network-diagnostics network-check-target-f29nk 1/1 Running 0 65d openshift-network-node-identity network-node-identity-cg7rr 2/2 Running 0 65d openshift-network-operator iptables-alerter-gl975 1/1 Running 0 65d openshift-ovn-kubernetes ovnkube-node-czhkg 8/8 Running 0 65d手動 drain 節點看有沒有什麼錯誤訊息
oc adm drain master-2 --insecure-skip-tls-verify --ignore-daemonsets --delete-emptydir-data執行結果：
node/master-2 already cordoned error: unable to drain node "master-2" due to error: cannot delete cannot delete Pods that declare no controller (use --force to override): openshift-etcd/revision-pruner-10-master-2, openshift-kube-scheduler/revision-pruner-6-master-2, continuing command... There are pending nodes to be drained: master-2 cannot delete cannot delete Pods that declare no controller (use --force to override): openshift-etcd/revision-pruner-10-master-2, openshift-kube-scheduler/revision-pruner-6-master-2revision-prunerPod 是 OpenShift Operator 自動產生的清理工作 (Housekeeping)。- 它們的作用是清理舊的 Static Pod 設定檔 (Revision)。
- 它們通常是暫時性的任務。如果因為 Drain 而被刪除，Operator 會在之後的協調循環 (Reconcile loop) 中，若有需要會再次建立它們，或者它們的任務其實早就結束了。
- 它們不承載任何使用者流量或重要資料。
因此，在這種情況下，使用
--force強制將其刪除是標準操作，不會對叢集造成傷害。強制驅逐該節點的 pods
oc adm drain master-2 \ --insecure-skip-tls-verify \ --ignore-daemonsets \ --delete-emptydir-data \ --force執行結果：
node/master-2 already cordoned Warning: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-wq2dg, openshift-dns/dns-default-rk5nw, openshift-dns/node-resolver-6b5xt, openshift-image-registry/node-ca-hshgn, openshift-machine-config-operator/machine-config-daemon-vtgwl, openshift-machine-config-operator/machine-config-server-tx5vx, openshift-monitoring/node-exporter-vcfsd, openshift-multus/multus-47z8l, openshift-multus/multus-additional-cni-plugins-zc8vb, openshift-multus/network-metrics-daemon-6h8k7, openshift-network-diagnostics/network-check-target-f29nk, openshift-network-node-identity/network-node-identity-cg7rr, openshift-network-operator/iptables-alerter-gl975, openshift-ovn-kubernetes/ovnkube-node-czhkg; deleting Pods that declare no controller: openshift-etcd/revision-pruner-10-master-2, openshift-kube-scheduler/revision-pruner-6-master-2 evicting pod openshift-kube-scheduler/revision-pruner-6-master-2 evicting pod openshift-etcd/revision-pruner-10-master-2 ^C以上命令如無法成功，請直些手動強制清除該 pod
oc delete pod revision-pruner-6-master-2 -n openshift-kube-scheduler --force --grace-period=0 --insecure-skip-tls-verify oc delete pod revision-pruner-10-master-2 -n openshift-etcd --force --grace-period=0 --insecure-skip-tls-verify再驅逐一次該節點的 pods
oc adm drain master-2 \ --insecure-skip-tls-verify \ --ignore-daemonsets \ --delete-emptydir-data \ --force執行結果：
Warning: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-wq2dg, openshift-dns/dns-default-rk5nw, openshift-dns/node-resolver-6b5xt, openshift-image-registry/node-ca-hshgn, openshift-machine-config-operator/machine-config-daemon-vtgwl, openshift-machine-config-operator/machine-config-server-tx5vx, openshift-monitoring/node-exporter-vcfsd, openshift-multus/multus-47z8l, openshift-multus/multus-additional-cni-plugins-zc8vb, openshift-multus/network-metrics-daemon-6h8k7, openshift-network-diagnostics/network-check-target-f29nk, openshift-network-node-identity/network-node-identity-cg7rr, openshift-network-operator/iptables-alerter-gl975, openshift-ovn-kubernetes/ovnkube-node-czhkg node/master-2 drained