antony@notes:~/kubernetes$ cat "Quick-Start-Kube-ovn-on-Kubeadm-K8s.md"
Quick Start Kube-ovn on Kubeadm K8s
Quick Start Kube-ovn on HA Kubeadm K8s
1. 建置 K8s 前置作業
:::danger K8s 叢集中的每台節點需做以下設定 :::
對於其他 Linux 發行版，請確保
geneve、openvswitch、ip_tables、iptable_nat、overlay和br_netfilterkernel module 存在。檢查 OS Kernel 是否有 geneve, openvswitch, ip_tables, iptable_nat 模組
# 定義需要檢查的模組 MODULES="geneve|openvswitch|ip_tables|iptable_nat|overlay|br_netfilter" # 執行檢查 sudo lsmod | grep -E "$MODULES"執行結果：
ip_tables 32768 0 x_tables 65536 1 ip_tables bridge 421888 1 br_netfilter overlay 212992 0永久添加
cat <<EOF | sudo tee /etc/modules-load.d/kube-ovn.conf geneve openvswitch ip_tables iptable_nat EOF cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF現在立刻添加
sudo modprobe geneve sudo modprobe openvswitch sudo modprobe ip_tables sudo modprobe iptable_nat sudo modprobe overlay sudo modprobe br_netfilter檢查
sudo lsmod | grep -E "$MODULES"執行結果：
iptable_nat 12288 0 openvswitch 217088 0 nsh 12288 1 openvswitch nf_conncount 24576 1 openvswitch nf_nat 61440 2 openvswitch,iptable_nat nf_conntrack 196608 3 nf_nat,openvswitch,nf_conncount geneve 49152 0 ip6_udp_tunnel 16384 1 geneve udp_tunnel 32768 1 geneve br_netfilter 32768 0 bridge 421888 1 br_netfilter overlay 212992 0 ip_tables 32768 1 iptable_nat x_tables 65536 2 ip_tables,iptable_nat libcrc32c 12288 6 nf_conntrack,nf_nat,openvswitch,btrfs,nf_tables,raid456
核心啟動需要開啟 IPv6，如果核心啟動參數包含
ipv6.disable=1需要設定為0。檢查當前 kernel 參數
cat /proc/cmdline執行結果：
BOOT_IMAGE=/vmlinuz-6.8.0-60-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro ipv6.disable=1設定
ipv6.disable=0sudo nano /etc/default/grub需修改的檔案內容如下：
... GRUB_CMDLINE_LINUX="ipv6.disable=0"更新 Grub
sudo update-grub重新開機
sudo reboot確認值為
0cat /proc/sys/net/ipv6/conf/all/disable_ipv6執行結果：
0
kube-proxy 正常運作，Kube-OVN 可以透過 Service ClusterIP 存取 kube-apiserver。
確認 kubelet 設定參數開啟了 CNI，且配置在標準路徑下, kubelet 啟動時應包含下列參數：
--network-plugin=cni--cni-bin-dir=/opt/cni/bin--cni-conf-dir=/etc/cni/net.d
確認未安裝其他網路插件，或其他網路插件已被清除，檢查
/etc/cni/net.d/路徑下無其他網路插件設定檔。如果之前安裝過其他網路插件，建議刪除後重新啟動機器清理殘留網路資源。確認 swap 是否關閉
cat /proc/swaps執行結果如下
Filename Type Size Used Priority永久添加必要 Kernel Parameters
cat <<EOF | sudo tee /etc/sysctl.d/99-k8s-cri.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF立即載入 Kernel Parameters
sudo sysctl --system驗證
net.bridge.bridge-nf-call-iptables,net.bridge.bridge-nf-call-ip6tables, 和net.ipv4.ip_forward的值都設成1sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward執行結果：
net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1定義要安裝 K8s 軟體套件版本
KUBERNETES_VERSION=v1.35 CRIO_VERSION=v1.35安裝添加儲存庫的依賴項
sudo apt-get update sudo apt-get install -y software-properties-common curl添加 Kubernetes 和 CRI-O 儲存庫
curl -fsSL https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/$KUBERNETES_VERSION/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list curl -fsSL https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://download.opensuse.org/repositories/isv:/cri-o:/stable:/$CRIO_VERSION/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list安裝套件
sudo apt-get update sudo apt-get install -y cri-o kubelet kubeadm kubectl crun runc關閉 ufw Systemd Service
sudo systemctl disable --now ufw設定 kubelet service node ip
INTERFACE=ens18 IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1) sudo sed -i \ -e '/^KUBELET_EXTRA_ARGS=$/s/$/\"--node-ip='"$IPV4_IP"'\"/' \ /etc/default/kubelet檢查是否正確修改
cat /etc/default/kubelet執行結果：
KUBELET_EXTRA_ARGS="--node-ip=172.20.6.61"啟動 kubelet 服務
sudo systemctl enable --now kubelet.service安裝 chrony
sudo apt-get install -y chrony設定 chronyd 服務開機自動啟動
sudo systemctl enable chrony設定跟 Google 校時
sudo nano /etc/chrony/chrony.conf修改內容如下：
server time.google.com prefer啟用 chronyd 服務
sudo systemctl start chrony手動確定已校時
sudo chronyc sources執行結果如下：
MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* time4.google.com 1 6 377 22 +271us[ +332us] +/- 3178usSSH 連到其他節點做步驟 1 ~ 22 的設定
2. 透過 kubeadm 開始建立 HA K8s
ssh 連到第一台 Control-plane node
ssh sudo_user@ipaddderss設定 kube-vip
export VIP="172.20.6.60" export VIPSUBSET="16" export INTERFACE="ens18" KVVERSION="v1.0.1" alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION" sudo mkdir -p /etc/kubernetes/manifests/ kube-vip manifest pod \ --interface $INTERFACE \ --address $VIP \ --vipSubnet $VIPSUBSET \ --controlplane \ --arp \ --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip-static-pod.yaml拷貝到另外兩台 Control plane nodes
ssh sudo_user@control_plane_node_2_node_hostname 'sudo mkdir -p /etc/kubernetes/manifests/' ssh sudo_user@control_plane_node_3_node_hostname 'sudo mkdir -p /etc/kubernetes/manifests/' scp /etc/kubernetes/manifests/kube-vip-static-pod.yaml sudo_user@control_plane_node_2_hostname: scp /etc/kubernetes/manifests/kube-vip-static-pod.yaml sudo_user@control_plane_node_3_node_hostname: ssh sudo_user@control_plane_node_3_node_hostname 'sudo mv kube-vip-static-pod.yaml /etc/kubernetes/manifests/' ssh sudo_user@control_plane_node_3_node_hostname 'sudo mv kube-vip-static-pod.yaml /etc/kubernetes/manifests/'修改 kube-vip 的 Service Account，只需在第一台 Control-plane 需要修改
sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' \ /etc/kubernetes/manifests/kube-vip-static-pod.yaml編輯 kubeadm 初始化設定檔
mkdir -p ${HOME}/k8s/config; \ nano "$HOME"/k8s/config/kubeadm-init.yaml檔案內容如下：
apiVersion: kubeadm.k8s.io/v1beta4 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 172.20.6.61 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/crio/crio.sock imagePullPolicy: IfNotPresent imagePullSerial: true name: taroko-m1.kubeantony.com taints: [] timeouts: controlPlaneComponentHealthCheck: 4m0s discovery: 5m0s etcdAPICall: 2m0s kubeletHealthCheck: 4m0s kubernetesAPICall: 1m0s tlsBootstrap: 5m0s upgradeManifests: 5m0s --- apiServer: certSANs: - 127.0.0.1 - api-taroko.kubeantony.com extraVolumes: - name: tz-config hostPath: /etc/localtime mountPath: /etc/localtime readOnly: true apiVersion: kubeadm.k8s.io/v1beta4 caCertificateValidityPeriod: 87600h0m0s certificateValidityPeriod: 8760h0m0s certificatesDir: /etc/kubernetes/pki clusterName: taroko-ovn controllerManager: extraArgs: - name: bind-address value: "0.0.0.0" - name: secure-port value: "10257" extraVolumes: - name: tz-config hostPath: /etc/localtime mountPath: /etc/localtime readOnly: true #dns: # imageRepository: harbor.example.com/library/coredns encryptionAlgorithm: RSA-2048 etcd: local: dataDir: /var/lib/etcd extraArgs: - name: listen-metrics-urls value: http://0.0.0.0:2381 #imageRepository: harbor.example.com/library kind: ClusterConfiguration kubernetesVersion: 1.35.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/16 podSubnet: 10.244.0.0/16 #proxy: # disabled: true scheduler: extraArgs: - name: bind-address value: "0.0.0.0" - name: secure-port value: "10259" extraVolumes: - name: tz-config hostPath: /etc/localtime mountPath: /etc/localtime readOnly: true #controlPlaneEndpoint: 172.20.6.60:6443 controlPlaneEndpoint: api-taroko.kubeantony.com:6443 --- apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: cacheTTL: 0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook webhook: cacheAuthorizedTTL: 0s cacheUnauthorizedTTL: 0s cgroupDriver: systemd clusterDNS: - 10.96.0.10 clusterDomain: cluster.local containerRuntimeEndpoint: "" cpuManagerReconcilePeriod: 0s crashLoopBackOff: {} evictionPressureTransitionPeriod: 0s fileCheckFrequency: 0s healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 0s imageMaximumGCAge: "168h" imageMinimumGCAge: "2m0s" kind: KubeletConfiguration logging: flushFrequency: 0 options: json: infoBufferSize: "0" text: infoBufferSize: "0" verbosity: 0 memorySwap: {} nodeStatusReportFrequency: 0s nodeStatusUpdateFrequency: 0s rotateCertificates: true runtimeRequestTimeout: 0s shutdownGracePeriod: 30s shutdownGracePeriodCriticalPods: 10s staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 0s syncFrequency: 0s volumeStatsAggPeriod: 0s maxPods: 110 systemReserved: memory: "1Gi" kubeReserved: memory: "2Gi" --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration metricsBindAddress: "0.0.0.0:10249" mode: "nftables" nftables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 1s syncPeriod: 30sDry run set up the Kubernetes control plane，用以檢視是否設定檔有誤
sudo kubeadm init \ --dry-run \ --upload-certs \ --config="$HOME"/k8s/config/kubeadm-init.yaml正確執行結果：
... [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/tmp/kubeadm-init-dryrun1288532023/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of control-plane nodes running the following command on each as root: kubeadm join api-taroko.kubeantony.com:6443 --token tvxdmf.8yjcy5j5wgi0tkk8 \ --discovery-token-ca-cert-hash sha256:7c7ff5930966bcb4e5ef89f4008fbc5a1578fceff088cd0dcf5b67ecb6ae4c07 \ --control-plane --certificate-key 962bae62c93cbd3f706b07014d837be78b7f120e0e75adb6f82ad942f18a092a Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join api-taroko.kubeantony.com:6443 --token tvxdmf.8yjcy5j5wgi0tkk8 \ --discovery-token-ca-cert-hash sha256:7c7ff5930966bcb4e5ef89f4008fbc5a1578fceff088cd0dcf5b67ecb6ae4c07Set up the Kubernetes control plane
sudo kubeadm init \ --upload-certs \ --config="$HOME"/k8s/config/kubeadm-init.yaml正確執行結果 :
[init] Using Kubernetes version: v1.35.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [api-taroko.kubeantony.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local taroko-m1.kubeantony.com] and IPs [10.96.0.1 172.20.6.61 127.0.0.1] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost taroko-m1.kubeantony.com] and IPs [172.20.6.61 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost taroko-m1.kubeantony.com] and IPs [172.20.6.61 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "super-admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml" [patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests" [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 501.521647ms [control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s [control-plane-check] Checking kube-apiserver at https://172.20.6.61:6443/livez [control-plane-check] Checking kube-controller-manager at https://0.0.0.0:10257/healthz [control-plane-check] Checking kube-scheduler at https://0.0.0.0:10259/livez [control-plane-check] kube-controller-manager is healthy after 505.527348ms [control-plane-check] kube-scheduler is healthy after 1.008943332s [control-plane-check] kube-apiserver is healthy after 3.001586313s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace [upload-certs] Using certificate key: 064ae9995157a471491e92d05c1498092c59b389d99878b87ad08eb5c5a1231f [mark-control-plane] Marking the node taroko-m1.kubeantony.com as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [bootstrap-token] Using token: 2mjfj9.wluywejrou2isf9n [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of control-plane nodes running the following command on each as root: kubeadm join api-taroko.kubeantony.com:6443 --token 2mjfj9.wluywejrou2isf9n \ --discovery-token-ca-cert-hash sha256:a7b161d19a125ccc6efc5511a81ea5dfb4563667a8d5fd922aa5f31093609094 \ --control-plane --certificate-key 064ae9995157a471491e92d05c1498092c59b389d99878b87ad08eb5c5a1231f Please note that the certificate-key gives access to cluster sensitive data, keep it secret! As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use "kubeadm init phase upload-certs --upload-certs" to reload certs afterward. Then you can join any number of worker nodes by running the following on each as root: kubeadm join api-taroko.kubeantony.com:6443 --token 2mjfj9.wluywejrou2isf9n \ --discovery-token-ca-cert-hash sha256:a7b161d19a125ccc6efc5511a81ea5dfb4563667a8d5fd922aa5f31093609094設定 kubeconfig
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config printf "\nsource <(kubectl completion bash)\nalias k=kubectl\ncomplete -o default -F __start_kubectl k\n" >> ~/.bashrc source ~/.bashrc第一台 control plan 安裝好後再將 kube-vip 權限調整回來
sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip-static-pod.yaml sudo systemctl daemon-reload sudo systemctl restart kubelet確認 kube-system pods 運作正常
k -n kube-system get pods執行結果：
NAME READY STATUS RESTARTS AGE coredns-7d764666f9-5mnc5 1/1 Running 0 58s coredns-7d764666f9-62cqv 1/1 Running 0 58s etcd-taroko-m1.kubeantony.com 1/1 Running 0 65s kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 65s kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 67s kube-proxy-xww8m 1/1 Running 0 59s kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 66s kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 21s下載 kube-ovn CNI 安裝腳本
mkdir -p ${HOME}/k8s/addon/cni/kube-ovn wget -P ${HOME}/k8s/addon/cni/kube-ovn \ https://raw.githubusercontent.com/kubeovn/kube-ovn/refs/tags/v1.15.2/dist/images/install.sh修改設定參數
nano ${HOME}/k8s/addon/cni/kube-ovn/install.sh檔案內容：
IFACE=ens18 POD_CIDR="10.244.0.0/16" # Do NOT overlap with NODE/SVC/JOIN CIDR POD_GATEWAY="10.244.0.1" SVC_CIDR="10.96.0.0/16"執行安裝腳本
cd ${HOME}/k8s/addon/cni/kube-ovn; \ sudo KUBECONFIG="$HOME"/.kube/config bash install.sh執行結果：
------------------------------- Kube-OVN Version: v1.15.2 Default Network Mode: geneve Default Subnet CIDR: 10.244.0.0/16 Join Subnet CIDR: 100.64.0.0/16 Enable SVC LB: true Enable Networkpolicy: true Enable EIP and SNAT: true Enable Mirror: false ------------------------------- [Step 1/6] Label kube-ovn-master node and label datapath type node/taroko-m1.kubeantony.com labeled ------------------------------- [Step 2/6] Install OVN components Install OVN DB in 172.20.6.61 customresourcedefinition.apiextensions.k8s.io/vpc-dnses.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/switch-lb-rules.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/vpc-nat-gateways.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/vpc-egress-gateways.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/iptables-eips.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/iptables-fip-rules.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/iptables-dnat-rules.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/iptables-snat-rules.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/ovn-eips.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/ovn-fips.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/ovn-snat-rules.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/ovn-dnat-rules.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/vpcs.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/ips.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/vips.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/subnets.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/ippools.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/vlans.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/provider-networks.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/security-groups.kubeovn.io created customresourcedefinition.apiextensions.k8s.io/qos-policies.kubeovn.io created serviceaccount/ovn-ovs created clusterrole.rbac.authorization.k8s.io/system:ovn-ovs created clusterrolebinding.rbac.authorization.k8s.io/ovn-ovs created serviceaccount/ovn created clusterrole.rbac.authorization.k8s.io/system:ovn created clusterrolebinding.rbac.authorization.k8s.io/ovn created rolebinding.rbac.authorization.k8s.io/ovn created serviceaccount/kube-ovn-cni created clusterrole.rbac.authorization.k8s.io/system:kube-ovn-cni created role.rbac.authorization.k8s.io/secret-reader-ovn-ipsec created clusterrolebinding.rbac.authorization.k8s.io/kube-ovn-cni created rolebinding.rbac.authorization.k8s.io/kube-ovn-cni created rolebinding.rbac.authorization.k8s.io/kube-ovn-cni-secret-reader created serviceaccount/kube-ovn-app created clusterrole.rbac.authorization.k8s.io/system:kube-ovn-app created clusterrolebinding.rbac.authorization.k8s.io/kube-ovn-app created rolebinding.rbac.authorization.k8s.io/kube-ovn-app created service/ovn-nb created service/ovn-sb created service/ovn-northd created deployment.apps/ovn-central created daemonset.apps/ovs-ovn created Waiting for deployment "ovn-central" rollout to finish: 0 of 1 updated replicas are available... deployment "ovn-central" successfully rolled out Waiting for daemon set "ovs-ovn" rollout to finish: 0 of 1 updated pods are available... daemon set "ovs-ovn" successfully rolled out ------------------------------- [Step 3/6] Install Kube-OVN configmap/ovn-vpc-nat-config created configmap/ovn-vpc-nat-gw-config created deployment.apps/kube-ovn-controller created daemonset.apps/kube-ovn-cni created deployment.apps/kube-ovn-monitor created service/kube-ovn-monitor created service/kube-ovn-controller created service/kube-ovn-cni created Waiting for deployment "kube-ovn-controller" rollout to finish: 0 of 1 updated replicas are available... deployment "kube-ovn-controller" successfully rolled out Waiting for daemon set "kube-ovn-cni" rollout to finish: 0 of 1 updated pods are available... daemon set "kube-ovn-cni" successfully rolled out ------------------------------- Check to delete multus pods to reload CNI config [Step 4/6] Delete pod that not in host network mode pod "coredns-7d764666f9-bdk5d" deleted from kube-system namespace pod "coredns-7d764666f9-r94v9" deleted from kube-system namespace Waiting for deployment "coredns" rollout to finish: 1 out of 2 new replicas have been updated... Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available... Waiting for deployment "coredns" rollout to finish: 1 out of 2 new replicas have been updated... Waiting for deployment "coredns" rollout to finish: 0 of 2 updated replicas are available... Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available... Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available... Waiting for deployment "coredns" rollout to finish: 1 of 2 updated replicas are available... deployment "coredns" successfully rolled out Install Kube-ovn-pinger service/kube-ovn-pinger created daemonset.apps/kube-ovn-pinger created Waiting for daemon set "kube-ovn-pinger" rollout to finish: 0 of 1 updated pods are available... daemon set "kube-ovn-pinger" successfully rolled out pod/kube-ovn-pinger-lgml7 condition met ------------------------------- [Step 5/6] Add kubectl plugin PATH [Step 6/6] Run network diagnose Defaulted container "cni-server" out of: cni-server, hostpath-init (init), install-cni (init) tar: Removing leading `/' from member names NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES coredns-7d764666f9-c66hz 1/1 Running 0 34s 10.244.0.5 taroko-m1.kubeantony.com <none> <none> coredns-7d764666f9-nj725 1/1 Running 0 35s 10.244.0.4 taroko-m1.kubeantony.com <none> <none> etcd-taroko-m1.kubeantony.com 1/1 Running 0 4m6s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 4m4s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 4m4s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-ovn-cni-v8nl5 1/1 Running 0 46s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-ovn-controller-6bd89b7549-xj2l6 1/1 Running 0 46s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-ovn-monitor-5d9bf9858c-sw8lq 1/1 Running 0 46s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-ovn-pinger-lgml7 1/1 Running 0 19s 10.244.0.6 taroko-m1.kubeantony.com <none> <none> kube-proxy-8qvlw 1/1 Running 0 3m59s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 4m4s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 3m22s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> ovn-central-55d8dfb46c-wqktr 1/1 Running 0 77s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> ovs-ovn-tctvd 1/1 Running 0 77s 172.20.6.61 taroko-m1.kubeantony.com <none> <none> NAME CREATED AT vpcs.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT vpc-nat-gateways.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT vpc-egress-gateways.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT subnets.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT ips.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT vlans.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT provider-networks.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT security-groups.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT vips.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT vpc-dnses.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT switch-lb-rules.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT ippools.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT ovn-eips.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT ovn-fips.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT ovn-dnat-rules.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT ovn-snat-rules.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT iptables-eips.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT iptables-fip-rules.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT iptables-snat-rules.kubeovn.io 2026-02-05T10:23:16Z NAME CREATED AT iptables-dnat-rules.kubeovn.io 2026-02-05T10:23:16Z NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 4m5s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4m6s NAME AGE ovn 78s NAME CREATED AT system:ovn 2026-02-05T10:23:17Z NAME ROLE AGE ovn ClusterRole/system:ovn 78s NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME taroko-m1.kubeantony.com Ready control-plane 4m8s v1.35.0 172.20.6.61 <none> Ubuntu 24.04.3 LTS 6.8.0-90-generic cri-o://1.35.0 switch 7aeda674-777a-48ca-8f85-657035f13057 (ovn-default) port coredns-7d764666f9-nj725.kube-system addresses: ["52:93:cd:e8:0d:18 10.244.0.4"] port coredns-7d764666f9-c66hz.kube-system addresses: ["3a:e4:49:26:c4:51 10.244.0.5"] port kube-ovn-pinger-lgml7.kube-system addresses: ["ca:41:f2:66:e8:0b 10.244.0.6"] port ovn-default-ovn-cluster type: router router-port: ovn-cluster-ovn-default switch 7736cdb4-13b5-40d8-aa59-2f7682578e08 (join) port join-ovn-cluster type: router router-port: ovn-cluster-join port node-taroko-m1.kubeantony.com addresses: ["ca:d5:9f:ca:6e:3e 100.64.0.2"] router 0484e79c-fb16-403a-8feb-f9e9c1ca7578 (ovn-cluster) port ovn-cluster-join mac: "92:a1:79:05:14:a6" ipv6-lla: "fe80::90a1:79ff:fe05:14a6" networks: ["100.64.0.1/16"] port ovn-cluster-ovn-default mac: "96:27:4d:84:37:3b" ipv6-lla: "fe80::9427:4dff:fe84:373b" networks: ["10.244.0.1/16"] Routing Policies 31000 ip4.dst == 10.244.0.0/16 allow 31000 ip4.dst == 100.64.0.0/16 allow 30000 ip4.dst == 172.20.6.61 reroute 100.64.0.2 29000 ip4.src == $ovn.default.taroko.m1.kubeantony.com_ip4 reroute 100.64.0.2 IPv4 Routes Route Table <main>: 0.0.0.0/0 100.64.0.1 dst-ip UUID LB PROTO VIP IPs 40a8ae8f-4e37-4ce6-aba2-2ad07cf5e062 cluster-tcp-load tcp 10.96.0.10:53 10.244.0.4:53,10.244.0.5:53 tcp 10.96.0.10:9153 10.244.0.4:9153,10.244.0.5:9153 tcp 10.96.0.1:443 172.20.6.61:6443 tcp 10.96.100.143:6641 172.20.6.61:6641 tcp 10.96.146.40:6643 172.20.6.61:6643 tcp 10.96.210.107:10665 172.20.6.61:10665 tcp 10.96.220.97:8080 10.244.0.6:8080 tcp 10.96.222.234:10661 172.20.6.61:10661 tcp 10.96.247.174:10660 172.20.6.61:10660 tcp 10.96.68.175:6642 172.20.6.61:6642 b298161a-2218-4447-a410-1c86548f3ba4 cluster-udp-load udp 10.96.0.10:53 10.244.0.4:53,10.244.0.5:53 _uuid : 26e09350-9f60-4dbd-a61b-b9448f3ab531 action : drop direction : from-lport external_ids : {parent=ovn.sg.kubeovn_deny_all, vendor=kube-ovn} label : 0 log : false match : "inport == @ovn.sg.kubeovn_deny_all && ip" meter : [] name : [] options : {} priority : 2003 sample_est : [] sample_new : [] severity : [] tier : 2 _uuid : 193e7062-2a24-459b-a696-5806a78aab34 action : drop direction : to-lport external_ids : {parent=ovn.sg.kubeovn_deny_all, vendor=kube-ovn} label : 0 log : false match : "outport == @ovn.sg.kubeovn_deny_all && ip" meter : [] name : [] options : {} priority : 2003 sample_est : [] sample_new : [] severity : [] tier : 2 Chassis "d4527da6-6748-420f-9839-646dbded1df7" hostname: taroko-m1.kubeantony.com Encap geneve ip: "172.20.6.61" options: {csum="true", is_default="true"} Port_Binding kube-ovn-pinger-lgml7.kube-system Port_Binding node-taroko-m1.kubeantony.com Port_Binding coredns-7d764666f9-c66hz.kube-system Port_Binding coredns-7d764666f9-nj725.kube-system 54cf Name: OVN_Southbound Cluster ID: 218b (218b1cca-afbe-4670-94c0-18f16abb94e3) Server ID: 54cf (54cf652c-99c9-431b-9bb3-d0dcdffa4bf3) Address: tcp:[172.20.6.61]:6644 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election won: 64441 ms ago Election timer: 5000 Log: [2, 84] Entries not yet committed: 0 Entries not yet applied: 0 Connections: Disconnections: 0 Servers: 54cf (54cf at tcp:[172.20.6.61]:6644) (self) next_index=2 match_index=83 status: ok 8f1c Name: OVN_Northbound Cluster ID: e5d5 (e5d5a2ef-3b2d-493b-8bff-96396e4d6461) Server ID: 8f1c (8f1c7a0c-ef84-4d5f-b157-96c2f1ffd614) Address: tcp:[172.20.6.61]:6643 Status: cluster member Role: leader Term: 1 Leader: self Vote: self Last Election won: 64879 ms ago Election timer: 5000 Log: [2, 101] Entries not yet committed: 0 Entries not yet applied: 0 Connections: Disconnections: 12 Servers: 8f1c (8f1c at tcp:[172.20.6.61]:6643) (self) next_index=2 match_index=100 status: ok ds kube-proxy ready kube-proxy ready deployment ovn-central ready deployment kube-ovn-controller ready ds kube-ovn-cni ready ds ovs-ovn ready deployment coredns ready ovn-nb leader check ok ovn-sb leader check ok ovn-northd leader check ok service/test-node-port created ### kube-ovn-controller recent log Defaulted container "kube-ovn-controller" out of: kube-ovn-controller, hostpath-init (init) ### start to diagnose node taroko-m1.kubeantony.com #### ovn-controller log: Defaulted container "pinger" out of: pinger, hostpath-init (init) 2026-02-05T10:24:09.776Z|00073|binding|INFO|Claiming lport coredns-7d764666f9-c66hz.kube-system for this chassis. 2026-02-05T10:24:09.776Z|00074|binding|INFO|coredns-7d764666f9-c66hz.kube-system: Claiming 3a:e4:49:26:c4:51 10.244.0.5 2026-02-05T10:24:09.777Z|00075|binding|INFO|Setting lport coredns-7d764666f9-c66hz.kube-system ovn-installed in OVS 2026-02-05T10:24:09.784Z|00076|binding|INFO|Setting lport coredns-7d764666f9-c66hz.kube-system up in Southbound 2026-02-05T10:24:10.345Z|00077|binding|INFO|Releasing lport coredns-7d764666f9-r94v9.kube-system from this chassis (sb_readonly=0) 2026-02-05T10:24:10.345Z|00078|binding|INFO|Setting lport coredns-7d764666f9-r94v9.kube-system down in Southbound 2026-02-05T10:24:15.878Z|00079|binding|INFO|Claiming lport kube-ovn-pinger-lgml7.kube-system for this chassis. 2026-02-05T10:24:15.878Z|00080|binding|INFO|kube-ovn-pinger-lgml7.kube-system: Claiming ca:41:f2:66:e8:0b 10.244.0.6 2026-02-05T10:24:15.882Z|00081|binding|INFO|Setting lport kube-ovn-pinger-lgml7.kube-system ovn-installed in OVS 2026-02-05T10:24:15.889Z|00082|binding|INFO|Setting lport kube-ovn-pinger-lgml7.kube-system up in Southbound #### ovs-vswitchd log: Defaulted container "pinger" out of: pinger, hostpath-init (init) 2026-02-05T10:23:56.412Z|00041|bridge|INFO|bridge br-int: added interface ovn0 on port 2 2026-02-05T10:23:57.868Z|00042|bridge|INFO|bridge br-int: added interface 7a4d9583419a_h on port 3 2026-02-05T10:23:57.983Z|00043|bridge|INFO|bridge br-int: added interface 99fecbc31137_h on port 4 2026-02-05T10:24:00.369Z|00044|connmgr|INFO|br-int<->unix#3: 657 flow_mods in the 9 s starting 10 s ago (634 adds, 23 deletes) 2026-02-05T10:24:00.463Z|00045|bridge|INFO|bridge br-int: added interface d10dda60b2bd_h on port 5 2026-02-05T10:24:09.337Z|00046|bridge|INFO|bridge br-int: deleted interface 7a4d9583419a_h on port 3 2026-02-05T10:24:09.775Z|00047|bridge|INFO|bridge br-int: added interface fc4fb4fc28e6_h on port 6 2026-02-05T10:24:10.345Z|00048|bridge|INFO|bridge br-int: deleted interface 99fecbc31137_h on port 4 2026-02-05T10:24:15.876Z|00049|bridge|INFO|bridge br-int: added interface 8ed870f01d82_h on port 7 2026-02-05T10:24:18.249Z|00002|ofproto_dpif_xlate(handler1)|WARN|dropping packet received on port mirror0, which is reserved exclusively for mirroring on bridge br-int while processing icmp6,in_port=1,vlan_tci=0x0000,dl_src=d6:f9:e3:fe:29:40,dl_dst=33:33:00:00:00:02,ipv6_src=fe80::d4f9:e3ff:fefe:2940,ipv6_dst=ff02::2,ipv6_label=0x00000,nw_tos=0,nw_ecn=0,nw_ttl=255,nw_frag=no,icmp_type=133,icmp_code=0 #### ovs-vsctl show results: Defaulted container "pinger" out of: pinger, hostpath-init (init) 4dcba15a-02ed-4933-860e-d52da853027b Bridge br-int fail_mode: secure datapath_type: system Port d10dda60b2bd_h Interface d10dda60b2bd_h Port ovn0 Interface ovn0 type: internal Port br-int Interface br-int type: internal Port mirror0 Interface mirror0 type: internal Port fc4fb4fc28e6_h Interface fc4fb4fc28e6_h Port "8ed870f01d82_h" Interface "8ed870f01d82_h" ovs_version: "3.5.3" #### pinger diagnose results: Defaulted container "pinger" out of: pinger, hostpath-init (init) I0205 18:24:40.105197 53 pinger.go:21] ------------------------------------------------------------------------------- Kube-OVN: Version: v1.15.2 Build: 2026-01-29_08:13:34 Commit: git-0622723 Go Version: go1.25.6 Arch: amd64 ------------------------------------------------------------------------------- I0205 18:24:40.105233 53 pinger.go:24] current capabilities: cap_net_bind_service,cap_net_raw=ep I0205 18:24:40.153647 53 config.go:202] pinger config is &{KubeConfigFile: KubeClient:0xc000637a40 Port:8080 Interval:5 Mode:job ExitCode:0 InternalDNS:kubernetes.default ExternalDNS: NodeName:taroko-m1.kubeantony.com HostIP:172.20.6.61 PodName:kube-ovn-pinger-lgml7 PodNamespace:kube-system PodIP:10.244.0.6 PodProtocols:[IPv4] LabelSelector:app=kube-ovn-pinger ExternalAddress:1.1.1.1,2606:4700:4700::1111 NetworkMode:kube-ovn EnableMetrics:true PollTimeout:2 PollInterval:15 SystemRunDir:/var/run/openvswitch DatabaseVswitchName:Open_vSwitch DatabaseVswitchSocketRemote:unix:/var/run/openvswitch/db.sock DatabaseVswitchFileDataPath:/etc/openvswitch/conf.db DatabaseVswitchFileLogPath:/var/log/openvswitch/ovsdb-server.log DatabaseVswitchFilePidPath:/var/run/openvswitch/ovsdb-server.pid DatabaseVswitchFileSystemIDPath:/etc/openvswitch/system-id.conf ServiceVswitchdFileLogPath:/var/log/openvswitch/ovs-vswitchd.log ServiceVswitchdFilePidPath:/var/run/openvswitch/ovs-vswitchd.pid ServiceOvnControllerFileLogPath:/var/log/ovn/ovn-controller.log ServiceOvnControllerFilePidPath:/var/run/ovn/ovn-controller.pid EnableVerboseConnCheck:false TCPConnCheckPort:8100 UDPConnCheckPort:8101 TargetIPPorts:tcp-172.20.6.61-31185 LogPerm:640} I0205 18:24:40.157925 53 ovn.go:36] ovsdb-server and ovs-vswitchd are up I0205 18:24:40.159364 53 ovn.go:52] ovn-controller is up I0205 18:24:40.159374 53 ovn.go:60] start to check port binding I0205 18:24:40.161788 53 ovn.go:66] ports in ovs: coredns-7d764666f9-c66hz.kube-system, coredns-7d764666f9-nj725.kube-system, kube-ovn-pinger-lgml7.kube-system, node-taroko-m1.kubeantony.com I0205 18:24:40.165161 53 ovn.go:73] ports in sb: coredns-7d764666f9-c66hz.kube-system, coredns-7d764666f9-nj725.kube-system, kube-ovn-pinger-lgml7.kube-system, node-taroko-m1.kubeantony.com I0205 18:24:40.165170 53 ovn.go:84] ovs and ovn-sb port binding check passed I0205 18:24:40.165173 53 ping.go:383] start to check apiserver connectivity I0205 18:24:40.165690 53 ping.go:394] connect to apiserver success in 0.51ms I0205 18:24:40.165698 53 ping.go:174] start to check pod connectivity I0205 18:24:40.489218 53 ping.go:220] ping pod: kube-ovn-pinger-lgml7 10.244.0.6, count: 3, loss count 0, average rtt 0.11ms I0205 18:24:40.489239 53 ping.go:106] start to check node connectivity I0205 18:24:40.752376 53 ping.go:151] ping node: taroko-m1.kubeantony.com 172.20.6.61, count: 3, loss count 0, average rtt 0.09ms I0205 18:24:40.752395 53 ping.go:339] start to check dns connectivity I0205 18:24:40.753552 53 ping.go:356] resolve dns kubernetes.default to [10.96.0.1] in 1.15ms I0205 18:24:40.753563 53 ping.go:291] start to check Service or externalIPPort connectivity I0205 18:24:40.753575 53 ping.go:298] checking targetIPPort tcp-172.20.6.61-31185 I0205 18:24:40.754269 53 ping.go:321] TCP connectivity to targetIPPort 172.20.6.61:31185 success I0205 18:24:40.754281 53 ping.go:255] start to check ping external to 1.1.1.1 I0205 18:24:41.025593 53 ping.go:271] ping external address: 1.1.1.1, total count: 3, loss count 0, average rtt 3.43ms ### finish diagnose node taroko-m1.kubeantony.com kubectl delete svc -l app=conn-check -n kube-system service "test-node-port" deleted from kube-system namespace ------------------------------- ,,,, ,::, ,,::,,,, ,,,,,::::::::::::,,,,, ,,,::::::::::::::::::::::,,, ,,::::::::::::::::::::::::::::,, ,,::::::::::::::::::::::::::::::::,, ,::::::::::::::::::::::::::::::::::::, ,:::::::::::::,, ,,:::::,,,::::::::::, ,,:::::::::::::, ,::, ,:::::::::, ,:::::::::::::, :x, ,:: :, ,:::::::::, ,:::::::::::::::, ,,, ,::, ,, ,::::::::::, ,:::::::::::::::::,,,,,,:::::,,,,::::::::::::, ,:, ,:, ,xx, ,:::::, ,:, ,:: :::, ,x ,::::::::::::::::::::::::::::::::::::::::::::, :x: ,:xx: , :xx, :xxxxxxxxx, :xx, ,xx:,xxxx, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx:, ,xx, :x: :xxx:x::, ::xxxx: :xx:, ,:xxx :xx, ,xx: ,xxxxx:, :x ,::::::::::::::::::::::::::::::::::::::::::::, :xxxxx, :xx, :x: :xxx,,:xx,:xx:,:xx, ,,,,,,,,,xxx, ,xx: :xx:xx: ,xxx,:xx::x ,::::::,,::::::::,,::::::::,,:::::::,,,::::::, :x:,xxx: ,xx, :xx :xx: ,xx,xxxxxx:, ,xxxxxxx:,xxx:, ,xxx, :xxx: ,xxx, :xxxx ,::::, ,::::, ,:::::, ,,::::, ,::::, :x: ,:xx,,:xx::xxxx,,xxx::xx: :xx::::x: ,,,,,, ,xxxxxxxxx, ,xx: ,xxx, :xxx ,::::, ,::::, ,::::, ,::::, ,::::, ,:, ,:, ,,::,,:, ,::::,, ,:::::, ,,:::::, ,, :x: ,:: ,::::, ,::::, ,::::, ,::::, ,::::, ,,,,, ,::::, ,::::, ,::::, ,:::, ,,,,,,,,,,,,, ,::::, ,::::, ,::::, ,:::, ,,,:::::::::::::::, ,::::, ,::::, ,::::, ,::::, ,,,,:::::::::,,,,,,,:::, ,::::, ,::::, ,::::, ,::::::::::::,,,,, ,,,, ,::::, ,,,, ,,,::::,,,, ,::::, ,,::, Thanks for choosing Kube-OVN! For more advanced features, please read https://kubeovn.github.io/docs/stable/en/ If you have any question, please file an issue https://github.com/kubeovn/kube-ovn/issues/new/choose確認 kube-ovn 安裝成功
k -n kube-system get pods執行結果：
NAME READY STATUS RESTARTS AGE coredns-7d764666f9-c66hz 1/1 Running 0 2m47s coredns-7d764666f9-nj725 1/1 Running 0 2m48s etcd-taroko-m1.kubeantony.com 1/1 Running 0 6m19s kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 6m17s kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 6m17s kube-ovn-cni-v8nl5 1/1 Running 0 2m59s kube-ovn-controller-6bd89b7549-xj2l6 1/1 Running 0 2m59s kube-ovn-monitor-5d9bf9858c-sw8lq 1/1 Running 0 2m59s kube-ovn-pinger-lgml7 1/1 Running 0 2m32s kube-proxy-8qvlw 1/1 Running 0 6m12s kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 6m17s kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 5m35s ovn-central-55d8dfb46c-wqktr 1/1 Running 0 3m30s ovs-ovn-tctvd 1/1 Running 0 3m30s新增其他 Control-plane 節點進 K8s 叢集
獲得 Control-plane node Kubeadm Join Command
kubeadm_init_config_file="${HOME}/k8s/config/kubeadm-init.yaml" certificate_key=$(sudo kubeadm init phase upload-certs --upload-certs --config ${kubeadm_init_config_file} | tail -n 1) join_cmd=$(sudo kubeadm token create --print-join-command) join_cp_cmd="sudo $join_cmd --control-plane --certificate-key $certificate_key" echo $join_cp_cmd執行結果：
sudo kubeadm join api-taroko.kubeantony.com:6443 --token j1d0fh.3se96b07m4h4gst6 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6 --control-plane --certificate-key 1e2dd91966c34bbc34602c07415bc30cf58348e96275b72f475b6c0001986e47以可以執行 sudo 的 user 透過 ssh 連線至其他 Control-plane node
ssh sudo_user@ipaddderss將 Control-plane 2 節點加入 K8s 叢集
sudo kubeadm join api-taroko.kubeantony.com:6443 --token j1d0fh.3se96b07m4h4gst6 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6 --control-plane --certificate-key 1e2dd91966c34bbc34602c07415bc30cf58348e96275b72f475b6c0001986e47ssh 連到第三台 control plane node
ssh sudo_user@ipaddderss將 Control-plane 3 節點加入 K8s 叢集
sudo kubeadm join api-taroko.kubeantony.com:6443 --token j1d0fh.3se96b07m4h4gst6 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6 --control-plane --certificate-key 1e2dd91966c34bbc34602c07415bc30cf58348e96275b72f475b6c0001986e47
新增 Worker 節點進 K8s 叢集
在第一台 Controlplane 將 join worker node 指令 print 出來
sudo kubeadm token create --print-join-command執行結果：
kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6以可以執行 sudo 的 user 透過 ssh 連線至 worker node 1
ssh sudo_user@ipaddderss將 worker node 1 加入 K8s 叢集
sudo kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6以可以執行 sudo 的 user 透過 ssh 連線至 worker node 2
ssh sudo_user@ipaddderss將 worker node 2 加入 K8s 叢集
sudo kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6以可以執行 sudo 的 user 透過 ssh 連線至 worker node 3
ssh sudo_user@ipaddderss將 worker node 3 加入 K8s 叢集
sudo kubeadm join api-taroko.kubeantony.com:6443 --token q2djge.km9ms68fttqidlj7 --discovery-token-ca-cert-hash sha256:8f41e3f883eed0f27055c5146da5c0de5a739ef2d4282417d2ea2bdc8c0ccaf6
回到第一台 Control-plane 執行以下命令，將所有 worker nodes 都貼 label
kubectl label node taroko-w1.kubeantony.com node-role.kubernetes.io/worker= kubectl label node taroko-w2.kubeantony.com node-role.kubernetes.io/worker= kubectl label node taroko-w3.kubeantony.com node-role.kubernetes.io/worker=檢視 K8s 狀態是否正常
檢視 K8s node 狀態
kubectl get nodes執行結果：
NAME STATUS ROLES AGE VERSION taroko-m1.kubeantony.com Ready control-plane 20m v1.35.0 taroko-m2.kubeantony.com Ready control-plane 7m35s v1.35.0 taroko-m3.kubeantony.com Ready control-plane 7m1s v1.35.0 taroko-w1.kubeantony.com Ready worker 105s v1.35.0 taroko-w2.kubeantony.com Ready worker 99s v1.35.0 taroko-w3.kubeantony.com Ready worker 94s v1.35.0檢視 kube-system namespace 底下所有 pods 狀態
kubectl -n kube-system get pods執行結果：
NAME READY STATUS RESTARTS AGE coredns-7d764666f9-c66hz 1/1 Running 0 18m coredns-7d764666f9-nj725 1/1 Running 0 18m etcd-taroko-m1.kubeantony.com 1/1 Running 0 21m etcd-taroko-m2.kubeantony.com 1/1 Running 0 8m13s etcd-taroko-m3.kubeantony.com 1/1 Running 0 7m40s kube-apiserver-taroko-m1.kubeantony.com 1/1 Running 0 21m kube-apiserver-taroko-m2.kubeantony.com 1/1 Running 0 8m13s kube-apiserver-taroko-m3.kubeantony.com 1/1 Running 0 7m40s kube-controller-manager-taroko-m1.kubeantony.com 1/1 Running 0 21m kube-controller-manager-taroko-m2.kubeantony.com 1/1 Running 0 8m13s kube-controller-manager-taroko-m3.kubeantony.com 1/1 Running 0 7m40s kube-ovn-cni-qrx5m 1/1 Running 0 2m13s kube-ovn-cni-r2bjz 1/1 Running 0 7m40s kube-ovn-cni-v8nl5 1/1 Running 0 18m kube-ovn-cni-wcqdd 1/1 Running 0 2m18s kube-ovn-cni-xfgqv 1/1 Running 0 2m24s kube-ovn-cni-zkwxj 1/1 Running 0 8m14s kube-ovn-controller-6bd89b7549-xj2l6 1/1 Running 0 18m kube-ovn-monitor-5d9bf9858c-sw8lq 1/1 Running 0 18m kube-ovn-pinger-dtm8w 1/1 Running 0 112s kube-ovn-pinger-h5sqd 1/1 Running 0 104s kube-ovn-pinger-lbd48 1/1 Running 0 113s kube-ovn-pinger-lgml7 1/1 Running 0 17m kube-proxy-5kf45 1/1 Running 0 7m40s kube-proxy-8qvlw 1/1 Running 0 21m kube-proxy-c2vbq 1/1 Running 0 2m13s kube-proxy-ct7jn 1/1 Running 0 8m14s kube-proxy-nfb9w 1/1 Running 0 2m18s kube-proxy-xnkwj 1/1 Running 0 2m24s kube-scheduler-taroko-m1.kubeantony.com 1/1 Running 0 21m kube-scheduler-taroko-m2.kubeantony.com 1/1 Running 0 8m13s kube-scheduler-taroko-m3.kubeantony.com 1/1 Running 0 7m40s kube-vip-taroko-m1.kubeantony.com 1/1 Running 0 20m kube-vip-taroko-m2.kubeantony.com 1/1 Running 0 8m13s kube-vip-taroko-m3.kubeantony.com 1/1 Running 0 7m40s ovn-central-55d8dfb46c-wqktr 1/1 Running 0 18m ovs-ovn-8rg8h 1/1 Running 0 2m18s ovs-ovn-b2zmj 1/1 Running 0 2m24s ovs-ovn-ch9jw 1/1 Running 0 7m40s ovs-ovn-ft9m8 1/1 Running 0 2m13s ovs-ovn-nq67b 1/1 Running 0 8m14s ovs-ovn-tctvd 1/1 Running 0 18m
檢測 K8s 網路延遲和吞吐量
安裝 cilium 指令
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}透過 cilium 指令檢測 K8s 網路頻寬
cilium connectivity perf \ --crr \ --host-to-pod \ --pod-to-host \ --node-selector-client kubernetes.io/hostname=taroko-w1.kubeantony.com \ --node-selector-server kubernetes.io/hostname=taroko-w2.kubeantony.com執行結果：
🔥 [taroko-ovn] Deleting connectivity check deployments... ✨ [taroko-ovn] Creating namespace cilium-test-1 for connectivity check... ℹ️ Nodes used for performance testing: ℹ️ * Node name: taroko-w2.kubeantony.com, zone: ℹ️ * Node name: taroko-w1.kubeantony.com, zone: ✨ [taroko-ovn] Deploying perf-client deployment... ✨ [taroko-ovn] Deploying perf-client-other-node deployment... ✨ [taroko-ovn] Deploying perf-server deployment... ✨ [taroko-ovn] Deploying perf-client-host-net deployment... ✨ [taroko-ovn] Deploying perf-client-other-node-host-net deployment... ✨ [taroko-ovn] Deploying perf-server-host-net deployment... ⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client to become ready... ⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client-other-node to become ready... ⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-server to become ready... ⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client-host-net to become ready... ⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-client-other-node-host-net to become ready... ⌛ [taroko-ovn] Waiting for deployment cilium-test-1/perf-server-host-net to become ready... ℹ️ Cilium version: 0.0.0 🏃[cilium-test-1] Running 1 tests ... [=] [cilium-test-1] Test [network-perf] [1/1] ................................ 🔥 Network Performance Test Summary [cilium-test-1]: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 📋 Scenario | Node | Test | Duration | Min | Mean | Max | P50 | P90 | P99 | Transaction rate OP/s -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 📋 pod-to-pod | same-node | TCP_CRR | 10s | 60µs | 85.91µs | 677µs | 82µs | 98µs | 125µs | 11615.84 📋 pod-to-pod | same-node | TCP_RR | 10s | 14µs | 29.22µs | 695µs | 28µs | 33µs | 48µs | 34070.14 📋 pod-to-host | same-node | TCP_CRR | 10s | 74µs | 88.54µs | 179µs | 86µs | 96µs | 120µs | 474.50 📋 pod-to-host | same-node | TCP_RR | 10s | 18µs | 29.87µs | 1.801ms | 29µs | 33µs | 49µs | 33340.49 📋 host-to-pod | same-node | TCP_CRR | 10s | 82µs | 96.1µs | 189µs | 92µs | 113µs | 137µs | 11.60 📋 host-to-pod | same-node | TCP_RR | 10s | 16µs | 29.97µs | 716µs | 29µs | 33µs | 48µs | 33217.61 📋 host-to-host | same-node | TCP_CRR | 10s | 59µs | 76.15µs | 683µs | 73µs | 84µs | 118µs | 13101.67 📋 host-to-host | same-node | TCP_RR | 10s | 15µs | 28.35µs | 700µs | 28µs | 31µs | 46µs | 35113.98 📋 pod-to-pod | other-node | TCP_CRR | 10s | 242µs | 293.67µs | 1.817ms | 290µs | 323µs | 381µs | 2641.99 📋 pod-to-pod | other-node | TCP_RR | 10s | 96µs | 121.1µs | 1.509ms | 118µs | 137µs | 167µs | 8245.27 📋 pod-to-host | other-node | TCP_CRR | 10s | 246µs | 1.07221ms | 3.071576s | 293µs | 329µs | 400µs | 926.89 📋 pod-to-host | other-node | TCP_RR | 10s | 93µs | 135.28µs | 1.03699s | 118µs | 137µs | 169µs | 7382.95 📋 host-to-pod | other-node | TCP_CRR | 10s | 246µs | 299.94µs | 2.051ms | 296µs | 325µs | 386µs | 3331.08 📋 host-to-pod | other-node | TCP_RR | 10s | 98µs | 121.49µs | 1.927ms | 118µs | 137µs | 168µs | 8218.90 📋 host-to-host | other-node | TCP_CRR | 10s | 190µs | 230.35µs | 1.95ms | 227µs | 255µs | 305µs | 4336.79 📋 host-to-host | other-node | TCP_RR | 10s | 85µs | 107.45µs | 3.69ms | 106µs | 120µs | 148µs | 9292.06 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------- 📋 Scenario | Node | Test | Duration | Throughput Mb/s ---------------------------------------------------------------------------------------- 📋 pod-to-pod | same-node | TCP_STREAM | 10s | 11263.46 📋 pod-to-pod | same-node | TCP_STREAM_MULTI | 10s | 41560.29 📋 pod-to-host | same-node | TCP_STREAM | 10s | 10561.07 📋 pod-to-host | same-node | TCP_STREAM_MULTI | 10s | 40874.81 📋 host-to-pod | same-node | TCP_STREAM | 10s | 11463.51 📋 host-to-pod | same-node | TCP_STREAM_MULTI | 10s | 42231.56 📋 host-to-host | same-node | TCP_STREAM | 10s | 26166.77 📋 host-to-host | same-node | TCP_STREAM_MULTI | 10s | 96386.74 📋 pod-to-pod | other-node | TCP_STREAM | 10s | 2823.65 📋 pod-to-pod | other-node | TCP_STREAM_MULTI | 10s | 4573.82 📋 pod-to-host | other-node | TCP_STREAM | 10s | 2886.42 📋 pod-to-host | other-node | TCP_STREAM_MULTI | 10s | 4405.81 📋 host-to-pod | other-node | TCP_STREAM | 10s | 2828.99 📋 host-to-pod | other-node | TCP_STREAM_MULTI | 10s | 5137.81 📋 host-to-host | other-node | TCP_STREAM | 10s | 9404.16 📋 host-to-host | other-node | TCP_STREAM_MULTI | 10s | 9407.35 ---------------------------------------------------------------------------------------- ✅ [cilium-test-1] All 1 tests (32 actions) successful, 0 tests skipped, 0 scenarios skipped.