antony@notes:~/openshift$ cat "OpenShift-設定讓不同的叢集節點中特定-namespace-底下所有-Pods-對外上網後顯示同一個-source-ip.md"

OpenShift 設定讓不同的叢集節點中特定 namespace 底下所有 Pods 對外上網後顯示同一個 source ip

2025-06-05· openshift

OpenShift 設定讓不同的叢集節點中特定 namespace 底下所有 Pods 對外上網後顯示同一個 source ip

先備知識

OpenShift 的 Egress IP 功能允許您確保一個或多個 Pod（來自一個或多個專案）的流量，在離開叢集網路時，透過 SNAT 機制，讓屬於特定 namespace 底下 Pods 的流量對外呈現固定 Source IP。

Step1: 建立 Project（Namespace）test-egress

oc new-project test-egress

Step2: 為特定節點指派託管 Egress IP

標記 worker-1 節點以託管 Egress IP 位址

oc label nodes worker-1 k8s.ovn.org/egress-assignable=""

Step3: 為 Namespace 指派 Egress IP

mkdir ~/egressip

編輯與設定 EgressIP YAML File

echo 'apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
  name: egressip-test
spec:
  egressIPs:
  - 10.10.7.19
  namespaceSelector:
    matchLabels:
      kubernetes.io/metadata.name: "test-egress"' > ~/egressip/egressip-test.yaml

egressIPs：列出要分配給 Pods 對外呈現的固定來源 IP。
namespaceSelector：以 Label Selector 的方式決定哪些 namespace 可套用此 Egress IP。
kubernetes.io/metadata.name: "test"，表示只選擇名稱為 test-egress 的 namesapce。
podSelector（可選）：可進一步針對特定 Pod 加以篩選；若不填，則預設會對該 namespace 內所有 Pods 生效。

建立 EgressIP

oc apply -f ~/egressip/egressip-test.yaml

檢查 EgressIP 是否指派到節點上

$ oc get egressip

執行結果 :

NAME            EGRESSIPS    ASSIGNED NODE   ASSIGNED EGRESSIPS
egressip-test   10.10.7.19   worker-1        10.10.7.19

實際到 worker-1 節點檢查

$ ssh core@worker-1 'ip addr show | grep -B 6 10.10.7.19'

執行結果 :

5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether bc:24:11:7f:f2:b0 brd ff:ff:ff:ff:ff:ff
    inet 10.10.7.16/24 brd 10.10.7.255 scope global noprefixroute br-ex
       valid_lft forever preferred_lft forever
    inet 169.254.0.2/17 brd 169.254.127.255 scope global br-ex
       valid_lft forever preferred_lft forever
    inet 10.10.7.19/32 scope global br-ex

Step4: 建立測試應用

編輯 Deployment YAML File

echo 'apiVersion: apps/v1
kind: Deployment
metadata:
  name: source-ip-app
  labels:
    app: source-ip-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: source-ip-app
  strategy: {}
  template:
    metadata:
      labels:
        app: source-ip-app
    spec:
      containers:
      - name: echoserver
        image: registry.k8s.io/echoserver:1.10
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - worker-1
                - worker-2
                - worker-3
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - source-ip-app
            topologyKey: "kubernetes.io/hostname"' > ~/egressip/source-ip-app.yaml

設定 Namespace 的 default ServiceAccount 使用 anyuid SCC

oc adm policy add-scc-to-user anyuid -z default -n test-egress

建立測試應用

oc apply -f ~/egressip/source-ip-app.yaml

檢視應用 Pods 是否正常

oc get pods -o wide

執行結果 :

NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
source-ip-app-b44758d6d-t894p   1/1     Running   0          3s    10.128.2.27   worker-1   <none>           <none>
source-ip-app-b44758d6d-tzpjj   1/1     Running   0          3s    10.131.0.14   worker-2   <none>           <none>
source-ip-app-b44758d6d-xvsqg   1/1     Running   0          4s    10.130.0.9    worker-3   <none>           <none>

Step5: 在 OpenShift 叢集外部的主機建立外部 HTTP server

安裝 httpd 套建

sudo yum install -y httpd

將 httpd 啟在 8080 Port

sudo sed -i 's|Listen 80|Listen 8080|g' /etc/httpd/conf/httpd.conf

建立測試 HTML 頁面

echo '<html>
<head/>
<body>OK</body>
</html>' | sudo tee /var/www/html/index.html

啟動 httpd Server 服務

sudo systemctl start httpd

確認 httpd Server 服務運作正常

sudo systemctl is-active httpd

正確執行結果 :

active

測試頁面是否正常運作

curl http://localhost:8080/index.html

執行結果 :

<html>
<head/>
<body>OK</body>
</html>

Step6: 測試連線

撰寫 Bash Script 來完成以下目的 :
以 round-robin 的方式對名稱為 source-ip-app-... 前綴的 Pod 執行 curl http://10.10.7.11:8080/index.html
編輯 Bash Script

nano ~/egressip/test.sh

檔案內容如下 :

#!/bin/bash
mapfile -t pods < <(oc -n test-egress get pods -o name | grep source-ip-app-)

if [[ ${#pods[@]} -eq 0 ]]; then
  echo "找不到符合條件的 Pod，請確認 Pod 名稱前綴是否正確" >&2
  exit 1
fi

for i in {1..20}; do
  for pod in "${pods[@]}"; do
    pod_name="${pod#pod/}"
    echo "對 Pod ${pod_name} 執行 curl 請求..."
    oc exec "${pod_name}" -- curl -s http://10.10.7.11:8080/index.html \
      && echo " ==> ${pod_name} 執行成功" \
      || echo " ==> ${pod_name} 執行失敗"
    sleep 1
  done
done

為 Bash Script 賦予執行權限

chmod +x ~/egressip/test.sh

執行腳本開始測試

~/egressip/test.sh

執行結果 :

對 Pod source-ip-app-b44758d6d-t894p 執行 curl 請求...
<html>
<head/>
<body>OK</body>
</html>
 ==> source-ip-app-b44758d6d-t894p 執行成功
對 Pod source-ip-app-b44758d6d-tzpjj 執行 curl 請求...
<html>
<head/>
<body>OK</body>
</html>
 ==> source-ip-app-b44758d6d-tzpjj 執行成功
對 Pod source-ip-app-b44758d6d-xvsqg 執行 curl 請求...
...以下省略

檢視 httpd access log

sudo tail -f /var/log/httpd/access_log

執行結果 :

127.0.0.1 - - [05/Jun/2025:16:51:26 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.76.1"
10.10.7.19 - - [05/Jun/2025:17:01:56 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:01:57 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:01:59 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:00 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:02 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:04 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:05 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:07 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:08 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:10 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:11 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:13 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:15 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:16 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
...以下結果省略

Step7: 清理環境

oc delete -R -f ~/egressip/
oc label node worker-1 k8s.ovn.org/egress-assignable-
oc adm policy remove-scc-from-user anyuid -z default -n test-egress
oc delete project test-egress
# 以下命令請在 httpd Server 的主機上執行
sudo systemcctl stop httpd
sudo yum remove -y httpd

參考資料

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/networking/ovn-kubernetes-network-plugin#configuring-egress-ips-ovn

OpenShift 如何在 Deployment 不改設定的情況下，設 Deployment pods 跑在特定的某些節點上在 Openshift 上建立應用並驗證掛載 NFS Storage