Antony 筆記：技術原理與實務經驗 – OpenShift

透過 VMware Aria Operations 監控 openshift

Tue, 13 Jan 2026 02:13:56 +0800

1. 先決條件

在使用 VMware Aria Operations Management Pack for Kubernetes 來監控 Kubernetes 叢集之前，您必須先準備好您的 VMware Aria Operations 環境。您必須確保 VMware Aria Operations 符合下列一般需求。

確保您已安裝 VMware Aria Operations (SaaS) 或 VMware Aria Operations 8.1 或更高版本。
確認您已安裝 VMware Aria Operations Management Pack for Kubernetes。
確保您的環境中已部署一個 Openshift 叢集。

2. 環境準備

2.1. 在 Kubernetes 中安裝 VMware Aria Operations Management Pack

2.1.1. 先決條件

已安裝 Kubernetes
已安裝 VMware Aria Operations
確認擁有最新版本的管理套件，且該管理套件與您的 VMware Aria Operations 版本相容。
確保擁有安裝管理套件的權限。

2.1.2. 開始安裝

從 VMware Marketplace 下載該管理套件的 PAK 檔案。
使用管理員權限登入 VMware Aria Operations。
從左側選單中，點擊 Data Sources > Integrations。前往右側窗格中的 Repository 索引標籤。
在 Repository 索引標籤上，點擊 Add。
此時會出現「Add Solutions Wizard Options」(新增解決方案精靈選項)。該精靈包含三個頁面，您可以在此尋找並上傳 PAK 檔案、接受 EULA (使用者授權合約) 並進行安裝，以及檢視安裝過程。點擊 Browse (瀏覽) 以找到您的暫存資料夾並選取 PAK 檔案。
- 在升級至最新版本時，您可以勾選 Install the PAK file even if it is already installed (即使已安裝也要安裝 PAK 檔案) 和 Reset Default Content (重設預設內容) 這兩個選項。
點擊 Upload (上傳)。上傳過程可能需要幾分鐘。
閱讀並接受 EULA，然後點擊 Next (下一步)。
當 VMware Aria Operations Management Pack for Kubernetes 安裝完成後，點擊 Finish (完成)。
從 Data Sources -> Integrations -> ADD → Kubernetes 索引標籤設定此項整合。

先在 openshift 建立一個 read only 的 serviceAccount

# 建立 ServiceAccount
oc create serviceaccount vrops-mp-read-only-user -n openshift-logging
# 賦予必要讀取權限
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z vrops-mp-read-only-user -n openshift-logging
oc adm policy add-cluster-role-to-user cluster-reader -z vrops-mp-read-only-user -n openshift-logging
# 建立長期 Token (Secret)
cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: vrops-mp-read-only-user-token
namespace: openshift-logging
annotations:
kubernetes.io/service-account.name: "vrops-mp-read-only-user"
EOF

獲取 sa 的 token

echo $(oc get secret vrops-mp-read-only-user-token -n openshift-logging -o jsonpath='{.data.token}' | base64 -d)

執行結果：

eyJhbGciOiJSUzI1NiIsImtpZCI6Ijk5Vi1DZlc5M1Y0eUh5UEtBYTA4bG9OaC1jMFFTT0s1cElCa0lOZ3NuaTAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZyb3BzLW1wLXJlYWQtb25seS11c2VyLXRva2VuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZyb3BzLW1wLXJlYWQtb25seS11c2VyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNDI2MmJmNTctMzA5OC00ZjYxLWFlMDAtNDE3M2UzMmYzZGJjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dnJvcHMtbXAtcmVhZC1vbmx5LXVzZXIifQ.lpM7IEWiiB4sbrCLYB-Re_R4d7-g3sELisU4npjJyp_pwzJB0LehOo-pwnlwWH33FtTOqrsD26hYlYO5POmEpGjAUt49DoL9-SkoOfCjcQy82lGKbj_F6kMVqyddVEcHBIUWIuHAckvekSerLxH1M4dIztBnTFXulKJ_GIY72VA6aQOt0kGX35-dCH8GK-nHUJIu5cKcbrWV1NAfkzLBpFqlkVnjcOgHpf2jXJ3-3ozGQ4wMvCeZlG9YA86rY8Xn1XrlbztQx2klURLuy4b2RgEt7sfZuilrZScrFoPI68a7Av5prb8EQeRnzEZSsagNP5RfH5PG1qmRW64R_BrAlg

依序輸入以下欄位內容：

Cloud Account Information
- Name: openshift

Connection Information

Control Plane URL: https://api.topgun.kubeantony.com:6443
- 執行 oc config view 找到 clusters.cluster.server 的值
Collector Service: Prometheus Server

Credential: <點選右邊的 + 按鈕>

Credential Kind: Token Auth
Credential name: ocp-prometheus
Bearer Token: 將步驟 11 得到的 token 複製貼上

Prometheus Server: https://prometheus-k8s-openshift-monitoring.apps.topgun.kubeantony.com/

執行以下命令獲取：

echo https://$(oc get routes -n openshift-monitoring prometheus-k8s -o jsonpath='{.status.ingress[0].host}')/

命令執行結果：

https://prometheus-k8s-openshift-monitoring.apps.topgun.kubeantony.com/

Prometheus endpoint token:

執行以下命令獲取：

oc create sa monitor-reader -n default
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z monitor-reader -n default
oc adm policy add-cluster-role-to-user cluster-reader -z monitor-reader -n default
# 建立長期 Token (Secret)
cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: monitor-reader-token
namespace: default
annotations:
kubernetes.io/service-account.name: "monitor-reader"
EOF
echo $(oc get secret monitor-reader-token -n default -o jsonpath='{.data.token}' | base64 -d)

命令執行結果：

eyJhbGciOiJSUzI1NiIsImtpZCI6Ild2Y3ZIRzk4cEhGSlJFeGt0dk1lS1hQQzlldDNIQkdnbXhFd0NLTmRLNVEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im1vbml0b3ItcmVhZGVyLXRva2VuIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6Im1vbml0b3ItcmVhZGVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYTE2ZTNhYTYtOWI3NC00NjQ3LTlkZjctZDE5MWJjM2RiNTcxIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6bW9uaXRvci1yZWFkZXIifQ.wo6tD4uW-4dXgJ2hy92MnIim0V2ZLu2Yzt9ZyCZ3Y9qcZ29brh9JV6fNgNarK1eY7qIcDQKNr5IFxOHL3dZzZUI5ZlfW-rI9HYpZHbbUKDqr1b7yf6_ZtW-kTAiSdehn0DGXynPCNDpfW7MpwrsD_HUGzuYmXB2U3uTZe2soeI4zQsFSgWNjmna12oXsQ59k7uvXevjcyC4_SVI3g5BryDUhrdeAKBcZpYs5udOPgw1-j0Th0BS25xOOG4sQO3weP2CiRtYwm2RLXpct_zJXTlAMyxo05UG7vwpHOfh8nsSP0TdXuRNA4Z4-A6WmI9VGCQdohsC6kOQgeVTcZ7yL0KMrz8TE3i6_V9Uo9tiH0Mc-fBM68Tb6r6VlNgGfHM4iGeuC4gBrYM2Nn0EC2GqOlRcZ_odifGZbkIpe0LO07kAF-OwEVRra2qlcPdvD5dEOnXYf3Nh2LyuupNKAspZ1sORw3pfix8C6lwBIFqo3OdmkNZCJHfr4BaDqBaA5Gl4Z8uSCUn5Gn2_F1kcq9iECZheQNhD4pFyMebNZyrKDbu1s67N8rjDR79urOzuIriB1FbElVoG1AHP2aNcX1bDOcNgtOptpRGdxgB1gkdgRpkVbq8PAu-W6dwU2NPEjljUaIe4Myf8vzhhJVq_DzYKOOhFTMKAPAyEzrFC3jABcXYQ

Collector / Group: 選 CloudProxy-xx-xx-xx-xx-xxx

Advanced Settings
- Java Process Monitoring: Activatied
- Delete nonexistent object older than: 2 hours
- cAdvisor Install Check: Deactivated
點選左下角 SAVE 按鈕

確認連接成功

How to Create a User Using Htpasswd in Openshift

Wed, 31 Dec 2025 05:26:48 +0800

安裝 htpasswd 套件
```
yum install -y httpd-tools
```
建立 htpasswd 使用者檔案
```
htpasswd -c -B -b users.htpasswd mgr-haha haha
```
格式如下： htpasswd -c -B -b </path/to/users.htpasswd> <username> <password>

密碼會以雜湊形式存儲在 users.htpasswd 檔案中。
創建 Htpasswd secret。定義一個包含 HTPasswd 使用者檔案的 secret。
```
oc -n openshift-config create secret generic htpass-secret \
--from-file=htpasswd=users.htpasswd 
```
格式如下： oc create secret generic htpass-secret --from-file=htpasswd=</path/to/users.htpasswd> -n openshift-config

建立 HTPasswd identity CR

nano htpasswd.yml

檔案內容如下：

apiVersion: config.openshift.io/v1
kind: OAuth
metadata:
name: cluster
spec:
identityProviders:
- name: mgr-haha_htpasswd_provider
challenge: true
login: true
mappingMethod: claim
type: HTPasswd
htpasswd:
fileData:
name: htpass-secret

部署 HTPasswd identity CR
```
oc apply -f htpasswd.yml
```

為新的使用者設定叢集管理員角色

oc adm policy add-role-to-user cluster-admin mgr-haha

將叢集的 CA 內容存成 ca.crt 檔案

oc -n openshift-config-managed get configmap kube-root-ca.crt \
-o jsonpath='{.data.ca\.crt}' > ca.crt

測試新使用者登入 ocp

oc login -u=mgr-haha -p haha --certificate-authority=ca.crt

正確執行結果：

Login successful.
You have one project on this server: "default"
Using project "default".

為 mgr-test 創建客戶端證書，使用內部 OpenShift CA

openssl req -nodes -newkey rsa:4096 \
-keyout /tmp/mgr-test.key -subj "/O=system:admin/CN=mgr-test" \
-out /tmp/mgr-test.csr

將 CSR 提交給 OpenShift，以便稍後用內部 CA 簽署：

cat << EOF | oc create -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: mgr-test-access
spec:
signerName: kubernetes.io/kube-apiserver-client
groups:
- system:authenticated
request: $(cat /tmp/mgr-test.csr | base64 -w0)
usages:
- client auth
EOF

批准證書請求

oc adm certificate approve mgr-test-access

證書已獲批准，我們現在可以從叢集中獲取它：

oc get csr mgr-test-access -o jsonpath='{.status.certificate}' \
| base64 -d > /tmp/mgr-test.crt

為 test-mgr 創建 kubeconfig 檔案

export OPENSHIFT_API_SERVER_ENDPOINT=api.topgun.kubeantony.com:6443
openssl s_client -showcerts -connect ${OPENSHIFT_API_SERVER_ENDPOINT} </dev/null 2>/dev/null|openssl x509 -outform PEM > /tmp/ocp-apiserver-cert.crt
export USER=mgr-test
# 1. 設定使用者憑證 (加入 config)
oc config set-credentials ${USER} \
--kubeconfig="/tmp/${USER}config" \
--client-certificate="/tmp/${USER}.crt" \
--client-key="/tmp/${USER}.key" \
--embed-certs=true
# 2. 設定叢集資訊 (加入 config)
oc config set-cluster topgun-ocp-cluster \
--kubeconfig="/tmp/${USER}config" \
--certificate-authority="/tmp/ocp-apiserver-cert.crt" \
--embed-certs=true \
--server="https://${OPENSHIFT_API_SERVER_ENDPOINT}"
# 3. 設定 Context (加入 config)
oc config set-context topgun-ocp \
--kubeconfig="/tmp/${USER}config" \
--cluster=topgun-ocp-cluster \
--namespace=default \
--user=${USER}
# 4. 切換 Context (加入 config)
oc config use-context topgun-ocp \
--kubeconfig="/tmp/${USER}config"

確認當前身份

oc --kubeconfig=/tmp/mgr-testconfig auth whoami

執行結果：

ATTRIBUTE VALUE
Username mgr-test
Groups [system:admin system:authenticated]

參考文件

ocp to vmware loginsight

Mon, 29 Dec 2025 03:42:18 +0800

oc new-project vmware-vrealize-log-insight
oc create configmap vrli-fluent-config --from-file=fluent.conf
oc create configmap fluentd-gemfile-config --from-file=Gemfile

Install Elasticsearch Metrics Exporter on Openshift

Thu, 27 Nov 2025 11:47:08 +0800

1. 環境資訊

openshift version 4.19.7 in disconnected network
需要將 Elasticsearch Metrics Exporter 部署在 logging 節點

logging 節點資訊如下：

oc get nodes -l "node-role.kubernetes.io/logging"

執行結果：

NAME STATUS ROLES AGE VERSION
docpl01.ocp.cbsd.scsb.com.tw Ready logging 24d v1.32.6
docpl02.ocp.cbsd.scsb.com.tw Ready logging 24d v1.32.6
docpl03.ocp.cbsd.scsb.com.tw Ready logging 24d v1.32.6

logging 節點的 taints 資訊 :

oc get nodes -l "node-role.kubernetes.io/logging" -o yaml | grep -A 6 -i "taints"

執行結果：

 taints:
- effect: NoSchedule
key: logging
value: reserved
- effect: NoExecute
key: logging
value: reserved
--
taints:
- effect: NoSchedule
key: logging
value: reserved
- effect: NoExecute
key: logging
value: reserved
--
taints:
- effect: NoSchedule
key: logging
value: reserved
- effect: NoExecute
key: logging
value: reserved

2. 前置作業

先決條件：須在可到 public 網路的 linux 主機作業，並有安裝 podman 套件

mkdir -p ~/openshift/addon/monitoring/es-exporter; cd ~/openshift/addon/monitoring/es-exporter

下載 helm 命令執行檔

wget https://get.helm.sh/helm-v4.0.1-linux-amd64.tar.gz

下載 prometheus-elasticsearch-exporter helm chart 檔案

helm pull oci://ghcr.io/prometheus-community/charts/prometheus-elasticsearch-exporter

下載 prometheus-elasticsearch-exporter default values.yaml

wget https://raw.githubusercontent.com/prometheus-community/helm-charts/refs/heads/main/charts/prometheus-elasticsearch-exporter/values.yaml

獲取 prometheus-elasticsearch-exporter container image 名稱
```
cat default-values.yaml | grep repository
```
執行結果：
```
repository: quay.io/prometheuscommunity/elasticsearch-exporter
```
tag 的版本預設會被指定在 Chart.yaml 中

獲取 prometheus-elasticsearch-exporter container image tag 版本

tar -zxvf prometheus-elasticsearch-exporter-7.0.0.tgz
grep -i appversion prometheus-elasticsearch-exporter/Chart.yaml

執行結果：

appVersion: v1.9.0

下載並打包 prometheus-elasticsearch-exporter container image

podman pull quay.io/prometheuscommunity/elasticsearch-exporter:v1.9.0
podman save quay.io/prometheuscommunity/elasticsearch-exporter:v1.9.0 > elasticsearch-exporter-v1.9.0.tar

準備啟用 openshift logging 收集 user workload 的 configmap 設定檔

echo 'apiVersion: v1
kind: ConfigMap
metadata:
name: cluster-monitoring-config
namespace: openshift-monitoring
data:
config.yaml: |
enableUserWorkload: true' > cm.yaml

先撰寫客製化的 helm values.yaml 範本 (實際到客戶端會再根據環境修改成對應的設定值)

nano custom-values.yaml

執行結果：

image:
repository: quay.kubeantony.com:8443/library/prometheuscommunity/elasticsearch-exporter
tag: "v1.9.0"
pullPolicy: IfNotPresent
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000790001
seccompProfile:
type: "RuntimeDefault"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 100m
memory: 128Mi
nodeSelector:
node-role.kubernetes.io/logging: ""
tolerations: []
es:
uri: https://admin:admin123@elasticsearch-sample-elastic.apps.topgun.kubeantony.com
sslSkipVerify: true

撰寫 prometheus 收集 elasticsearch metrics 需要的 service monitor yaml 設定檔

nano es-service-monitor.yaml

檔案內容如下：

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: ealsticsearch-monitor
namespace: elastic
spec:
endpoints:
- interval: 30s
port: http
scheme: http
selector:
matchLabels:
app.kubernetes.io/instance: prometheus-elasticsearch-exporter

將整個專案目錄壓縮成一個檔案，之後可直接提供給客戶
```
cd ..; tar -zcvf es-exporter.tar.gz es-exporter/
```

3. 開始設定與安裝 Elasticsearch Metrics Exporter

將 es-exporter.tar.gz 壓縮檔上傳至客戶 ocp 環境的 bastion 主機
```
scp es-exporter.tar.gz <user>:<host_ip>:<folder>
```
for example: scp es-exporter.tar.gz ansible@10.21.104.120:.
解壓縮 es-exporter.tar.gz 檔案
```
tar -zxvf /PATH/TO/es-exporter.tar.gz -C <DST_Folder>
```
for example: tar -zxvf /home/ansible/es-exporter.tar.gz -C elk/
切換至工作目錄
```
cd ~/elk/es-exporter/
```
將 prometheus-elasticsearch-exporter container image 的打包檔還原成 image
```
podman load < elasticsearch-exporter-v1.9.0.tar
```

修改 container image 名稱用以符合客戶環境

podman tag quay.io/prometheuscommunity/elasticsearch-exporter:v1.9.0 docph01.cbsd.scsb.com.tw/ocp4/elk/elasticsearch-exporter:v1.9.0

確認是否登入 Image Registry

podman login --get-login docph01.cbsd.scsb.com.tw

執行結果：

admin

將 container image 推送至客戶 Image Registry

podman push docph01.cbsd.scsb.com.tw/ocp4/elk/elasticsearch-exporter:v1.9.0

安裝 helm 指令

tar -zxvf helm-v4.0.1-linux-amd64.tar.gz; \
mv linux-amd64/helm /usr/local/bin/

修改 helm values.yaml 用以符合客戶 ocp 環境

nano custom-values.yaml

主要會需要設定以下項目：

image.repository: container image 名稱
podSecurityContext.runAsUser: pod 須以什麼 user 執行，可根據以下命令得知 namespace 底下可用的 uid 範圍
```
oc get ns elastic -o yaml | grep uid-
```
執行結果：
```
openshift.io/sa.scc.uid-range: 1000780000/10000
```
resources: 設定 cpu 和記憶體可用或上限的資源
nodeSelector: 設定 pod 要跑在帶有什麼 label 的節點上
tolerations: 如果該節點有 taints 需設定 tolerations 來容忍這些 taints，否則 pod 將無法被指派到該節點
es.url: elasticsearch url 的位置

檔案內容：

image:
repository: docph01.cbsd.scsb.com.tw/ocp4/elk/elasticsearch-exporter
tag: "v1.9.0"
pullPolicy: IfNotPresent
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000780001
seccompProfile:
type: "RuntimeDefault"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 100m
memory: 128Mi
nodeSelector:
node-role.kubernetes.io/logging: ""
tolerations:
- effect: NoSchedule
key: logging
operator: Equal
value: reserved
- effect: NoExecute
key: logging
operator: Equal
value: reserved
es:
uri: https://elastic:1B91L1oPibVZ430p95wS3ljs@elasticsearch-elastic.apps.ocp.cbsd.scsb.com.tw
sslSkipVerify: true

查找 kubeconfig 位置

echo $KUBECONFIG

執行結果：

/source/OCP_4.14_CLI_and_YAMLs/OCP_YAMLs/ign-workdir/auth/kubeconfig

透過 helm 安裝 prometheus-elasticsearch-exporter

helm --kubeconfig=/source/OCP_4.14_CLI_and_YAMLs/OCP_YAMLs/ign-workdir/auth/kubeconfig \
upgrade
--install prometheus-elasticsearch-exporter ./prometheus-elasticsearch-exporter-7.0.0.tgz \
-f custom-values.yaml

檢視 prometheus-elasticsearch-exporter pod 運作狀態

oc -n elastic get pods

執行結果：

NAME READY STATUS RESTARTS AGE
elasticsearch-es-data-nodes-0 1/1 Running 2 (5h7m ago) 5h8m
elasticsearch-es-data-nodes-1 1/1 Running 2 (5h7m ago) 5h8m
elasticsearch-es-data-nodes-2 1/1 Running 2 (5h7m ago) 5h8m
elasticsearch-es-master-nodes-0 1/1 Running 2 (5h7m ago) 5h8m
elasticsearch-es-master-nodes-1 1/1 Running 2 (5h7m ago) 5h8m
elasticsearch-es-master-nodes-2 1/1 Running 2 (5h7m ago) 5h8m
kibana-kb-688b58f67f-8jq4n 1/1 Running 0 5h8m
prometheus-elasticsearch-exporter-7b9d89dcd9-xx68g 1/1 Running 0 4h51m

設定 openshift monitoring 允許收集自定義的 metrics
```
oc apply -f cm.yaml
```
設定 openshift monitoring 要去收集 elasticsearch metrics expose 出來的 metrics
```
oc apply -f es-service-monitor.yaml
```

檢視 servicemonitors 是否正常部署

oc -n elastic get servicemonitors

執行結果：

NAME AGE
ealsticsearch-monitor 4h54m

4. 驗證 Prometheus 是否可收集到 Elasticsearch 的 metrics

創建一個 serviceAccount 並賦予相應的權限來驗證 prometheus 有收集到 elasticsearch 的 metrics
```
oc create sa monitor-reader
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z monitor-reader
```

為 serviceAccount 建立 secret 來獲取永久的 token

cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
name: monitor-reader-token
annotations:
kubernetes.io/service-account.name: "monitor-reader"
EOF

獲取 monitor-reader serviceAccount 的 token

echo "Bearer $(oc get secret monitor-reader-token -o jsonpath='{.data.token}' | base64 -d)"

執行結果：

Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ilc4Y0dnOC1JMTVWZ1Zwd3kzVFpxdjlFeUJybWlnREJFcUlkZGdOanZFdlEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtb25pdG9yLXJlYWRlci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtb25pdG9yLXJlYWRlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFhNjllNzJhLTU3YzAtNDJiYi1iMTU0LTVjMWUwYzgyYjllZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpvcGVuc2hpZnQtbW9uaXRvcmluZzptb25pdG9yLXJlYWRlciJ9.ahjR-seVoZI6YiKNCKovP9TaIqAuy4rTRbYaZEb1xwHfy3FTowvrtZyi5zdYNHx4r5-Ry7M7edQUPFY4CCTdd8sbt3q-jD2rAankCIanKzCf02D_QVvPB2so1zdxdsVDBgx71tLTsx7k2wzzsefoL0QRnPcYA-nJJjMxeg2Rrzn2g8dNQm65sTl-ZAfunZq5UlqIs-UYhrM5DGZ0_EbQRtLgEdW91h_OGa0KFEtwsjXL_f8k4cPZydk63du4qwXeW0MLrHAQXRc6UAkG7Pl3GuPeqUdaSIafwywPOVv_BHBRKJl8FKtnBcYRuV7MFrqLsdjkfGO8XcWciIQzKn_2x33wb4kLGVeVKebU_72xVzqdgxCFSXNjO-tz3IF2DITByXxVGRoNv0y--sPxgldznDLSqUsoKBnpkjlnzA_rGUvTr7cy9r-Rj5uRDgO8pUFrptTldhlJxm5Dtfej_-jGuXysDkREBGs4SKond98WcKVYdIRWzjvY0Kgha7bdx6gmmysRdd2-5y_ffS6ruO-jikl82A-Bt2-7BjK2MdzNvDReD6SJ8zQXkG6_-oUPY_i0UR3PTidO71pORW4q0YxjVKD-si3wnot98A6HDCnCXkUAOgAA5SD8k0uzegeHL9Gbktqghuy9eL2PJyomSknxQoqQx9L4j2HsjOUkg70zIKA

獲得 thanos-querier 的 host 跟 path

oc -n openshift-monitoring get route

執行結果：

NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
alertmanager-main alertmanager-main-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw /api alertmanager-main web reencrypt/Redirect None
prometheus-k8s prometheus-k8s-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw /api prometheus-k8s web reencrypt/Redirect None
prometheus-k8s-federate prometheus-k8s-federate-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw /federate prometheus-k8s web reencrypt/Redirect None
thanos-querier thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw /api thanos-querier web reencrypt/Redirect None

查找 thanos-querier fqdn 對應的 ip

dig thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw +short

執行結果：

10.21.107.50

因當前操作的實體主機網路無法直接存取透過 route 對外的 openshift 服務，故建立 ssh tunnel
```
ssh -L 10000:10.21.107.50:443 ansible@10.21.104.120
```
輸入密碼登入，並執行 top 避免被自動登出，造成 ssh tunnel 終止

在本機 /etc/hosts 添加一筆名稱解析

sudo nano /etc/hosts

添加以下檔案內容：

127.0.0.1 thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw

在實體主機的 Google Chrome 瀏覽器安裝 ModHeader 插件

設定 ModHeader 插件

點選 +Mod，並選擇 Request headers，然後依序填入以下設定：

name: Authorization
value: 複製貼上第三步驟(獲取 monitor-reader `serviceAccount` 的 token)的執行結果

點選 Filter，並選擇 Request URL filters，然後輸入以下內容：

https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw:10000/.*

打開 Google Chrome 瀏覽器，連線測試是否可正常 query

https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw:10000/api/v1/query?query=up

確認網路可正常連線到 ocp 叢集裡面的 thanos-querier 服務後，查詢 elasticsearch 的 metrics name 確認是否有成功收集
```
https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw:10000/api/v1/query?query=elasticsearch_thread_pool_completed_count
```

5. 設定 Grafana 呈現 Prometheus 收集到的 Elasticsearch Metrics

打開 Google Chrome 瀏覽器，並連線至 Grafana
```
https://10.21.104.120
```
登入 Grafana
新增 Data Sources，點選左側選單 -> Connections -> Data Sources -> 右上角 Add new data source
選擇 prometheus

輸入 prometheus 連線資訊：

name: ocp-prometheus
Connection
Prometheus server URL: https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw
TLS settings
勾選 Skip TLS certificate validation
展開 HTTP headers
Add header
Header: Authorization
Value: 複製貼上第三步驟(獲取 monitor-reader `serviceAccount` 的 token)的執行結果

確認沒問題後，滑鼠滾到頁面最下面點選 Save & test 按鈕

確認有出現綠色區塊的連線成功資訊
匯入 DashBorad，點選左側選單 Dashboard -> Elastic Folder ->

修復 ocp 叢集憑證過期

Fri, 21 Nov 2025 05:30:24 +0800

1. 環境資訊

Red Hat OpenShift Container Platform (RHOCP) 4.19.10

2. 問題描述

ocp 叢集的美台節點都關機一個月以上，再次重新開機時，遇到 Control-plane 上的核心元件 (kube-apiserver, kube-scheduler…等)的憑證過期。

無法透過 oc 指令跟 kube-apiserver 溝通

oc get nodes

執行結果：

Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-11-21T11:47:17+08:00 is after 2025-10-17T04:33:31Z

3. 問題根原因

Control Plane 憑證已經過期超過一個月，由於叢集被關機超過 2 個月以上，導致憑證自動輪替 (Rotation) 機制失效時。

kube-apiserver 的 kubelet-client 憑證已過期，API Server 無法與 Kubelet 進行 TLS 驗證。

4. 診斷步驟

列出系統當前時間

TZ=GMT date

執行結果：

Fri Nov 21 03:52:02 AM GMT 2025

列出在 openshift 叢集中所有過期的憑證

echo -e "NAMESPACE\tNAME\tEXPIRY_DATE" && \
oc get secrets --insecure-skip-tls-verify -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | \
while read namespace name cert; do
if ! echo "$cert" | base64 -d | openssl x509 -noout -checkend 0 > /dev/null; then
enddate=$(echo "$cert" | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//')
echo -e "$namespace\t$name\t$enddate"
fi
done | column -t

執行結果：

NAMESPACE NAME EXPIRY_DATE
openshift-config-managed kube-controller-manager-client-cert-key Oct 17 04:33:39 2025 GMT
openshift-config-managed kube-scheduler-client-cert-key Oct 17 04:33:39 2025 GMT
openshift-kube-apiserver-operator aggregator-client-signer Sep 18 04:10:54 2025 GMT
openshift-kube-apiserver aggregator-client Sep 18 04:10:54 2025 GMT
openshift-kube-apiserver check-endpoints-client-cert-key Oct 17 04:33:39 2025 GMT
openshift-kube-apiserver control-plane-node-admin-client-cert-key Oct 17 04:33:35 2025 GMT
openshift-kube-apiserver external-loadbalancer-serving-certkey Oct 17 04:33:31 2025 GMT
openshift-kube-apiserver internal-loadbalancer-serving-certkey Oct 17 04:33:34 2025 GMT
openshift-kube-apiserver kubelet-client Oct 17 04:33:35 2025 GMT
openshift-kube-apiserver localhost-serving-cert-certkey Oct 17 04:33:31 2025 GMT
openshift-kube-apiserver service-network-serving-certkey Oct 17 04:33:31 2025 GMT
openshift-kube-controller-manager-operator csr-signer Sep 18 04:10:55 2025 GMT
openshift-kube-controller-manager-operator csr-signer-signer Sep 18 04:10:55 2025 GMT
openshift-kube-controller-manager csr-signer Sep 18 04:10:55 2025 GMT
openshift-kube-controller-manager kube-controller-manager-client-cert-key Oct 17 04:33:39 2025 GMT
openshift-kube-scheduler kube-scheduler-client-cert-key Oct 17 04:33:39 2025 GMT
openshift-operator-lifecycle-manager pprof-cert Sep 18 04:45:10 2025 GMT

4. 解決問題

觸發 OpenShift 叢集內所有 Master 和 Worker 節點的滾動式重新開機

oc adm reboot-machine-config-pool mcp/worker mcp/master \
--insecure-skip-tls-verify

執行結果：

machineconfig.machineconfiguration.openshift.io/95-oc-initiated-reboot-worker rolling reboot initiated
machineconfigpool.machineconfiguration.openshift.io/worker rolling reboot initiated
machineconfig.machineconfiguration.openshift.io/95-oc-initiated-reboot-master rolling reboot initiated
machineconfigpool.machineconfiguration.openshift.io/master rolling reboot initiated

檢視被 drian 的節點

oc get nodes --insecure-skip-tls-verify

執行結果：

NAME STATUS ROLES AGE VERSION
master-1 Ready control-plane,master 65d v1.32.7
master-2 NotReady,SchedulingDisabled control-plane,master 65d v1.32.7
master-3 Ready control-plane,master 65d v1.32.7
worker-1 Ready worker 65d v1.32.7
worker-2 Ready worker 65d v1.32.7
worker-3 NotReady,SchedulingDisabled worker 65d v1.32.7

如果節點很久都沒自動重新開機，請執行以下步驟：

檢視該節點上是否還有 pod 沒被驅逐

oc get pods --all-namespaces \
--insecure-skip-tls-verify \
--field-selector spec.nodeName=master-2

執行結果：

NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-cluster-node-tuning-operator tuned-wq2dg 1/1 Running 0 65d
openshift-dns dns-default-rk5nw 2/2 Running 0 65d
openshift-dns node-resolver-6b5xt 1/1 Running 0 65d
openshift-etcd etcd-master-2 5/5 Running 0 65d
openshift-etcd revision-pruner-10-master-2 0/1 Pending 0 64d
openshift-image-registry node-ca-hshgn 1/1 Running 0 65d
openshift-kube-apiserver kube-apiserver-master-2 5/5 Running 0 65d
openshift-kube-controller-manager kube-controller-manager-master-2 4/4 Running 2 (64d ago) 65d
openshift-kube-scheduler openshift-kube-scheduler-master-2 3/3 Running 0 65d
openshift-kube-scheduler revision-pruner-6-master-2 0/1 Pending 0 64d
openshift-machine-config-operator kube-rbac-proxy-crio-master-2 1/1 Running 12 (65d ago) 65d
openshift-machine-config-operator machine-config-daemon-vtgwl 2/2 Running 0 65d
openshift-machine-config-operator machine-config-server-tx5vx 1/1 Running 0 65d
openshift-monitoring node-exporter-vcfsd 2/2 Running 0 65d
openshift-multus multus-47z8l 1/1 Running 0 65d
openshift-multus multus-additional-cni-plugins-zc8vb 1/1 Running 0 65d
openshift-multus network-metrics-daemon-6h8k7 2/2 Running 0 65d
openshift-network-diagnostics network-check-target-f29nk 1/1 Running 0 65d
openshift-network-node-identity network-node-identity-cg7rr 2/2 Running 0 65d
openshift-network-operator iptables-alerter-gl975 1/1 Running 0 65d
openshift-ovn-kubernetes ovnkube-node-czhkg 8/8 Running 0 65d

手動 drain 節點看有沒有什麼錯誤訊息

oc adm drain master-2 --insecure-skip-tls-verify --ignore-daemonsets --delete-emptydir-data

執行結果：

node/master-2 already cordoned
error: unable to drain node "master-2" due to error: cannot delete cannot delete Pods that declare no controller (use --force to override): openshift-etcd/revision-pruner-10-master-2, openshift-kube-scheduler/revision-pruner-6-master-2, continuing command...
There are pending nodes to be drained:
master-2
cannot delete cannot delete Pods that declare no controller (use --force to override): openshift-etcd/revision-pruner-10-master-2, openshift-kube-scheduler/revision-pruner-6-master-2

revision-pruner Pod 是 OpenShift Operator 自動產生的清理工作 (Housekeeping)。

它們的作用是清理舊的 Static Pod 設定檔 (Revision)。

它們通常是暫時性的任務。如果因為 Drain 而被刪除，Operator 會在之後的協調循環 (Reconcile loop) 中，若有需要會再次建立它們，或者它們的任務其實早就結束了。

它們不承載任何使用者流量或重要資料。

因此，在這種情況下，使用 --force 強制將其刪除是標準操作，不會對叢集造成傷害。

強制驅逐該節點的 pods

oc adm drain master-2 \
--insecure-skip-tls-verify \
--ignore-daemonsets \
--delete-emptydir-data \
--force

執行結果：

node/master-2 already cordoned
Warning: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-wq2dg, openshift-dns/dns-default-rk5nw, openshift-dns/node-resolver-6b5xt, openshift-image-registry/node-ca-hshgn, openshift-machine-config-operator/machine-config-daemon-vtgwl, openshift-machine-config-operator/machine-config-server-tx5vx, openshift-monitoring/node-exporter-vcfsd, openshift-multus/multus-47z8l, openshift-multus/multus-additional-cni-plugins-zc8vb, openshift-multus/network-metrics-daemon-6h8k7, openshift-network-diagnostics/network-check-target-f29nk, openshift-network-node-identity/network-node-identity-cg7rr, openshift-network-operator/iptables-alerter-gl975, openshift-ovn-kubernetes/ovnkube-node-czhkg; deleting Pods that declare no controller: openshift-etcd/revision-pruner-10-master-2, openshift-kube-scheduler/revision-pruner-6-master-2
evicting pod openshift-kube-scheduler/revision-pruner-6-master-2
evicting pod openshift-etcd/revision-pruner-10-master-2
^C

以上命令如無法成功，請直些手動強制清除該 pod

oc delete pod revision-pruner-6-master-2 -n openshift-kube-scheduler --force --grace-period=0 --insecure-skip-tls-verify
oc delete pod revision-pruner-10-master-2 -n openshift-etcd --force --grace-period=0 --insecure-skip-tls-verify

再驅逐一次該節點的 pods

oc adm drain master-2 \
--insecure-skip-tls-verify \
--ignore-daemonsets \
--delete-emptydir-data \
--force

執行結果：

Warning: ignoring DaemonSet-managed Pods: openshift-cluster-node-tuning-operator/tuned-wq2dg, openshift-dns/dns-default-rk5nw, openshift-dns/node-resolver-6b5xt, openshift-image-registry/node-ca-hshgn, openshift-machine-config-operator/machine-config-daemon-vtgwl, openshift-machine-config-operator/machine-config-server-tx5vx, openshift-monitoring/node-exporter-vcfsd, openshift-multus/multus-47z8l, openshift-multus/multus-additional-cni-plugins-zc8vb, openshift-multus/network-metrics-daemon-6h8k7, openshift-network-diagnostics/network-check-target-f29nk, openshift-network-node-identity/network-node-identity-cg7rr, openshift-network-operator/iptables-alerter-gl975, openshift-ovn-kubernetes/ovnkube-node-czhkg
node/master-2 drained

Step by Step install OpenShift 4.18 with the Agent-based Installer (Online)

Tue, 18 Nov 2025 10:12:50 +0800

Preface

本篇文章會介紹，如何在多台 VM 上使用 agent-based 的方式安裝 3 台 Control-plane Node 和 3 台 Worker Node 架構的 OpenShift Container Platform 4.18

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

1. 先備知識

1.1. 什麼是 Agent-based Installer

Agent-based 安裝方式是一個 OpenShift 安裝工具的子命令，可以生成一張可開機的 ISO，讓你彈性地啟動本地伺服器，自行部署叢集，甚至適用於全離線網路（air‑gapped）環境。
它結合了 Assisted Installation 的便利與離線操作能力，整張 ISO 已包含部署叢集所需的所有資訊和 Release image。
此安裝方式支持 Zero Touch Provisioning（ZTP），可利用宣告式方式，自動化部署裸機設備到新的邊緣站點，同時設定格式與其他安裝方式一致。

1.2. 理解 Agent-based Installer 安裝流程

在 Control-plane node 中，其中一台會在開機一開始啟動 Assisted Service，並最終成為啟動節點（bootstrap host），這台節點被稱為 rendezvous host（節點 0）。
Assisted Service 會檢查所有節點是否符合安裝需求，並啟動 OpenShift Container Platform 叢集的部署流程；每個節點都會將 Red Hat Enterprise Linux CoreOS（RHCOS）映像寫入硬碟。
除了 rendezvous host 外的其他節點會重啟並開始安裝，rendezvous host 也會在稍後重啟並加入叢集；當所有節點都完成重啟並加入時，bootstrapping 完成，叢集部署成功。

可以透過 openshift-install agent create image 子命令，在離線環境安裝 OpenShift Container Platform，不同架構如下：

單節點 OpenShift 叢集（SNO） 一台同時包含 master 與 worker 身份的節點。
三節點緊湊型叢集 由三台 master 節點組成，同時也是 worker 節點。
高可用 OpenShift 叢集（HA） 至少三台 master 節點，搭配任意數量的 worker 節點，具備高可用架構。

1.3. 各台主機扮演的角色說明

主機名稱	角色和服務	說明
bastion	Cluster installer/bastion	發動安裝 OCP 的跳板機, 同時提供 load balance, DNS Server 的服務
master-1	rendezvous host/bootstrap	安裝時重要的角色, 透過 installer 將 ocp cluster 角色先部署在此, 再透過 scale out 延伸到 master node
master-[1-3]	Master node	ocp 重要的控制節點須為三個
worker-[1-3]	Worker/compute node	執行 Application 的節點
bastion	DNS Server	提供名稱解析和反解析(網址轉成 IP 或是將 IP 轉回網址)
bastion	HA Proxy	提供 load balance 的服務

2. 準備安裝

2.1. 先決條件

2.1.1. 硬體資源需求

2.1.2. 環境架構

請在虛擬化平台(Proxmox/VMWare…)產出以下 7 台 VM，並記錄每台 VM 對應的 Mac Address(除了 bastion)

1 台 bastion (RHEL)
3 台 master (RHCOS)
3 台 worker (RHCOS)

2.1.3. 主機名稱設定格式

HOSTNAME.CLUSTER_NAME.DOMAIN_NAME

example:

bastion.topgun.kubeantony.com

HOSTNAME 就是 bastion
CLUSTER_NAME 就是 topgun
DOMAIN_NAME 就是 kubeantony.com

2.1.4. 軟體授權

Red Hat Enterprise Linux subscription
- 60 天個人免費試用連結 : https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux/server/trial
Red Hat OpenShift Container Platform subscription
- 60 天個人免費試用連結 : https://www.redhat.com/en/technologies/cloud-computing/openshift/ocp-self-managed-trial

3. 安裝與設定 Bastion 主機

Step 0: 安裝與設定 Red Hat Enterprise Linux 9

Install Red Hat Enterprise Linux 9 - server-world

Step 1: 下載 “安裝 OCP 程式” 和 “管理 OpenShift 的 CLI 工具”

至 RedHat 官網連結下載以下兩個項目的檔案 (一定要有帳號)

OpenShift v4.18.13 Linux Installer
OpenShift v4.18.13 Linux Client

OVERSION="4.18.13"
wget --show-progress -qO "openshift-install-linux-${OVERSION}.tar.gz" "<installer 的下載網址>" && wget --show-progress -qO "oc-${OVERSION}-linux.tar.gz" "<Clinet 的下載網址>"

oc-4.18.13-linux.tar.gz，是 OpenShift CLI 的壓縮檔

openshift-install-linux-4.18.13.tar.gz，是安裝 OCP 會用到的程式

網址需用兩個雙引號包住，不然會報錯。

解壓縮檔案，並將 oc、openshift-install 和 kubectl 加至 PATH 環境變數

tar -xvf oc-${OVERSION}-linux.tar.gz && \
tar -xvf openshift-install-linux-${OVERSION}.tar.gz && \
sudo mv oc kubectl openshift-install /usr/local/bin

螢幕輸出 :

README.md
kubectl
oc
README.md
openshift-install

確認 oc Command 版本

oc version

螢幕輸出 :

Client Version: 4.18.13
Kustomize Version: v5.4.2

確認 openshift-install Command 版本

openshift-install version

螢幕輸出 :

openshift-install 4.18.13
built from commit 9357b668a760d53a34f7094840d1e9f773127441
release image quay.io/openshift-release-dev/ocp-release@sha256:a93c65b0f9de1d2e29641fbeebc07178733db1cacc7bde178033d7b9183540bc
release architecture amd64

確認 kubectl Command 版本

kubectl version --client --output=yaml

螢幕輸出 :

clientVersion:
buildDate: "2025-05-06T22:17:45Z"
compiler: gc
gitCommit: 35f7af703663a7459e0bc494e69ed2cc80543d04
gitTreeState: clean
gitVersion: v1.31.1
goVersion: go1.22.12 (Red Hat 1.22.12-2.el9_5) X:strictfipsruntime
major: "1"
minor: "31"
platform: linux/amd64
kustomizeVersion: v5.4.2

Step 2: 安裝 DNS Server

sudo yum -y install bind

螢幕輸出 :

...以上省略
Complete!

編輯 DNS Server 設定檔 named.conf

sudo nano /etc/named.conf

要改的地方有兩個部分

將 listen-on port 53 和 allow-query 的值, 改成 any

檔案內容 :

options {
listen-on port 53 { any; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
secroots-file "/var/named/data/named.secroots";
recursing-file "/var/named/data/named.recursing";
allow-query { any; };

假設我 bastion 這台機器的 Hostname 是 bastion.topgun.kubeantony.com IP 是 192.168.11.21，在 /etc/named.conf 新增以下內容至檔案的最後面 :

zone "kubeantony.com" {
type master;
file "/etc/named/zones/db.kubeantony.com";
};
zone "11.168.192.in-addr.arpa" {
type master;
file "/etc/named/zones/db.kubeantony.com.reverse";
};

在本次範例中

Cluster name 是 topgun
Base Domain Name 是 kubeantony.com

設定 DNS Server 名稱解析

sudo mkdir /etc/named/zones && \
sudo nano /etc/named/zones/db.kubeantony.com

檔案內容如下 :

$TTL 1W
@ IN SOA bastion.topgun.kubeantony.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS bastion.topgun.kubeantony.com.
IN MX 10 smtp.kubeantony.com.
;
;
bastion.topgun.kubeantony.com. IN A 192.168.11.21
smtp.kubeantony.com. IN A 192.168.11.21
;
api.topgun.kubeantony.com. IN A 192.168.11.21
api-int.topgun.kubeantony.com. IN A 192.168.11.21
;
*.apps.topgun.kubeantony.com. IN A 192.168.11.21
;
master-1.topgun.kubeantony.com. IN A 192.168.11.23
master-2.topgun.kubeantony.com. IN A 192.168.11.24
master-3.topgun.kubeantony.com. IN A 192.168.11.25
;
worker-1.topgun.kubeantony.com. IN A 192.168.11.26
worker-2.topgun.kubeantony.com. IN A 192.168.11.27
worker-3.topgun.kubeantony.com. IN A 192.168.11.28
;
;EOF

Note: 紅字部分為須依照環境的規劃來設定

設定 DNS Server 反解析

sudo nano /etc/named/zones/db.kubeantony.com.reverse

檔案內容如下 :

$TTL 1W
@ IN SOA bastion.topgun.kubeantony.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS bastion.topgun.kubeantony.com.
;
21.11.168.192.in-addr.arpa. IN PTR api.topgun.kubeantony.com.
21.11.168.192.in-addr.arpa. IN PTR api-int.topgun.kubeantony.com.
;
23.11.168.192.in-addr.arpa. IN PTR master-1.topgun.kubeantony.com.
24.11.168.192.in-addr.arpa. IN PTR master-2.topgun.kubeantony.com.
25.11.168.192.in-addr.arpa. IN PTR master-3.topgun.kubeantony.com.
;
26.11.168.192.in-addr.arpa. IN PTR worker-1.topgun.kubeantony.com.
27.11.168.192.in-addr.arpa. IN PTR worker-2.topgun.kubeantony.com.
28.11.168.192.in-addr.arpa. IN PTR worker-3.topgun.kubeantony.com.
;
;EOF

Note: 紅字部分為須依照環境的規劃來設定

啟動 DNS Server

sudo systemctl enable named --now

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/named.service → /usr/lib/systemd/system/named.service.

檢查

sudo systemctl status named

螢幕輸出 :

● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 16:50:31 CST; 1s ago
...以下省略

將 bastion 機器的 IP 新增為 DNS Server

sudo nmcli connection modify 'ens18' ipv4.dns '192.168.11.21' +ipv4.dns '8.8.8.8'
sudo systemctl restart NetworkManager

檢查

sudo cat /etc/resolv.conf

螢幕輸出

# Generated by NetworkManager
search topgun.kubeantony.com
nameserver 192.168.11.21
nameserver 8.8.8.8

關閉防火牆

sudo systemctl disable firewalld.service --now

檢查是否關閉

sudo systemctl status firewalld.service --no-pager

螢幕輸出 :

● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
...以下省略

驗證 DNS Server 正解

# 定義 A Zone File
AZF="/etc/named/zones/db.kubeantony.com"
# 從 DNS Zone File 中找出所有 A 記錄的主機名稱
ARecord=$(sudo grep -w 'A' $AZF | cut -d " " -f 1)
# 對每個主機名稱進行查詢，使用系統中第一個 nameserver 為查詢伺服器
for i in $ARecord; do dig @$(awk '/^nameserver/ {print $2; exit}' /etc/resolv.conf) ${i} +short; done

螢幕輸出 :

192.168.11.21
192.168.11.21
192.168.11.21
192.168.11.21
192.168.11.21
192.168.11.23
192.168.11.24
192.168.11.25
192.168.11.26
192.168.11.27
192.168.11.28

驗證 DNS Server 反解

# 指定 PTR Zone File 檔案路徑
PZF="/etc/named/zones/db.kubeantony.com.reverse"
# 擷取所有 PTR 記錄，將其轉換為標準 IPv4 格式，並去除重複
PTRRecord=$(sudo grep 'PTR' $PZF | awk '{split($1,a,"."); print a[4]"."a[3]"."a[2]"."a[1]}' | sort -u)
# 針對每個 IP 進行反解查詢（reverse DNS lookup），使用第一個 nameserver 查詢
for i in $PTRRecord; do dig -x ${i} @$(awk '/^nameserver/ {print $2; exit}' /etc/resolv.conf) +short; done

螢幕輸出 :

api-int.topgun.kubeantony.com.
api.topgun.kubeantony.com.
master-1.topgun.kubeantony.com.
master-2.topgun.kubeantony.com.
master-3.topgun.kubeantony.com.
worker-1.topgun.kubeantony.com.
worker-2.topgun.kubeantony.com.
worker-3.topgun.kubeantony.com.

Step 3: 安裝 HAProxy 服務

sudo yum -y install haproxy

螢幕輸出 :

...以上省略
Complete!

設定 HAProxy

直接清掉預設的設定檔

cat /dev/null | sudo tee /etc/haproxy/haproxy.cfg

再編輯設定檔

sudo nano /etc/haproxy/haproxy.cfg

將以下內容複製到檔案中 :

global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
defaults
mode http
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen api-server-6443
bind *:6443
mode tcp
server master-1 master-1.topgun.kubeantony.com:6443 check inter 1s
server master-2 master-2.topgun.kubeantony.com:6443 check inter 1s
server master-3 master-3.topgun.kubeantony.com:6443 check inter 1s
listen machine-config-server-22623
bind *:22623
mode tcp
server master-1 master-1.topgun.kubeantony.com:22623 check inter 1s
server master-2 master-2.topgun.kubeantony.com:22623 check inter 1s
server master-3 master-3.topgun.kubeantony.com:22623 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server worker-1 worker-1.topgun.kubeantony.com:443 check inter 1s
server worker-2 worker-2.topgun.kubeantony.com:443 check inter 1s
server worker-3 worker-3.topgun.kubeantony.com:443 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server worker-1 worker-1.topgun.kubeantony.com:80 check inter 1s
server worker-2 worker-2.topgun.kubeantony.com:80 check inter 1s
server worker-3 worker-3.topgun.kubeantony.com:80 check inter 1s

Note: 紅字部分為須依照環境的規劃來設定

設定允許 HAProxy 可以使用 TCP Port

sudo setsebool -P haproxy_connect_any=1

If you are using HAProxy as a load balancer and SELinux is set to enforcing, you must ensure that the HAProxy service can bind to the configured TCP port by running setsebool -P haproxy_connect_any=1.

啟動 HAProxy 服務，並設為開機自動啟動

sudo systemctl enable --now haproxy.service

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

檢查

sudo systemctl status haproxy.service

螢幕輸出 :

● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 17:26:21 CST; 11min ago
...以下省略

Step 4: 下載 pull-secret

至以下連結下載，注意 : 需先登入帳號

Download Pull secret

可點選 “Download pull secret"，或是點 “Copy pull secret” 直接將 pull-secret 複製到剪貼簿

Step5: 編輯必要叢集設定檔

產生目錄

mkdir ~/ocp4; cd ocp4

設定 install-config.yaml

nano ~/ocp4/install-config.yaml

檔案內容如下 :

apiVersion: v1
baseDomain: kubeantony.com
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: topgun
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 192.168.11.0/24
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '{"auths":{...}'
sshKey: '...'

`install-config.yaml` 欄位介紹

區塊	鍵名 / 值	說明
版本與網域	`apiVersion: v1`	設定檔格式版本
	`baseDomain: kubeantony.com`	集群基礎網域，與 metadata.name 組成完整 FQDN
	`metadata.name: topgun`	集群名稱，完整 FQDN 為 `topgun.kubeantony.com`
控制平面	`controlPlane.name: master`	控制節點名稱
	`controlPlane.replicas: 3`	控制節點數量
	`controlPlane.architecture: amd64`	架構
	`controlPlane.hyperthreading: Enabled`	是否啟用超執行緒
工作節點	`compute[0].name: worker`	工作節點名稱
	`compute[0].replicas: 3`	工作節點數量
	`compute[0].architecture: amd64`	架構
	`compute[0].hyperthreading: Enabled`	是否啟用超執行緒
網路設定	`networking.networkType: OVNKubernetes`	CNI 使用 OVN
	`clusterNetwork.cidr: 10.128.0.0/14`	Pod 網段
	`clusterNetwork.hostPrefix: 23`	每節點最大 Pod 數（約 512）
	`machineNetwork.cidr: 192.168.11.0/24`	節點網路範圍
	`serviceNetwork: 172.30.0.0/16`	Service 網段
平台	`platform.none: {}`	表示為裸機或 Agent-based 安裝
憑證與金鑰	`pullSecret: {...}`	Red Hat container registry 認證用
	`sshKey: '...'`	安裝後 SSH 公鑰

設定 `agent-config.yaml`

nano agent-config.yaml

檔案內容如下 :

apiVersion: v1alpha1
kind: AgentConfig
metadata:
name: topgun
rendezvousIP: 192.168.11.13
additionalNTPSources:
- 192.168.11.11
hosts:
- hostname: master-1
role: master
interfaces:
- name: ens18
macAddress: BC:24:11:B9:6B:C1
networkConfig:
interfaces:
- name: ens18
type: ethernet
state: up
mac-address: BC:24:11:B9:6B:C1
ipv4:
enabled: true
address:
- ip: 192.168.11.13
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.11.11
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.11.1
next-hop-interface: ens18
table-id: 254
rootDeviceHints:
deviceName: /dev/sda
- hostname: master-2
role: master
interfaces:
- name: ens18
macAddress: BC:24:11:5A:26:6D
networkConfig:
interfaces:
- name: ens18
type: ethernet
state: up
mac-address: BC:24:11:5A:26:6D
ipv4:
enabled: true
address:
- ip: 192.168.11.14
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.11.11
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.11.1
next-hop-interface: ens18
table-id: 254
rootDeviceHints:
deviceName: /dev/sda
- hostname: master-3
role: master
interfaces:
- name: ens18
macAddress: BC:24:11:2B:C6:4B
networkConfig:
interfaces:
- name: ens18
type: ethernet
state: up
mac-address: BC:24:11:2B:C6:4B
ipv4:
enabled: true
address:
- ip: 192.168.11.15
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.11.11
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.11.1
next-hop-interface: ens18
table-id: 254
rootDeviceHints:
deviceName: /dev/sda
- hostname: worker-1
role: worker
interfaces:
- name: ens18
macAddress: BC:24:11:7F:F2:B0
networkConfig:
interfaces:
- name: ens18
type: ethernet
state: up
mac-address: BC:24:11:7F:F2:B0
ipv4:
enabled: true
address:
- ip: 192.168.11.16
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.11.11
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.11.1
next-hop-interface: ens18
table-id: 254
rootDeviceHints:
deviceName: /dev/sda
- hostname: worker-2
role: worker
interfaces:
- name: ens18
macAddress: BC:24:11:BE:DC:4E
networkConfig:
interfaces:
- name: ens18
type: ethernet
state: up
mac-address: BC:24:11:BE:DC:4E
ipv4:
enabled: true
address:
- ip: 192.168.11.17
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.11.11
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.11.1
next-hop-interface: ens18
table-id: 254
rootDeviceHints:
deviceName: /dev/sda
- hostname: worker-3
role: worker
interfaces:
- name: ens18
macAddress: BC:24:11:05:55:16
networkConfig:
interfaces:
- name: ens18
type: ethernet
state: up
mac-address: BC:24:11:05:55:16
ipv4:
enabled: true
address:
- ip: 192.168.11.18
prefix-length: 24
dhcp: false
dns-resolver:
config:
server:
- 192.168.11.11
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 192.168.11.1
next-hop-interface: ens18
table-id: 254
rootDeviceHints:
deviceName: /dev/sda

備份設定檔

cp agent-config.yaml agent-config.yaml.bk
cp install-config.yaml install-config.yaml.bk

產生安裝用 ISO

需要先安裝 nmstae 套件

sudo yum install -y nmstate

再產生 ISO

openshift-install --dir ~/ocp4/ agent create image

螢幕輸出 :

INFO Configuration has 3 master replicas and 3 worker replicas
INFO The rendezvous host IP (node0 IP) is 192.168.1.91
INFO Extracting base ISO from release payload
INFO Verifying cached file
INFO Using cached Base ISO /home/bbg/.cache/agent/image_cache/coreos-x86_64.iso
INFO Consuming Install Config from target directory
INFO Consuming Agent Config from target directory
INFO Generated ISO at /home/bbg/ocp4/agent.x86_64.iso.

Step6: 開始安裝 RedHat OpenShift

1. 將 ISO 上傳至虛擬化平台，並掛載到 VM 上

注意！如果是在 vsphere 的環境，需要幫每一台 VM 新增以下參數：

點選 vm -> edit settings -> Advanced Parameters

Attribute: disk.EnableUUID
Value: true

然後點選 ADD

2. 在虛擬化平台將各節點透過 ISO 開機

3. 追蹤和驗證安裝進度 (在 Bastion 主機執行以下指令)

確認哪些節點已安裝完畢，並需要重新開機:

openshift-install --dir ~/ocp4/ agent wait-for bootstrap-complete --log-level=debug

注意! 以上指令會提示你哪台 VM 已將所有東西都安裝進硬碟裡面，然後進入自動重新開機的階段，當你看到類似的提示訊息後，大約過 30 秒 ~ 2 分鐘內，該主機會自動重開，請確保該 VM 會透過硬碟重新開機，不要再透過 ISO 又開機進去。

正確安裝的螢幕輸出如下 :

...
INFO Host: master-3, reached installation stage Writing image to disk: 100%
INFO Host: worker-3, reached installation stage Waiting for control plane
INFO Bootstrap Kube API Initialized
INFO Host: master-1, reached installation stage Waiting for control plane: Waiting for masters to join bootstrap control plane
INFO Uploaded logs for host master-2 cluster 84a2dfac-6f61-4de2-93e8-185d5e342f02
INFO Host: master-2, reached installation stage Rebooting
INFO Host: master-3, reached installation stage Rebooting
INFO Host: master-1, reached installation stage Waiting for bootkube
INFO Host: master-3, reached installation stage Done
INFO Node master-2 has been rebooted 1 times before completing installation
INFO Node master-3 has been rebooted 1 times before completing installation
INFO Host: worker-2, reached installation stage Rebooting
INFO Host: worker-1, reached installation stage Rebooting
INFO Host: master-1, reached installation stage Waiting for bootkube: waiting for ETCD bootstrap to be complete
INFO Bootstrap configMap status is complete
INFO Bootstrap is complete
INFO cluster bootstrap is complete

確認重開機後，OpenShift 自動安裝完成

openshift-install --dir ~/ocp4 agent wait-for install-complete --log-level=debug

螢幕輸出 :

...
INFO Cluster is installed
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run
INFO export KUBECONFIG=/home/bbg/ocp4/auth/kubeconfig
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.topgun.kubeantony.com
INFO Login to the console with user: "kubeadmin", and password: "JyAcY-VDp6D-DYdHp-T4Twr"

設定 KubeConfig

mkdir ~/.kube && \
cp ~/ocp4/auth/kubeconfig ~/.kube/config

檢查叢集節點狀態

oc get nodes

螢幕輸出

NAME STATUS ROLES AGE VERSION
master-1 Ready control-plane,master 25m v1.31.8
master-2 Ready control-plane,master 67m v1.31.8
master-3 Ready control-plane,master 66m v1.31.8
worker-1 Ready worker 29m v1.31.8
worker-2 Ready worker 32m v1.31.8
worker-3 Ready worker 24m v1.31.8

檢視整個叢集核心元件是否健康

oc get co

螢幕輸出 :

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.18.13 True False False 165m
baremetal 4.18.13 True False False 3h10m
cloud-controller-manager 4.18.13 True False False 3h13m
cloud-credential 4.18.13 True False False 3h14m
cluster-autoscaler 4.18.13 True False False 3h10m
config-operator 4.18.13 True False False 3h10m
console 4.18.13 True False False 172m
control-plane-machine-set 4.18.13 True False False 3h10m
csi-snapshot-controller 4.18.13 True False False 3h2m
dns 4.18.13 True False False 76s
etcd 4.18.13 True False False 3h8m
image-registry 4.18.13 True False False 3h1m
ingress 4.18.13 True False False 59s
insights 4.18.13 True False False 3h10m
kube-apiserver 4.18.13 True False False 3h7m
kube-controller-manager 4.18.13 True False False 3h7m
kube-scheduler 4.18.13 True False False 3h6m
kube-storage-version-migrator 4.18.13 True False False 3h2m
machine-api 4.18.13 True False False 3h9m
machine-approver 4.18.13 True False False 3h10m
machine-config 4.18.13 True False False 3h10m
marketplace 4.18.13 True False False 3h10m
monitoring 4.18.13 True False False 172m
network 4.18.13 True False False 3h10m
node-tuning 4.18.13 True False False 175m
olm 4.18.13 True False False 177m
openshift-apiserver 4.18.13 True False False 3h1m
openshift-controller-manager 4.18.13 True False False 3h6m
openshift-samples 4.18.13 True False False 3h1m
operator-lifecycle-manager 4.18.13 True False False 3h9m
operator-lifecycle-manager-catalog 4.18.13 True False False 3h9m
operator-lifecycle-manager-packageserver 4.18.13 True False False 3h2m
service-ca 4.18.13 True False False 3h10m
storage 4.18.13 True False False 3h10m

在 Openshift 中設定 Cluster Observability Operator 監控一個服務

Thu, 18 Sep 2025 08:06:14 +0800

0. 本篇文章 Openshift 環境資訊

Openshift version: 4.18.22
Cluster Observability Operator version : 1.2.2
- 安裝 Cluster Observability Operator 文件

1. Preface

可以透過設定由Cluster Observability Operator 所管理的監控堆疊 (monitoring stacks)，來監控服務的指標 (metrics)。

若要測試監控服務，請遵循以下步驟：

部署一個定義了 service endpoint 的範例服務。
建立一個 ServiceMonitor 物件，用以指定 COO 應如何監控該服務。
建立一個 MonitoringStack 物件來探索 (discover) 前述的 ServiceMonitor 物件。

2. 為 Cluster Observability Operator 部署範例服務

此設定檔會在使用者自訂的 ns1-coo 專案中，部署一個名為 prometheus-coo-example-app 的範例服務。該服務會公開 (expose) 自訂的 version 指標。

2.1. 先決條件

您能以具備 cluster-admin 叢集角色的使用者身分，或是以具 namespace 管理權限的使用者身分存取叢集。

2.2. 開始部署與設定

建立並切換專案目錄
```
mkdir sample; cd sample
```

建立一個名為 prometheus-coo-example-app.yaml 的 YAML 檔案，其中包含以下用於 namespace、deployment 和 service 的設定細節：

cat <<EOF > prometheus-coo-example-app.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ns1-coo
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: prometheus-coo-example-app
name: prometheus-coo-example-app
namespace: ns1-coo
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-coo-example-app
template:
metadata:
labels:
app: prometheus-coo-example-app
spec:
containers:
- image: ghcr.io/rhobs/prometheus-example-app:0.4.2
imagePullPolicy: IfNotPresent
name: prometheus-coo-example-app
---
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-coo-example-app
name: prometheus-coo-example-app
namespace: ns1-coo
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
name: web
selector:
app: prometheus-coo-example-app
type: ClusterIP
EOF

建立範例應用

oc apply -f prometheus-coo-example-app.yaml

確認 pod 狀態是 running

oc -n ns1-coo get pods

正確執行結果：

NAME READY STATUS RESTARTS AGE
prometheus-coo-example-app-57b7fcf6c6-w7b2b 1/1 Running 0 8s

好的，這段技術文件的繁體中文翻譯如下：

3. 指定服務如何由 Cluster Observability Operator 監控

要使用您在「為 Cluster Observability Operator 部署範例服務」一節中所建立的範例服務公開的指標 (metrics)，您必須設定監控元件以從 /metrics 端點 (endpoint) 抓取指標。

您可以透過建立一個 ServiceMonitor 物件來指定如何監控服務，或建立一個 PodMonitor 物件來指定如何監控 Pod，以完成此設定。ServiceMonitor 物件需要一個 Service 物件，而 PodMonitor 物件則不需要，這讓 MonitoringStack 物件能夠直接從 Pod 公開的指標端點抓取指標。

此程序展示如何為 ns1-coo namespace 中一個名為 prometheus-coo-example-app 的範例服務建立 ServiceMonitor 物件。

3.1. 先決條件

您需以具有 cluster-admin 叢集角色 (cluster role) 的使用者身分，或具有該命名空間管理權限的使用者身分存取叢集。
您已安裝 Cluster Observability Operator。
您已在 ns1-coo 命名空間中部署 prometheus-coo-example-app 範例服務。

3.2. 開始部署與設定

建立一個名為 example-coo-app-service-monitor.yaml 的 YAML 檔案，其中包含以下 ServiceMonitor 物件的設定詳細資訊：

定義一個 ServiceMonitor 物件

MonitoringStack 物件將會引用它，以抓取 (scrape) 由 prometheus-coo-example-app 範例服務所暴露 (exposed) 的指標資料

cat <<EOF > example-coo-app-service-monitor.yaml
apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: prometheus-coo-example-monitor
name: prometheus-coo-example-monitor
namespace: ns1-coo
spec:
endpoints:
- interval: 30s
port: web
scheme: http
selector:
matchLabels:
app: prometheus-coo-example-app
EOF

透過執行以下指令，將此設定應用至叢集：
```
oc apply -f example-coo-app-service-monitor.yaml
```
透過執行以下指令並觀察輸出結果，來驗證 ServiceMonitor 資源是否已成功建立：
```
oc -n ns1-coo get servicemonitors.monitoring.rhobs
```
執行結果：
```
NAME AGE
prometheus-coo-example-monitor 77s
```

4. 為 Cluster Observability Operator 建立一個 `MonitoringStack` 物件

若要抓取由目標 prometheus-coo-example-app 服務所暴露的指標資料，您需要建立一個 MonitoringStack 物件，該物件需引用您在「為 Cluster Observability Operator 指定服務的監控方式」一節中所建立的 ServiceMonitor 物件。接著，這個 MonitoringStack 物件便能發現該服務，並從中抓取其暴露的指標資料。

4.1. 先決條件

您需以具有 cluster-admin 叢集角色的使用者身分，或是在該命名空間中具有管理權限的使用者身分存取叢集。
您已安裝叢集可觀察性 Operator (Cluster Observability Operator)。
您已在 ns1-coo 命名空間中部署了 prometheus-coo-example-app 範例服務。
您已在 ns1-coo 命名空間中建立了一個名為 prometheus-coo-example-monitor 的 ServiceMonitor 物件。

4.2. 開始部署與設定

為 MonitoringStack 物件的設定建立一個 YAML 檔案。在此範例中，將檔案命名為 example-coo-monitoring-stack.yaml。

cat <<EOF > example-coo-monitoring-stack.yaml
apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
name: example-coo-monitoring-stack
namespace: ns1-coo
spec:
logLevel: debug
retention: 1d
resourceSelector:
matchLabels:
k8s-app: prometheus-coo-example-monitor
EOF

加入以下 MonitoringStack 物件的設定詳細資訊：
```
oc apply -f example-coo-monitoring-stack.yaml
```
執行以下指令並檢視輸出，以驗證 MonitoringStack 物件是否可用：
```
oc -n ns1-coo get monitoringstack
```
執行結果：
```
NAME AGE
example-coo-monitoring-stack 9s
```

執行以下指令，從 Prometheus 擷取作用中目標 (active targets) 的資訊，並篩選輸出，僅列出標籤為 app=prometheus-coo-example-app 的目標。此舉可以驗證帶有此特定標籤的目標，確實已被 Prometheus 發現並正在主動監控。

oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'

正確執行結果：

{
"__address__": "10.247.0.35:8080",
"__meta_kubernetes_endpoint_address_target_kind": "Pod",
"__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-57b7fcf6c6-w7b2b",
"__meta_kubernetes_endpoint_node_name": "worker-3",
"__meta_kubernetes_endpoint_port_name": "web",
"__meta_kubernetes_endpoint_port_protocol": "TCP",
"__meta_kubernetes_endpoint_ready": "true",
"__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2025-09-18T06:21:52Z",
"__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true",
"__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app",
"__meta_kubernetes_endpoints_labelpresent_app": "true",
"__meta_kubernetes_endpoints_name": "prometheus-coo-example-app",
"__meta_kubernetes_namespace": "ns1-coo",
"__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.247.0.35/23\"],\"mac_address\":\"0a:58:0a:f7:00:23\",\"gateway_ips\":[\"10.247.0.1\"],\"routes\":[{\"dest\":\"10.244.0.0/14\",\"nextHop\":\"10.247.0.1\"},{\"dest\":\"10.98.0.0/16\",\"nextHop\":\"10.247.0.1\"},{\"dest\":\"169.254.0.5/32\",\"nextHop\":\"10.247.0.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.247.0.1\"}],\"ip_address\":\"10.247.0.35/23\",\"gateway_ip\":\"10.247.0.1\",\"role\":\"primary\"}}",
"__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.247.0.35\"\n ],\n \"mac\": \"0a:58:0a:f7:00:23\",\n \"default\": true,\n \"dns\": {}\n}]",
"__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2",
"__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default",
"__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true",
"__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true",
"__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true",
"__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true",
"__meta_kubernetes_pod_controller_kind": "ReplicaSet",
"__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-57b7fcf6c6",
"__meta_kubernetes_pod_host_ip": "10.10.7.18",
"__meta_kubernetes_pod_ip": "10.247.0.35",
"__meta_kubernetes_pod_label_app": "prometheus-coo-example-app",
"__meta_kubernetes_pod_label_pod_template_hash": "57b7fcf6c6",
"__meta_kubernetes_pod_labelpresent_app": "true",
"__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
"__meta_kubernetes_pod_name": "prometheus-coo-example-app-57b7fcf6c6-w7b2b",
"__meta_kubernetes_pod_node_name": "worker-3",
"__meta_kubernetes_pod_phase": "Running",
"__meta_kubernetes_pod_ready": "true",
"__meta_kubernetes_pod_uid": "275239d6-4353-4f0d-8887-4b0046efd0df",
"__meta_kubernetes_service_label_app": "prometheus-coo-example-app",
"__meta_kubernetes_service_labelpresent_app": "true",
"__meta_kubernetes_service_name": "prometheus-coo-example-app",
"__metrics_path__": "/metrics",
"__scheme__": "http",
"__scrape_interval__": "30s",
"__scrape_timeout__": "10s",
"job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0"
}

5. Validating the monitoring stack

若要驗證 monitoring stack 是否正常運作，請存取範例服務，然後檢視收集到的指標資料。

5.1. 先決條件

您需以具有 cluster-admin 叢集角色的使用者身分，或是在該命名空間中具有管理權限的使用者身分存取叢集。
您已安裝叢集可觀察性 Operator (Cluster Observability Operator)。
您已在 ns1-coo 命名空間中部署了 prometheus-coo-example-app 範例服務。
您已在 ns1-coo 命名空間中建立了一個名為 prometheus-coo-example-monitor 的 ServiceMonitor 物件。
您已在 ns1-coo 命名空間中建立了一個名為 example-coo-monitoring-stack 的 MonitoringStack 物件。

5.2. 開始驗證

建立一個路由 (Route) 以暴露 (expose) prometheus-coo-example-app 範例服務。請在您的終端機中執行以下指令：
```
oc expose svc prometheus-coo-example-app -n ns1-coo
```

檢視 route 物件

oc -n ns1-coo get route

執行結果：

NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
prometheus-coo-example-app prometheus-coo-example-app-ns1-coo.apps.topgun.example.com prometheus-coo-example-app web None

透過您的瀏覽器或命令列存取此路由，以產生指標資料

curl http://prometheus-coo-example-app-ns1-coo.apps.topgun.example.com; echo

執行結果：

Hello from example application.

在 Prometheus Pod 上執行查詢，以回傳 HTTP 請求總數 (total HTTP requests) 的指標：

oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total' | jq

執行結果：

{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"__name__": "http_requests_total",
"code": "200",
"endpoint": "web",
"instance": "10.247.0.35:8080",
"job": "prometheus-coo-example-app",
"method": "get",
"namespace": "ns1-coo",
"pod": "prometheus-coo-example-app-57b7fcf6c6-w7b2b",
"service": "prometheus-coo-example-app"
},
"value": [
1758178540.022,
"5"
]
},
{
"metric": {
"__name__": "http_requests_total",
"code": "404",
"endpoint": "web",
"instance": "10.247.0.35:8080",
"job": "prometheus-coo-example-app",
"method": "get",
"namespace": "ns1-coo",
"pod": "prometheus-coo-example-app-57b7fcf6c6-w7b2b",
"service": "prometheus-coo-example-app"
},
"value": [
1758178540.022,
"0"
]
}
]
}
}

6. 抓取多個 namespace 中的目標

若要抓取多個命名空間中的目標，請在 MonitoringStack 物件中設定 namespace 與 resource selector 。

6.1. 先決條件

您需要以具有 cluster-admin 叢集角色的使用者，或具有該 namespace 管理權限的使用者身分存取叢集。
您已安裝 Cluster Observability Operator。

6.2. 開始部署與設定

部署以下的 namespace 與 MonitoringStack YAML 檔案：

cat <<EOF > example-coo-monitoring-stack-v2.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ns1-coo
labels:
monitoring.rhobs/stack: multi-ns
---
apiVersion: monitoring.rhobs/v1alpha1
kind: MonitoringStack
metadata:
name: example-coo-monitoring-stack
namespace: ns1-coo
spec:
logLevel: debug
retention: 1d
resourceSelector:
matchLabels:
k8s-app: prometheus-coo-example-monitor
namespaceSelector:
matchLabels:
monitoring.rhobs/stack: multi-ns
EOF

建立 MonitoringStack

oc apply -f example-coo-monitoring-stack-v2.yaml

在 ns1-coo 命名空間中部署一個範例應用程式，該應用程式帶有一個持續觸發的警報：

cat <<EOF > prometheus-coo-example-app-v2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: prometheus-coo-example-app
name: prometheus-coo-example-app
namespace: ns1-coo
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-coo-example-app
template:
metadata:
labels:
app: prometheus-coo-example-app
spec:
containers:
- image: ghcr.io/rhobs/prometheus-example-app:0.4.2
imagePullPolicy: IfNotPresent
name: prometheus-coo-example-app
---
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-coo-example-app
name: prometheus-coo-example-app
namespace: ns1-coo
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
name: web
selector:
app: prometheus-coo-example-app
type: ClusterIP
---
apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: prometheus-coo-example-monitor
name: prometheus-coo-example-monitor
namespace: ns1-coo
spec:
endpoints:
- interval: 30s
port: web
scheme: http
selector:
matchLabels:
app: prometheus-coo-example-app
---
apiVersion: monitoring.rhobs/v1
kind: PrometheusRule
metadata:
name: example-alert
namespace: ns1-coo
labels:
k8s-app: prometheus-coo-example-monitor
spec:
groups:
- name: example
rules:
- alert: VersionAlert
for: 1m
expr: version{job="prometheus-coo-example-app"} > 0
labels:
severity: warning
EOF

重新部署範例應用

oc apply -f prometheus-coo-example-app-v2.yaml

在另一個標記為 monitoring.rhobs/stack: multi-ns 的 namespace 中部署相同的範例應用程式：

cat <<EOF > prometheus-coo-example-app-ns2.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ns2-coo
labels:
monitoring.rhobs/stack: multi-ns
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: prometheus-coo-example-app
name: prometheus-coo-example-app
namespace: ns2-coo
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-coo-example-app
template:
metadata:
labels:
app: prometheus-coo-example-app
spec:
containers:
- image: ghcr.io/rhobs/prometheus-example-app:0.4.2
imagePullPolicy: IfNotPresent
name: prometheus-coo-example-app
---
apiVersion: v1
kind: Service
metadata:
labels:
app: prometheus-coo-example-app
name: prometheus-coo-example-app
namespace: ns2-coo
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
name: web
selector:
app: prometheus-coo-example-app
type: ClusterIP
---
apiVersion: monitoring.rhobs/v1
kind: ServiceMonitor
metadata:
labels:
k8s-app: prometheus-coo-example-monitor
name: prometheus-coo-example-monitor
namespace: ns2-coo
spec:
endpoints:
- interval: 30s
port: web
scheme: http
selector:
matchLabels:
app: prometheus-coo-example-app
EOF

部署新的範例應用

oc apply -f prometheus-coo-example-app-ns2.yaml

4.5.3. 驗證

請驗證 Prometheus 實例已新增目標，且警報正在觸發。您可以使用連接埠轉發 (port-forward) 指令，來公開由 MonitoringStack 實例所部署的 Prometheus 或 Alertmanager 使用者介面。

Prometheus

oc port-forward -n ns1-coo pod/prometheus-example-coo-monitoring-stack-0 9090

打開瀏覽器連接至 http://localhost:9090/targets 驗證目標是否正在被抓取

Alertmanager

oc port-forward -n ns1-coo pod/alertmanager-example-coo-monitoring-stack-0 9093

http://localhost:9093/#/alerts，來驗證目標警報是否正在觸發

7. 參考資料

Configuring the Cluster Observability Operator to monitor a service - Red Hat Docs

Install Red Hat Openshift Logging on ocp 4.18

Thu, 18 Sep 2025 08:04:36 +0800

1. 先備知識

1.1. Openshift Logging 簡介

OpenShift Container Platform 的 Operator 使用自訂資源 (Custom Resources, CRs) 來管理應用程式及其元件。您可以透過 CR 提供高階的配置與設定。Operator 會根據其邏輯中內建的最佳實踐，將高階指令轉譯為低階操作。自訂資源定義 (Custom Resource Definition, CRD) 則會定義一個 CR，並列出所有供該 Operator 使用者使用的配置。安裝 Operator 時會建立 CRD，以便用來產生 CR。

若要開始使用日誌功能 (logging)，您必須安裝下列 Operator：

Loki Operator：用於管理您的日誌儲存。
Red Hat OpenShift Logging Operator：用於管理日誌的收集與轉發。
Cluster Observability Operator (COO)：用於管理視覺化。

您可以使用 OpenShift Container Platform 網站主控台或 OpenShift 容器平台命令列介面 (CLI) 來安裝或設定日誌功能。

重要提示：必須在設定 Loki Operator 之後，才能設定 Red Hat OpenShift Logging Operator。

1.2. 先決條件

如果是在離線環境的 openshift，請確定有正確設定 catalogsources CR 和 imagecontentsourcepolicies CR，可透過以下兩個命令檢查

確認 catalogsources

oc -n openshift-marketplace get catalogsources.operators.coreos.com

執行結果：

NAME DISPLAY TYPE PUBLISHER AGE
cs-redhat-operator-index grpc 2d17h

確認對應的 operator pod 正常運作

oc -n openshift-marketplace get pods -l "olm.catalogSource=cs-redhat-operator-index"

執行結果：

NAME READY STATUS RESTARTS AGE
cs-redhat-operator-index-wm57c 1/1 Running 0 2d17h

確認 imagecontentsourcepolicies 有定義 openshift-logging operator 的 image 要去哪一個本地的 image registry 下載

oc -n openshift-marketplace get imagecontentsourcepolicies.operator.openshift.io operator-0 -o json \
| jq -r '"SOURCE\tMIRRORS", (.spec.repositoryDigestMirrors[] | "\(.source)\t\(.mirrors | join(", "))")' \
| column -ts $'\t'

執行結果：

SOURCE MIRRORS
registry.redhat.io/cluster-observability-operator quay.kubeantony.com:8443/cluster-observability-operator
registry.redhat.io/rhosdt quay.kubeantony.com:8443/rhosdt
registry.redhat.io/rhel8 quay.kubeantony.com:8443/rhel8
registry.redhat.io/openshift-logging quay.kubeantony.com:8443/openshift-logging
registry.redhat.io/openshift4 quay.kubeantony.com:8443/openshift4
registry.redhat.io/quay quay.kubeantony.com:8443/quay

查詢叢集中，所有可從 OperatorHub 安裝的 Operator 套件，並檢查是否有 cluster-logging

oc get packagemanifests.packages.operators.coreos.com

執行結果：

NAME CATALOG AGE
elasticsearch-operator 2d22h
opentelemetry-product 2d22h
loki-operator 2d22h
cluster-logging 2d22h
quay-operator 2d22h
cluster-observability-operator 2d22h

2. 使用 CLI 安裝 Loki Operator

2.1. 先決條件

您擁有管理員權限。
您已安裝 OpenShift 命令列介面 (oc)。
您能夠存取一個受支援的物件儲存空間。例如：AWS S3、Google Cloud Storage、Azure、Swift、Minio 或 OpenShift Data Foundation。
預先準備好要安裝的 loki size 對應的硬體資源

2.2. 開始安裝

建立工作目錄，並切換當前目錄至新建立的工作目錄
```
mkdir -p ~/wulin/logging/loki;
cd ~/wulin/logging/loki
```

幫 Loki Operator 建立 Namespace

cat <<EOF > loki-operator-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: openshift-operators-redhat
labels:
openshift.io/cluster-monitoring: "true"
EOF

建立 Namespace：
```
oc apply -f loki-operator-ns.yaml
```

建立 OperatorGroup 物件

cat <<EOF > loki-operator-group.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: loki-operator
namespace: openshift-operators-redhat
spec:
upgradeStrategy: Default
EOF

必須指定 openshift-operators-redhat 作為 namespace。

建立 OperatorGroup
```
oc apply -f loki-operator-group.yaml
```
幫 Loki Operator 建立 Subscription
```
cat <<EOF > loki-operator-subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: loki-operator
namespace: openshift-operators-redhat
spec:
channel: stable-6.3
installPlanApproval: Manual
name: loki-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
EOF
```
1. 您必須將 openshift-operators-redhat 指定為 namespace。
2. 將 stable-6.3 指定為頻道。
3. 如果訂閱中的核准策略設定為 Automatic (自動)，那麼一旦所選的頻道中有新的 Operator 版本可用，更新程序就會立即啟動。如果核准策略設定為 Manual (手動)，您就必須手動核准待處理的更新。
4. 將 redhat-operators 指定為其值。如果您的 OpenShift Container Platform 叢集安裝在受限網路 (也稱為離線叢集) 中，請指定您在設定 Operator Lifecycle Manager (OLM) 時所建立的 CatalogSource 物件名稱。

建立 Subscription

oc apply -f loki-operator-subscription.yaml

檢查是否有待處理的 Operator 安裝更新計畫
```
oc -n openshift-operators-redhat get installplans
```
執行結果：
```
NAME CSV APPROVAL APPROVED
install-w2988 loki-operator.v6.3.0 Manual false
```
如果沒看到 installplans，去看 openshift-marketplace namespace 底下應該會有一個 job pod 噴錯

手動核准待處理的安裝計畫

oc patch installplan $(oc -n openshift-operators-redhat get installplans.operators.coreos.com -o name | cut -d "/" -f 2) \
-n openshift-operators-redhat \
--type merge \
--patch '{"spec":{"approved":true}}'

查詢 Operator 目前的版本狀態

oc -n openshift-operators-redhat get csv

正確執行結果：

NAME DISPLAY VERSION REPLACES PHASE
loki-operator.v6.3.0 Loki Operator 6.3.0 Succeeded

確認 operator pod 狀態

oc -n openshift-operators-redhat get pods

執行結果：

NAME READY STATUS RESTARTS AGE
loki-operator-controller-manager-5bff6f97d-7wtnz 2/2 Running 0 16m

幫 LokiStack 建 namespace
```
cat <<EOF > lokistack-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: openshift-logging
labels:
openshift.io/cluster-monitoring: "true"
EOF
```
1. openshift-logging namespace 專門用於所有 logging 的 workload。
2. openshift.io/cluster-monitoring: "true" 一個字串值，用於指定標籤，以確保 cluster monitoring 能夠抓取 openshift-logging namespace。
建立 namespace
```
oc apply -f lokistack-ns.yaml
```

建立一個 Secret，其中包含用以存取 object storage 空間的憑證。例如，建立一個用以存取 Minio s3 的 Secret。

cat <<EOF > lokistack-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: logging-loki-s3
namespace: openshift-logging
stringData:
access_key_id: admin
access_key_secret: admin123
bucketnames: test
endpoint: https://antony-minio.example.com
region: us-west-2
EOF

建立 Secret
```
oc apply -f lokistack-secret.yaml
```
建立 LokiStack CR
```
cat <<EOF > lokistack-cr.yaml
apiVersion: loki.grafana.com/v1
kind: LokiStack
metadata:
name: logging-loki
namespace: openshift-logging
spec:
size: 1x.extra-small
storage:
schemas:
- version: v13
effectiveDate: "2025-09-11"
secret:
name: logging-loki-s3
type: s3
storageClassName: managed-nfs-storage
tenants:
mode: openshift-logging
EOF
```
1. metadata.name: 請使用名稱 logging-loki。
2. metadata.namespace: 必須將 openshift-logging 指定為 namespace。
3. spec.size: 指定部署規模。Loki 生產環境執行個體支援的規模選項為 1x.extra-small、1x.small 或 1x.medium。此外，自 logging 6.1 版起也支援 1x.pico。
4. spec.storage.schemas.effectiveDate: 對於新的安裝，此日期應設定為相當於「昨天」的日期，因為這將會是 schema (結構綱要) 生效的起始日期。
5. spec.storage.secret.name: 指定您的日誌儲存 Secret 名稱。
6. spec.storage.secret.type: 指定對應的儲存類型。
7. spec.storageClassName: 指定一個用於暫存空間的儲存類別 (Storage Class) 名稱。為獲得最佳效能，請指定一個會分配 block storage 的儲存類別。可以使用 oc get storageclasses 指令來列出叢集所有可用的儲存類別。
8. openshift-logging 模式是預設的租用模式 (tenancy mode)，此模式會為稽核 (audit)、基礎架構 (infrastructure) 和應用程式 (application) 等日誌類型建立不同的租戶 (tenant)。這使得個別使用者及使用者群組對不同日誌串流 (log streams) 的存取控制成為可能。
建立 LokiStack CR
```
oc apply -f lokistack-cr.yaml
```
切換 current project
```
oc project openshift-logging
```

確認 LokiStack pod 狀態

oc get pods

正確執行結果：

NAME READY STATUS RESTARTS AGE
logging-loki-compactor-0 1/1 Running 0 3m37s
logging-loki-distributor-6bcbc8bc5b-bwjlj 1/1 Running 0 3m38s
logging-loki-distributor-6bcbc8bc5b-bznfk 1/1 Running 0 3m38s
logging-loki-gateway-6f9657d5f7-5mrfd 2/2 Running 0 3m37s
logging-loki-gateway-6f9657d5f7-v4h75 2/2 Running 1 (3m31s ago) 3m37s
logging-loki-index-gateway-0 1/1 Running 0 3m37s
logging-loki-index-gateway-1 1/1 Running 0 3m16s
logging-loki-ingester-0 1/1 Running 0 3m38s
logging-loki-ingester-1 1/1 Running 0 2m35s
logging-loki-querier-75c5755c58-c582z 1/1 Running 0 3m38s
logging-loki-querier-75c5755c58-wtfv7 1/1 Running 0 3m38s
logging-loki-query-frontend-7894d6df5c-g7vpj 1/1 Running 0 3m37s
logging-loki-query-frontend-7894d6df5c-hlt4v 1/1 Running 0 3m37s

3. 使用 CLI 安裝 Red Hat OpenShift Logging Operator 6.3

請使用 OpenShift CLI (oc)，將 Red Hat OpenShift Logging Operator 安裝至 OpenShift Container Platform 叢集，以收集日誌並將其轉發至日誌儲存區。

3.1. 先決條件

擁有管理員權限。
已安裝 OpenShift CLI (oc)。
已安裝並設定 Loki Operator。
已建立 openshift-logging 命名空間。

3.2. 開始安裝

建立工作目錄，並切換當前目錄至新建立的工作目錄

mkdir -p ~/wulin/logging/openshift-logging;
cd ~/wulin/logging/openshift-logging

建立 OperatorGroup YAML 檔

cat <<EOF > logging-operator-group.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: cluster-logging
namespace: openshift-logging
spec:
upgradeStrategy: Default
EOF

建立 OperatorGroup
```
oc apply -f logging-operator-group.yaml
```

為 Red Hat OpenShift Logging Operator 建立 Subscription YAML 檔

cat <<EOF > logging-operator-subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: cluster-logging
namespace: openshift-logging
spec:
channel: stable-6.3
installPlanApproval: Automatic
name: cluster-logging
source: cs-redhat-operator-index
sourceNamespace: openshift-marketplace
EOF

建立 Subscription

oc apply -f logging-operator-subscription.yaml

確認 operator pod 已啟動

oc get pods

執行結果：

NAME READY STATUS RESTARTS AGE
cluster-logging-operator-86bf5bc9b6-fxf28 1/1 Running 0 17h

3. 使用 CLI 安裝 logging UI plugin

3.1. 先決條件

您擁有管理員權限。
您已安裝 OpenShift CLI (oc)。
您已安裝並設定 Loki Operator。

3.2. 開始安裝 Cluster Observability Operator

Cluster Observability Operator 是 OpenShift Container Platform 的一個選用元件，專為建立和管理高度可自訂的監控堆疊而設計。它使叢集管理員能夠廣泛地自動化設定和管理監控需求，與預設的 OpenShift Container Platform 監控系統相比，它能為每個命名空間 (namespace) 提供更客製化且詳細的視圖。

COO 會部署以下監控元件：

Prometheus - 一個高可用性的 Prometheus 實例，能夠使用遠端寫入 (remote write) 將指標 (metrics) 傳送到外部端點。
Thanos Querier (選用) - 能夠從一個中央位置查詢多個 Prometheus 實例。
Alertmanager (選用) - 為不同服務提供警報設定功能。
UI plugins (選用) - 透過用於監控、日誌記錄、分散式追蹤和疑難排解的外掛程式來增強可觀測性。
Korrel8r (選用) - 提供可觀測性訊號的關聯性分析，由開源的 Korrel8r 專案提供支援。
Incident detection (選用) - 將相關的警報分組為事件，以協助您識別警報爆發的根本原因。

建立工作目錄，並切換當前目錄至新建立的工作目錄
```
mkdir -p ~/wulin/logging/coo;
cd ~/wulin/logging/coo
```

建立 OperatorGroup YAML 檔

cat <<EOF > coo-operator-group.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: openshift-cluster-observability-operator
namespace: openshift-cluster-observability-operator
spec:
upgradeStrategy: Default
EOF

建立 OperatorGroup
```
oc apply -f coo-operator-group.yaml
```

為 Cluster Observability Operator 建立 Subscription YAML 檔

cat <<EOF > coo-subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: cluster-observability-operator
namespace: openshift-cluster-observability-operator
spec:
channel: stable
installPlanApproval: Manual
name: cluster-observability-operator
source: cs-redhat-operator-index
sourceNamespace: openshift-marketplace
startingCSV: cluster-observability-operator.v1.2.2
EOF

建立 Subscription
```
oc apply -f coo-subscription.yaml
```

檢查是否有待處理的 Operator 安裝更新計畫

oc -n openshift-cluster-observability-operator get installplans.operators.coreos.com

執行結果：

NAME CSV APPROVAL APPROVED
install-cxddp cluster-observability-operator.v1.2.2 Manual false

如果沒看到 installplans，去看 openshift-marketplace namespace 底下應該會有一個 job pod 噴錯

手動核准待處理的安裝計畫

oc patch installplan $(oc -n openshift-cluster-observability-operator get installplans.operators.coreos.com -o name | cut -d "/" -f 2) \
-n openshift-cluster-observability-operator \
--type merge \
--patch '{"spec":{"approved":true}}'

查詢 Operator 目前的版本狀態

oc -n openshift-cluster-observability-operator get csv

正確執行結果：

NAME DISPLAY VERSION REPLACES PHASE
cluster-logging.v6.3.0 Red Hat OpenShift Logging 6.3.0 Succeeded
cluster-observability-operator.v1.2.2 Cluster Observability Operator 1.2.2 cluster-observability-operator.v1.2.1 Succeeded
loki-operator.v6.3.0 Loki Operator 6.3.0 Succeeded

確認 operator pod 狀態

oc -n openshift-cluster-observability-operator get pods

執行結果：

NAME READY STATUS RESTARTS AGE
obo-prometheus-operator-54d44bf564-czp56 1/1 Running 0 19s
obo-prometheus-operator-admission-webhook-86c9c76864-qg9hb 1/1 Running 0 18s
obo-prometheus-operator-admission-webhook-86c9c76864-xbpcz 1/1 Running 0 19s
observability-operator-77c8cf97cb-6qqqq 1/1 Running 0 18s
perses-operator-bfd8678b7-czxlk 1/1 Running 0 17s

建立 UIPlugin CR

cat <<EOF > coo-ui-plugin.yaml
apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
name: logging
spec:
type: Logging
logging:
lokiStack:
name: logging-loki
logsLimit: 50
timeout: 30s
schema: otel
EOF

建立 UIPlugin
```
oc apply -f coo-ui-plugin.yaml
```
驗證：
- 前往 Red Hat OpenShift Logging 網站主控台，若有彈出式訊息提示，請重新整理頁面。
- 導覽至 Observe > Logs 面板，您可以在此執行 LogQL 查詢。您也可以從特定 Pod 的 Aggregated Logs 頁籤中，查詢該 Pod 的個別日誌。

4. 參考資料

Installing Logging - Red Hat Docs

設定 Openshift audit log 透過 openshift-logging 拋轉到 Elasticsearch

Thu, 18 Sep 2025 03:30:48 +0800

1. 先決條件

已在 openshift 叢集中安裝好以下工具
- Elasticsearch + Kibana
- Openshift logging operator

2. 開始設定

幫 log collector 建立 service account

oc create sa logging-collector -n openshift-logging

請將必要的權限指派給服務帳號，以便 collector 能夠收集並轉發日誌。在此範例中，收集器將被授予權限，以同時收集來自基礎架構、應用程式和審計的日誌。

oc adm policy add-cluster-role-to-user logging-collector-logs-writer -z logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-application-logs -z logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z logging-collector -n openshift-logging
oc adm policy add-cluster-role-to-user collect-audit-logs -z logging-collector -n openshift-logging

建立 elasticsearch tls secret

oc -n elastic get secret elasticsearch-es-http-certs-internal -o json | \
jq 'del(.metadata.namespace, .metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.ownerReferences, .metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' | \
oc -n openshift-logging create -f -

打開瀏覽器連線並登入 kibana 網站

oc -n elastic get route

執行結果如下：

NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
elasticsearch-sample elasticsearch-sample-elastic.apps.topgun.example.com elasticsearch-es-http <all> passthrough/Redirect None
kibana-sample kibana-sample-elastic.apps.topgun.example.com kibana-kb-http <all> passthrough/Redirect None

我的範例是：kibana-sample-elastic.apps.topgun.example.com

在 左側選單 -> Management/Dev tools，執行以下語法，建立一個名為 openshift_log_forwarder 的角色，這個角色只對 app-*, infra-*, audit-* 這些 OpenShift 常用的索引名稱開頭的索引有寫入權限。

PUT /_security/role/openshift_log_forwarder
{
 "cluster": [
 "monitor",
 "manage_index_templates"
 ],
 "indices": [
 {
 "names": [
 "app-*",
 "infra-*",
 "audit-*"
 ],
 "privileges": [
 "write",
 "create_index"
 ],
 "allow_restricted_indices": false
 }
 ]
}

在 左側選單 -> Management/Stack Management -> Security/Users -> Create user，建立一個使用者 (例如 ocp-forwarder-user)，並將剛才建立的角色 openshift_log_forwarder 指派給它。
設定帳號資訊
- Username: ocp-forwarder-user
- Full name: ocp-forwarder-user
- Password: password
- Confirm password: password
- Roles: 將下拉選單展開選擇剛剛自建的 openshift_log_forwarder role

建立 elasticsearch user secret yaml 檔

cat <<EOF > es-userinfo.yaml
apiVersion: v1
kind: Secret
metadata:
name: elasticsearch-user
namespace: openshift-logging
type: Opaque
stringData:
username: ocp-forwarder-user
password: password
EOF

建立 secret
```
oc apply -f es-userinfo.yaml
```

建立 ClusterLogForwarder CR YAML 檔

cat <<EOF > clf_elasticsearch.yaml
apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: collector
namespace: openshift-logging
spec:
managementState: Managed
outputs:
- name: audit-elasticsearch
type: elasticsearch
elasticsearch:
url: https://elasticsearch-es-http.elastic.svc.cluster.local:9200
version: 8
index: audit-write
authentication:
username:
key: username
secretName: elasticsearch-user
password:
key: password
secretName: elasticsearch-user
tls:
insecureSkipVerify: true
pipelines:
- name: audit
inputRefs:
- audit
outputRefs:
- audit-elasticsearch
serviceAccount:
name: logging-collector
EOF

建立 ClusterLogForwarder
```
oc apply -f clf_elasticsearch.yaml
```

確認 pod 狀態

oc get pods

正確執行結果：

NAME READY STATUS RESTARTS AGE
cluster-logging-operator-86bf5bc9b6-b27wh 1/1 Running 0 127m
collector-5qvt4 1/1 Running 0 26m
collector-6vpxt 1/1 Running 0 26m
collector-hnzhm 1/1 Running 0 26m
collector-s8z9g 1/1 Running 0 26m
collector-tz2lm 1/1 Running 0 26m
collector-x5l5t 1/1 Running 0 26m

檢視 collector pod log 是否有異常

oc logs collector-x5l5t

正確執行結果：

Creating the directory used for persisting Vector state /var/lib/vector/openshift-logging/collector
Starting Vector process...
2025-09-12T09:02:20.158032Z WARN sink{component_kind="sink" component_id=output_audit_elasticsearch component_type=elasticsearch}: vector_core::tls::settings: The `verify_certificate` option is DISABLED, this may lead to security vulnerabilities.
2025-09-12T09:02:20.158081Z WARN sink{component_kind="sink" component_id=output_audit_elasticsearch component_type=elasticsearch}: vector_core::tls::settings: The `verify_hostname` option is DISABLED, this may lead to security vulnerabilities.
2025-09-12T09:02:20.287291Z WARN vector::internal_events::file::source: Currently ignoring file too small to fingerprint. file=/var/log/ovn/acl-audit-log.log

Lab 內部測試可把資料副本調成 1

在 左側選單 -> Management/Dev tools，執行以下語法
```
PUT /your-index-name/_settings
{
"index": {
"number_of_replicas": 0
}
}
```
打開瀏覽器連到 Kibana UI 檢視是否有收到 log

Step by Step install OpenShift 4.19 with the Agent-based Installer in Disconnected environment

Thu, 18 Sep 2025 01:46:44 +0800

0. Preface

本篇文章會介紹，如何在全離線的環境下，在多台 VM 上使用 agent-based 的方式安裝 3 台 Control-plane Node 和 3 台 Worker Node 架構的 OpenShift Container Platform 4.19

簡易架構圖如下：

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

1. 環境規劃

主機名稱	角色和服務	說明
bastion	Cluster installer/bastion	發動安裝 OCP 的跳板機, 同時提供 load balance, DNS Server,Project Quay 的服務
master-1	rendezvous host/bootstrap	安裝時重要的角色, 透過 installer 將 ocp cluster 角色先部署在此, 再透過 scale out 延伸到 master node
master-[1-3]	Master node	ocp 重要的控制節點須為三個
worker-[1-3]	Worker/compute node	執行 Application 的節點
bastion	DNS Server	提供名稱解析和反解析(網址轉成 IP 或是將 IP 轉回網址)
bastion	HA Proxy	提供 load balance 的服務
bastion	Project Quay	提供 Image Registry 服務

1.1. 硬體資源需求

參考連結 : Minimum resource requirements - Red Hat Docs

1.2. 環境架構

請在虛擬化平台(Proxmox/vCenter…)產出以下 7 台 VM，並記錄每台 VM 對應的 Mac Address(除了 bastion)

1 台 bastion (RHEL)
3 台 master (RHCOS)
3 台 worker (RHCOS)

1.3. 主機名稱設定格式

HOSTNAME.CLUSTER_NAME.DOMAIN_NAME

example:

bastion.topgun.kubeantony.com

HOSTNAME 就是 bastion
CLUSTER_NAME 就是 topgun
DOMAIN_NAME 就是 kubeantony.com

1.4. 軟體授權

Red Hat Enterprise Linux subscription
- 60 天個人免費試用連結 : https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux/server/trial
Red Hat OpenShift Container Platform subscription
- 60 天個人免費試用連結 : https://www.redhat.com/en/technologies/cloud-computing/openshift/ocp-self-managed-trial

2. 安裝與設定 Bastion 主機

2.1. 安裝與設定 Red Hat Enterprise Linux 9

Install Red Hat Enterprise Linux 9 - server-world

2.2. 下載 “安裝 OCP 程式” 和 “管理 OpenShift 的 CLI 工具”

至 Red Hat 官網連結下載以下兩個項目的檔案 (一定要有帳號)

OpenShift v4.19.10 Linux Installer
OpenShift v4.19.10 Linux Client

請在 bastion 主機執行以下命令

OVERSION="4.19.10"
wget --show-progress -qO "openshift-install-linux-${OVERSION}.tar.gz" "<installer 的下載網址>" && wget --show-progress -qO "oc-${OVERSION}-linux.tar.gz" "<Clinet 的下載網址>"

oc-4.19.10-linux.tar.gz，是 OpenShift CLI 的壓縮檔

openshift-install-linux-4.19.10.tar.gz，是安裝 OCP 會用到的程式

注意! 下載網址需用兩個雙引號包住，不然會報錯。

2.3. 解壓縮檔案，並將 oc、openshift-install 和 kubectl 加至 PATH 環境變數

tar -xvf oc-${OVERSION}-linux.tar.gz && \
tar -xvf openshift-install-linux-${OVERSION}.tar.gz && \
sudo mv oc kubectl openshift-install /usr/local/bin

螢幕輸出 :

README.md
kubectl
oc
README.md
openshift-install

2.4. 確認 oc Command 版本

oc version

螢幕輸出 :

Client Version: 4.19.10
Kustomize Version: v5.5.0

2.5. 確認 openshift-install Command 版本

openshift-install version

螢幕輸出 :

openshift-install 4.19.10
built from commit 87bc6d06e8abd759e92112b434a180c2ddff41f1
release image quay.io/openshift-release-dev/ocp-release@sha256:2f9145136fb387d43c7fff55b30a036c14eb96b0992c292274b6f543c6c33857
release architecture amd64

2.6. 確認 kubectl Command 版本

kubectl version --client --output=yaml

螢幕輸出 :

openshift-install 4.19.10
built from commit 87bc6d06e8abd759e92112b434a180c2ddff41f1
release image quay.io/openshift-release-dev/ocp-release@sha256:2f9145136fb387d43c7fff55b30a036c14eb96b0992c292274b6f543c6c33857
release architecture amd64
[bigred@bastion ~]$ kubectl version --client --output=yaml
clientVersion:
buildDate: "2025-08-26T15:03:13Z"
compiler: gc
gitCommit: 298429ba9831d1d72b89edd9beb82a6ee665c3b7
gitTreeState: clean
gitVersion: v1.32.1
goVersion: go1.23.9 (Red Hat 1.23.9-1.el9_6) X:strictfipsruntime
major: "1"
minor: "32"
platform: linux/amd64
kustomizeVersion: v5.5.0

2.7. 安裝 DNS Server

sudo yum -y install bind

螢幕輸出 :

...以上省略
Complete!

2.8. 編輯 DNS Server 設定檔 named.conf

sudo nano /etc/named.conf

要改的地方有兩個部分

將 listen-on port 53 和 allow-query 的值, 改成 any

檔案內容 :

options {
listen-on port 53 { any; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
secroots-file "/var/named/data/named.secroots";
recursing-file "/var/named/data/named.recursing";
allow-query { any; };

假設我 bastion 這台機器的 Hostname 是 bastion.topgun.kubeantony.com IP 是 192.168.11.21，在 /etc/named.conf 新增以下內容至檔案的最後面 :

zone "kubeantony.com" {
type master;
file "/etc/named/zones/db.kubeantony.com";
};
zone "11.168.192.in-addr.arpa" {
type master;
file "/etc/named/zones/db.kubeantony.com.reverse";
};

在本次範例中

Cluster name 是 topgun
Base Domain Name 是 kubeantony.com

2.9. 設定 DNS Server 名稱解析

sudo mkdir /etc/named/zones && \
sudo nano /etc/named/zones/db.kubeantony.com

檔案內容如下 :

$TTL 1W
@ IN SOA bastion.topgun.kubeantony.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS bastion.topgun.kubeantony.com.
IN MX 10 smtp.kubeantony.com.
;
;
bastion.topgun.kubeantony.com. IN A 192.168.11.21
smtp.kubeantony.com. IN A 192.168.11.21
quay.kubeantony.com. IN A 192.168.11.21
;
api.topgun.kubeantony.com. IN A 192.168.11.21
api-int.topgun.kubeantony.com. IN A 192.168.11.21
;
*.apps.topgun.kubeantony.com. IN A 192.168.11.21
;
master-1.topgun.kubeantony.com. IN A 192.168.11.23
master-2.topgun.kubeantony.com. IN A 192.168.11.24
master-3.topgun.kubeantony.com. IN A 192.168.11.25
;
worker-1.topgun.kubeantony.com. IN A 192.168.11.26
worker-2.topgun.kubeantony.com. IN A 192.168.11.27
worker-3.topgun.kubeantony.com. IN A 192.168.11.28
;
;EOF

Note: 紅字部分為須依照環境的規劃來設定

2.10. 設定 DNS Server 反解析

sudo nano /etc/named/zones/db.kubeantony.com.reverse

檔案內容如下 :

$TTL 1W
@ IN SOA bastion.topgun.kubeantony.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS bastion.topgun.kubeantony.com.
;
21.11.168.192.in-addr.arpa. IN PTR api.topgun.kubeantony.com.
21.11.168.192.in-addr.arpa. IN PTR api-int.topgun.kubeantony.com.
;
23.11.168.192.in-addr.arpa. IN PTR master-1.topgun.kubeantony.com.
24.11.168.192.in-addr.arpa. IN PTR master-2.topgun.kubeantony.com.
25.11.168.192.in-addr.arpa. IN PTR master-3.topgun.kubeantony.com.
;
26.11.168.192.in-addr.arpa. IN PTR worker-1.topgun.kubeantony.com.
27.11.168.192.in-addr.arpa. IN PTR worker-2.topgun.kubeantony.com.
28.11.168.192.in-addr.arpa. IN PTR worker-3.topgun.kubeantony.com.
;
;EOF

Note: 紅字部分為須依照環境的規劃來設定

2.11. 啟動 DNS Server

sudo systemctl enable named --now

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/named.service → /usr/lib/systemd/system/named.service.

檢查

sudo systemctl status named

螢幕輸出 :

● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 16:50:31 CST; 1s ago
...以下省略

2.12. 將 bastion 機器的 IP 新增為 DNS Server

sudo nmcli connection modify 'ens18' ipv4.dns '192.168.11.21' +ipv4.dns '8.8.8.8'
sudo systemctl restart NetworkManager

檢查

sudo cat /etc/resolv.conf

螢幕輸出

# Generated by NetworkManager
search topgun.kubeantony.com
nameserver 192.168.11.21
nameserver 8.8.8.8

2.13. 關閉防火牆

sudo systemctl disable firewalld.service --now

檢查是否關閉

sudo systemctl status firewalld.service --no-pager

螢幕輸出 :

● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
...以下省略

2.14. 驗證 DNS Server 正解

# 定義 A Zone File
AZF="/etc/named/zones/db.kubeantony.com"
# 從 DNS Zone File 中找出所有 A 記錄的主機名稱
ARecord=$(sudo grep -w 'A' $AZF | cut -d " " -f 1)
# 對每個主機名稱進行查詢，使用系統中第一個 nameserver 為查詢伺服器
for i in $ARecord; do dig @$(awk '/^nameserver/ {print $2; exit}' /etc/resolv.conf) ${i} +short; done

螢幕輸出 :

192.168.11.21
192.168.11.21
192.168.11.21
192.168.11.21
192.168.11.21
192.168.11.23
192.168.11.24
192.168.11.25
192.168.11.26
192.168.11.27
192.168.11.28

2.15. 驗證 DNS Server 反解

# 指定 PTR Zone File 檔案路徑
PZF="/etc/named/zones/db.kubeantony.com.reverse"
# 擷取所有 PTR 記錄，將其轉換為標準 IPv4 格式，並去除重複
PTRRecord=$(sudo grep 'PTR' $PZF | awk '{split($1,a,"."); print a[4]"."a[3]"."a[2]"."a[1]}' | sort -u)
# 針對每個 IP 進行反解查詢（reverse DNS lookup），使用第一個 nameserver 查詢
for i in $PTRRecord; do dig -x ${i} @$(awk '/^nameserver/ {print $2; exit}' /etc/resolv.conf) +short; done

螢幕輸出 :

api-int.topgun.kubeantony.com.
api.topgun.kubeantony.com.
master-1.topgun.kubeantony.com.
master-2.topgun.kubeantony.com.
master-3.topgun.kubeantony.com.
worker-1.topgun.kubeantony.com.
worker-2.topgun.kubeantony.com.
worker-3.topgun.kubeantony.com.

2.16. 安裝 HAProxy 服務

sudo yum -y install haproxy

螢幕輸出 :

...以上省略
Complete!

2.17. 設定 HAProxy

直接清掉預設的設定檔

cat /dev/null | sudo tee /etc/haproxy/haproxy.cfg

再編輯設定檔

sudo nano /etc/haproxy/haproxy.cfg

將以下內容複製到檔案中 :

global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
defaults
mode http
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen api-server-6443
bind *:6443
mode tcp
server master-1 master-1.topgun.kubeantony.com:6443 check inter 1s
server master-2 master-2.topgun.kubeantony.com:6443 check inter 1s
server master-3 master-3.topgun.kubeantony.com:6443 check inter 1s
listen machine-config-server-22623
bind *:22623
mode tcp
server master-1 master-1.topgun.kubeantony.com:22623 check inter 1s
server master-2 master-2.topgun.kubeantony.com:22623 check inter 1s
server master-3 master-3.topgun.kubeantony.com:22623 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server worker-1 worker-1.topgun.kubeantony.com:443 check inter 1s
server worker-2 worker-2.topgun.kubeantony.com:443 check inter 1s
server worker-3 worker-3.topgun.kubeantony.com:443 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server worker-1 worker-1.topgun.kubeantony.com:80 check inter 1s
server worker-2 worker-2.topgun.kubeantony.com:80 check inter 1s
server worker-3 worker-3.topgun.kubeantony.com:80 check inter 1s

Note: 紅字部分為須依照環境的規劃來設定

2.18. 設定允許 HAProxy 可以使用 TCP Port

sudo setsebool -P haproxy_connect_any=1

If you are using HAProxy as a load balancer and SELinux is set to enforcing, you must ensure that the HAProxy service can bind to the configured TCP port by running setsebool -P haproxy_connect_any=1.

2.19. 啟動 HAProxy 服務，並設為開機自動啟動

sudo systemctl enable --now haproxy.service

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

檢查

sudo systemctl status haproxy.service

螢幕輸出 :

● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 17:26:21 CST; 11min ago
...以下省略

2.20. 下載 pull-secret

執行以下命令

mkdir ~/ocp4; cd ~/ocp4;
nano pull-secret.txt

至以下連結下載，注意 : 需先登入帳號
- Download Pull secret

可點選 “Download pull secret"，或是點 “Copy pull secret” 直接將 pull-secret.txt 複製到剪貼簿

3. 安裝 Project Quay Container Registry

3.1. 先決條件

2 個或以上的 virtual CPU
4 GB 或以上的記憶體
系統上至少需要約 30 GB 或以上的磁碟空間，可細分如下：
- 約 10 GB 的磁碟空間用於 Red Hat Enterprise Linux (RHEL) 作業系統。
- 約 10 GB 的磁碟空間用於 Podman 儲存，以運行三個容器 (container)。
- 約 10 GB 的磁碟空間用於 Project Quay 本地儲存。
已安裝 podman
這台機器需可上網

3.2. 規劃一顆獨立的硬碟給 Project Quay

在 VM 新增一顆硬碟專門給 Project Quay 使用

檢查新的硬碟名稱

lsblk

執行結果 :

...
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdb 8:16 0 100G 0 disk

將硬碟名稱設為變數
```
D="/dev/sdb"
```
使用 parted 建立 GPT 分割區，從 1 MiB 到磁碟尾，並標記為 LVM 用途
```
sudo parted -s ${D} mklabel gpt
sudo parted -s ${D} unit MiB mkpart primary 1 100%
```
將剛建立的分割區初始化為 LVM PV
```
sudo pvcreate ${D}1
```
建立名為 vg-data 的 VG
```
sudo vgcreate vg-data ${D}1
```
從 VG 中建立一 LV，利用全部剩餘空間並命名為 lv-quay
```
sudo lvcreate -l +100%FREE -n lv-quay /dev/vg-data
```
格式化 LV 為 XFS 檔案系統，以提供穩健且高效的儲存
```
sudo mkfs.xfs /dev/vg-data/lv-quay
```

建立 Project Quay 要使用的儲存目錄

sudo mkdir -p /data/quay/{root,sqlite-storage,quay-storage}

修改目錄權限

sudo chown -R $(id -u):$(id -g) /data/quay

將目錄設為開機自動掛載 lv 裝置上

echo '/dev/vg-data/lv-quay /data/quay xfs defaults 0 0' | sudo tee -a /etc/fstab

將目錄掛載到 lv 裝置上

sudo mount -a && sudo systemctl daemon-reload

3.3. 確認已準備好 DNS A record

請執行以下命令查詢

dig quay.kubeantony.com +short

執行結果：

192.168.11.21

3.4. 透過 Mirror Registry 安裝 Project Quay

Mirror Registry: 這應用程式能讓使用者透過一個簡單的命令列介面 (CLI) 工具，輕易地安裝 Quay 及其所需元件。其目的是提供一個映像檔倉庫 (registry)，用以存放 OpenShift 的映像檔。

建立並切換專案工作目錄

mkdir ~/project-quay; cd ~/project-quay/

下載並準備 mirror-registry 工具

wget https://mirror.openshift.com/pub/cgw/mirror-registry/latest/mirror-registry-amd64.tar.gz

將下載的工具包解壓縮到工作目錄中

tar xvzf mirror-registry-amd64.tar.gz -C ~/project-quay/

執行安裝指令，並設定 Quay 的資料儲存路徑、主機名稱及初始管理員帳號密碼

~/project-quay/mirror-registry install \
--quayRoot /data/quay/root \
--quayStorage /data/quay/sqlite-storage \
--sqliteStorage /data/quay/quay-storage \
--initUser admin \
--initPassword redhat123 \
--quayHostname quay.kubeantony.com

成功執行結果 :

...
INFO[2025-09-16 23:50:07] Quay installed successfully, config data is stored in /data/quay/root
INFO[2025-09-16 23:50:07] Quay is available at https://quay.kubeantony.com:8443 with credentials (admin, redhat123)

檢查 Quay、Redis 等相關 app container 是否已成功啟動

podman ps -a

正確執行結果：

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dab9b2b04011 registry.access.redhat.com/ubi8/pause:8.10-5 infinity 3 minutes ago Up 3 minutes 0.0.0.0:8443->8443/tcp 31e6ad76f0aa-infra
cd50dd26d5d3 registry.redhat.io/rhel8/redis-6:1-190 run-redis 3 minutes ago Up 3 minutes 0.0.0.0:8443->8443/tcp, 6379/tcp quay-redis
e5386bb42056 registry.redhat.io/quay/quay-rhel8:v3.12.10 registry 2 minutes ago Up 3 minutes 0.0.0.0:8443->8443/tcp, 7443/tcp, 8080/tcp quay-app

將 Quay 自動產生的根憑證 (rootCA.pem) 複製到系統的信任憑證目錄

sudo cp -v /data/quay/root/quay-rootCA/rootCA.pem /etc/pki/ca-trust/source/anchors/

更新系統的 CA 信任清單，讓 HTTPS 連線時能信任此憑證
```
sudo update-ca-trust
```
使用 podman 登入 Quay 倉庫，確認帳號與連線皆正常
```
podman login -u admin -p redhat123 quay.kubeantony.com:8443
```
正確執行結果：
```
Login Succeeded!
```

透過 systemd 查詢 Quay 相關服務的運行狀態

systemctl --user list-units --type=service | grep "quay"

執行結果：

 quay-app.service loaded active running Quay Container
quay-pod.service loaded active exited Infra Container for Quay
quay-redis.service loaded active running Redis Podman Container for Quay

透過 systemd 重啟 Quay Pod (此操作會連帶重啟 Quay App 與 Redis 容器):
systemctl --user restart quay-pod

2. 透過 `oc-mirror` 下載並打包 openshift 叢集所需之 container image，並上傳至本地 project quay

切換工作目錄
```
cd ~/ocp4
```

下載 oc-mirror CLI 工具

wget https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/oc-mirror.rhel9.tar.gz

將 oc-mirror 執行檔解壓縮至 $PATH 路徑中，使其可直接執行

mkdir -p $HOME/.local/bin
tar xvzf ./oc-mirror.rhel9.tar.gz -C $HOME/.local/bin
chmod +x $HOME/.local/bin/oc-mirror

建立存放 image 倉庫身份憑證檔案的標準目錄
```
mkdir -p $HOME/.docker
```
將 Red Hat Pull Secret 移至標準認證路徑，用於從官方倉庫拉取 image
```
mv -v pull-secret.txt $HOME/.docker/config.json
```

登入本地 Quay 倉庫，認證資訊會自動加入 config.json

podman login -u admin \
-p redhat123 \
--authfile $HOME/.docker/config.json \
quay.kubeantony.com:8443

備份已包含 Red Hat 和本地 Quay 認證的組合檔案
```
cp -v $HOME/.docker/config.json ~/ocp4/pull-secret.json
```

列出指定 OpenShift 版本 (4.18) 可用的 OperatorHub Catalog

oc-mirror list operators --catalogs --version=4.19

執行結果：

Available OpenShift OperatorHub catalogs:
OpenShift 4.19:
registry.redhat.io/redhat/redhat-operator-index:v4.19
registry.redhat.io/redhat/certified-operator-index:v4.19
registry.redhat.io/redhat/community-operator-index:v4.19
registry.redhat.io/redhat/redhat-marketplace-index:v4.19

列出特定 Catalog 中的所有 Operator Packages，並將結果存檔

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.19 > ~/ocp4/redhat-operator-index-package.out

從清單中搜尋特定 Operator，以確認其 package 名稱和可用的 channel

cat ~/ocp4/redhat-operator-index-package.out | grep -E "opentelemetry-product|cluster-logging|cluster-observability-operator|quay-operator"

執行結果：

cluster-logging stable-6.3
cluster-observability-operator stable
opentelemetry-product stable
quay-operator stable-3.15

查詢與 OpenShift 4.19 版本相關的所有可用的 channel

oc-mirror list releases --version 4.19 --channels

執行結果：

Listing channels for version 4.19.
fast-4.18
stable-4.18
candidate-4.19
fast-4.19
stable-4.19
candidate-4.20
candidate-4.18
eus-4.18

列出 stable-4.19 channel 可安裝的 openshift 版本

oc-mirror list releases --channel stable-4.19

執行結果：

Channel: stable-4.19
Architecture: amd64
...
4.19.3
4.19.4
4.19.5
4.19.6
4.19.7
4.19.9
4.19.10

建立存放鏡像 tar 檔的本地目錄
```
mkdir -p /data/quay/mirror_img
```

編輯 ImageSetConfiguration 設定檔，定義要鏡像的內容

nano imageset-config.yaml

檔案內容如下：

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
 # 設定鏡像檔案的本地存放路徑
 local:
 path: /data/quay/mirror_img
mirror:
 # 鏡像 OpenShift 平台的核心映像檔
 platform:
 channels:
 - name: stable-4.19  # 指定 OCP 版本
 type: ocp
 # 鏡像指定的 Operator
 operators:
 - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.19
 # 指定要下載的 packages 及其 channel
 packages:
 - name: opentelemetry-product
 channels:
 - name: stable
 - name: cluster-logging
 channels:
 - name: stable-6.3
 - name: cluster-observability-operator
 channels:
 - name: stable
 - name: loki-operator
 channels:
 - name: stable-6.3
 # 鏡像額外的映像檔
 additionalImages:
 - name: registry.redhat.io/ubi8/ubi:latest
 helm: {}

執行鏡像程序：根據設定檔從網路下載所有 image，並打包成一個 tar 檔存放在本地
```
oc mirror --verbose 3 \
--config=imageset-config.yaml \
file:///data/quay/mirror_img
```

確認 tar 檔已成功建立

ls -lh /data/quay/mirror_img

執行結果：

total 39G
-rw-r--r--. 1 bigred bigred 39G Sep 15 16:35 mirror_seq1_000000.tar
drwxr-xr-x. 3 bigred bigred 17 Sep 15 16:11 oc-mirror-workspace
drwxr-x---. 2 bigred bigred 28 Sep 15 16:35 publish

將本地 tar 檔中的所有 image 推送至目標 Project Quay 倉庫

oc mirror --verbose 3 \
--from=/data/quay/mirror_img/mirror_seq1_000000.tar \
docker://quay.kubeantony.com:8443

4. 編輯必要叢集設定檔

4.1. 設定 `install-config.yaml`

nano ~/ocp4/install-config.yaml

每個環境一定會不同的欄位：
networking.machineNetwork.cidr，這是叢集節點的網路。

pullSecret 欄位的內容：jq -c . ~/ocp4/pull-secret.json。

sshKey 欄位的內容：cat ~/.ssh/id_rsa.pub，如果沒有 ssh public key，可透過這命令產生：ssh-keygen -t rsa -P ''。
additionalTrustBundle 欄位的內容 : cat /data/quay/root/quay-rootCA/rootCA.pem。
透過指令將憑證用正確的格式寫入檔案：
安裝 yq
sudo wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/local/bin/yq
sudo chmod +x /usr/local/bin/yq
定義檔案路徑變數，方便閱讀
CERT_FILE="/data/quay/root/quay-rootCA/rootCA.pem"
CONFIG_FILE="$HOME/ocp4/install-config.yaml"
讀取憑證內容並透過 yq 寫入 YAML 檔案
CERT_CONTENT=$(cat "$CERT_FILE") yq -i '.additionalTrustBundle = strenv(CERT_CONTENT)' "$CONFIG_FILE"
imageContentSources.mirrors：將 quay.kubeantony.com:8443 換成你的 quay 的 FQDN:PORT

檔案內容如下：

apiVersion: v1
baseDomain: kubeantony.com
compute:
- architecture: amd64
hyperthreading: Enabled
name: worker
replicas: 3
controlPlane:
architecture: amd64
hyperthreading: Enabled
name: master
replicas: 3
metadata:
name: topgun
networking:
clusterNetwork:
- cidr: 10.244.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 172.20.6.0/16
networkType: OVNKubernetes
serviceNetwork:
- 10.98.0.0/16
platform:
none: {}
pullSecret: '...'
sshKey: '...'
additionalTrustBundlePolicy: Always
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIID3zCCAsegAwIBAgIUA/xNrN5qECVnywPZoUuA+VblAwowDQYJKoZIhvcNAQEL
BQAwbTELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAlZBMREwDwYDVQQHDAhOZXcgWW9y
azENMAsGA1UECgwEUXVheTERMA8GA1UECwwIRGl2aXNpb24xHDAaBgNVBAMME3F1
YXkua3ViZWFudG9ueS5jb20wHhcNMjUwNzAxMTU1MDEzWhcNMjgwNDIwMTU1MDEz
WjBtMQswCQYDVQQGEwJVUzELMAkGA1UECAwCVkExETAPBgNVBAcMCE5ldyBZb3Jr
MQ0wCwYDVQQKDARRdWF5MREwDwYDVQQLDAhEaXZpc2lvbjEcMBoGA1UEAwwTcXVh
eS5rdWJlYW50b255LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ALPGUbzh8ic5Qz8B3z0G66fyXsTYGYoFf37VXVL8oCxliKWEwz9kDutWyTjWI1Su
3JHFzxjq0gWkeC0vtVmfTFmNtId1zQrXAk6np0HPrgW73msy5UW5c7q98BoAS7Aq
wLDH7NsnUVclBnnn3yozLNRFVF/DrgY8JDhcPoeuFljb0pwMCEwnkYYi7RR+7Tmx
Kkt5hTaraRyH2i/ja6pjNmTtR2TVQKfsKgy5yCZOIcrEjVZMRIdqna1z1EhFdW8R
GGiBZwScfkjimUV8HIXWs+QfS7EpQSmC/CK+dR2IPawVoimyN9pff1U60p23Y3dV
vNf8TH0Crj6zfpaWMix1o3ECAwEAAaN3MHUwCwYDVR0PBAQDAgLkMBMGA1UdJQQM
MAoGCCsGAQUFBwMBMB4GA1UdEQQXMBWCE3F1YXkua3ViZWFudG9ueS5jb20wEgYD
VR0TAQH/BAgwBgEB/wIBATAdBgNVHQ4EFgQUKDUsDN+UCyJaNeA/xv/4/DUrspAw
DQYJKoZIhvcNAQELBQADggEBAIjUZ2aUCapDbfjuEJeOQOIuS4ZfrM0RSAx/kMHl
uV22Wz2bbjLGX8EC9Y1zajYV00vVPC/2sH8hGvnUiznTngY5KSWypsx42BY9eMbN
lCATIuayyByWSdqn9BxZ8E7yzwSm1B9529aVFuuT8yNmEx+Xhe1wtJRYZQBArPZA
rbrouCUqrLm0LNKo5L/rMKlWdYY5QwpzY7UQcENCe4wO8xfFnAOP5uJTrSUp1R1N
Q5JiA+RPoSOZ/MkfTOO4ibW8DlqZt+s3NSpbfHIFnPuc9uK4EoAs7TDEHo53ORct
x5TvHNuXmfqYMIqaDMfw2FlpvzhfZHucI+a/Nb46XcAb4cg=
-----END CERTIFICATE-----
imageContentSources:
- mirrors:
- quay.kubeantony.com:8443/openshift/release
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
- quay.kubeantony.com:8443/openshift/release-images
source: quay.io/openshift-release-dev/ocp-release

4.2. 設定 `agent-config.yaml`

nano ~/ocp4/agent-config.yaml

每個環境一定會不同的欄位：

rendezvousIP，請設為第一台 control-plane node 的 ip

additionalNTPSources，你環境的 ntp server

hosts.interfaces.macAddress，請填入每台節點的 mac address

hosts.networkConfig，請填入相關網路設定

檔案內容如下：

apiVersion: v1alpha1
kind: AgentConfig
metadata:
 name: topgun
rendezvousIP: 192.168.11.23
additionalNTPSources:
- 192.168.11.11
hosts:
 - hostname: master-1
 role: master
 interfaces:
 - name: ens18
 macAddress: BC:24:11:99:B8:1B
 networkConfig:
 interfaces:
 - name: ens18
 type: ethernet
 state: up
 mac-address: BC:24:11:99:B8:1B
 ipv4:
 enabled: true
 address:
 - ip: 192.168.11.23
 prefix-length: 24
 dhcp: false
 dns-resolver:
 config:
 server:
 - 192.168.11.21
 routes:
 config:
 - destination: 0.0.0.0/0
 next-hop-address: 192.168.11.253
 next-hop-interface: ens18
 table-id: 254
 rootDeviceHints:
 deviceName: /dev/sda
 - hostname: master-2
 role: master
 interfaces:
 - name: ens18
 macAddress: BC:24:11:F5:E5:E4
 networkConfig:
 interfaces:
 - name: ens18
 type: ethernet
 state: up
 mac-address: BC:24:11:F5:E5:E4
 ipv4:
 enabled: true
 address:
 - ip: 192.168.11.24
 prefix-length: 24
 dhcp: false
 dns-resolver:
 config:
 server:
 - 192.168.11.21
 routes:
 config:
 - destination: 0.0.0.0/0
 next-hop-address: 192.168.11.253
 next-hop-interface: ens18
 table-id: 254
 rootDeviceHints:
 deviceName: /dev/sda
 - hostname: master-3
 role: master
 interfaces:
 - name: ens18
 macAddress: BC:24:11:9C:1C:0F
 networkConfig:
 interfaces:
 - name: ens18
 type: ethernet
 state: up
 mac-address: BC:24:11:9C:1C:0F
 ipv4:
 enabled: true
 address:
 - ip: 192.168.11.25
 prefix-length: 24
 dhcp: false
 dns-resolver:
 config:
 server:
 - 192.168.11.21
 routes:
 config:
 - destination: 0.0.0.0/0
 next-hop-address: 192.168.11.253
 next-hop-interface: ens18
 table-id: 254
 rootDeviceHints:
 deviceName: /dev/sda
 - hostname: worker-1
 role: worker
 interfaces:
 - name: ens18
 macAddress: BC:24:11:28:F0:89
 networkConfig:
 interfaces:
 - name: ens18
 type: ethernet
 state: up
 mac-address: BC:24:11:28:F0:89
 ipv4:
 enabled: true
 address:
 - ip: 192.168.11.26
 prefix-length: 24
 dhcp: false
 dns-resolver:
 config:
 server:
 - 192.168.11.21
 routes:
 config:
 - destination: 0.0.0.0/0
 next-hop-address: 192.168.11.253
 next-hop-interface: ens18
 table-id: 254
 rootDeviceHints:
 deviceName: /dev/sda
 - hostname: worker-2
 role: worker
 interfaces:
 - name: ens18
 macAddress: BC:24:11:77:1E:9E
 networkConfig:
 interfaces:
 - name: ens18
 type: ethernet
 state: up
 mac-address: BC:24:11:77:1E:9E
 ipv4:
 enabled: true
 address:
 - ip: 192.168.11.27
 prefix-length: 24
 dhcp: false
 dns-resolver:
 config:
 server:
 - 192.168.11.21
 routes:
 config:
 - destination: 0.0.0.0/0
 next-hop-address: 192.168.11.253
 next-hop-interface: ens18
 table-id: 254
 rootDeviceHints:
 deviceName: /dev/sda
 - hostname: worker-3
 role: worker
 interfaces:
 - name: ens18
 macAddress: BC:24:11:F0:74:F3
 networkConfig:
 interfaces:
 - name: ens18
 type: ethernet
 state: up
 mac-address: BC:24:11:F0:74:F3
 ipv4:
 enabled: true
 address:
 - ip: 192.168.11.28
 prefix-length: 24
 dhcp: false
 dns-resolver:
 config:
 server:
 - 192.168.11.21
 routes:
 config:
 - destination: 0.0.0.0/0
 next-hop-address: 192.168.11.253
 next-hop-interface: ens18
 table-id: 254
 rootDeviceHints:
 deviceName: /dev/sda

4.3. 產生安裝用 ISO

需要先安裝 nmstae 套件
```
sudo yum install -y nmstate
```

備份設定檔

cp agent-config.yaml agent-config.yaml.bk
cp install-config.yaml install-config.yaml.bk

再產生 ISO

openshift-install --dir ~/ocp4/ agent create image

螢幕輸出 :

INFO Configuration has 3 master replicas and 3 worker replicas
INFO The rendezvous host IP (node0 IP) is 192.168.11.23
INFO Extracting base ISO from release payload
INFO Base ISO obtained from release and cached at [/home/bigred/.cache/agent/image_cache/coreos-x86_64.iso]
INFO Consuming Install Config from target directory
INFO Consuming Agent Config from target directory
INFO Generated ISO at /home/bigred/ocp4/agent.x86_64.iso.

5. 開始安裝 RedHat OpenShift

5.1. 安裝步驟

將 ISO 上傳至虛擬化平台，並掛載到 VM 上
在虛擬化平台將各節點透過 ISO 開機

追蹤和驗證安裝進度 (在 Bastion 主機執行以下指令) 確認哪些節點已安裝完畢，並需要重新開機:

openshift-install --dir ~/ocp4/ agent wait-for bootstrap-complete --log-level=debug

正確安裝的螢幕輸出如下 :

...
INFO Host: master-3, reached installation stage Writing image to disk: 100%
INFO Host: worker-3, reached installation stage Waiting for control plane
INFO Bootstrap Kube API Initialized
INFO Host: master-1, reached installation stage Waiting for control plane: Waiting for masters to join bootstrap control plane
INFO Uploaded logs for host master-2 cluster 84a2dfac-6f61-4de2-93e8-185d5e342f02
INFO Host: master-2, reached installation stage Rebooting
INFO Host: master-3, reached installation stage Rebooting
INFO Host: master-1, reached installation stage Waiting for bootkube
INFO Host: master-3, reached installation stage Done
INFO Node master-2 has been rebooted 1 times before completing installation
INFO Node master-3 has been rebooted 1 times before completing installation
INFO Host: worker-2, reached installation stage Rebooting
INFO Host: worker-1, reached installation stage Rebooting
INFO Host: master-1, reached installation stage Waiting for bootkube: waiting for ETCD bootstrap to be complete
INFO Bootstrap configMap status is complete
INFO Bootstrap is complete
INFO cluster bootstrap is complete

確認重開機後，OpenShift 自動安裝完成

openshift-install --dir ~/ocp4 agent wait-for install-complete --log-level=debug

螢幕輸出 :

...
INFO Cluster is installed
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run
INFO export KUBECONFIG=/home/bbg/ocp4/auth/kubeconfig
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.topgun.kubeantony.com
INFO Login to the console with user: "kubeadmin", and password: "JyAcY-VDp6D-DYdHp-T4Twr"

設定 KubeConfig

mkdir ~/.kube && \
cp ~/ocp4/auth/kubeconfig ~/.kube/config

為當前使用者設定 oc 指令設定 Bash 自動補全

oc completion bash > ~/.oc_completion.sh
echo "source ~/.oc_completion.sh" >> ~/.bash_rc
source ~/.bash_rc

檢查叢集節點狀態

oc get nodes

螢幕輸出

NAME STATUS ROLES AGE VERSION
master-1 Ready control-plane,master 33m v1.32.7
master-2 Ready control-plane,master 46m v1.32.7
master-3 Ready control-plane,master 47m v1.32.7
worker-1 Ready worker 40m v1.32.7
worker-2 Ready worker 37m v1.32.7
worker-3 Ready worker 41m v1.32.7

檢視整個叢集核心元件是否健康

oc get co

螢幕輸出 :

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.19.10 True False False 28m
baremetal 4.19.10 True False False 45m
cloud-controller-manager 4.19.10 True False False 47m
cloud-credential 4.19.10 True False False 48m
cluster-autoscaler 4.19.10 True False False 45m
config-operator 4.19.10 True False False 46m
console 4.19.10 True False False 32m
control-plane-machine-set 4.19.10 True False False 45m
csi-snapshot-controller 4.19.10 True False False 45m
dns 4.19.10 True False False 45m
etcd 4.19.10 True False False 43m
image-registry 4.19.10 True False False 35m
ingress 4.19.10 True False False 37m
insights 4.19.10 True False False 45m
kube-apiserver 4.19.10 True False False 40m
kube-controller-manager 4.19.10 True False False 41m
kube-scheduler 4.19.10 True False False 43m
kube-storage-version-migrator 4.19.10 True False False 46m
machine-api 4.19.10 True False False 45m
machine-approver 4.19.10 True False False 45m
machine-config 4.19.10 True False False 45m
marketplace 4.19.10 True False False 45m
monitoring 4.19.10 True False False 35m
network 4.19.10 True False False 46m
node-tuning 4.19.10 True False False 35m
olm 4.19.10 True False False 45m
openshift-apiserver 4.19.10 True False False 36m
openshift-controller-manager 4.19.10 True False False 41m
openshift-samples 4.19.10 True False False 31m
operator-lifecycle-manager 4.19.10 True False False 45m
operator-lifecycle-manager-catalog 4.19.10 True False False 45m
operator-lifecycle-manager-packageserver 4.19.10 True False False 36m
service-ca 4.19.10 True False False 46m
storage 4.19.10 True False False 45m

5.2. Openshihift 完成安裝後一定要做的設定

禁用預設的 OperatorHub 來源，以便使用自訂的鏡像來源

oc patch OperatorHub cluster --type json \
-p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

列出 oc-mirror 指令產生的資訊

ls -l $(ls -td ~/ocp4/oc-mirror-workspace/results-*/ | head -n 1)

執行結果：

total 60
-rwxr-xr-x. 1 bigred bigred 234 Sep 15 17:07 catalogSource-cs-redhat-operator-index.yaml
drwxr-xr-x. 2 bigred bigred 6 Sep 15 17:01 charts
-rwxr-xr-x. 1 bigred bigred 1407 Sep 15 17:07 imageContentSourcePolicy.yaml
-rw-r--r--. 1 bigred bigred 49707 Sep 15 17:07 mapping.txt
drwxr-xr-x. 2 bigred bigred 52 Sep 15 17:07 release-signatures

套用映像檔內容來源策略，將映像檔請求重新導向至本地鏡像倉庫

oc create -f $(ls -td ~/ocp4/oc-mirror-workspace/results-*/ | head -n 1)imageContentSourcePolicy.yaml

建立新的 CatalogSource，讓 Operator Lifecycle Manager (OLM) 從本地鏡像倉庫找到 Operator

oc create -f $(ls -td ~/ocp4/oc-mirror-workspace/results-*/ | head -n 1)/catalogSource-cs-redhat-operator-index.yaml

匯出全域叢集拉取密鑰 (pull-secret) 到當前目錄下的 .dockerconfigjson 檔案
```
oc extract secret/pull-secret -n openshift-config --confirm --to=.
```

從拉取密鑰中移除 cloud.openshift.com 的認證資訊

jq 'del(.auths["cloud.openshift.com"])' .dockerconfigjson > .new-dockerconfigjson

使用更新後的內容（已移除 cloud.openshift.com）來更新叢集中的拉取密鑰

oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=./.new-dockerconfigjson

重新啟動 Insights Operator 以套用新的設定

oc -n openshift-insights rollout restart deployment insights-operator

檢查 Insights Operator 的 Pod 狀態，確認重啟成功

oc get pod -n openshift-insights -l app=insights-operator

執行結果：

NAME READY STATUS RESTARTS AGE
insights-operator-58bd6cc4c7-lpc2q 1/1 Running 0 5s

檢查 Insights Cluster Operator (CO) 的健康狀態，確認其正常運作

oc get co insights

執行結果：

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
insights 4.19.10 True False False 51m

6. 參考文章

在 Openshift 上透過 Kustomize 安裝 Elasticsearch

Wed, 17 Sep 2025 01:50:08 +0800

0. 安裝環境

模擬客戶環境，不透過 StorageClass，而是手動管理 PV/PVC
透過 LVM + NFS 作為底層的 Storage
全離線環境 (Openshift 叢集節點都無法到 public 網路)

1. 環境確認

SSH 連線到 NFS Server 執行以下命令

1.1. 確認 NFS Server `fsid=0` 對應的掛載點

cat /etc/exports | grep fsid=0

執行結果：

/nfs 10.21.107.0/24(rw,no_root_squash,fsid=0,insecure,sync)

1.2. 確認要用哪個 LVM VG

vgs

執行結果：

VG #PV #LV #SN Attr VSize VFree
nfsvg 1 32 0 wz--n- <2.50t <698.00g
rootvg 2 7 0 wz--n- <197.80g <30.80g

2. NFS 掛載環境準備

2.1. Create Folders for Elasticsearch

ocp log 資料要在 /nfs/ocpnfs_data/ 目錄底下 (因環境問題故放在 /nfs/apnfs_data)

mkdir -p /nfs/apnfs_data/rh-elastic-es-master-nodes-{0,1,2}
mkdir -p /nfs/apnfs_data/rh-elastic-es-data-nodes-{0,1,2}

2.2. Create LVs

lvcreate -L 10G -n rh-elastic-es-master-nodes-0 /dev/nfsvg
lvcreate -L 10G -n rh-elastic-es-master-nodes-1 /dev/nfsvg
lvcreate -L 10G -n rh-elastic-es-master-nodes-2 /dev/nfsvg
lvcreate -L 100G -n rh-elastic-es-data-nodes-0 /dev/nfsvg
lvcreate -L 100G -n rh-elastic-es-data-nodes-1 /dev/nfsvg
lvcreate -L 100G -n rh-elastic-es-data-nodes-2 /dev/nfsvg

2.3. Format xfs filesystem

mkfs.xfs /dev/nfsvg/rh-elastic-es-master-nodes-0
mkfs.xfs /dev/nfsvg/rh-elastic-es-master-nodes-1
mkfs.xfs /dev/nfsvg/rh-elastic-es-master-nodes-2
mkfs.xfs /dev/nfsvg/rh-elastic-es-data-nodes-0
mkfs.xfs /dev/nfsvg/rh-elastic-es-data-nodes-1
mkfs.xfs /dev/nfsvg/rh-elastic-es-data-nodes-2

2.4. Edit fstab

nano /etc/fstab

檔案內容如下：

/dev/nfsvg/rh-elastic-es-data-nodes-0 /nfs/apnfs_data/rh-elastic-es-data-nodes-0 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-data-nodes-1 /nfs/apnfs_data/rh-elastic-es-data-nodes-1 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-data-nodes-2 /nfs/apnfs_data/rh-elastic-es-data-nodes-2 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-master-nodes-0 /nfs/apnfs_data/rh-elastic-es-master-nodes-0 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-master-nodes-1 /nfs/apnfs_data/rh-elastic-es-master-nodes-1 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-master-nodes-2 /nfs/apnfs_data/rh-elastic-es-master-nodes-2 xfs defaults 0 0

2.5. mount

mount -a && systemctl daemon-reload

2.6. Check

lsblk /dev/sdc | tail -n 6

執行結果：

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sdc 8:32 0 2.5T 0 disk
└─sdc1 8:33 0 2.5T 0 part
...
├─nfsvg-rh--elastic--es--master--nodes--0 253:33 0 10G 0 lvm /nfs/ocpnfs_data/rh-elastic-es-master-nodes-0
├─nfsvg-rh--elastic--es--master--nodes--1 253:34 0 10G 0 lvm /nfs/ocpnfs_data/rh-elastic-es-master-nodes-1
├─nfsvg-rh--elastic--es--master--nodes--2 253:35 0 10G 0 lvm /nfs/ocpnfs_data/rh-elastic-es-master-nodes-2
├─nfsvg-rh--elastic--es--data--nodes--0 253:36 0 100G 0 lvm /nfs/ocpnfs_data/rh-elastic-es-data-nodes-0
├─nfsvg-rh--elastic--es--data--nodes--1 253:37 0 100G 0 lvm /nfs/ocpnfs_data/rh-elastic-es-data-nodes-1
└─nfsvg-rh--elastic--es--data--nodes--2 253:38 0 100G 0 lvm /nfs/ocpnfs_data/rh-elastic-es-data-nodes-2

2.7. 編輯 /etc/exports

nano /etc/exports

執行結果：

...以上省略
# ECK
/nfs/apnfs_data/rh-elastic-es-master-nodes-0 10.21.107.0/24(rw,no_root_squash,fsid=60,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-master-nodes-1 10.21.107.0/24(rw,no_root_squash,fsid=61,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-master-nodes-2 10.21.107.0/24(rw,no_root_squash,fsid=62,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-data-nodes-0 10.21.107.0/24(rw,no_root_squash,fsid=63,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-data-nodes-1 10.21.107.0/24(rw,no_root_squash,fsid=64,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-data-nodes-2 10.21.107.0/24(rw,no_root_squash,fsid=65,insecure,sync)

2.8. 重載 NFS server 配置並立即生效 export

systemctl reload nfs-server
exportfs -arv

執行結果：

exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-data-nodes-2
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-data-nodes-1
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-data-nodes-0
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-master-nodes-2
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-master-nodes-1
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-master-nodes-0
...

3. 安裝 ELK

3.1. 將事前準備好的 ELK 全離線安裝包上傳到 bastion 機和 Harbor

3.2. 將 image 改 tag 並 push 到 harbor

3.3. 修改 ELK 設定檔

cd elk/
oc create -f crds.yaml
oc new-project elastic-system
oc new-project elastic
cd env/dev

3.3.1. 確認要安裝的節點名稱

oc get nodes -l "node-role.kubernetes.io/logging" -o name

執行結果：

node/docpl01.ocp.cbsd.scsb.com.tw
node/docpl02.ocp.cbsd.scsb.com.tw
node/docpl03.ocp.cbsd.scsb.com.tw

3.3.2. 確認 Taints

oc get nodes -l "node-role.kubernetes.io/logging" -o yaml | grep -iA 6 taints

執行結果：

 taints:
- effect: NoSchedule
key: logging
value: reserved
- effect: NoExecute
key: logging
value: reserved
--
taints:
- effect: NoSchedule
key: logging
value: reserved
- effect: NoExecute
key: logging
value: reserved
--
taints:
- effect: NoSchedule
key: logging
value: reserved
- effect: NoExecute
key: logging
value: reserved

3.3.3. 編輯 kustomize 設定檔

nano kustomization.yaml

需修改以下項目

elk images name, tag

PV nodeSelector

PV NFS Path

PV NFS Server IP

tolearation for elk and kibana

3.4. dry run 確認設定檔是否有誤

oc apply -k . --dry-run=server

3.5. 將 elk 部署進 ocp

oc apply -k .

3.6. 確認 pod 用哪個 user 在跑

oc get pods -o yaml | grep -i runasuser | sort -u
runAsUser: 1000830000

3.7. 進到 NFS Server 修改對應的權限

chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-data-nodes-0
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-data-nodes-1
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-data-nodes-2
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-master-nodes-0
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-master-nodes-1
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-master-nodes-2

3.8. 確認 pod 狀態

oc get pods

執行結果：

NAME READY STATUS RESTARTS AGE
elasticsearch-es-data-nodes-0 1/1 Running 2 (6m53s ago) 7m21s
elasticsearch-es-data-nodes-1 1/1 Running 2 (6m52s ago) 7m21s
elasticsearch-es-data-nodes-2 1/1 Running 2 (6m50s ago) 7m21s
elasticsearch-es-master-nodes-0 1/1 Running 2 (6m52s ago) 7m21s
elasticsearch-es-master-nodes-1 1/1 Running 2 (6m51s ago) 7m21s
elasticsearch-es-master-nodes-2 1/1 Running 2 (6m50s ago) 7m21s
kibana-kb-6967c5f584-hxqg6 1/1 Running 0 5m46s

3.9. 獲取 URL

oc get route

執行結果：

NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
elasticsearch-sample elasticsearch-sample-elastic.apps.ocp.cbsd.scsb.com.tw elasticsearch-es-http <all> passthrough/Redirect None
kibana-sample kibana-sample-elastic.apps.ocp.cbsd.scsb.com.tw kibana-kb-http <all> passthrough/Redirect None

3.10. login

account: elastic passed: 0Hv2Xln8Fb2V571Hw3ik0ed6

密碼獲取命令

oc -n elastic get secrets elasticsearch-es-elastic-user \
-o go-template='{{.data.elastic | base64decode}}{{"\n"}}'

在 Openshift 上建立應用並驗證掛載 NFS Storage

Thu, 03 Jul 2025 01:56:44 +0800

請在 NFS Server 的主機中執行 Step 1~4 的操作命令，Step 6 請在能夠執行 oc 命令管理 OpenShift 的主機操作

本篇文章的 NFS Server 裝在 RHEL 9.6 上

Step1: 安裝 NFS Server

sudo dnf install -y nfs-utils

Step2: 設定儲存空間

# 1. 在 VM 新增一顆硬碟專門給 NFS Server 使用
# 2. 檢查新的硬碟名稱
$ lsblk
...
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:16 0 70G 0 disk
# 3. 將硬碟名稱設為變數
$ D="/dev/sda"
# 4. 使用 parted 建立 GPT 分割區，從 1 MiB 到磁碟尾，並標記為 LVM 用途
$ sudo parted -s ${D} mklabel gpt
$ sudo parted -s ${D} unit MiB mkpart primary 1 100%
# 5. 將剛建立的分割區初始化為 LVM PV
$ sudo pvcreate ${D}1
# 6. 建立名為 vg-nfs 的 VG
$ sudo vgcreate vg-nfs ${D}1
# 7. 從 VG 中建立一 LV，利用全部剩餘空間並命名為 lv-nfs
$ sudo lvcreate -l +100%FREE -n lv-nfs /dev/vg-nfs
# 8. 格式化 LV 為 XFS 檔案系統，以提供穩健且高效的儲存
$ sudo mkfs.xfs /dev/vg-nfs/lv-nfs
# 9. 建立 NFS Server 要共享給 Client 的目錄
$ sudo mkdir -p /data/nfs
# 10. 修改目錄權限
$ sudo chown -R 65534:65534 /data/nfs
# 11. 將目錄設為開機自動掛載 lv 裝置上
$ echo '/dev/vg-nfs/lv-nfs /data/nfs xfs defaults 0 0' | sudo tee -a /etc/fstab
# 12. 將目錄掛載到 lv 裝置上
$ sudo mount -a && sudo systemctl daemon-reload
# 13. 設定共享目錄
$ echo '/data/nfs 192.168.1.0/24(rw,sync,no_subtree_check,no_root_squash)' | sudo tee /etc/exports
# 14. 設定 NFS Server 使用 4.2 版本，以匯出分享 SELinux context。無適合的版本 client 端掛載時會出現 mount.nfs: Protocol not supported 的訊息。
$ echo 'RPCNFSDARGS="-V 4.2"' | sudo tee /etc/sysconfig/nfs
# 15. 查看目前啟動的 nfs 版本，因 server 指定使用 4.2，若出現 -4.2 表示 nfs server 沒有成功啟動。
$ sudo cat /proc/fs/nfsd/versions
+3 +4 +4.1 +4.2

Step3: 啟動 NFS Server

sudo systemctl enable --now nfs-server
sudo exportfs -arv

Step4: 設定 NFS Server 使用 fsid

# 1. 先手動 resizze vm 硬碟的儲存空間
# 2. 使用 parted 將第 1 區擴展到磁碟剩餘最大空間（自動確認、低互動）
$ sudo parted -s -a optimal /dev/sda "resizepart 1 100%"
# 3. 設定 LVM 更新 PV 已可使用空間
$ sudo pvresize /dev/sda1
# 4. 檢查目前 PV 與剩餘空間狀態
$ sudo pvs
PV VG Fmt Attr PSize PFree
/dev/sda1 vg-nfs lvm2 a-- <80.00g 10.00g
/dev/sdb2 rhel lvm2 a-- <69.00g 0
# 5. 在 vg-nfs 卷組中創建新的 5G 的 LV lv-share1
$ sudo lvcreate -L 5G -n lv-share1 /dev/vg-nfs
# 6. 顯示所有 LV 與大小資訊
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root rhel -wi-ao---- <69.00g
lv-nfs vg-nfs -wi-ao---- <70.00g
lv-share1 vg-nfs -wi-ao---- 5.00g
# 7. 在新的 LV 上建立 XFS 檔案系統
$ sudo mkfs.xfs /dev/vg-nfs/lv-share1
# 8. 建立掛載點資料夾
$ sudo mkdir -p /data/nfs/share1
# 9. 設定資料夾所有者為 nobody:nobody (UID/GID 65534)，通常對應 NFS anonymous user
$ sudo chown -R 65534:65534 /data/nfs/share1
# 10. 將新 LV 自動掛載設定加入 fstab
$ echo '/dev/vg-nfs/lv-share1 /data/nfs/share1 xfs defaults 0 0' | sudo tee -a /etc/fstab
# 11. 掛載全部 fstab 條目，並重載 systemd daemon
$ sudo mount -a; sudo systemctl daemon-reload
# 12. 驗證 /dev/sda 與 LVM 的結構
$ lsblk /dev/sda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 80G 0 disk
└─sda1 8:1 0 80G 0 part
├─vg--nfs-lv--nfs 253:1 0 70G 0 lvm /data/nfs
└─vg--nfs-lv--share1 253:2 0 5G 0 lvm /data/nfs/share1
# 13. 編輯 /etc/exports，NFSv4 範例設定，使用 fsid 唯一識別
$ sudo nano /etc/exports
/data/nfs 192.168.1.0/24(rw,fsid=0,sync,no_subtree_check,no_root_squash)
/data/nfs/share1 192.168.1.0/24(rw,fsid=1,sync,no_subtree_check,no_root_squash)
# 14. 重載 NFS server 配置並立即生效 export
$ sudo systemctl reload nfs-server
$ sudo exportfs -arv
exporting 192.168.1.0/24:/data/nfs/share1
exporting 192.168.1.0/24:/data/nfs

Step5: 在其他機器測試掛載

$ mkdir ~/test
$ sudo mount -t nfs -o hard,nfsvers=4.2 192.168.1.90:/ ~/test/
$ ls -l ~/test/
total 0
drwxr-xr-x. 2 nfsnobody nfsnobody 6 Jun 18 07:35 share1
$ sudo umount ~/test

Step6: 在 OCP 中建應用測試掛載

啟用 NFS Client Provisioner

# 1. 使用 NFS Client provisioner 建立動態供應儲存的 template，並指定 NFS Server 和共享目錄位置
$ oc process -f https://raw.githubusercontent.com/openshift-examples/external-storage-nfs-client/main/openshift-template-nfs-client-provisioner.yaml \
-p NFS_SERVER=192.168.1.90 \
-p NFS_PATH=/ | oc apply -f -
# 2. 建立本地目錄以儲存 StorageClass 等 YAML 定義
$ mkdir ~/test-mssql
# 3. 建立一個新的 StorageClass，使用 pathPattern 區分命名空間與自訂儲存目錄路徑
$ cat <<EOF | tee ~/test-mssql/sc-nfs-share1.yaml | oc apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage-share1
provisioner: redhat-emea-ssa-team/hetzner-ocp4
parameters:
archiveOnDelete: "false"
pathPattern: "share1/\${.PVC.namespace}/\${.PVC.annotations.nfs.io/storage-path}"
EOF
# 4. 確認新的 StorageClass 已建立
$ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
managed-nfs-storage (default) redhat-emea-ssa-team/hetzner-ocp4 Delete Immediate false 30m
managed-nfs-storage-share1 redhat-emea-ssa-team/hetzner-ocp4 Delete Immediate false 3h24m
# 5. 建立一個新的專案用來部署 MSSQL Server
$ oc new-project mssql
# 6. 建立密碼的 secret 給 MSSQL Server 的使用者
$ oc create secret generic mssql --from-literal=SA_PASSWORD="1qaz@WSX"
# 7. 建立 PersistentVolumeClaim，並使用剛建立的 NFS StorageClass
$ cat <<EOF | tee ~/test-mssql/mssql-pvc.yaml | oc apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mssql-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: managed-nfs-storage-share1
EOF
# 8. 驗證 PVC 是否成功綁定 PV
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
mssql-data Bound pvc-e00bad5b-9a86-4344-94d3-49c7bde4b8e1 8Gi RWO managed-nfs-storage-share1 <unset> 52s
# 9. 建立 MSSQL Server 的 Deployment 與 Service，並將資料掛載到 NFS PVC
$ cat <<EOF | tee ~/test-mssql/mssql-deployment.yaml | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: mssql-deployment
spec:
selector:
matchLabels:
app: mssql
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: mssql
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mssql
image: mcr.microsoft.com/mssql/server:2017-CU8-ubuntu
ports:
- containerPort: 1433
env:
- name: MSSQL_PID
value: "Developer"
- name: ACCEPT_EULA
value: "Y"
- name: SA_PASSWORD
valueFrom:
secretKeyRef:
name: mssql
key: SA_PASSWORD
volumeMounts:
- name: mssqldb
mountPath: /var/opt/mssql
volumes:
- name: mssqldb
persistentVolumeClaim:
claimName: mssql-data
---
apiVersion: v1
kind: Service
metadata:
name: mssql
spec:
selector:
app: mssql
ports:
- protocol: TCP
port: 1433
targetPort: 1433
EOF
# 10. 確認 Pod 是否成功建立並執行中
$ oc get pods
NAME READY STATUS RESTARTS AGE
mssql-deployment-7578c99777-bq7dw 1/1 Running 0 3s
# 11. 檢查 NFS Server 上是否已產生 SQL Server 的儲存目錄
$ ls -l /data/nfs/share1/
total 0
drwxrwxrwx. 6 root root 59 Jun 18 15:00 mssql
# 12. 檢查 MSSQL Server 儲存目錄內部結構，包含 data/log/secrets/system 等子目錄
$ ls -al /data/nfs/share1/mssql/
total 4
drwxrwxrwx. 6 root root 59 Jun 18 15:00 .
drwxr-xr-x. 4 nobody nobody 34 Jun 18 14:58 ..
drwxr-xr-x. 2 1000750000 root 156 Jun 18 15:00 data
drwxr-xr-x. 2 1000750000 root 4096 Jun 18 15:00 log
drwxr-xr-x. 2 1000750000 root 25 Jun 18 15:00 secrets
drwxr-xr-x. 5 1000750000 root 74 Jun 18 15:00 .system

清理環境

$ oc delete -R -f ~/test-mssql/
$ oc delete project mssql
$ sudo rm -rf ~/test-mssql/
# execute the following command on NFS Server
$ sudo rm -rf /data/nfs/share1/mssql/

不啟用 StorageClass

# 1. 建立本機目錄以儲存 YAML 定義
$ mkdir ~/test-mssql-manually/
# 2. 建立新的專案（namespace）mssql
$ oc new-project mssql
# 3. 建立 secret，用於 MSSQL 的 SA 密碼
$ oc create secret generic mssql --from-literal=SA_PASSWORD="1qaz@WSX"
# 4. 建立 PVC 與手動對應的 PV，使用 NFS 掛載 RWX 模式
$ cat <<EOF | tee ~/test-mssql-manually/mssql-storage.yaml | oc apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mssql-data
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: ""
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: mssql-data-pv
spec:
capacity:
storage: 5Gi
volumeMode: Filesystem
accessModes:
- ReadWriteMany
mountOptions:
- hard
- nfsvers=4.2
nfs:
path: /share1
server: 192.168.1.90
claimRef:
namespace: mssql
name: mssql-data
EOF
# 5. 確認 PVC 已成功綁定 PV
$ oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
mssql-data Bound mssql-data-pv 5Gi RWX <unset> 10s
# 6. 建立 Deployment 與 Service，掛載 NFS 作為資料目錄
$ cat <<EOF | tee ~/test-mssql-manually/mssql-deployment.yaml | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: mssql-deployment
spec:
selector:
matchLabels:
app: mssql
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: mssql
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mssql
image: mcr.microsoft.com/mssql/server:2017-CU8-ubuntu
ports:
- containerPort: 1433
env:
- name: MSSQL_PID
value: "Developer"
- name: ACCEPT_EULA
value: "Y"
- name: SA_PASSWORD
valueFrom:
secretKeyRef:
name: mssql
key: SA_PASSWORD
volumeMounts:
- name: mssqldb
mountPath: /var/opt/mssql
volumes:
- name: mssqldb
persistentVolumeClaim:
claimName: mssql-data
---
apiVersion: v1
kind: Service
metadata:
name: mssql
spec:
selector:
app: mssql
ports:
- protocol: TCP
port: 1433
targetPort: 1433
EOF
# 7. 檢查 Pod 狀態（第一次會失敗）
$ oc get pods
NAME READY STATUS RESTARTS AGE
mssql-deployment-7578c99777-qcvgv 0/1 Error 2 (30s ago) 32s
# 8. 查看失敗原因：目錄建立失敗，權限錯誤 (Errno 13)
$ oc logs mssql-deployment-7578c99777-qcvgv
/opt/mssql/bin/sqlservr: Error: The secrets directory [var/opt/mssql/secrets/] could not be created. Errno [13]
# 9. 確認 app container 是以哪個 UID 執行
$ oc get pods mssql-deployment-7578c99777-qcvgv -o yaml | grep -i runasuser
runAsUser: 1000780000
# 10. 修正 NFS 路徑權限，將目錄擁有者改為指定 UID
$ sudo chown -R 1000780000 /data/nfs/share1/
# 11. 重啟 Deployment 讓 Pod 重建並應用正確的目錄權限
$ oc rollout restart deployment mssql-deployment
# 12. 確認 Pod 成功運行
$ oc get pods
NAME READY STATUS RESTARTS AGE
mssql-deployment-9fcdd78f8-7d8r8 1/1 Running 0 3s
# 13. 驗證 NFS Server 上的目錄內容是否成功建立
$ ls -l /data/nfs/share1/
total 4
drwxr-xr-x. 2 1000780000 root 156 Jun 18 15:20 data
drwxr-xr-x. 2 1000780000 root 4096 Jun 18 15:20 log
drwxr-xr-x. 2 1000780000 root 25 Jun 18 15:20 secrets

清理環境

$ oc delete -R -f ~/test-mssql-manually/
$ oc delete project mssql
$ sudo rm -rf ~/test-mssql-manually/
# execute the following command on NFS Server
$ sudo rm -rf /data/nfs/share1/*/
$ sudo rm -rf /data/nfs/share1/.system

OpenShift 設定讓不同的叢集節點中特定 namespace 底下所有 Pods 對外上網後顯示同一個 source ip

Thu, 05 Jun 2025 09:35:24 +0800

先備知識

OpenShift 的 Egress IP 功能允許您確保一個或多個 Pod（來自一個或多個專案）的流量，在離開叢集網路時，透過 SNAT 機制，讓屬於特定 namespace 底下 Pods 的流量對外呈現固定 Source IP。

Step1: 建立 Project（Namespace）test-egress

oc new-project test-egress

Step2: 為特定節點指派託管 Egress IP

標記 worker-1 節點以託管 Egress IP 位址

oc label nodes worker-1 k8s.ovn.org/egress-assignable=""

Step3: 為 Namespace 指派 Egress IP

mkdir ~/egressip

編輯與設定 EgressIP YAML File

echo 'apiVersion: k8s.ovn.org/v1
kind: EgressIP
metadata:
name: egressip-test
spec:
egressIPs:
- 10.10.7.19
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: "test-egress"' > ~/egressip/egressip-test.yaml

egressIPs：列出要分配給 Pods 對外呈現的固定來源 IP。

namespaceSelector：以 Label Selector 的方式決定哪些 namespace 可套用此 Egress IP。

kubernetes.io/metadata.name: "test"，表示只選擇名稱為 test-egress 的 namesapce。

podSelector（可選）：可進一步針對特定 Pod 加以篩選；若不填，則預設會對該 namespace 內所有 Pods 生效。

建立 EgressIP

oc apply -f ~/egressip/egressip-test.yaml

檢查 EgressIP 是否指派到節點上

$ oc get egressip

執行結果 :

NAME EGRESSIPS ASSIGNED NODE ASSIGNED EGRESSIPS
egressip-test 10.10.7.19 worker-1 10.10.7.19

實際到 worker-1 節點檢查

$ ssh core@worker-1 'ip addr show | grep -B 6 10.10.7.19'

執行結果 :

5: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether bc:24:11:7f:f2:b0 brd ff:ff:ff:ff:ff:ff
inet 10.10.7.16/24 brd 10.10.7.255 scope global noprefixroute br-ex
valid_lft forever preferred_lft forever
inet 169.254.0.2/17 brd 169.254.127.255 scope global br-ex
valid_lft forever preferred_lft forever
inet 10.10.7.19/32 scope global br-ex

Step4: 建立測試應用

編輯 Deployment YAML File

echo 'apiVersion: apps/v1
kind: Deployment
metadata:
name: source-ip-app
labels:
app: source-ip-app
spec:
replicas: 3
selector:
matchLabels:
app: source-ip-app
strategy: {}
template:
metadata:
labels:
app: source-ip-app
spec:
containers:
- name: echoserver
image: registry.k8s.io/echoserver:1.10
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- worker-1
- worker-2
- worker-3
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- source-ip-app
topologyKey: "kubernetes.io/hostname"' > ~/egressip/source-ip-app.yaml

設定 Namespace 的 default ServiceAccount 使用 anyuid SCC

oc adm policy add-scc-to-user anyuid -z default -n test-egress

建立測試應用

oc apply -f ~/egressip/source-ip-app.yaml

檢視應用 Pods 是否正常

oc get pods -o wide

執行結果 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
source-ip-app-b44758d6d-t894p 1/1 Running 0 3s 10.128.2.27 worker-1 <none> <none>
source-ip-app-b44758d6d-tzpjj 1/1 Running 0 3s 10.131.0.14 worker-2 <none> <none>
source-ip-app-b44758d6d-xvsqg 1/1 Running 0 4s 10.130.0.9 worker-3 <none> <none>

Step5: 在 OpenShift 叢集外部的主機建立外部 HTTP server

安裝 httpd 套建

sudo yum install -y httpd

將 httpd 啟在 8080 Port

sudo sed -i 's|Listen 80|Listen 8080|g' /etc/httpd/conf/httpd.conf

建立測試 HTML 頁面

echo '<html>
<head/>
<body>OK</body>
</html>' | sudo tee /var/www/html/index.html

啟動 httpd Server 服務

sudo systemctl start httpd

確認 httpd Server 服務運作正常

sudo systemctl is-active httpd

正確執行結果 :

active

測試頁面是否正常運作

curl http://localhost:8080/index.html

執行結果 :

<html>
<head/>
<body>OK</body>
</html>

Step6: 測試連線

撰寫 Bash Script 來完成以下目的 :
以 round-robin 的方式對名稱為 source-ip-app-... 前綴的 Pod 執行 curl http://10.10.7.11:8080/index.html
編輯 Bash Script

nano ~/egressip/test.sh

檔案內容如下 :

#!/bin/bash
mapfile -t pods < <(oc -n test-egress get pods -o name | grep source-ip-app-)
if [[ ${#pods[@]} -eq 0 ]]; then
echo "找不到符合條件的 Pod，請確認 Pod 名稱前綴是否正確" >&2
exit 1
fi
for i in {1..20}; do
for pod in "${pods[@]}"; do
pod_name="${pod#pod/}"
echo "對 Pod ${pod_name} 執行 curl 請求..."
oc exec "${pod_name}" -- curl -s http://10.10.7.11:8080/index.html \
&& echo " ==> ${pod_name} 執行成功" \
|| echo " ==> ${pod_name} 執行失敗"
sleep 1
done
done

為 Bash Script 賦予執行權限

chmod +x ~/egressip/test.sh

執行腳本開始測試

~/egressip/test.sh

執行結果 :

對 Pod source-ip-app-b44758d6d-t894p 執行 curl 請求...
<html>
<head/>
<body>OK</body>
</html>
==> source-ip-app-b44758d6d-t894p 執行成功
對 Pod source-ip-app-b44758d6d-tzpjj 執行 curl 請求...
<html>
<head/>
<body>OK</body>
</html>
==> source-ip-app-b44758d6d-tzpjj 執行成功
對 Pod source-ip-app-b44758d6d-xvsqg 執行 curl 請求...
...以下省略

檢視 httpd access log

sudo tail -f /var/log/httpd/access_log

執行結果 :

127.0.0.1 - - [05/Jun/2025:16:51:26 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.76.1"
10.10.7.19 - - [05/Jun/2025:17:01:56 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:01:57 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:01:59 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:00 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:02 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:04 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:05 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:07 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:08 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:10 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:11 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:13 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:15 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
10.10.7.19 - - [05/Jun/2025:17:02:16 +0800] "GET /index.html HTTP/1.1" 200 39 "-" "curl/7.47.0"
...以下結果省略

Step7: 清理環境

oc delete -R -f ~/egressip/
oc label node worker-1 k8s.ovn.org/egress-assignable-
oc adm policy remove-scc-from-user anyuid -z default -n test-egress
oc delete project test-egress
# 以下命令請在 httpd Server 的主機上執行
sudo systemcctl stop httpd
sudo yum remove -y httpd

參考資料

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/networking/ovn-kubernetes-network-plugin#configuring-egress-ips-ovn

OpenShift 如何在 Deployment 不改設定的情況下，設 Deployment pods 跑在特定的某些節點上

Thu, 05 Jun 2025 09:30:54 +0800

Step1: 建立 Project（Namespace）test

oc new-project test

Step2: 給 worker 節點加上標籤，用以後續 Pod 排程條件

oc label node worker-1 app=nginx
oc label node worker-3 app=nginx

Step3: 在 Namespace 層級套用 nodeSelector，Namespace 底下所有沒有指定 nodeSelector 的 Pod 都會自動帶入這個條件

oc annotate namespace test openshift.io/node-selector="app=nginx"

如果 Pods 有指定 nodeSelector 則不影響

Step4(可選): 允許該 Namespace 的 default ServiceAccount 使用 anyuid SCC，讓 app container 可以以 root (UID 0) 執行。

oc adm policy add-scc-to-user anyuid -z default -n test

Step5(可選) : 設定 namespaace 層級 PSA

# 對 namespace "test" 加上 Pod Security Audit 等級為 "privileged" 的標籤
oc label namespace test \
pod-security.kubernetes.io/audit=privileged \
--overwrite

# 對 namespace "test" 加上 Pod Security Warn 等級為 "privileged" 的標籤
oc label namespace test \
pod-security.kubernetes.io/warn=privileged \
--overwrite

Step6: 建立一個測試用 NGINX Deployment，並在 PodSpec 內指定 runAsUser: 0，讓 NGINX 以 root 身份啟動。

echo 'apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: test
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: docker.io/library/nginx
ports:
- containerPort: 80
securityContext:
runAsUser: 0' | oc apply -f -

Step7: 確認 Pod 執行節點與狀態，檢查是否都被正確排程到貼了 app=nginx 標籤的 Node。

oc get pods -o wide

執行結果 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-77979fb7f9-6hh44 1/1 Running 0 9s 10.130.0.15 worker-3 <none> <none>
nginx-77979fb7f9-nf6wg 1/1 Running 0 9s 10.128.2.98 worker-1 <none> <none>
nginx-77979fb7f9-xz24w 1/1 Running 0 9s 10.130.0.16 worker-3 <none> <none>

Step8: 清理環境

oc annotate namespace test openshift.io/node-selector-
oc delete deploy nginx
oc adm policy remove-scc-from-user anyuid -z default -n test
oc label node worker-1 app-
oc label node worker-2 app-
oc delete project test

Disconnected Installation Mirroring

Wed, 14 Aug 2024 04:16:58 +0800

1. 關於離線安裝 Mirroring

可以使用 mirror registry 來確保您的叢集僅使用滿足組織對外部內容控制的 container images 。在限制網路中佈建的基礎結構上安裝叢集之前，您必須將所需的 container images mirror 到該環境中。若要 mirror container images ，必須有一個 mirror registry 。

2. 建立 Container image registry

如果已有 container image registry ，例如 Red Hat Quay，您可以將它用作 mirror registry。

2.1. Prerequisites

一個 OpenShift Container Platform 訂閱.
Red Hat Enterprise Linux (RHEL) 8 and 9 with Podman 3.4.2 or later and OpenSSL installed.
Fully qualified domain name for the Red Hat Quay service, 必須透過 DNS 伺服器解析。
Key-based SSH connectivity on the target host. SSH keys are automatically generated for local installs. For remote hosts, you must generate your own SSH keys.
2 or more vCPUs.
8 GB of RAM.
About 12 GB for OpenShift Container Platform 4.14 release images, or about 358 GB for OpenShift Container Platform 4.14 release images and OpenShift Container Platform 4.14 Red Hat Operator images. Up to 1 TB per stream or more is suggested.

2.2. Procedure

Download the mirror-registry.tar.gz package for the latest version of the mirror registry for Red Hat OpenShift found on the OpenShift console Downloads page.

$ tar -zxvf mirror-registry.tar.gz
image-archive.tar
execution-environment.tar
mirror-registry

Install the mirror registry for Red Hat OpenShift on your local host with your current user account by using the mirror-registry tool. For a full list of available flags, see “mirror registry for Red Hat OpenShift flags”.

$ hostname -f
bastion.ocp4.example.com
$ sudo mkdir /root/ocpmirror
# Check Disk Space
$ lsblk /dev/rhel/root
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
rhel-root 253:0 0 349G 0 lvm /
$ sudo lvs /dev/rhel/root
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root rhel -wi-ao---- 349.00g
## 密碼必須至少有 8 個字元，且不得包含空白字元
$ sudo ./mirror-registry install \
--quayHostname $(hostname -f) \
--quayRoot /root/ocpmirror \
--initUser bigred \
--initPassword srebigred
...
INFO[2024-08-07 10:11:06] Quay installed successfully, config data is stored in /root/ocpmirror
INFO[2024-08-07 10:11:06] Quay is available at https://bastion.ocp4.example.com:8443 with credentials (bigred, srebigred)
$ sudo cp -v /root/ocpmirror/quay-rootCA/rootCA.pem /etc/pki/ca-trust/source/anchors/ && \
sudo update-ca-trust
$ podman login -u bigred -p srebigred $(hostname -f):8443
Login Succeeded!
# 安裝完成後，可以查看已建立的服務:
$ systemctl -a | grep quay
quay-app.service loaded active running Quay Container
quay-pod.service loaded active exited Infra Container for Quay
quay-postgres.service loaded active running PostgreSQL Podman Container for Quay
quay-redis.service loaded active running Redis Podman Container for Quay
# 這些服務會在系統重新啟動時自動啟動。quay-redis, quay-db, 和 quay-app 服務依賴於 quay-pod 服務。可以使用一個命令重新啟動所有服務：
$ sudo systemctl restart quay-pod

3. Mirroring images for a disconnected installation

3.1. 透過下載二進位檔安裝 OpenShift CLI

導覽到 Red Hat Customer Portal 上的 OpenShift Container Platform downloads page。
從 Product Variant 下拉清單中選擇架構。
從 Version 下拉清單中選擇 4.14.33 for RHEL 9 的版本。
按一下 OpenShift v4.14.33 Linux Client 項目旁的立即下載，並儲存檔案。
解壓縮檔案並將 oc 和 kubectl 命令移到 PATH 路徑底下

$ tar -zxvf oc-4.14.33-linux.tar.gz && \
sudo mv oc kubectl /usr/local/bin

3.2. 設定允許 Mirror Images 的憑證

建立 container image registry 憑證檔案，允許從 Red Hat mirror 影像到自建的 container image registry。

在安裝主機上完成下列步驟 :

從 Red Hat OpenShift Cluster Manager 下載您的 registry.redhat.io pull secret。
以 JSON 格式產生您的 pull secret：

$ mkdir -p $HOME/.docker
$ cat pull-secret.txt | jq . > $HOME/.docker/config.json

把私有 Registry 的身分驗證訊息也加到同一個檔案中：

$ podman login -u bigred -p srebigred \
--authfile $HOME/.docker/config.json $(hostname -f):8443
# 檢查
$ cat $HOME/.docker/config.json
{
"auths": {
"bastion.ocp4.example.com:8443": {
"auth": "YmlncmVkOnNyZWJpZ3JlZA=="
},
...以下省略

3.3. Mirroring the OpenShift Container Platform image repository

設定所需的環境變數：

$ OCP_RELEASE=4.14.33 \
LOCAL_REGISTRY='bastion.ocp4.example.com:8443' \
LOCAL_REPOSITORY='ocp4/openshift4' \
PRODUCT_REPO='openshift-release-dev' \
LOCAL_SECRET_JSON="$HOME/.podman/config.json" \
RELEASE_NAME="ocp-release" \
ARCHITECTURE='x86_64'

Mirror the version images to the mirror registry:

oc adm release mirror -a ${LOCAL_SECRET_JSON} \
--from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
--to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
--to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

執行結果 :

...
Success
Update image: bastion.ocp4.example.com:8443/ocp4/openshift4:4.14.33-x86_64
Mirror prefix: bastion.ocp4.example.com:8443/ocp4/openshift4
Mirror prefix: bastion.ocp4.example.com:8443/ocp4/openshift4:4.14.33-x86_64
To use the new mirrored repository to install, add the following section to the install-config.yaml:
imageContentSources:
- mirrors:
- bastion.ocp4.example.com:8443/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- bastion.ocp4.example.com:8443/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
To use the new mirrored repository for upgrades, use the following to create an ImageContentSourcePolicy:
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
name: example
spec:
repositoryDigestMirrors:
- mirrors:
- bastion.ocp4.example.com:8443/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- bastion.ocp4.example.com:8443/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

3.4. Mirroring Operator catalogs for use with disconnected clusters

extract and mirror the content to the mirror registry

# Install oc-mirror
$ sudo tar -zxvf oc-mirror.rhel9.tar.gz -C /usr/local/bin/ && \
sudo chmod +x /usr/local/bin/oc-mirror

Setting up and configuring the Image Registry on OpenShift

Thu, 14 Dec 2023 03:05:28 +0800

Preface

本篇文章會介紹，如何在 OpenShift 上安裝與設定 Image Registry，並透過 Neuvector Scan 它

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

Prerequisites

有一座 OpenShift Container Platform 4.12
並在 OCP 上完成布署 Neuvector 5.1.3

1. Changing the image registry’s management state

Change managementState Image Registry Operator configuration from Removed to Managed. For example:

$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}'

2. Set the image registry storage to an empty directory

$ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'

Configure this option for only non-production clusters.

3. Exposing the registry

To expose the registry using the defaultRoute:

$ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge

4. Get the default registry route:

$ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')

5. Accessing the registry

$ ssh core@master.ocp4.example.com
$ oc login -u kubeadmin -p bbDNC-bVPgX-Jh5Iv-ViEaa https://api-int.ocp4.example.com:6443
$ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
$ podman login -u kubeadmin -p $(oc whoami -t) $HOST

6. Perform `podman pull` and `podman push` operations against your registry:

$ podman pull docker.io/library/alpine:latest
$ podman tag docker.io/library/alpine:latest default-route-openshift-image-registry.apps.ocp4.example.com/openshift/alpine:latest
$ podman push default-route-openshift-image-registry.apps.ocp4.example.com/openshift/alpine:latest

7. Neuvector Registry Scanning Configuration

7.1. Provide token of the service account which has access to all namespaces

## 1. Create service account
$ oc project default
$ oc create sa nvqa
$ oc get sa
## 2. Assign cluster admin role to service account to read all registry
$ oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:default:nvqa
## 3. Get token for the service account
$ oc sa create token nvqa

得到的 Token 應該跟以下螢幕輸出類似 :

eyJhbGciOiJSUzI1NiIsImtpZCI6InNON1dvNG5jQjlyNXZmdV9WUER2YV9iZE40VUdCVEFPU1c5YTZiRHdaM1kifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTcwMjUyMzAzMywiaWF0IjoxNzAyNTE5NDMzLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6ImRlZmF1bHQiLCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoibnZxYSIsInVpZCI6IjAwN2I1ZDgyLWZmNDMtNDliNi05ZDkyLTBhZjgzNjBiOTU5MCJ9fSwibmJmIjoxNzAyNTE5NDMzLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpudnFhIn0.IgVmNC_OsTRUw2WnLv4JNfYfQEgJonTumTfi7dvBb5zNW5GQuyTrzmdT5klMqrVLpT7IFmi1IV4nM530dNY8SwXeCXwjNLdVZd8O1mNqrnkj5uLRVyF617TDVy8JjqZ1GKFWo8vfYQ-ObCOWJwsOhVpU62DKV7LOxLstT2JltqtHeuuHkadih1qoZxDp1Rpfjy4O_ywSn5vUNgvZq6bwx-DCjTbcO0hu832z3NKHWaK_hGdYDx5zusCy5_iWMR0PYTQj9mghCcvH72ID0ggr_lLKT2P6mFsNPX5YenjAlBi-rMBP8cV6tv9c98OZJs2SP7AuH-mqUc3m7-18rssfS3LGBmzF0ludAwYTIaUxXXjDeIVvVZEsypX5kaUlC_79ljnpgSOen9DJwl5Ps6LIzRktgTaVsphRiko6zNp0P38irgyXFK44SsMqHSZaAwh23hw4F3S1H6SUkzKgR5SwDhrqFFH4ZWz46Ijkd83cr5wtWrMxf8JP1SkChgYLD7CFrK-hA1fEw0c3Y4NoqVltAckBzXSkSBklsvUDAoQmhH5RsxHfcTX5loCf16xis_2u9mxmzWhdfB5F9Wl2czcZF-ij90SPGXyjbAhgyKcMOHK5CTQIkKOahSLK5Qfm8S1yU_dSPDRz30hsmgpNnXdCurp8fl_fdAwW8DxvvuY2asQ

7.2. 打開瀏覽器，並連線至 Neuvector Web UI

7.3. 新增 Image Registry

點選左側選單 “Assets” -> “Registries” -> “Add”

7.3. 輸入 Image Registry 資訊

Registry type: OpenShift Registry
Name: ocpregistry
Registry: https://default-route-openshift-image-registry.apps.ocp4.example.com
點選 Authenticated by Token
Token: 將第 7.1. 步驟產生的 Token 全部複製貼上到這個欄位中
filter: openshift/alpine:latest
點選右下角 Submit 按鈕

7.4. Start Scan OpenShift Image Registry

7.4.1. 點選 Start Scan 按鈕，開始掃描 Regstiry

7.4.2. 成功得到 Scan Image 後的結果資訊

Ref

Step by Step Install OCP 4.12 (Online)

Tue, 01 Aug 2023 00:53:58 +0800

Preface

本篇文章會介紹，如何在多台 VM 上使用 user provisioned infrastructure 的方式安裝 1 台 Master Node 和 2 台 Worker Node 架構的 OpenShift Container Platform 4.12

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

安裝流程

安裝和設定 bastion 虛擬主機
安裝和設定 bootstrap 虛擬主機
安裝和設定 master 虛擬主機
安裝和設定 worker1 和 worker2 虛擬主機
開始安裝 OpenShift Container Platform 4.12

各台主機扮演的角色說明

角色和服務	說明
Cluster installer/bastion	發動安裝 OCP 的跳板機, 同時提供 load balance, DNS Server, Web server 的服務
bootstrap	安裝時重要的角色, 透過 installer 將 ocp cluster 角色先部署在此, 再透過 scale out 延伸到 master node
Master node	ocp 資源調節重要節點須為三個
Worker/compute node	運行 Application 的節點
ignition file	安裝時透過 installer 編寫的 yaml 檔, 後續透過執行此檔案安裝 ocp
DNS Server	提供名稱解析和反解析(網址轉成 IP 或是將 IP 轉回網址)
HA Proxy	提供 load balance 的服務
Web Server	讓 bootstrap 、master 和 worker 主機可以透過這個網站伺服器下載 ignition file

Prerequisites

硬體資源需求

環境架構

1 台 bastion (RHEL)
1 台 bootstrap (RHCOS)
1 台 master (RHCOS)
2 台 worker (RHCOS)

主機名稱設定格式

HOSTNAME.CLUSTER_NAME.DOMAIN_NAME

example:

bastion.ocp4.example.com

HOSTNAME 就是 bastion
CLUSTER_NAME 就是 ocp4
DOMAIN_NAME 就是 example.com

Setup bastion Machine

Step 0: 安裝與設定 Red Hat Enterprise Linux

Performing a standard RHEL 8 installation - RedHat Docs

Step 1: 下載 “安裝 OCP 程式” 和 “管理 OpenShift 的 CLI 工具”

至 RedHat 官網連結下載以下兩個項目的檔案 (一定要有帳號)

OpenShift v4.12.24 Linux Installer
OpenShift v4.12 Linux Client

$ wget --show-progress -qO 'openshift-install-linux-4.12.24.tar.gz' "<installer 的下載網址>" && wget --show-progress -qO 'oc-4.12.24-linux.tar.gz' "<Clinet 的下載網址>"

螢幕輸出 :

openshift-install-linux-4.12. 100%[==============================================>] 329.92M 1.87MB/s in 3m 25s
oc-4.12.24-linux.tar.gz 100%[==============================================>] 52.25M 11.1MB/s in 5.7s

oc-4.12.24-linux.tar.gz，是 OpenShift CLI 的壓縮檔 openshift-install-linux-4.12.24.tar.gz，是安裝 OCP 會用到的程式

確認兩個檔案的 checksum

$ [[ "dd032e41022453424377b1bbcc40a8f712059c1e937f04c0e336389b70173b81" == "$(sha256sum oc-4.12.24-linux.tar.gz | cut -d ' ' -f 1)" ]] && echo 'oc-4.12.24-linux.tar.gz ok' && [[ "9f4362464784100ea4e950513989df912fb93420465408c496361e009c73efdc" == "$(sha256sum openshift-install-linux-4.12.24.tar.gz | cut -d ' ' -f 1)" ]] && echo "openshift-install-linux-4.12.24.tar.gz ok"

螢幕輸出 :

oc-4.12.24-linux.tar.gz ok
openshift-install-linux-4.12.24.tar.gz ok

解壓縮檔案，並將 oc、openshift-install 和 kubectl 加至 PATH 環境變數

$ tar -xvf oc-4.12.24-linux.tar.gz && \
tar -xvf openshift-install-linux-4.12.24.tar.gz && \
sudo mv oc kubectl openshift-install /usr/local/bin

螢幕輸出 :

README.md
kubectl
oc
README.md
openshift-install

確認 oc Command 版本

$ oc version

螢幕輸出 :

Client Version: 4.12.24
Kustomize Version: v4.5.7

確認 openshift-install Command 版本

$ openshift-install version

螢幕輸出 :

openshift-install 4.12.24
built from commit 194f806390b2b20668f5ac32d7918f9773b29a5e
release image quay.io/openshift-release-dev/ocp-release@sha256:b0b11eedf91175459b5d7aefcf3936d0cabf00f01ced756677483f5f26227328
release architecture amd64

確認 kubectl Command 版本

$ kubectl version --output=yaml

螢幕輸出 :

clientVersion:
buildDate: "2023-06-26T20:27:30Z"
compiler: gc
gitCommit: ea7c11aac7fa39d9eb733b62251ee1bc4a242c99
gitTreeState: clean
gitVersion: v1.25.2
goVersion: go1.19.10 X:strictfipsruntime
major: "1"
minor: "25"
platform: linux/amd64
kustomizeVersion: v4.5.7

Step 2: 安裝 DNS Server

$ sudo yum -y install bind

螢幕輸出 :

...以上省略
Complete!

編輯 DNS Server 設定檔 named.conf

$ sudo nano /etc/named.conf

要改的地方有兩個部分

將 listen-on port 53 和 allow-query 的值, 改成 any

檔案內容 :

options {
listen-on port 53 { any; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
secroots-file "/var/named/data/named.secroots";
recursing-file "/var/named/data/named.recursing";
allow-query { any; };

假設我 bastion 這台機器的 Hostname 是 bastion.ocp4.example.com IP 是 192.168.11.80，在 /etc/named.conf 新增以下內容至檔案的最後面 :

zone "example.com" {
type master;
file "/etc/named/zones/db.example.com";
};
zone "11.168.192.in-addr.arpa" {
type master;
file "/etc/named/zones/db.reverse";
};

在本次範例中

Cluster name 是 ocp4
Base Domain Name 是 example.com

設定 DNS Server 名稱解析

$ sudo mkdir /etc/named/zones && sudo nano /etc/named/zones/db.example.com

檔案內容如下 :

$TTL 1W
@ IN SOA bastion.ocp4.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS bastion.ocp4.example.com.
IN MX 10 smtp.example.com.
;
;
bastion.ocp4.example.com. IN A 192.168.11.80
smtp.example.com. IN A 192.168.11.80
;
api.ocp4.example.com. IN A 192.168.11.80
api-int.ocp4.example.com. IN A 192.168.11.80
;
*.apps.ocp4.example.com. IN A 192.168.11.80
;
bootstrap.ocp4.example.com. IN A 192.168.11.81
;
master.ocp4.example.com. IN A 192.168.11.82
;
worker1.ocp4.example.com. IN A 192.168.11.83
worker2.ocp4.example.com. IN A 192.168.11.84
;
;EOF

Note: 紅字部分為須依照環境的規劃來設定

設定 DNS Server 反解析

$ sudo nano /etc/named/zones/db.reverse

檔案內容如下 :

$TTL 1W
@ IN SOA bastion.ocp4.example.com. root (
2019070700 ; serial
3H ; refresh (3 hours)
30M ; retry (30 minutes)
2W ; expiry (2 weeks)
1W ) ; minimum (1 week)
IN NS bastion.ocp4.example.com.
;
80.11.168.192.in-addr.arpa. IN PTR api.ocp4.example.com.
80.11.168.192.in-addr.arpa. IN PTR api-int.ocp4.example.com.
;
81.11.168.192.in-addr.arpa. IN PTR bootstrap.ocp4.example.com.
;
82.11.168.192.in-addr.arpa. IN PTR master.ocp4.example.com.
;
83.11.168.192.in-addr.arpa. IN PTR worker1.ocp4.example.com.
84.11.168.192.in-addr.arpa. IN PTR worker2.ocp4.example.com.
;
;EOF

Note: 紅字部分為須依照環境的規劃來設定

啟動 DNS Server

$ sudo systemctl enable named --now

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/named.service → /usr/lib/systemd/system/named.service.

檢查

$ sudo systemctl status named

螢幕輸出 :

● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 16:50:31 CST; 1s ago
...以下省略

將 bastion 機器的 IP 新增為 DNS Server

$ sudo nmcli connection modify 'ens18' ipv4.dns '192.168.11.80' +ipv4.dns '8.8.8.8'
$ sudo systemctl restart NetworkManager

檢查

$ sudo cat /etc/resolv.conf

螢幕輸出

# Generated by NetworkManager
search ocp4.example.com
nameserver 192.168.11.80
nameserver 8.8.8.8

關閉防火牆

$ sudo systemctl disable firewalld.service --now

檢查是否關閉

$ sudo systemctl status firewalld.service --no-pager

螢幕輸出 :

● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
...以下省略

驗證 DNS Server 正解

host bastion.ocp4.example.com && \
host api.ocp4.example.com && \
host api-int.ocp4.example.com && \
host 111.apps.ocp4.example.com && \
host bootstrap.ocp4.example.com && \
host master.ocp4.example.com && \
host worker1.ocp4.example.com && \
host worker2.ocp4.example.com

螢幕輸出 :

bastion.ocp4.example.com has address 192.168.11.80
api.ocp4.example.com has address 192.168.11.80
api-int.ocp4.example.com has address 192.168.11.80
111.apps.ocp4.example.com has address 192.168.11.80
bootstrap.ocp4.example.com has address 192.168.11.81
master.ocp4.example.com has address 192.168.11.82
worker1.ocp4.example.com has address 192.168.11.83
worker2.ocp4.example.com has address 192.168.11.84

驗證 DNS Server 反解

host 192.168.11.80 && \
host 192.168.11.81 && \
host 192.168.11.82 && \
host 192.168.11.83 && \
host 192.168.11.84

螢幕輸出 :

80.11.168.192.in-addr.arpa domain name pointer api-int.ocp4.example.com.
80.11.168.192.in-addr.arpa domain name pointer api.ocp4.example.com.
81.11.168.192.in-addr.arpa domain name pointer bootstrap.ocp4.example.com.
82.11.168.192.in-addr.arpa domain name pointer master.ocp4.example.com.
83.11.168.192.in-addr.arpa domain name pointer worker1.ocp4.example.com.
84.11.168.192.in-addr.arpa domain name pointer worker2.ocp4.example.com.

Step 3: 安裝 HAProxy 服務

$ sudo yum -y install haproxy

螢幕輸出 :

...以上省略
Complete!

設定 HAProxy

直接清掉預設的設定檔

$ cat /dev/null | sudo tee /etc/haproxy/haproxy.cfg

再編輯設定檔

$ sudo nano /etc/haproxy/haproxy.cfg

將以下內容複製到檔案中 :

global
log 127.0.0.1 local2
pidfile /var/run/haproxy.pid
maxconn 4000
daemon
defaults
mode http
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen api-server-6443
bind *:6443
mode tcp
server bootstrap bootstrap.ocp4.example.com:6443 check inter 1s backup
server master master.ocp4.example.com:6443 check inter 1s
listen machine-config-server-22623
bind *:22623
mode tcp
server bootstrap bootstrap.ocp4.example.com:22623 check inter 1s backup
server master master.ocp4.example.com:22623 check inter 1s
listen ingress-router-443
bind *:443
mode tcp
balance source
server worker1 worker1.ocp4.example.com:443 check inter 1s
server worker2 worker2.ocp4.example.com:443 check inter 1s
listen ingress-router-80
bind *:80
mode tcp
balance source
server worker1 worker1.ocp4.example.com:80 check inter 1s
server worker2 worker2.ocp4.example.com:80 check inter 1s

Note: 紅字部分為須依照環境的規劃來設定

設定允許 HAProxy 可以使用 TCP Port

$ sudo setsebool -P haproxy_connect_any=1

If you are using HAProxy as a load balancer and SELinux is set to enforcing, you must ensure that the HAProxy service can bind to the configured TCP port by running setsebool -P haproxy_connect_any=1.

啟動 HAProxy 服務，並設為開機自動啟動

$ sudo systemctl enable --now haproxy.service

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

檢查

$ sudo systemctl status haproxy.service

螢幕輸出 :

● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 17:26:21 CST; 11min ago
...以下省略

Step 4: 安裝 Web Server

$ sudo yum install -y httpd

將網站的 TCP Port 改成 8080

$ sudo nano /etc/httpd/conf/httpd.conf

修改以下檔案內容，將 80 Port 改為 8080 Port

Listen 8080

或是用 sed 直接修改 : sudo sed -i 's|Listen 80|Listen 8080|g' /etc/httpd/conf/httpd.conf

修改 ssl.conf 檔名

$ sudo mv /etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf.bak

沒修改此檔名，可能會再啟動 httpd 服務時，遇到以下錯誤訊息 : Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443

“could not bind to address” error - 解決辦法連結

啟動 httpd 服務，並設為開機自動啟動

$ sudo systemctl enable --now httpd

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.

檢查

$ sudo systemctl status httpd

螢幕輸出 :

● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2023-07-17 20:34:55 CST; 6s ago
...以下省略

Step 5: 產生 SSH Key

$ ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519

螢幕輸出 :

Generating public/private ed25519 key pair.
Created directory '/home/rancher/.ssh'.
Your identification has been saved in /home/rancher/.ssh/id_ed25519.
Your public key has been saved in /home/rancher/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:fwnYQ9U9iwOe+KzbaLTz3FbX3gZnGMg5MQTFemFGjPk rancher@bastion.ocp4.example.com
The key's randomart image is:
+--[ED25519 256]--+
| .X=. . |
| ooO ...|
| +*o*. o|
| =.oEo.. |
| S *. ..o.|
| o = .o.=|
| . + o .=o|
| =+... +|
| .o++.. . |
+----[SHA256]-----+

啟動 SSH 代理

$ eval "$(ssh-agent -s)"

將 SSH private key 加入 ssh-agent

$ ssh-add ~/.ssh/id_ed25519

Step 6: 下載 pull-secret

至以下連結下載，注意 : 需先登入帳號

Download Pull secret

可點選 “Download pull secret"，或是點 “Copy pull secret” 直接將 pull-secret 複製到剪貼簿

Step 7: 撰寫 install-config.yaml

$ mkdir ocpinstall && nano ocpinstall/install-config.yaml

Note: install-config.yaml 這個檔案名稱不可亂改，只能取這個名稱

將以下檔案內容，修改符合當前環境之設定值

apiVersion: v1
baseDomain: example.com
compute:
- hyperthreading: Enabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Enabled
name: master
replicas: 1
metadata:
name: ocp4
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
fips: false
pullSecret: '{"auths": ...}'
sshKey: 'ssh-ed25519 AAAA...'

Note: 紅字部分為須依照環境的規劃來設定

baseDomain ，The base domain of the cluster.

使用 user-provisioned 的方式建立 OCP 時，不要改 compute.replicas = 0 這個數字

In installer-provisioned installations, the parameter controls the number of compute machines that the cluster creates and manages for you.
In user-provisioned installations, you must manually deploy the compute machines before you finish installing the cluster.

controlPlane.replicas = 1 ，設定 ControlPlane Node 的數量為 1 台

metadata.name，設定 OCP 的 Cluster Name

pullSecret ，將上一個步驟複製的 pullSecret 複製到這

sshKey，bastion 主機的 SSH 公鑰，The SSH public key for the core user in Red Hat Enterprise Linux CoreOS (RHCOS).(可在 bastion 主機，透過 SSH 免密碼的方式登入 RHCOS 的機器)

檢視 SSH 公鑰命令 : cat ~/.ssh/id_ed25519.pub

備份 install-config.yaml

$ cp ~/ocpinstall/install-config.yaml ~/ocpinstall/install-config.yaml.bak

install-config.yaml 之後會不見, 會透過 openshift-install 指令變成 XXX.ign 檔案, 以防萬一有地方寫錯之後要修改, 這邊先備份

Step 8: Creating the Kubernetes manifest and Ignition config files

$ openshift-install create manifests --dir ~/ocpinstall

--dir，specify the installation directory that contains the install-config.yaml file you created.

螢幕輸出 :

INFO Consuming Install Config from target directory
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
INFO Manifests created in: /home/rancher/ocpinstall/manifests and /home/rancher/ocpinstall/openshift

Because you must modify some cluster definition files and manually start the cluster machines, you must generate the Kubernetes manifest and Ignition config files that the cluster needs to configure the machines.

The installation configuration file transforms into the Kubernetes manifests. The manifests wrap into the Ignition configuration files, which are later used to configure the cluster machines.

設定 master Node 不要 Run Pod

$ sed -i 's|mastersSchedulable: true|mastersSchedulable: false|g' ocpinstall/manifests/cluster-scheduler-02-config.yml

Create the Ignition configuration files

$ openshift-install create ignition-configs --dir ocpinstall

螢幕輸出 :

INFO Consuming Openshift Manifests from target directory
INFO Consuming Master Machines from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Common Manifests from target directory
INFO Ignition-Configs created in: /home/rancher/ocpinstall and /home/rancher/ocpinstall/auth

將 Ignition configuration files 放至網站

$ sudo mkdir /var/www/html/ocp4 && \
sudo cp -R ~/ocpinstall/* /var/www/html/ocp4 && \
sudo chown -R apache: /var/www/html/ocp4 && \
sudo chmod 755 /var/www/html/ocp4

檢查

打開瀏覽器，連線至

http://192.168.11.80:8080/ocp4/

Setup bootstrap machine

Installing RHCOS by using an ISO image

1. 取得 bootstrap.ign 啟動設定檔案的 SHA512 雜湊值

在 bastion 主機執行以下命令

$ sha512sum ~/ocpinstall/bootstrap.ign

螢幕輸出 :

e11591138d028175664c5779ca696c568aaf6dc3849a0facb7e76f160184284e451131968786d2c81648d0425668a922e159541c98287be07438a596d6d84d2b ocpinstall/bootstrap.ign

在稍後的步驟中，這個 SHA512 雜湊值會被提供給「coreos-installer」，以驗證叢集節點上 Ignition 設定檔的真實性。

2. 透過 openshift-install 命令獲取正確版本的 RHCOS ISO 檔

在 bastion 機器執行以下命令

$ openshift-install coreos print-stream-json | grep '\.iso[^.]'

螢幕輸出 :

... 以上省略
"location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202306132230-0/x86_64/rhcos-412.86.202306132230-0-live.x86_64.iso",

下載 x86_64 版本的 RHCOS ISO 檔，並透過這個 ISO 檔，將 RHCOS 裝起來

下載命令 :

$ wget https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202306132230-0/x86_64/rhcos-412.86.202306132230-0-live.x86_64.iso

3. 使用上個步驟下載好的 ISO 檔安裝 RHCOS

開始安裝後，不用指定任何選項或中斷實時啟動順序。等待安裝程式進入 RHCOS 實機環境的命令提示字元。

4. 允許 SSH 使用密碼登入和設定網路、主機名稱

設定 SSH Server 允許透過密碼登入

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service

更改 core 使用者密碼

$ echo -e "rancher\nrancher" | sudo passwd core

設定固定 IP: 192.168.11.81/24
Gateway: 192.168.11.254
DNS: 192.168.11.80, 8.8.8.8
關閉 Ipv6

$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.81/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled
$ sudo nmcli connection up 'Wired connection 1'

設定 hostname: boostrap.ocp4.example.com

$ sudo hostnamectl set-hostname bootstrap-ocp4-example.com
$ sudo systemctl restart NetworkManager

Setup master machine

Installing RHCOS by using an ISO image

1. 取得 master.ign 啟動設定檔案的 SHA512 雜湊值

在 bastion 主機執行以下命令

$ sha512sum ocpinstall/master.ign

螢幕輸出 :

46be965b1657dc170203ce83b9ff2ffddf34004da83cecf2440e0c50dec06ea32296c0b2c21a8324fb321ba6592ee5d12108c362d5b98ac3e5a0edf3d8fce7a8 ocpinstall/master.ign

在稍後的步驟中，這個 SHA512 雜湊值會被提供給「coreos-installer」，以驗證叢集節點上 Ignition 設定檔的真實性。

2. 使用與 bootstrap 機器相同的 ISO 檔安裝 RHCOS

3. 允許 SSH 使用密碼登入和設定網路、主機名稱

設定 SSH Server 允許透過密碼登入

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service

更改 core 使用者密碼

$ echo -e "rancher\nrancher" | sudo passwd core

設定固定 IP: 192.168.11.82/24
Gateway: 192.168.11.254
DNS: 192.168.11.80, 8.8.8.8
關閉 Ipv6

$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.82/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled
$ sudo nmcli connection up 'Wired connection 1'

設定 hostname: master.ocp4.example.com

$ sudo hostnamectl set-hostname master-ocp4-example.com
$ sudo systemctl restart NetworkManager

Setup worker1 machine

Installing RHCOS by using an ISO image

1. 取得 worker.ign 啟動設定檔案的 SHA512 雜湊值

在 bastion 主機執行以下命令

$ sha512sum ocpinstall/worker.ign

螢幕輸出 :

c61b89807d86ef9974aaa0451015205bf9e7534b9506b52fb8883c3411c1ba1c5ec0e3080c7d35bb782cd847aa64533552b3f2132ce9b0c8fc6cf4e62933f8c6 ocpinstall/worker.ign

在稍後的步驟中，這個 SHA512 雜湊值會被提供給「coreos-installer」，以驗證叢集節點上 Ignition 設定檔的真實性。

2. 使用與 bootstrap 機器相同的 ISO 檔安裝 RHCOS

3. 允許 SSH 使用密碼登入和設定網路、主機名稱

設定 SSH Server 允許透過密碼登入

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service

更改 core 使用者密碼

$ echo -e "rancher\nrancher" | sudo passwd core

設定固定 IP: 192.168.11.83/24
Gateway: 192.168.11.254
DNS: 192.168.11.80, 8.8.8.8
關閉 Ipv6

$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.83/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled
$ sudo nmcli connection up 'Wired connection 1'

設定 hostname: worker1.ocp4.example.com

$ sudo hostnamectl set-hostname worker1-ocp4-example.com
$ sudo systemctl restart NetworkManager

Setup worker2 machine

Installing RHCOS by using an ISO image

1. 使用與 bootstrap 機器相同的 ISO 檔安裝 RHCOS

2. 允許 SSH 使用密碼登入和設定網路、主機名稱

設定 SSH Server 允許透過密碼登入

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service

更改 core 使用者密碼

$ echo -e "rancher\nrancher" | sudo passwd core

設定固定 IP: 192.168.11.84/24
Gateway: 192.168.11.254
DNS: 192.168.11.80, 8.8.8.8
關閉 Ipv6

$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.84/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled
$ sudo nmcli connection up 'Wired connection 1'

設定 hostname: worker2.ocp4.example.com

$ sudo hostnamectl set-hostname worker2-ocp4-example.com
$ sudo systemctl restart NetworkManager

Starting the OpenShift Container Platform bootstrap process

1. 在 bootstrap 機器執行以下安裝命令

$ sudo coreos-installer install \
--firstboot-args 'rd.neednet=1' \
--copy-network \
--ignition-url=http://192.168.11.80:8080/ocp4/bootstrap.ign \
/dev/sda \
--ignition-hash=sha512-e11591138d028175664c5779ca696c568aaf6dc3849a0facb7e76f160184284e451131968786d2c81648d0425668a922e159541c98287be07438a596d6d84d2b

螢幕輸出 :

Installing Red Hat Enterprise Linux CoreOS 412.86.202305030814-0 (Ootpa) x86_64 (512-byte sectors)
> Read disk 4.1 GiB/4.1 GiB (100%)
Writing Ignition config
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/Wired connection 1.nmconnection to installed system
Install complete.

重新開機

$ sudo reboot

2. 在 master 機器執行以下安裝命令

$ sudo coreos-installer install \
--firstboot-args 'rd.neednet=1' \
--copy-network \
--ignition-url=http://192.168.11.80:8080/ocp4/master.ign \
/dev/sda \
--ignition-hash=sha512-46be965b1657dc170203ce83b9ff2ffddf34004da83cecf2440e0c50dec06ea32296c0b2c21a8324fb321ba6592ee5d12108c362d5b98ac3e5a0edf3d8fce7a8

螢幕輸出 :

> Read disk 4.1 GiB/4.1 GiB (100%)
Writing Ignition config
Writing first-boot kernel arguments
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/Wired connection 1.nmconnection to installed system
Install complete.

重新開機

$ sudo reboot

3. Monitor the bootstrap process

回到 bastion 主機，透過 ssh 連線至 bootstrap 主機

先刪除舊的公鑰紀錄

$ rm .ssh/known_hosts

ssh 連線至 bootstrap 主機

$ ssh -i .ssh/id_ed25519 core@192.168.11.81

檢視 bootstrap 主機安裝到哪個步驟

$ journalctl -b -f -u bootkube.service

約等 20 分鐘後的正確螢幕輸出 :

...以上省略
Jul 18 02:36:10 bootstrap.ocp4.example.com bootkube.sh[10836]: 0718 02:36:10.911230 1 waitforceo.go:64] Cluster etcd operator bootstrapped successfullI0718 02:36:10.913059 1 waitforceo.go:58] cluster-etcd-operator bootstrap etcd
Jul 18 02:36:11 bootstrap.ocp4.example.com bootkube.sh[7433]: bootkube.service complete
Jul 18 02:36:11 bootstrap.ocp4.example.com systemd[1]: bootkube.service: Succeeded.

4. 等待 bootstrap process 安裝完成

$ openshift-install --dir ocpinstall wait-for bootstrap-complete --log-level=debug

螢幕輸出 :

...以上省略
INFO Waiting up to 30m0s (until 10:40AM) for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
DEBUG Time elapsed per stage:
DEBUG Bootstrap Complete: 25m47s
INFO Time elapsed: 25m47s

看到 INFO It is now safe to remove the bootstrap resources 代表安裝完成

5. 加入 worker Node

在 worker1 和 worker2 主機執行以下命令

$ sudo coreos-installer install \
--firstboot-args 'rd.neednet=1' \
--copy-network \
--ignition-url=http://192.168.11.80:8080/ocp4/worker.ign \
/dev/sda \
--ignition-hash=sha512-c61b89807d86ef9974aaa0451015205bf9e7534b9506b52fb8883c3411c1ba1c5ec0e3080c7d35bb782cd847aa64533552b3f2132ce9b0c8fc6cf4e62933f8c6

螢幕輸出 :

> Read disk 4.1 GiB/4.1 GiB (100%)
Writing Ignition config
Writing first-boot kernel arguments
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/Wired connection 1.nmconnection to installed system
Install complete.

分別重啟 Node

$ suddo reboot

6. 使用 CLI 登錄 OCP Cluster

在 Bastion 主機執行以下命令

$ mkdir .kube && \
cp ocpinstall/auth/kubeconfig .kube/config

檢視 OCP Cluster Node 的狀態

$ oc get nodes

螢幕輸出 :

NAME STATUS ROLES AGE VERSION
master.ocp4.example.com Ready control-plane,master 21m v1.25.11+1485cc9

在某些 CSR 獲得批准之前，工作節點還無法被加入 OCP 叢集。

7. 核准 Worker Node 的 CSRs (憑證請求)

檢視 CSRs 的狀態

$ oc get csr

csr is short for “certificate signing request”

螢幕輸出 :

NAME AGE SIGNERNAME REQUESTOR
REQUESTEDDURATION CONDITION
csr-5kwr7 10m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending
csr-hmrpl 10m kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending

核准 CSRs

$ oc adm certificate approve csr-5kwr7
$ oc adm certificate approve csr-hmrpl

或者也可以執行以下命令，一次核准所有的 CSRs

$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

超級注意 ! 由於 CSRs 會自動輪換，請在將機器新增到集群後的一小時內核准 CSRs。

8. 檢視 Worker Node 是否成功加入 OCP 叢集

$ oc get nodes

螢幕輸出 :

NAME STATUS ROLES AGE VERSION
master.ocp4.example.com Ready control-plane,master,worker 3h16m v1.25.11+1485cc9
worker1.ocp4.example.com Ready worker 16m v1.25.11+1485cc9
worker2.ocp4.example.com Ready worker 16m v1.25.11+1485cc9

It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

9. 檢視整個叢集是否健康

$ oc get co

螢幕輸出 :

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.12.24 True False False 121m
baremetal 4.12.24 True False False 9d
cloud-controller-manager 4.12.24 True False False 9d
cloud-credential 4.12.24 True False False 9d
cluster-autoscaler 4.12.24 True False False 9d
config-operator 4.12.24 True False False 9d
console 4.12.24 True False False 15h
control-plane-machine-set 4.12.24 True False False 9d
csi-snapshot-controller 4.12.24 True False False 9d
dns 4.12.24 True False False 9d
etcd 4.12.24 True False False 9d
image-registry 4.12.24 True False False 9d
ingress 4.12.24 True False False 15h
insights 4.12.24 True False False 9d
kube-apiserver 4.12.24 True False False 9d
kube-controller-manager 4.12.24 True False False 9d
kube-scheduler 4.12.24 True False False 9d
kube-storage-version-migrator 4.12.24 True False False 9d
machine-api 4.12.24 True False False 9d
machine-approver 4.12.24 True False False 9d
machine-config 4.12.24 True False False 9d
marketplace 4.12.24 True False False 9d
monitoring 4.12.24 True False False 18h
network 4.12.24 True False False 9d
node-tuning 4.12.24 True False False 9d
openshift-apiserver 4.12.24 True False False 18h
openshift-controller-manager 4.12.24 True False False 8d
openshift-samples 4.12.24 True False False 9d
operator-lifecycle-manager 4.12.24 True False False 9d
operator-lifecycle-manager-catalog 4.12.24 True False False 9d
operator-lifecycle-manager-packageserver 4.12.24 True False False 9d
service-ca 4.12.24 True False False 9d
storage 4.12.24 True False False 9d

10. 登入 Web Console

帳號是 : kubeadmin
密碼會放在 Bastion 主機的 ~/ocpinstall/auth/ 目錄下的 kubeadmin-password 檔案中
網址可以透過以下命令獲得

$ oc whoami --show-console

螢幕輸出 :

https://console-openshift-console.apps.ocp4.example.com

成功登入後的畫面

TroubleShooting

Issue: Canary route checks for the default ingress controller are failing

$ oc get clusteroperators ingress -oyaml

完整錯誤訊息如下 :

 The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-7bcdc5b9c7-kfcmb" cannot be scheduled: 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.), DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 0/1 of replicas are available), CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)

開始找真正的原因

執行以下命令，測試 pod 是否可以透過 curl 命令使用 port 8080 連線到自己。所有的 curl 請求都應該傳回 200 狀態碼，如果狀態碼不等於 200，表示 pod 尚未運作。

$ for i in `oc get po -n openshift-ingress-canary | grep -vi name | awk '{print $1}' ` ; do echo $i ; oc exec -n openshift-ingress-canary $i -- curl -vI http://localhost:8080/ ; done

如果問題是最近才發生的，請等個幾分鐘（1或2分鐘），然後再檢查一次。
如果問題一直存在，請重新啟動 pod。
如果都沒問題，就繼續往下一個步驟繼續找問題的真正原因

執行以下命令，檢查 pod 是否可以使用 service 端點進行 curl。所有的 curl 請求都應該傳回 200 狀態碼，如果狀態碼不等於 200，表示 service IP 無法路由傳入的流量。

$ oc get svc -n openshift-ingress-canary
$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator $i -- curl http://<service-cluster-IP>:8080 -v ; done

service-cluster-IP，要記得換

此服務由名為「openshift-ingress-canary」Namespace 中的 daemonset 「ingress-canary」管理。如果狀態碼不等於 200，請重啟「cluster-version-operator」Pod，然後重新啟動 daemonset。

執行以下命令，檢查對路由的連線，Curl 請求應該會得到 302 的回應碼。

$ oc get route -n openshift-ingress-canary
$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator $i -- curl http://canary-openshift-ingress-canary.apps.<cluster-name>.<domain-name> -vIL ; done

cluster-name 和 domain-name 要記得換

檢查 load-balancer 是否將流量導向 router pod。
如果路由不存在或狀態碼不等於 302，請重新啟動 openshift-ingress Namespace 中的 cluster-version operator 和 router pods。

參考文章

Upgrade process fails in Openshift Container Platform 4 because the Ingress Operator failed: CanaryChecksRepetitiveFailures

Issue: Router-Default 的 pod 無法被指派到叢集中的任何一台 Node 上

$ oc get pods -n openshift-ingress:
NAME READY STATUS RESTARTS AGE
router-default-84ff4c69cd-wrzxj 0/1 Pending 0 18m

Ingress operator 報告了有關污點不容忍和節點放置的錯誤

執行以下命令看 ingress 的 message 訊息

$ oc get co/ingress -o yaml

看看是不是跟以下錯誤訊息類似

- lastTransitionTime: '2021-08-11T16:32:23Z'
message: 'Some ingresscontrollers are degraded: ingresscontroller "default" is
degraded: DegradedConditions: One or more other status conditions indicate a
degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled:
Pod "router-default-84ff4c69cd-jlkh7" cannot be scheduled: 0/9 nodes are available:
3 node(s) didn''t match Pod''s node affinity, 3 node(s) had taint {node-role.kubernetes.io/infra:
}, that the pod didn''t tolerate, 3 node(s) had taint {node-role.kubernetes.io/master:
}, that the pod didn''t tolerate. Make sure you have sufficient worker nodes.),
DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 1/2 of
replicas are available)'
reason: IngressControllersDegraded
status: 'True'
type: Degraded

真正發生問題的原因

Ingresscontroller/default 有透過 node Selector 指定要到 mater node 上，但是 master node 上又有 Taints，所以 Pod 沒有任何一台主機可以用

$ oc get deploy -n openshift-ingress -o yaml | grep -A1 nodeS
nodeSelector:
node-role.kubernetes.io/master: ""

解決辦法

必需將 router-default pod 放置在 Worker 節點上。
在 openshift-ingress-operator namespace 中對 ingresscontroller/default 資源進行 oc patch 推送失敗，或者在 oc edit 編輯後也無法更改。

執行以下命令產生一個新的 default IngressController 的 Yaml 檔

echo 'apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
nodePlacement:
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
replicas: 2
status: {}
kind: List
metadata:
resourceVersion: ""
selfLink: ""' > router-default.yaml

覆蓋現有的 `ingresscontroller/default`

$ oc apply -f router-default.yaml

稍微等幾分鐘後，檢視問題是否被修復

$ oc get all -n openshift-ingress

螢幕輸出 :

NAME READY STATUS RESTARTS AGE
pod/router-default-75c46bd896-7dzvp 1/1 Running 0 16h
pod/router-default-75c46bd896-tvvpx 1/1 Running 0 16h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/router-internal-default ClusterIP 172.30.249.206 <none> 80/TCP,443/TCP,1936/TCP 9d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/router-default 2/2 2 2 9d
NAME DESIRED CURRENT READY AGE
replicaset.apps/router-default-5b9868bbcc 0 0 0 18h
replicaset.apps/router-default-75c46bd896 2 2 2 16h
replicaset.apps/router-default-7bcdc5b9c7 0 0 0 9d

Antony 筆記：技術原理與實務經驗 – OpenShift

透過 VMware Aria Operations 監控 openshift

1. 先決條件

2. 環境準備

2.1. 在 Kubernetes 中安裝 VMware Aria Operations Management Pack

2.1.1. 先決條件

2.1.2. 開始安裝

How to Create a User Using Htpasswd in Openshift

參考文件

ocp to vmware loginsight

Install Elasticsearch Metrics Exporter on Openshift

1. 環境資訊

2. 前置作業

3. 開始設定與安裝 Elasticsearch Metrics Exporter

4. 驗證 Prometheus 是否可收集到 Elasticsearch 的 metrics

5. 設定 Grafana 呈現 Prometheus 收集到的 Elasticsearch Metrics

修復 ocp 叢集憑證過期

1. 環境資訊

2. 問題描述

3. 問題根原因

4. 診斷步驟

4. 解決問題

Step by Step install OpenShift 4.18 with the Agent-based Installer (Online)

Preface

1. 先備知識

1.1. 什麼是 Agent-based Installer

1.2. 理解 Agent-based Installer 安裝流程

1.3. 各台主機扮演的角色說明

2. 準備安裝

2.1. 先決條件

2.1.1. 硬體資源需求

2.1.2. 環境架構

2.1.3. 主機名稱設定格式

2.1.4. 軟體授權

3. 安裝與設定 Bastion 主機

Step 0: 安裝與設定 Red Hat Enterprise Linux 9

Step 1: 下載 “安裝 OCP 程式” 和 “管理 OpenShift 的 CLI 工具”

至 RedHat 官網連結 下載以下兩個項目的檔案 (一定要有帳號)

解壓縮檔案，並將 oc、openshift-install 和 kubectl 加至 PATH 環境變數

確認 oc Command 版本

確認 openshift-install Command 版本

確認 kubectl Command 版本

Step 2: 安裝 DNS Server

編輯 DNS Server 設定檔 named.conf

設定 DNS Server 名稱解析

設定 DNS Server 反解析

啟動 DNS Server

將 bastion 機器的 IP 新增為 DNS Server

關閉防火牆

驗證 DNS Server 正解

驗證 DNS Server 反解

Step 3: 安裝 HAProxy 服務

設定 HAProxy

設定允許 HAProxy 可以使用 TCP Port

啟動 HAProxy 服務，並設為開機自動啟動

Step 4: 下載 pull-secret

Step5: 編輯必要叢集設定檔

產生目錄

設定 install-config.yaml

install-config.yaml 欄位介紹

設定 agent-config.yaml

備份設定檔

產生安裝用 ISO

Step6: 開始安裝 RedHat OpenShift

1. 將 ISO 上傳至虛擬化平台，並掛載到 VM 上

2. 在虛擬化平台將各節點透過 ISO 開機

3. 追蹤和驗證安裝進度 (在 Bastion 主機執行以下指令)

確認重開機後，OpenShift 自動安裝完成

設定 KubeConfig

檢查叢集節點狀態

檢視整個叢集核心元件是否健康

在 Openshift 中設定 Cluster Observability Operator 監控一個服務

0. 本篇文章 Openshift 環境資訊

1. Preface

2. 為 Cluster Observability Operator 部署範例服務

2.1. 先決條件

2.2. 開始部署與設定

3. 指定服務如何由 Cluster Observability Operator 監控

3.1. 先決條件

3.2. 開始部署與設定

至 RedHat 官網連結下載以下兩個項目的檔案 (一定要有帳號)

`install-config.yaml` 欄位介紹

設定 `agent-config.yaml`

4. 為 Cluster Observability Operator 建立一個 `MonitoringStack` 物件

2. 透過 `oc-mirror` 下載並打包 openshift 叢集所需之 container image，並上傳至本地 project quay

4.1. 設定 `install-config.yaml`

4.2. 設定 `agent-config.yaml`