Skip to content

antony@notes:~/openshift$ cat "在-Openshift-上透過-Kustomize-安裝-Elasticsearch.md"

在 Openshift 上透過 Kustomize 安裝 Elasticsearch

2025-09-17· openshift

在 Openshift 上透過 Kustomize 安裝 Elasticsearch

0. 安裝環境

  • 模擬客戶環境，不透過 StorageClass，而是手動管理 PV/PVC
  • 透過 LVM + NFS 作為底層的 Storage
  • 全離線環境 (Openshift 叢集節點都無法到 public 網路)

1. 環境確認

SSH 連線到 NFS Server 執行以下命令

1.1. 確認 NFS Server fsid=0 對應的掛載點 

cat /etc/exports | grep fsid=0

執行結果：

/nfs		10.21.107.0/24(rw,no_root_squash,fsid=0,insecure,sync)

1.2. 確認要用哪個 LVM VG 

vgs

執行結果：

VG     #PV #LV #SN Attr   VSize    VFree
nfsvg    1  32   0 wz--n-   <2.50t <698.00g
rootvg   2   7   0 wz--n- <197.80g  <30.80g

2. NFS 掛載環境準備

2.1. Create Folders for Elasticsearch

ocp log 資料要在 /nfs/ocpnfs_data/ 目錄底下 (因環境問題故放在 /nfs/apnfs_data)

mkdir -p /nfs/apnfs_data/rh-elastic-es-master-nodes-{0,1,2}
mkdir -p /nfs/apnfs_data/rh-elastic-es-data-nodes-{0,1,2}

2.2. Create LVs 

lvcreate -L 10G -n rh-elastic-es-master-nodes-0 /dev/nfsvg
lvcreate -L 10G -n rh-elastic-es-master-nodes-1 /dev/nfsvg
lvcreate -L 10G -n rh-elastic-es-master-nodes-2 /dev/nfsvg
lvcreate -L 100G -n rh-elastic-es-data-nodes-0 /dev/nfsvg
lvcreate -L 100G -n rh-elastic-es-data-nodes-1 /dev/nfsvg
lvcreate -L 100G -n rh-elastic-es-data-nodes-2 /dev/nfsvg

2.3. Format xfs filesystem 

mkfs.xfs /dev/nfsvg/rh-elastic-es-master-nodes-0
mkfs.xfs /dev/nfsvg/rh-elastic-es-master-nodes-1
mkfs.xfs /dev/nfsvg/rh-elastic-es-master-nodes-2
mkfs.xfs /dev/nfsvg/rh-elastic-es-data-nodes-0
mkfs.xfs /dev/nfsvg/rh-elastic-es-data-nodes-1
mkfs.xfs /dev/nfsvg/rh-elastic-es-data-nodes-2

2.4. Edit fstab 

nano /etc/fstab

檔案內容如下：

/dev/nfsvg/rh-elastic-es-data-nodes-0 /nfs/apnfs_data/rh-elastic-es-data-nodes-0 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-data-nodes-1 /nfs/apnfs_data/rh-elastic-es-data-nodes-1 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-data-nodes-2 /nfs/apnfs_data/rh-elastic-es-data-nodes-2 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-master-nodes-0 /nfs/apnfs_data/rh-elastic-es-master-nodes-0 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-master-nodes-1 /nfs/apnfs_data/rh-elastic-es-master-nodes-1 xfs defaults 0 0
/dev/nfsvg/rh-elastic-es-master-nodes-2 /nfs/apnfs_data/rh-elastic-es-master-nodes-2 xfs defaults 0 0

2.5. mount 

mount -a && systemctl daemon-reload

2.6. Check 

lsblk /dev/sdc | tail -n 6

執行結果：

NAME                                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sdc                                                 8:32   0  2.5T  0 disk
└─sdc1                                              8:33   0  2.5T  0 part
...
  ├─nfsvg-rh--elastic--es--master--nodes--0                        253:33   0   10G  0 lvm  /nfs/ocpnfs_data/rh-elastic-es-master-nodes-0
  ├─nfsvg-rh--elastic--es--master--nodes--1                        253:34   0   10G  0 lvm  /nfs/ocpnfs_data/rh-elastic-es-master-nodes-1
  ├─nfsvg-rh--elastic--es--master--nodes--2                        253:35   0   10G  0 lvm  /nfs/ocpnfs_data/rh-elastic-es-master-nodes-2
  ├─nfsvg-rh--elastic--es--data--nodes--0                          253:36   0  100G  0 lvm  /nfs/ocpnfs_data/rh-elastic-es-data-nodes-0
  ├─nfsvg-rh--elastic--es--data--nodes--1                          253:37   0  100G  0 lvm  /nfs/ocpnfs_data/rh-elastic-es-data-nodes-1
  └─nfsvg-rh--elastic--es--data--nodes--2                          253:38   0  100G  0 lvm  /nfs/ocpnfs_data/rh-elastic-es-data-nodes-2

2.7. 編輯 /etc/exports 

nano /etc/exports

執行結果：

...以上省略
# ECK
/nfs/apnfs_data/rh-elastic-es-master-nodes-0 10.21.107.0/24(rw,no_root_squash,fsid=60,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-master-nodes-1 10.21.107.0/24(rw,no_root_squash,fsid=61,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-master-nodes-2 10.21.107.0/24(rw,no_root_squash,fsid=62,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-data-nodes-0 10.21.107.0/24(rw,no_root_squash,fsid=63,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-data-nodes-1 10.21.107.0/24(rw,no_root_squash,fsid=64,insecure,sync)
/nfs/apnfs_data/rh-elastic-es-data-nodes-2 10.21.107.0/24(rw,no_root_squash,fsid=65,insecure,sync)

2.8. 重載 NFS server 配置並立即生效 export 

systemctl reload nfs-server
exportfs -arv

執行結果：

exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-data-nodes-2
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-data-nodes-1
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-data-nodes-0
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-master-nodes-2
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-master-nodes-1
exporting 10.21.107.0/24:/nfs/ocpnfs_data/rh-elastic-es-master-nodes-0
...

3. 安裝 ELK

3.1. 將事前準備好的 ELK 全離線安裝包上傳到 bastion 機和 Harbor

3.2. 將 image 改 tag 並 push 到 harbor

3.3. 修改 ELK 設定檔 

cd elk/
oc create -f crds.yaml
oc new-project elastic-system
oc new-project elastic
cd env/dev

3.3.1. 確認要安裝的節點名稱 

oc get nodes -l "node-role.kubernetes.io/logging" -o name

執行結果：

node/docpl01.ocp.cbsd.scsb.com.tw
node/docpl02.ocp.cbsd.scsb.com.tw
node/docpl03.ocp.cbsd.scsb.com.tw

3.3.2. 確認 Taints 

oc get nodes -l "node-role.kubernetes.io/logging" -o yaml | grep -iA 6 taints

執行結果：

    taints:
    - effect: NoSchedule
      key: logging
      value: reserved
    - effect: NoExecute
      key: logging
      value: reserved
--
    taints:
    - effect: NoSchedule
      key: logging
      value: reserved
    - effect: NoExecute
      key: logging
      value: reserved
--
    taints:
    - effect: NoSchedule
      key: logging
      value: reserved
    - effect: NoExecute
      key: logging
      value: reserved

3.3.3. 編輯 kustomize 設定檔 

nano kustomization.yaml

需修改以下項目

  1. elk images name, tag
  2. PV nodeSelector
  3. PV NFS Path
  4. PV NFS Server IP
  5. tolearation for elk and kibana

3.4. dry run 確認設定檔是否有誤 

oc apply -k . --dry-run=server

3.5. 將 elk 部署進 ocp 

oc apply -k .

3.6. 確認 pod 用哪個 user 在跑 

oc get pods -o yaml | grep -i runasuser | sort -u
        runAsUser: 1000830000

3.7. 進到 NFS Server 修改對應的權限 

chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-data-nodes-0
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-data-nodes-1
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-data-nodes-2
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-master-nodes-0
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-master-nodes-1
chown -R 1000830000 /nfs/apnfs_data/rh-elastic-es-master-nodes-2

3.8. 確認 pod 狀態 

oc get pods

執行結果：

NAME                              READY   STATUS    RESTARTS        AGE
elasticsearch-es-data-nodes-0     1/1     Running   2 (6m53s ago)   7m21s
elasticsearch-es-data-nodes-1     1/1     Running   2 (6m52s ago)   7m21s
elasticsearch-es-data-nodes-2     1/1     Running   2 (6m50s ago)   7m21s
elasticsearch-es-master-nodes-0   1/1     Running   2 (6m52s ago)   7m21s
elasticsearch-es-master-nodes-1   1/1     Running   2 (6m51s ago)   7m21s
elasticsearch-es-master-nodes-2   1/1     Running   2 (6m50s ago)   7m21s
kibana-kb-6967c5f584-hxqg6        1/1     Running   0               5m46s

3.9. 獲取 URL 

oc get route

執行結果：

NAME                   HOST/PORT                                                PATH   SERVICES                PORT    TERMINATION            WILDCARD
elasticsearch-sample   elasticsearch-sample-elastic.apps.ocp.cbsd.scsb.com.tw          elasticsearch-es-http   <all>   passthrough/Redirect   None
kibana-sample          kibana-sample-elastic.apps.ocp.cbsd.scsb.com.tw                 kibana-kb-http          <all>   passthrough/Redirect   None

3.10. login

account: elastic passed: 0Hv2Xln8Fb2V571Hw3ik0ed6

密碼獲取命令

oc -n elastic get secrets elasticsearch-es-elastic-user \
  -o go-template='{{.data.elastic | base64decode}}{{"\n"}}'
在 Openshift 上建立應用並驗證掛載 NFS StorageStep by Step install OpenShift 4.19 with the Agent-based Installer in Disconnected environment