Skip to content

antony@notes:~/openshift$ cat "Install-Elasticsearch-Metrics-Exporter-on-Openshift.md"

Install Elasticsearch Metrics Exporter on Openshift

2025-11-27· openshift

Install Elasticsearch Metrics Exporter on Openshift

1. 環境資訊

  • openshift version 4.19.7 in disconnected network
  • 需要將 Elasticsearch Metrics Exporter 部署在 logging 節點
  • logging 節點資訊如下：
    oc get nodes -l "node-role.kubernetes.io/logging"
    執行結果：
    NAME                           STATUS   ROLES     AGE   VERSION
docpl01.ocp.cbsd.scsb.com.tw   Ready    logging   24d   v1.32.6
docpl02.ocp.cbsd.scsb.com.tw   Ready    logging   24d   v1.32.6
docpl03.ocp.cbsd.scsb.com.tw   Ready    logging   24d   v1.32.6
  • logging 節點的 taints 資訊 :
    oc get nodes -l "node-role.kubernetes.io/logging" -o yaml | grep -A 6 -i "taints"
    執行結果：
        taints:
    - effect: NoSchedule
      key: logging
      value: reserved
    - effect: NoExecute
      key: logging
      value: reserved
--
    taints:
    - effect: NoSchedule
      key: logging
      value: reserved
    - effect: NoExecute
      key: logging
      value: reserved
--
    taints:
    - effect: NoSchedule
      key: logging
      value: reserved
    - effect: NoExecute
      key: logging
      value: reserved

2. 前置作業

  1. 先決條件：須在可到 public 網路的 linux 主機作業，並有安裝 podman 套件
  2. 建立專案目錄並切換當前目錄
    mkdir -p ~/openshift/addon/monitoring/es-exporter; cd ~/openshift/addon/monitoring/es-exporter
  3. 下載 helm 命令執行檔
    wget https://get.helm.sh/helm-v4.0.1-linux-amd64.tar.gz
  4. 下載 prometheus-elasticsearch-exporter helm chart 檔案
    helm pull oci://ghcr.io/prometheus-community/charts/prometheus-elasticsearch-exporter
  5. 下載 prometheus-elasticsearch-exporter default values.yaml
    wget https://raw.githubusercontent.com/prometheus-community/helm-charts/refs/heads/main/charts/prometheus-elasticsearch-exporter/values.yaml
  6. 獲取 prometheus-elasticsearch-exporter container image 名稱
    cat default-values.yaml | grep repository
    執行結果：
    repository: quay.io/prometheuscommunity/elasticsearch-exporter

    tag 的版本預設會被指定在 Chart.yaml 中

  7. 獲取 prometheus-elasticsearch-exporter container image tag 版本
    tar -zxvf prometheus-elasticsearch-exporter-7.0.0.tgz
grep -i appversion prometheus-elasticsearch-exporter/Chart.yaml
    執行結果：
    appVersion: v1.9.0
  8. 下載並打包 prometheus-elasticsearch-exporter container image
    podman pull quay.io/prometheuscommunity/elasticsearch-exporter:v1.9.0
podman save quay.io/prometheuscommunity/elasticsearch-exporter:v1.9.0 > elasticsearch-exporter-v1.9.0.tar
  9. 準備啟用 openshift logging 收集 user workload 的 configmap 設定檔
    echo 'apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true' > cm.yaml
  10. 先撰寫客製化的 helm values.yaml 範本 (實際到客戶端會再根據環境修改成對應的設定值)
    nano custom-values.yaml
    執行結果：
    image:
  repository: quay.kubeantony.com:8443/library/prometheuscommunity/elasticsearch-exporter
  tag: "v1.9.0"
  pullPolicy: IfNotPresent

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000790001
  seccompProfile:
    type: "RuntimeDefault"

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 100m
    memory: 128Mi

nodeSelector:
  node-role.kubernetes.io/logging: ""

tolerations: []

es:
  uri: https://admin:admin123@elasticsearch-sample-elastic.apps.topgun.kubeantony.com
  sslSkipVerify: true
  11. 撰寫 prometheus 收集 elasticsearch metrics 需要的 service monitor yaml 設定檔
    nano es-service-monitor.yaml
    檔案內容如下：
    apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ealsticsearch-monitor
  namespace: elastic
spec:
  endpoints:
  - interval: 30s
    port: http
    scheme: http
  selector:
    matchLabels:
      app.kubernetes.io/instance: prometheus-elasticsearch-exporter
  12. 將整個專案目錄壓縮成一個檔案，之後可直接提供給客戶
    cd ..; tar -zcvf es-exporter.tar.gz es-exporter/

3. 開始設定與安裝 Elasticsearch Metrics Exporter

  1. 將 es-exporter.tar.gz 壓縮檔上傳至客戶 ocp 環境的 bastion 主機

    scp es-exporter.tar.gz <user>:<host_ip>:<folder>

    for example: scp es-exporter.tar.gz ansible@10.21.104.120:.

  2. 解壓縮 es-exporter.tar.gz 檔案

    tar -zxvf /PATH/TO/es-exporter.tar.gz -C <DST_Folder>

    for example: tar -zxvf /home/ansible/es-exporter.tar.gz -C elk/

  3. 切換至工作目錄

    cd ~/elk/es-exporter/

  4. 將 prometheus-elasticsearch-exporter container image 的打包檔還原成 image

    podman load < elasticsearch-exporter-v1.9.0.tar

  5. 修改 container image 名稱用以符合客戶環境

    podman tag quay.io/prometheuscommunity/elasticsearch-exporter:v1.9.0 docph01.cbsd.scsb.com.tw/ocp4/elk/elasticsearch-exporter:v1.9.0

  6. 確認是否登入 Image Registry

    podman login --get-login docph01.cbsd.scsb.com.tw

    執行結果：

    admin

  7. 將 container image 推送至客戶 Image Registry

    podman push docph01.cbsd.scsb.com.tw/ocp4/elk/elasticsearch-exporter:v1.9.0

  8. 安裝 helm 指令

    tar -zxvf helm-v4.0.1-linux-amd64.tar.gz; \
mv linux-amd64/helm /usr/local/bin/

  9. 修改 helm values.yaml 用以符合客戶 ocp 環境

    nano custom-values.yaml

    主要會需要設定以下項目：

    • image.repository: container image 名稱
    • podSecurityContext.runAsUser: pod 須以什麼 user 執行，可根據以下命令得知 namespace 底下可用的 uid 範圍
      oc get ns elastic -o yaml | grep uid-
      執行結果：
      openshift.io/sa.scc.uid-range: 1000780000/10000
    • resources: 設定 cpu 和 記憶體可用或上限的資源
    • nodeSelector: 設定 pod 要跑在帶有什麼 label 的節點上
    • tolerations: 如果該節點有 taints 需設定 tolerations 來容忍這些 taints，否則 pod 將無法被指派到該節點
    • es.url: elasticsearch url 的位置

    檔案內容：

    image:
  repository: docph01.cbsd.scsb.com.tw/ocp4/elk/elasticsearch-exporter
  tag: "v1.9.0"
  pullPolicy: IfNotPresent

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000780001
  seccompProfile:
    type: "RuntimeDefault"

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 100m
    memory: 128Mi

nodeSelector:
  node-role.kubernetes.io/logging: ""

tolerations:
  - effect: NoSchedule
    key: logging
    operator: Equal
    value: reserved
  - effect: NoExecute
    key: logging
    operator: Equal
    value: reserved

es:
  uri: https://elastic:1B91L1oPibVZ430p95wS3ljs@elasticsearch-elastic.apps.ocp.cbsd.scsb.com.tw
  sslSkipVerify: true

  10. 查找 kubeconfig 位置

    echo $KUBECONFIG

    執行結果：

    /source/OCP_4.14_CLI_and_YAMLs/OCP_YAMLs/ign-workdir/auth/kubeconfig

  11. 透過 helm 安裝 prometheus-elasticsearch-exporter

    helm --kubeconfig=/source/OCP_4.14_CLI_and_YAMLs/OCP_YAMLs/ign-workdir/auth/kubeconfig \
  upgrade
    --install prometheus-elasticsearch-exporter ./prometheus-elasticsearch-exporter-7.0.0.tgz \
    -f custom-values.yaml

  12. 檢視 prometheus-elasticsearch-exporter pod 運作狀態

    oc -n elastic get pods

    執行結果：

    NAME                                                 READY   STATUS    RESTARTS       AGE
elasticsearch-es-data-nodes-0                        1/1     Running   2 (5h7m ago)   5h8m
elasticsearch-es-data-nodes-1                        1/1     Running   2 (5h7m ago)   5h8m
elasticsearch-es-data-nodes-2                        1/1     Running   2 (5h7m ago)   5h8m
elasticsearch-es-master-nodes-0                      1/1     Running   2 (5h7m ago)   5h8m
elasticsearch-es-master-nodes-1                      1/1     Running   2 (5h7m ago)   5h8m
elasticsearch-es-master-nodes-2                      1/1     Running   2 (5h7m ago)   5h8m
kibana-kb-688b58f67f-8jq4n                           1/1     Running   0              5h8m
prometheus-elasticsearch-exporter-7b9d89dcd9-xx68g   1/1     Running   0              4h51m

  13. 設定 openshift monitoring 允許收集自定義的 metrics

    oc apply -f cm.yaml

  14. 設定 openshift monitoring 要去收集 elasticsearch metrics expose 出來的 metrics

    oc apply -f es-service-monitor.yaml

  15. 檢視 servicemonitors 是否正常部署

    oc -n elastic get servicemonitors

    執行結果：

    NAME                    AGE
ealsticsearch-monitor   4h54m

4. 驗證 Prometheus 是否可收集到 Elasticsearch 的 metrics

  1. 創建一個 serviceAccount 並賦予相應的權限來驗證 prometheus 有收集到 elasticsearch 的 metrics

    oc create sa monitor-reader
oc adm policy add-cluster-role-to-user cluster-monitoring-view -z monitor-reader

  2. 為 serviceAccount 建立 secret 來獲取永久的 token

    cat <<EOF | oc apply -f -
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: monitor-reader-token
  annotations:
    kubernetes.io/service-account.name: "monitor-reader"
EOF

  3. 獲取 monitor-reader serviceAccount 的 token

    echo "Bearer $(oc get secret monitor-reader-token -o jsonpath='{.data.token}' | base64 -d)"

    執行結果：

    Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ilc4Y0dnOC1JMTVWZ1Zwd3kzVFpxdjlFeUJybWlnREJFcUlkZGdOanZFdlEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtb25pdG9yLXJlYWRlci10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtb25pdG9yLXJlYWRlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFhNjllNzJhLTU3YzAtNDJiYi1iMTU0LTVjMWUwYzgyYjllZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpvcGVuc2hpZnQtbW9uaXRvcmluZzptb25pdG9yLXJlYWRlciJ9.ahjR-seVoZI6YiKNCKovP9TaIqAuy4rTRbYaZEb1xwHfy3FTowvrtZyi5zdYNHx4r5-Ry7M7edQUPFY4CCTdd8sbt3q-jD2rAankCIanKzCf02D_QVvPB2so1zdxdsVDBgx71tLTsx7k2wzzsefoL0QRnPcYA-nJJjMxeg2Rrzn2g8dNQm65sTl-ZAfunZq5UlqIs-UYhrM5DGZ0_EbQRtLgEdW91h_OGa0KFEtwsjXL_f8k4cPZydk63du4qwXeW0MLrHAQXRc6UAkG7Pl3GuPeqUdaSIafwywPOVv_BHBRKJl8FKtnBcYRuV7MFrqLsdjkfGO8XcWciIQzKn_2x33wb4kLGVeVKebU_72xVzqdgxCFSXNjO-tz3IF2DITByXxVGRoNv0y--sPxgldznDLSqUsoKBnpkjlnzA_rGUvTr7cy9r-Rj5uRDgO8pUFrptTldhlJxm5Dtfej_-jGuXysDkREBGs4SKond98WcKVYdIRWzjvY0Kgha7bdx6gmmysRdd2-5y_ffS6ruO-jikl82A-Bt2-7BjK2MdzNvDReD6SJ8zQXkG6_-oUPY_i0UR3PTidO71pORW4q0YxjVKD-si3wnot98A6HDCnCXkUAOgAA5SD8k0uzegeHL9Gbktqghuy9eL2PJyomSknxQoqQx9L4j2HsjOUkg70zIKA

  4. 獲得 thanos-querier 的 host 跟 path

    oc -n openshift-monitoring get route

    執行結果：

    NAME                      HOST/PORT                                                                PATH        SERVICES            PORT   TERMINATION          WILDCARD
alertmanager-main         alertmanager-main-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw         /api        alertmanager-main   web    reencrypt/Redirect   None
prometheus-k8s            prometheus-k8s-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw            /api        prometheus-k8s      web    reencrypt/Redirect   None
prometheus-k8s-federate   prometheus-k8s-federate-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw   /federate   prometheus-k8s      web    reencrypt/Redirect   None
thanos-querier            thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw            /api        thanos-querier      web    reencrypt/Redirect   None

  5. 查找 thanos-querier fqdn 對應的 ip

    dig thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw +short

    執行結果：

    10.21.107.50

  6. 因當前操作的實體主機網路無法直接存取透過 route 對外的 openshift 服務，故建立 ssh tunnel

    ssh -L 10000:10.21.107.50:443 ansible@10.21.104.120

    輸入密碼登入，並執行 top 避免被自動登出，造成 ssh tunnel 終止

  7. 在本機 /etc/hosts 添加一筆名稱解析

    sudo nano /etc/hosts

    添加以下檔案內容：

    127.0.0.1 thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw

  8. 在實體主機的 Google Chrome 瀏覽器安裝 ModHeader 插件

  9. 設定 ModHeader 插件

    • 點選 +Mod，並選擇 Request headers，然後依序填入以下設定：
      name: Authorization
value: 複製貼上第三步驟(獲取 monitor-reader `serviceAccount` 的 token)的執行結果
    • 點選 Filter，並選擇 Request URL filters，然後輸入以下內容：
      https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw:10000/.*

  10. 打開 Google Chrome 瀏覽器，連線測試是否可正常 query

    https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw:10000/api/v1/query?query=up

  11. 確認網路可正常連線到 ocp 叢集裡面的 thanos-querier 服務後，查詢 elasticsearch 的 metrics name 確認是否有成功收集

    https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw:10000/api/v1/query?query=elasticsearch_thread_pool_completed_count

5. 設定 Grafana 呈現 Prometheus 收集到的 Elasticsearch Metrics

  1. 打開 Google Chrome 瀏覽器，並連線至 Grafana

    https://10.21.104.120

  2. 登入 Grafana

  3. 新增 Data Sources，點選左側選單 -> Connections -> Data Sources -> 右上角 Add new data source

  4. 選擇 prometheus

  5. 輸入 prometheus 連線資訊：

    name: ocp-prometheus

Connection
Prometheus server URL: https://thanos-querier-openshift-monitoring.apps.ocp.cbsd.scsb.com.tw

TLS settings
勾選 Skip TLS certificate validation

展開 HTTP headers
Add header
Header: Authorization
Value: 複製貼上第三步驟(獲取 monitor-reader `serviceAccount` 的 token)的執行結果

    確認沒問題後，滑鼠滾到頁面最下面點選 Save & test 按鈕

  6. 確認有出現綠色區塊的連線成功資訊

  7. 匯入 DashBorad，點選左側選單 Dashboard -> Elastic Folder ->

修復 ocp 叢集憑證過期ocp to vmware loginsight