設定 Alertmanager 觸發告警時去執行 Bash Script

1. 安裝 kube-prometheus-stack

:::info 安裝 kube‑prometheus stack 的核心元件：

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• Prometheus blackbox-exporter
• Prometheus Adapter for Kubernetes Metrics APIs
• kube-state-metrics
• Grafana :::

1.1. 先決條件

• K8s 已安裝完畢
• 已安裝好 StorageClass
• 已安裝好 IngressClass

1.2. 開始安裝

1. 安裝 helm

   curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

2. 建立 K8s namespace

   kubectl create ns monitoring-system

3. 將 Prometheus 社群維護的 Helm 圖表倉庫加入本地 Helm 用戶端，以便拉取 Prometheus 相關的 Helm Chart

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 

4. 更新本地緩存中所有已加入倉庫的圖表清單，確保可以取得最新版本的 Chart

   helm repo update

5. 建立並切換工作目錄

   mkdir ~/monitoring-system; cd ~/monitoring-system

6. 編輯 kube-prometheus-stack helm 安裝設定檔

   nano values.yaml

   檔案內容如下 ：

   kubelet:
     enabled: true
     serviceMonitor:
       cAdvisorMetricRelabelings: []
   prometheus:
     prometheusSpec:
       storageSpec:
         volumeClaimTemplate:
           spec:
             accessModes:
             - ReadWriteOnce
             resources:
               requests:
                 storage: 50Gi
             storageClassName: nfs-client
       additionalScrapeConfigsSecret:
         enabled: true
         name: additional-scrape-configs
         key: additional.yaml
     service:
       type: NodePort
     ingress:
       enabled: true
       ingressClassName: "cilium"
   alertmanager:
     alertmanagerSpec:
       storage:
         volumeClaimTemplate:
           spec:
             storageClassName: nfs-client
             accessModes:
             - ReadWriteOnce
             resources:
               requests:
                 storage: 50Gi
       alertmanagerConfigSelector:
         matchLabels:
           resource: prometheus
     service:
       type: NodePort
     ingress:
       enabled: true
       ingressClassName: "cilium"
   grafana:
     persistence:
       enabled: true
       type: sts
       storageClassName: "nfs-client"
       accessModes:
         - ReadWriteOnce
       size: 20Gi
       finalizers:
         - kubernetes.io/pvc-protection
     service:
       type: NodePort
     ingress:
       enabled: true
       ingressClassName: "cilium"

   主要會設定