Skip to content

antony@notes:~/suse$ cat "Basic-HA-Troubleshooting.md"

Basic HA Troubleshooting

2024-01-03· suse ·SLE HAE

Basic HA Troubleshooting

Preface

本篇文章會主要會介紹在 SLE HAE v15 上如何進行基本的故障修復

可以透過點擊以下目錄，選擇想看的內容，跳轉至特定章節

:::warning

:::spoiler {state=“open”} 目錄

[TOC]

:::

General Troubleshooting

  • Cluster nodes are SLE 15 servers
    • Standard SLE troubleshooting techniques and tools can be used
  • Check the basics:
    • 時間有沒有校時
    • 名稱解析
  • Monitor bonded network interfaces
    • The system may be using its last working interface
  • A cluster is inherently less stable than a single server:
    • It is running SBD/Watchdog and could self fence
    • Other nodes are monitoring it with the intention of shooting it

Maintanance Tasks

  • The following tasks will enable you to potentially catch a potential problem early
  • Perform regular health checks
  • Follow documented procedures to clean up after a failure
  • Using Maintanance mode
  • Manage logs
  • Patching nodes

Command Line Troubleshooting Tools

Clean up after Resouce Failures

當 resource 失敗達到叢集屬性 migration-threshold 指定的次數時，它將被遷移到不同的節點

  • 將為失敗的節點建立一個 -INFINITY 的分數
  • To display the scores:
    • crm resource scores
    • 請記住，分數也會被 constraints 設定
    • To view constraints:
      • crm resources constraints <resouce-name>
  • Before cleaning up resources:
    • 調查問題原因
    • 尋找解決方案
    • 解決問題

如果要通知叢集可以再次在該節點上執行資源：

  • crm resource cleanup <resource-name> [node]
  • 可以提供一個可選的節點參數，以便僅在該節點上執行 cleanup 程序。
  • 如果未指定可選節點，則會清除所有節點上資源的 fail counts ( 失敗計數 ) 和 negative scores ( 負分數 )

Remove Relocation Constraints created by a move or ban

Resouce Agent Tracing

Maintenance Mode

Maintenance Mode Targets

LevelResult
Resource- 這個資源將被設定為 unmanaged ( 不管理 )，並且監控的動作會被暫停，在底層運作的項目 (service, IP address/filesystem) 可以被手動啟動或停止
Node- 節點上的所有資源都設定為 unmanaged ( 不管理 )
- 不會為該節點分配資源
Cluster- 在叢集中的所有資源全部設定為 unmanaged ( 不管理 )
- 叢集不會進行資源分配
- Poison Pills/STONITH 將被暫停

Managing Maintenance Mode

Cluster Health Check

Cluster Report

hb_report 目前快被廢棄了，但是考試的時候，如果看到這兩個選項放在一起，要選 hb_report

Cluster Report Content

Cluster Report 檔案包含有關叢集和單一節點的信息

  1. 收集報告
$ sudo crm report -f "2024/1/3 12:00" report_2024-1-3

螢幕輸出 :

sle-ha-2: INFO: The report is saved in ./report_2024-1-3.tar.bz2
sle-ha-2: INFO: Report timespan: 01/03/24 12:00:00 - 01/03/24 15:43:32
sle-ha-2: INFO: Thank you for taking time to create this report.
  1. 建立存放報告的目錄區
$ mkdir log
  1. 把報告解壓縮至特定目錄區
$ sudo tar -xvjf report_2024-1-3.tar.bz2 -C log/

螢幕輸出 :

report_2024-1-3/
report_2024-1-3/journal.log
...以下省略
  1. 檢視報告的目錄區內容
$ ls -l log/report_2024-1-3/

螢幕輸出 :

total 604
-rw-r--r-- 1 root root  17339 Jan  3 15:43 analysis.txt
-rw-r--r-- 1 root root    937 Nov 17 11:08 corosync.conf
-rw-r--r-- 1 root root    953 Jan  3 15:43 crm_mon.txt
-rw-r--r-- 1 root root   5144 Jan  3 15:43 description.txt
-rw-r--r-- 1 root root    802 Jan  3 15:43 events.txt
-rw-r--r-- 1 root root 329585 Jan  3 15:43 ha-log.txt
-rw-r--r-- 1 root root     25 Jan  3 15:43 ha-log.txt.info
-rw-r--r-- 1 root root 231691 Jan  3 15:43 journal.log
-rw-r--r-- 1 root root     18 Jan  3 15:43 members.txt
drwxr-xr-x 1 root root    656 Jan  3 15:43 sle-ha-1
drwxr-xr-x 1 root root    644 Jan  3 15:43 sle-ha-2
-rw-r--r-- 1 root root   2116 Jan  3 15:43 sysinfo.txt

sle-ha-1 和 sle-ha-2 為包含節點特定詳細資料的目錄

Cluster Report Node Specific Details 

$ ls -l log/report_2024-1-3/sle-ha-1

螢幕輸出 :

total 2032
-rw-r--r-- 1 root root  22743 Jan  3 15:43 cib.xml
lrwxrwxrwx 1 root root     16 Jan  3 15:43 corosync.conf -> ../corosync.conf
-rw-r--r-- 1 root root   3301 May 18  2023 crm.conf
-rw-r--r-- 1 root root     22 Jan  3 15:43 crm.conf.info
lrwxrwxrwx 1 root root     14 Jan  3 15:43 crm_mon.txt -> ../crm_mon.txt
-rw-r--r-- 1 root root  63899 Jan  3 15:43 crmsh.log
-rw-r--r-- 1 root root     30 Jan  3 15:43 crmsh.log.info
-rw-r--r-- 1 root root      0 Jan  3 15:43 crm_verify.txt
-rw-r--r-- 1 root root     71 Jan  3 15:43 dlm_dump.txt
-rw-r--r-- 1 root root    117 Jan  2 17:32 drbd.conf
drwxr-xr-x 1 root root    104 Jan  2 17:34 drbd.d
-rw-r--r-- 1 root root      0 Jan  3 15:43 events.txt
-rw-r--r-- 2 root root 485436 Jan  3 15:43 ha-log.txt
-rw-r--r-- 1 root root     24 Jan  3 15:43 ha-log.txt.info
-rw-r--r-- 1 root root 184740 Jan  3 15:43 journal.log
lrwxrwxrwx 1 root root     14 Jan  3 15:43 members.txt -> ../members.txt
-rw-r--r-- 2 root root 485436 Jan  3 15:43 messages
-rw-r--r-- 1 root root     25 Jan  3 15:43 ocfs2.txt
-rw-r--r-- 1 root root 689875 Jan  3 15:43 pacemaker.log
-rw-r--r-- 1 root root     40 Jan  3 15:43 pacemaker.log.info
drwxr-xr-x 1 root root    960 Jan  3 15:43 pengine
-rw-r--r-- 1 root root    147 Jan  3 15:43 permissions.txt
-rw-r--r-- 1 root root    986 May 18  2023 profiles.yml
-rw-r--r-- 1 root root     25 Jan  3 15:43 profiles.yml.info
-rw-r--r-- 1 root root      0 Jan  3 15:43 RUNNING
-rw-r--r-- 1 root root   4274 Nov 17 11:08 sbd
-rw-r--r-- 1 root root   1164 Jan  3 15:43 sbd.txt
lrwxrwxrwx 1 root root     14 Jan  3 15:43 sysinfo.txt -> ../sysinfo.txt
-rw-r--r-- 1 root root  55689 Jan  3 15:43 sysstats.txt
-rw-r--r-- 1 root root     39 Jan  3 15:43 time.txt
drwxr-xr-x 1 root root     14 Jan  3 15:43 trace_ra

Cluster File Differences

Log Files

Cluster Logging

systemd Logs

journalctl

Syntax: journalctl OPTIONS

ModeDescription
-xAdd message explanations where available
-r反向輸出，以便先顯示最新的 Log 資訊。
-u corosyncCluster communication issues
-u pacemakerPacemaker related details
-u sbdStartup issues if SBD_STARTMODE=‘clean’
-fFollow the journal
–since=DATEShow entries not older than the specified date
–until=DATEShow entries not newer than the specified date

Managing the Epoch

The Cluster Epoch

Updating the Cluster Epoch 

## on sle-ha-2 node
$ sudo cibadmin --query --local | tr -s " " "\n" | grep -w epoch
epoch="49"

$ sudo cibadmin -B

$ sudo cibadmin --query --local | tr -s " " "\n" | grep -w epoch
epoch="50"

## on sle-ha-1 node
$ sudo cibadmin --query --local | tr -s " " "\n" | grep -w epoch
epoch="50"

SBD Startup Behavior

SBD Behaviour

Working with the CIB

Shadow CIBs

Configure and Create Cluster ResourcesRegistering Oracle Linux Clients