Skip to content

antony@notes:~/openshift$ cat "Step-by-Step-Install-OCP-4.12-Online.md"

Step by Step Install OCP 4.12 (Online)

2023-08-01· openshift ·OpenShift Container Platform

Preface

本篇文章會介紹，如何在多台 VM 上使用 user provisioned infrastructure 的方式安裝 1 台 Master Node 和 2 台 Worker Node 架構的 OpenShift Container Platform 4.12

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

:::warning

:::spoiler 文章目錄

[TOC]

:::

安裝流程

  1. 安裝和設定 bastion 虛擬主機
  2. 安裝和設定 bootstrap 虛擬主機
  3. 安裝和設定 master 虛擬主機
  4. 安裝和設定 worker1 和 worker2 虛擬主機
  5. 開始安裝 OpenShift Container Platform 4.12

各台主機扮演的角色說明

角色和服務說明
Cluster installer/bastion發動安裝 OCP 的跳板機, 同時提供 load balance, DNS Server, Web server 的服務
bootstrap安裝時重要的角色, 透過 installer 將 ocp cluster 角色先部署在此, 再透過 scale out 延伸到 master node
Master nodeocp 資源調節重要節點須為三個
Worker/compute node運行 Application 的節點
ignition file安裝時透過 installer 編寫的 yaml 檔, 後續透過執行此檔案安裝 ocp
DNS Server提供名稱解析和反解析(網址轉成 IP 或是將 IP 轉回網址)
HA Proxy提供 load balance 的服務
Web Server讓 bootstrap 、master 和 worker 主機可以透過這個網站伺服器下載 ignition file

Prerequisites

硬體資源需求

環境架構

  • 1 台 bastion (RHEL)
  • 1 台 bootstrap (RHCOS)
  • 1 台 master (RHCOS)
  • 2 台 worker (RHCOS)

主機名稱設定格式 

HOSTNAME.CLUSTER_NAME.DOMAIN_NAME

example:

bastion.ocp4.example.com
  • HOSTNAME 就是 bastion
  • CLUSTER_NAME 就是 ocp4
  • DOMAIN_NAME 就是 example.com

Setup bastion Machine

Step 0: 安裝與設定 Red Hat Enterprise Linux

Step 1: 下載 “安裝 OCP 程式” 和 “管理 OpenShift 的 CLI 工具”

至 RedHat 官網連結 下載以下兩個項目的檔案 (一定要有帳號)

  1. OpenShift v4.12.24 Linux Installer
  2. OpenShift v4.12 Linux Client
$ wget --show-progress -qO 'openshift-install-linux-4.12.24.tar.gz' "<installer 的下載網址>" && wget --show-progress -qO 'oc-4.12.24-linux.tar.gz' "<Clinet 的下載網址>"

螢幕輸出 :

openshift-install-linux-4.12. 100%[==============================================>] 329.92M  1.87MB/s    in 3m 25s
oc-4.12.24-linux.tar.gz       100%[==============================================>]  52.25M  11.1MB/s    in 5.7s

oc-4.12.24-linux.tar.gz，是 OpenShift CLI 的壓縮檔 openshift-install-linux-4.12.24.tar.gz，是安裝 OCP 會用到的程式

確認兩個檔案的 checksum 

$ [[ "dd032e41022453424377b1bbcc40a8f712059c1e937f04c0e336389b70173b81" == "$(sha256sum oc-4.12.24-linux.tar.gz | cut -d ' ' -f 1)" ]] && echo 'oc-4.12.24-linux.tar.gz ok' && [[ "9f4362464784100ea4e950513989df912fb93420465408c496361e009c73efdc" == "$(sha256sum  openshift-install-linux-4.12.24.tar.gz | cut -d ' ' -f 1)" ]] && echo "openshift-install-linux-4.12.24.tar.gz ok"

螢幕輸出 :

oc-4.12.24-linux.tar.gz ok
openshift-install-linux-4.12.24.tar.gz ok

解壓縮檔案，並將 oc、openshift-install 和 kubectl 加至 PATH 環境變數 

$ tar -xvf oc-4.12.24-linux.tar.gz && \
tar -xvf openshift-install-linux-4.12.24.tar.gz && \
sudo mv oc kubectl openshift-install /usr/local/bin

螢幕輸出 :

README.md
kubectl
oc
README.md
openshift-install

確認 oc Command 版本 

$ oc version

螢幕輸出 :

Client Version: 4.12.24
Kustomize Version: v4.5.7

確認 openshift-install Command 版本 

$ openshift-install version

螢幕輸出 :

openshift-install 4.12.24
built from commit 194f806390b2b20668f5ac32d7918f9773b29a5e
release image quay.io/openshift-release-dev/ocp-release@sha256:b0b11eedf91175459b5d7aefcf3936d0cabf00f01ced756677483f5f26227328
release architecture amd64

確認 kubectl Command 版本 

$ kubectl version --output=yaml

螢幕輸出 :

clientVersion:
  buildDate: "2023-06-26T20:27:30Z"
  compiler: gc
  gitCommit: ea7c11aac7fa39d9eb733b62251ee1bc4a242c99
  gitTreeState: clean
  gitVersion: v1.25.2
  goVersion: go1.19.10 X:strictfipsruntime
  major: "1"
  minor: "25"
  platform: linux/amd64
kustomizeVersion: v4.5.7

Step 2: 安裝 DNS Server 

$ sudo yum -y install bind

螢幕輸出 :

...以上省略
Complete!

編輯 DNS Server 設定檔 named.conf 

$ sudo nano /etc/named.conf

要改的地方有兩個部分

  1. 將 listen-on port 53 和 allow-query 的值, 改成 any

檔案內容 :

options {
        listen-on port 53 { any; };
        listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        secroots-file   "/var/named/data/named.secroots";
        recursing-file  "/var/named/data/named.recursing";
        allow-query     { any; };
  1. 假設我 bastion 這台機器的 Hostname 是 bastion.ocp4.example.com IP 是 192.168.11.80，在 /etc/named.conf 新增以下內容至檔案的最後面 :
zone "example.com" {
        type master;
        file "/etc/named/zones/db.example.com";
};

zone "11.168.192.in-addr.arpa" {
        type master;
        file "/etc/named/zones/db.reverse";
};

:::info

在本次範例中

設定 DNS Server 名稱解析 

$ sudo mkdir /etc/named/zones && sudo nano /etc/named/zones/db.example.com

檔案內容如下 :

$TTL 1W
@       IN      SOA     bastion.ocp4.example.com.       root (
                        2019070700      ; serial
                        3H              ; refresh (3 hours)
                        30M             ; retry (30 minutes)
                        2W              ; expiry (2 weeks)
                        1W )            ; minimum (1 week)
        IN      NS      bastion.ocp4.example.com.
        IN      MX 10   smtp.example.com.
;
;
bastion.ocp4.example.com.               IN      A       192.168.11.80
smtp.example.com.                       IN      A       192.168.11.80
;
api.ocp4.example.com.                   IN      A       192.168.11.80
api-int.ocp4.example.com.               IN      A       192.168.11.80
;
*.apps.ocp4.example.com.                IN      A       192.168.11.80
;
bootstrap.ocp4.example.com.             IN      A       192.168.11.81
;
master.ocp4.example.com.                IN      A       192.168.11.82
;
worker1.ocp4.example.com.               IN      A       192.168.11.83
worker2.ocp4.example.com.               IN      A       192.168.11.84
;
;EOF

:::danger

Note: 紅字部分為須依照環境的規劃來設定

:::

設定 DNS Server 反解析 

$ sudo nano /etc/named/zones/db.reverse

檔案內容如下 :

$TTL 1W
@       IN      SOA     bastion.ocp4.example.com.       root (
                        2019070700      ; serial
                        3H              ; refresh (3 hours)
                        30M             ; retry (30 minutes)
                        2W              ; expiry (2 weeks)
                        1W )            ; minimum (1 week)
        IN      NS      bastion.ocp4.example.com.
;
80.11.168.192.in-addr.arpa.     IN      PTR     api.ocp4.example.com.
80.11.168.192.in-addr.arpa.     IN      PTR     api-int.ocp4.example.com.
;
81.11.168.192.in-addr.arpa.     IN      PTR     bootstrap.ocp4.example.com.
;
82.11.168.192.in-addr.arpa.      IN      PTR     master.ocp4.example.com.
;
83.11.168.192.in-addr.arpa.     IN      PTR     worker1.ocp4.example.com.
84.11.168.192.in-addr.arpa.     IN      PTR     worker2.ocp4.example.com.
;
;EOF

:::danger

Note: 紅字部分為須依照環境的規劃來設定

:::

啟動 DNS Server 

$ sudo systemctl enable named --now

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/named.service → /usr/lib/systemd/system/named.service.

檢查

$ sudo systemctl status named

螢幕輸出 :

● named.service - Berkeley Internet Name Domain (DNS)
   Loaded: loaded (/usr/lib/systemd/system/named.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-17 16:50:31 CST; 1s ago

...以下省略

將 bastion 機器的 IP 新增為 DNS Server 

$ sudo nmcli connection modify 'ens18' ipv4.dns '192.168.11.80' +ipv4.dns '8.8.8.8'

$ sudo systemctl restart NetworkManager

檢查

$ sudo cat /etc/resolv.conf

螢幕輸出

# Generated by NetworkManager
search ocp4.example.com
nameserver 192.168.11.80
nameserver 8.8.8.8

關閉防火牆 

$ sudo systemctl disable firewalld.service --now

檢查是否關閉

$ sudo systemctl status firewalld.service --no-pager

螢幕輸出 :

● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

...以下省略

驗證 DNS Server 正解 

host bastion.ocp4.example.com && \
host api.ocp4.example.com && \
host api-int.ocp4.example.com && \
host 111.apps.ocp4.example.com && \
host bootstrap.ocp4.example.com && \
host master.ocp4.example.com && \
host worker1.ocp4.example.com && \
host worker2.ocp4.example.com

螢幕輸出 :

bastion.ocp4.example.com has address 192.168.11.80
api.ocp4.example.com has address 192.168.11.80
api-int.ocp4.example.com has address 192.168.11.80
111.apps.ocp4.example.com has address 192.168.11.80
bootstrap.ocp4.example.com has address 192.168.11.81
master.ocp4.example.com has address 192.168.11.82
worker1.ocp4.example.com has address 192.168.11.83
worker2.ocp4.example.com has address 192.168.11.84

驗證 DNS Server 反解 

host 192.168.11.80 && \
host 192.168.11.81 && \
host 192.168.11.82 && \
host 192.168.11.83 && \
host 192.168.11.84

螢幕輸出 :

80.11.168.192.in-addr.arpa domain name pointer api-int.ocp4.example.com.
80.11.168.192.in-addr.arpa domain name pointer api.ocp4.example.com.
81.11.168.192.in-addr.arpa domain name pointer bootstrap.ocp4.example.com.
82.11.168.192.in-addr.arpa domain name pointer master.ocp4.example.com.
83.11.168.192.in-addr.arpa domain name pointer worker1.ocp4.example.com.
84.11.168.192.in-addr.arpa domain name pointer worker2.ocp4.example.com.

Step 3: 安裝 HAProxy 服務 

$ sudo yum -y install haproxy

螢幕輸出 :

...以上省略
Complete!

設定 HAProxy

直接清掉預設的設定檔

$ cat /dev/null | sudo tee /etc/haproxy/haproxy.cfg

再編輯設定檔

$ sudo nano /etc/haproxy/haproxy.cfg

將以下內容複製到檔案中 :

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon
defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000
listen api-server-6443
  bind *:6443
  mode tcp
  server bootstrap bootstrap.ocp4.example.com:6443 check inter 1s backup
  server master master.ocp4.example.com:6443 check inter 1s
listen machine-config-server-22623
  bind *:22623
  mode tcp
  server bootstrap bootstrap.ocp4.example.com:22623 check inter 1s backup
  server master master.ocp4.example.com:22623 check inter 1s
listen ingress-router-443
  bind *:443
  mode tcp
  balance source
  server worker1 worker1.ocp4.example.com:443 check inter 1s
  server worker2 worker2.ocp4.example.com:443 check inter 1s
listen ingress-router-80
  bind *:80
  mode tcp
  balance source
  server worker1 worker1.ocp4.example.com:80 check inter 1s
  server worker2 worker2.ocp4.example.com:80 check inter 1s

:::danger

Note: 紅字部分為須依照環境的規劃來設定

:::

設定允許 HAProxy 可以使用 TCP Port 

$ sudo setsebool -P haproxy_connect_any=1

If you are using HAProxy as a load balancer and SELinux is set to enforcing, you must ensure that the HAProxy service can bind to the configured TCP port by running setsebool -P haproxy_connect_any=1.

啟動 HAProxy 服務，並設為開機自動啟動 

$ sudo systemctl enable --now haproxy.service

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.

檢查

$ sudo systemctl status haproxy.service

螢幕輸出 :

● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-17 17:26:21 CST; 11min ago
...以下省略

Step 4: 安裝 Web Server 

$ sudo yum install -y httpd

將網站的 TCP Port 改成 8080 

$ sudo nano /etc/httpd/conf/httpd.conf

修改以下檔案內容，將 80 Port 改為 8080 Port

Listen 8080

或是用 sed 直接修改 : sudo sed -i 's|Listen 80|Listen 8080|g' /etc/httpd/conf/httpd.conf

修改 ssl.conf 檔名 

$ sudo mv /etc/httpd/conf.d/ssl.conf /etc/httpd/conf.d/ssl.conf.bak

沒修改此檔名，可能會再啟動 httpd 服務時，遇到以下錯誤訊息 : Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443

“could not bind to address” error - 解決辦法連結

啟動 httpd 服務，並設為開機自動啟動 

$ sudo systemctl enable --now httpd

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.

檢查

$ sudo systemctl status httpd

螢幕輸出 :

● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2023-07-17 20:34:55 CST; 6s ago
...以下省略

Step 5: 產生 SSH Key 

$ ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519

螢幕輸出 :

Generating public/private ed25519 key pair.
Created directory '/home/rancher/.ssh'.
Your identification has been saved in /home/rancher/.ssh/id_ed25519.
Your public key has been saved in /home/rancher/.ssh/id_ed25519.pub.
The key fingerprint is:
SHA256:fwnYQ9U9iwOe+KzbaLTz3FbX3gZnGMg5MQTFemFGjPk rancher@bastion.ocp4.example.com
The key's randomart image is:
+--[ED25519 256]--+
|          .X=. . |
|          ooO ...|
|          +*o*. o|
|         =.oEo.. |
|        S *. ..o.|
|         o = .o.=|
|        . + o .=o|
|         =+...  +|
|        .o++.. . |
+----[SHA256]-----+

啟動 SSH 代理 

$ eval "$(ssh-agent -s)"

將 SSH private key 加入 ssh-agent 

$ ssh-add ~/.ssh/id_ed25519

Step 6: 下載 pull-secret

至以下連結下載，注意 : 需先登入帳號

可點選 “Download pull secret"，或是點 “Copy pull secret” 直接將 pull-secret 複製到剪貼簿

Step 7: 撰寫 install-config.yaml 

$ mkdir ocpinstall && nano ocpinstall/install-config.yaml

:::info

Note: install-config.yaml 這個檔案名稱不可亂改，只能取這個名稱

:::

將以下檔案內容，修改符合當前環境之設定值

apiVersion: v1
baseDomain: example.com 
compute: 
- hyperthreading: Enabled 
  name: worker
  replicas: 0 
controlPlane: 
  hyperthreading: Enabled 
  name: master
  replicas: 1 
metadata:
  name: ocp4 
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14 
    hostPrefix: 23 
  networkType: OVNKubernetes 
  serviceNetwork: 
  - 172.30.0.0/16
platform:
  none: {} 
fips: false 
pullSecret: '{"auths": ...}' 
sshKey: 'ssh-ed25519 AAAA...'

:::danger

Note: 紅字部分為須依照環境的規劃來設定

:::

  1. baseDomain ，The base domain of the cluster.
  2. 使用 user-provisioned 的方式建立 OCP 時，不要改 compute.replicas = 0 這個數字

:::info

:::

  1. controlPlane.replicas = 1 ，設定 ControlPlane Node 的數量為 1 台
  2. metadata.name，設定 OCP 的 Cluster Name
  3. pullSecret ，將上一個步驟複製的 pullSecret 複製到這
  4. sshKey，bastion 主機的 SSH 公鑰，The SSH public key for the core user in Red Hat Enterprise Linux CoreOS (RHCOS).(可在 bastion 主機，透過 SSH 免密碼的方式登入 RHCOS 的機器)

:::info

:::

備份 install-config.yaml 

$ cp ~/ocpinstall/install-config.yaml ~/ocpinstall/install-config.yaml.bak

install-config.yaml 之後會不見, 會透過 openshift-install 指令變成 XXX.ign 檔案, 以防萬一有地方寫錯之後要修改, 這邊先備份

Step 8: Creating the Kubernetes manifest and Ignition config files 

$ openshift-install create manifests --dir ~/ocpinstall

--dir，specify the installation directory that contains the install-config.yaml file you created.

螢幕輸出 :

INFO Consuming Install Config from target directory
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
INFO Manifests created in: /home/rancher/ocpinstall/manifests and /home/rancher/ocpinstall/openshift

Because you must modify some cluster definition files and manually start the cluster machines, you must generate the Kubernetes manifest and Ignition config files that the cluster needs to configure the machines.

The installation configuration file transforms into the Kubernetes manifests. The manifests wrap into the Ignition configuration files, which are later used to configure the cluster machines.

設定 master Node 不要 Run Pod 

$ sed -i 's|mastersSchedulable: true|mastersSchedulable: false|g' ocpinstall/manifests/cluster-scheduler-02-config.yml

Create the Ignition configuration files 

$ openshift-install create ignition-configs --dir ocpinstall

螢幕輸出 :

INFO Consuming Openshift Manifests from target directory
INFO Consuming Master Machines from target directory
INFO Consuming Worker Machines from target directory
INFO Consuming OpenShift Install (Manifests) from target directory
INFO Consuming Common Manifests from target directory
INFO Ignition-Configs created in: /home/rancher/ocpinstall and /home/rancher/ocpinstall/auth

將 Ignition configuration files 放至網站 

$ sudo mkdir /var/www/html/ocp4 && \
sudo cp -R ~/ocpinstall/* /var/www/html/ocp4 && \
sudo chown -R apache: /var/www/html/ocp4 && \
sudo chmod 755 /var/www/html/ocp4

檢查

打開瀏覽器，連線至

http://192.168.11.80:8080/ocp4/

Setup bootstrap machine

Installing RHCOS by using an ISO image

1. 取得 bootstrap.ign 啟動設定檔案的 SHA512 雜湊值

在 bastion 主機執行以下命令

$ sha512sum ~/ocpinstall/bootstrap.ign

螢幕輸出 :

e11591138d028175664c5779ca696c568aaf6dc3849a0facb7e76f160184284e451131968786d2c81648d0425668a922e159541c98287be07438a596d6d84d2b  ocpinstall/bootstrap.ign

在稍後的步驟中，這個 SHA512 雜湊值會被提供給「coreos-installer」，以驗證叢集節點上 Ignition 設定檔的真實性。

2. 透過 openshift-install 命令獲取正確版本的 RHCOS ISO 檔

在 bastion 機器執行以下命令

$ openshift-install coreos print-stream-json | grep '\.iso[^.]'

螢幕輸出 :

... 以上省略
"location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202306132230-0/x86_64/rhcos-412.86.202306132230-0-live.x86_64.iso",

下載 x86_64 版本的 RHCOS ISO 檔，並透過這個 ISO 檔，將 RHCOS 裝起來

下載命令 :

$ wget https://rhcos.mirror.openshift.com/art/storage/prod/streams/4.12/builds/412.86.202306132230-0/x86_64/rhcos-412.86.202306132230-0-live.x86_64.iso

3. 使用上個步驟下載好的 ISO 檔安裝 RHCOS

開始安裝後，不用指定任何選項或中斷實時啟動順序。等待安裝程式進入 RHCOS 實機環境的命令提示字元。

4. 允許 SSH 使用密碼登入 和 設定網路、主機名稱 

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service
$ echo -e "rancher\nrancher" | sudo passwd core
$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.81/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled

$ sudo nmcli connection up 'Wired connection 1'
$ sudo hostnamectl set-hostname bootstrap-ocp4-example.com

$ sudo systemctl restart NetworkManager

Setup master machine

Installing RHCOS by using an ISO image

1. 取得 master.ign 啟動設定檔案的 SHA512 雜湊值

在 bastion 主機執行以下命令

$ sha512sum ocpinstall/master.ign

螢幕輸出 :

46be965b1657dc170203ce83b9ff2ffddf34004da83cecf2440e0c50dec06ea32296c0b2c21a8324fb321ba6592ee5d12108c362d5b98ac3e5a0edf3d8fce7a8  ocpinstall/master.ign

在稍後的步驟中，這個 SHA512 雜湊值會被提供給「coreos-installer」，以驗證叢集節點上 Ignition 設定檔的真實性。

2. 使用與 bootstrap 機器相同的 ISO 檔安裝 RHCOS

3. 允許 SSH 使用密碼登入 和 設定網路、主機名稱 

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service
$ echo -e "rancher\nrancher" | sudo passwd core
$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.82/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled

$ sudo nmcli connection up 'Wired connection 1'
$ sudo hostnamectl set-hostname master-ocp4-example.com

$ sudo systemctl restart NetworkManager

Setup worker1 machine

Installing RHCOS by using an ISO image

1. 取得 worker.ign 啟動設定檔案的 SHA512 雜湊值

在 bastion 主機執行以下命令

$ sha512sum ocpinstall/worker.ign

螢幕輸出 :

c61b89807d86ef9974aaa0451015205bf9e7534b9506b52fb8883c3411c1ba1c5ec0e3080c7d35bb782cd847aa64533552b3f2132ce9b0c8fc6cf4e62933f8c6  ocpinstall/worker.ign

在稍後的步驟中，這個 SHA512 雜湊值會被提供給「coreos-installer」，以驗證叢集節點上 Ignition 設定檔的真實性。

2. 使用與 bootstrap 機器相同的 ISO 檔安裝 RHCOS

3. 允許 SSH 使用密碼登入 和 設定網路、主機名稱 

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service
$ echo -e "rancher\nrancher" | sudo passwd core
$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.83/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled

$ sudo nmcli connection up 'Wired connection 1'
$ sudo hostnamectl set-hostname worker1-ocp4-example.com

$ sudo systemctl restart NetworkManager

Setup worker2 machine

Installing RHCOS by using an ISO image

1. 使用與 bootstrap 機器相同的 ISO 檔安裝 RHCOS

2. 允許 SSH 使用密碼登入 和 設定網路、主機名稱 

$ sudo nano /etc/ssh/sshd_config

將第 68 行的 PasswordAuthentication no 改成 PasswordAuthentication yes

$ sudo systemctl daemon-reload && sudo systemctl restart sshd.service
$ echo -e "rancher\nrancher" | sudo passwd core
$ sudo nmcli connection modify 'Wired connection 1' ipv4.method manual ipv4.addresses 192.168.11.84/24 gw4 192.168.11.254 ipv4.dns 192.168.11.80 +ipv4.dns 8.8.8.8 ipv6.method disabled

$ sudo nmcli connection up 'Wired connection 1'
$ sudo hostnamectl set-hostname worker2-ocp4-example.com

$ sudo systemctl restart NetworkManager

Starting the OpenShift Container Platform bootstrap process

1. 在 bootstrap 機器執行以下安裝命令 

$ sudo coreos-installer install \
--firstboot-args 'rd.neednet=1' \
--copy-network \
--ignition-url=http://192.168.11.80:8080/ocp4/bootstrap.ign \
/dev/sda \
--ignition-hash=sha512-e11591138d028175664c5779ca696c568aaf6dc3849a0facb7e76f160184284e451131968786d2c81648d0425668a922e159541c98287be07438a596d6d84d2b

螢幕輸出 :

Installing Red Hat Enterprise Linux CoreOS 412.86.202305030814-0 (Ootpa) x86_64 (512-byte sectors)
> Read disk 4.1 GiB/4.1 GiB (100%)
Writing Ignition config
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/Wired connection 1.nmconnection to installed system
Install complete.

重新開機 

$ sudo reboot

2. 在 master 機器執行以下安裝命令 

$ sudo coreos-installer install \
--firstboot-args 'rd.neednet=1' \
--copy-network \
--ignition-url=http://192.168.11.80:8080/ocp4/master.ign \
/dev/sda \
--ignition-hash=sha512-46be965b1657dc170203ce83b9ff2ffddf34004da83cecf2440e0c50dec06ea32296c0b2c21a8324fb321ba6592ee5d12108c362d5b98ac3e5a0edf3d8fce7a8

螢幕輸出 :

> Read disk 4.1 GiB/4.1 GiB (100%)
Writing Ignition config
Writing first-boot kernel arguments
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/Wired connection 1.nmconnection to installed system
Install complete.

重新開機 

$ sudo reboot

3. Monitor the bootstrap process

回到 bastion 主機，透過 ssh 連線至 bootstrap 主機

先刪除舊的公鑰紀錄

$ rm .ssh/known_hosts

ssh 連線至 bootstrap 主機

$ ssh -i .ssh/id_ed25519 core@192.168.11.81

檢視 bootstrap 主機安裝到哪個步驟

$ journalctl -b -f -u bootkube.service

約等 20 分鐘後的正確螢幕輸出 :

...以上省略
Jul 18 02:36:10 bootstrap.ocp4.example.com bootkube.sh[10836]: 0718 02:36:10.911230       1 waitforceo.go:64] Cluster etcd operator bootstrapped successfullI0718 02:36:10.913059       1 waitforceo.go:58] cluster-etcd-operator bootstrap etcd
Jul 18 02:36:11 bootstrap.ocp4.example.com bootkube.sh[7433]: bootkube.service complete
Jul 18 02:36:11 bootstrap.ocp4.example.com systemd[1]: bootkube.service: Succeeded.

4. 等待 bootstrap process 安裝完成 

$ openshift-install --dir ocpinstall wait-for bootstrap-complete --log-level=debug

螢幕輸出 :

...以上省略
INFO Waiting up to 30m0s (until 10:40AM) for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
DEBUG Time elapsed per stage:
DEBUG Bootstrap Complete: 25m47s
INFO Time elapsed: 25m47s

看到 INFO It is now safe to remove the bootstrap resources 代表安裝完成

5. 加入 worker Node

在 worker1 和 worker2 主機執行以下命令

$ sudo coreos-installer install \
--firstboot-args 'rd.neednet=1' \
--copy-network \
--ignition-url=http://192.168.11.80:8080/ocp4/worker.ign \
/dev/sda \
--ignition-hash=sha512-c61b89807d86ef9974aaa0451015205bf9e7534b9506b52fb8883c3411c1ba1c5ec0e3080c7d35bb782cd847aa64533552b3f2132ce9b0c8fc6cf4e62933f8c6

螢幕輸出 :

> Read disk 4.1 GiB/4.1 GiB (100%)
Writing Ignition config
Writing first-boot kernel arguments
Copying networking configuration from /etc/NetworkManager/system-connections/
Copying /etc/NetworkManager/system-connections/Wired connection 1.nmconnection to installed system
Install complete.

分別重啟 Node 

$ suddo reboot

6. 使用 CLI 登錄 OCP Cluster

在 Bastion 主機執行以下命令

$ mkdir .kube && \
cp ocpinstall/auth/kubeconfig .kube/config

檢視 OCP Cluster Node 的狀態 

$ oc get nodes

螢幕輸出 :

NAME                      STATUS   ROLES                  AGE   VERSION
master.ocp4.example.com   Ready    control-plane,master   21m   v1.25.11+1485cc9

在某些 CSR 獲得批准之前，工作節點還無法被加入 OCP 叢集。

7. 核准 Worker Node 的 CSRs (憑證請求)

檢視 CSRs 的狀態 

$ oc get csr

csr is short for “certificate signing request”

螢幕輸出 :

NAME        AGE   SIGNERNAME                                    REQUESTOR
                                    REQUESTEDDURATION   CONDITION
csr-5kwr7   10m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
csr-hmrpl   10m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending

核准 CSRs 

$ oc adm certificate approve csr-5kwr7
$ oc adm certificate approve csr-hmrpl

或者也可以執行以下命令，一次核准所有的 CSRs

$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

:::danger

超級注意 ! 由於 CSRs 會自動輪換，請在將機器新增到集群後的一小時內核准 CSRs。 :::

8. 檢視 Worker Node 是否成功加入 OCP 叢集 

$ oc get nodes

螢幕輸出 :

NAME                       STATUS   ROLES                         AGE     VERSION
master.ocp4.example.com    Ready    control-plane,master,worker   3h16m   v1.25.11+1485cc9
worker1.ocp4.example.com   Ready    worker                        16m     v1.25.11+1485cc9
worker2.ocp4.example.com   Ready    worker                        16m     v1.25.11+1485cc9

It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

9. 檢視整個叢集是否健康 

$ oc get co

螢幕輸出 :

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.12.24   True        False         False      121m
baremetal                                  4.12.24   True        False         False      9d
cloud-controller-manager                   4.12.24   True        False         False      9d
cloud-credential                           4.12.24   True        False         False      9d
cluster-autoscaler                         4.12.24   True        False         False      9d
config-operator                            4.12.24   True        False         False      9d
console                                    4.12.24   True        False         False      15h
control-plane-machine-set                  4.12.24   True        False         False      9d
csi-snapshot-controller                    4.12.24   True        False         False      9d
dns                                        4.12.24   True        False         False      9d
etcd                                       4.12.24   True        False         False      9d
image-registry                             4.12.24   True        False         False      9d
ingress                                    4.12.24   True        False         False      15h
insights                                   4.12.24   True        False         False      9d
kube-apiserver                             4.12.24   True        False         False      9d
kube-controller-manager                    4.12.24   True        False         False      9d
kube-scheduler                             4.12.24   True        False         False      9d
kube-storage-version-migrator              4.12.24   True        False         False      9d
machine-api                                4.12.24   True        False         False      9d
machine-approver                           4.12.24   True        False         False      9d
machine-config                             4.12.24   True        False         False      9d
marketplace                                4.12.24   True        False         False      9d
monitoring                                 4.12.24   True        False         False      18h
network                                    4.12.24   True        False         False      9d
node-tuning                                4.12.24   True        False         False      9d
openshift-apiserver                        4.12.24   True        False         False      18h
openshift-controller-manager               4.12.24   True        False         False      8d
openshift-samples                          4.12.24   True        False         False      9d
operator-lifecycle-manager                 4.12.24   True        False         False      9d
operator-lifecycle-manager-catalog         4.12.24   True        False         False      9d
operator-lifecycle-manager-packageserver   4.12.24   True        False         False      9d
service-ca                                 4.12.24   True        False         False      9d
storage                                    4.12.24   True        False         False      9d

10. 登入 Web Console 

$ oc whoami --show-console

螢幕輸出 :

https://console-openshift-console.apps.ocp4.example.com

成功登入後的畫面

TroubleShooting

Issue: Canary route checks for the default ingress controller are failing 

$ oc get clusteroperators ingress -oyaml

完整錯誤訊息如下 :

   The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled: Pod "router-default-7bcdc5b9c7-kfcmb" cannot be scheduled: 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/master: }, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.), DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 0/1 of replicas are available), CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)

開始找真正的原因

  1. 執行以下命令，測試 pod 是否可以透過 curl 命令使用 port 8080 連線到自己。所有的 curl 請求都應該傳回 200 狀態碼，如果狀態碼不等於 200，表示 pod 尚未運作。
$ for i in `oc get po  -n openshift-ingress-canary | grep -vi name | awk '{print $1}' ` ; do echo $i ; oc exec -n openshift-ingress-canary $i  --    curl -vI http://localhost:8080/ ; done
  1. 執行以下命令，檢查 pod 是否可以使用 service 端點進行 curl。所有的 curl 請求都應該傳回 200 狀態碼，如果狀態碼不等於 200，表示 service IP 無法路由傳入的流量。
$ oc get svc -n openshift-ingress-canary

$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator $i -- curl http://<service-cluster-IP>:8080 -v ; done

service-cluster-IP，要記得換

  1. 執行以下命令，檢查對路由的連線，Curl 請求應該會得到 302 的回應碼。
$ oc get route -n openshift-ingress-canary    

$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator $i -- curl http://canary-openshift-ingress-canary.apps.<cluster-name>.<domain-name> -vIL ; done

cluster-name 和 domain-name 要記得換

參考文章

Issue: Router-Default 的 pod 無法被指派到叢集中的任何一台 Node 上 

$ oc get pods -n openshift-ingress:

NAME                             READY  STATUS   RESTARTS  AGE
router-default-84ff4c69cd-wrzxj  0/1    Pending  0         18m

執行以下命令看 ingress 的 message 訊息 

$ oc get co/ingress -o yaml

看看是不是跟以下錯誤訊息類似

- lastTransitionTime: '2021-08-11T16:32:23Z'
    message: 'Some ingresscontrollers are degraded: ingresscontroller "default" is
      degraded: DegradedConditions: One or more other status conditions indicate a
      degraded state: PodsScheduled=False (PodsNotScheduled: Some pods are not scheduled:
      Pod "router-default-84ff4c69cd-jlkh7" cannot be scheduled: 0/9 nodes are available:
      3 node(s) didn''t match Pod''s node affinity, 3 node(s) had taint {node-role.kubernetes.io/infra:
      }, that the pod didn''t tolerate, 3 node(s) had taint {node-role.kubernetes.io/master:
      }, that the pod didn''t tolerate. Make sure you have sufficient worker nodes.),
      DeploymentReplicasAllAvailable=False (DeploymentReplicasNotAvailable: 1/2 of
      replicas are available)'
    reason: IngressControllersDegraded
    status: 'True'
    type: Degraded

真正發生問題的原因 

$ oc get deploy -n openshift-ingress -o yaml | grep -A1 nodeS
        nodeSelector:
          node-role.kubernetes.io/master: ""

解決辦法

執行以下命令產生一個新的 default IngressController 的 Yaml 檔 

echo 'apiVersion: v1
items:
- apiVersion: operator.openshift.io/v1
  kind: IngressController
  metadata:
    name: default
    namespace: openshift-ingress-operator
  spec:
    nodePlacement:
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/worker: ""
    replicas: 2
  status: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""' > router-default.yaml

覆蓋現有的 ingresscontroller/default 

$ oc apply -f router-default.yaml

稍微等幾分鐘後，檢視問題是否被修復 

$ oc get all -n openshift-ingress

螢幕輸出 :

NAME                                  READY   STATUS    RESTARTS   AGE
pod/router-default-75c46bd896-7dzvp   1/1     Running   0          16h
pod/router-default-75c46bd896-tvvpx   1/1     Running   0          16h

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                   AGE
service/router-internal-default   ClusterIP   172.30.249.206   <none>        80/TCP,443/TCP,1936/TCP   9d

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/router-default   2/2     2            2           9d

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/router-default-5b9868bbcc   0         0         0       18h
replicaset.apps/router-default-75c46bd896   2         2         2       16h
replicaset.apps/router-default-7bcdc5b9c7   0         0         0       9d

參考文章

Setting up and configuring the Image Registry on OpenShift