Skip to content

antony@notes:~/suse$ cat "SUSE-實作面試.md"

SUSE - 實作面試

2023-06-06· suse

SUSE - 實作面試

題目

  1. 安裝 Rancher 與 K8S 叢集，Rancher 版本為 2.7.1, K8S 為 1.23，CRI 為 RKE2，CNI 為 Calico。
  2. 安裝 NeuVector 在 K8S Cluster 中，版本為 5.1.1，CRI 要改為 K3S。
  3. 將 NFS 的空間用三顆 disk 組成 LVM 後，再透過 NFS share出來給環境使用。
  4. 將 Kubernetes、MySQL Cluster NDB 、異地備援專題的實作建立於 K8S Cluster 中。

解答

Lab 環境介紹

  • dns-nfs: 192.168.0.31
  • rsm1: 192.168.0.32
  • rke2-1: 192.168.0.33
  • rke2-2: 192.168.0.34
  • DNS: 192.168.0.31
  • Hosts must add
    • 192.168.0.32 rancher.example.com

Install DNS 

~> sudo zypper in -t pattern dhcp_dns_server
  • install, in ，Install packages.
  • -t, --type <TYPE>，Type of package (package, patch, pattern, product).
    • pattern means a group of software packages.

login 

> ssh rancher@192.168.0.32

下載安裝 Rancher 的程式 

~> curl -sfL https://get.rke2.io --output install.sh

賦予執行的權限

~> chmod +x install.sh

config rke2 basic parameters

建立 Rancher rke2 的目錄

~> sudo mkdir -p /etc/rancher/rke2/

編輯 yaml 檔

~> sudo nano /etc/rancher/rke2/config.yaml

檔案內容

node-name:
  - "rsm1"
token: my-shared-secret

install rke2 with 1.23.9 

~> sudo INSTALL_RKE2_CHANNEL=v1.23.9+rke2r1 ./install.sh

螢幕輸出 :

[WARN]  /usr/local is read-only or a mount point; installing to /opt/rke2
[INFO]  finding release for channel v1.23.9+rke2r1
[INFO]  using v1.23.9+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.23.9+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /opt/rke2
[INFO]  updating tarball contents to reflect install path
[INFO]  moving systemd units to /etc/systemd/system
[INFO]  install complete; you may want to run:  export PATH=$PATH:/opt/rke2/bin

將 /opt/rke2/bin 加到 PATH 環境變數

~> export PATH=$PATH:/opt/rke2/bin

enable rke2 and setup kubeconfig 

~> sudo systemctl enable rke2-server

螢幕輸出 :

Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /etc/systemd/system/rke2-server.service.

啟動 rke2-server

~> sudo systemctl start rke2-server

將當前使用者設為 Rancher 管理員

~> mkdir .kube
~> sudo cp /etc/rancher/rke2/rke2.yaml .kube/config
~> sudo chown rancher .kube/config

將 kubectl 複製到 /usr/local/bin/ 目錄下

~> sudo cp /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/

check pod status 

~> kubectl get po -A

螢幕輸出

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-rsm1                           1/1     Running     0          34m
kube-system   etcd-rsm1                                               1/1     Running     0          35m
kube-system   helm-install-rke2-canal-lqg7v                           0/1     Completed   0          35m
kube-system   helm-install-rke2-coredns-4pqwm                         0/1     Completed   0          35m
kube-system   helm-install-rke2-ingress-nginx-g8q7r                   0/1     Completed   0          35m
kube-system   helm-install-rke2-metrics-server-5spzg                  0/1     Completed   0          35m
kube-system   kube-apiserver-rsm1                                     1/1     Running     0          34m
kube-system   kube-controller-manager-rsm1                            1/1     Running     0          34m
kube-system   kube-proxy-rsm1                                         1/1     Running     0          34m
kube-system   kube-scheduler-rsm1                                     1/1     Running     0          34m
kube-system   rke2-canal-b5vhg                                        2/2     Running     0          34m
kube-system   rke2-coredns-rke2-coredns-545d64676-x4nbx               1/1     Running     0          34m
kube-system   rke2-coredns-rke2-coredns-autoscaler-5dd676f5c7-lc9gf   1/1     Running     0          34m
kube-system   rke2-ingress-nginx-controller-b2txr                     1/1     Running     0          34m
kube-system   rke2-metrics-server-6564db4569-7t5rk                    1/1     Running     0          34m

install helm3

下載 helm3

~> wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz

螢幕輸出

--2023-03-07 02:40:48--  https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz
Resolving get.helm.sh (get.helm.sh)... 152.199.39.108, 2606:2800:247:1cb7:261b:1f9c:2074:3c
Connecting to get.helm.sh (get.helm.sh)|152.199.39.108|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13633605 (13M) [application/x-tar]
Saving to: ‘helm-v3.8.2-linux-amd64.tar.gz’

helm-v3.8.2-linux-amd64.t 100%[=====================================>]  13.00M  10.6MB/s    in 1.2s

2023-03-07 02:40:50 (10.6 MB/s) - ‘helm-v3.8.2-linux-amd64.tar.gz’ saved [13633605/13633605]

解壓縮 helm3

~> tar zxvf helm-v3.8.2-linux-amd64.tar.gz

螢幕輸出

linux-amd64/
linux-amd64/helm
linux-amd64/LICENSE
linux-amd64/README.md

將 helm 拷貝至 /usr/local/bin/ 目錄下

~> sudo cp linux-amd64/helm /usr/local/bin/

install rancher and cert-manager 

~> helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
"rancher-stable" has been added to your repositories

螢幕輸出

"rancher-stable" has been added to your repositories

建立 cattle-system Namespace

~> kubectl create namespace cattle-system

螢幕輸出

namespace/cattle-system created

安裝 cert-manager

~> kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml

螢幕輸出

customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created

安裝 jetstack

~> helm repo add jetstack https://charts.jetstack.io

螢幕輸出

"jetstack" has been added to your repositories

更新 helm 儲存庫

~> helm repo update

螢幕輸出

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈

安裝 cert-manager

~> helm install cert-manager jetstack/cert-manager \
space ce> --namespace cert-manager \
> --create-namespace \
> --version v1.7.1

螢幕輸出

NAME: cert-manager
LAST DEPLOYED: Tue Mar  7 04:27:00 2023
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.7.1 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/

檢視 Cert-manager pod 狀態

~> kubectl get pods --namespace cert-manager

螢幕輸出

NAME                                     READY   STATUS    RESTARTS   AGE
cert-manager-76d44b459c-45vrr            1/1     Running   0          101s
cert-manager-cainjector-9b679cc6-z5gq7   1/1     Running   0          101s
cert-manager-webhook-57c994b6b9-cl72k    1/1     Running   0          101s

安裝 rancher 2.7.1

~> helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.example.com --version 2.7.1

螢幕輸出

NAME: rancher
LAST DEPLOYED: Tue Mar  7 04:30:45 2023
NAMESPACE: cattle-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Rancher Server has been installed.

NOTE: Rancher may take several minutes to fully initialize. Please standby while Certificates are being issued, Containers are started and the Ingress rule comes up.

Check out our docs at https://rancher.com/docs/

If you provided your own bootstrap password during installation, browse to https://rancher.example.com to get started.

If this is the first time you installed Rancher, get started by running this command and clicking the URL it generates:


echo https://rancher.example.com/dashboard/?setup=$(kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}')


To get just the bootstrap password on its own, run:


kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{ "\n" }}'



Happy Containering!

check rancher status 

~> watch -n 1 kubectl -n cattle-system get po

螢幕輸出

NAME                               READY   STATUS      RESTARTS       AGE
helm-operation-67hnc               0/2     Completed   0              43s
helm-operation-g2f84               0/2     Completed   0              93s
helm-operation-hthw8               0/2     Completed   0              36s
rancher-5799876645-5wr79           1/1     Running     0              3m49s
rancher-5799876645-pr5ns           1/1     Running     1 (2m4s ago)   3m49s
rancher-5799876645-vp422           1/1     Running     0              3m49s
rancher-webhook-69bb6bc7cf-nkdq7   1/1     Running     0              33s

連線 Rancher

打開瀏覽器，輸入 example.rancher.com

:::spoiler 示意圖 :::

複製命令

~> kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'

螢幕輸出

4xb47zj5mhtrvf7crxk7fsn644rx842gl7b2rmkfsg5pscwjvsnhcs

選擇 “Set a specific password to use”，輸入自訂密碼後，勾選 “By checking the box, you accept the End User Lincense Agreement & Terms & Corditions”

:::spoiler 示意圖 :::

建立 K8s

按 “Create”

將 RKE2/K3s 打勾，並且點選 “Custom”

填入 “Cluster Name”，Kubernetes Version 選擇 “v.123.15+rke2r1”，再點 “Create”

勾選 “Insecure”，允許不安全的連線，並點選第 2 點的框框複製

到 rke2-1 主機執行

rke2-1:~> curl -fL https://rancher.example.com/system-agent-install.sh | sudo  sh -s - --server https://rancher.example.com --label 'cattle.io/os=linux' --token pkqfn6d6ghkltw4dvn8zdwsk7q5s2w9z5h46vsgwmfp8zng2k6rlfm --ca-checksum c93bc39496a549d53f0a8e74d757795202a8cff487cabd9d26bd449375ecd482 --etcd --controlplane --worker

螢幕輸出

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29802    0 29802    0     0  3596k      0 --:--:-- --:--:-- --:--:-- 3637k
[INFO]  Label: cattle.io/os=linux
[INFO]  Role requested: etcd
[INFO]  Role requested: controlplane
[INFO]  Role requested: worker
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[WARN]  /usr/local is read-only or a mount point; installing to /opt/rancher-system-agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://rancher.example.com/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://rancher.example.com/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading rancher-system-agent-uninstall.sh script from https://rancher.example.com/assets/system-agent-uninstall.sh
[INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO]  Generating Cattle ID
[INFO]  Successfully downloaded Rancher connection information
[INFO]  systemd: Creating service file
[INFO]  Creating environment file /etc/systemd/system/rancher-system-agent.env
[INFO]  Enabling rancher-system-agent.service
Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service.
[INFO]  Starting/restarting rancher-system-agent.service

LVM 實做流程

建立 PV (Physical Volume) 

~> sudo pvcreate /dev/{sdb,sdc,sdd}

檢查

~> sudo pvscan

螢幕輸出

  PV /dev/sdb    VG nfsvg           lvm2 [49.98 GiB / 0    free]
  PV /dev/sdc    VG nfsvg           lvm2 [49.98 GiB / 0    free]
  PV /dev/sdd    VG nfsvg           lvm2 [49.98 GiB / 0    free]
  PV /dev/sda2   VG system          lvm2 [399.99 GiB / 334.99 GiB free]
  Total: 4 [549.94 GiB] / in use: 4 [549.94 GiB] / in no VG: 0 [0   ]

建立 VG (Volume Group) 

~> sudo vgcreate -s 16M nfsvg /dev/sd{b,c,d}

檢查

~> sudo vgdisplay nfsvg

螢幕輸出

  --- Volume group ---
  VG Name               nfsvg
  System ID
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               149.95 GiB
  PE Size               16.00 MiB
  Total PE              9597
  Alloc PE / Size       9597 / 149.95 GiB
  Free  PE / Size       0 / 0
  VG UUID               tUhEIt-JEGs-xZxx-j4lq-CODo-781Z-3Jlfjj

建立 LV (Logical Volume) 

~> sudo lvcreate -n nfslv -L 500M nfsvg

檢查

~> sudo lvdisplay /dev/nfsvg/nfslv

螢幕輸出

  --- Logical volume ---
  LV Path                /dev/nfsvg/nfslv
  LV Name                nfslv
  VG Name                nfsvg
  LV UUID                WWbbDs-nQ7Y-JaXJ-ROMD-v6SU-6NEf-WVIOG0
  LV Write Access        read/write
  LV Creation host, time dns-nfs, 2023-03-07 23:20:10 +0800
  LV Status              available
  # open                 1
  LV Size                512.00 MiB
  Current LE             32
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           254:2
  • 因為使用 16MB 的 PE，因此不會剛好等於 500MB， LV 自動選擇接近 500MB 的數值來建立， 因此會得到使用 512MB 的容量。

檢視還有多少 PE 可以用

~> sudo vgdisplay nfsvg

螢幕輸出

  --- Volume group ---
  VG Name               nfsvg
  System ID
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  3
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               149.95 GiB
  PE Size               16.00 MiB
  Total PE              9597
  Alloc PE / Size       9597 / 149.95 GiB
  Free  PE / Size       0 / 0
  VG UUID               tUhEIt-JEGs-xZxx-j4lq-CODo-781Z-3Jlfjj

為 LV 增加容量

~> sudo lvresize -l +9565 /dev/nfsvg/nfslv

螢幕輸出

  Size of logical volume nfsvg/nfslv changed from 512.00 MiB (32 extents) to 149.95 GiB (9597 extents).
  Logical volume nfsvg/nfslv successfully resized.

補充 :

幫 LV resize 特定大小

$ sudo lvresize -L 25G <LV Name>

幫 LV resize 到最大值

$ sudo lvresize -l +100%FREE <LV Name>

檢查是否符合預期

~> sudo lvscan

螢幕輸出

  ACTIVE            '/dev/nfsvg/nfslv' [149.95 GiB] inherit
  ACTIVE            '/dev/system/home' [25.00 GiB] inherit
  ACTIVE            '/dev/system/root' [40.00 GiB] inherit

Format LVM

將 LVM 格式化為 xfs 檔案系統

~> sudo mkfs.xfs /dev/nfsvg/nfslv

測試 LVM 可否正常使用

# 建立工作目錄
~> sudo mkdir /opt/www

# 將 LV 掛載到 /opt/www 目錄
~> sudo mount /dev/nfsvg/nfslv /opt/www

~> cd /opt/www
~> echo "testing lvm partition" | sudo tee  dummy.txt
~> cat dummy.txt
testing lvm partition
~> 
~> sudo rm -f  dummy.txt

resize btrfs filesystem 到最大值 

~> sudo btrfs filesystem resize max /

永久使用 LVM 

~> echo '/dev/nfsvg/nfslv  /opt/www  xfs  defaults 0 0' | sudo  tee -a /etc/fstab
~> sudo mount -a

安裝 NFS Server 

~> sudo zypper -n install nfs-kernel-server

設定共享資料夾 

~> echo '/opt/www 192.168.0.31/24(rw,sync,no_subtree_check,no_root_squash)' | sudo tee /etc/exports

啟動 NFS Server 

~> sudo systemctl start nfs-server
~> sudo systemctl enable nfs-server

修改權限 

~> sudo chown -R rancher:users /opt/www

再另一台機器測試 

~> sudo mkdir -p /opt/www
~> sudo mount -t nfs 192.168.0.31:/opt/www /opt/www
~> echo 'haha' > /opt/www/test

回到原來的機器 

> cat /opt/www/test

螢幕輸出

haha

成功~

Overview of SUSE Linux Enterprise在 SUSE Custom Center 啟用 SUSE Liberty Linux 訂閱