antony@notes:~/rancher$ cat "Full-Offline-Installation-Harbor-RKE2-Rancher.md"
Full Offline Installation - Harbor, RKE2, Rancher
Full Offline Installation - Harbor, RKE2, Rancher
Preface
本篇文章主要介紹如何在離線模式下，安裝 Harbor v2.7.0, RKE2 v1.25.9, Rancher Prime v2.7.3 和 Neuvector 5.1.3，主要分為兩大部分
- 準備離線安裝所需之檔案
- 開始離線安裝
可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節
:::warning
:::spoiler 文章目錄
[TOC]
:::
準備離線安裝
Requirements for Docker, Hardware, and Networking
- 一台機器，最低規格: 4 CORE, 8G RAM, 250G Disk (建議 300G 以上)
- Disk 檔案系統為 Btrfs，需啟動 LVM，將
/目錄的可用空間設為 270G ，/home目錄 30G - 須能上網
- 檢查 Docker 儲存 Image 的目錄應為
/var/lib/docker
$ sudo docker info -f '{{ .DockerRootDir}}'- 檢查目錄的儲存容量
邏輯層
$ lsblk -o NAME,SIZE,MOUNTPOINTS實體層
$ df -hT- 檢查 Docker 儲存 Image 的目錄目前已使用多大的儲存空間
$ sudo du -sh `sudo docker info -f '{{ .DockerRootDir}}'`Prepare
Create Working Directory
$ mkdir -p work/{harbor,rke2,rancher,minio,compressed_files}準備 Harbor v2.7.0 全離線安裝包
- 切換工作目錄
$ cd ~/work/harbor- 下載 Harbor 壓縮檔，版本 v2.7.0
$ wget https://github.com/goharbor/harbor/releases/download/v2.7.0/harbor-offline-installer-v2.7.0.tgz- 下載 Docker Compose 套件，版本 1.18
$ sudo wget https://github.com/docker/compose/releases/download/1.18.0/docker-compose-Linux-x86_64- 將以上離線安裝 Harbor 所需之套件壓縮成一個檔案
$ cd ..; tar -czf compressed_files/harbor-offline.tar.gz harbor/- 檢查是否符合預期
$ ls -lh compressed_files/螢幕輸出
total 762M
-rw-r--r-- 1 762M harbor-offline.tar.gz解壓縮命令 :
tar -xzf compressed_files/harbor-offline.tar.gz
準備 RKE2 v1.25.9 全離線安裝包
- 切換工作目錄，並切換為 root 使用者
$ cd ~/work/rke2 && sudo su- 下載離線安裝 RKE2 所需 Image 之壓縮檔
$ curl -# -OL https://github.com/rancher/rke2/releases/download/v1.25.9%2Brke2r1/rke2-images.linux-amd64.tar.zst
$ curl -# -OL https://github.com/rancher/rke2/releases/download/v1.25.9%2Brke2r1/rke2.linux-amd64.tar.gz
$ curl -# -OL https://github.com/rancher/rke2/releases/download/v1.25.9%2Brke2r1/sha256sum-amd64.txt- 下載官網提供的離線安裝 RKE2 所需之安裝腳本，並賦予它執行權限
$ curl -sfL https://get.rke2.io --output install.sh && chmod +x install.sh- 檢查當前目錄下的檔案
$ ls -lh螢幕輸出
total 939M
-rwxr-xr-x 1 root root 22K May 23 13:48 install.sh
-rw-r--r-- 1 root root 915M May 23 13:46 rke2-images.linux-amd64.tar.zst
-rw-r--r-- 1 root root 25M May 23 13:47 rke2.linux-amd64.tar.gz
-rw-r--r-- 1 root root 3.6K May 23 13:47 sha256sum-amd64.txt- 將以上離線安裝 RKE2 所需之檔案，壓縮成一個檔案
$ cd ..; tar -czf compressed_files/rke2-offline.tar.gz rke2/- 檢查是否符合預期
$ ls -lh compressed_files/螢幕輸出
total 1.7G
-rw-r--r-- 1 762M harbor-offline.tar.gz
-rw-r--r-- 1 937M rke2-offline.tar.gz解壓縮命令 :
tar -xzf compressed_files/rke2-offline.tar.gz
- 離開 root 使用者
$ exit準備 Rancher Prime v2.7.3 全離線安裝包
$ cd ~/work/rancher
# 安裝 helm
$ wget https://get.helm.sh/helm-v3.8.2-linux-amd64.tar.gz
$ tar zxvf helm-v3.8.2-linux-amd64.tar.gz
$ sudo cp linux-amd64/helm /usr/local/bin/
$ helm repo add rancher-prime https://charts.rancher.com/server-charts/prime
$ helm repo update
$ helm fetch rancher-prime/rancher --version=v2.7.3
$ wget https://github.com/rancher/rancher/releases/download/v2.7.3/rancher-images.txt
$ wget https://github.com/rancher/rancher/releases/download/v2.7.3/rancher-load-images.sh
$ wget https://github.com/rancher/rancher/releases/download/v2.7.3/rancher-save-images.sh- 準備 cert-manager 環境
$ wget https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm fetch jetstack/cert-manager --version v1.11.0
$ sudo docker pull quay.io/jetstack/cert-manager-controller:v1.11.0
$ sudo docker pull quay.io/jetstack/cert-manager-webhook:v1.11.0
$ sudo docker pull quay.io/jetstack/cert-manager-cainjector:v1.11.0
$ sudo docker pull quay.io/jetstack/cert-manager-ctl:v1.11.0
# 之後要再改 tag push 到 harbor
$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx stable 7067317e3807 10 days ago 142MB
quay.io/jetstack/cert-manager-controller v1.7.1 db7725ef729d 12 months ago 65.9MB
quay.io/jetstack/cert-manager-webhook v1.7.1 36736adc2f4b 12 months ago 52.5MB
quay.io/jetstack/cert-manager-cainjector v1.7.1 623ec0d31539 12 months ago 44.5MB
quay.io/jetstack/cert-manager-ctl v1.7.1 4e60e4ce697f 12 months ago 57.2MB
$ sudo docker tag quay.io/jetstack/cert-manager-controller:v1.7.1 harbor.example.com/rancher/cert-manager-controller:v1.7.1
$ sudo docker tag quay.io/jetstack/cert-manager-webhook:v1.7.1 harbor.example.com/rancher/cert-manager-webhook:v1.7.1
$ sudo docker tag quay.io/jetstack/cert-manager-cainjector:v1.7.1 harbor.example.com/rancher/cert-manager-cainjector:v1.7.1
$ sudo docker tag quay.io/jetstack/cert-manager-ctl:v1.7.1 harbor.example.com/rancher/cert-manager-ctl:v1.7.1- 下載 neuvector image
$ sudo docker pull neuvector/controller:5.1.3
$ sudo docker pull neuvector/manager:5.1.3
$ sudo docker pull neuvector/enforcer:5.1.3
$ sudo docker pull neuvector/scanner:latest
$ sudo docker pull neuvector/updater:latest
$ sudo docker tag neuvector/scanner harbor.example.com/rancher/neuvector/scanner
$ sudo docker tag neuvector/updater harbor.example.com/rancher/neuvector/updater
$ sudo docker tag neuvector/manager:5.1.1 harbor.example.com/rancher/neuvector/manager:5.1.1
$ sudo docker tag neuvector/enforcer:5.1.1 harbor.example.com/rancher/neuvector/enforcer:5.1.1
$ sudo docker tag neuvector/controller:5.1.1 harbor.example.com/rancher/neuvector/controller:5.1.1- 下載所有 rancher 會用到的 image
$ chmod +x rancher-save-images.sh
$ chmod +x rancher-load-images.sh
$ ll
total 13432
-rw-r--r-- 1 rancher users 66038 Feb 20 10:20 cert-manager-v1.7.1.tgz
-rw-r--r-- 1 rancher users 13633605 Apr 14 2022 helm-v3.8.2-linux-amd64.tar.gz
-rw-r--r-- 1 rancher users 13996 Feb 20 10:10 rancher-2.7.1.tgz
-rw-r--r-- 1 rancher users 19406 Jan 25 06:21 rancher-images.txt
-rwxr-xr-x 1 rancher users 4115 Jan 25 06:22 rancher-load-images.sh
-rwxr-xr-x 1 rancher users 1757 Jan 25 06:21 rancher-save-images.sh
drwxr-xr-x 2 rancher rancher 50 Apr 14 2022 linux-amd64
-rw-r--r-- 1 rancher rancher 358698 Feb 4 2022 cert-manager.crds.yaml
# 預計下載會等兩個小時
# registry.rancher.com rancher-prime 來源
$ sudo ./rancher-save-images.sh --image-list ./rancher-images.txt --images rancher-images.tar.gz --source-registry registry.rancher.com | tee result.log- 壓縮全離線安裝 rancher 會需要的所有檔案
$ tar -czvf rancher-offline.tar.gz cert-manager.crds.yaml image/rancher-image.tar.gz helm-v3.8.2-linux-amd64.tar.gz rancher-2.7.3.tgz cert-manager-v1.11.0.tgz rancher-images.txt rancher-load-images.sh push.sh程式自動化
:::spoiler load 和 Save 自動化程式
#!/bin/bash
[ -f detail-result.txt ] && cat /dev/null > detail-result.txt
[ -f result.txt ] && cat /dev/null > result.txt
exec {BASH_XTRACEFD}>>detail-result.txt
set -x
## load Image
### var
all_image_txt=$(cat rancher-images.txt cert-manager-images.txt neuvector-images.txt)
while read line
do
if sudo docker images "${line}" | grep -q "${line%:*}"; then
if ! sudo docker rmi "${line}" &> /dev/null; then
echo rmi "${line}" Error && exit 1
fi
fi
done <<< $(cat rancher-images.txt | sed 's|registry.rancher.com/||g')
while read image
do
if ! sudo docker pull "$image" 1> /dev/null 2> ./result.txt; then
echo pull "$image" Error && exit 1
fi
done <<< $(cat rancher-images.txt cert-manager-images.txt neuvector-images.txt)
## Save Image
### var
all_image=$(cat rancher-images.txt cert-manager-images.txt | tr '\n' ' ')
neuvector_image=$(cat neuvector-images.txt | sed 's|docker.io/||g' | tr '\n' ' ')
for n in $all_image $neuvector_image
do
if ! sudo docker images "$n" | grep -q "${n%:*}"; then
echo "$n" pull Error >> pull_error.txt
if ! sudo docker pull "$n" 1> /dev/null 2> still_pull_error.txt; then
echo "$n" pull Error twice >> pull_error.txt && exit 1
fi
fi
done
sudo docker save $all_image | gzip --stdout > rancher-cert_manager-image.tar.gz
[ "$?" != "0" ] && echo "save rancher image error" && exit 1
## 壓成一個壓縮檔
tar -czf ../compressed_files/rancher-offline.tar.gz rancher-cert_manager-image.tar.gz rancher-2.7.3.tgz cert-manager-v1.11.0.tgz helm-v3.8.2-linux-amd64.tar.gz cert-manager.crds.yaml
sudo docker save $neuvector_image | gzip --stdout > ../compressed_files/neuvector-image.tar.gz
[ "$?" != "0" ] && echo "save neuvector image error" && exit 1
tar -czf ../compressed_files/full-offline.tar.gz ../compressed_files/harbor-offline.tar.gz ../compressed_files/neuvector-image.tar.gz ../compressed_files/rancher-offline.tar.gz ../compressed_files/rke2-offline.tar.gz
[ "$?" == "0" ] && split -b 7G ../compressed_files/full-offline.tar.gz ../compressed_files/full-offline.tar.gz.part:::
:::spoiler minio
minio
$ cd ~/work/minio
# 準備 docker-compose
$ sudo wget https://github.com/docker/compose/releases/download/1.18.0/docker-compose-Linux-x86_64- 準備 image
$ sudo docker pull minio/minio
$ sudo docker tag minio/minio harbor.example.com/library/minio
$ sudo docker push harbor.example.com/library/minio:::
開始離線安裝
建置系統環境確認
- Private Registry: 4 CORE, 6G RAM, 300G Disk
- Rancher Manager: 4 CORE, 8G RAM, 100G Disk
- Kubernetes Master: 4 CORE, 8G RAM, 100G Disk
- Worker(*3): 4 CORE, 8G RAM, 100G Disk
安裝前準備
- 確認 根目錄 可用空間
$ df -hT /螢幕輸出:
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/system-root btrfs 200G 28G 173G 14% /- 將全離線安裝包上傳至機器
$ scp -r split/ <遠端機器>檢查 split/ 目錄下之全離線安裝包檔案
ls -lh split/螢幕輸出
total 27G
-rw-r--r-- 1 root root 7.0G Jun 6 21:51 full_offline.tar.gz.aa
-rw-r--r-- 1 root root 7.0G Jun 6 21:51 full_offline.tar.gz.ab
-rw-r--r-- 1 root root 7.0G Jun 6 21:52 full_offline.tar.gz.ac
-rw-r--r-- 1 root root 5.9G Jun 6 21:53 full_offline.tar.gz.ad- 建立並切換工作目錄
$ mkdir -p compressed_files && cd compressed_files/- 解壓縮全離線安裝包 (耗時約 3 分鐘)
$ cat ../split/full_offline.tar.gz.a* | tar -zxvf -須注意全離線安裝包的路徑
螢幕輸出:
harbor-offline.tar.gz
rancher-offline.tar.gz
rke2-offline.tar.gz檢視檔案
ls -lh螢幕輸出 :
total 27G
-rw-r--r-- 1 1003 users 762M May 23 14:33 harbor-offline.tar.gz
-rw-r--r-- 1 1003 users 26G May 26 10:34 rancher-offline.tar.gz
-rw-r--r-- 1 root root 937M May 23 15:10 rke2-offline.tar.gz- 將 harbor、rancher、rke2 全離線安裝包傳送至對應之機器
$ scp harbor-offline.tar.gz <harbor 的機器>$ scp rke2-offline.tar.gz <rancher 的機器>
$ scp rancher-offline.tar.gz <rancher 的機器>$ scp rke2-offline.tar.gz <rke2 的機器>
