antony@notes:~/kubernetes$ cat "Kubernetes-Landlord.md"

Kubernetes Landlord

2023-01-07· kubernetes ·系統工程

Kubernetes Landlord

[TOC]

部署 Landlord on K8S

環境準備 CNT.2022.v4 K8S 叢集建置完成以下 K8S 公設完成部署

MetalLB
Ingress NGINX Controller
Local Path Provisioner

在 Windows 系統的 cmd 視窗登入 m1 主機

$ kubectl create ns landlord
namespace/landlord created

$ kubectl create -n landlord configmap kuser-conf --from-file /home/bigred/.kube/config
configmap/kuser-conf created

$ kubectl apply -f https://web.flymks.com/landlord/v1/landlord.yaml

確認狀態

$ kubectl get all -n landlord
NAME                           READY   STATUS    RESTARTS   AGE
pod/gateway-759f9676cd-fms9r   1/1     Running   0          12m
pod/kuser-7bcd86d848-nvz5t     1/1     Running   0          12m
pod/logger-7c68f4cf59-b255x    1/1     Running   0          12m
pod/mariadb-b8fdcc6c4-48pj6    1/1     Running   1          12m
pod/tenant-0                   1/1     Running   0          12m
pod/tenant-1                   1/1     Running   0          11m
pod/tenant-2                   1/1     Running   0          2m21s

NAME              TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)           AGE
service/gateway   LoadBalancer   10.98.0.163   192.168.61.221   20000:31662/TCP   12m
service/kuser     ClusterIP      None          <none>           <none>            12m
service/logger    ClusterIP      None          <none>           <none>            12m
service/mariadb   ClusterIP      None          <none>           <none>            12m
service/tenant    ClusterIP      None          <none>           <none>            12m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/gateway   1/1     1            1           12m
deployment.apps/kuser     1/1     1            1           12m
deployment.apps/logger    1/1     1            1           12m
deployment.apps/mariadb   1/1     1            1           12m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/gateway-759f9676cd   1         1         1       12m
replicaset.apps/kuser-7bcd86d848     1         1         1       12m
replicaset.apps/logger-7c68f4cf59    1         1         1       12m
replicaset.apps/mariadb-b8fdcc6c4    1         1         1       12m

NAME                      READY   AGE
statefulset.apps/tenant   3/3     12m

問題討論

為甚麼角色大多使用 deployment 設計 ?
- 因為當 node 發生異常，導致 pod 掛掉時，在 node 上的 kubelet 也會跟著去世，這時候如果我們是使用 deployment 設計的 pod ，這些 pod 就會由 deployment 的 ReplicaSets 去照顧 pod ，讓 pod 浴火重生， pv 與 pvc 的問題，可以由分散式的 CSI 解決。
為甚麼要有 Gateway 這個腳色？
- 如果今天沒有提供 gateway ，使用者就要
- 統一控管網路的入口。
為甚麼 Tenant 要用 StatefulSets 設計？
- 讓 pod 各自有自己的 pvc 與 pv 。
Tenant 是透過何種方式達到永存資料
- 用 local-path-pr
可不可以沒有 Logger ，將資料直接放入 Mariadb
- 如果沒有 Logger 會有安全性的問題，
- Logger 寫入 Mariadb 是用特殊方法寫入
沒有 Kuser 會如何
- 為了能讓一般使用者能連接 Kubernetes

Landlord 移除

$ kubectl delete -f https://web.flymks.com/landlord/v1/landlord.yaml

$ kubectl delete ns tenant-0 tenant-1 tenant-2

$ kubectl delete csr tenant-0-csr tenant-1-csr tenant-2-csr

實作 Landlord

下載 Landlord 實作包
$ wget https://web.flymks.com/landlord/v1/sre_landlord.zip
…
2022-06-12 22:29:50 (2.79 MB/s) - 'sre_landlord.zip' saved [18712/18712]

$ unzip sre_landlord.zip
Archive:  sre_landlord.zip
…

$ cd sre_landlord

Landlord 目錄結構

$ tree .
.
├── Dockerfile.landlord.kubectl
├── Dockerfile.landlord.kuser
├── Dockerfile.landlord.mariadb
├── LICENSE
├── app_kuser
│   ├── RBAC.md
│   ├── ckuser    ---------------> ## 判斷是否為 Kubernetes 使用者
│   ├── config    ---------------> ## 處理謙和過的憑證
│   ├── create    ---------------> ## 建立使用者憑證
│   ├── delete    ---------------> ## 刪除 Kubernetes 使用者
│   ├── go.mod
│   ├── go.sum
│   ├── main.go
│   └── template
│       ├── config.temp
│       ├── csr.temp
│       ├── role.temp
│       └── rolebind.temp
├── bin
│   ├── gateway
│   │   └── adduser
│   ├── logger
│   │   └── log_server
│   └── tenant
│       └── log_client
├── conf
│   └── mariadb
│       └── initdb
│           └── initdb.sql
└── landlord.yaml

9 directories, 21 files

提問：Golang 透過提交路由，執行內部 bash script。系統設計上，有哪些好處？

答：

希望使用者不用再打 create tenant-0 ，系統直接判定你是誰幫你建立所有你要的功能，這也是一種降低 Human Error
核心是希望權限由系統處理
Golang 不是分流，他主要的功能是 RestApi 的形式，透過網頁供他人存取，存取的程式都是 bash script
整個系統都會由 ckuser 來處理與判定，不會讓非 Kubernetes 使用者（禁止未登入 Cluster 的指令請求）

$ cat Dockerfile.landlord.kuser
FROM quay.io/flysangel/golang AS build-env

ADD ./app_kuser /src

RUN \
    cd /src && \
    CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o main


FROM quay.io/flysangel/image:alpine.base AS main

RUN \
    apk update && \
    apk add --no-cache openssl

RUN \
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl && \
    rm -r kubectl

RUN \
    mkdir -p /app_kuser/kuser

COPY --from=build-env /src/main /app_kuser/main
COPY app_kuser/template /app_kuser/template
COPY app_kuser/ckuser /app_kuser/ckuser
COPY app_kuser/config /app_kuser/config
COPY app_kuser/create /app_kuser/create
COPY app_kuser/delete /app_kuser/delete

WORKDIR /app_kuser

ENTRYPOINT ["./main"]

實作 Kuser

$ sudo podman build -t quay.k8s.org/quay/landlord.kuser:1.0.0 -f Dockerfile.landlord.kuser .

$ sudo podman login --tls-verify=false quay.k8s.org
Username: quay
Password: Quay12345
Login Succeeded!

$ sudo podman push --tls-verify=false quay.k8s.org/quay/landlord.kuser:1.0.0
Writing manifest to image destination
Storing signatures

$ cat Dockerfile.landlord.mariadb
FROM quay.io/flysangel/mariadb

COPY conf/mariadb/initdb/initdb.sql /docker-entrypoint-initdb.d

$ cat conf/mariadb/initdb/initdb.sql
CREATE TABLE IF NOT EXISTS history (
  Hostname VARCHAR(50),
  Username VARCHAR(50),
  Time VARCHAR(50),
  Command VARCHAR(255)
);
CREATE TABLE IF NOT EXISTS inotify (
  Hostname VARCHAR(50),
  Time VARCHAR(50),
  Event VARCHAR(20),
  Path VARCHAR(255)
);

練習

實作 Gateway

請設計一個 Container Image 名為 landlord.gateway，標籤為 1.0.0，並滿足以下功能

FROM quay.io/flysangel/image:alpine.sshd
複製 adduser 程式至 /bin
CMD ["/bin/bash", "-c", "/bin/adduser; /usr/sbin/sshd -D"]

將 Container Image 上傳至落地 Quay 在 K8S 使用 landlord.gateway 驗證 Pod 內是否有成功建立使用者

$ nano Dockerfile.landlord.gateway
FROM quay.io/flysangel/image:alpine.sshd

COPY bin/gateway/adduser /bin

CMD ["/bin/bash", "-c", "/bin/adduser; /usr/sbin/sshd -D"]

$ sudo podman build -t quay.k8s.org/quay/landlord.gateway:1.0.0 -f Dockerfile.landlord.gateway .

$ sudo podman images | grep gateway
quay.k8s.org/quay/landlord.gateway                        1.0.0             22cd776d7171  5 seconds ago   59.5 MB

$ sudo podman push --tls-verify=false quay.k8s.org/quay/landlord.gateway:1.0.0

$ nano pod-landlord.gateway.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gateway
  namespace: landlord
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gateway
  template:
    metadata:
      labels:
        app: gateway
    spec:
      containers:
      - env:
        - name: NS
          value: landlord
        - name: USERTAG
          value: tenant
        image: quay.k8s.org/quay/landlord.gateway:1.0.0
        imagePullPolicy: Always
        name: gateway
        securityContext:
          capabilities:
            add:
            - AUDIT_WRITE
            - SYS_CHROOT
            - CAP_NET_RAW
      restartPolicy: Always


$ kubectl apply -f Deployment-landlord.gateway.yml

$ kubectl get pod -n landlord
NAME                       READY   STATUS    RESTARTS   AGE
gateway-6898d75c8f-5f2rc   1/1     Running   0          4s

$ kubectl exec -it -n landlord gateway-6898d75c8f-5f2rc -- bash
bash-5.1#/bin/adduser
adduser: user 'tenant-0' in use
chpasswd: password for 'tenant-0' changed
adduser: user 'tenant-1' in use
chpasswd: password for 'tenant-1' changed
... 以下省略

bash-5.1# cat /etc/passwd
... 以上省略
tenant-0:x:1001:1001:Linux User,,,:/home/tenant-0:/bin/bash
tenant-1:x:1002:1002:Linux User,,,:/home/tenant-1:/bin/bash
tenant-2:x:1003:1003:Linux User,,,:/home/tenant-2:/bin/bash
...
tenant-98:x:1099:1099:Linux User,,,:/home/tenant-98:/bin/bash
tenant-99:x:1100:1100:Linux User,,,:/home/tenant-99:/bin/bash

實作 Logger

請設計一個 Container Image 名為 landlord.logger，標籤為 1.0.0，並滿足以下功能

FROM quay.io/flysangel/image:alpine.base
安裝 mysql client
複製 log_server 程式至 /bin
CMD ["/bin/bash", "-c", "/bin/log_server"]

將 Container Image 上傳至落地 Quay 在 K8S 使用 landlord.logger 驗證 Pod 內是否有執行 log_server

$ nano Dockerfile.landlord.logger
FROM quay.io/flysangel/image:alpine.base

RUN \
  apk update && \
  apk add mysql mysql-client

COPY bin/logger/log_server /bin
CMD ["/bin/bash", "-c", "/bin/log_server"

$ sudo podman build -t quay.k8s.org/quay/landlord.logger:1.0.0 -f Dockerfile.landlord.logger .

$ sudo podman images | grep logger
quay.k8s.org/quay/landlord.logger                         1.0.0             7e8992f656b8  19 seconds ago     235 MB

$ sudo podman push --tls-verify=false quay.k8s.org/quay/landlord.logger:1.0.0

$ nano Deployment-landlord.logger.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logger
  namespace: landlord
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logger
  template:
    metadata:
      labels:
        app: logger
    spec:
      containers:
      - env:
        - name: NS
          value: landlord
        - name: DB_SERVER
          value: mariadb
        - name: DB_NAME
          value: landlord
        - name: DB_ROOT
          value: landlord
        - name: DB_ROOT_PW
          value: mymariadblandlord
        image: quay.k8s.org/quay/landlord.logger:1.0.0
        imagePullPolicy: Always
        name: logger
      restartPolicy: Always

$ kubectl apply -f Deployment-landlord.logger.yml

$ kubectl get pod -n landlord -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP             NODE   NOMINATED NODE   READINESS GATES
gateway-6898d75c8f-5f2rc   1/1     Running   0          28m   10.244.1.146   w1     <none>           <none>
logger-64d95c7ddd-wp46c    1/1     Running   0          21s   10.244.1.147   w1     <none>           <none>

$ kubectl exec -it -n landlord logger-64d95c7ddd-wp46c -- bash
bash-5.1#cat /bin/log_server
#!/bin/bash

nc -k -l -p 5566 |
  while read -r line; do
    if echo "${line}" | grep 'HISTORY'; then
      HOSTNAME=$(echo "${line}" | grep 'HISTORY' | awk -F'|' '{print $2}')
      USERNAME=$(echo "${line}" | grep 'HISTORY' | awk -F'|' '{print $3}')
      TIME=$(echo "${line}" | grep 'HISTORY' | awk -F'|' '{print $5}')
      COMMAND=$(echo "${line}" | grep 'HISTORY' | awk -F'|' '{print $6}')
      mysql -h "${DB_SERVER}.${NS}" -u "${DB_ROOT}" -p"${DB_ROOT_PW}" "${DB_NAME}" \
        -e "INSERT INTO history VALUES ('${HOSTNAME}','${USERNAME}','${TIME}','${COMMAND}')"
    fi

    if echo "${line}" | grep 'INOTIFY'; then
      HOSTNAME=$(echo "${line}" | grep 'INOTIFY' | awk -F'|' '{print $2}')
      TIME=$(echo "${line}" | grep 'INOTIFY' | awk -F'|' '{print $3}')
      EVENT=$(echo "${line}" | grep 'INOTIFY' | awk -F'|' '{print $4}')
      INPATH=$(echo "${line}" | grep 'INOTIFY' | awk -F'|' '{print $5}')
      mysql -h "${DB_SERVER}.${NS}" -u "${DB_ROOT}" -p"${DB_ROOT_PW}" "${DB_NAME}" \
        -e "INSERT INTO inotify VALUES ('${HOSTNAME}','${TIME}','${EVENT}','${INPATH}')"
    fi
  done

實作 Tenant

請設計一個 Container Image 名為 landlord.tenant，標籤為 1.0.0，並滿足以下功能

FROM quay.io/flysangel/image:alpine.sshd
安裝 mysql client
安裝 kubectl
複製 log_client 程式至 /bin
CMD ["/bin/bash", "-c", "/bin/log_client & /usr/sbin/sshd -D"]

將 Container Image 上傳至落地 Quay 在 K8S 使用 landlord.tenant 驗證 Pod 內是否有 kubectl 命令

$ nano Dockerfile.landlord.tenant
FROM quay.io/flysangel/image:alpine.sshd

RUN \
    apk update && \
    apk add mysql mysql-client

RUN \
    curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && \
    install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl && \
    rm -r kubectl

COPY bin/tenant/log_client /bin

CMD ["/bin/bash", "-c", "/bin/log_client & /usr/sbin/sshd -D"]

$ sudo podman build -t quay.k8s.org/quay/landlord.tenant:1.0.0 -f Dockerfile.landlord.tenant .

$ sudo podman images | grep tenant
quay.k8s.org/quay/landlord.tenant                         1.0.0             b4fc9e216ae3  30 seconds ago     285 MB

$ sudo podman push --tls-verify=false quay.k8s.org/quay/landlord.tenant:1.0.0

$ nano StatefulSet-landlord.tenant.yml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: tenant
  namespace: landlord
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tenant
  serviceName: tenant
  template:
    metadata:
      labels:
        app: tenant
    spec:
      containers:
      - env:
        - name: NS
          value: landlord
        - name: LOG_SERVER
          value: logger
        - name: KUSER_SERVER
          value: kuser
        - name: EXCLUDE
          value: .kube\|.bash_history
        image: quay.k8s.org/quay/landlord.tenant:1.0.0
        imagePullPolicy: Always
        name: tenant
        securityContext:
          capabilities:
            add:
            - AUDIT_WRITE
            - SYS_CHROOT
            - CAP_NET_RAW
        volumeMounts:
        - mountPath: /home/wk
          name: wk
      restartPolicy: Always
  volumeClaimTemplates:
  - metadata:
      name: wk
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: local-path

$ kubectl apply -f StatefulSet-landlord.tenant.yml

$ kubectl get pods -n landlord
NAME                       READY   STATUS    RESTARTS   AGE
gateway-6898d75c8f-5f2rc   1/1     Running   0          101m
logger-64d95c7ddd-wp46c    1/1     Running   0          73m
tenant-0                   1/1     Running   0          33s
tenant-1                   1/1     Running   0          27s
tenant-2                   1/1     Running   0          20s

$ kubectl exec -it -n landlord tenant-0 -- bash
bash-5.1# which kubectl
/usr/local/bin/kubectl

bash-5.1# exit
exit

$ kubectl exec -it -n landlord tenant-1 -- bash
bash-5.1# which kubectl
/usr/local/bin/kubectl

bash-5.1# exit
exit

$ kubectl exec -it -n landlord tenant-2 -- bash
bash-5.1# which kubectl
/usr/local/bin/kubectl

bash-5.1# exit
exit

New

檢視 Dockerfile.mariadb-v1.0.0

$ cat Dockerfile.mariadb-v1.0.0
FROM quay.io/flysangel/mariadb

COPY conf/mariadb/initdb/initdb.sql /docker-entrypoint-initdb.d

image 啟動時，會先執行 /docker-entrypoint-initdb.d 底下的 *.sql 檔
這個功能只有 docker hub 做出來的 mariadb 做的 image 才有

$ cat conf/mariadb/initdb/initdb.sql
CREATE TABLE IF NOT EXISTS history (
  Hostname VARCHAR(50),
  Username VARCHAR(50),
  Time VARCHAR(50),
  Command VARCHAR(255)
);
CREATE TABLE IF NOT EXISTS inotify (
  Hostname VARCHAR(50),
  Time VARCHAR(50),
  Event VARCHAR(20),
  Path VARCHAR(255)
);

檢視 Mariadb YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mariadb
  namespace: landlord
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - args:
        - --character-set-server=utf8mb4
        - --collation-server=utf8mb4_general_ci
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: root
        - name: MYSQL_DATABASE
          value: landlord
        - name: MYSQL_USER
          value: landlord
        - name: MYSQL_PASSWORD
          value: mymariadblandlord
        - name: TZ
          value: Asia/Taipei
        image: quay.io/flysangel/landlord:mariadb-v1.0.0
        imagePullPolicy: Always
        name: mariadb
        volumeMounts:
        - mountPath: /var/lib/mysql
          name: mariadb
      restartPolicy: Always
      volumes:
      - name: mariadb
        persistentVolumeClaim:
          claimName: mariadb

ENV 的變數也是只有 docker hub 做出來的 mariadb 做的 image 才有
參考資料 : https://hub.docker.com/_/mariadb

實作 Mariadb

$ sudo podman build -t $IP/myquay/landlord:mariadb-v1.0.0 -f Dockerfile.mariadb-v1.0.0

-f, --file pathname or URL ，pathname or URL of a Dockerfile

檢視 log_client

#!/bin/bash

#set -x # debug mode
set -a # var export

# create user
KUSER=$(hostname)
adduser -s /bin/bash -h /home/"${KUSER}" -G wheel -u 1001 -D "${KUSER}"
grep '^%wheel.*NOPASSWD.*ALL' /etc/sudoers >/dev/null || sed -i '/%wheel.*NOPASSWD/s/^# //' /etc/sudoers
echo "${KUSER}:${KUSER}" | chpasswd
if [ -d /run/wk ]; then
  ln -s /run/wk /home/"${KUSER}"/wk
else
  mkdir /home/"${KUSER}"/wk
fi
chown -R "${KUSER}":wheel /home/"${KUSER}"

# create kube config
kubectl create namespace "${KUSER}"
kubectl create serviceaccount "${KUSER}" -n "${KUSER}"
envsubst </bin/template/role.yaml | kubectl apply -f -
kubectl create rolebinding "${KUSER}" -n "${KUSER}" --role="${KUSER}" --serviceaccount="${KUSER}":"${KUSER}"
envsubst </bin/template/secret.yaml | kubectl apply -f -
CA=$(kubectl get secret "${KUSER}" -n "${KUSER}" -o jsonpath='{.data.ca\.crt}' | base64 -d)
TOKEN=$(kubectl get secret "${KUSER}" -n "${KUSER}" -o jsonpath='{.data.token}' | base64 -d)
serviceaccountpath='/run/secrets/kubernetes.io/serviceaccount'
umount ${serviceaccountpath}
echo "${CA}" | tee ${serviceaccountpath}/ca.crt
echo "${TOKEN}" | tee ${serviceaccountpath}/token
echo "${KUSER}" | tee ${serviceaccountpath}/namespace
echo "
export KUBERNETES_SERVICE_HOST=${KUBERNETES_SERVICE_HOST}
export KUBERNETES_SERVICE_PORT=${KUBERNETES_SERVICE_PORT}
" | tee -a /etc/profile

# history collect
sed -i '/HISTCONTROL\|HISTTIMEFORMAT\|PROMPT_COMMAND/d' /etc/profile
echo "
readonly HISTCONTROL=''
readonly HISTTIMEFORMAT='|%Y-%m-%d %H:%M:%S|'
readonly PROMPT_COMMAND='echo HISTORY:\|\$(hostname)\|\$(whoami)\|\$(history 1) | timeout 0.1 nc ${LOG_SERVER}.${NS} 5566; history -a; history -n'
cd ~/wk
" | tee -a /etc/profile

# filesystem collect
inotifywait -rmq /home \
  --format '|%T|%e|%w%f' \
  --timefmt '%Y-%m-%d %H:%M:%S' \
  --event 'modify,move,create,delete' | while read -r line; do
  echo "${line}" | grep -q "${EXCLUDE}" ||
    echo "INOTIFY:|$(hostname)${line}" | timeout 0.1 nc "${LOG_SERVER}.${NS}" 5566
done

set +a # var unexport

tags: `系統工程`

Kubernetes-RBAC-Context Kubernetes Deployment

Kubernetes Landlord

Kubernetes Landlord

部署 Landlord on K8S

問題討論

Landlord 移除

實作 Landlord

Landlord 目錄結構

實作 Kuser

練習

實作 Gateway

實作 Logger

實作 Tenant

New

檢視 Dockerfile.mariadb-v1.0.0

檢視 Mariadb YAML

實作 Mariadb

檢視 log_client

tags: 系統工程

tags: `系統工程`