Skip to content

antony@notes:~/kubernetes$ cat "Expose-Mariadb-on-TarokoK8s-with-Nginx-Ingress.md"

Expose Mariadb on TarokoK8s with Nginx Ingress

2024-06-23· kubernetes

Expose Mariadb on TarokoK8s with Nginx Ingress

PreRequest

  • 已完成安裝 TarokoK8s (Kind)
  • 已完成 Base Images 的部署
  • 已完成 設定叢集管理主機

1. 在 JuiceFS 上建立 Mariadb 

# 1. 透過 bigred 使用者 SSH 登入 Kube-Kadm Console
$ ssh localhost -p 22100

# 2. 建立 Mariadb
$ kubectl create -R -f ~/wulin/wk/mariadb/
deployment.apps/mariadb created
persistentvolumeclaim/mariadb created
service/mdb created

# 3. 檢查 Mariadb 的 Pod 和 Service 狀態是否正常
$ kubectl get pods,svc
NAME                           READY   STATUS    RESTARTS   AGE
pod/mariadb-64bfb5b98d-4zgbv   1/1     Running   0          18m

NAME                 TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/kubernetes   ClusterIP      10.96.0.1    <none>        443/TCP    5h15m
service/mdb          LoadBalancer   10.99.8.79   10.89.0.101   3306/TCP   18m

# 4. 檢查 PVC 和 PV 的狀態
$ kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS    REASON   AGE
persistentvolume/pvc-2c5d6e24-053a-4f0f-ba74-bf1a2070b1c3   10Gi       RWO            Delete           Bound    s3-system/redis   standard                 118m
persistentvolume/pvc-60f9bff2-e9af-491b-bb33-e8ab8f883659   10Gi       RWX            Retain           Bound    default/mariadb   juicefs-minio            97m

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/mariadb   Bound    pvc-60f9bff2-e9af-491b-bb33-e8ab8f883659   10Gi       RWX            juicefs-minio   97m

2. 設定 Nginx Ingress Controller Exposing TCP and UDP services

2.1. 先備知識

While the Kubernetes Ingress resource only officially supports routing external HTTP(s) traffic to services, ingress-nginx can be configured to receive external TCP/UDP traffic from non-HTTP protocols and route them to internal services using TCP/UDP port mappings that are specified within a ConfigMap.

To support this, the --tcp-services-configmap and --udp-services-configmap flags can be used to point to an existing config map where the key is the external port to use and the value indicates the service to expose using the format: <service port>:<namespace/service name>:[PROXY]:[PROXY]

更詳細資訊請參考以下官網連結 : Exposing TCP and UDP services - Ingress-Nginx Controller Docs

2.2. 建立 Configmap 

# 1. 撰寫 Configmap 的內容
$ cat <<EOF > ~/wulin/wk/mariadb/tcp-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  "3306": default/mdb:3306
EOF

# 2. 建立 Configmap
$ kubectl create -f ~/wulin/wk/mariadb/tcp-configmap.yaml

2.3. 修改 Nginx Ingress Controller 的 Service 

# 1. 編輯要修改的內容
$ cat <<EOF > ~/wulin/wk/mariadb/patch-file-nginx-ingres-controller-svc.yaml
spec:
  ports:
  - name: proxied-tcp-3306
    port: 3306
    protocol: TCP
    targetPort: 3306
EOF

# 2. 修改 ingress-nginx-controller Service
$ kubectl -n ingress-nginx patch service/ingress-nginx-controller --patch-file ~/wulin/wk/mariadb/patch-file-nginx-ingres-controller-svc.yaml
service/ingress-nginx-controller patched

# 3. 設定透過 Ingress-NGINX controller 指派一個 Static IP 給 Ingress
$ kubectl -n ingress-nginx patch svc ingress-nginx-controller --type='merge' -p '{"spec":{"externalIPs":["10.89.0.12"]}}'

# 檢查可以發現多出 3306 Port number，EXTERNAL-IP 也被設為 10.89.0.12
$ kubectl -n ingress-nginx get svc ingress-nginx-controller
NAME                       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
ingress-nginx-controller   NodePort   10.108.137.173   10.89.0.12    3306:31657/TCP,80:30929/TCP,443:32449/TCP   45m

2.4. 將 Configmap 新增至 ingress controller’s deployment 的 args。 

# 1. 編輯要修改的內容
$ cat <<EOF > ~/wulin/wk/mariadb/patch-file-nginx-ingres-controller.yaml
spec:
  template:
    spec:
      containers:
      - name: controller
        args:
        - /nginx-ingress-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=\$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --watch-ingress-without-class=true
        - --enable-metrics=false
        ## 註解以下一行，此設定會導致 Ingress 建立出來的 Address 會變成 localhost
        #### - --publish-status-address=localhost
        ## 新增以下這行設定
        - --publish-service=\$(POD_NAMESPACE)/ingress-nginx-controller
        - --tcp-services-configmap=ingress-nginx/tcp-services
EOF

# 2. 修改 ingress-nginx-controller Service
$ kubectl -n ingress-nginx patch deploy ingress-nginx-controller --patch-file ~/wulin/wk/mariadb/patch-file-nginx-ingres-controller.yaml

# 3. 重啟 ingress-nginx-controller 的 Pods，用以確保 pod 使用最新的設定檔被部屬
$ kubectl -n ingress-nginx rollout restart deploy ingress-nginx-controller

# 4. 檢查 pods 狀態是否 Running
$ kubectl -n ingress-nginx get pods
NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-6dcf57c5f7-6pb75   1/1     Running   0          51s

補充 : ingres-controller 的 arguments 解釋

  • --publish-service When used together with update-status, the controller mirrors the address of this service’s endpoints to the load-balancer status of all Ingress objects it satisfies.

  • --tcp-services-configmap Name of the ConfigMap containing the definition of the TCP services to expose. The key in the map indicates the external port to be used. The value is a reference to a Service in the form namespace/name:port, where port can either be a port number or name. TCP ports 80 and 443 are reserved by the controller for servicing HTTP traffic.

  • --update-status Update the load-balancer status of Ingress objects this controller satisfies. Requires setting the publish-service parameter to a valid Service reference. (default true)

  • --publish-status-address Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter. 這個參數設定的 IP 會覆蓋掉 Ingress 的 Address，Kind 預設裝好的 Ingress Controller 會是 localhost。

  • 更多詳細的解釋在以下連結 : cli-arguments - ingress-nginx Docs

3. 建立 Ingress 

# 1. 編輯 Ingress 的內容
$ cat <<EOF > ~/wulin/wk/mariadb/ing-mariadb.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    app: mysql
  name: mariadb-ingress
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: mysql.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mdb
            port:
              number: 3306
EOF

# 2. 建立 mariadb-ingress
$ kubectl create -f ~/wulin/wk/mariadb/ing-mariadb.yaml

# 3. 檢查 mariadb-ingress 狀態
$ kubectl get ing
NAME              CLASS   HOSTS               ADDRESS      PORTS   AGE
mariadb-ingress   nginx   mysql.example.com   10.89.0.12   80      17m

# 4. 離開 Kube-Kadm Console
$ exit

# 5. 新增本機名稱解析
$ echo '10.89.0.12 mysql.example.com' | sudo tee -a /etc/hosts
10.89.0.12 mysql.example.com

4. 在 K8s 叢集外測試連接 Mariadb 

# 以下操作步驟都是由 M109 主機的 bigred 使用者執行
# 1. 安裝 mysql client
$ sudo apk -U add mysql-client

# 2. 確認 mysql client 版號
$ mysql --version
mysql  Ver 15.1 Distrib 10.11.8-MariaDB, for Linux (x86_64) using readline 5.1

# 3. 測試使用 Ingress 的域名登入 Mariadb
$ mysql -u bigred -pbigred -h mysql.example.com
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 11.4.2-MariaDB-ubu2404 mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

# 4. 顯示所有 DataBases
MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| dbtest             |
| information_schema |
+--------------------+
2 rows in set (0.002 sec)

# 5. 離開
MariaDB [(none)]> \q
Bye

5. TrobleShooting: 如果無法透過 Ingress 的域名登入 Mariadb 的故障排除步驟

5.1. Kind Cluster 

# m 變數為 control-plane 的 Container name
$ m=c24-control-plane

# mpid 為 control-plane 的 Container 在 Host 主機的 pid
$ mpid=$(sudo crun list | grep $(sudo podman ps -a --filter name=$m --format {{.ID}}) | cut -d " " -f 2)

# 以下操作步驟都是由 M109 主機的 bigred 使用者執行
# 1. 測試 pod 是否能正常連接
## 1.1. 得到 pod ip
$ kubectl get pods -o wide
NAME                       READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
mariadb-5b764d9495-nh66h   1/1     Running   0          77m   10.244.2.8   c24-worker   <none>           <none>

## 1.2. nc 測試
$ sudo nsenter -t $mpid -n nc -w 1 -v 10.103.39.45 3306; echo $?
10.103.39.45 (10.103.39.45:3306) open
Z
11.4.2-MariaDB-ubu2404<`PMt}:G��-��5qZRh9:?o0>mysql_native_password0

## 2.1. 得到 service ip
$ kubectl get svc mdb
NAME   TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
mdb    LoadBalancer   10.103.39.45   10.89.0.100   3306/TCP   78m

## 2.2. nc 測試
$  sudo nsenter -t $mpid -n nc -w 1 -v 10.103.39.45 3306; echo $?
10.103.39.45 (10.103.39.45:3306) open
Z
11.4.2-MariaDB-ubu2404EvM`xZ<��-��]?bPLiL}Vs`imysql_native_password0

## 3.1. 檢查 Ingress Controller Log 確認是否有錯誤訊息
$ kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | less
Deploy RMT Server on K3S and synchronize it with Disconnected SUMAKubernetes Deployments: Rolling vs Canary vs Blue-Green