antony@notes:~/kubernetes$ cat "Kubernetes-Resource-CPU-Memory.md"

Kubernetes-Resource-CPU-Memory

2022-09-18· kubernetes ·系統工程

Kubernetes-Resource-CPU-Memory

:::warning

:::spoiler 目錄

[TOC]

:::

Assigning Pods to Nodes

# 檢視所有 node 的 labels
$ kubectl get nodes --show-labels

# 編輯 nodeselect 的 yaml 檔
$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: p1
spec:
  containers:
  - name: c1
    image: quay.io/cloudwalker/busybox
    imagePullPolicy: Never
    tty: true
  nodeSelector:
    kubernetes.io/hostname : m1 '> ~/wulin/yaml/nodeselect.yaml 
    
# apply 它
$ kubectl apply -f  ~/wulin/yaml/nodeselect.yaml 
pod/p1 created

# 檢查是否跑在 m1 這台指定的 node 上
$ kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
p1     1/1     Running   0          35s   10.233.2.2           m1   <none>           <none>

# 刪除 p1 pod
$ kubectl delete pods p1
pod "p1" deleted

# 編輯 depnode 的 yaml 檔
$ nano ~/wulin/yaml/depnode.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: depnode
  labels:
    app: depnode
spec:
  replicas: 4
  selector:
    matchLabels:
      app: depnode
  template:
    metadata:
      labels:
        app: depnode
    spec:
      containers:
      - name: alp
        image: quay.io/cloudwalker/alpine
        tty: true

$ ka  -f  ~/wulin/yaml/depnode.yaml

$ kg pods -o wide --selector app=depnode
NAME                       READY   STATUS    RESTARTS   AGE    IP            NODE   NOMINATED NODE   READINESS GATES
depnode-7789fd7db7-2h6t5   1/1     Running   0       5m4s   10.244.2.12   w2     <none>           <none>
depnode-7789fd7db7-2vc6s   1/1     Running   0       5m4s   10.244.0.10   m1     <none>           <none>
depnode-7789fd7db7-vl6g2   1/1     Running   0        5m4s   10.244.1.11   w1     <none>           <none>
depnode-7789fd7db7-z4zxr   1/1     Running   0        5m4s   10.244.1.12   w1     <none>           <none>

$ kd -f ~/wulin/yaml/depnode.yaml

$ nano ~/wulin/yaml/depnode.yaml
apiVersion: apps/v1
kind: Deployment
.......
    spec:
      containers:
      - name: alp
        image: quay.io/cloudwalker/alpine
        tty: true
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - depnode
            topologyKey: kubernetes.io/hostname

$ ka  -f  ~/wulin/yaml/depnode.yaml

$ kg pods -o wide --selector app=depnode
NAME                       READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
depnode-7c65d66984-6cplq   1/1     Running   0          25s   10.244.2.13   w2       <none>           <none>
depnode-7c65d66984-hgbs5   1/1     Running   0          25s   10.244.0.11   m1       <none>           <none>
depnode-7c65d66984-pb9f9   0/1     Pending   0          25s   <none>        <none>   <none>           <none>
depnode-7c65d66984-qj7g4   1/1     Running   0          25s   10.244.1.13   w1       <none>           <none>

$ kd -f ~/wulin/yaml/depnode.yaml

Metrics Server

安裝 Metrics Server

$ wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

$ nano components.yaml
..........
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        - --kubelet-insecure-tls
        image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent

$ kubectl apply -f components.yaml

- --kubelet-insecure-tls ，所有 K8S 在網路上運作傳輸的封包和資料都會進行加密

檢視 Worker Node 資源使用狀態

$ kubectl top nodes
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
m1       84m             4%        603Mi                7%
w1       26m             1%        496Mi                6%
w2       27m             1%        459Mi                5%

CPU 的單位 : milliCPU
1 core = 1000m
m1 node 比較忙的原因是因為它裡面跑的 pod 都是在做系統維運的

檢視 pod 的資源使用狀態

$ kubectl top pods -n bobo
NAME         CPU(cores)   MEMORY(bytes)
bobo.fbs     0m           3Mi
bobo.goweb   1m           1Mi
bobo.mysql   7m           289Mi

Rsource Requests & Limits

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: podres
  namespace: default
spec:
  containers:
  - name: podres
    image: quay.io/cloudwalker/alpine
    command: ["/usr/bin/yes"]
    resources:
       requests:
          cpu: 100m
          memory: 640M
       limits:
          cpu: 500m'> ~/wulin/yaml/resource.yaml 

$ kubectl apply -f  ~/wulin/yaml/resource.yaml

Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler is implemented as a control loop, with a period controlled by the controller manager’s –horizontal-pod-autoscaler-sync-period flag (with a default value of 15 seconds)

HPA 一定會透過 Metrics Server 來得到 Deployment 的 pod 現在使用資源的狀態

建立 Horizontal Pod Autoscaler

$ mkdir ~/wulin/hpa; cd ~/wulin/hpa

$ nano hpa-dep.yaml

編輯 yaml 檔

kind: Deployment
apiVersion: apps/v1
metadata:
  name: hpa-dep
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hpa.pod
  template:
    metadata:
      labels:
        app: hpa.pod
    spec:
      containers:
      - name: alp
        image: quay.io/cloudwalker/alpine
        imagePullPolicy: IfNotPresent
        tty: true
        resources:
          limits:
            cpu: 1

單位 m 指的是 milli-cores，每 1000m = 1 vCore
設定可以用 m 或分數，例如：
設定 0.5 = 500m
設定 300m = 0.3
這裡設定的 1 代表的是 1000m，代表 1 個 pod 只能使用一顆 vCore 的運算資源

編輯 yaml 檔

$ nano hpa-sp.yml

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: hpa-sp
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hpa-dep
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 30

scaleTargetRef:，要監控的對象
minReplicas: 2，擴充的 pod 最少會有兩個
maxReplicas: 5，擴充的 pod 最多只能有五個
targetCPUUtilizationPercentage: 30，設定 30% 代表 HPA 將會維持 Pod 的平均 CPU 使用率為 30 %，只要超出 30%，即會自動擴展

建立與檢測 Horizontal Pod Autoscaler

$ kubectl apply -f  .
deployment.apps/hpa-dep created
horizontalpodautoscaler.autoscaling/hpa-sp created

$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
hpa-dep-765c997fc8-6vbmd   1/1     Running   0          22s
hpa-dep-765c997fc8-g9kgd   1/1     Running   0          22

$ kubectl get hpa --watch
NAME     REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-sp   Deployment/hpa-dep   0%/30%    2         5         2          4m47s
hpa-sp   Deployment/hpa-dep   0%/30%    2         5         2          5m16s

# 開啟新的 命令提示字元 視窗
$ ssh bigred@<m1 node ip>

$ kg pod
NAME                       READY   STATUS    RESTARTS   AGE
hpa-dep-765c997fc8-6vbmd   1/1     Running   0          7m21s
hpa-dep-765c997fc8-g9kgd   1/1     Running   0          7m21s

$ kubectl exec hpa-dep-765c997fc8-6vbmd -- timeout 240 yes >/dev/null &
$ kubectl exec hpa-dep-765c997fc8-g9kgd -- timeout 240 yes >/dev/null &

利用 yes 命令會霸佔 CPU 的特性，來測試 HPA

$ kubectl get hpa --watch
NAME     REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-sp   Deployment/hpa-dep   0%/30%    2         5         2          25m
hpa-sp   Deployment/hpa-dep   33%/30%   2         5         2          25m
hpa-sp   Deployment/hpa-dep   56%/30%   2         5         3          26m
hpa-sp   Deployment/hpa-dep   56%/30%   2         5         4          26m
hpa-sp   Deployment/hpa-dep   55%/30%   2         5         4          26m
hpa-sp   Deployment/hpa-dep   56%/30%   2         5         4          27m
hpa-sp   Deployment/hpa-dep   56%/30%   2         5         4          27m
hpa-sp   Deployment/hpa-dep   56%/30%   2         5         4          27m
hpa-sp   Deployment/hpa-dep   57%/30%   2         5         4          27m
hpa-sp   Deployment/hpa-dep   57%/30%   2         5         4          28m
hpa-sp   Deployment/hpa-dep   56%/30%   2         5         4          28m
hpa-sp   Deployment/hpa-dep   57%/30%   2         5         4          28m

可以看到 REPLICAS 擴充為 4

檢查 pod 有沒有 running

$ kubectl get pods
NAME                       READY   STATUS    RESTARTS   AGE
hpa-dep-765c997fc8-6vbmd   1/1     Running   0          26m
hpa-dep-765c997fc8-7rdcm   0/1     Pending   0          38s
hpa-dep-765c997fc8-g9kgd   1/1     Running   0          26m
hpa-dep-765c997fc8-wvvm2   0/1     Pending   0          23s

為何擴充的 pod 都是 pending ? 用 kubectl describe 看一下

$ kubectl describe pods hpa-dep-765c997fc8-7rdcm
Name:           hpa-dep-765c997fc8-7rdcm
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  53s   default-scheduler  0/5 nodes are available: 2 Insufficient cpu, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 2 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.

錯誤訊息說明，已經沒有 node 上有足夠的 CPU 讓 pod run，因為我們是用 VM 來 run K8S ，設定給 VM 用的 CPU 只有 2 Core
雖然 K8S 的 HPA 能自動幫我們橫向擴充 pod ，但是硬體資源 (CPU) 不夠力，擴充出來的 pod 也沒辦法 run，所以要特別注意。

可以根據 kubectl top來檢視 node 使用資源狀態

$ k top nodes
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
m1     1102m        55%    869Mi           11%
m2     270m         13%    803Mi           10%
m3     240m         12%    695Mi           8%
w1     1372m        68%    265Mi           3%
w2     1396m        69%    644Mi           8%

可以看到 w1 和 w2 node 的 CPU 使用率都飆到將近 70 % ，已經沒有足夠的運算資源，再讓多的 pod run 了，所以才會顯示 pending 的狀態

刪除 deployment 和 hpa

$ kd -f hpa-dep.yaml
$ kd -f hpa-sp.yml

修改 pod 可以使用的 CPU 運作資源

$ nano hpa-dep.yaml

...
spec:
  ...
  template:
    ...
    spec:
      containers:
        ...
        resources:
          limits:
            cpu: 400m

將最後的 CPU 設成 400m

$ nano hpa-sp.yml

spec:
...
  targetCPUUtilizationPercentage: 10

將 targetCPUUtilizationPercentage 的值改成 10

再次產生物件

$ ka -f .
deployment.apps/hpa-dep created
horizontalpodautoscaler.autoscaling/hpa-sp created

$ kg hpa --watch
NAME     REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-sp   Deployment/hpa-dep   0%/10%    2         5         2          47s

在另一個終端機，檢視

$ kg pod
NAME                       READY   STATUS    RESTARTS   AGE
hpa-dep-6f5bd9dd57-ggs9b   1/1     Running   0          56s
hpa-dep-6f5bd9dd57-kmxmb   1/1     Running   0          56s

讓 pod 飆 yes

$ kubectl exec hpa-dep-6f5bd9dd57-ggs9b -- timeout 240 yes >/dev/null &
$ kubectl exec hpa-dep-6f5bd9dd57-kmxmb -- timeout 240 yes >/dev/null &

回原來的終端機

$ kg hpa --watch
NAME     REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
hpa-sp   Deployment/hpa-dep   0%/10%    2         5         2          47s
hpa-sp   Deployment/hpa-dep   3%/10%    2         5         2          3m15s
hpa-sp   Deployment/hpa-dep   69%/10%   2         5         2          3m31s
hpa-sp   Deployment/hpa-dep   100%/10%   2         5         4          3m46s
hpa-sp   Deployment/hpa-dep   100%/10%   2         5         5          4m1s
hpa-sp   Deployment/hpa-dep   50%/10%    2         5         5          4m16s
hpa-sp   Deployment/hpa-dep   40%/10%    2         5         5          4m31s

可以看到 REPLICAS 擴充為 4

再到另一個終端機，檢視 pod 有無 runninf

$ kg po -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP           NODE   NOMINATED NODE   READINESS GATES
hpa-dep-6f5bd9dd57-7tkdp   1/1     Running   0          24s     10.244.3.7   w1     <none>           <none>
hpa-dep-6f5bd9dd57-cntd5   1/1     Running   0          9s      10.244.3.8   w1     <none>           <none>
hpa-dep-6f5bd9dd57-ggs9b   1/1     Running   0          3m55s   10.244.4.6   w2     <none>           <none>
hpa-dep-6f5bd9dd57-kmxmb   1/1     Running   0          3m55s   10.244.3.6   w1     <none>           <none>
hpa-dep-6f5bd9dd57-qlgr6   1/1     Running   0          24s     10.244.4.7   w2     <none>           <none>

通通健康的在跑，因為我們設定一個 pod 只能 run 400m ，且目前單一 node 總共有 2000m 可以用，所以當然可以讓擴充出來的 pod 快樂的 running

檢視 node 使用運作資源

$ k top nodes
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
m1     867m         43%    897Mi           11%
m2     241m         12%    807Mi           10%
m3     229m         11%    704Mi           8%
w1     1068m        53%    266Mi           3%
w2     1084m        54%    650Mi           8%

結束測試，刪除物件

$ kd -f .

Managing the Container Life Cycle

Liveness and Readiness Probes

(2022/09/05 10:35)

The kubelet uses liveness probes to know when to restart a Container. For example, liveness probes could catch a deadlock, where an application is running, but unable to make progress. Restarting a Container in such a state can help to make the application more available despite bugs.
The kubelet uses readiness probes to know when a Container is ready to start accepting traffic. A Pod is considered ready when all of its Containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.

共有三種檢測方式，分別是

ExecAction: Executes a specified command inside the Container. The diagnostic is considered successful if the command exits with a status code of 0.
TCPSocketAction: Performs a TCP check against the Container’s IP address on a specified port. The diagnostic is considered successful if the port is open.
- 像是 Linux 命令的 nc ，檢查 IP 的 port 有沒有開
HTTPGetAction: Performs an HTTP Get request against the Container’s IP address on a specified port and path. The diagnostic is considered successful if the response has a status code greater than or equal to 200 and less than 400.
- 像是 Linux 命令的 curl ，

Each probe has one of three results:

Success: The Container passed the diagnostic.
Failure: The Container failed the diagnostic.
Unknown: The diagnostic failed, so no action should be taken.

在老師的 K8S Cluster 中，探針會透過 Kubelet 和 CRI-O 監控 Container 的一支程式叫 conmon 來講話，看 Container 的狀態
如果遇到探針無法執行，需檢查 conmon 的版本 conmon --version 是不是 2.1.2，再 K8S 1.23 以後的版本，須將 conmon 升級至 2.1.4 版本
- 升級命令 : sudo apk upgrade conmon --no-cache --update-cache --allow-untrusted --repository http://dl-cdn.alpinelinux.org/alpine/edge/community

Liveness Probe - Exec

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    command: [/bin/sh]
    args:
    - -c
    - touch /tmp/healthy; sleep 20; rm -rf /tmp/healthy; sleep 300
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5 '>  ~/wulin/yaml/exec-liveness.yml

Container 跑的命令是先建立 /tmp/healthy 這個空檔案，再睡 20 秒，刪除檔案，再睡 300 秒
livenessProbe ，liveness 探針
- exec ，透過進入 Container 執行命令來健康檢查
  - 執行的命令是看 /tmp/healthy 檔案的內容
- initialDelaySeconds: 5，Container init 後，等 5 秒後，開始健康檢查
- periodSeconds: 5，每 5 秒檢查一次

# 佈署 exec-liveness.yml
$ kubectl apply -f  ~/wulin/yaml/exec-liveness.yml

# 檢視 exec-liveness Pod 的 Event
$ kubectl describe pod liveness-exec
.......
  Normal   Pulled     58s                kubelet            Successfully pulled image "k8s.gcr.io/busybox" in 7.253319528s
  Normal   Created    57s                kubelet            Created container liveness
  Normal   Started    56s                kubelet            Started container liveness
  Warning  Unhealthy  25s (x3 over 35s)  kubelet            Liveness probe failed: cat: cant open '/tmp/healthy': No such file or directory
  Normal   Killing    25s                kubelet            Container liveness failed liveness probe, will be restarted

$ kubectl get pod
NAME              READY   STATUS    RESTARTS   AGE
liveness-exec   1/1     Running    2          2m57s

移除 Pod
$ kubectl delete -f  ~/wulin/yaml/exec-liveness.yml

K8S Namespace LimitRange

$ kubectl create namespace memlimit

$ echo 'apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
spec:
  limits:
  - default:
      memory: 512Mi
    defaultRequest:
      memory: 256Mi
    max:
      memory: 2Gi
    min:
      memory: 256Mi
    type: Container  '> ~/wulin/yaml/limitrange.yaml

$ kubectl apply  -f  ~/wulin/yaml/limitrange.yaml -n memlimit

default.memory，
LimitRange ，會針對一個 POD 產生限制, 並不是限制所有 POD 的總和
注意 ! 如果 Namespace 在 LimitRange 產生之前的物件，是不會受到

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: memlimit-pod
spec:
  containers:
  - name: memlimit-pod-nginx
    image: k8s.gcr.io/nginx ' > ~/wulin/yaml/memlimit-pod.yaml

$ kubectl apply -f ~/wulin/yaml/memlimit-pod.yaml -n memlimit
pod/memlimit-pod created

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: memlimit-pod1
spec:
  containers:
  - name: memlimit-pod-nginx
    image: k8s.gcr.io/nginx
    resources:
      limits:
        memory: "2Gi"  ' > ~/wulin/yaml/memlimit-pod1.yaml

# 以下命令會執行成功
$ kubectl apply -f  ~/wulin/yaml/memlimit-pod1.yaml -n memlimit

K8S Namespace Resource Quota

編輯 yaml 檔

$ nano ~/wulin/yaml/nsrslimit.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: mynsrs
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: mynsrs
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 256Mi
    requests.nvidia.com/gpu: 1
    limits.cpu: "2"
    limits.memory: 1Gi
    limits.nvidia.com/gpu: 2
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: object-quota
  namespace: mynsrs
spec:
  hard:
    configmaps: "2"
    persistentvolumeclaims: "2"
    replicationcontrollers: "2"
    secrets: "10"
    services: "10"
    services.loadbalancers: "2"

yaml 檔中，如果要宣告多個物件，物件跟物間中間用 --- 區隔
在 ResourceQuota 中，如果有宣告建立物件的數量限制，代表該物件只能被建立幾個，未宣告的物件則不受數量限制。

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: nsrs-pod
spec:
  containers:
  - name: nsrs-pod-nginx
    image: k8s.gcr.io/nginx ' > ~/wulin/yaml/nsrs-pod.yaml

$ kubectl apply -f ~/wulin/yaml/nsrs-pod.yaml -n mynsrs
Error from server (Forbidden): error when creating "nsrs-pod.yaml": pods "nsrs-pod" is forbidden: failed quota: compute-quota: must specify limits.cpu,limits.memory,requests.cpu,requests.memory

當一個 Namespace 有分配 ResourceQuota 和對應的 Namespace 時，所有在該 Namespace 內使用元件時必須都要詳細指明用量，不然不給用！

重新設定運算資源

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: nsrs-pod
spec:
  containers:
  - name: nsrs-pod-nginx
    image: k8s.gcr.io/nginx 
    resources:
      requests:
         cpu: 1.0
         memory: 256Mi
      limits:
         cpu: 2.0
         memory: 512Mi ' > ~/wulin/yaml/nsrs-pod.yaml

$ kubectl apply -f ~/wulin/yaml/nsrs-pod.yaml -n mynsrs

$ kg po -n mynsrs
NAME       READY   STATUS    RESTARTS   AGE
nsrs-pod   1/1     Running   0          93s

$ kg resourcequotas -n mynsrs
NAME            AGE   REQUEST                                                                                                                                 LIMIT
compute-quota   10m   requests.cpu: 1/1, requests.memory: 256Mi/256Mi, requests.nvidia.com/gpu: 0/1                                                           limits.cpu: 2/2, limits.memory: 512Mi/1Gi, limits.nvidia.com/gpu: 0/2
object-quota    10m   configmaps: 1/2, persistentvolumeclaims: 0/2, replicationcontrollers: 0/2, secrets: 0/10, services: 0/10, services.loadbalancers: 0/2

在建立同樣運算資源的 pod 一次

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: nsrs-pod1
spec:
  containers:
  - name: nsrs-pod-nginx
    image: k8s.gcr.io/nginx 
    resources:
      requests:
         cpu: 1.0
         memory: 256Mi
      limits:
         cpu: 2.0
         memory: 512Mi ' > ~/wulin/yaml/nsrs-pod1.yaml

$ kubectl apply -f ~/wulin/yaml/nsrs-pod1.yaml -n mynsrs
Error from server (Forbidden): error when creating "nsrs-pod1.yaml": pods "nsrs-pod1" is forbidden: exceeded quota: compute-quota, requested: limits.cpu=2,requests.cpu=1,requests.memory=256Mi, used: limits.cpu=2,requests.cpu=1,requests.memory=256Mi, limited: limits.cpu=2,requests.cpu=1,requests.memory=256Mi

CKA 2022 考題 Kubernetes-RBAC-Context