antony@notes:~/kubernetes$ cat "CKS.md"

CKS

2025-04-09· kubernetes

CKS

Preface

本篇文章為 2024 更新後的 CKS 真題，包含題目模擬與題目解答

可以透過點擊以下目錄，選擇想看的內容，跳轉至特定章節

:::warning

:::spoiler 目錄

[TOC]

:::

Question #1: kube-bench 修復不安全項目

Context:

You must resolve issues that a CIS Benchmark tool found for the kubeadm provisioned cluster.

Task

Fix all issues via configuration and restart the affected components to ensure the new settings take effect.

Fix all of the following violations that were found against the kubelet:

:::danger

2.1.2 Ensure that the anonymous-auth argument is set to false (FAIL)
2.1.3 Ensure that the --authorization-mode argument is not set to AlwaysAllow (FAIL)

:::

:::info

Use Webhook authentication/authorization where possible.

:::

Fix all of the following violations that were found against etcd:

:::danger

2.2 Ensure that the --client-cert-auth argument is set to true (FAIL)

:::

Prepare Env

# 1. 安裝必要套件
$ apt update; apt install -y nano \
  yq \
  dnsutils

# 2. 安裝 kube-bench
$ curl -sL $(curl -sL https://api.github.com/repos/aquasecurity/kube-bench/releases/latest | jq -r .assets[].browser_download_url | grep 'linux_amd64.deb') -o kube-bench.deb
$ dpkg -i kube-bench.deb
$ rm -r kube-bench.deb

# 3. etcd
$ cp /etc/kubernetes/manifests/etcd.yaml .
$ sed -i 's|--client-cert-auth=true|--client-cert-auth=false|g' /etc/kubernetes/manifests/etcd.yaml

# 4. kubelet
$ cp /var/lib/kubelet/config.yaml .
$ yq -iy '.authentication.anonymous.enabled = true' /var/lib/kubelet/config.yaml
$ yq -iy '.authentication.webhook.enabled = false' /var/lib/kubelet/config.yaml
$ yq -iy '.authorization.mode = "AlwaysAllow"' /var/lib/kubelet/config.yaml
$ systemctl daemon-reload
$ systemctl restart kubelet.service

Answer

# 確認 kubelet 的參數需要修改
$ kube-bench run -s node
...
[FAIL] 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[FAIL] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
...

# 備份 kubelet 設定檔
$ cp /var/lib/kubelet/config.yaml ./kubelet-config.yaml.bk

# 修改 kubelet 設定檔
$ nano /var/lib/kubelet/config.yaml
...
authentication:
  anonymous:
    enabled: false    <--- 這裡改 false
  webhook:
    cacheTTL: 0s
    enabled: true     <--- 這裡改 true
...
authorization:
  mode: Webhook       <--- 這裡改 Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
...

# 重新加载 systemd 的設定文件，使得 systemd 能够讀取最新的設定資訊。
$ systemctl daemon-reload

# 重啟 kubelet 服務
$ systemctl restart kubelet

# 確認 ETCD 的參數需要修改
$ kube-bench run -s etcd
...
[FAIL] 2.2 Ensure that the --client-cert-auth argument is set to true (Automated)
...

# 備份 etcd 設定檔
$ cp /etc/kubernetes/manifests/etcd.yaml ./etcd.yaml.bk

# 修改 etcd 設定檔
$ nano /etc/kubernetes/manifests/etcd.yaml
...
spec:
  containers:
  - command:
    - etcd
    ...
    - --client-cert-auth=true   <--- 這裡改 true
    
# 檢查 etcd container 的狀態，確認建立時間是新的，且狀態是 Running
$ crictl ps --name etcd
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
ce981115becf6       3861cfcd7c04c       2 minutes ago       Running             etcd                0                   747918759d292       etcd-c30-control-plane

# 檢查 ETCD Pods 狀態
$ kubectl -n kube-system get pods
NAME                                        READY   STATUS    RESTARTS         AGE
etcd-c30-control-plane                      1/1     Running   0                2m17s
...

# 檢查 kubelet 是否修改正確
$ kube-bench run -s node
[PASS] 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated)
[PASS] 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)
...

# 檢查 ETCD 是否修改正確
$ kube-bench run -s etcd
[PASS] 2.2 Ensure that the --client-cert-auth argument is set to true (Automated)
...

Question #2: 啟用 API server 認證

Context

For testing purposes, the kubeadm provisioned cluster’s API server was configured to allow unauthenticated and unauthorized access.

Task

First, secure the cluster’s API server by configuring it as follows:

Forbid anonymous authentication
Use authorization mode Node,RBAC
Use admission controller NodeRestriction

:::warning

kubectl is configured to use unauthenticated and unauthorized access. You do not have to change it, but be aware that kubectl will stop working once you have secured the cluster.

:::

:::info

You can use the cluster’s original kubectl configuration file located at /etc/kubernetes/admin.conf to access the secured cluster.

:::

Next, to clean up, remove the ClusterRoleBinding system:anonymous.

Prepare Env

# 建立 anonymous ClusterRoleBinding
$ echo 'apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:anonymous
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous' | kubectl apply -f -

# 修改 kube-apiserver
$ cp /etc/kubernetes/manifests/kube-apiserver.yaml .
$ sed -i '/- --enable-admission-plugins=NodeRestriction/d' /etc/kubernetes/manifests/kube-apiserver.yaml
$ sed -i 's|--authorization-mode=Node,RBAC|--authorization-mode=AlwaysAllow\n    - --anonymous-auth=true|g' /etc/kubernetes/manifests/kube-apiserver.yaml

Answer

官網文件收尋關鍵字 : kube-apiserver 找到這篇文章 : kube-apiserver | Kubernetes 按 Ctrl + F 搜尋 : -admission

# 備份 kube-apiserver 設定檔
$ cp /etc/kubernetes/manifests/kube-apiserver.yaml ./kube-apiserver.yaml.bk

# 修改 kube-apiserver 設定檔
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
  containers:
  - command:
    ...
    - --authorization-mode=Node,RBAC               <-- 這裡改 Node,RBAC (逗號後面不能空格)
    - --anonymous-auth=false                       <-- 這裡改 false (實際考試這個參數會藏在所有參數的最底下)
    - --enable-admission-plugins=NodeRestriction   <-- 到官網文件複製此參數，並新增此行設定 (如果題目原本就有此參數，但值不是題目要求的，就直接刪除該值，換成跟題目一模一樣的 NodeRestriction)

# 重啟 kubelet service
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet.service

# 修改後，確認 kube-apiserver Container 的狀態
$ crictl ps --name kube-apiserver
CONTAINER           IMAGE               CREATED              STATE               NAME                ATTEMPT             POD ID              POD
a6b389f7f3c6f       7f6c51674d5ef       About a minute ago   Running             kube-apiserver      0                   f09b8f9c8a33f       kube-apiserver-c30-control-plane

# 確認 kube-apiserver Pods 的狀態
$ kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system get pods
NAME                                        READY   STATUS             RESTARTS         AGE
kube-apiserver-c30-control-plane            1/1     Running            0                3m11s

# 刪除 system:anonymous ClusterRoleBinding
$ kubectl --kubeconfig=/etc/kubernetes/admin.conf delete clusterrolebinding system:anonymous
clusterrolebinding.rbac.authorization.k8s.io "system:anonymous" deleted

Question #3: ImagePolicyWebhook Container images 掃描

Context

You must fully integrate a container image scanner into the kubeadm provisioned cluster.

Task

Given an incomplete configuration located at /etc/kubernetes/bouncer and a functional container image scanner with an HTTPS endpoint at https://smooth-yak.local/image_policy, perform the following tasks to implement a validating admission controller:

First, re-configure the API server to enable all admission plugin(s) to support the provided AdmissionConfiguration.

Next, re-configure the ImagePolicyWebhook configuration to deny images on backend failure.

Next, Complete the backend configuration to point to the container image scanner’s endpoint at https://smooth-yak.local/image_policy.

Finally, to test the configuration, deploy the test resource defined in ~/vulnerable.yaml, which is using an image that should be denied.

You may delete and re-create the resource as often as needed.

:::info

The container image scanner’s log file is located at /var/log/nginx/access_log.

:::

Prepare ENV

# Create a Service
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: imagepolicywebhook
  name: image-bouncer-webhook
spec:
  ports:
  - name: "443"
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: imagepolicywebhook
  type: ClusterIP
EOF

# We create an environment variable for the IP to use it later.
$ export SERVICE_IP=$(kubectl get svc -o jsonpath='{.items[0].spec.clusterIP}')

# Create Server Certificates
$ mkdir ~/ssl; cd ~/ssl
$ openssl genrsa -out webhook-server.key 2048
$ openssl req -new -key webhook-server.key -subj "/CN=system:node:imagepolicywebhook/O=system:nodes" -addext "subjectAltName = DNS:image-bouncer-webhook.default.svc.cluster.local,DNS:image-bouncer-webhook.default.svc,DNS:image-bouncer-webhook.default.pod.cluster.local,IP:$SERVICE_IP" -out webhook-server.csr

# To use the csr in a Kubernetes manifest, we need to change the encoding to base64 and export it as an environment variable.
$ export SIGNING_REQUEST=$(cat webhook-server.csr | base64 | tr -d "\n")

# We then send a CertificateSigningRequest manifest to the kube-apiserver to sign our certificate. The certificate usages in this manifest are the usages that the signer accept to sign.
$ cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: webhook-server
spec:
  request: $SIGNING_REQUEST
  signerName: kubernetes.io/kubelet-serving
  expirationSeconds: 864000  # ten days
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# This signing request is now ready to be approved. After approving it, we can write the signed certificate to a file.
$ kubectl get csr
NAME             AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
csr-4jpkv        3d    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:abcdef   <none>              Approved,Issued
webhook-server   56s   kubernetes.io/kubelet-serving                 kubernetes-admin          10d                 Pending

$ kubectl certificate approve webhook-server
certificatesigningrequest.certificates.k8s.io/webhook-server approved

$ kubectl get csr webhook-server -o=jsonpath={.status.certificate} | base64 --decode > webhook-server.crt

# Create a secret with the certificates for the server
$ kubectl create secret tls webhook-server --cert=webhook-server.crt --key=webhook-server.key
secret/webhook-server created

# Deploy the imagePolicyWebhook server
$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: imagepolicywebhook
  name: imagepolicywebhook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: imagepolicywebhook
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: imagepolicywebhook
    spec:
      volumes:
      - name: cert
        secret:
          secretName: webhook-server
          items:
            - key: tls.crt
              path: webhook-server.crt 
      - name: key
        secret:
          secretName: webhook-server
          items:
            - key: tls.key
              path: webhook-server.key
      containers:
      - image: stephang/imagepolicywebhook:latest
        name: imagepolicywebhook
        ports:
        - containerPort: 443
        volumeMounts:
        - name: cert
          readOnly: true
          mountPath: /etc/ssl/certs/
        - name: key
          readOnly: true
          mountPath: /etc/ssl/private/
EOF

# Prepare the kube-apiserver configuration
## To tell the kube-apiserver to use our webhook, it needs the server certificate, an admissionConfiguration and a kubeconfig to connect to the server. We provide the needed Data in the /etc/kubernetes/epconfig webhook directory.
$ mkdir -p /etc/kubernetes/admission-control && \
cp webhook-server.crt /etc/kubernetes/admission-control

# The AdmissionConfiguration points to a kubeConfigfile that will authenticate the webhook. This configuration will allow every image in our cluster when the webhook is not reachable.
$ cat <<EOF >/etc/kubernetes/admission-control/imagepolicy.conf
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: ImagePolicyWebhook
    configuration:
      imagePolicy:
        kubeConfigFile: /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
        allowTTL: 50
        denyTTL: 50
        retryBackoff: 500
        defaultAllow: true
EOF

# The kube-apiserver uses only external DNS server and will not find the internal DNS service adresses. To let the apiserver use the internal service, the server in the kubeconfig must be set to the IP of the service. For standard admission controller, we could use a special notation to encode the dns name in the kubeconfig. As the client certificate, we use the existing apiserver certificate.
$ cat <<EOF >/etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/admission-control/webhook-server.crt
    server: https://$SERVICE_IP
  name: webhook
contexts:
- context:
    cluster: webhook
    user: image-bouncer-webhook.default.svc
  name: webhook
current-context: webhook
kind: Config
users:
- name: image-bouncer-webhook.default.svc
  user:
    client-certificate: /etc/kubernetes/pki/apiserver.crt
    client-key: /etc/kubernetes/pki/apiserver.key
EOF

# Prepare the kube-apiserver
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
## 新增以下內容
metadata:
  annotations:
    glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
  dnsPolicy: ClusterFirstWithHostNet

# 準備測試 YAML 檔
$ cat <<EOF > /root/vulns-pod.yml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: alpine
  name: alpine
spec:
  containers:
  - args:
    - sleep
    - 1d
    image: alpine
    name: alpine
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
EOF

Answer

官網文件收尋關鍵字 : ImagePolicyWebhook 找到這篇文章 : Admission Controllers Reference | Kubernetes 章節 : ImagePolicyWebhook

# 修改 AdmissionConfiguration 設定檔
$ nano /etc/kubernetes/admission-control/imagepolicy
...
plugins:
  - name: ImagePolicyWebhook
    configuration:
      imagePolicy:
        ...
        defaultAllow: false  <- 這裡改成 false

# 編輯 werbhook 要用的 kubeconfig，把 server 改成正確的位置
$ nano /etc/kubernetes/admission-control/imagepolicy_backend.kubeconfig
clusters:
- cluster:
    certificate-authority: /etc/kubernetes/epconfig/webhook-server.crt
    server: https://image-bouncer-webhook.default.svc    <- 這裡改成題目要的

# 啟用 imagepolicywebhook
$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
## 新增以下內容
  containers:
  - command:
    - --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
    - --admission-control-config-file=/etc/kubernetes/admission-control/imagepolicy
    ...
    volumeMounts:
    - mountPath: /etc/kubernetes/admission-control
      name: webhook
      readOnly: true
    ...
  - hostPath:
      path: /etc/kubernetes/admission-control
      type: DirectoryOrCreate
    name: webhook

# 確認 API Server 正常
$ crictl ps
NAMESPACE            NAME                                        READY   STATUS    RESTARTS       AGE
kube-system          kube-apiserver-c30-control-plane            1/1     Running   0              84s

$ kubectl -n kube-system get pods
kube-apiserver-c30-control-plane            1/1     Running   0              91s

# 測試建立 Pod 看 ImagePolicyWebhook 是否正常工作
$ kubectl apply -f /root/vulns-pod.yml
Error from server (Forbidden): error when creating "/root/vulns-pod.yml": pods "alpine" is forbidden: image policy webhook backend denied one or more images: Only nginx images are allowed

## 考試的時候要用 kubectl get event 去找錯誤

Question #4: Dockerfile 和 yaml 檔案安全性偵測

Task

Check the Dockerfile located at /root/Dockerfile on the CLI server. This Dockerfile is based upon alpine:3.13.5.
Correct the two security issues within the file.

:::info

Note: If you need an OS user, you can use the user nobody with id 65535.

:::

Check the YAML file located at /root/scooby-gang-deploy.yml on the CLI server.
Correct the two security issues within the file.

Prepare ENV

$ cat <<'EOF' > /root/Dockerfile
FROM alpine:latest

COPY sunnydale.sh .

USER root
CMD ["./sunnydale.sh"]
EOF

$ cat <<'EOF' > /root/scooby-gang-deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        securityContext:
          {'capabilities': {'add': ['NET_BIND_SERVICE'], 'drop': ['all']}, 'privileged': true, 'readOnlyRootFilesystem': false, 'runAsUser': 0}
EOF

Answer

# 修改 Dockerfile
$ nano /root/Dockerfile
FROM ubuntu:22.04
ENTRYPOINT ["/sunnydale.sh"]
USER nobody
CMD ["./sunnydale.sh"]

# 修改 Deployment
$ nano /root/scooby-gang-deploy.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        securityContext:
          # privileged 設成 False
          # readOnlyRootFilesystem 設成 True
          # runAsUser 設成 65535
          {'capabilities': {'add': ['NET_BIND_SERVICE'], 'drop': ['all']}, 'privileged': false, 'readOnlyRootFilesystem': true, 'runAsUser': 65535}

Question #5: log audit

Context

You must implement auditing for the kubeadm provisioned cluster.

Task

First, reconfigure the cluster’s API server, so that:

The basic audit policy located at /etc/kubernetes/logpolicy/audit-policy.yaml is used.
Logs are stored at /var/log/kubernetes/audit-logs.txt.
A maximum of 2 logs are retained for 10 days.

:::info

The basic policy only specifies what not to log.

:::

Next, edit and extend the basic policy to log:

namespaces interactions at RequestResponse level.
The request body of deployments interactions in the namespace webapps.
ConfigMap and Secret interactions in all namespaces at the Metadata level.
All other requests at the Metadata level.

:::warning

Make sure the API server uses the extended policy. Failure to do so may result in a reduced score.

:::

Prepare ENV

$ sudo mkdir -p /etc/kubernetes/logpolicy
$ cat <<EOF > /etc/kubernetes/logpolicy/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"
rules:
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"
EOF

Answer

官網文件收尋關鍵字 : audit 找到這篇文章 : Auditing | Kubernetes 章節 : Audit policy 和 Log backend

# 確認 deployment 的 API Group
$ kubectl api-resources | grep deploy
deployments                         deploy       apps/v1                           true         Deployment

# 備份 audit-policy.yaml
$ cp /etc/kubernetes/logpolicy/audit-policy.yaml ./audit-policy.yaml.bk

# 修改 audit-policy.yaml
## 不要刪除原有規則, 可以在下面繼續追加題目要求的規則
$ nano /etc/kubernetes/logpolicy/audit-policy.yaml
…
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["namespaces"]
  - level: Request
    resources:
    - group: "apps"
      resources: ["deployments"]
    namespaces: ["webapps"]
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets", "configmaps"]
  - level: Metadata

# 啟用 Audit Log
# 備份 kube-apiserver 的 YAML 檔
$ cp /etc/kubernetes/manifests/kube-apiserver.yaml ./kube-apiserver.yaml

$ nano /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
  containers:
  - command:
    ...
    ## 新增以下內容 (實際考試的時候，一樣要仔細確認每個參數在實際環境是否已經存在，如果存在就把值改得跟題目要求一樣，如果沒有參數再手動加，volumeMounts 和 volumes 這兩個項目的內容，在實際考試環境可能已經存在)
    - --audit-policy-file=/etc/kubernetes/logpolicy/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit-logs.txt
    - --audit-log-maxage=10
    - --audit-log-maxbackup=2
    volumeMounts:
    - mountPath: /etc/kubernetes/logpolicy/audit-policy.yaml
      name: audit
      readOnly: true
    - mountPath: /var/log/kubernetes/
      name: audit-log
      readOnly: false
  volumes:
  - name: audit
    hostPath:
      path: /etc/kubernetes/logpolicy/audit-policy.yaml
      type: File
  - name: audit-log
    hostPath:
      path: /var/log/kubernetes/
      type: DirectoryOrCreate

# 重啟 kubelet service
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet

## 等待 kube-apiserver 自動重起，且恢復正常
$ crictl ps

$ kubectl get nodes

## 檢查 Audit log 已啟用，看到 audit log 一直噴就代表這題成功!
$ tail -f /var/log/kubernetes/audit-logs.txt
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"0969e8fc-8dbd-4cfa-b1bd-44ed9d915344","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/events","verb":"create","user":{"username":"system:node:c30-worker","groups":["system:nodes","system:authenticated"]},"sourceIPs":["10.89.0.10"],"userAgent":"kubelet/v1.30.0 (linux/amd64) kubernetes/7c48c2b","objectRef":{"resource":"events","namespace":"default","name":"nginx-bf5d5cf98-gk9br.180afb34a972b0aa","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2024-11-24T18:33:30.277486Z","stageTimestamp":"2024-11-24T18:33:30.279054Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
...

Question #6: Container Security Context

Context

Container Security Context 應在特定 namespace 中修改 Deployment。

Task

依照下列要求修改 sec-ns namespace 裡的 Deployment secdep

使用 ID 為 30000 的使用者啟動 Container（設定使用者 ID 為: 30000）
不允許程序獲得超出其父程序的特權（禁止 allowPrivilegeEscalation）
以 readOnly 方式載入 Container 的根檔案系統（對根檔案的唯讀權限）

Prepare ENV

$ kubectl create ns sec-ns

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secdep
  labels:
    app: nginx
  namespace: sec-ns
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
    spec:
      containers:
      - name: sec-ctx-demo-1
        image: registry.k8s.io/pause:3.9
        ports:
      - name: sec-ctx-demo-2
        image: registry.k8s.io/pause:3.9
        ports:
EOF

Answer

官網文件收尋關鍵字 : Security Context 找到這篇文章 : Configure a Security Context for a Pod or Container | Kubernetes 章節 : Set the security context for a Pod 和 Log backend

# 備份 Deployment 的 yaml 檔
$ kubectl -n sec-ns get deployment secdep -o yaml > secdep.yaml

# 修改 Deployment
$ kubectl -n sec-ns edit deployment secdep
spec:
  template:
    spec:
      containers:
      - image: registry.k8s.io/pause:3.9
        imagePullPolicy: IfNotPresent
        name: sec-ctx-demo-1
        resources: {}
        securityContext:                                 ## 新增此行
          allowPrivilegeEscalation: false                ## 新增此行
          readOnlyRootFilesystem: true                   ## 新增此行
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - image: registry.k8s.io/pause:3.9
        imagePullPolicy: IfNotPresent
        name: sec-ctx-demo-2
        resources: {}
        securityContext:                                 ## 新增此行
          allowPrivilegeEscalation: false                ## 新增此行
          readOnlyRootFilesystem: true                   ## 新增此行
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:                                   ## 新增此行
        runAsUser: 30000                                 ## 新增此行

# 檢查修改後 pod 是否正常
$ kubectl -n sec-ns get pods
NAME                      READY   STATUS    RESTARTS   AGE
secdep-56c5959674-n5wb5   2/2     Running   0          2m13s

Question #7: Networkpolicy

Task

A default deny network policy is a policy that blocks all traffic in a namespace by default.

Create a network policy called default-deny that blocks all incoming traffic to all Pods in the mordor namespace.

Create a network policy called mtdoom-np that allows specific traffic on port 80 to reach the mtdoom Pod in the mordor namespace.

The policy should allow incoming traffic:

From all Pods in the frodo namespace.
From all Pods with the label app=sam, regardless of which namespace they are in.

考試時有測試的 pod 可以用 ping 命令

Prepare ENV

$ kubectl create ns mordor

$ kubectl create ns frodo --dry-run=client -o yaml | kubectl label --local -f - app=frodo --dry-run=client -o yaml | kubectl apply -f -

$ kubectl -n mordor run mtdoom --image=nginx

$ kubectl -n frodo run haha --image=nginx

$ kubectl run hello --image=nginx -l app=sam

Answer

$ nano default-denyall.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: mordor
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  
$ kubectl apply -f default-denyall.yaml

$ kubectl -n mordor get pods --show-labels
NAME     READY   STATUS    RESTARTS   AGE   LABELS
mtdoom   1/1     Running   0          13m   run=mtdoom

$ kubectl get ns frodo --show-labels
NAME    STATUS   AGE   LABELS
frodo   Active   19m   app=frodo,kubernetes.io/metadata.name=frodo

$ nano ingress-netpol.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: mtdoom-np
  namespace: mordor
spec:
  podSelector:
    matchLabels:
      run: mtdoom
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          app: frodo
    - podSelector:
        matchLabels:
          app: sam
    ports:
    - protocol: TCP
      port: 80

$ kubectl apply -f ingress-netpol.yaml

$ ip=$(kubectl -n mordor get pod mtdoom --template={{.status.podIP}})

$ kubectl -n frodo exec -it haha -- curl -s http://${ip} | grep nginx
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
<a href="http://nginx.org/">nginx.org</a>.<br/>
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>

$ kubectl exec -it hello -- curl -s http://${ip} | grep nginx
<title>Welcome to nginx!</title>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
<a href="http://nginx.org/">nginx.org</a>.<br/>
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>

$ kubectl run qqq --image=nginx

$ kubectl exec qqq -- curl -m 1 -s http://${ip} | grep nginx
command terminated with exit code 28

Question #8: Ingress TLS

Context

您必須使用儲存在 TLS Secret 中的 SSL 文件，來保護 Web Server 的安全存取。

Task

在 clever-cactus namespace中為名為 clever-cactus 的現有Deployment 建立名為 clever-cactus 的 TLS Secret 。

使用以下 SSL 檔案：

證書: /ca-cert/web.k8s.local.crt
密鑰: /ca-cert/web.k8s.local.key

Deployment 已設定為使用 TLS Secret。請勿修改現有的 Deployment。

service、deployment 和 secret 都好了，只要建有 TLS 的 Ingress，並且要啟用 ssl-redirect。

Answer

第一篇官網文章 :

官網文件收尋關鍵字 : ssl-redirect
找到這篇文章 : NGINX Ingress Controller + front F5 Load Balancer - General ...
不要點進去文章!!!

第二篇官網文章 :

官網文件收尋關鍵字 : ingress 找到這篇文章 : Ingress | Kubernetes 章節 : TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-example-ingress
spec:
  tls:
  - hosts:
      - https-example.foo.com
    secretName: testsecret-tls
  rules:
  - host: https-example.foo.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: service1
            port:
              number: 80

Question #9: ServiceAccount

Task

要關自動產生的 token 檔案，但是 pod 要把 token mount 回來，並且 volumemount 要設 readOnly

Prepare ENV

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-robot
EOF

Answer

官網文件收尋關鍵字 : serviceaccount 找到這篇文章 : Configure Service Accounts for Pods | Kubernetes 章節 : Opt out of API credential automounting 和 Launch a Pod using service account token projection

# 設定 serviceaccount 取消 API 憑證自動掛載
$ kubectl edit sa build-robot
apiVersion: v1
automountServiceAccountToken: false    ## 新增此行
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ServiceAccount","metadata":{"annotations":{},"name":"build-robot","namespace":"default"}}
  creationTimestamp: "2024-11-24T19:35:28Z"
  name: build-robot
  namespace: default
  resourceVersion: "16129"
  uid: 1e4d3c5a-8cb8-4612-a690-5d0960f2034e
 
# 設定 pod
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  annotations:
    glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
      readOnly: true
  serviceAccountName: build-robot
  volumes:
  - name: vault-token
    projected:
      sources:
      - serviceAccountToken:
          path: vault-token
          expirationSeconds: 7200
          audience: vault
EOF

$ kubectl exec nginx -- ls -l /var/run/secrets/tokens
total 0
lrwxrwxrwx 1 root root 18 Nov 24 19:43 vault-token -> ..data/vault-token

Question #10: Upgrade Worker node

Context

The kubeadm provisioned cluster was recently upgraded, leaving one node on a slightly older version due to workload compatibility concerns.

Task

Upgrade the cluster node compute-0 to match the version of the control plane node. Use a command like the following to connect to the compute node:

[candidate@cks000034] $ ssh compute-0

Prepare ENV

$ apt update
$ apt install -y gpg-agent software-properties-common

$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | tee /etc/apt/sources.list.d/kubernetes.list

$ kubelet --version
Kubernetes v1.31.2
[註] 把 kubeadm kubelet kubectl 套件安裝回來

$ apt update
$ apt install -y --allow-change-held-packages kubeadm=1.31.2-1.1 kubelet=1.31.2-1.1
*** kubelet (Y/I/N/O/D/Z) [default=N] ? Y
Installing new version of config file /etc/default/kubelet ...

$ apt-mark hold kubeadm kubelet

Answer

官網文件收尋關鍵字 : upgrade linux 找到這篇文章 : Upgrading Linux nodes | Kubernetes 章節 : Upgrading worker nodes

# 確認 cluster 狀態
$ kubectl get nodes
NAME                   STATUS   ROLES           AGE     VERSION
c30-up-control-plane   Ready    control-plane   4m29s   v1.30.3
c30-up-worker          Ready    <none>          4m10s   v1.30.2

# Drain the worker node
$ kubectl drain c30-up-worker --ignore-daemonsets

# 連線到 worker node
$ ssh compute-0

# 確認要升級的套件名稱
$ apt-cache madison kubeadm | grep '1.31'
   kubeadm | 1.31.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages
   kubeadm | 1.31.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.31/deb  Packages

# Upgrade kubeadm
$ apt-mark unhold kubeadm && \
apt-get update && apt-get install -y kubeadm='1.30.3-1.1' && \
apt-mark hold kubeadm

# Call "kubeadm upgrade"
$ kubeadm upgrade node

# Upgrade kubelet
$ apt-mark unhold kubelet && \
apt-get update && apt-get install -y kubelet='1.30.3-1.1' && \
apt-mark hold kubelet

# Restart the kubelet
$ systemctl daemon-reload && \
systemctl restart kubelet

# Uncordon the node
$ kubectl uncordon c30-up-worker

# Check node status
$ kubectl get nodes
NAME                   STATUS   ROLES           AGE   VERSION
c30-up-control-plane   Ready    control-plane   32m   v1.30.3
c30-up-worker          Ready    <none>          32m   v1.30.3

Question #11: BOM

Task

有一個 pod 裡面有 3 個 Container，要找出特定版本的特定套件在哪一個 Container images，並透過 bom 命令，將 container images 的套件相依的清單資訊倒到一個檔案。

Prepare ENV

$ curl -sL $(curl -sL https://api.github.com/repos/kubernetes-sigs/bom/releases/latest | jq -r .assets[].browser_download_url | grep 'bom-amd64-linux$') -o bom
install -o root -g root -m 0755 bom /usr/local/bin/bom
rm -r bom*

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: alpine
  name: alpine
  annotations:
    glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
  containers:
  - args:
    - sleep
    - 1d
    image: alpine
    name: alpine
  - args:
    - sleep
    - 1d
    image: busybox:1.28
    name: busybox
    resources: {}
  - name: nginx
    image: nginx:1.14.2
    ports:
    - containerPort: 80
  dnsPolicy: ClusterFirst
  restartPolicy: Always
EOF

Answer

## 先得到 image 的名稱
$ kubectl get pods alpine -o yaml | grep image: | tail -n 3
    image: docker.io/library/alpine:latest
    image: docker.io/library/busybox:1.28
    image: docker.io/library/nginx:1.14.2

$ bom generate --scan-images=false --image=docker.io/library/alpine:latest -o /root/alp.spdx

$ bom document outline --purl /root/alp.spdx

Question #12: Docker Daemon Troubleshooting

Task

要把 docker 群組裡面的其中一個 user 移除，sock 的權限是錯的，另一個送他(把題目背出來)

Answer

$ gpasswd -d developer docker

$ vim /usr/lib/systemd/system/docker.socket
## 將 SocketGroup 欄位設為 root

$ 修改docker.service，將[Service]下面ExecStart裡面的 -H tcp://0.0.0.0:2375 刪掉

Question #13: Cilium NetworkPolicy、MTLS

Context

使用 Cilium 執行以下任務，以保護現有應用程式的內部和外部網路流量。

Task

:::info

您可以使用瀏覽器存取 Cilium 的文件。

:::

首先，在 nodebb namespace 中建立一個名為 nodebb 的 L4 CiliumNetworkPolicy，並以以下方式設定它：

允許在 ingress-nginx namespace 中運行的所有 Pod 存取 nodebb Deployment 的 Pod
要求相互驗證

然後，將前一步建立的網路策略擴展如下：

允許主機存取 nodebb Deployment 的 Pod
不要使用相互身份驗證

Answer

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: nodebb
  namespace: nodebb
spec:
  endpointSelector:
    matchLabels:
      app: nodebb
  ingress:
  - fromEndpoints:
    - matchLabels:
        kubernetes.io/metadata.name: ingress-nginx
    authentication:
      mode: "required"
    fromEntities:
      - "host"

Question #14: Pod security standard

Task

題目會給你一個 YAML 檔去 apply，但會有很多錯誤訊息，慢慢改

Prepare ENV

$ kubectl create ns mytest

$ kubectl patch namespace mytest -p '{"metadata": {"labels": {"pod-security.kubernetes.io/enforce": "restricted"}}}' 

$ kubectl get ns mytest --show-labels
NAME     STATUS   AGE   LABELS
mytest   Active   10s   kubernetes.io/metadata.name=mytest,pod-security.kubernetes.io/enforce=restricted

$ mkdir -p /cks/pss/ && echo 'apiVersion: v1
kind: Pod
metadata:
  name: restricted-pod
  namespace: mytest
  annotations:
    glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
  containers:
  - name: restricted-container
    image: registry.k8s.io/pause:3.9
    securityContext:
      privileged: true' > /cks/pss/pod.yaml

Answer

# 查看具體遇到那些錯誤
$ kubectl apply -f /cks/pss/pod.yaml
Error from server (Forbidden): error when creating "/cks/pss/pod.yaml": pods "baseline-pod" is forbidden: violates PodSecurity "restricted:latest": privileged (container "baseline-container" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "baseline-container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "baseline-container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "baseline-container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "baseline-container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

# 修改 YAML 檔
$ nano /cks/pss/pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: restricted-pod
  namespace: mytest
  annotations:
    glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
spec:
  containers:
  - name: restricted-container
    image: registry.k8s.io/pause:3.9
    securityContext:
      ### 新增以下設定
      privileged: false
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - "ALL"
      runAsNonRoot: true
      seccompProfile:
        type: "RuntimeDefault"

# 部署修改好後的 Pod YAML
$ kubectl apply -f /cks/pss/pod.yaml

# 檢查可成功部署
$ kubectl get -f /cks/pss/pod.yaml
NAME             READY   STATUS    RESTARTS   AGE
restricted-pod   1/1     Running   0          4s

Question #15: TLS Secrets 創建與掛載

Context

您必須使用儲存在 TLS Secret 中的 SSL 文件，來保護 Web Server 的安全存取。

Task

在 clever-cactus namespace中為名為 clever-cactus 的現有Deployment 建立名為 clever-cactus 的 TLS Secret 。

使用以下 SSL 檔案：

證書: /root/nginx/nginx.crt
密鑰: /root/nginx/nginx.key

Deployment 已設定為使用 TLS Secret。請勿修改現有的 Deployment。

Prepare ENV

$ kubectl create ns clever-cactus

$ mkdir -p nginx

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout ./nginx/nginx.key -out ./nginx/nginx.crt

$ cat <<EOF > default.conf
server {
    listen 443 ssl;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    server_name localhost;
    ssl_certificate /etc/nginx/ssl/tls.crt;
    ssl_certificate_key /etc/nginx/ssl/tls.key;
}
EOF

$ kubectl -n clever-cactus create configmap nginx-conf --from-file=default.conf

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: clever-cactus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        glasbreaker.image-policy.k8s.io/ticket-1234: "break-glass"
    spec:
      volumes:
      - name: tls-volume
        secret:
          secretName: clever-cactus
      - name: config-volume
        configMap:
          name: nginx-conf
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 443
        volumeMounts:
        - mountPath: /etc/nginx/ssl
          name: tls-volume
        - mountPath: /etc/nginx/conf.d
          name: config-volume
EOF

Answer

官網文件收尋關鍵字 : secret 找到這篇文章 : Secrets | Kubernetes 章節 : TLS Secrets

$ kubectl -n clever-cactus get pods
NAME                    READY   STATUS              RESTARTS   AGE
nginx-b9b7995fc-hfhm4   0/1     ContainerCreating   0          90s

$ kubectl -n clever-cactus describe pod nginx-b9b7995fc-hfhm4
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    46s                default-scheduler  Successfully assigned clever-cactus/nginx-b9b7995fc-hfhm4 to c30-worker
  Warning  FailedMount  14s (x7 over 46s)  kubelet            MountVolume.SetUp failed for volume "clever-cactus" : secret "nginx-tls" not found

$ kubectl create secret tls clever-cactus -n clever-cactus \
  --cert=/root/nginx/nginx.crt \
  --key=/root/nginx/nginx.key
  
$ kubectl -n clever-cactus get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-5b4dcf49bd-ksl8z   1/1     Running   0          17s

Question #16: 考 Falco 送他

REF

Create a normal user with Client certificate in Kubernetes Kubernetes CPU throttling