antony@notes:~/kubernetes$ cat "Kubernetes-RBAC-Context.md"
Kubernetes-RBAC-Context
Kubernetes-RBAC-Context
:::warning
:::spoiler 目錄
[TOC]
:::
前言

- Linux 系統可以讓多人同時協同作業,使用者登入後,工作目錄會在自己的家目錄。
- 在 K8S 的系統中,也能夠實現多使用者同時進入 K8S 的叢集做操作,登入 K8S 時,靠的是所在機器(Linux/Windows 作業系統)的使用者家目錄下
~/.kube/config這個放憑證的檔案。 - 這個憑證會決定使用者在 K8S 可以進入哪個 Namespace ,可以"建立"還是"刪除"還是"查詢"物件…等。
K8S Multi-Tenancy 管理套件
下載老師準備好的環境
$ cd; wget http://www.oc99.org/zip/k8suser.zip; unzip k8suser.zip; cd k8suser
$ ls -l
total 40
-rw-r--r-- 1 bigred bigred 328 Apr 8 22:03 clusterbind.yaml
-rw-r--r-- 1 bigred bigred 507 Apr 8 22:03 clusterole.yaml
-rw-r--r-- 1 bigred bigred 269 Nov 1 2021 context.temp
-rw-r--r-- 1 bigred bigred 277 Oct 30 2021 csr.yaml
-rwxr-xr-x 1 bigred bigred 74 Apr 6 16:00 deluser.sh
-rwxr-xr-x 1 bigred bigred 975 Apr 8 21:57 mkcontext.sh
-rwxr-xr-x 1 bigred bigred 1045 Apr 6 16:01 mkubeuser.sh
-rw-r--r-- 1 bigred bigred 986 Apr 7 17:30 role.temp
-rw-r--r-- 1 bigred bigred 214 Apr 7 17:20 role.yaml
-rw-r--r-- 1 bigred bigred 302 Oct 27 2021 rolebind.yaml建立 K8S User 及 憑証
建立 bigboss 使用者
$ ./mkubeuser.sh bigboss
K8S bigboss created憑證放在 kuser 目錄
$ ls -l kuser/
total 12
-rw-r--r-- 1 bigred bigred 1115 Sep 12 10:13 bigboss.crt
-rw-r--r-- 1 bigred bigred 911 Sep 12 10:13 bigboss.csr
-rw------- 1 bigred bigred 1675 Sep 12 10:13 bigboss.keybigboss.key,私鑰bigboss.csr,放的是使用者的資訊,包含私鑰,姓名和服務單位的資料bigboss.crt,K8S 根據你用bigboss.csr這個檔案申請認證後,產生的憑證檔crtstand forcertificate
檢查 K8S 的所有使用者
$ kubectl config view | grep -A 10 -e '^users'
users:
- name: bigboss
user:
client-certificate: /home/bigred/k8suser/kuser/bigboss.crt
client-key: /home/bigred/k8suser/kuser/bigboss.key
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTEDREDACTED,表示沒有自己特別指定目錄將使用者登入資訊放在另一個檔案- 放在
~/.kube/config這個檔案裡面
- 放在
看 bigred 使用者在 K8S 系統的憑證
$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
...
server: https://192.168.61.4:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: bigboss
user:
client-certificate: /home/bigred/k8suser/kuser/bigboss.crt
client-key: /home/bigred/k8suser/kuser/bigboss.key
- name: kubernetes-admin
user:
client-certificate-data:
...clusters,設定 K8S 叢集 (結尾是 s ,代表 K8S 的 Cluster 可以有很多個)cluster.server,登入 K8S 的 IP 位址 :https://192.168.61.4:6443clusters.name,K8S 叢集的名字 :kubernetes
contexts.,設定 K8S 的入口 (結尾是 s ,代表 K8S 的入口可以有很多個)context.cluster,入口在哪一個 K8S 叢集 ( 名字就叫kubernetes)context.user,可以進入入口的使用者 :kubernetes-adminname,入口的名字 :kubernetes-admin@kubernetes
current-context,當前的入口 :kubernetes-admin@kuberneteskind: Config,設定使用者登入的資訊- 入口憑證檔的重點: 當前的入口名字、哪位使用者可以進去、進到哪一個 K8S Cluster
認識 Role-based Access Control
- Subjects: The set of users and processes that want to access the Kubernetes API.
- Resources: The set of Kubernetes API Objects available in the cluster. Examples include Pods, Deployments, Services, Nodes, and PersistentVolumes, among others.
- Verbs: The set of operations that can be executed to the resources above. Different verbs are available (examples: get, watch, create, delete, etc.), but ultimately all of them are Create, Read, Update or Delete (CRUD) operations.
Subject,指的是 K8S 的 User (分為 User 和 ServiceAccount)- Users: These are global, and meant for humans or processes living outside the cluster.
- 人和 K8S Cluster 外面的程序
- ServiceAccounts: These are namespaced and meant for intra-cluster processes running inside pods.
- 在 K8S Cluster 裡面的 pod 裡的程序
- Users: These are global, and meant for humans or processes living outside the cluster.
Resources,指的是 K8S 的物件Verbs,指的是可以對 K8S 的物件做什麼動作- 像是可以看 pod 的資訊、新增 pod 或是 刪除 pod …等動作
watch,是一直監控物件的狀態get,是看單一物件的狀態list,看單一種物件的狀態
產生 Role
# 檢視 RBAC 的版本
$ kubectl api-versions | grep rbac
rbac.authorization.k8s.io/v1
# 建立 finance Namespace
$ kubectl create namespace finance
# 編輯 Role 的 yaml 檔
$ echo $'kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: finance
name: finance-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods", "services"]
verbs: ["get", "watch", "list"] ' > role.yaml
# 建立 Role
$ kubectl create -f role.yaml
role.rbac.authorization.k8s.io/finance-reader created
# 檢視 Role 資訊
$ kubectl get roles --namespace=finance
NAME CREATED AT
finance-reader 2020-10-21T13:48:50ZRole,設定一個 Namespace 裡對什麼物件可以有什麼動作rules.resources,主要設定pods和services的物件rules.verbs,針對剛剛的物件,設定動作get、watch和list
產生 RoleBinding
# 建立 RoleBinding
$ echo $'kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: finance-read-access
namespace: finance
subjects:
- kind: User
name: bigboss
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role #this must be Role or ClusterRole
name: finance-reader
apiGroup: rbac.authorization.k8s.io ' | kubectl apply -f -
# 檢視 Rolebinding 資訊
$ kubectl get rolebindings --namespace=finance
NAME ROLE AGE
finance-read-access Role/finance-reader 8m13sRoleBinding,在單一 Namespace 中,將 Role 跟 K8S 的 User 綁在一起subjects.kind,設定 K8S Username: bigboss,指定 K8S 的 bigboss User
roleRef,設定 Role 或是 ClusterRolename: finance-reader,設定finance-reader這個 Role
Create Cluster Role
# 建立 ClusterRole
$ echo $'kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
# "namespace" omitted since ClusterRoles are not namespaced
name: cluster-pods-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "watch", "list"] ' | kubectl create -f -
# 檢查 ClusterRole 狀態
$ kubectl get clusterroles | grep cluster-pods-reader
NAME CREATED AT
cluster-pods-reader 2022-09-12T03:41:26ZCluster Role,設定在全部的 Namespace 中對什麼物件可以有什麼動作apiGroups,有設定這個才能設定要限制 resourcekubectl api-resources,可以看 resources 在哪個apiGroups下
Cluster Role Binding
# 建立 ClusterRoleBinding
$ echo $'kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-cluster-pods
subjects:
- kind: User
name: bigboss
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-pods-reader
apiGroup: rbac.authorization.k8s.io ' | kubectl create -f -
$ kubectl get clusterrolebindings | grep read-cluster-pods
NAME ROLE AGE
read-cluster-pods ClusterRole/cluster-pods-reader 16s
$ kubectl get pods --as=bigboss
No resources found in default namespaceclusterRolebinding,設定 K8S User (包含 service account)與 role 綁定而擁有存取權限
切換 K8S Namespace
# 執行以下命令, 可知目前使用的 K8S Context 的 namespace 沒有設定
# 用的是預設 default 的 Namespace
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* kubernetes-admin@kubernetes kubernetes kubernetes-admin
# 只要從這個路口登入,Namespace 就會直接切換到 finance namespace
$ kubectl config set-context $(kubectl config current-context) --namespace=finance
Context "kubernetes-admin@kubernetes" modified.
# 只會顯示 finance namespace 中的 POD
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld 1/1 Running 0 13h
# 因內定為 finance namespace, 所以 bigboss 可以讀取
$ kubectl get pods --as bigboss
NAME READY STATUS RESTARTS AGE
helloworld 1/1 Running 0 13h切換回 K8S Default Namespace
# 切換到 default namespace
$ kubectl config set-context $(kubectl config current-context) --namespace=
Context "kubernetes-admin@kubernetes" modified.
$ kubectl get pods
No resources found in default namespace.建立 K8S Context
建立 finance-context Context
$ kubectl config set-context finance-context \
--cluster=kubernetes --namespace=finance --user=bigbossset-context finance-context,建立新的 K8S 的入口,後面接的是入口名稱finance-context--cluster,設定入口所在的 K8S 叢集--namespace,設定進入入口後的 Namespace--user,設定哪一個 K8S 的使用者可以進入新建立的入口
檢視新的入口資訊
$ kubectl config view | grep -A 10 contexts:
contexts:
- context:
cluster: kubernetes
namespace: finance
user: bigboss
name: finance-context
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetesContexts下面成功多了剛剛新建立的finance-context
切換至 finance-context
$ kubectl config use-context finance-contextuse-context,設定當前的入口
在 finance-context 入口操作
# 檢視 nodes 資訊
$ k get nodes
Error from server (Forbidden): nodes is forbidden: User "bigboss" cannot list resource "nodes" in API group "" at the cluster scope
# 檢視 pods 資訊
$ k get pods
NAME READY STATUS RESTARTS AGE
helloworld 1/1 Running 0 124m
# 檢視在 kube-system 中 pods 的資訊
$ k get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-565d847f94-rf5tn 1/1 Running 2 2d2h
coredns-565d847f94-t5px2 1/1 Running 2 2d2h
etcd-m1 1/1 Running 2 2d2h
kube-apiserver-m1 1/1 Running 2 2d2h
kube-controller-manager-m1 1/1 Running 2 2d2h
kube-proxy-2pgpr 1/1 Running 2 2d2h
kube-proxy-fh9c2 1/1 Running 2 2d2h
kube-proxy-qblhl 1/1 Running 2 2d2h
kube-scheduler-m1 1/1 Running 2 2d2hK8S Multi-Tenancy 實作
前置動作
$ mkdir tmp; cd tmp
$ unzip ~/k8suser.zip
$ cp k8suser/*.yaml ~/k8suser/建立 K8S User 及 憑証
$ ./mkubeuser.sh rbean
K8S rbean created
$ ls -lt kuser/
total 24
-rw-r--r-- 1 bigred bigred 1111 Sep 12 13:49 rbean.crt
-rw-r--r-- 1 bigred bigred 907 Sep 12 13:49 rbean.csr
-rw------- 1 bigred bigred 1679 Sep 12 13:49 rbean.key
...檢查是否符合預期
$ k config view | grep -A12 -e '^users'
users:
- name: bigboss
user:
client-certificate: /home/bigred/k8suser/kuser/bigboss.crt
client-key: /home/bigred/k8suser/kuser/bigboss.key
- name: kubernetes-admin
user:
client-certificate-data: REDACTED
client-key-data: REDACTED
- name: rbean
user:
client-certificate: /home/bigred/k8suser/kuser/rbean.crt
client-key: /home/bigred/k8suser/kuser/rbean.key建立 rbean 使用者的入口
$ ./mkcontext.sh rbean
namespace/rbean created
- context:
cluster: default
namespace: rbean
user: rbean
name: rbean-context
Warning: resource roles/finance-reader is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
role.rbac.authorization.k8s.io/finance-reader configured
rolebinding.rbac.authorization.k8s.io/rbean-rbind created
clusterrole.rbac.authorization.k8s.io/rbean-clusterole created
clusterrolebinding.rbac.authorization.k8s.io/rbean-clusterbind created檢查
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://192.168.61.4:6443
name: kubernetes
contexts:
...
- context:
cluster: default
namespace: rbean
user: rbean
name: rbean-context
current-context: kubernetes-admin@kubernetes
...檢視 mkubeuser.sh
#!/bin/bash
export STU=${1}
export DIR_CSR=~/k8suser/kuser
[ "$#" != "1" ] && echo "mkubeuser.sh user" && exit 1
[ ! -d ${DIR_CSR} ] && mkdir -p ${DIR_CSR}
which envsubst &>/dev/null
[ $? = 1 ] && sudo apk update && sudo apk add gettext &>/dev/null
if [ -f ${DIR_CSR}/${STU}.key ]; then
echo "K8S ${STU} existed"
else
openssl genrsa -out ${DIR_CSR}/${STU}.key 2048 &>/dev/null
openssl req -new -key ${DIR_CSR}/${STU}.key -out ${DIR_CSR}/${STU}.csr -subj "/CN=${STU}/O=${STU}"
export BASE64_CSR=$(cat ${DIR_CSR}/${STU}.csr | base64 | tr -d '\n')
cat ~/k8suser/csr.yaml | envsubst | kubectl apply -f - &>/dev/null
kubectl certificate approve ${STU}-csr &>/dev/null
sleep 5
kubectl get csr ${STU}-csr -o jsonpath='{.status.certificate}' | base64 -d > ${DIR_CSR}/${STU}.crt
kubectl config set-credentials ${STU} --client-certificate=${DIR_CSR}/${STU}.crt --client-key=${DIR_CSR}/${STU}.key &>/dev/null
kubectl config view | grep "name: ${STU}" &>/dev/null
[ "$?" == "0" ] && echo "K8S ${STU} created"
fi檢視 K8S User Context
$ cat kuser/rbean.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F
........
server: https://192.168.61.4:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: rbean
user: rbean
name: rbean-context
current-context: rbean-context
........
kind: Config
preferences: {}
users:
- name: rbean
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCakNDQWU2Z0F..........
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0........client-certificate-data存放 公鑰 的憑証檔client-key-data,存放使用者的 私鑰
檢視 Namespace RBAC 設定
$ kubectl get role -n rbean
NAME CREATED AT
rbean-role 2022-04-08T14:50:29Z
$ kubectl get clusterrole -n rbean | grep rbean
rbean-clusterole 2022-04-08T14:50:29Z
$ kubectl describe role rbean-role -n rbean
Name: rbean-clusterole
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
csidrivers.storage.k8s.io [] [] [*]
csinodes.storage.k8s.io [] [] [*]
storageclasses.storage.k8s.io [] [] [*]
volumeattachments.storage.k8s.io [] [] [*]
nodes [] [] [create delete deletecollection get list patch update watch]
persistentvolumes [] [] [create delete deletecollection get list patch update watch]建立 Linux User 及 設定 K8S 憑証
$ echo -e "rbean\nrbean" | sudo adduser rbean
$ sudo mkdir /home/rbean/.kube; sudo cp ~/k8suser/kuser/rbean.conf /home/rbean/.kube/config; sudo chown -R rbean:rbean /home/rbean/.kube
$ dir /home/rbean/.kube
total 16K
drwxr-sr-x 2 rbean rbean 4.0K Apr 7 10:00 .
drwxr-sr-x 3 rbean rbean 4.0K Apr 7 10:00 ..
-rw-r--r-- 1 rbean rbean 5.5K Apr 7 10:00 config
$ exit/home/rbean/.kube/config,這個檔案為路口憑證檔
檢視 K8S User 憑証
在 Windows 系統的 cmd 視窗, 執行以下命令
$ ssh rbean@<alp.m1 IP>
$ kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://192.168.61.4:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
namespace: rbean
user: rbean
name: rbean-context
current-context: rbean-context
kind: Config
preferences: {}
users:
- name: rbean
user:
client-certificate-data: REDACTED
client-key-data: REDACTED測試 K8S User 憑証
$ kg all
No resources found in rbean namespace.
$ kg pv
No resources found
$ kg sc
No resources found
$ kubectl run a1 --rm -it --image=quay.io/cloudwalker/alpine
If you don't see a command prompt, try pressing enter.
/ # ping -c 1 www.hinet.net
PING www.hinet.net (163.28.83.113): 56 data bytes
ping: permission denied (are you root?)
/ # exit
$ exitbobo 網站應用系統
佈署 bobo 網站應用系統
在 bigred 終端機, 執行以下命令
$ wget http://www.oc99.org/zip/bobowas.zip; unzip bobowas.zip
# 佈署 alp.mysql, alp.goweb 及 alp.myfbs image
$ cd ~/bobowas/k8simg/
$ ./go-k8s-images.sh
[images build]
alp.myfbs image ok
alp.mysql image ok
alp.goweb image ok
[images deploy]
w1: localhost/alp.myfbs image ok
w2: localhost/alp.myfbs image ok
w1: localhost/alp.mysql image ok
w2: localhost/alp.mysql image ok
w1: localhost/alp.goweb image ok
w2: localhost/alp.goweb image ok在 rbean 終端機, 執行以下命令
$ wget http://www.oc99.org/zip/bobowas.zip; unzip bobowas.zip
$ cd /home/bigred/bobowas/k8sobj/
$ ./go-k8s-object.sh
bobo.fbs(w1) pod created
bobo.mysql(w2) pod created
bobo.goweb(w1) pod created
bobo.gosvc service created
$ kg all
NAME READY STATUS RESTARTS AGE
pod/bobo.fbs 1/1 Running 0 15s
pod/bobo.goweb 1/1 Running 0 15s
pod/bobo.mysql 1/1 Running 0 15s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/svc-gosrv ClusterIP 10.98.0.237 192.168.61.4 80/TCP 15sK8S Namespace Resource Quota
在 bigred 終端機, 執行以下命令
$ nano ~/bobowas/k8sobj/nsrq.yamlapiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: rbean
spec:
hard:
requests.cpu: "1"
requests.memory: 256Mi
requests.nvidia.com/gpu: 1
limits.cpu: "2"
limits.memory: 1Gi
limits.nvidia.com/gpu: 2
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: object-quota
namespace: rbean
spec:
hard:
pods: "3"
configmaps: "2"
persistentvolumeclaims: "2"
replicationcontrollers: "2"
secrets: "10"
services: "10"
services.loadbalancers: "2"產生 resourcequota
$ ka -f nsrq.yaml檢視 Resource Quota
在 rbean 終端機, 執行以下命令
$ kubectl get resourcequotas
NAME AGE REQUEST LIMIT
compute-quota 2m36s requests.cpu: 0/1, requests.memory: 0/256Mi, requests.nvidia.com/gpu: 0/1 limits.cpu: 0/2, limits.memory: 0/1Gi, limits.nvidia.com/gpu: 0/2
object-quota 2m36s configmaps: 1/2, persistentvolumeclaims: 0/2, pods: 3/3, replicationcontrollers: 0/2, secrets: 0/10, services: 1/10, services.loadbalancers: 0/2- Resource Quota 建立後,並不會限制已經建立的 pod 。
產生 a1 的 pod 測試有沒有被限制
$ kubectl run a1 --rm -it --image=quay.io/cloudwalker/alpine
Error from server (Forbidden): pods "a1" is forbidden: failed quota: compute-quota: must specify limits.cpu for: a1; limits.memory for: a1; requests.cpu for: a1; requests.memory for: a1- 錯誤訊息說明 : 建立 pod 時,要設定使用的資源
重新編輯 yaml 檔
$ kubectl run a1 --image=quay.io/cloudwalker/alpine --dry-run=client -o yaml > pod.yaml編輯 yaml 檔
$ nano pod.yamlapiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: a1
name: a1
spec:
containers:
- image: quay.io/cloudwalker/alpine
name: a1
resources:
requests:
cpu: 1.0
memory: 256Mi
limits:
cpu: 2.0
memory: 512Mi
dnsPolicy: ClusterFirst
restartPolicy: Always
status: {}再重新產生 pod
$ kubectl apply -f pod.yaml
Error from server (Forbidden): error when creating "pod.yaml": pods "a1" is forbidden: exceeded quota: object-quota, requested: pods=1, used: pods=3, limited: pods=3- 這時候噴的錯誤就是超過 resourcequota 的限制