antony@notes:~/security$ cat "Persistent-Data-for-NeuVector-Policy-and-Configuration.md"

Persistent Data for NeuVector Policy and Configuration

2023-09-27· security ·NeuVector 5.2.0

Persistent Data for NeuVector Policy and Configuration

Preface

本篇文章會採取 Step by Step 和 ScreenShot 的介紹，

如何將 Neuvector 的 Policy and Configuration 透過 Kubernetes Persistent Volume 永久保存資料

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

:::warning

:::spoiler 文章目錄

[TOC]

:::

Intro

預設情況下，NeuVector 將各種設定檔儲存在 Controller-Pods 的 /var/neuvector/config/backup 中。

NeuVector 可以將 configuration ( 設定 ) 和 policy ( 規則 ) 儲存到硬碟上，這樣即使 Controller 故障，設定和規則也不會遺失。這可以通過將一個 Kubernetes volume 掛載到 Controller pod 中的 /var/neuvector/ 目錄區來實現。

如果 K8S 集群完全故障，當創建新 K8S 集群時，設定和規則會自動恢復。設定和規則也可以手動從 Volume 中恢復或刪除。

如果在 HA 配置中執行多個 Controller-Pods，只要其中一個 Controller-Pod 始終處於運作狀態，所有資料都會在 Controller 之間同步。

如果希望儲存 violations ( 違規 )、threats ( 威脅 )、vulnerabilities ( 漏洞 ) 和 events ( 事件 ) 等 Logs，請在 “Settings” 中啟用 SYSLOG server。

如果沒有使用「Kuberentes Persistent Volume」，NeuVector 就不會將設定或規則儲存到硬碟上。在停止所有 Neuvector-Controller-Pods 之前，請先備份 Controller 的設定和規則，這可以在「Settings」>「Configuration」中完成。

如果要確保設定和規則在 Neuvector Controller 更新時不會遺失，可以將 Neuvector Controller 部署在高可用性 (HA) 配置中，以 3 或 5 個 Neuvector-Controller-Pods 運行。

環境介紹

將 Neuvector 5.2.0 安裝在 OpenShift 4.12 之上

Step1: Create Persistent Volume Claim

$ nano neuvector-pvc.yaml

檔案內容如下 :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: neuvector-data
  namespace: neuvector
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 1Gi

:::danger

Note:

如果在您的環境中有 StorageClass 則可跳過 Step2 建立 Persistent Volume
NeuVector 對於 accessModes 的要求必須是 ReadWriteMany(RWX).

:::

建立 PVC

$ oc create -f neuvector-pvc.yaml

Step2: Create Persistent Volume

$ nano neuvector-pv.yaml

檔案內容如下 :

apiVersion: v1
kind: PersistentVolume
metadata:
  name: neuvector-data
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/var/neuvector"

建立 PV

$ oc create -f neuvector-pv.yaml

檢查 PV 和 PVC 是否 Bound 在一起

$ oc get pv,pvc

螢幕輸出 :

NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
persistentvolume/neuvector-data   1Gi        RWX            Retain           Bound    neuvector/neuvector-data                           18m

NAME                                   STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/neuvector-data   Bound    neuvector-data   1Gi        RWX                           18

Step3: Modify the NeuVector-Controller sample yaml file

If Neuvector is existed

:::danger

:::spoiler {state=“open”} Note: 可以先依照以下步驟，匯出 Neuvector 的所有設定檔

Settings –> Configuration –> Export –> All –> Submit

如果 Volume 重新掛載到 Neuvector Controller 之後，有發現設定檔有殘缺，就可以 Import 回去

點 Choose File –> 再點擊向上的箭頭，匯入檔案

:::

執行以下命令修改 neuvector-controller-pod

$ oc edit deploy neuvector-controller-pod

修改檔案內容如下 :

...
metadata:
  name: neuvector-controller-pod
spec:
  template:
    spec:
      containers:
      - env:
        - name: CTRL_PERSIST_CONFIG                ## 新增這個 ENV
...
      volumes:
      - #hostPath:                                 ## 註解掉這行
        #  path: /var/neuvector                    ## 註解掉這行
        #  type: ""                                ## 註解掉這行
        name: nv-share
        persistentVolumeClaim:                     ## 新增 PVC
          claimName: neuvector-data                ## 指定 Pods 使用哪個 PVC

Rollout Restart Controller Pods

$ oc rollout restart deploy neuvector-controller-pod

檢視 Pods 被滾動式重新佈署之後的狀態

$ watch -n 1 oc get pods -l app=neuvector-controller-pod

成功部屬後的螢幕輸出

NAME                                        READY   STATUS    RESTARTS   AGE
neuvector-controller-pod-7bd8dd947f-8tw66   1/1     Running   0          8m34s
neuvector-controller-pod-7bd8dd947f-qtghq   1/1     Running   0          9m50s
neuvector-controller-pod-7bd8dd947f-vf94c   1/1     Running   0          7m19s

If Neuvector doesn’t existed

$ nano neuvector.yaml

修改檔案內容如下 :

...
spec:
  template:
    spec:
      volumes:
        - name: nv-share
#         hostPath:                        // replaced by persistentVolumeClaim
#           path: /var/neuvector        // replaced by persistentVolumeClaim
          persistentVolumeClaim:
            claimName: neuvector-data

Add the following environment variable in the Controller

            - name: CTRL_PERSIST_CONFIG

Step4: Build Neuvector

$ oc create -f neuvector.yaml

檢視 Neuvector Pods 運作狀況

$ oc get pods

螢幕輸出 :

NAME                                        READY   STATUS    RESTARTS   AGE
neuvector-controller-pod-74d88c4df6-7kqxk   1/1     Running   0          4m4s
neuvector-controller-pod-74d88c4df6-m9spn   1/1     Running   0          4m3s
neuvector-controller-pod-74d88c4df6-vp96s   1/1     Running   0          4m3s
neuvector-enforcer-pod-4zsj8                1/1     Running   0          4m3s
neuvector-enforcer-pod-f5znw                1/1     Running   0          4m3s
neuvector-enforcer-pod-tqhz5                1/1     Running   0          4m3s
neuvector-manager-pod-6bb5555c7b-85sdk      1/1     Running   0          4m4s
neuvector-operator-86dd64f497-nqmbg         1/1     Running   2          19d
neuvector-scanner-pod-56c798bb86-8ktcl      1/1     Running   0          4m3s
neuvector-scanner-pod-56c798bb86-rvftv      1/1     Running   0          4m3s

Step5: 建立掃描 Image Registry 的規則

Assets -> Registries -> Add

Name: quay.io
Registry: https://quay.io/
User Name: hahappyman
Password: ***
Filter: hahappyman/alpine-*

勾選 Scan Layers -> Submit

點擊 Start Scan 按鈕

檢視掃描 Image Registry 的相關設定是否被保存到 Node 上

$ cat <<EOF | oc debug node/worker2.ocp4.example.com 2> /dev/null
chroot /host
/bin/bash -c "[[ -d /var/neuvector/config/backup/ ]] && cat /var/neuvector/config/backup/registry.backup || ls -al /var/neuvector"
EOF

因為本次範例 PV 使用的是 HostPath ，資料可能只存在特定的 Node 上，Neuvector 的 Controller 會自行同步，所以有的節點的 /var/neuvector 目錄底下沒有資料，如以下螢幕輸出

total 4
drwxr-xr-x.  2 root root    6 Aug  1 04:25 .
drwxr-xr-x. 26 root root 4096 Aug  1 04:25 ..

正確螢幕輸出如下

{"auth_token":"","auth_with_token":false,"aws_key":null,"cfg_type":2,"creater_domains":null,"disable_files":false,"domains":null,"filters":["hahappyman/*"],"gcr_key":null,"gitlab_api_url":"","gitlab_private_token":"","ibmcloud_account":"","ibmcloud_token_url":"","jfrog_aql":false,"jfrog_mode":"","name":"quay.io","parsed_filters":[{"organization":"hahappyman","repository":".*","tag":".*"}],"password":"tZS7XXX6T0bkk0Pim8y3//qb0xmTVx5eGboJcrck","poll_period":0,"registry":"https://quay.io/","repo_limit":200,"rescan_image":true,"scan_layers":true,"schedule":"manual","tag_limit":20,"type":"Docker Registry","username":"hahappyman"}

此時就可以看到掃描 Image Registry 的設定被儲存在 Node 上
Image 掃描的結果也會被儲存到 /var/neuvector/registry/ 目錄下

Step6: 建立 Admission Control 規則

Policy -> Admission Control -> Add

Criterion:
- Run as privileged
- Namespace is one of [test]
Mode: Protect

勾選 Status: Enabled，並切換到 Protect Mode

測試 Admission Control 規則

$ oc run nginx --image=nginx:stable -n test --privileged=true

螢幕輸出 :

Error from server: admission webhook "neuvector-validating-admission-webhook.neuvector.svc" denied the request: Creation of Kubernetes Pod is denied.

檢視 Admission Control 規則是否被保存到 Node 上

$ cat <<EOF | oc debug node/worker2.ocp4.example.com 2> /dev/null
chroot /host
/bin/bash -c "[[ -d /var/neuvector/config/backup/ ]] && cat /var/neuvector/config/backup/admission_control.backup || ls -al /var/neuvector"
EOF

螢幕輸出 :

{"id":1000,"category":"Kubernetes","comment":"","criteria":[{"name":"runAsPrivileged","op":"=","value":"true","value_slice":null,"path":"runAsPrivileged"},{"name":"namespace","op":"containsAny","value":"test","value_slice":null,"path":"namespace"}],"disable":false,"critical":false,"cfg_type":2,"rule_type":"deny","use_as_risky_role_tag":false,"rule_mode":"protect"}

...以下省略

Step7: 刪除 Neuvector

$ oc delete -f neuvector.yaml

須確認 Pods 都已被刪除

Step8: 重新建立 Neuvector

$ oc create -f neuvector.yaml

檢視 Pods 運作狀態

$ oc get pods

螢幕輸出 :

NAME                                        READY   STATUS    RESTARTS   AGE
neuvector-controller-pod-74d88c4df6-h7npj   1/1     Running   0          3m43s
neuvector-controller-pod-74d88c4df6-k6fvg   1/1     Running   0          3m43s
neuvector-controller-pod-74d88c4df6-th8lj   1/1     Running   0          3m43s
neuvector-enforcer-pod-b2n8r                1/1     Running   0          3m43s
neuvector-enforcer-pod-mzvkb                1/1     Running   0          3m43s
neuvector-enforcer-pod-rdvq8                1/1     Running   0          3m43s
neuvector-manager-pod-6bb5555c7b-zlq78      1/1     Running   0          3m43s
neuvector-operator-86dd64f497-nqmbg         1/1     Running   2          19d
neuvector-scanner-pod-56c798bb86-qpfc6      1/1     Running   0          3m43s
neuvector-scanner-pod-56c798bb86-v29w8      1/1     Running   0          3m43

Step9: 確認設定和規則都還存在

連線至 Neuvector Web Console

掃描 Image Registry 的相關設定

Admission Control 規則

Config 會被備份的種類

$ ls -l /var/neuvector/config/backup

螢幕輸出 :

total 1332
-rw-------. 1 root root   2615 Sep 19 08:11 admission_control.backup
-rw-------. 1 root root     96 Sep 19 07:53 compliance.backup
-rw-------. 1 root root    378 Sep 19 07:53 crd.backup
-rw-------. 1 root root  14903 Sep 19 07:53 dlp_group.backup
-rw-------. 1 root root   2539 Sep 19 07:53 dlp_rule.backup
-rw-------. 1 root root  29463 Sep 19 07:53 domain.backup
-rw-------. 1 root root     41 Sep 19 07:53 eula_oss.backup
-rw-------. 1 root root    488 Sep 19 07:53 federation.backup
-rw-------. 1 root root 433060 Sep 19 07:53 file_monitor.backup
-rw-------. 1 root root 554801 Sep 19 07:53 file_rule.backup
-rw-------. 1 root root  80549 Sep 19 07:53 group.backup
-rw-------. 1 root root  72943 Sep 19 08:06 policy.backup
-rw-------. 1 root root  93486 Sep 19 07:53 process_profile.backup
-rw-------. 1 root root    464 Sep 19 07:53 pwd_profile.backup
-rw-------. 1 root root    664 Sep 19 07:53 registry.backup
-rw-------. 1 root root   1738 Sep 19 07:53 response_rule.backup
-rw-------. 1 root root   1295 Sep 19 07:53 system.backup
-rw-------. 1 root root    532 Sep 19 08:02 user.backup
-rw-r--r--. 1 root root     45 Sep 19 07:53 version.backup
-rw-------. 1 root root     76 Sep 19 07:53 vulnerability.backup
-rw-------. 1 root root  14903 Sep 19 07:53 waf_group.backup
-rw-------. 1 root root   1481 Sep 19 07:53 waf_rule.backup

Ref

Backups and Persistent Data - Neuvector docs

Introduction to NeuVector Helm Install Neuvector on community Kubernetes