跳至內容

antony@notes:~/security$ cat "Offline-Install-Neuvector-5.2.0-on-Openshift-Container-Platform-4.12.md"

Offline Install Neuvector 5.2.0 on Openshift Container Platform 4.12

2024-09-09· 安全性 ·NeuVector 5.2.0

Preface

本篇文章會介紹，

  1. 如何在 OpenShift Container Platform 4.12 上安裝 Neuvector 5.2.0
  2. 設定 Neuvector web console 使用自簽憑證 ( Self-Signed Certificate )

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

安裝前環境準備

1. 確認 Image Registry 的位址，並再修改 Image 的 Tag 後，將 Image 上傳至 Image Registry

  • 要確定在叢集中的每一台 Node 都可以至該 Image Registry Pull Image 下來
docker login -u <user_name> -p `oc whoami -t` docker-registry.default.svc:5000
docker tag docker.io/neuvector/enforcer:<version> docker-registry.default.svc:5000/neuvector/enforcer:<version>
docker tag docker.io/neuvector/controller:<version> docker-registry.default.svc:5000/neuvector/controller:<version>
docker tag docker.io/neuvector/manager:<version> docker-registry.default.svc:5000/neuvector/manager:<version>
docker tag docker.io/neuvector/scanner docker-registry.default.svc:5000/neuvector/scanner
docker tag docker.io/neuvector/updater docker-registry.default.svc:5000/neuvector/updater
docker push docker-registry.default.svc:5000/neuvector/enforcer:<version>
docker push docker-registry.default.svc:5000/neuvector/controller:<version>
docker push docker-registry.default.svc:5000/neuvector/manager:<version>
docker push docker-registry.default.svc:5000/neuvector/scanner
docker push docker-registry.default.svc:5000/neuvector/updater
docker logout docker-registry.default.svc:5000

2. 檢查環境有無 defaultNodeSelector 

$ oc get scheduler cluster -o yaml | grep -A10 defaultNodeSelector

如果沒有螢幕輸出，則跳過本步驟

螢幕輸出 :

spec:
  defaultNodeSelector: node-role.kubernetes.io/worker=

解決辦法 

$ oc annotate namespace neuvector openshift.io/node-selector=

如果 Neuvector 的 Enforcer DaemonSets 已建立，則須刪除 Pod，讓 Pod 重建

$ oc delete pod -l app=neuvector-enforcer-pod

如果不做本步驟，你可能會再建立完 Neuvector 後，看到 Enforcer 的 Pod 出現 Pending 的狀況，並且看到類似以下的錯誤訊息

0/11 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/11 nodes are available: 1 Preemption is not helpful for scheduling, 10 No preemption victims found for incoming pod.

3. 檢查 CRI-O Socket 的位置 

$ for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
  echo
  echo "$i"
  echo
  cat <<EOF | oc debug node/"$i" 2> /dev/null
chroot /host
ps -eaf | grep kubelet | tr ' ' '\n' |  grep -w -- '--container-runtime-endpoint'
EOF
done

螢幕輸出 :

master.ocp4.example.com

--container-runtime-endpoint=/var/run/crio/crio.sock

worker1.ocp4.example.com

--container-runtime-endpoint=/var/run/crio/crio.sock

worker2.ocp4.example.com

--container-runtime-endpoint=/var/run/crio/crio.sock

4. 檢查在叢集中的每一台 Node 上有無額外的 Taints 

$ for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
  echo
  echo "$i"
  echo
  if [[ -z $(oc get node "$i" -o=jsonpath='{.spec.taints}') ]]; then
    echo "  No Taints on the $i"
  else
    echo "  $(oc get node "$i" -o=jsonpath='{.spec.taints}')"
  fi
done

螢幕輸出 :

master.ocp4.example.com

  [{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"}]

worker1.ocp4.example.com

  No Taints on the worker1.ocp4.example.com

worker2.ocp4.example.com

  No Taints on the worker2.ocp4.example.com
spec:
  template:
    spec:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
        - effect: NoSchedule
          key: mykey
          value: myvalue

如果有額外的 Taints 請在紅字的地方添加。這需要與污點節點上定義的所有污點信息相匹配。否則，Enforcer 的 Pod 將無法在污點節點上部署

一次檢查全部的腳本 

$ curl -s https://raw.githubusercontent.com/braveantony/bash-script/main/precheck.sh | bash
#!/bin/bash

# check Env before installing Neuvector on OCP

## check Taint

echo "Starting Check The Taint Info On a Node"

for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
  echo
  echo "$i"
  echo
  if [[ -z $(oc get node "$i" -o=jsonpath='{.spec.taints}') ]]; then
    echo "  No Taints on the $i"
  else
    echo "  $(oc get node "$i" -o=jsonpath='{.spec.taints}')"
  fi
done

echo
echo "Note: All taint info must match to schedule Enforcers on nodes"
echo "=================="

## check CRI-O run-time

echo "Starting Check CRI-O run-time"

for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
  echo
  echo "$i"
  echo
  cat <<EOF | oc debug node/"$i" 2> /dev/null
chroot /host
ps -eaf | grep kubelet | tr ' ' '\n' |  grep -w -- '--container-runtime-endpoint'
EOF
done

echo "=================="

## check defaultNodeSelector

echo "Starting Check defaultNodeSelector"

if [[ -z $(oc get scheduler cluster -o yaml | grep defaultNodeSelector) ]]; then
  echo "  No defaultNodeSelector"
else
  oc get scheduler cluster -o yaml | grep -A10 defaultNodeSelector
fi

echo "=================="
echo "End Check"

開始安裝 Neuvector

1. Login as a normal user 

$ oc login -u <user_name>

2. Create a new project 

$ oc new-project neuvector

3. Login as system:admin account 

$ oc login -u system:admin

4. Create Service Accounts and Grant Access to the Privileged SCC 

$ oc create sa controller -n neuvector
$ oc create sa enforcer -n neuvector
$ oc create sa basic -n neuvector
$ oc create sa updater -n neuvector
$ oc -n neuvector adm policy add-scc-to-user privileged -z controller -z enforcer

5. Create the custom resources (CRD) for NeuVector security rules. For OpenShift 4.6+ (Kubernetes 1.19+): 

$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/crd-k8s-1.19.yaml
$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/waf-crd-k8s-1.19.yaml
$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/dlp-crd-k8s-1.19.yaml
$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/admission-crd-k8s-1.19.yaml

6. Add read permission to access the kubernetes API and OpenShift RBACs. 

oc create clusterrole neuvector-binding-app --verb=get,list,watch,update --resource=nodes,pods,services,namespaces
oc create clusterrole neuvector-binding-rbac --verb=get,list,watch --resource=rolebindings.rbac.authorization.k8s.io,roles.rbac.authorization.k8s.io,clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,imagestreams.image.openshift.io
oc adm policy add-cluster-role-to-user neuvector-binding-app system:serviceaccount:neuvector:controller
oc adm policy add-cluster-role-to-user neuvector-binding-rbac system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-admission --verb=get,list,watch,create,update,delete --resource=validatingwebhookconfigurations,mutatingwebhookconfigurations
oc adm policy add-cluster-role-to-user neuvector-binding-admission system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-customresourcedefinition --verb=watch,create,get,update --resource=customresourcedefinitions
oc adm policy add-cluster-role-to-user neuvector-binding-customresourcedefinition system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-nvsecurityrules --verb=list,delete --resource=nvsecurityrules,nvclustersecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvsecurityrules system:serviceaccount:neuvector:controller
oc adm policy add-cluster-role-to-user view system:serviceaccount:neuvector:controller --rolebinding-name=neuvector-binding-view
oc create clusterrole neuvector-binding-nvwafsecurityrules --verb=list,delete --resource=nvwafsecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvwafsecurityrules system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-nvadmissioncontrolsecurityrules --verb=list,delete --resource=nvadmissioncontrolsecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvadmissioncontrolsecurityrules system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-nvdlpsecurityrules --verb=list,delete --resource=nvdlpsecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvdlpsecurityrules system:serviceaccount:neuvector:controller
oc create role neuvector-binding-scanner --verb=get,patch,update,watch --resource=deployments -n neuvector
oc adm policy add-role-to-user neuvector-binding-scanner system:serviceaccount:neuvector:updater system:serviceaccount:neuvector:controller -n neuvector --role-namespace neuvector
oc create clusterrole neuvector-binding-csp-usages --verb=get,create,update,delete --resource=cspadapterusagerecords
oc adm policy add-cluster-role-to-user neuvector-binding-csp-usages system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-co --verb=get,list --resource=clusteroperators
oc adm policy add-cluster-role-to-user neuvector-binding-co system:serviceaccount:neuvector:enforcer system:serviceaccount:neuvector:controller

7. Run the following command to check if the neuvector/controller, neuvector/enforcer and neuvector/updater service accounts are added successfully. 

$ oc get ClusterRoleBinding neuvector-binding-app neuvector-binding-rbac neuvector-binding-admission neuvector-binding-customresourcedefinition neuvector-binding-nvsecurityrules neuvector-binding-view neuvector-binding-nvwafsecurityrules neuvector-binding-nvadmissioncontrolsecurityrules neuvector-binding-nvdlpsecurityrules neuvector-binding-csp-usages neuvector-binding-co -o wide

OpenShift Deployment Examples for NeuVector 

# neuvector yaml version for NeuVector 5.2.x on CRI-O
apiVersion: v1
kind: Service
metadata:
  name: neuvector-svc-crd-webhook
  namespace: neuvector
spec:
  ports:
  - port: 443
    targetPort: 30443
    protocol: TCP
    name: crd-webhook
  type: ClusterIP
  selector:
    app: neuvector-controller-pod

---

apiVersion: v1
kind: Service
metadata:
  name: neuvector-svc-admission-webhook
  namespace: neuvector
spec:
  ports:
  - port: 443
    targetPort: 20443
    protocol: TCP
    name: admission-webhook
  type: ClusterIP
  selector:
    app: neuvector-controller-pod

---

apiVersion: v1
kind: Service
metadata:
  name: neuvector-service-webui
  namespace: neuvector
spec:
  ports:
    - port: 8443
      name: manager
      protocol: TCP
  type: ClusterIP
  selector:
    app: neuvector-manager-pod

---

apiVersion: v1
kind: Service
metadata:
  name: neuvector-svc-controller
  namespace: neuvector
spec:
  ports:
  - port: 18300
    protocol: "TCP"
    name: "cluster-tcp-18300"
  - port: 18301
    protocol: "TCP"
    name: "cluster-tcp-18301"
  - port: 18301
    protocol: "UDP"
    name: "cluster-udp-18301"
  clusterIP: None
  selector:
    app: neuvector-controller-pod

---

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: neuvector-route-webui
  namespace: neuvector
spec:
  to:
    kind: Service
    name: neuvector-service-webui
  port:
    targetPort: manager
  tls:
    termination: passthrough

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neuvector-manager-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-manager-pod
  replicas: 1
  template:
    metadata:
      labels:
        app: neuvector-manager-pod
    spec:
      serviceAccountName: basic
      serviceAccount: basic
      containers:
        - name: neuvector-manager-pod
          image: image-registry.openshift-image-registry.svc:5000/neuvector/manager:<version>
          env:
            - name: CTRL_SERVER_IP
              value: neuvector-svc-controller.neuvector
      restartPolicy: Always

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neuvector-controller-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-controller-pod
  minReadySeconds: 60
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 3
  template:
    metadata:
      labels:
        app: neuvector-controller-pod
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - neuvector-controller-pod
              topologyKey: "kubernetes.io/hostname"
      serviceAccountName: controller
      serviceAccount: controller
      containers:
        - name: neuvector-controller-pod
          image: image-registry.openshift-image-registry.svc:5000/neuvector/controller:<version>
          securityContext:
            privileged: true
          readinessProbe:
            exec:
              command:
              - cat
              - /tmp/ready
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: CLUSTER_JOIN_ADDR
              value: neuvector-svc-controller.neuvector
            - name: CLUSTER_ADVERTISED_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CLUSTER_BIND_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /var/neuvector
              name: nv-share
              readOnly: false
            - mountPath: /var/run/crio/crio.sock
              name: runtime-sock
              readOnly: true
            - mountPath: /host/proc
              name: proc-vol
              readOnly: true
            - mountPath: /host/cgroup
              name: cgroup-vol
              readOnly: true
            - mountPath: /etc/config
              name: config-volume
              readOnly: true
      terminationGracePeriodSeconds: 300
      restartPolicy: Always
      volumes:
        - name: nv-share
          hostPath:
            path: /var/neuvector
        - name: runtime-sock
          hostPath:
            path: /var/run/crio/crio.sock
        - name: proc-vol
          hostPath:
            path: /proc
        - name: cgroup-vol
          hostPath:
            path: /sys/fs/cgroup
        - name: config-volume
          projected:
            sources:
              - configMap:
                  name: neuvector-init
                  optional: true
              - secret:
                  name: neuvector-init
                  optional: true

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: neuvector-enforcer-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-enforcer-pod
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: neuvector-enforcer-pod
    spec:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
      hostPID: true
      serviceAccountName: enforcer
      serviceAccount: enforcer
      containers:
        - name: neuvector-enforcer-pod
          image: image-registry.openshift-image-registry.svc:5000/neuvector/enforcer:<version>
          securityContext:
            privileged: true
          env:
            - name: CLUSTER_JOIN_ADDR
              value: neuvector-svc-controller.neuvector
            - name: CLUSTER_ADVERTISED_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CLUSTER_BIND_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /lib/modules
              name: modules-vol
              readOnly: true
            - mountPath: /var/run/crio/crio.sock
              name: runtime-sock
              readOnly: true
            - mountPath: /host/proc
              name: proc-vol
              readOnly: true
            - mountPath: /host/cgroup
              name: cgroup-vol
              readOnly: true
      terminationGracePeriodSeconds: 1200
      restartPolicy: Always
      volumes:
        - name: modules-vol
          hostPath:
            path: /lib/modules
        - name: runtime-sock
          hostPath:
            path: /var/run/crio/crio.sock
        - name: proc-vol
          hostPath:
            path: /proc
        - name: cgroup-vol
          hostPath:
            path: /sys/fs/cgroup

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neuvector-scanner-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-scanner-pod
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 2
  template:
    metadata:
      labels:
        app: neuvector-scanner-pod
    spec:
      serviceAccountName: basic
      serviceAccount: basic
      containers:
        - name: neuvector-scanner-pod
          image: image-registry.openshift-image-registry.svc:5000/neuvector/scanner:latest
          imagePullPolicy: Always
          env:
            - name: CLUSTER_JOIN_ADDR
              value: neuvector-svc-controller.neuvector
      restartPolicy: Always

---

apiVersion: batch/v1
kind: CronJob
metadata:
  name: neuvector-updater-pod
  namespace: neuvector
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: neuvector-updater-pod
        spec:
          serviceAccountName: updater
          serviceAccount: updater
          containers:
          - name: neuvector-updater-pod
            image: image-registry.openshift-image-registry.svc:5000/neuvector/updater:latest
            imagePullPolicy: Always
            command:
            - /bin/sh
            - -c
            - TOKEN=`cat /var/run/secrets/kubernetes.io/serviceaccount/token`; /usr/bin/curl -kv -X PATCH -H "Authorization:Bearer $TOKEN" -H "Content-Type:application/strategic-merge-patch+json" -d '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'`date +%Y-%m-%dT%H:%M:%S%z`'"}}}}}' 'https://kubernetes.default/apis/apps/v1/namespaces/neuvector/deployments/neuvector-scanner-pod'
          restartPolicy: Never

8. 修改 Sample Yaml 檔

8-1. 修改 Image Registry 的位置 8-2. 修改 CRI-O 的位置，如果 CRI-O Socket 的位置是 /var/run/crio/crio.sock，則不需修改 8-3. 新增多餘的 Taints，除了

8-4. 修改 neuvector-service-webui 這個 Service 的 type 為 NodePort

9. 建立 Neuvector 

$ oc create -f neuvector.yaml

螢幕輸出 :

service/neuvector-svc-crd-webhook created
service/neuvector-svc-admission-webhook created
service/neuvector-service-webui created
service/neuvector-svc-controller created
route.route.openshift.io/neuvector-route-webui created
deployment.apps/neuvector-manager-pod created
deployment.apps/neuvector-controller-pod created
daemonset.apps/neuvector-enforcer-pod created
deployment.apps/neuvector-scanner-pod created
cronjob.batch/neuvector-updater-pod created

10. 檢視 Neuvector 運作狀態 

$ oc get pods,svc -o wide

螢幕輸出 :

NAME                                           READY   STATUS    RESTARTS   AGE   IP             NODE                       NOMINATED NODE   READINESS GATES
pod/neuvector-controller-pod-6655ddf6b-kf69d   1/1     Running   0          10m   10.129.1.168   worker2.ocp4.example.com   <none>           <none>
pod/neuvector-controller-pod-6655ddf6b-l89tf   1/1     Running   0          10m   10.130.0.40    worker1.ocp4.example.com   <none>           <none>
pod/neuvector-controller-pod-6655ddf6b-ljhbq   1/1     Running   0          10m   10.129.1.170   worker2.ocp4.example.com   <none>           <none>
pod/neuvector-enforcer-pod-5d2j9               1/1     Running   0          10m   10.130.0.38    worker1.ocp4.example.com   <none>           <none>
pod/neuvector-enforcer-pod-vc6tk               1/1     Running   0          10m   10.129.1.166   worker2.ocp4.example.com   <none>           <none>
pod/neuvector-enforcer-pod-vpg56               1/1     Running   0          10m   10.128.0.249   master.ocp4.example.com    <none>           <none>
pod/neuvector-manager-pod-896b46c9f-bgm8n      1/1     Running   0          10m   10.129.1.169   worker2.ocp4.example.com   <none>           <none>
pod/neuvector-scanner-pod-56c798bb86-8dh2m     1/1     Running   0          10m   10.130.0.39    worker1.ocp4.example.com   <none>           <none>
pod/neuvector-scanner-pod-56c798bb86-kvgrh     1/1     Running   0          10m   10.129.1.167   worker2.ocp4.example.com   <none>           <none>
pod/web                                        1/1     Running   0          47m   10.129.1.163   worker2.ocp4.example.com   <none>           <none>

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE   SELECTOR
service/neuvector-service-webui           NodePort    172.30.139.142   <none>        8443:32744/TCP                  10m   app=neuvector-manager-pod
service/neuvector-svc-admission-webhook   ClusterIP   172.30.188.201   <none>        443/TCP                         10m   app=neuvector-controller-pod
service/neuvector-svc-controller          ClusterIP   None             <none>        18300/TCP,18301/TCP,18301/UDP   10m   app=neuvector-controller-pod
service/neuvector-svc-crd-webhook         ClusterIP   172.30.208.182   <none>        443/TCP                         10m   app=neuvector-controller-p

看 neuvector-manager-pod-*-* 的 Pod 在哪一台 Node 上 以及 neuvector-service-webui 這個 Service 開的 Port 好是多少

11.Access Neuvector WebUI 

https://<worker2.ocp4.example.com 的 IP 位址>:32744

Neuvector Dashboard

Replacing Self-Signed Certificate

Prerequest

The NeuVector web console supports 2 different self-signed certificate types, specifically,

The self-signed certificate can be replaced with either of these PKCS types.

what is difference between “PKCS1” and “PKCS8”?

PKCS is short for “Public Key Cryptography Standards”，中文翻譯為 “公鑰加密標準”

私鑰格式支持的加密算法私鑰檔案內容的開頭
PKCS#1RSA—–BEGIN RSA PRIVATE KEY—–
PKCS#8RSA、DSA、ECDSA 等—–BEGIN PRIVATE KEY—–

Create Certificate Authority

我們需要建立自己的 root CA certificate，讓瀏覽器信任自簽發憑證。

$ mkdir ssl && cd ssl

執行以下 openssl 指令，以建立 rootCA.key 和 rootCA.crt。 請將 SUSE.Lab 替換為您的網域名稱或 IP 位址。

$ openssl req -x509 \
            -sha256 -days 356 \
            -nodes \
            -newkey rsa:2048 \
            -subj "/CN=SUSE.Lab/C=TW/L=Taipei" \
            -keyout rootCA.key -out rootCA.crt

等下會使用 rootCA.key 和 rootCA.crt 來簽署 SSL 憑證。

Create Self-Signed Certificates using OpenSSL

1. Create the Server Private Key 

$ openssl genrsa -out server.key 2048

2. Create Certificate Signing Request Configuration 

$ cat > csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[dn]
C = TW
ST = Taiwan
L = Taipei
O = SUSE
OU = IT Department
emailAddress = admin@example.com
CN = ocp4.example.com

[req_ext]
subjectAltName = @alt_names

[alt_names]
DNS.1 = ocp4.example.com
DNS.2 = *.apps.ocp4.example.com
DNS.3 = api.ocp4.example.com
DNS.4 = api-int.ocp4.example.com
IP.1 = 192.168.11.80
EOF

3. Generate Certificate Signing Request (CSR) Using Server Private Key 

$ openssl req -new -key server.key -out server.csr -config csr.conf

4. Create a external file 

$ cat > cert.conf <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = *.apps.ocp4.example.com
EOF

5. Generate SSL certificate With self signed CA

執行以下指令產生 SSL 憑證，這個憑證是由我們自己的憑證機構所建立的 rootCA.crt 和 rootCA.key 簽署的。

$ openssl x509 -req \
    -in server.csr \
    -CA rootCA.crt -CAkey rootCA.key \
    -CAcreateserial -out server.crt \
    -days 365 \
    -sha256 -extfile cert.conf

上述指令會產生 server.crt 憑證，我們將與 server.key 一起使用這個憑證，以在 Neuvector web console 中啟用 SSL 功能。

Create the secret from the generated key and certificate files

執行以下命令，把要給 Neuvector 用的私鑰和憑證做成 secret

$ oc create secret generic https-cert -n neuvector --from-file=server.key --from-file=server.crt

Edit the yaml directly for the manager and controller deployments to add the mounts 

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - mountPath: /etc/neuvector/certs/ssl-cert.key
          name: cert
          readOnly: true
          subPath: server.key
        - mountPath: /etc/neuvector/certs/ssl-cert.pem
          name: cert
          readOnly: true
          subPath: server.crt
      volumes:
      - name: cert
        secret:
          defaultMode: 420
          secretName: https-cert
# neuvector yaml version for NeuVector 5.2.x on CRI-O
apiVersion: v1
kind: Service
metadata:
  name: neuvector-svc-crd-webhook
  namespace: neuvector
spec:
  ports:
  - port: 443
    targetPort: 30443
    protocol: TCP
    name: crd-webhook
  type: ClusterIP
  selector:
    app: neuvector-controller-pod

---

apiVersion: v1
kind: Service
metadata:
  name: neuvector-svc-admission-webhook
  namespace: neuvector
spec:
  ports:
  - port: 443
    targetPort: 20443
    protocol: TCP
    name: admission-webhook
  type: ClusterIP
  selector:
    app: neuvector-controller-pod

---

apiVersion: v1
kind: Service
metadata:
  name: neuvector-service-webui
  namespace: neuvector
spec:
  ports:
    - port: 443
      targetPort: 8443
      name: manager
      protocol: TCP
  type: ClusterIP
  selector:
    app: neuvector-manager-pod

---

apiVersion: v1
kind: Service
metadata:
  name: neuvector-svc-controller
  namespace: neuvector
spec:
  ports:
  - port: 18300
    protocol: "TCP"
    name: "cluster-tcp-18300"
  - port: 18301
    protocol: "TCP"
    name: "cluster-tcp-18301"
  - port: 18301
    protocol: "UDP"
    name: "cluster-udp-18301"
  clusterIP: None
  selector:
    app: neuvector-controller-pod

---

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: neuvector-route-webui
  namespace: neuvector
spec:
  to:
    kind: Service
    name: neuvector-service-webui
  port:
    targetPort: manager
  tls:
    termination: passthrough

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neuvector-manager-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-manager-pod
  replicas: 1
  template:
    metadata:
      labels:
        app: neuvector-manager-pod
    spec:
      serviceAccountName: basic
      serviceAccount: basic
      containers:
        - name: neuvector-manager-pod
          image: docker.io/neuvector/manager:5.2.0
          env:
            - name: CTRL_SERVER_IP
              value: neuvector-svc-controller.neuvector
          volumeMounts:
          - mountPath: /etc/neuvector/certs/ssl-cert.key
            name: cert
            readOnly: true
            subPath: server.key
          - mountPath: /etc/neuvector/certs/ssl-cert.pem
            name: cert
            readOnly: true
            subPath: server.crt
      volumes:
        - name: cert
          secret:
            defaultMode: 420
            secretName: https-cert
      restartPolicy: Always

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neuvector-controller-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-controller-pod
  minReadySeconds: 60
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 3
  template:
    metadata:
      labels:
        app: neuvector-controller-pod
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - neuvector-controller-pod
              topologyKey: "kubernetes.io/hostname"
      serviceAccountName: controller
      serviceAccount: controller
      containers:
        - name: neuvector-controller-pod
          image: docker.io/neuvector/controller:5.2.0
          securityContext:
            privileged: true
          readinessProbe:
            exec:
              command:
              - cat
              - /tmp/ready
            initialDelaySeconds: 5
            periodSeconds: 5
          env:
            - name: CLUSTER_JOIN_ADDR
              value: neuvector-svc-controller.neuvector
            - name: CLUSTER_ADVERTISED_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CLUSTER_BIND_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /var/neuvector
              name: nv-share
              readOnly: false
            - mountPath: /var/run/crio/crio.sock
              name: runtime-sock
              readOnly: true
            - mountPath: /host/proc
              name: proc-vol
              readOnly: true
            - mountPath: /host/cgroup
              name: cgroup-vol
              readOnly: true
            - mountPath: /etc/config
              name: config-volume
              readOnly: true
            - mountPath: /etc/neuvector/certs/ssl-cert.key
              name: cert
              readOnly: true
              subPath: server.key
            - mountPath: /etc/neuvector/certs/ssl-cert.pem
              name: cert
              readOnly: true
              subPath: server.crt
      terminationGracePeriodSeconds: 300
      restartPolicy: Always
      volumes:
        - name: nv-share
          hostPath:
            path: /var/neuvector
        - name: runtime-sock
          hostPath:
            path: /var/run/crio/crio.sock
        - name: proc-vol
          hostPath:
            path: /proc
        - name: cgroup-vol
          hostPath:
            path: /sys/fs/cgroup
        - name: cert
          secret:
            defaultMode: 420
            secretName: https-cert
        - name: config-volume
          projected:
            sources:
              - configMap:
                  name: neuvector-init
                  optional: true
              - secret:
                  name: neuvector-init
                  optional: true

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: neuvector-enforcer-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-enforcer-pod
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: neuvector-enforcer-pod
    spec:
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          key: node-role.kubernetes.io/control-plane
      hostPID: true
      serviceAccountName: enforcer
      serviceAccount: enforcer
      containers:
        - name: neuvector-enforcer-pod
          image: docker.io/neuvector/enforcer:5.2.0
          securityContext:
            privileged: true
          env:
            - name: CLUSTER_JOIN_ADDR
              value: neuvector-svc-controller.neuvector
            - name: CLUSTER_ADVERTISED_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: CLUSTER_BIND_ADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /lib/modules
              name: modules-vol
              readOnly: true
            - mountPath: /var/run/crio/crio.sock
              name: runtime-sock
              readOnly: true
            - mountPath: /host/proc
              name: proc-vol
              readOnly: true
            - mountPath: /host/cgroup
              name: cgroup-vol
              readOnly: true
      terminationGracePeriodSeconds: 1200
      restartPolicy: Always
      volumes:
        - name: modules-vol
          hostPath:
            path: /lib/modules
        - name: runtime-sock
          hostPath:
            path: /var/run/crio/crio.sock
        - name: proc-vol
          hostPath:
            path: /proc
        - name: cgroup-vol
          hostPath:
            path: /sys/fs/cgroup

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: neuvector-scanner-pod
  namespace: neuvector
spec:
  selector:
    matchLabels:
      app: neuvector-scanner-pod
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 2
  template:
    metadata:
      labels:
        app: neuvector-scanner-pod
    spec:
      serviceAccountName: basic
      serviceAccount: basic
      containers:
        - name: neuvector-scanner-pod
          image: docker.io/neuvector/scanner:latest
          imagePullPolicy: Always
          env:
            - name: CLUSTER_JOIN_ADDR
              value: neuvector-svc-controller.neuvector
      restartPolicy: Always

---

apiVersion: batch/v1
kind: CronJob
metadata:
  name: neuvector-updater-pod
  namespace: neuvector
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: neuvector-updater-pod
        spec:
          serviceAccountName: updater
          serviceAccount: updater
          containers:
          - name: neuvector-updater-pod
            image: docker.io/neuvector/updater:latest
            imagePullPolicy: Always
            command:
            - /bin/sh
            - -c
            - TOKEN=`cat /var/run/secrets/kubernetes.io/serviceaccount/token`; /usr/bin/curl -kv -X PATCH -H "Authorization:Bearer $TOKEN" -H "Content-Type:application/strategic-merge-patch+json" -d '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'`date +%Y-%m-%dT%H:%M:%S%z`'"}}}}}' 'https://kubernetes.default/apis/apps/v1/namespaces/neuvector/deployments/neuvector-scanner-pod'
          restartPolicy: Never

What is Route in OpenShift?

An OpenShift Container Platform route exposes a service at a host name, such as www.example.com, so that external clients can reach it by name.

Route Types

What is a Passthrough Route 

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: neuvector-route-webui
  namespace: neuvector
spec:
  to:
    kind: Service
    name: neuvector-service-webui
  port:
    targetPort: manager
  tls:
    termination: passthrough

建立 Neuvector 

$ oc create -f neuvector.yaml

執行以下命令檢查 Pod 運行狀態和 route

$ oc get pod,route

螢幕輸出 :

NAME                                            READY   STATUS    RESTARTS   AGE
pod/neuvector-controller-pod-76bbbfc5dc-5t8lk   1/1     Running   0          167m
pod/neuvector-controller-pod-76bbbfc5dc-8znrb   1/1     Running   0          167m
pod/neuvector-controller-pod-76bbbfc5dc-t5tbb   1/1     Running   0          167m
pod/neuvector-enforcer-pod-4kgrn                1/1     Running   0          167m
pod/neuvector-enforcer-pod-kwsrr                1/1     Running   0          167m
pod/neuvector-enforcer-pod-lwnkp                1/1     Running   0          167m
pod/neuvector-manager-pod-6bb5555c7b-cx7lr      1/1     Running   0          167m
pod/neuvector-scanner-pod-56c798bb86-fvp8l      1/1     Running   0          167m
pod/neuvector-scanner-pod-56c798bb86-scqjk      1/1     Running   0          167m

NAME                                             HOST/PORT                                               PATH   SERVICES                  PORT      TERMINATION   WILDCARD
route.route.openshift.io/neuvector-route-webui   neuvector-route-webui-neuvector.apps.ocp4.example.com          neuvector-service-webui   manager   passthrough   None

在本機測試 Neuvector web console

  1. 將 CA 憑證複製到指定目錄
$ sudo cp rootCA.crt /etc/pki/ca-trust/source/anchors/
  1. 更新 Linux 系統的 CA (Certificate Authority) 信任我們自己建的 CA 憑證
$ sudo update-ca-trust extract
  1. 驗證我們自己建的 CA 憑證是否由可信任的根憑證機構 (CA) 簽發
$ openssl verify /etc/pki/ca-trust/source/anchors/rootCA.crt

螢幕輸出 :

/etc/pki/ca-trust/source/anchors/rootCA.crt: OK
  1. 訪問 Neuvector web console
$ curl https://neuvector-route-webui-neuvector.apps.ocp4.example.com

oc get route neuvector-route-webui 可以得到 Domain name

螢幕輸出 :

This and all future requests should be directed to <a href="/index.html?v=2358c7ec6b">this URI</a>.

透過 Firefox 瀏覽器連線至 Neuvector web console

參考文件

Neuvector

Self-Signed Certificate

OpenShift Route

NeuVector Onboarding and Best Practices GuidePods status UnexpectedAdmissionError