Antony 筆記：技術原理與實務經驗 – 安全性

KCSA 真題

Sun, 28 Sep 2025 13:51:06 +0800

Question #1

What is Grafana?

A. A cloud-native distributed tracing system for monitoring microservices architectures. B. A container orchestration platform for managing and scaling applications. C. A platform for monitoring and visualizing time-series data. D. A cloud-native security tool for scanning and detecting vulnerabilities in Kubernetes clusters.

Question #2

Which step would give an attacker a foothold in a cluster but no long term persistence?

A. Modify file on host filesystem. B. Create restarting container on host using Docker. C. Modify Kubernetes objects stored within etcd. D. Starting a process in a running container.

Question #3

What is the reasoning behind considering the Cloud as the trusted computing base of a Kubernetes cluster?

A. A vulnerability in the Cloud layer has a negligible impact on containers due to Linux isolation mechanisms. B. A Kubernetes cluster can only be trusted if the underlying Cloud provider is certified against international standards. C. A Kubernetes cluster can only be as secure as the security posture of its Cloud hosting. D. The Cloud enforces security controls at the Kubernetes cluster level, so application developers can focus on applications only.

Question #4

By default, in a Kubeadm cluster, which authentication methods are enabled?

A. OIDC, Bootstrap tokens, and Service Account Tokens

B. X509 Client Certs, Bootstrap Tokens, and Service Account Tokens

C. X509 Client Certs, OIDC, and Service Account Tokens

D. X509 Client Certs, Webhook Authentication, and Service Account Tokens

Question #5

Which technology can be used to apply security policy for internal cluster traffic at the application layer of the network?

A. Container Runtime B. Network Policy C. Service Mesh D. Ingress Controller

Question #6

Which of the following statement regarding a container run with privileged: true is correct?

A. A container run with privileged: true within a Namespace can access all Secrets used within that Namespace. B. A container run with privileged: true within a cluster can access all Secrets used within that cluster. C. A container run with privileged: true has no additional access to Secrets than if it were run with privileged: false. D. A container run with privileged: true on a node can access all Secrets used on that node.

Question #7

Which of the following is a measure for data plane isolation in a Kubernetes multi-tenancy scenario?

A. Enforce Object Count Quotas via the ResourceQuota admission controller. B. Assign a dedicated namespace to Pods from each tenant. C. Assign a dedicated set of workers to run Pods from each tenant. D. Enforce Roles and RoleBindings tied to specific namespaces only, forbid cluster-wide roles.

Question #8

How can a user enforce the Pod Security Standard without third party tools?

A. It is only possible to enforce the Pod Security Standard with additional tools within the cloud native ecosystem. B. Use the PodSecurity admission controller. C. No additional measures have to be taken to enforce the Pod Security Standard. D. Through implementing Kyverno or OPA Policies.

Question #9

Which of the following statements best describe container image signing and verification in the cloud environment?

A. Container image signatures and their verification ensure their authenticity and integrity against tampering. B. Container image signatures affect the performance of containerized applications, as they increase the size of images with additional metadata. C. Container image signatures are concerned with defining developer ownership of applications within multi-tenant environments. D. Container image signatures are mandatory in cloud environments, as cloud providers would deny the execution of unsigned container images.

Question #10

Which other controllers are part of the kube controller manager inside the Kubernetes cluster?

A. Replication controller, Endpoints controller, Namespace controller, and ServiceAccounts controller. B. Namespace controller, ConfigMap controller, and Secret controller. C. Pod, Service, and Ingress controller. D. Job controller, CronJob controller, and DaemonSet controller.

A 參考連結: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

Question #11

On a client machine, what directory (by default) contains sensitive credential information?

A. /opt/kubernetes/secrets/ B. $HOME/.kube C. $HOME/.config/kubernetes/ D. /etc/kubernetes/

Question #12

What mechanism can I use to block unsigned images from running in my cluster?

A. Enabling Admission Controllers to validate image signatures. B. Using PodSecurityPolicy (PSP) to enforce image signing and validation. C. Configuring Container Runtime Interface (CRI) to enforce image signing and validation. D. Using Pod Security Standards (PSS) to enforce validation of signatures.

Question #13

What is the main reason an organization would use a Cloud Workload Protection Platform (CWPP) solution?

A. To manage networking between containerized workloads in the Kubernetes cluster. B. To automate the deployment and management of containerized workloads. C. To protect containerized workloads from known vulnerabilities and malware threats. D. To optimize resource utilization and scalability of containerized workloads.

Question #14

What is the purpose of an egress NetworkPolicy?

A. To secure the Kubernetes cluster against unauthorized access. B. To control the incoming network traffic to a Kubernetes cluster. C. To control the outbound network traffic from a Kubernetes cluster. D. To control the outgoing network traffic from one or more Kubernetes Pods.

Question #16

What information is stored in etcd?

A. Pod data contained in Persistent Volume Claims (e.g., hostPath). B. Application logs and monitoring data for auditing and troubleshooting purposes. C. Etcd manages the configuration data, state data, and metadata for Kubernetes. D. Sensitive user data such as usernames and passwords.

Question #17

A container image is trojanized by an attacker by compromising the build server. Based on the STRIDE threat modeling framework, which threat category best defines this threat?

A. Denial Of Service B. Repudiation C. Tampering D. Spoofing

Question #18

Which label should be added to the Namespace to block any privileged Pods from being created in that Namespace?

A. privileged: true B. pod.security.kubernetes.io/enforce: baseline C. privileged: false D. pod.security.kubernetes.io/privileged: false

Question #19

You are responsible for securing the kubelet component in a Kubernetes cluster. Which of the following statements about kubelet security is correct?

A. Kubelet does not have any built-in security features. B. Kubelet runs as a privileged container by default. C. Kubelet requires root access to interact with the host system. D. Kubelet supports TLS authentication and encryption for secure communication with the API server.

Question #20

Which of the following statements best describes the role of the Scheduler in Kubernetes?

A. The Scheduler is responsible for monitoring and managing the health of the Kubernetes cluster. B. The Scheduler is responsible for ensuring the security of the Kubernetes cluster and its components. C. The Scheduler is responsible for managing the deployment and scaling of applications in the Kubernetes cluster. D. The Scheduler is responsible for assigning Pods to nodes based on resource availability and other constraints.

Question #21

How do Kubernetes namespaces impact the application of policies when using Pod Security Admission?

A. Different policies can be applied to specific namespaces. B. Namespaces are ignored; Pod Security Admission policies apply cluster-wide only. C. The default namespace enforces the strictest security policies by default. D. Each namespace can have only one active policy.

Question #22

Which of the following statements correctly describes a container breakout?

A. A container breakout is the process of escaping the container and gaining access to the cloud provider’s infrastructure. B. A container breakout is the process of escaping the container and gaining access to the host operating system. C. A container breakout is the process of escaping a container when it reaches its resource limits. D. A container breakout is the process of escaping the container and gaining access to the Pod’s network traffic.

Question #23

What was the name of the precursor to Pod Security Standards?

A. Kubernetes Security Context B. Container Security Standards C. Pod Security Policy D. Container Runtime Security

Question #24

Given a standard Kubernetes cluster architecture comprising a single control plane node (hosting both etcd and the control plane as Pods) and three worker nodes, which of the following data flows crosses a trust boundary?

A. From API Server to Container Runtime B. From kubelet to Controller Manager C. From kubelet to API Server D. From kubelet to Container Runtime

Question #25

In Kubernetes, what is Public Key Infrastructure used for?

A. To monitor and analyze performance metrics of a Kubernetes cluster. B. To manage certificates and ensure secure communication in a Kubernetes cluster. C. To automate the scaling of containers in a Kubernetes cluster. D. To manage networking in a Kubernetes cluster.

Question #26

When should soft multitenancy be used over hard multitenancy?

A. When the priority is enabling complete isolation between tenants. B. When the priority is enabling fine-grained control over tenant resources. C. When the priority is enabling resource sharing and efficiency between tenants. D. When the priority is enabling strict security boundaries between tenants.

Question #27

You want to minimize security issues in running Kubernetes Pods. Which of the following actions can help achieve this goal?

A. Sharing sensitive data among Pods in the same cluster to improve collaboration. B. Implement Pod Security standards in the Pod’s YAML configuration. C. Deploying Pods with randomly generated names to obfuscate their identities. D. Running Pods with elevated privileges to maximize their capabilities.

Question #28

Which of the following is a control for Supply Chain Risk Management according to NIST 800-53 Rev. 5?

A. Incident Response B. Supply Chain Risk Management Plan C. Access Control D. System and Communications Protection

Question #29

Which standard approach to security is augmented by the 4C’s of Cloud Native security?

A. Secure-by-Design B. Least Privilege C. Zero Trust D. Defense-in-Depth

Question #30

In a Kubernetes environment, what kind of Admission Controller can modify resource manifests when applied to the Kubernetes API to fix misconfigurations automatically?

A. ValidatingAdmissionController B. PodSecurityPolicy C. MutatingAdmissionController D. ResourceQuota

Question #31

A container running in a Kubernetes cluster has permission to modify host processes on the underlying node. What combination of privileges and capabilities is most likely to have led to this privilege escalation?

A. hostNetwork and NET_RAW B. There is no combination of privileges and capabilities that permits this. C. hostPath and AUDIT_WRITE D. hostPID and SYS_PTRACE

Question #32

What kind of organization would need to be compliant with PCI DSS?

A. Merchants that process credit card payments. B. Retail stores that only accept cash payments. C. Government agencies that collect personally identifiable information. D. Non-profit organizations that handle sensitive customer data.

Question #33

As a Kubernetes and Cloud Native Security Associate, a user can set up audit logging in a cluster. What is the risk of logging every event at the full RequestResponse level?

A. Reduced storage requirements and faster performance. B. Increased storage requirements and potential impact on performance. C. Improved security and easier incident investigation. D. No risk, as it provides the most comprehensive audit trail.

Question #34

In a Kubernetes cluster, what are the security risks associated with using ConfigMaps for storing secrets?

A. Storing secrets in ConfigMaps does not allow for fine-grained access control via RBAC. B. ConfigMaps store sensitive information in etcd encoded in base64 format automatically, which does not ensure confidentiality of data. C. Storing secrets in ConfigMaps can expose sensitive information as they are stored in plaintext and can be accessed by unauthorized users. D. Using ConfigMaps for storing secrets might make applications incompatible with the Kubernetes cluster.

Question #35

In the event that kube-proxy is in a CrashLoopBackOff state, what impact does it have on the Pods running on the same worker node?

A. The Pod cannot mount persistent volumes through CSI drivers. B. The Pod’s resource utilization increases significantly. C. The Pods cannot communicate with other Pods in the cluster. D. The Pod’s security context restrictions cannot be enforced.

Question #36

Which of the following statements on static Pods is true?

A. The kubelet schedules static Pods local to its node without going through the kube-scheduler, making tracking and managing them difficult. B. The kubelet can run a maximum of 5 static Pods on each node. C. The kubelet only deploys static Pods when the kube-scheduler is unresponsive. D. The kubelet can run static Pods that span multiple nodes, provided that it has the necessary privileges from the API server.

Question #37

A user runs a command with kubectl to apply a change to a deployment. What is the first Kubernetes component that the request reaches?

A. kubelet B. Kubernetes Scheduler C. Kubernetes Controller Manager D. Kubernetes API Server

Question #38

Which of the following represents a baseline security measure for containers?

A. Configuring a static IP for each container. B. Run containers as the root user. C. Configuring persistent storage for containers. D. Implementing access control to restrict container access.

Question #39

Which way of defining security policy brings consistency, minimizes toil, and reduces the probability of misconfiguration?

A. Manually configuring security controls for each individual resource, regularly. B. Using a declarative approach to define security policies as code. C. Implementing security policies through manual scripting on an ad-hoc basis. D. Relying on manual audits and inspections for security policy enforcement.

Question #40

Why might NetworkPolicy resources have no effect in a Kubernetes cluster?

A. NetworkPolicy resources are only enforced for unprivileged Pods. B. NetworkPolicy resources are only enforced if the user has the right RBAC permissions. C. NetworkPolicy resources are only enforced if the networking plugin supports them. D. NetworkPolicy resources are only enforced if the Kubernetes scheduler supports them.

Question #41

What is the difference between gVisor and Firecracker?

A. gVisor is a user-space kernel that provides isolation and security for containers. At the same time, Firecracker is a lightweight virtualization technology for creating and managing secure, multi-tenant container and function-as-a-service (FaaS) workloads. B. gVisor is a lightweight virtualization technology for creating and managing secure, multi-tenant container and function-as-a-service (FaaS) workloads. At the same time, Firecracker is a user-space kernel that provides isolation and security for containers. C. gVisor and Firecracker are two names for the same technology, which provides isolation and security for containers. D. gVisor and Firecracker are both container runtimes that can be used interchangeably.

Question #42

An attacker has access to the network segment that the cluster is on. What happens when a compromised Pod attempts to connect to the API server?

A. The compromised Pod is automatically isolated from the network to prevent any connections to the API server. B. The compromised Pod is allowed to connect to the API server without any restrictions. C. The compromised Pod connects to the API server and is granted elevated privileges by default. D. The compromised Pod attempts to connect to the API server, but its requests may be blocked due to network policies.

Question #43

Why does the default base64 encoding that Kubernetes applies to the contents of Secret resources provide inadequate protection?

A. Base64 encoding does not encrypt the contents of the Secret, only obfuscates it. B. Base64 encoding is not supported by all Secret Stores. C. Base64 encoding relies on a shared key which can be easily compromised. D. Base64 encoding is vulnerable to brute-force attacks.

Question #44

A cluster is failing to pull more recent versions of images from k8s.gcr.io. Why may this be?

A. The authentication credentials for accessing k8s.gcr.io are incorrectly scoped. B. There is a network connectivity issue between the cluster and k8s.gcr.io. C. The container image registry k8s.gcr.io has been deprecated. D. There is a bug in the container runtime or the image pull process.

Question #45

A Kubernetes cluster tenant can launch privileged pods in contravention of the restricted Pod Security Standard mandated for cluster tenants and enforced by the built-in PodSecurity admission controller. The tenant has full CRUD permissions on the namespace object and the namespaced resources. How did the tenant achieve this?

A. By deleting the PodSecurity admission controller deployment running in their namespace. B. The scope of the tenant role means privilege escalation is impossible. C. By using higher-level access credentials obtained from another namespace. D. By tampering with the namespace labels.

Question #46

Is it possible to restrict permissions so that a controller can only change the image of a deployment (without changing anything else about it, e.g., environment variables, commands, replicas, secrets)?

A. No, because granting access to the spec.containers.image field always grants access to the rest of the spec object. B. Yes, with a ‘managed fields’ annotation. C. Not with RBAC, but it is possible with an admission webhook. D. Yes, by granting permission to the /image subresource.

Question #47

Which of the following statements is true concerning the use of microVMs over user-space kernel implementations for advanced container sandboxing?

A. MicroVMs offer higher isolation than user-space kernel implementations at the cost of a higher per-instance memory footprint. B. MicroVMs provide reduced application compatibility and higher per-system call overhead than user-space kernel implementations. C. MicroVMs allow for easier container management and orchestration than user-space kernel implementation. D. MicroVMs offer lower isolation and security compared to user-space kernel implementations.

Question #48

In a cluster that contains Nodes with multiple container runtimes installed, how can a Pod be configured to be created on a specific runtime?

A. By setting the container runtime as an environment variable in the Pod. B. By specifying the container runtime in the Pod’s YAML file. C. By using a command-line flag when creating the Pod. D. By modifying the Docker daemon configuration.

Question #49

Which of the following is a valid security risk caused by having no egress controls in a Kubernetes cluster?

A. Data exfiltration. B. Increased attack surface. C. Unauthorized access to external resources. D. Denial of Service.

Question #50

What is the purpose of the Supplier Assessments and Reviews control in the NIST 800-53 Rev. 5 set of controls for Supply Chain Risk Management?

A. To establish contractual agreements with suppliers. B. To identify potential suppliers for the organization. C. To evaluate and monitor existing suppliers for adherence to security requirements. D. To conduct regular audits of suppliers’ financial performance.

Question #51

When using a cloud provider’s managed Kubernetes service, who is responsible for maintaining the etcd cluster?

A. Kubernetes administrator B. Application developer C. Namespace administrator D. Cloud provider

Question #52

In order to reduce the attack surface of the Scheduler, which default parameter should be set to false?

A. –secure-kubeconfig B. –scheduler-name C. –bind-address D. –profiling

Question #53

A cluster administrator wants to enforce the use of a different container runtime depending on the application a workload belongs to. How could a mutating admission controller be configured to achieve this?

A. By manually modifying the container runtime for each workload after it has been created. B. By modifying the kube-apiserver configuration file to specify the desired container runtime for each application. C. By configuring a mutating admission controller webhook that intercepts new workload creation requests and modifies the container runtime based on the application label. D. By configuring a validating admission controller webhook that verifies the container runtime based on the application label and rejects requests that do not comply.

Question #54

An attacker has successfully overwhelmed the Kubernetes API server in a cluster with a single control plane node by flooding it with requests. How would implementing a high-availability mode with multiple control plane nodes mitigate this attack?

A. By implementing rate limiting and throttling mechanisms on the API server to restrict the number of requests allowed. B. By implementing network segmentation to isolate the API server from the rest of the cluster, preventing the attack from spreading. C. By increasing the resources allocated to the API server, allowing it to handle a higher volume of requests. D. By distributing the workload across multiple API servers, reducing the load on each server.

Question #55

In which order are the validating and mutating admission controllers run while the Kubernetes API server processes a request?

A. Validating admission controllers run before mutating admission controllers. B. Mutating admission controllers run before validating admission controllers. C. The order of execution varies and is determined by the cluster configuration. D. Validating and mutating admission controllers run simultaneously.

Question #56

Question: Which security knowledge-base focuses specifically on offensive tools, techniques, and procedures?

A. NIST Cybersecurity Framework B. MITRE ATT&CK C. CIS Controls D. OWASP Top 10

Question #57

Question: What does the ‘cluster-admin’ ClusterRole enable when used in a RoleBinding?

A. It gives full control over every resource in the cluster and in all namespaces. B. It gives full control over every resource in the role binding’s namespace, including the namespace itself. C. It gives full control over every resource in the role binding’s namespace, not including the namespace object for isolation purposes. D. It allows read/write access to most resources in the role binding’s namespace. This role does not allow write access to resource quota, to the namespace itself, and to EndpointSlices (or Endpoints).

Question #58

Question: To restrict the kubelet’s rights to the Kubernetes API, what authorization mode should be set on the Kubernetes API server?

A. AlwaysAllow B. Node C. Webhook D. kubelet

Question #59

Question: What is a multi-stage build?

A. A build process that involves multiple repositories for storing container images. B. A build process that involves multiple developers collaborating on building an image. C. A build process that involves multiple containers running simultaneously to speed up the image creation. D. A build process that involves multiple stages of image creation, allowing for smaller, optimized images.

Question #60

Question: Which information does a user need to verify a signed container image?

A. The image’s SHA-256 hash and the public key of the signing authority. B. The image’s digital signature and the private key of the signing authority. C. The image’s digital signature and the public key of the signing authority. D. The image’s SHA-256 hash and the private key of the signing authority.

Question #61

Which of the following snippets from a RoleBinding correctly associates user bob with Role pod-reader?

subjects:
-kind: User
name: pod-reader
apiGroup:rbac.authorization.k8s.io
roleRef:
kind: Role
name: bob
apiGroup:rbac.authorization.k8s.io

subjects:
kind: User
name: bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup:rbac.authorization.k8s.io

subjects:
kind: Group
name: bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io

subjects:
kind: User
name: bob
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup:rbac.authorization.k8s.io

Question #62

Which of the following is a benefit of signing container images?

A. Prevents container images with vulnerabilities from running B. Preserves the confidentiality of code within the container image C. Ensures the authenticity and integrity of the cotnainer image D. Ensure that container images are free of malicious code

Question #63

Which organisation provides Kubernetes Security Benchmarks for both self-hosted and managed platform

A. The Open Web Application Security Project (OWASP) B. The Cloud Native Computing FOundation (CNCF) C. The Center for Internet Security (CIS) D. The National Institute of Standards and Technology (NIST)

Question #64

An attacker has disabled the log forward for a Kubernetes cluster with the aim of hiding evidence of their malicious activity from incident. Which attack type in STRIDE would this fall under

A. Repudiation B. Spoofing C. Tampering D. Denial of Service

Question #65

A Kubernetes cluster runs on a cloud platform. The platform’s metadata API provides information about the cluster (e.g.). What should be done to mitigate the risk associated with cloud metadata API access?

A. Regularly audit the access logs of the cloud metadata API B. Restrict Pod access to the cloud metadata API via network policies C. Turn off the cloud metadata API D. Ensure that all sensitive data in the cloud metadata API is encrypted

Question #66

Which of the following is a recommendation in the NSA and CISA Kubernetes Hardening Guidance on namespaces?

A. Use the default namespace for all workloads B. Use Pods should not be placed in Kube-system or kube-public C. Share the same namespace for all workloads to improve resource utilization D. Assign a single and unique namespace to each tenant

Question #67

What is the name for the process of assessing the validity of vulnerabilities detected in the code or dependencies of a piece of software

A. Penetration Testing B. Static Code Analysis C. Vulnerability Triage D. Vulnerability Scanning

Question #68

The user is configuring user authentication to a Kubernetes cluster. WHich of the following standards can be used to configure user authentication to a Kubernetes cluster without the use of an authentication webhook?

A. SAML B. OAuth 2.0 C. Kerberos D. OpenID Connect

Question #69

A user is responsible for managing a multi-tenant Kubernetes platform’s security. They are worried about data condifentiality across to different tenants, and the impact of a compromise in case of an attacker eavesdropping traffic at the node level. Why would they implement a service mesh for this specific use case?

A. To improve network security by encrypting traffic between services through mutual TLS B. To enhance observability and monitoring of services within the Kubernetes platform C. To improve the performance and scalability of services within the Kubernetes platform D. To simplify the management and configuration of services in a Kubernetes cluster

Question #70

What is the effect of allowing privilege escalation at the container security context level?

A. Privilege escalation facilitates smoother cluster scaling by allowing Pods to adapt to increased workloads B. Privilege escalation can allow a container to gain more privileges than its parent process, which may be used for exploitation C. Privilege escalation optimizes resource utilization within the cluster by enabling Pods to share resources via namespaces D. Privilege escalation ensures better isolation between containers in a Pod, limiting the impact of security breaches

Question #71

What is a trust boundary in the context of Kubernetes

A. The boundary between two components where trust levels change B. The boundary between two Services defined by different teams C. The boundary between two Nodes in different subnets D. The boundary between two Pods from different applications

Question #72

Which explicitly describes the respective security and operations obligations of both the cloud provider and the cloud user?

A. Segregation of Duties Model B. Least Privilege Model C. Shared Responsibility Model D. Zero Trust Model

Question #73

A malicious actor with root access on a node wants to run a container without it being visible in the Kubernetes API. How can attacker achieve this?

A. By Exploiting a vulnberability in the Kubernetes control plane components B. By running the container using the underlying container runtime C. By creating a static Pod using a filesystem-hosted static Pod manifest D. By leveraging a Kubernetes admission controller to hide the controller

Question #74

A user has a client X.509 certificate with Subject including O=system:masters that lets them authenticate to the Kubernetes API server. What consequence of this?

A. The user will have no access to the Kubernetes cluster B. The user will have read-only access to the Kubernetes cluster C. The user will have limited access to specific namespaces in the Kubernetes cluster D. The user will have full administrator access to the Kubernetes cluster

Question #75

AppArmor profiles preventing access to sensitive resources have been loaded onto a Node, but the Pods running on the Node can still acr resources. How can the Pods be configured correctly constrained by the AppArmor profiles?

A. Edit the Pod YAML files to include the correct AppArmor profile annotations. B. Restart the Pods to apply the profiles to the Pods running on the Node. C. Reload the AppArmor profiles on the Node. D. Enable AppArmor for the Pods running on the Node.

Question #76

In a multi-tenant Kubernetes environment where each team has distinct access requirements and workloads, which combination of s enhance client security and maintain workload isolation?

A. Enable anonymous authentication, enforce mutual TLS (mTLS), and segregate workloads using pod affinities. B. Rely solely on basic authentication with a shared password across all teams, utilize node taints and tolerations, and tur optimize performance. C. Use service account tokens for all external and internal clients, store client credentials in ConfigMaps, and rely on na workload isolation. D. Utilize bearer tokens with a third-party identity service, employ network policies for Pod-level isolation, and leverag interactions.

Question #77

What is the operational impact of a failing kube-controller-manager Pod in the Kubernetes control plane?

A. Pods can no longer be scheduled on the worker node running the failing kube-controller-manager Pod. Other worker nodes are not impacted. B. Due to the lack of egress network policies, Pods can no longer communicate with external networks. C. The container runtime on the worker node running the failing kube-controller-manager Pod evicts all existing containers from the same node. Other worker nodes are not impacted. D. The cluster does not correctly manage the life-cycle of cluster resources, which makes it subject to a Denial of Service.

Question #78

In a Kubernetes environment, how do Linux namespaces ensure the isolation of containers?

A. Linux namespaces ensure that processes within containers are only granted specific privileges without granting all the B. Linux namespaces provide a mechanism for isolating groups of Kubernetes resources within a single cluster, includir Services. C. Linux namespaces provide process isolation of filesystems, network resources, process IDs, and other Operating System separate namespaces for processes within containers. D. Linux namespaces constrain resources that are allocated to processes within containers, such as CPU/memory

Question #79

In Kubernetes, which of the following conditions must be met for a user to be able to create or update a Role?

A. The user is granted a role to create/update RoleBinding or ClusterRoleBinding objects. B. The user is granted explicit permission to perform the ‘bind’ verb on the roles or ClusterRoles resource in the rbac.authenticate.kubernetes.io API group. C. The user is granted explicit permission to perform the ’escalate’ verb on its own User resource in the authenticate.kubernetes.io D. The user already has all the permissions contained in the role at the same scope as the object being modified.

Question #80

In the STRIDE threat modelling framework, what does the letter D stand for?

A. Denial of Service B. Data Tampering C. Disclosure D. Deception

Question #81

A user is responsible for securing their Kubernetes pipeline and ensuring that the components are signed, and their signatures are validated. What is the purpose of pipeline attestation?

A. Pipeline attestation verifies the integrity of the software supply chain and ensures that the components have not been breaked B. Pipeline attestation is a process of automatically deploying and testing new versions of components in the pipeline C. Pipeline attestation is a security feature that encrypts the communication between the pipeline components. D. Pipeline attestation ensures that all components in the pipeline are up to date and have the latest security patches

Question #82

Which components should be able to access etcd at the network level directly?

A. Only worker nodes in the cluster. B. All Pods running in the cluster. C. Only the Kubernetes API server. D. All Kubernetes control plane components.

Question #83

A user needs to maintain the audit policy of a Kubernetes cluster and wants to make sure that they log the most information in regard to. Which level do they select for the Pod resource?

A. RequestResponse B. RequestResponseMetadata C. Request D. Metadata

Question #84

Why is setting resource limits and requests for Kubernetes pods important to prevent internal Denial of Service scenarios?

A. To optimize the network performance of the cluster B. To ensure even distribution of storage resources among pods C. To prevent a single pod from consuming excessive resources, impacting overall cluster stability D. To facilitate rapid scaling of applications in response to demand

Question #85

What is the primary security advantage of using client certificate authentication for Kubernetes API server access?

A. To enable rapid deployment of Kubernetes resources B. To provide a secure method of authentication that minimizes the risk of credential theft C. To optimize the network traffic flow within the Kubernetes cluster D. To facilitate the integration of Kubernetes with cloud-based storage services

Question #86

Which of the following is a key benefit of implementing a Service Mesh in a cloud-native application architecture from a security perspective?

A. Automatic scaling of application resources B. Simplified service discovery and load balancing C. Enhanced traffic encryption and fine-grained access control D. Reduced need for application-level logging

Question #87

What is a key security measure for the Kubernetes Controller Manager to prevent unauthorized control of cluster components?

A. Configuring Horizontal Pod Autoscaling for the Controller Manager B. Enabling Role-Based Access Control (RBAC) for the Controller Manager’s operations C. Implementing network policies specific to the Controller Manager D. Regularly updating container runtime environments in the cluster

Question #88

Why is it important to enable and configure Kubelet client certificate rotation in a Kubernetes cluster?

A. To improve the performance efficiency of the Kubelet B. To ensure efficient resource allocation by the Kubelet C. To facilitate smoother upgrades of Kubelet software D. To enhance security through regular updating of authentication credentials

Question #89

What is the role of a Certificate Authority (CA) in the PKI setup of a Kubernetes cluster?

A. To provide load balancing between different Kubernetes services B. To manage the distribution of secrets within the Kubernetes cluster C. To issue and manage digital certificates for secure communication within the cluster D. To optimize network traffic routing within the Kubernetes environment

Question #90

In Kubernetes, why is it important to configure Pod Security Admissions to audit mode?

A. To automatically scale pods based on the detected security risk B. To evaluate the impact of proposed security policies without enforcing them C. To enhance the data throughput performance of pods D. To manage the allocation of persistent storage resources to pods

Question #91

Why is pod isolation crucial in a multi-tenant Kubernetes environment?

A. To optimize the load balancing of network traffic among different tenants B. To manage the allocation of storage resources to different tenant namespaces C. To prevent a compromised or malicious pod in one tenant environment from affecting others D. To ensure uniform resource utilization across different tenant environments

Question #92

Why is the secure management of secrets in persistent storage important in Kubernetes threat modeling?

A. To facilitate seamless scaling of applications based on storage demands B. To prevent unauthorized access to sensitive information like credentials and keys stored as secrets C. To optimize the load balancing across different storage resources D. To improve the network communication efficiency between storage volumes and pods

Question #93

Which practice is most effective in securing workload and application code in a cloud-native environment against unauthorized access and data breaches?

A. Implementing auto-scaling for workloads B. Performing regular network penetration testing C. Enforcing role-based access control (RBAC) at the application level D. Using container orchestration tools for deployment automation

Question #94

What is a crucial measure in Kubernetes to secure communication and data transfer, thereby mitigating the risk posed by an attacker on the network?

A. Increasing the number of pod replicas to distribute network traffic B. Implementing network policies to control and restrict pod-to-pod communication C. Automating pod deployments to reduce configuration errors D. Using load balancers to evenly distribute incoming network traffic

Question #95

Why is restricting access to Kubernetes’ cluster-level resources critical in mitigating the risk of privilege escalation?

A. To enhance the performance of cluster components B. To optimize resource distribution among different cluster services C. To prevent unauthorized users or compromised applications from gaining extensive control over the cluster D. To facilitate seamless communication between cluster services

Question #96

Which security practice is most effective in preventing unauthorized or compromised container images from being deployed in a Kubernetes environment?

A. Regularly updating Kubernetes and its dependencies B. Implementing continuous integration/continuous deployment (CI/CD) pipelines C. Enforcing image scanning and vulnerability assessment D. Using network policies to control pod-to-pod communication

Question #97

What is the primary security benefit of isolating Kubernetes service traffic using a Service Mesh like Istio or Linkerd?

A. To enhance the auto-scaling capabilities of services based on traffic B. To enable automatic routing and load balancing of service requests C. To provide encrypted communication and fine-grained access control between services D. To improve the efficiency of resource utilization among services

Question #98

Why is setting Pod Security Admissions (PSA) to ‘audit’ mode beneficial during the initial implementation phase?

A. To improve the data processing efficiency within the cluster B. To manage the allocation of resources based on security policy evaluation C. To record policy violations for assessment, allowing administrators to fine-tune security policies before enforcement D. To optimize the distribution of network traffic among microservices

Question #99

Why is configuring access controls on a Kubernetes image repository crucial for platform security?

A. To optimize the distribution of images across multiple clusters B. To ensure only authorized personnel and systems can push or pull images C. To enhance the speed of image deployment to production environments D. To facilitate the integration of the repository with external monitoring tools

Question #100

When assessing the security of a cloud-native infrastructure, which of the following would not typically be the direct responsibility of the cloud service provider?

A. Physical security of data center facilities B. Patching the host operating system of virtual machines C. Encrypting data at rest within the customer’s application D. Ensuring the availability of network infrastructure

Question #101

In Kubernetes, why is it important to restrict the use of host network and host PID namespaces in pods?

A. To improve the load balancing of network traffic among pods B. To enhance the isolation of pods from the underlying host system C. To facilitate better resource allocation to pods D. To streamline the deployment process of pods

Question #102

What is an effective security measure in Kubernetes to mitigate risks from an attacker present on the network?

A. Implementing resource quotas to limit the use of cluster resources by pods B. Encrypting data in transit between pods and services within the cluster C. Automating the deployment process of pods and services D. Utilizing load balancers to distribute network traffic evenly across pods

Question #103

What is the primary purpose of enabling mutual TLS (mTLS) authentication on the Kubernetes API Server?

A. To increase the processing speed of API requests B. To ensure load balancing of API requests across multiple servers C. To provide encrypted and authenticated communication between clients and the API server D. To facilitate automatic scaling of the API server based on traffic

Question #104

Which component of the 4Cs of Cloud Native Security is directly responsible for implementing security policies and controls within the application code to prevent vulnerabilities?

A. Code Security B. Container Security C. Cloud Infrastructure Security D. Cluster Security

Question #105

Why is the ‘Restricted’ policy in Kubernetes Pod Security Standards important for handling sensitive workloads?

A. To optimize resource allocation for sensitive applications B. To implement strict security constraints that minimize the risk of security breaches in sensitive applications C. To increase the computational speed of applications handling sensitive data D. To ensure high availability of sensitive applications in the cluster

Question #106

In the context of Kubernetes cluster security, how can the Scheduler component be secured to prevent unauthorized scheduling of pods?

A. By implementing Horizontal Pod Autoscaling for the Scheduler B. By applying Network Policies specific to the Scheduler C. By enforcing Role-Based Access Control (RBAC) for scheduling decisions D. By using a Service Mesh to manage traffic to the Scheduler

Question #107

Pod Root access is uing what PID

A. 0 B. 1000 C. 1001 D. 5000

Question #108

Use images scanning is setting in?

A. CRI B. PodSecurityPolicy (PSP) C. PodSecurityContext (PSC) D. Pod Security Admission (PSA)

Question #109

Signed container images pull from private registry is what permission for security.

A. Read only B. Read Write C. No Need set any permission D. Write only

Question #110

POD Admission controller take action in what stage?

A. Before authentication Before Authorization B. Before authentication After Authorization C. After authentication Before Authorization D. After authentication After Authorization

Question #111

API server send request to kubelet use what certificate

A. Use root certificates B. Use CA certificates C. Use server certificates D. Use client certificates

2025 考試實戰心得

多的考題

STRIDE 的 D 是什麼
trust boundary 的描述定義(題目給好幾個定義描述去挑)，原本題庫也有，但換了個問法，關鍵在於跨兩個安全信任等級不同的區域
考 Networkpolicy egress 的 yaml 寫法，要讓兩個 pod 能夠通
NIST 800-53 的定義要多了解一下，它會換個方式問
cloud vender 跟客戶對於 K8S 的安全性，怎麼分責的定義(題目給好幾個定義描述去挑)
有一個人叫 Bob，他的 role 的 yaml 內容會長什麼樣?考 role yaml 的格式，四個選項，選格式是正確的。

認識與實作 Kyverno

Mon, 28 Jul 2025 09:10:06 +0800

Kyverno（希臘語中「治理」的意思）是一個為雲端原生環境設計的策略引擎。它最初是為 Kubernetes 所打造，現在也能在 Kubernetes 叢集之外，作為一個統一的策略語言來使用。

Kyverno 讓平台工程師（platform engineers）能夠自動化處理安全性、合規性及最佳實踐的驗證，並為應用程式團隊（application teams）提供安全的自助服務。

其眾多功能特色包含：

將策略定義為基於 YAML 的宣告式 Kubernetes 資源，無需學習新的語言！
可作為 Kubernetes 的 admission controller、基於 CLI 的掃描器，以及在執行期間（at runtime）強制執行策略。
驗證（validate）、變更（mutate）、產生（generate）或清理（cleanup/remove）任何 Kubernetes 資源。
驗證 container image 及其元數據（metadata），以確保軟體供應鏈（software supply chain）的安全性。
策略可應用於任何 JSON 格式的負載（payload），包含 Terraform 資源、雲端資源及服務授權。
使用來自 CNCF Policy WG（策略工作小組）的開放報告格式來產出策略報告。
彈性的策略例外管理機制。
提供工具對策略進行全面的單元測試（unit testing）與端對端測試（e2e testing）。
使用如 git 和 kustomize 等熟悉的工具，將策略作為程式碼資源（policies as code）來管理。

How Kyverno Works

Kyverno 在 Kubernetes 叢集中作為一個 dynamic admission controller 運行。Kyverno 從 Kubernetes API server 接收 validating 和 mutating admission webhook 的 HTTP 回呼（callbacks），並應用相符的策略來回傳結果，以強制執行准入策略或拒絕請求。

Kyverno 策略可以使用資源的 kind、name、label selectors 等多種方式來匹配資源。

Mutating policies（變更型策略）可以寫成 overlays（類似於 Kustomize）的形式，或是 RFC 6902 JSON Patch 的格式。Validating policies（驗證型策略）也使用 overlay 風格的語法，並支援模式匹配（pattern matching）和條件式（if-then-else）處理。

策略的強制執行情況會透過 Kubernetes events 來記錄。對於那些被允許的請求，或是在 Kyverno 策略被引入前就已存在的資源，Kyverno 會在叢集中建立 Policy Reports。這些報告會包含一份持續更新的清單，列出由某個策略匹配到的資源、它們的狀態等等。

下圖顯示了 Kyverno 的高階邏輯架構 :

Webhook 是一個伺服器，負責處理來自 Kubernetes API server 的傳入 AdmissionReview 請求，並將它們傳送給 Engine 進行處理。它由 Webhook Controller 進行動態設定，該控制器會監看已安裝的策略，並修改 webhooks，使其只請求那些被策略所匹配的資源。

Cert Renewer 負責監看並更新 webhook 所需的憑證，這些憑證以 Kubernetes Secrets 的形式儲存。

Background Controller 透過協調（reconciling）一個名為 UpdateRequests 的中介資源，來處理所有 generate（產生）和 mutate-existing（變更現有）的策略。

而 Report Controllers 則負責從它們的中介資源——Admission Reports 和 Background Scan Reports——來建立及協調 Policy Reports。

Kyverno 也支援高可用性（high availability）。一個高可用性的 Kyverno 安裝，是指其選擇安裝的 controllers 會被設定為以多個副本（replicas）運行。根據 controller 的不同，額外的 replicas 也可能用來提升 Kyverno 的可擴展性（scalability）。請參閱高可用性頁面，以獲取更多關於各種 Kyverno controllers、它們的元件以及各自如何處理可用性的詳細資訊。

實作 Validating Admission Policy

Sat, 26 Jul 2025 07:03:20 +0800

What is Admission Control in K8s

admission controller 是一段程式碼，它會在請求經過認證 (authenticated) 與授權 (authorized) 之後，但在資源物件被永存到 etcd 之前，攔截傳送到 Kubernetes API server 的請求。

Kubernetes 的許多重要功能都必須啟用 admission controller 才能獲得妥善的支援。因此，如果一個 Kubernetes API server 沒有配置一組正確的 admission controllers，那它就是一個不完整的伺服器，而且無法支援您所期望的所有功能。

它們是什麼？

Admission controllers 是 Kubernetes API server 內的一段程式碼，它會檢查為了修改資源而傳入的請求中的資料。

Admission controllers 適用於建立 (create)、刪除 (delete) 或修改 (modify) 物件的請求。Admission controllers 也可以阻擋自訂的動詞 (verb)，例如透過 API server proxy 連接到一個 pod 的請求。Admission controllers 不會（也無法）阻擋讀取（get、watch 或 list）物件的請求，因為讀取操作會繞過准入控制層 (admission control layer)。

准入控制機制 (Admission control mechanisms) 可能是驗證型 (validating)、變更型 (mutating)，或兩者皆是。變更型 (Mutating) 的 controller 可以修改正在被變更的資源資料；驗證型 (validating) 的 controller 則不行。

在 Kubernetes 1.33 中，admission controllers 包含列表中的項目，它們被編譯進 kube-apiserver 的二進位檔中，並且只能由叢集管理員進行設定。

准入控制擴充點 (Admission control extension points)

在完整的清單中，有三個特殊的 controller：

MutatingAdmissionWebhook
ValidatingAdmissionWebhook
ValidatingAdmissionPolicy

這兩個 webhook controller 會（分別）執行在 API 中設定好的變更型 (mutating) 和驗證型 (validating) 的 admission control webhooks。而 ValidatingAdmissionPolicy 提供了一種方法，可以將宣告式的驗證碼嵌入到 API 之中，而無需依賴任何外部的 HTTP 呼叫。

您可以使用這三個 admission controller，在准入階段客製化叢集的行為。

准入控制階段 (Admission control phases)

准入控制的過程分為兩個階段 :

第一個階段會執行變更型 (mutating) 的 admission controllers。
第二個階段則執行驗證型 (validating) 的 admission controllers。

請再次注意，有些 controller 同時具備這兩種功能。如果在任一階段中有任何一個 controller 拒絕了請求，則整個請求會被立即駁回，並將錯誤回傳給終端使用者。

最後，admission controllers 除了有時會變更目標物件之外，也可能產生副作用 (side effects)；也就是說，在處理請求的過程中變更相關的資源。增加配額 (quota) 的用量，就是說明為何需要此機制的典型範例。任何這類的副作用都需要一個對應的回收 (reclamation) 或調節 (reconciliation) 過程，因為任何一個特定的 admission controller 都無法確信該請求最終會通過所有其他的 admission controllers。

這些呼叫的順序可參考下方說明 :

What is Validating Admission Policy?

Validating admission policies 使用 Common Expression Language（CEL）來宣告 policy 的驗證規則。Validating admission policies 具有高度的可配置性，使 policy 撰寫者能夠定義可參數化並依照叢集管理員需求套用到特定資源的 policies。

Validating admission policies 用以取代 validating admission webhooks。

構成一個 VAP 策略需要哪些資源

一個策略 (policy) 通常由三種資源組成：

ValidatingAdmissionPolicy 描述了一個策略的抽象邏輯（可以想成：「這個策略確保某個特定的 label 被設定為一個特定的值」）。
一個參數資源 (parameter resource) 提供資訊給 ValidatingAdmissionPolicy，使其成為一個具體的陳述（可以想成：「owner 這個 label 的值必須以 .company.com 結尾」）。一個原生型別（例如 ConfigMap 或 CRD）定義了參數資源的結構 (schema)。ValidatingAdmissionPolicy 物件會指明它們期望使用哪種 Kind 作為其參數資源。
ValidatingAdmissionPolicyBinding 將上述的資源連結在一起，並定義其作用範圍。如果您只想要求 Pods 必須設定 owner 這個 label，那麼 binding 就是用來指定此限制的地方。

一個策略至少必須定義一個 ValidatingAdmissionPolicy 和一個對應的 ValidatingAdmissionPolicyBinding 才會生效。

如果一個 ValidatingAdmissionPolicy 不需要透過參數來設定，只要將其 spec.paramKind 欄位留空不指定即可。

Validating Admission Policy 快速開始

建立 ValidatingAdmissionPolicy

mkdir ~/validatingadmissionpolicy/; cd ~/validatingadmissionpolicy/
nano basic-example-policy.yaml

內容如下 :

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: "demo-policy.example.com"
spec:
failurePolicy: Fail
matchConstraints:
resourceRules:
- apiGroups: ["apps"]
apiVersions: ["v1"]
operations: ["CREATE", "UPDATE"]
resources: ["deployments"]
validations:
- expression: "object.spec.replicas <= 5"

apiVersion: admissionregistration.k8s.io/v1
- 這指定了該物件所使用的 Kubernetes API 版本。admissionregistration.k8s.io/v1 是 ValidatingAdmissionPolicy 目前穩定的 API 群組與版本。
kind: ValidatingAdmissionPolicy
- 這表明我們要建立的資源種類是 ValidatingAdmissionPolicy。
metadata:
- name: "demo-policy.example.com"
  - 這是此策略物件的唯一名稱。使用類似網域名稱的格式是一種很好的實踐，可以避免命名衝突。
spec:
- 這是定義策略核心行為的地方。
- failurePolicy: Fail
  - 這個欄位決定了當驗證表達式 (expression) 執行失敗或回傳結果為 false 時該如何處理。
  - Fail (目前設定)：表示請求將被拒絕。這是最常見的設定，用來強制執行策略。
  - Ignore：表示如果驗證失敗，將忽略此策略並放行該請求。
- matchConstraints:
  - 這個區塊用來定義此策略適用於哪些 API 請求。只有符合這裡所有條件的請求才會觸發後續的驗證。
  - resourceRules:
    - apiGroups: ["apps"]：指定目標資源所屬的 API 群組。Deployments 隸屬於 apps 群組。
    - apiVersions: ["v1"]：指定目標資源的 API 版本。
    - operations: ["CREATE", "UPDATE"]: 指定此策略僅在建立和更新資源時觸發。它不會在 DELETE 操作時觸發。
    - resources: ["deployments"]：明確指定此規則只針對 deployments 這種資源。
- validations:
  - 這個區塊包含了實際的驗證邏輯。
  - expression: "object.spec.replicas <= 5"
    - 這是一段使用 Common Expression Language (CEL) 撰寫的表達式。
    - object：代表正在被驗證的物件本身（在此情境下就是那個 Deployment）。
    - object.spec.replicas：存取該 Deployment 物件的 spec 欄位下的 replicas 數量。
    - <= 5：檢查 replicas 的值是否小於或等於 5。
    - 如果此表達式回傳 true，請求通過驗證。如果回傳 false，則根據 failurePolicy 的設定（Fail）來拒絕請求。

總結與效果

當你將這個 YAML 檔案應用到你的 Kubernetes 叢集後：

✅ 允許的操作：任何使用者試圖建立或更新一個 Deployment，只要其 .spec.replicas 的值小於或等於 5，操作都會成功。
```
# 這個會成功
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment-ok
spec:
 replicas: 3 # <= 5
 ...
```
❌ 被阻擋的操作：如果有人試圖將 replicas 設定為 6 或更高，kubectl apply 或 kubectl create 指令將會立即失敗，並收到來自 API 伺服器的錯誤訊息，說明該操作違反了 demo-policy.example.com 這條策略。
```
# 這個會失敗
apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment-bad
spec:
 replicas: 10 # > 5
 ...
```

可以在 CEL Playground 中快速測試 CEL 表達式。

Pods status UnexpectedAdmissionError

Sat, 17 May 2025 05:18:30 +0800

Environment

K3s: v1.31.3+k3s1
單節點，此節點同時扮演 control-plane, etcd, worker 三種角色

Issue

當 deployment pod resources 中有宣告使用 gpu，在節點重新開機後，原來的 Pod 狀態會變成 UnexpectedAdmissionError 或 Unknown 或 ContainerStatusUnknown，並且會看在 Pod Event 中看到 kubelet 噴出以下訊息

Allocate failed due to no healthy devices present; cannot allocate unhealthy devices nvidia.com/gpu, which is unexpected

但是過幾分鐘後，又有正常 Running 的 Pod

複現問題

建立測試應用

echo 'apiVersion: apps/v1
kind: Deployment
metadata:
name: gpu-deployment-sample
spec:
replicas: 1
selector:
matchLabels:
app: gpu-sample
template:
metadata:
labels:
app: gpu-sample
spec:
containers:
- name: cuda-container
image: quay.io/cloudwalker/alpine
command: ["sh", "-c", "sleep infinity"]
resources:
limits:
nvidia.com/gpu: 1' | kubectl apply -f -

檢視 pod 運作狀態

kubectl get pods -l app=gpu-sample

執行結果

NAME READY STATUS RESTARTS AGE
gpu-deployment-sample-7dd75997f5-qv9l8 1/1 Running 0 22s

重開機

reboot

再次檢視運作狀態

kubectl get pods -l app=gpu-sample

執行結果 :

NAME READY STATUS RESTARTS AGE
gpu-deployment-sample-7dd75997f5-qv9l8 0/1 Unknown 0 46m
gpu-deployment-sample-7dd75997f5-stbjz 1/1 Running 0 42m

檢視 Event

kubectl get events \
--field-selector involvedObject.name=gpu-deployment-sample-7dd75997f5-qv9l8 \
--sort-by='.lastTimestamp'

執行結果 :

LAST SEEN TYPE REASON OBJECT MESSAGE
49m Normal Scheduled pod/gpu-deployment-sample-7dd75997f5-qv9l8 Successfully assigned default/gpu-deployment-sample-7dd75997f5-qv9l8 to testing
49m Normal Pulling pod/gpu-deployment-sample-7dd75997f5-qv9l8 Pulling image "quay.io/cloudwalker/alpine"
49m Normal Pulled pod/gpu-deployment-sample-7dd75997f5-qv9l8 Successfully pulled image "quay.io/cloudwalker/alpine" in 705ms (705ms including waiting). Image size: 3751656 bytes.
49m Normal Created pod/gpu-deployment-sample-7dd75997f5-qv9l8 Created container cuda-container
49m Normal Started pod/gpu-deployment-sample-7dd75997f5-qv9l8 Started container cuda-container
44m Warning UnexpectedAdmissionError pod/gpu-deployment-sample-7dd75997f5-qv9l8 Allocate failed due to no healthy devices present; cannot allocate unhealthy devices nvidia.com/gpu, which is unexpected
44m Warning FailedMount pod/gpu-deployment-sample-7dd75997f5-qv9l8 MountVolume.SetUp failed for volume "kube-api-access-ns48f" : object "default"/"kube-root-ca.crt" not registered

查看 rs 狀態

kubectl get rs -l app=gpu-sample -o wide

執行結果 :

NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
gpu-deployment-sample-7dd75997f5 1 1 1 73m cuda-container quay.io/cloudwalker/alpine app=gpu-sample,pod-template-hash=7dd75997f5

Offline Install Neuvector 5.2.0 on Openshift Container Platform 4.12

Mon, 09 Sep 2024 02:29:36 +0800

Preface

本篇文章會介紹，

如何在 OpenShift Container Platform 4.12 上安裝 Neuvector 5.2.0
設定 Neuvector web console 使用自簽憑證 ( Self-Signed Certificate )

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

安裝前環境準備

1. 確認 Image Registry 的位址，並再修改 Image 的 Tag 後，將 Image 上傳至 Image Registry

要確定在叢集中的每一台 Node 都可以至該 Image Registry Pull Image 下來

docker login -u <user_name> -p `oc whoami -t` docker-registry.default.svc:5000
docker tag docker.io/neuvector/enforcer:<version> docker-registry.default.svc:5000/neuvector/enforcer:<version>
docker tag docker.io/neuvector/controller:<version> docker-registry.default.svc:5000/neuvector/controller:<version>
docker tag docker.io/neuvector/manager:<version> docker-registry.default.svc:5000/neuvector/manager:<version>
docker tag docker.io/neuvector/scanner docker-registry.default.svc:5000/neuvector/scanner
docker tag docker.io/neuvector/updater docker-registry.default.svc:5000/neuvector/updater
docker push docker-registry.default.svc:5000/neuvector/enforcer:<version>
docker push docker-registry.default.svc:5000/neuvector/controller:<version>
docker push docker-registry.default.svc:5000/neuvector/manager:<version>
docker push docker-registry.default.svc:5000/neuvector/scanner
docker push docker-registry.default.svc:5000/neuvector/updater
docker logout docker-registry.default.svc:5000

2. 檢查環境有無 defaultNodeSelector

$ oc get scheduler cluster -o yaml | grep -A10 defaultNodeSelector

如果沒有螢幕輸出，則跳過本步驟

螢幕輸出 :

spec:
defaultNodeSelector: node-role.kubernetes.io/worker=

如果有類似以上螢幕輸出，則使用以下解決辦法

解決辦法

$ oc annotate namespace neuvector openshift.io/node-selector=

如果 Neuvector 的 Enforcer DaemonSets 已建立，則須刪除 Pod，讓 Pod 重建

$ oc delete pod -l app=neuvector-enforcer-pod

如果不做本步驟，你可能會再建立完 Neuvector 後，看到 Enforcer 的 Pod 出現 Pending 的狀況，並且看到類似以下的錯誤訊息

0/11 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/11 nodes are available: 1 Preemption is not helpful for scheduling, 10 No preemption victims found for incoming pod.

3. 檢查 CRI-O Socket 的位置

檢查的目的是 Neuvector 的 Controller 和 Enforcer 的 Pod 會需要 mount CRI-O Socket 的位置，所以路徑要對
如果 CRI-O Socket 的位置，不是 /var/run/crio/crio.sock，則需要修改 Neuvector 的 Yaml 檔

$ for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
echo
echo "$i"
echo
cat <<EOF | oc debug node/"$i" 2> /dev/null
chroot /host
ps -eaf | grep kubelet | tr ' ' '\n' | grep -w -- '--container-runtime-endpoint'
EOF
done

螢幕輸出 :

master.ocp4.example.com
--container-runtime-endpoint=/var/run/crio/crio.sock
worker1.ocp4.example.com
--container-runtime-endpoint=/var/run/crio/crio.sock
worker2.ocp4.example.com
--container-runtime-endpoint=/var/run/crio/crio.sock

4. 檢查在叢集中的每一台 Node 上有無額外的 Taints

$ for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
echo
echo "$i"
echo
if [[ -z $(oc get node "$i" -o=jsonpath='{.spec.taints}') ]]; then
echo " No Taints on the $i"
else
echo " $(oc get node "$i" -o=jsonpath='{.spec.taints}')"
fi
done

螢幕輸出 :

master.ocp4.example.com
[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"}]
worker1.ocp4.example.com
No Taints on the worker1.ocp4.example.com
worker2.ocp4.example.com
No Taints on the worker2.ocp4.example.com

如果有出現除了以上 Taint 以外的 Taints 的話，需要更改 Neuvector 的 Yaml 檔裡面 Enforcer Pod 的部分

spec:
template:
spec:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
- effect: NoSchedule
key: mykey
value: myvalue

如果有額外的 Taints 請在紅字的地方添加。這需要與污點節點上定義的所有污點信息相匹配。否則，Enforcer 的 Pod 將無法在污點節點上部署

一次檢查全部的腳本

$ curl -s https://raw.githubusercontent.com/braveantony/bash-script/main/precheck.sh | bash

#!/bin/bash
# check Env before installing Neuvector on OCP
## check Taint
echo "Starting Check The Taint Info On a Node"
for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
echo
echo "$i"
echo
if [[ -z $(oc get node "$i" -o=jsonpath='{.spec.taints}') ]]; then
echo " No Taints on the $i"
else
echo " $(oc get node "$i" -o=jsonpath='{.spec.taints}')"
fi
done
echo
echo "Note: All taint info must match to schedule Enforcers on nodes"
echo "=================="
## check CRI-O run-time
echo "Starting Check CRI-O run-time"
for i in $(oc get node -o=jsonpath='{.items[*].metadata.name}')
do
echo
echo "$i"
echo
cat <<EOF | oc debug node/"$i" 2> /dev/null
chroot /host
ps -eaf | grep kubelet | tr ' ' '\n' | grep -w -- '--container-runtime-endpoint'
EOF
done
echo "=================="
## check defaultNodeSelector
echo "Starting Check defaultNodeSelector"
if [[ -z $(oc get scheduler cluster -o yaml | grep defaultNodeSelector) ]]; then
echo " No defaultNodeSelector"
else
oc get scheduler cluster -o yaml | grep -A10 defaultNodeSelector
fi
echo "=================="
echo "End Check"

開始安裝 Neuvector

1. Login as a normal user

$ oc login -u <user_name>

2. Create a new project

$ oc new-project neuvector

3. Login as system:admin account

$ oc login -u system:admin

4. Create Service Accounts and Grant Access to the Privileged SCC

$ oc create sa controller -n neuvector
$ oc create sa enforcer -n neuvector
$ oc create sa basic -n neuvector
$ oc create sa updater -n neuvector
$ oc -n neuvector adm policy add-scc-to-user privileged -z controller -z enforcer

5. Create the custom resources (CRD) for NeuVector security rules. For OpenShift 4.6+ (Kubernetes 1.19+):

$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/crd-k8s-1.19.yaml
$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/waf-crd-k8s-1.19.yaml
$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/dlp-crd-k8s-1.19.yaml
$ oc apply -f https://raw.githubusercontent.com/neuvector/manifests/main/kubernetes/5.2.0/admission-crd-k8s-1.19.yaml

6. Add read permission to access the kubernetes API and OpenShift RBACs.

oc create clusterrole neuvector-binding-app --verb=get,list,watch,update --resource=nodes,pods,services,namespaces
oc create clusterrole neuvector-binding-rbac --verb=get,list,watch --resource=rolebindings.rbac.authorization.k8s.io,roles.rbac.authorization.k8s.io,clusterrolebindings.rbac.authorization.k8s.io,clusterroles.rbac.authorization.k8s.io,imagestreams.image.openshift.io
oc adm policy add-cluster-role-to-user neuvector-binding-app system:serviceaccount:neuvector:controller
oc adm policy add-cluster-role-to-user neuvector-binding-rbac system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-admission --verb=get,list,watch,create,update,delete --resource=validatingwebhookconfigurations,mutatingwebhookconfigurations
oc adm policy add-cluster-role-to-user neuvector-binding-admission system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-customresourcedefinition --verb=watch,create,get,update --resource=customresourcedefinitions
oc adm policy add-cluster-role-to-user neuvector-binding-customresourcedefinition system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-nvsecurityrules --verb=list,delete --resource=nvsecurityrules,nvclustersecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvsecurityrules system:serviceaccount:neuvector:controller
oc adm policy add-cluster-role-to-user view system:serviceaccount:neuvector:controller --rolebinding-name=neuvector-binding-view
oc create clusterrole neuvector-binding-nvwafsecurityrules --verb=list,delete --resource=nvwafsecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvwafsecurityrules system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-nvadmissioncontrolsecurityrules --verb=list,delete --resource=nvadmissioncontrolsecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvadmissioncontrolsecurityrules system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-nvdlpsecurityrules --verb=list,delete --resource=nvdlpsecurityrules
oc adm policy add-cluster-role-to-user neuvector-binding-nvdlpsecurityrules system:serviceaccount:neuvector:controller
oc create role neuvector-binding-scanner --verb=get,patch,update,watch --resource=deployments -n neuvector
oc adm policy add-role-to-user neuvector-binding-scanner system:serviceaccount:neuvector:updater system:serviceaccount:neuvector:controller -n neuvector --role-namespace neuvector
oc create clusterrole neuvector-binding-csp-usages --verb=get,create,update,delete --resource=cspadapterusagerecords
oc adm policy add-cluster-role-to-user neuvector-binding-csp-usages system:serviceaccount:neuvector:controller
oc create clusterrole neuvector-binding-co --verb=get,list --resource=clusteroperators
oc adm policy add-cluster-role-to-user neuvector-binding-co system:serviceaccount:neuvector:enforcer system:serviceaccount:neuvector:controller

7. Run the following command to check if the neuvector/controller, neuvector/enforcer and neuvector/updater service accounts are added successfully.

$ oc get ClusterRoleBinding neuvector-binding-app neuvector-binding-rbac neuvector-binding-admission neuvector-binding-customresourcedefinition neuvector-binding-nvsecurityrules neuvector-binding-view neuvector-binding-nvwafsecurityrules neuvector-binding-nvadmissioncontrolsecurityrules neuvector-binding-nvdlpsecurityrules neuvector-binding-csp-usages neuvector-binding-co -o wide

OpenShift Deployment Examples for NeuVector

# neuvector yaml version for NeuVector 5.2.x on CRI-O
apiVersion: v1
kind: Service
metadata:
name: neuvector-svc-crd-webhook
namespace: neuvector
spec:
ports:
- port: 443
targetPort: 30443
protocol: TCP
name: crd-webhook
type: ClusterIP
selector:
app: neuvector-controller-pod
---
apiVersion: v1
kind: Service
metadata:
name: neuvector-svc-admission-webhook
namespace: neuvector
spec:
ports:
- port: 443
targetPort: 20443
protocol: TCP
name: admission-webhook
type: ClusterIP
selector:
app: neuvector-controller-pod
---
apiVersion: v1
kind: Service
metadata:
name: neuvector-service-webui
namespace: neuvector
spec:
ports:
- port: 8443
name: manager
protocol: TCP
type: ClusterIP
selector:
app: neuvector-manager-pod
---
apiVersion: v1
kind: Service
metadata:
name: neuvector-svc-controller
namespace: neuvector
spec:
ports:
- port: 18300
protocol: "TCP"
name: "cluster-tcp-18300"
- port: 18301
protocol: "TCP"
name: "cluster-tcp-18301"
- port: 18301
protocol: "UDP"
name: "cluster-udp-18301"
clusterIP: None
selector:
app: neuvector-controller-pod
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: neuvector-route-webui
namespace: neuvector
spec:
to:
kind: Service
name: neuvector-service-webui
port:
targetPort: manager
tls:
termination: passthrough
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-manager-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-manager-pod
replicas: 1
template:
metadata:
labels:
app: neuvector-manager-pod
spec:
serviceAccountName: basic
serviceAccount: basic
containers:
- name: neuvector-manager-pod
image: image-registry.openshift-image-registry.svc:5000/neuvector/manager:<version>
env:
- name: CTRL_SERVER_IP
value: neuvector-svc-controller.neuvector
restartPolicy: Always
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-controller-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-controller-pod
minReadySeconds: 60
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
replicas: 3
template:
metadata:
labels:
app: neuvector-controller-pod
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- neuvector-controller-pod
topologyKey: "kubernetes.io/hostname"
serviceAccountName: controller
serviceAccount: controller
containers:
- name: neuvector-controller-pod
image: image-registry.openshift-image-registry.svc:5000/neuvector/controller:<version>
securityContext:
privileged: true
readinessProbe:
exec:
command:
- cat
- /tmp/ready
initialDelaySeconds: 5
periodSeconds: 5
env:
- name: CLUSTER_JOIN_ADDR
value: neuvector-svc-controller.neuvector
- name: CLUSTER_ADVERTISED_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CLUSTER_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- mountPath: /var/neuvector
name: nv-share
readOnly: false
- mountPath: /var/run/crio/crio.sock
name: runtime-sock
readOnly: true
- mountPath: /host/proc
name: proc-vol
readOnly: true
- mountPath: /host/cgroup
name: cgroup-vol
readOnly: true
- mountPath: /etc/config
name: config-volume
readOnly: true
terminationGracePeriodSeconds: 300
restartPolicy: Always
volumes:
- name: nv-share
hostPath:
path: /var/neuvector
- name: runtime-sock
hostPath:
path: /var/run/crio/crio.sock
- name: proc-vol
hostPath:
path: /proc
- name: cgroup-vol
hostPath:
path: /sys/fs/cgroup
- name: config-volume
projected:
sources:
- configMap:
name: neuvector-init
optional: true
- secret:
name: neuvector-init
optional: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: neuvector-enforcer-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-enforcer-pod
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: neuvector-enforcer-pod
spec:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
hostPID: true
serviceAccountName: enforcer
serviceAccount: enforcer
containers:
- name: neuvector-enforcer-pod
image: image-registry.openshift-image-registry.svc:5000/neuvector/enforcer:<version>
securityContext:
privileged: true
env:
- name: CLUSTER_JOIN_ADDR
value: neuvector-svc-controller.neuvector
- name: CLUSTER_ADVERTISED_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CLUSTER_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- mountPath: /lib/modules
name: modules-vol
readOnly: true
- mountPath: /var/run/crio/crio.sock
name: runtime-sock
readOnly: true
- mountPath: /host/proc
name: proc-vol
readOnly: true
- mountPath: /host/cgroup
name: cgroup-vol
readOnly: true
terminationGracePeriodSeconds: 1200
restartPolicy: Always
volumes:
- name: modules-vol
hostPath:
path: /lib/modules
- name: runtime-sock
hostPath:
path: /var/run/crio/crio.sock
- name: proc-vol
hostPath:
path: /proc
- name: cgroup-vol
hostPath:
path: /sys/fs/cgroup
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-scanner-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-scanner-pod
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
replicas: 2
template:
metadata:
labels:
app: neuvector-scanner-pod
spec:
serviceAccountName: basic
serviceAccount: basic
containers:
- name: neuvector-scanner-pod
image: image-registry.openshift-image-registry.svc:5000/neuvector/scanner:latest
imagePullPolicy: Always
env:
- name: CLUSTER_JOIN_ADDR
value: neuvector-svc-controller.neuvector
restartPolicy: Always
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: neuvector-updater-pod
namespace: neuvector
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: neuvector-updater-pod
spec:
serviceAccountName: updater
serviceAccount: updater
containers:
- name: neuvector-updater-pod
image: image-registry.openshift-image-registry.svc:5000/neuvector/updater:latest
imagePullPolicy: Always
command:
- /bin/sh
- -c
- TOKEN=`cat /var/run/secrets/kubernetes.io/serviceaccount/token`; /usr/bin/curl -kv -X PATCH -H "Authorization:Bearer $TOKEN" -H "Content-Type:application/strategic-merge-patch+json" -d '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'`date +%Y-%m-%dT%H:%M:%S%z`'"}}}}}' 'https://kubernetes.default/apis/apps/v1/namespaces/neuvector/deployments/neuvector-scanner-pod'
restartPolicy: Never

8. 修改 Sample Yaml 檔

8-1. 修改 Image Registry 的位置 8-2. 修改 CRI-O 的位置，如果 CRI-O Socket 的位置是 /var/run/crio/crio.sock，則不需修改 8-3. 新增多餘的 Taints，除了

[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"}]
[{"effect":"NoSchedule","key":"node-role.kubernetes.io/controlplane"}]

8-4. 修改 neuvector-service-webui 這個 Service 的 type 為 NodePort

9. 建立 Neuvector

$ oc create -f neuvector.yaml

螢幕輸出 :

service/neuvector-svc-crd-webhook created
service/neuvector-svc-admission-webhook created
service/neuvector-service-webui created
service/neuvector-svc-controller created
route.route.openshift.io/neuvector-route-webui created
deployment.apps/neuvector-manager-pod created
deployment.apps/neuvector-controller-pod created
daemonset.apps/neuvector-enforcer-pod created
deployment.apps/neuvector-scanner-pod created
cronjob.batch/neuvector-updater-pod created

10. 檢視 Neuvector 運作狀態

$ oc get pods,svc -o wide

螢幕輸出 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/neuvector-controller-pod-6655ddf6b-kf69d 1/1 Running 0 10m 10.129.1.168 worker2.ocp4.example.com <none> <none>
pod/neuvector-controller-pod-6655ddf6b-l89tf 1/1 Running 0 10m 10.130.0.40 worker1.ocp4.example.com <none> <none>
pod/neuvector-controller-pod-6655ddf6b-ljhbq 1/1 Running 0 10m 10.129.1.170 worker2.ocp4.example.com <none> <none>
pod/neuvector-enforcer-pod-5d2j9 1/1 Running 0 10m 10.130.0.38 worker1.ocp4.example.com <none> <none>
pod/neuvector-enforcer-pod-vc6tk 1/1 Running 0 10m 10.129.1.166 worker2.ocp4.example.com <none> <none>
pod/neuvector-enforcer-pod-vpg56 1/1 Running 0 10m 10.128.0.249 master.ocp4.example.com <none> <none>
pod/neuvector-manager-pod-896b46c9f-bgm8n 1/1 Running 0 10m 10.129.1.169 worker2.ocp4.example.com <none> <none>
pod/neuvector-scanner-pod-56c798bb86-8dh2m 1/1 Running 0 10m 10.130.0.39 worker1.ocp4.example.com <none> <none>
pod/neuvector-scanner-pod-56c798bb86-kvgrh 1/1 Running 0 10m 10.129.1.167 worker2.ocp4.example.com <none> <none>
pod/web 1/1 Running 0 47m 10.129.1.163 worker2.ocp4.example.com <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/neuvector-service-webui NodePort 172.30.139.142 <none> 8443:32744/TCP 10m app=neuvector-manager-pod
service/neuvector-svc-admission-webhook ClusterIP 172.30.188.201 <none> 443/TCP 10m app=neuvector-controller-pod
service/neuvector-svc-controller ClusterIP None <none> 18300/TCP,18301/TCP,18301/UDP 10m app=neuvector-controller-pod
service/neuvector-svc-crd-webhook ClusterIP 172.30.208.182 <none> 443/TCP 10m app=neuvector-controller-p

看 neuvector-manager-pod-*-* 的 Pod 在哪一台 Node 上以及 neuvector-service-webui 這個 Service 開的 Port 好是多少

11.Access Neuvector WebUI

https://<worker2.ocp4.example.com 的 IP 位址>:32744

預設帳號: admin
預設密碼: admin

Neuvector Dashboard

Replacing Self-Signed Certificate

Prerequest

The NeuVector web console supports 2 different self-signed certificate types, specifically,

PKCS8 (Private-Key Information Syntax Standard)
PKCS1 (RSA Cryptography Standard).

The self-signed certificate can be replaced with either of these PKCS types.

what is difference between “PKCS1” and “PKCS8”?

PKCS is short for “Public Key Cryptography Standards”，中文翻譯為 “公鑰加密標準”

PKCS1，用來定義 RSA 格式的公私鑰，是最基本的格式。
- 通過 openssl 生成的金鑰，默認私鑰是 PKCS1 格式，不會輸出公鑰（因為公鑰可以通過私鑰導出）
PKCS8，是一種用於處理所有算法（而不僅僅是 RSA）的私鑰的標準。

私鑰格式	支持的加密算法	私鑰檔案內容的開頭
PKCS#1	RSA	`—–BEGIN RSA PRIVATE KEY—–`
PKCS#8	RSA、DSA、ECDSA 等	`—–BEGIN PRIVATE KEY—–`

Create Certificate Authority

我們需要建立自己的 root CA certificate，讓瀏覽器信任自簽發憑證。

$ mkdir ssl && cd ssl

執行以下 openssl 指令，以建立 rootCA.key 和 rootCA.crt。請將 SUSE.Lab 替換為您的網域名稱或 IP 位址。

$ openssl req -x509 \
-sha256 -days 356 \
-nodes \
-newkey rsa:2048 \
-subj "/CN=SUSE.Lab/C=TW/L=Taipei" \
-keyout rootCA.key -out rootCA.crt

等下會使用 rootCA.key 和 rootCA.crt 來簽署 SSL 憑證。

Create Self-Signed Certificates using OpenSSL

1. Create the Server Private Key

$ openssl genrsa -out server.key 2048

2. Create Certificate Signing Request Configuration

建立一個 csr.conf 檔案，把所有生成 CSR 所需的資訊都放進去。
把 ocp4.example.com 替換成你自己的網域名稱或 IP 地址。

$ cat > csr.conf <<EOF
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn
[dn]
C = TW
ST = Taiwan
L = Taipei
O = SUSE
OU = IT Department
emailAddress = admin@example.com
CN = ocp4.example.com
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = ocp4.example.com
DNS.2 = *.apps.ocp4.example.com
DNS.3 = api.ocp4.example.com
DNS.4 = api-int.ocp4.example.com
IP.1 = 192.168.11.80
EOF

3. Generate Certificate Signing Request (CSR) Using Server Private Key

執行以下命令生成 server.csr

$ openssl req -new -key server.key -out server.csr -config csr.conf

4. Create a external file

執行以下命令建立 cert.conf 檔案，目的是透過這個檔案，來產生 SSL 憑證。
將 *.apps.ocp4.example.com 替換為你的網域名稱或 IP 地址。

$ cat > cert.conf <<EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.apps.ocp4.example.com
EOF

5. Generate SSL certificate With self signed CA

執行以下指令產生 SSL 憑證，這個憑證是由我們自己的憑證機構所建立的 rootCA.crt 和 rootCA.key 簽署的。

$ openssl x509 -req \
-in server.csr \
-CA rootCA.crt -CAkey rootCA.key \
-CAcreateserial -out server.crt \
-days 365 \
-sha256 -extfile cert.conf

上述指令會產生 server.crt 憑證，我們將與 server.key 一起使用這個憑證，以在 Neuvector web console 中啟用 SSL 功能。

Create the secret from the generated key and certificate files

執行以下命令，把要給 Neuvector 用的私鑰和憑證做成 secret

$ oc create secret generic https-cert -n neuvector --from-file=server.key --from-file=server.crt

Edit the yaml directly for the manager and controller deployments to add the mounts

spec:
template:
spec:
containers:
volumeMounts:
- mountPath: /etc/neuvector/certs/ssl-cert.key
name: cert
readOnly: true
subPath: server.key
- mountPath: /etc/neuvector/certs/ssl-cert.pem
name: cert
readOnly: true
subPath: server.crt
volumes:
- name: cert
secret:
defaultMode: 420
secretName: https-cert

# neuvector yaml version for NeuVector 5.2.x on CRI-O
apiVersion: v1
kind: Service
metadata:
name: neuvector-svc-crd-webhook
namespace: neuvector
spec:
ports:
- port: 443
targetPort: 30443
protocol: TCP
name: crd-webhook
type: ClusterIP
selector:
app: neuvector-controller-pod
---
apiVersion: v1
kind: Service
metadata:
name: neuvector-svc-admission-webhook
namespace: neuvector
spec:
ports:
- port: 443
targetPort: 20443
protocol: TCP
name: admission-webhook
type: ClusterIP
selector:
app: neuvector-controller-pod
---
apiVersion: v1
kind: Service
metadata:
name: neuvector-service-webui
namespace: neuvector
spec:
ports:
- port: 443
targetPort: 8443
name: manager
protocol: TCP
type: ClusterIP
selector:
app: neuvector-manager-pod
---
apiVersion: v1
kind: Service
metadata:
name: neuvector-svc-controller
namespace: neuvector
spec:
ports:
- port: 18300
protocol: "TCP"
name: "cluster-tcp-18300"
- port: 18301
protocol: "TCP"
name: "cluster-tcp-18301"
- port: 18301
protocol: "UDP"
name: "cluster-udp-18301"
clusterIP: None
selector:
app: neuvector-controller-pod
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: neuvector-route-webui
namespace: neuvector
spec:
to:
kind: Service
name: neuvector-service-webui
port:
targetPort: manager
tls:
termination: passthrough
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-manager-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-manager-pod
replicas: 1
template:
metadata:
labels:
app: neuvector-manager-pod
spec:
serviceAccountName: basic
serviceAccount: basic
containers:
- name: neuvector-manager-pod
image: docker.io/neuvector/manager:5.2.0
env:
- name: CTRL_SERVER_IP
value: neuvector-svc-controller.neuvector
volumeMounts:
- mountPath: /etc/neuvector/certs/ssl-cert.key
name: cert
readOnly: true
subPath: server.key
- mountPath: /etc/neuvector/certs/ssl-cert.pem
name: cert
readOnly: true
subPath: server.crt
volumes:
- name: cert
secret:
defaultMode: 420
secretName: https-cert
restartPolicy: Always
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-controller-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-controller-pod
minReadySeconds: 60
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
replicas: 3
template:
metadata:
labels:
app: neuvector-controller-pod
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- neuvector-controller-pod
topologyKey: "kubernetes.io/hostname"
serviceAccountName: controller
serviceAccount: controller
containers:
- name: neuvector-controller-pod
image: docker.io/neuvector/controller:5.2.0
securityContext:
privileged: true
readinessProbe:
exec:
command:
- cat
- /tmp/ready
initialDelaySeconds: 5
periodSeconds: 5
env:
- name: CLUSTER_JOIN_ADDR
value: neuvector-svc-controller.neuvector
- name: CLUSTER_ADVERTISED_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CLUSTER_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- mountPath: /var/neuvector
name: nv-share
readOnly: false
- mountPath: /var/run/crio/crio.sock
name: runtime-sock
readOnly: true
- mountPath: /host/proc
name: proc-vol
readOnly: true
- mountPath: /host/cgroup
name: cgroup-vol
readOnly: true
- mountPath: /etc/config
name: config-volume
readOnly: true
- mountPath: /etc/neuvector/certs/ssl-cert.key
name: cert
readOnly: true
subPath: server.key
- mountPath: /etc/neuvector/certs/ssl-cert.pem
name: cert
readOnly: true
subPath: server.crt
terminationGracePeriodSeconds: 300
restartPolicy: Always
volumes:
- name: nv-share
hostPath:
path: /var/neuvector
- name: runtime-sock
hostPath:
path: /var/run/crio/crio.sock
- name: proc-vol
hostPath:
path: /proc
- name: cgroup-vol
hostPath:
path: /sys/fs/cgroup
- name: cert
secret:
defaultMode: 420
secretName: https-cert
- name: config-volume
projected:
sources:
- configMap:
name: neuvector-init
optional: true
- secret:
name: neuvector-init
optional: true
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: neuvector-enforcer-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-enforcer-pod
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: neuvector-enforcer-pod
spec:
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
hostPID: true
serviceAccountName: enforcer
serviceAccount: enforcer
containers:
- name: neuvector-enforcer-pod
image: docker.io/neuvector/enforcer:5.2.0
securityContext:
privileged: true
env:
- name: CLUSTER_JOIN_ADDR
value: neuvector-svc-controller.neuvector
- name: CLUSTER_ADVERTISED_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CLUSTER_BIND_ADDR
valueFrom:
fieldRef:
fieldPath: status.podIP
volumeMounts:
- mountPath: /lib/modules
name: modules-vol
readOnly: true
- mountPath: /var/run/crio/crio.sock
name: runtime-sock
readOnly: true
- mountPath: /host/proc
name: proc-vol
readOnly: true
- mountPath: /host/cgroup
name: cgroup-vol
readOnly: true
terminationGracePeriodSeconds: 1200
restartPolicy: Always
volumes:
- name: modules-vol
hostPath:
path: /lib/modules
- name: runtime-sock
hostPath:
path: /var/run/crio/crio.sock
- name: proc-vol
hostPath:
path: /proc
- name: cgroup-vol
hostPath:
path: /sys/fs/cgroup
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-scanner-pod
namespace: neuvector
spec:
selector:
matchLabels:
app: neuvector-scanner-pod
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
replicas: 2
template:
metadata:
labels:
app: neuvector-scanner-pod
spec:
serviceAccountName: basic
serviceAccount: basic
containers:
- name: neuvector-scanner-pod
image: docker.io/neuvector/scanner:latest
imagePullPolicy: Always
env:
- name: CLUSTER_JOIN_ADDR
value: neuvector-svc-controller.neuvector
restartPolicy: Always
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: neuvector-updater-pod
namespace: neuvector
spec:
schedule: "0 0 * * *"
jobTemplate:
spec:
template:
metadata:
labels:
app: neuvector-updater-pod
spec:
serviceAccountName: updater
serviceAccount: updater
containers:
- name: neuvector-updater-pod
image: docker.io/neuvector/updater:latest
imagePullPolicy: Always
command:
- /bin/sh
- -c
- TOKEN=`cat /var/run/secrets/kubernetes.io/serviceaccount/token`; /usr/bin/curl -kv -X PATCH -H "Authorization:Bearer $TOKEN" -H "Content-Type:application/strategic-merge-patch+json" -d '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'`date +%Y-%m-%dT%H:%M:%S%z`'"}}}}}' 'https://kubernetes.default/apis/apps/v1/namespaces/neuvector/deployments/neuvector-scanner-pod'
restartPolicy: Never

What is Route in OpenShift?

An OpenShift Container Platform route exposes a service at a host name, such as www.example.com, so that external clients can reach it by name.

Route Types

What is a Passthrough Route

apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: neuvector-route-webui
namespace: neuvector
spec:
to:
kind: Service
name: neuvector-service-webui
port:
targetPort: manager
tls:
termination: passthrough

spec.tls.termination: passthrough，With passthrough termination, encrypted traffic is sent straight to the destination without the router providing TLS termination. Therefore no key or certificate is required on the route.

建立 Neuvector

$ oc create -f neuvector.yaml

執行以下命令檢查 Pod 運行狀態和 route

$ oc get pod,route

螢幕輸出 :

NAME READY STATUS RESTARTS AGE
pod/neuvector-controller-pod-76bbbfc5dc-5t8lk 1/1 Running 0 167m
pod/neuvector-controller-pod-76bbbfc5dc-8znrb 1/1 Running 0 167m
pod/neuvector-controller-pod-76bbbfc5dc-t5tbb 1/1 Running 0 167m
pod/neuvector-enforcer-pod-4kgrn 1/1 Running 0 167m
pod/neuvector-enforcer-pod-kwsrr 1/1 Running 0 167m
pod/neuvector-enforcer-pod-lwnkp 1/1 Running 0 167m
pod/neuvector-manager-pod-6bb5555c7b-cx7lr 1/1 Running 0 167m
pod/neuvector-scanner-pod-56c798bb86-fvp8l 1/1 Running 0 167m
pod/neuvector-scanner-pod-56c798bb86-scqjk 1/1 Running 0 167m
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/neuvector-route-webui neuvector-route-webui-neuvector.apps.ocp4.example.com neuvector-service-webui manager passthrough None

在本機測試 Neuvector web console

$ sudo cp rootCA.crt /etc/pki/ca-trust/source/anchors/

更新 Linux 系統的 CA (Certificate Authority) 信任我們自己建的 CA 憑證

$ sudo update-ca-trust extract

驗證我們自己建的 CA 憑證是否由可信任的根憑證機構 (CA) 簽發

$ openssl verify /etc/pki/ca-trust/source/anchors/rootCA.crt

螢幕輸出 :

/etc/pki/ca-trust/source/anchors/rootCA.crt: OK

訪問 Neuvector web console

$ curl https://neuvector-route-webui-neuvector.apps.ocp4.example.com

oc get route neuvector-route-webui 可以得到 Domain name

螢幕輸出 :

This and all future requests should be directed to <a href="/index.html?v=2358c7ec6b">this URI</a>.

透過 Firefox 瀏覽器連線至 Neuvector web console

將 rootCA.crt 放入瀏覽器
- 設定 > 隱私權與安全性 > 往下滑點選 檢視憑證 > 匯入 > 選擇 rootCA.crt 後按 開啟 > 確定
輸入以下網址 : https://neuvector-route-webui-neuvector.apps.ocp4.example.com/

參考文件

Neuvector

Self-Signed Certificate

OpenShift Route

NeuVector Onboarding and Best Practices Guide

Wed, 21 Aug 2024 09:14:24 +0800

Preface

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

Neuvector 各 Container 所扮演的角色

Manager: 顯示 web-based console 的 stateless container。通常只需要一個，而且可以在任何地方執行。Manager 失敗不會影響 Controller 或 enforcer 的任何作業。但是，一些通知（事件）和最近的連線資料會被 Manager 快取到記憶體中，因此檢視這些資料會受到影響。
Controller: NeuVector 的「control plane」，應該部署在 HA 的架構中，以便在節點故障時不會遺失設定。Controller 可在任何地方執行，但在許多情況下，由於其關鍵性，客戶會選擇將其放置在「management」、master node或 infra node 上。
enforcer: 此 container 作為 daemonset 部署，因此每個要保護的節點上都有一個 Enforcer。通常會部署到每個 Worker 節點，但也可以部屬到 master 和 infra 節點。注意：如果 Enforcer 不在叢集中的節點上，且連線來自該節點上的 Pod，這些 workloads 在 NeuVector 中會顯示為「unmanaged」workloads。
scanner: 根據 Controller 的指示，使用內建的 CVE 資料庫執行弱點掃描。可部署多個 scanner 以增加掃描容量。 scanner 可在任何地方執行，但通常在 Controller 執行的節點上執行。scanner 節點的大小考量請參閱下文。當 scanner 用於建立階段掃描時，也可以獨立調用，例如在 pipeline 中觸發掃描、擷取結果並停止 scanner 。 scanner 包含最新的 CVE 資料庫，因此應每日更新。
updater: 當需要更新 CVE 資料庫時，updater 會透過 Kubernetes cron job 觸發 scanner 的更新。請務必針對您的環境進行設定。

Architecture

最簡單的部署模式是讓 Kubernetes 或 orchestrator 根據每個 Container 的記憶體（可能還有 cpu）需求，決定將每個 Contiainer 放置在哪裡。對於每個節點資源相近的叢集，而且可以保證所有 workloads 都有足夠的空間，這樣就沒問題了。其他考慮因素，例如獨立節點的維護週期、網路隔離和大小，都可能影響 NeuVector containers 的放置位置（在每個受保護節點上執行的 Enforcer 除外）。下圖顯示了部署模式範例以及 NeuVector 部署的彈性 :

Kubernetes 可以將 NeuVector Container 放置在任何節點 (在此例中為 worker 節點) 上的部署，除非也啟用了調度至 Master node。在 EKS、AKS 等公有雲管理服務中，NeuVector 只能在 worker 節點上執行。最常見的部署。

NeuVector control plane 的 Container 和 scanner 是透過 taints/tolerations 或 labels 放置在 Master node 上。Master node 可根據 control plane 的需求適當調整 node 資源的大小。

類似於 2，但會選擇 node 來執行 NeuVector control plane 和 scanners。這些節點可以是 worker node，而且大小適當。請注意，其中一個 NeuVector node 也允許 workloads 在上面執行，而其他節點則不允許 workloads 在上面執行。

NeuVectorcontrol plane 和 scanners 在 Master node 或專用節點上執行，但由於 Master node 或專用節點上沒有 workloads ，因此不會在主節點或專用節點上部署 Enforcer。在這種情況下，NeuVector 不會監控在主節點上執行的 system containers，這與公有雲的情況類似。

Q&A

Q: 是否可以在同一個節點上執行兩個或兩個以上的 Controller？ A: 可以，但如果節點領便當，該節點上的所有 Controller 都會丟失。範例部署的 yaml 中有一個「affinity」設定，預設會偏好在不同節點上部署 Controller。
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: neuvector-controller-pod
namespace: neuvector
spec:
template:
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- neuvector-controller-pod
topologyKey: "kubernetes.io/hostname"
```
這個 AntiAffinity 規則告訴 Kubernetes 的 scheduler，當指派新的 Pod 時，應該儘量避免將它們指派到已經有標籤 app=neuvector-controller-pod 的 Pod 所在的節點上。
Q: 我們應該在 master 節點和 infra 節點上部署 Enforcer 嗎？ A: 是的，如果可能，建議這樣做，這樣它也可以監控這些節點上的 Container 和網路流量。

Sizing and Scale

NeuVector Containers 需要足夠的記憶體和 cpu 才能正常運作。這是規劃部署架構時最重要的考量。確保每種 NeuVector Containers 的最低需求 (通常為 1GB RAM) 已經分配且可用。在某些情況下，這應該增加，例如 :

Large image scanning。Scanner Containers 必須有足夠的記憶體，才能將要掃描的 Container images 拉入記憶體並展開。如果要掃瞄大於 1GB 的 Container images ，請將掃瞄機的記憶體增加至略高於預期的最大 Container images 大小。
在保護模式下，預期網路連線數很高。Enforcer 在保護（線上防火牆封鎖）模式下需要 CPU 和記憶體來保持和檢查連線及可能的有效負載 (DLP)。增加記憶體並將一個 CPU 核心專用於 Enforcer 可確保足夠的封包過濾能力。

Tip：在 NeuVector Container 上設定不足的 Pod 資源限制，可能會導致意想不到的行為。我們建議您不要在 NeuVector Pod 上設定任何記憶體或 CPU 約束（最大值），如果有需要，請確保根據 staging/production 環境（目標 workloads 規模）中的實際效能特性，有足夠的可用空間。

其他擴充考量

K8s 叢集中的節點數量。隨著叢集中節點數量的增加，從每個 enforcer 到 controllers 的網路連接將共同增加。此外，在 Web 控制台中檢視節點的實際管理可能會變得更具挑戰性。
節點上的 Pod（ workloads ）數量。節點上 Pod 的數量會影響資源消耗，並可能影響網路流量 (節點上 Pod 與 Pod 之間或與其他節點之間的網路流量)。NeuVector Enforcer 也將會有更多的 Pods 來收集掃描資料和合規性資料，以傳送給 controllers。
叢集中的 Namespaces、Containers 和其他資產數量。隨著 Containers、 Namespaces、群組和其他資產的增加，主控台中的各種顯示會變得更繁忙，可能需要更多的時間來載入。大多數畫面都有篩選器可用，以快速隔離要檢視的 Namespaces或 Containers/pod。
Registry 中要掃描 Container images 的數量。如果 Registry 中的 Container images 數量超過一千個，且/或掃描或重新掃描整個 Registry/repository 所需的時間超過預期，請考慮在不同節點上部署多個 scanner pod。請記住，每個 scanner pod 會在掃描期間消耗其主機上的資源。自動擴展功能可設定為自動擴展可用 scanner pod 的數量，以滿足需求。
聯邦中的叢集數量。多叢集 Primary 會在其與遠端叢集之間啟動雙向連線。此外，如果部署了聯邦規則，這些規則會在發生任何變更時推送至所有遠端叢集。

提示：在與預期生產環境相似的環境中，觀察所有 pod（包括 NeuVector）的記憶體和 CPU 消耗統計，以確保沒有觀察到任何不利影響。

提示：在網路拓樸中，選擇一個或多個 namespaces 和核取標記，以將檢視限制為選取的物件。這將使地圖的載入速度更快。

Q&A

Q：運行 Controller 的 Host 主機最低規格為何？ A：假設 Controller 將與其他系統的 Containers 或 workloads 一起執行，我們建議至少使用 16 GB RAM 和 4 個 CPU 核心。
Q：NeuVector 最多可支援多少節點？ A：NeuVector 沒有硬性的限制，但是根據環境和主機資源的不同，超過 500 個節點的叢集可能會遇到實際的困難。

Deployment and Initial Configuration

Q&A

Q：樣本部署預設值需要做哪些變更？ A：變更可能包括：
1. Manager Service 的存取方法。Loadbalancer、nodeport、ingress controllers …等。
2. 如果從不同的 Registries 拉取重新命名或重新標記的 Container Image。就要修改 Container Image Path’s/Registries/names/version。
3. Container run-time 的 volume mounts。預設 containerd、CRI-O、docker 及其他需要變更磁碟區掛載的 run-times。
4. 多叢集 (主叢集/遠端叢集)。如果需要多叢集，請啟用 service access。
5. Controller、scanner Replicas 的數量。Controller 的 Replcas 預設值為 3。
6. 用於 Control-plane 節點部署的 Taints/tolerations。用於控制 Controller 被部署在哪些節點，或 enforcers 是否應該部署在 master node 上。

Managing NeuVector

大多數 NeuVector 部署至少部分是透過網頁式主控台來管理。然而，REST API、CLI、configMaps 和 CRD 都可用於 non-console-based (非控制台管理)。

Tip：在「My Profile」功能表 (右上方) 中，增加「Session Timeout」，這樣您就不會在 5 分鐘後登出。

簡單的部署可以使用整合式 load balancer (例如 EKS、AKS、GKE、IKS 上的 load balancer) 來啟用 Manager 服務的外部存取，或是開放一個 nodePort 供存取。請注意，nodePort 存取有其安全性考量，因為它會在每個節點上開放連接埠供外部存取，而不是強制透過 load balancer 或 ingress controller 進入入口。

Tip：使用 load balancer 或 ingress controller (例如 nginx) 來控制 NeuVector console 的存取。

Tip：SSL 連線可以在入口 ingress controller 終止，並使用 HTTP 連線到 manager。在 Manager deployment 的 YAML 檔中使用環境變數關閉到 manager 的 SSL

Q&A

Q：Manager container 可以在叢集外執行嗎？ A：可以，但這並不常見。Manager 需要將 controller 的 REST API 暴露在叢集之外。
Q：NeuVector Manager 服務可以使用 Istio 等 service mesh 類型的 ingress controller 嗎？ A：不支援。請使用其他 Kubernetes ingress 方法。
Q：Manager 可以管理多個叢集嗎？ A：可以，這需要部署 Multi-cluster federation (多叢集聯邦)的 controllers。可能還需要單獨的授權。
Q：自簽證書可以更換嗎？ A：可以。

Policy Modes – Discover, Monitor, Protect

每個 Group 的 policy 模式會決定 NeuVector 在偵測到 processes、網路連線和檔案活動時會做什麼。一般而言，「Discover」模式應該用於測試和暫存環境，以建立保護應用程式 workloads 的白名單規則。Monitor 或 Protect 模式應在生產環境中使用，以回應 Security Events (安全事件)。也可在測試/暫存環境中使用「Monitor」或「Protect」模式，觀察 NeuVector 在生產環境中的預期 run-time 行為。

Tip：在「Discover」模式下部署 workloads 後，執行測試 script 或流量，以運用 Container 中的所有功能（網路連線、process、檔案）。當幾天未建立新規則時，請將群組切換至 Monitor 模式，並觀察至少一週。尋找 Notification -> Security Events，看看合法活動是否被警告。加入適當的網路或程序規則，將合法活動列入白名單。如果需要 Protect (封鎖) 模式，請將群組切換至 Protect 模式，並在接下來幾天特別注意任何被封鎖的活動。

Tip：預設會啟用 Container 中的程序和檔案保護的 Zero-drift 模式。這對於允許有限 functions/processes 的強化 Container 非常有用。

Q&A

Q：在轉移至「Monitor」或「Protect」模式之前，我應該讓群組處於「Discover」模式多久？ A：在執行所有應用程式測試，並確定 NeuVector 已學會所有網路連線和程序之後，最快也要幾個小時。也可能需要數天或一週的時間。舉例來說，有些開放原始碼工具會定期與外部連線以檢查更新，而您可能看不到這些連線。您應該決定是否允許這些外部連線，如果允許，就為它們加入白名單規則。好的做法是將規則匯出為 CRD yaml 檔案，並與應用程式開發人員一起檢閱，以確認預期的行為。

Backups and Persistent Data

NeuVector 的設定以及任何狀態資料 (連線、通知等) 都會在可用的 controllers 之間同步。但是，如果所有 controllers 都領便當，設定和狀態資料就會遺失。若要讓 NeuVector 在領便當後自動復原叢集的組態，請啟用 persistent volume。當 controllers 啟動時，它們會從 persistent volume 中提取最新備份的設定資訊。

Tip：建立 RWX persistent volume 來自動備份 NeuVector 設定，並在任何 NeuVector、主機作業系統或 orchestrator 更新（重新開機）之前，定期透過 Web Console 或 REST API 進行手動匯出備份。某些公共雲端儲存系統不支援 RWX，因此可能需要部署 NFS 等獨立儲存設備。

Q&A

Q：匯出的備份檔案可以匯入不同的群集進行設定嗎？ A：不建議這樣做，因為名稱、namespace、IP 位址、system container 或其他組態設定的變更，可能在目標群集中無法運作。請使用 Helm、configMaps 和 CRD 來自動設定叢集。

Integration Into CI/CD Pipeline

安全應該儘可能整合並自動化到 CI/CD pipeline 中。漏洞與合規性管理將在下一節說明。整合可使用 NeuVector 的 plug-ins (外掛程式)與支援介面，或使用 REST API 自訂。 Admission control 是 Pipeline 與生產環境之間的重要橋樑，建議啟用。

Tip：在 Policy -> Admission Control 中啟用並測試 admission controller。然後建立一些簡單的規則來阻止未經授權的部署。即使尚未設定 registry scan，根據 registry 的名稱、namespace 或其他一般條件的規則，也可針對 Container image 的部署提供保障。

Enterprise Integration: Alerts, Notifications, SIEM/SYSLOG, Webhooks

NeuVector 在「Notifications」功能表中顯示 alerts，並透過 SYSLOG、webhooks 或 Prometheus exporter 匯出事件。事件也可以使用 REST API 匯出。NeuVector 會顯示每種事件類型 (security events、risk reports 及一般事件) 的最新事件。不過，這些資料僅限於每種類型最近的 4K 事件。預期事件將透過 SYSLOG 或其他方式匯出，以進行永久儲存、alerting 及進階處理。

Tip：使用 webhooks 將特殊事件通知直接傳送至 webhook endpoint (例如 Slack) 或叢集內的自訂 webhook receptor container，以便進行額外處理。

Tip: 提示：若要與 alerting/paging 系統、case management (案件管理)系統或 SIEM 自訂整合，請使用 REST API、webhooks 或其組合。

Policy Migration – Staging to Production

一旦應用程式經過測試，且 NeuVector 的執行時 security rules 經過驗證，就應該將規則複製到生產環境中。

Tip：使用 NeuVector CRD 將 security rules 從暫存環境匯出、檢閱、登入及遷移到生產環境。

Tip：在生產環境中，在「Settings」->「Configuration」中將「New Services Mode」設定為「Monitor」或「Protect」，以防止任何未知服務啟動而不產生警示。在部署任何新的 workloads 之前，請確保透過 CRD、REST API 或主控台部署白名單規則 (程序、網路、檔案)，以便 workloads 可以在受保護狀態下不中斷地開始執行。

Q&A

Q：在轉移到生產環境之前，我們應該在暫存環境中運行多久？ A：確保您的暫存環境擁有與生產環境相同的 node、orchestrator、system containers 及其他重要資產。也應該測試預期在生產中的 Initial application workloads，但預期新的和更新的應用程式會持續發生。對於初始部署，這通常需要在暫存環境中進行幾週的測試，而對於新增或更新的 application workloads，則需要幾天到幾週的時間。
Q：我們應該在生產中以「Monitor」或「Protect」模式執行嗎？ A：一開始，我們建議您在生產中以 Monitor 模式執行所有群組數天或數週，直到您對通知中偵測到的任何安全事件感到滿意為止。然後，您可以切換到 Protect 模式，只針對希望 NeuVector 阻擋網路、程序和檔案違規的群組。這可以根據具有出口連線的群組、關鍵資料庫，或是您 100% 確信所有預期行為都已列入白名單的 workloads 來決定。
Q：我可以使用「Settings」->「Configuration」中的 export/import 設定檔案來進行遷移嗎？答：不建議這樣做，因為名稱、namespace、IP 位址、system containers 或其他設定可能有變，無法在目標叢集中運作。使用 CRD 來進行遷移，方法是匯出群組規則，並在 yamls 中編輯這些規則，以反映生產環境中的任何變更，然後再進行部署。ConfigMaps 可用於在暫存環境和生產環境中一致配置其他設定。

References

Using CRD: https://open-docs.neuvector.com/policy/usingcrd

Vulnerability and Compliance Management

每家公司都會有不同的流程和標準來管理 Vulnerability (漏洞)和 Compliance (合規性) 測試。NeuVector 可彈性適應您的流程。

弱點與法規遵循管理的主要最佳實務包括：

要求開發人員在有修復方法時修復關鍵弱點，並盡可能在建立階段就停止或警示。
如果在 registry 中的現有 (已核准) Container images 或生產環境的 Container 中發現新的弱點，通知開發人員或適當的團隊。
提供寬限期，讓開發人員修復關鍵弱點，但確保運作中的 workloads 受到 NeuVector 白名單規則與「virtual patching」的保護。請參閱以下有關 virtual patching 的 Q&A 與參考連結， virtual patching 可保護執行漏洞的容器。
允許開發人員與 devops 團隊申請政策的例外情況，並能夠忽略某些漏洞掃描警示 (根據 CVE 編號)。

Tip：利用欄位「已修復」、「發佈日期」和自訂「Author/developer」的 metadata，自動執行政策，要求開發者修復已發佈超過 7 天 (寬限期) 的弱點 (已修復)。

Q&A

Q：為什麼 NeuVector 的掃描報告與我使用的其他 scanner 不同？ A：每家掃描廠商都維護自己的 CVE 資料庫，以及對 CVE 來源的詮釋，例如嚴重性/關鍵性等級。
Q：NeuVector 是否可以在 Container images 推送到 registry 時立即進行掃描？ A：有些 registry (例如 Openshift) 支援 imagestreams，可讓 NeuVector 在 Container images 推送時自動掃描。對於其他 registry，則可設定定期掃描，每隔幾分鐘或幾小時掃描一次新 Container images。對於真正的按需掃描，可使用 REST API 在特定 Container images 推送至 registry 後觸發掃描。
Q：什麼是 Virtual Patching (虛擬修補)？ A：NeuVector 所使用的這個詞彙，是指在生產環境中運行的 Container，如果存在 critical 的漏洞，NeuVector 在 Monitor 或 Protect 模式下運行時，會對其進行「virtually patched (虛擬修補)」，因為任何利用漏洞的嘗試都會立即被偵測到並封鎖，因為它會創建未經授權的 process、網路連線或檔案存取。

Compliance Management

NeuVector 的 Compliance (合規性)檢查包括 CIS benchmarks (docker、kubernetes、openshift 等) 以及自訂合規性檢查 (在 Container 或主機上執行的 script)。這些都可以針對 PCI、GDPR 等各種產業標準進行 tag 與報告。

Q&A

Q：我可以自訂合規性報告嗎？ A：可以，透過標記適當的合規檢查，可以建立 PCI、GDPR 及其他的標準報告。每項報告都可以自訂，以包含或排除某些檢查。

Improving the Security Risk Score in the Dashboard

Dashboard 提供整體安全風險評分，該評分根據 Container 的 Policy 模式、ingress/egress 連線、vulnerabilities (弱點)、privileged/root Container 和 admission control。使用分數旁邊的精靈工具來逐步改善您的分數。

Tip：通常不可能將風險分數降至零，因為任何執行中的 Container、Kubernetes system container 或 egress 連線都代表風險。任何在「良好」範圍（小於 20）內的分數都被認為是可以接受的。

Network Rules – Ingress, Egress, Segmentation

網路 Segmentation (分割)、檢查和保護是 run-time 最重要的安全防護措施。應仔細檢閱和調整網路規則，以達到所需的行為（允許、警示、封鎖）。NeuVector 在「發現」模式下學習到的網路規則，在某些情況下會變得支離破碎，例如 :

連線來源或目的地總是在變，因為使用的是隨機連接埠，例如透過 load balancer 或 ingress 連接
部屬同一應用程式的 Pod 或同一應用程式的新版本在命名慣例中有版本號或隨機字串，導致 NeuVector 認為這是一個新應用程式，並為其建立新規則。

Tip：透過在 Policy -> Network Rules 中篩選應用程式，檢視並編輯影響每個應用程式的網路規則。如果您注意到許多規則重複出現，且出發地或目的地的 IP 位址、連接埠或節點不斷變更，請考慮如何根據通訊協定、標籤或其他條件 (有或沒有通配符) 建立更高層級的網路規則，以整合這些規則。

Egress 的連接可能是高風險的來源，因此應加以評估，並在可能的情況下，聲明只允許存取特定目的地。

Tip：為需要存取叢及外部的 workloads 建立自訂出口規則。使用 address=<destination> 建立目標 (目的地) 自訂群組，並建立相對應的網路規則，允許從 Pod 群組存取到目標群組，可能的話使用應用程式通訊協定 (例如 MySQL、redis、mongodb、SSL 等)。

在部署過程中套用的 Pod 標籤也可以用來強制執行規則。舉例來說，scope=cde、scope=non_cde、external_access=allowed 都是標籤的範例，這些標籤可以套用到 Pod，並在 NeuVector 中為這些 Pod 建立規則。可以建立符合標籤的自訂群組，並將規則套用至該群組。

Q&A

Q：網路規則可以訂製嗎？ A：可以，自訂和學習的網路規則可以透過 console 或 REST API 排序。聯邦規則和 CRD 建立的規則無法編輯，且總是先進行評估（即優先於其他規則）。
Q：建立自訂群組時，可以使用通配符嗎？ Q：可以，通配符通常支援在條件中使用，例如 address=*.google.com。如果需要更靈活的匹配，也可以使用 regex 表達式，用 ~ 符號表示，例如 label~my.label*-xyz。
Q：我可以將自訂群組的 policymode 變更為 Monitor 或 Protect 嗎？ A：目前自訂群組的政策模式是無法設定的，因為所引用的底層 Pod 可能處於不同的模式 (Discover、Monitor、Protect)。
Q：DLP 網路 payload 檢查 regex 引擎可否用於其他政策？ A：可以，它可以用於 secrets 檢查、檢查 HTTP 標頭或路徑，或其他用途。
Q：某些群組可以設定為 block，而其他群組只設定為警示嗎？ A：可以，NeuVector 中每個已學習的「Group」都可以有不同的模式：「Discover」、「Monitor」或「Protect」。
Q：是否可以將「Group」設定為封鎖外部 egress 連線，但只在叢集或 namespace 中的其他 pod 違規時發出警示？ A：目前不可以。群組對於所有流量只能使用一種模式。我們正考慮在未來的版本中使用更細緻的政策。

Automation – REST API

使用 REST API 可直接在 Controller 上執行任何動作，而無需透過 Web console。Manager container 使用 REST API 存取 Controller。REST API 的預設連接埠為 10443，可從叢集內部存取。若要從群集外部對 Controller 進行呼叫，請在外部將 REST API expose 為服務。

Tip：在 script 中使用 REST API 來自動執行 Policy 備份、根據可疑活動的封包擷取、觸發 Container images 掃描、讀取掃描結果等工作。

Q&A

Q： authentication (驗證) 和 authorization (授權) 如何執行？ A：api token 請求需要使用者帳號和密碼進行驗證。使用者的角色決定該使用者可透過 api 進行哪些動作。
Q：token 可使用多久？ A：token 的持續時間與使用者個人資料中設定的使用者逾時時間相同。例如，如果超時設定為 10 分鐘 (600 秒)，只要在 10 分鐘內使用，token 就會有效。10 分鐘未使用後，它就會過期。

Automation – Security Policy, CRD

NeuVector Custom Resource Definition (CRD) 提供了一個強大的機制，可在 NeuVector 中宣告並自動化 security rules (安全規則)。它可讓安全、開發與開發人員協同討論與檢視允許的應用程式行為。

Tip：使用 NeuVector 來學習應用程式在 staging/test 中的行為，然後將規則匯出為 CRD，以便與開發人員和 devops 團隊一起檢閱與核准。然後檢查 CRD，像管理其他「code (程式碼)」一樣管理安全政策。

CRD 用來「宣告」一組 Groups、rules 和 NeuVector objects 的 policy 模式。一旦宣告後，就只能套用更新的 CRD 來編輯。

Tip：使用 CRD 設定不與特定應用程式行為掛鉤的「global」安全規則，例如在「所有 Container」上防止 SSH 或 SCP (Container 是 NeuVector 中的保留群組名稱，可用於此)。或是允許外部 api 存取特定標籤的 Pod。

Q&A

Q：如何匯出應用程式的 CRD？ A：移至 Groups，選擇要匯出的 Group，然後按一下匯出 Group Policy。選取 Groups 的所有 rules，以及相關 Groups (即在 network rules [網路規則] 中定義為選取群組的來源或目的地的群組) 將會匯出。
Q：在 CRD 中應用之前，我可以變更自訂群組的「policy mode」嗎？ A：目前，自訂 Group 的 policy mode 必須為 null，因為所引用的底層 Pod 可能處於不同的模式 (Discover、Monitor、Protect)。

Reviewing Notifications and Reducing False Positives

在 NeuVector 部署的最初幾天或幾週內，應檢查事件以確定它們是否為誤報。有幾種方法可以減少誤報：

檢視「Notifications」中的事件，如果適用，選擇「Review Rule」按鈕，立即新增違規事件的白名單規則。
檢閱事件通知後，在適當的政策功能表中手動建立白名單規則 - admission control、process (Group 下)、network rules 等。
建立 Response rule，以「抑制通知」這些類型的事件。

Tip：盡可能建立白名單規則，以允許被報告為違反的行為。若要暫時壓制通知，請使用 Response rule，此規則稍後可輕鬆停用或移除。

Updating – NeuVector, Nodes, Host OS, Orchestrator Platforms

NeuVector 支援對 critical containers 進行滾動式的更新，但對於環境的任何更新，都應該特別小心。controllers 之間會維持一個狀態，如果所有 controllers 同時不可用，這個狀態就會消失。因此，在升級 hosts/nodes (重新開機) 或 orchestrator (例如 Kubernetes) 時，請特別小心，即使已啟用節點 draining process。

Tip：對於主機作業系統或如 Kubernetes 等 orchestrator 平台的更新，需要重新啟動節點或排出 Pod 時，請確保至少有一個 NeuVector Controller 一直處於活動狀態。當重新啟動有 Controller 的節點時，請觀察新 Controller，以確保它至少有 60 秒（幾分鐘更好）處於 available (可用) 的狀態，以確保它有時間與領導 Controller 同步狀態，然後再重新啟動下一個有 Controller 的節點。

Tip：在更新 NeuVector、其運行的主機或 orchestrator 之前，請務必手動備份整個設定檔。您可以在 Settings -> Configuration -> Export All 中匯出備份。

Appendix - Pre-Deployment Checklist

收集所需的資訊
- NeuVector 版本和 K8s 版本。
- 拉取 NeuVector Container image 的 Dockerhub ID。
- 目標節點的 CPU/記憶體設定文件。
- 正在使用的 Container run-time。
- manager 使用 Ingress 的方法對外。
- SYSLOG servers、LDAP/AD、SSO/SAML servers 的整合資訊。
檢閱 NeuVector 官方文件
- 特別是第 1 節 Deployment Preparation
- 第 2 節 Deploying。每個協調平台也在本節有特定的部署說明。
準備目標環境
- 如果不是從 NeuVector docker hub registry 動態拉取，請預先拉取 images。
- 建立 RWX 儲存空間（如果使用持久化儲存空間來備份設定） ‧
確保網路連通性
- 測試從 Cluster 到 Registry 或 docker hub 拉取映像的能力。
- 啟用並測試從集群內部存取 Registry 以進行 Registry 掃描（如適用）。
- 透過 load balancer、route、IP/port（預設埠 8443），存取集群中的 manager service 的控制台。
- 檢查網路或本機防火牆（如 firewalld）是否阻擋 NeuVector 所需連接埠的存取。
- 啟用 SYSLOG（預設埠514）和 webhook 通知所需的 outbound connections。
建立部署流程與範本
- 決定部署方法： Helm、kubectl/yaml 檔案、Operator…
- 檢閱範例 yaml 檔案和值與設定選項
- Kubernetes - https://open-docs.neuvector.com/deploying/kubernetes
- 從 Rancher Manager 部署 - https://opendocs.neuvector.com/deploying/rancher
- Openshift - https://open-docs.neuvector.com/deploying/openshift
- 在 Helm 中編輯 yamls 或部署值（如適用）：
  - Manager 存取方式 - LoadBalancer、ingress、NodePort…
  - 啟用/停用多叢集 master 和 remote services
  - Container Images 名稱、版本標籤或 NeuVector images 的路徑
  - 若為 containerd 或 CRI-O 則為 Container run-time
  - 控制 NeuVector Pod 部署位置的 Taints/tolerations 或節點標籤。

Ref

NeuVector Onboarding and Best Practices Guide

Neuvector Project Plan

Tue, 20 Aug 2024 03:38:56 +0800

gantt
%% 宣告這是甘特圖
title Neuvector Project Plan
%% 給它一個標題
dateFormat MM-DD
%% 日期格式
axisFormat %m-%d
%% 縱軸的日期格式
section Deploy
%% 宣告一個區塊
Testing : active, work, 09-01, 2d
Dev : active, dev, after work, 2d
Production : active, production, 09-07, 2d
%%語法：
%%甘特圖上的任務名稱 : [crit], [active|done], 任務名稱, [日期|after 任務名稱], 長度
section 功能驗測
Testing : active, b1, after dev, 1d
Dev : active, b2, after b1, 1d
Production :active, b3, after production, 1d
section Report
Report : active, b4, after b3, 1d
%%todayMarker off

Helm Install Neuvector on community Kubernetes

Sat, 06 Jul 2024 08:02:38 +0800

Preface

本篇文章會介紹以下內容

如何透過 Helm 安裝 Neuvector 5.2.0

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

測試環境

master node resource:
- os: Alpine Linux 3.18.2
- memory: 4 G
- CPU: 2 Core
- cri-o: 1.27
w1 node resource:
- os: Alpine Linux 3.18.2
- memory: 8 G
- CPU: 2 Core
- cri-o: 1.27
w2 node resource:
- os: Alpine Linux 3.18.2
- memory: 8 G
- CPU: 2 Core
- cri-o: 1.27

確認 K8S Node 狀態

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
c1m1 Ready control-plane 45m v1.27.4 120.96.143.212 <none> Alpine Linux v3.18 6.1.39-0-virt cri-o://1.27.0
c1w1 Ready worker 45m v1.27.4 120.96.143.219 <none> Alpine Linux v3.18 6.1.39-0-virt cri-o://1.27.0
c1w2 Ready worker 45m v1.27.4 120.96.143.220 <none> Alpine Linux v3.18 6.1.39-0-virt cri-o://1.27.0

Install Neuvector

準備 Neuvector 5.2.0 的 Image 壓縮檔

下載連結
將 image 都準備到 Kubernetes 中的每一台 Node 上
或是執行以下 Command ，程式會在每一台主機座以下動作，
- 到我的下載連結把 Image 的壓縮檔下載下來
- 透過 sudo podman load 將 Image 壓縮檔還原成 Image

$ kubectl apply -f https://raw.githubusercontent.com/braveantony/k8s-yaml/main/prepare-neuvector-image.yaml

注意，以上方法只適用於 K8S 的 CRI 是 CRI-O，並且每台主機有裝 Podman
此方法會占用大約 4 G 的記憶體
看 log 命令 : kubectl logs -f -l app=install-neuvector-image -c install-neuvector-image

Install Helm3

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

check helm version

$ helm version

螢幕輸出 :

version.BuildInfo{Version:"v3.12.2", GitCommit:"1e210a2c8cc5117d1055bfaa5d40f51bbc2e345e", GitTreeState:"clean", GoVersion:"go1.20.5"}

Adding chart repo

$ helm repo add neuvector https://neuvector.github.io/neuvector-helm/

螢幕輸出 :

"neuvector" has been added to your repositories

透過 Helm 安裝 Neuvector 5.2.0

$ helm install neuvector --namespace neuvector --create-namespace neuvector/core \
--set crio.enabled=true \
--set tag=5.2.0 \
--set controller.replicas=1

螢幕輸出 :

NAME: neuvector
LAST DEPLOYED: Mon Jul 24 11:18:20 2023
NAMESPACE: neuvector
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Get the NeuVector URL by running these commands:
NODE_PORT=$(kubectl get --namespace neuvector -o jsonpath="{.spec.ports[0].nodePort}" services neuvector-service-webui)
NODE_IP=$(kubectl get nodes --namespace neuvector -o jsonpath="{.items[0].status.addresses[0].address}")
echo https://$NODE_IP:$NODE_PORT

可以透過 Helm 給的資訊，取得 NeuVector URL

檢查 Neuvector Pod 運作狀態

$ kubectl -n neuvector get pod

螢幕輸出 :

NAME READY STATUS RESTARTS AGE
neuvector-controller-pod-85bd66d98-g2jwx 1/1 Running 0 10m
neuvector-enforcer-pod-bfnht 1/1 Running 0 10m
neuvector-enforcer-pod-f6xvb 1/1 Running 0 10m
neuvector-enforcer-pod-zpb5p 1/1 Running 0 10m
neuvector-manager-pod-5894fd44b9-hdq6n 1/1 Running 0 10m
neuvector-scanner-pod-6c68cc78d6-6rg55 1/1 Running 0 3m10s
neuvector-scanner-pod-6c68cc78d6-k96hk 1/1 Running 0 3m9s

將 neuvector-scanner-pod 這個 Deployment 的 ImagePullPolicy 改為 IfNotPresent

$ kubectl patch deployment neuvector-scanner-pod -n neuvector -p '{"spec": {"template": {"spec":
{"containers": [{"name": "neuvector-scanner-pod", "imagePullPolicy": "IfNotPresent"}]}}}}'

讓 scanner deplotment 的 Pod 不要被指派到 master Node 上

$ kubectl patch deployment neuvector-scanner-pod -n neuvector --type='json' -p='[{"op": "add", "path": "/spec/template/spec/affinity", "value": {"nodeAffinity": {"requiredDuringSchedulingIgnoredDuringExecution": {"nodeSelectorTerms": [{"matchExpressions": [{"key": "node-role.kubernetes.io/worker", "operator": "In", "values": [""]}]}]}}}}]'

將 scanner pod 數量縮減為 2

$ kubectl -n neuvector scale deploy neuvector-scanner-pod --replicas=2

檢查修改後 Neuvector Pod 運作狀態

$ kubectl -n neuvector get pod -o wide

螢幕輸出 :

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
neuvector-controller-pod-85bd66d98-qwbb4 1/1 Running 0 54s 10.244.161.134 c1w1 <none> <none>
neuvector-enforcer-pod-6jp5p 1/1 Running 0 54s 10.244.54.135 c1w2 <none> <none>
neuvector-enforcer-pod-9pp2f 1/1 Running 0 54s 10.244.161.133 c1w1 <none> <none>
neuvector-enforcer-pod-fk6k9 1/1 Running 0 54s 10.244.74.9 c1m1 <none> <none>
neuvector-manager-pod-5894fd44b9-kwtgv 1/1 Running 0 54s 10.244.54.134 c1w2 <none> <none>
neuvector-scanner-pod-7cd7d597df-755bv 1/1 Running 0 13s 10.244.161.137 c1w1 <none> <none>
neuvector-scanner-pod-7cd7d597df-9w55b 1/1 Running 0 11s 10.244.54.137 c1w2 <none> <none>

檢查 Neuvector Service

$ kubectl -n neuvector get svc

螢幕輸出 :

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
neuvector-service-webui NodePort 10.98.0.202 <none> 8443:30039/TCP 8m33s
neuvector-svc-admission-webhook ClusterIP 10.98.0.62 <none> 443/TCP 8m33s
neuvector-svc-controller ClusterIP None <none> 18300/TCP,18301/TCP,18301/UDP 8m33s
neuvector-svc-crd-webhook ClusterIP 10.98.0.88 <none> 443/TCP 8m33s

預設會有一個 Service 開 NodePort

$ NODE_PORT=$(kubectl get --namespace neuvector -o jsonpath="{.spec.ports[0].nodePort}" services neuvector-service-webui)
$ NODE_IP=$(kubectl get nodes --namespace neuvector -o jsonpath="{.items[0].status.addresses[0].address}")
$ echo https://$NODE_IP:$NODE_PORT

螢幕輸出 :

https://120.96.143.212:30039

打開瀏覽器輸入 NeuVector 的 URL

https://120.96.143.212:30039

帳號密碼預設都是 : admin 勾選同意 License 後，按 login 登入

Neuvector Dashboard 畫面

解除安裝 Neuvector

$ helm uninstall neuvector --namespace neuvector

螢幕輸出 :

release "neuvector" uninstalled

參考資料

NeuVector Helm charts

helm install neuvector --namespace neuvector --create-namespace neuvector/core \
--set containerd.enabled=true \
--set runtimePath=/run/containerd/containerd.sock \
--set containerd.path=/run/containerd/containerd.sock \
--set tag=5.3.3 \
--set controller.replicas=1 \
--set manager.ingress.enabled=true \
--set manager.ingress.host=neuvector.example.com \
--set manager.ingress.ingressClassName=nginx \
--set manager.ingress.path=/

Increase SSH Connection Timeout in Linux

Tue, 02 Jan 2024 09:19:20 +0800

1. SSH Server 設定

$ cat <<EOF | sudo tee -a /etc/ssh/sshd_config
ClientAliveInterval 1200
ClientAliveCountMax 9
EOF

The ClientAliveInterval parameter specifies the time in seconds that the server will wait before sending a null packet to the client system to keep the connection alive.

the ClientAliveCountMax parameter defines the number of client alive messages which are sent without getting any messages from the client. If this limit is reached while the messages are being sent, the sshd daemon will drop the session, effectively terminating the ssh session.

Timeout value = ClientAliveInterval * ClientAliveCountMax

以上設定等於連續連接 9 小時都沒發送命令後才會斷線

2. 重啟 SSHD 服務

$ sudo systemctl restart sshd

Persistent Data for NeuVector Policy and Configuration

Wed, 27 Sep 2023 09:09:32 +0800

Preface

本篇文章會採取 Step by Step 和 ScreenShot 的介紹，

如何將 Neuvector 的 Policy and Configuration 透過 Kubernetes Persistent Volume 永久保存資料

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

Intro

預設情況下，NeuVector 將各種設定檔儲存在 Controller-Pods 的 /var/neuvector/config/backup 中。

NeuVector 可以將 configuration ( 設定 ) 和 policy ( 規則 ) 儲存到硬碟上，這樣即使 Controller 故障，設定和規則也不會遺失。這可以通過將一個 Kubernetes volume 掛載到 Controller pod 中的 /var/neuvector/ 目錄區來實現。

如果 K8S 集群完全故障，當創建新 K8S 集群時，設定和規則會自動恢復。設定和規則也可以手動從 Volume 中恢復或刪除。

如果在 HA 配置中執行多個 Controller-Pods，只要其中一個 Controller-Pod 始終處於運作狀態，所有資料都會在 Controller 之間同步。

如果希望儲存 violations ( 違規 )、threats ( 威脅 )、vulnerabilities ( 漏洞 ) 和 events ( 事件 ) 等 Logs，請在 “Settings” 中啟用 SYSLOG server。

如果沒有使用「Kuberentes Persistent Volume」，NeuVector 就不會將設定或規則儲存到硬碟上。在停止所有 Neuvector-Controller-Pods 之前，請先備份 Controller 的設定和規則，這可以在「Settings」>「Configuration」中完成。

如果要確保設定和規則在 Neuvector Controller 更新時不會遺失，可以將 Neuvector Controller 部署在高可用性 (HA) 配置中，以 3 或 5 個 Neuvector-Controller-Pods 運行。

環境介紹

將 Neuvector 5.2.0 安裝在 OpenShift 4.12 之上

Step1: Create Persistent Volume Claim

$ nano neuvector-pvc.yaml

檔案內容如下 :

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: neuvector-data
namespace: neuvector
spec:
accessModes:
- ReadWriteMany
volumeMode: Filesystem
resources:
requests:
storage: 1Gi

Note:

如果在您的環境中有 StorageClass 則可跳過 Step2 建立 Persistent Volume
NeuVector 對於 accessModes 的要求必須是 ReadWriteMany(RWX).

建立 PVC

$ oc create -f neuvector-pvc.yaml

Step2: Create Persistent Volume

$ nano neuvector-pv.yaml

檔案內容如下 :

apiVersion: v1
kind: PersistentVolume
metadata:
name: neuvector-data
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteMany
hostPath:
path: "/var/neuvector"

建立 PV

$ oc create -f neuvector-pv.yaml

檢查 PV 和 PVC 是否 Bound 在一起

$ oc get pv,pvc

螢幕輸出 :

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/neuvector-data 1Gi RWX Retain Bound neuvector/neuvector-data 18m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/neuvector-data Bound neuvector-data 1Gi RWX 18

Step3: Modify the NeuVector-Controller sample yaml file

If Neuvector is existed

Settings –> Configuration –> Export –> All –> Submit

如果 Volume 重新掛載到 Neuvector Controller 之後，有發現設定檔有殘缺，就可以 Import 回去

點 Choose File –> 再點擊向上的箭頭，匯入檔案

執行以下命令修改 neuvector-controller-pod

$ oc edit deploy neuvector-controller-pod

修改檔案內容如下 :

...
metadata:
name: neuvector-controller-pod
spec:
template:
spec:
containers:
- env:
- name: CTRL_PERSIST_CONFIG ## 新增這個 ENV
...
volumes:
- #hostPath: ## 註解掉這行
# path: /var/neuvector ## 註解掉這行
# type: "" ## 註解掉這行
name: nv-share
persistentVolumeClaim: ## 新增 PVC
claimName: neuvector-data ## 指定 Pods 使用哪個 PVC

Rollout Restart Controller Pods

$ oc rollout restart deploy neuvector-controller-pod

檢視 Pods 被滾動式重新佈署之後的狀態

$ watch -n 1 oc get pods -l app=neuvector-controller-pod

成功部屬後的螢幕輸出

NAME READY STATUS RESTARTS AGE
neuvector-controller-pod-7bd8dd947f-8tw66 1/1 Running 0 8m34s
neuvector-controller-pod-7bd8dd947f-qtghq 1/1 Running 0 9m50s
neuvector-controller-pod-7bd8dd947f-vf94c 1/1 Running 0 7m19s

If Neuvector doesn’t existed

$ nano neuvector.yaml

修改檔案內容如下 :

...
spec:
template:
spec:
volumes:
- name: nv-share
# hostPath: // replaced by persistentVolumeClaim
# path: /var/neuvector // replaced by persistentVolumeClaim
persistentVolumeClaim:
claimName: neuvector-data

Add the following environment variable in the Controller

 - name: CTRL_PERSIST_CONFIG

Step4: Build Neuvector

$ oc create -f neuvector.yaml

檢視 Neuvector Pods 運作狀況

$ oc get pods

螢幕輸出 :

NAME READY STATUS RESTARTS AGE
neuvector-controller-pod-74d88c4df6-7kqxk 1/1 Running 0 4m4s
neuvector-controller-pod-74d88c4df6-m9spn 1/1 Running 0 4m3s
neuvector-controller-pod-74d88c4df6-vp96s 1/1 Running 0 4m3s
neuvector-enforcer-pod-4zsj8 1/1 Running 0 4m3s
neuvector-enforcer-pod-f5znw 1/1 Running 0 4m3s
neuvector-enforcer-pod-tqhz5 1/1 Running 0 4m3s
neuvector-manager-pod-6bb5555c7b-85sdk 1/1 Running 0 4m4s
neuvector-operator-86dd64f497-nqmbg 1/1 Running 2 19d
neuvector-scanner-pod-56c798bb86-8ktcl 1/1 Running 0 4m3s
neuvector-scanner-pod-56c798bb86-rvftv 1/1 Running 0 4m3s

Step5: 建立掃描 Image Registry 的規則

Assets -> Registries -> Add

Name: quay.io
Registry: https://quay.io/
User Name: hahappyman
Password: ***
Filter: hahappyman/alpine-*

勾選 Scan Layers -> Submit

點擊 Start Scan 按鈕

檢視掃描 Image Registry 的相關設定是否被保存到 Node 上

$ cat <<EOF | oc debug node/worker2.ocp4.example.com 2> /dev/null
chroot /host
/bin/bash -c "[[ -d /var/neuvector/config/backup/ ]] && cat /var/neuvector/config/backup/registry.backup || ls -al /var/neuvector"
EOF

因為本次範例 PV 使用的是 HostPath ，資料可能只存在特定的 Node 上，Neuvector 的 Controller 會自行同步，所以有的節點的 /var/neuvector 目錄底下沒有資料，如以下螢幕輸出

total 4
drwxr-xr-x. 2 root root 6 Aug 1 04:25 .
drwxr-xr-x. 26 root root 4096 Aug 1 04:25 ..

正確螢幕輸出如下

{"auth_token":"","auth_with_token":false,"aws_key":null,"cfg_type":2,"creater_domains":null,"disable_files":false,"domains":null,"filters":["hahappyman/*"],"gcr_key":null,"gitlab_api_url":"","gitlab_private_token":"","ibmcloud_account":"","ibmcloud_token_url":"","jfrog_aql":false,"jfrog_mode":"","name":"quay.io","parsed_filters":[{"organization":"hahappyman","repository":".*","tag":".*"}],"password":"tZS7XXX6T0bkk0Pim8y3//qb0xmTVx5eGboJcrck","poll_period":0,"registry":"https://quay.io/","repo_limit":200,"rescan_image":true,"scan_layers":true,"schedule":"manual","tag_limit":20,"type":"Docker Registry","username":"hahappyman"}

此時就可以看到掃描 Image Registry 的設定被儲存在 Node 上
Image 掃描的結果也會被儲存到 /var/neuvector/registry/ 目錄下

Step6: 建立 Admission Control 規則

Policy -> Admission Control -> Add

Criterion:
- Run as privileged
- Namespace is one of [test]
Mode: Protect

勾選 Status: Enabled，並切換到 Protect Mode

測試 Admission Control 規則

$ oc run nginx --image=nginx:stable -n test --privileged=true

螢幕輸出 :

Error from server: admission webhook "neuvector-validating-admission-webhook.neuvector.svc" denied the request: Creation of Kubernetes Pod is denied.

檢視 Admission Control 規則是否被保存到 Node 上

$ cat <<EOF | oc debug node/worker2.ocp4.example.com 2> /dev/null
chroot /host
/bin/bash -c "[[ -d /var/neuvector/config/backup/ ]] && cat /var/neuvector/config/backup/admission_control.backup || ls -al /var/neuvector"
EOF

螢幕輸出 :

{"id":1000,"category":"Kubernetes","comment":"","criteria":[{"name":"runAsPrivileged","op":"=","value":"true","value_slice":null,"path":"runAsPrivileged"},{"name":"namespace","op":"containsAny","value":"test","value_slice":null,"path":"namespace"}],"disable":false,"critical":false,"cfg_type":2,"rule_type":"deny","use_as_risky_role_tag":false,"rule_mode":"protect"}
...以下省略

Step7: 刪除 Neuvector

$ oc delete -f neuvector.yaml

須確認 Pods 都已被刪除

Step8: 重新建立 Neuvector

$ oc create -f neuvector.yaml

檢視 Pods 運作狀態

$ oc get pods

螢幕輸出 :

NAME READY STATUS RESTARTS AGE
neuvector-controller-pod-74d88c4df6-h7npj 1/1 Running 0 3m43s
neuvector-controller-pod-74d88c4df6-k6fvg 1/1 Running 0 3m43s
neuvector-controller-pod-74d88c4df6-th8lj 1/1 Running 0 3m43s
neuvector-enforcer-pod-b2n8r 1/1 Running 0 3m43s
neuvector-enforcer-pod-mzvkb 1/1 Running 0 3m43s
neuvector-enforcer-pod-rdvq8 1/1 Running 0 3m43s
neuvector-manager-pod-6bb5555c7b-zlq78 1/1 Running 0 3m43s
neuvector-operator-86dd64f497-nqmbg 1/1 Running 2 19d
neuvector-scanner-pod-56c798bb86-qpfc6 1/1 Running 0 3m43s
neuvector-scanner-pod-56c798bb86-v29w8 1/1 Running 0 3m43

Step9: 確認設定和規則都還存在

連線至 Neuvector Web Console

掃描 Image Registry 的相關設定

Admission Control 規則

Config 會被備份的種類

$ ls -l /var/neuvector/config/backup

螢幕輸出 :

total 1332
-rw-------. 1 root root 2615 Sep 19 08:11 admission_control.backup
-rw-------. 1 root root 96 Sep 19 07:53 compliance.backup
-rw-------. 1 root root 378 Sep 19 07:53 crd.backup
-rw-------. 1 root root 14903 Sep 19 07:53 dlp_group.backup
-rw-------. 1 root root 2539 Sep 19 07:53 dlp_rule.backup
-rw-------. 1 root root 29463 Sep 19 07:53 domain.backup
-rw-------. 1 root root 41 Sep 19 07:53 eula_oss.backup
-rw-------. 1 root root 488 Sep 19 07:53 federation.backup
-rw-------. 1 root root 433060 Sep 19 07:53 file_monitor.backup
-rw-------. 1 root root 554801 Sep 19 07:53 file_rule.backup
-rw-------. 1 root root 80549 Sep 19 07:53 group.backup
-rw-------. 1 root root 72943 Sep 19 08:06 policy.backup
-rw-------. 1 root root 93486 Sep 19 07:53 process_profile.backup
-rw-------. 1 root root 464 Sep 19 07:53 pwd_profile.backup
-rw-------. 1 root root 664 Sep 19 07:53 registry.backup
-rw-------. 1 root root 1738 Sep 19 07:53 response_rule.backup
-rw-------. 1 root root 1295 Sep 19 07:53 system.backup
-rw-------. 1 root root 532 Sep 19 08:02 user.backup
-rw-r--r--. 1 root root 45 Sep 19 07:53 version.backup
-rw-------. 1 root root 76 Sep 19 07:53 vulnerability.backup
-rw-------. 1 root root 14903 Sep 19 07:53 waf_group.backup
-rw-------. 1 root root 1481 Sep 19 07:53 waf_rule.backup

Ref

Backups and Persistent Data - Neuvector docs

Introduction to NeuVector

Tue, 19 Sep 2023 09:09:56 +0800

Preface

本篇文章會介紹以下內容

甚麼是 NeuVector ，它的功能有哪些?

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

5.x Overview

NeuVector 提供雲原生的 Kubernetes 安全平台

Features

1. CI/CD Vulnerability Management & Admission Control

透過 Jenkins 掃描 Image，掃描 Registries 並執行 Admission Control Rule (准入控制規則)，以便部署到生產中。

2. Violation Protection

發現行為並創建「白名單」的策略，用來偵測違反正常行為。

3. Threat Detection

檢測常見的應用攻擊，如對 Container 的 DDoS 和 DNS 攻擊。

4. DLP and WAF Sensors

檢查網路流量以防止敏感數據丟失，並檢測常見的 OWASP Top10 WAF 攻擊。

5. Run-time Vulnerability Scanning

掃描 Registries、Image，以及在 Kubernetes 和主機上運行的 Container ，以尋找常見的（CVE）和應用程式特定的安全漏洞。

6. Compliance & Auditing

自動運行 Docker Bench 測試和 Kubernetes CIS Benchmarks。

7. Endpoint/Host Security

檢測權限升級，監控主機和 Container 內的程序和檔案活動，並監控 Container 檔案系統的可疑活動。

8. Multi-cluster Management

從一個控制台監控和管理多個 Kubernetes 叢集。

Manage SUSE NeuVector Using REST API

Fri, 11 Aug 2023 09:31:26 +0800

Preface

本篇文章會介紹，

透過 Swagger 給的 API 指令與 Neuvector 溝通

可以透過點擊展開以下目錄，選擇想看的內容，跳轉至特定章節

Setup Neuvector

Expose REST API

To expose the REST API for access from outside of the Kubernetes cluster, enable port 10443.

apiVersion: v1
kind: Service
metadata:
name: neuvector-service-rest
namespace: neuvector
spec:
ports:
- port: 10443
name: controller
protocol: TCP
type: NodePort
selector:
app: neuvector-controller-pod

建立 Service

$ oc create -f neuvector-service-rest.yaml

檢視服務狀態

$ oc get svc neuvector-service-rest

螢幕輸出 :

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
neuvector-service-rest NodePort 172.30.84.78 <none> 10443:32215/TCP 3h8m

Swagger

Install Swagger with Podman

$ podman run -d -p 80:8080 docker.io/swaggerapi/swagger-ui

-d，將 Container 推到背景執行
-p，將 Container 的 8080 Port 對應到 Host 主機的 80 Port

Access Swagger Web

打開瀏覽器連線至 Podman Host 主機的 80 Port

http://192.168.11.211/

匯入設定檔

在上方搜尋欄輸入以下網址，並點選 “Explore” 按鈕

https://raw.githubusercontent.com/neuvector/neuvector/main/controller/api/apis.yaml

Setup Authentication

Create an authentication token

在 swagger 的網站找到 Authentication 標題底下的 /v1/auth Login Authentication，點擊展開後，再點選 Try it out 按鈕，就可以對 body 的部分進行修改，

修改後點 Execute 就可以產出 Sample 指令，再根據實際環境進行修改

以下是一個實際範例，執行以下命令，生成身分驗證的 Token

curl -s -k -X 'POST' \
'https://192.168.11.82:32215/v1/auth' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"client_ip": "",
"password": {
"username": "admin",
"password": "admin"
},
"Token": {
"token": "",
"state": "",
"redirect_endpoint": ""
}
}' | jq -r '.token.token' > token.json

檢視 token.json 檔案內容

$ cat token.json

螢幕輸出 :

sUeU1TK648M4Y2RsScwiVceXB3zDuAPtIbzEHO_r++vMUbdriGCkjCVlcvVjZuK2dzV9hLgce_oKhJlhGXexL_4kmdJ0REtiwrTZb94uudgdjXz8QlkyzXsjbGpHVnqjMDhVqQSoJZ7WMQ+mslJOd6oHPiff476Xyl0+vwTj9tPlEkdeZkZcKjKH37aFbtbvrinCUpwIVl9kuJEV72OmL8p5kcjufydHPW0E6auxiLLmVKp46gki4hPt7CjgiSbtMG4vfanNMnOFpibmwRp9tRfDFnb4Q7z8b1iiuAouXqftJ1VMHuv2LKX_wuKTQAmj0e020_jwmR+MVgTwqVDQ5Uu0NmaeouSTuKbj39clCa+HzXWARNv735QsBDOAeoPt4r6BZTu1mPvo2A5ztcroAL77rZvtEpV0s2xTzxlVAYsXAK1Nu5asUDRqrWbtMkRJIbzXVBq6JYJE8w18+txwbo+JazJQU5vOWxIL93aQtzIveper+vmWjYHd5ieBOvHNlfjt4WJEnfxZ0A61RANPFEfb8HCNQCeFnv4heNV8GuAqstm5BxmsZVWIxW3_b+p0TFaqAl3Sfz1LlewbVFERGKoBNsWRYMsOPD5vgwgZP73r+SLJCVJGIyPbr1RugjOyackA_MFVjtjuLJzINR_IBpp6eiIctWF3Oqlj8Z2a_GhAA8HtYvcXFoM49E8V8IyQxN_KLGkSw_7HbI6LY63gKUzrAeisMuD9iBFkvqxURYEjhmu6F0UI6OFMR55S72GV4Q6afHnPRvmMiPDMuQ1h_bfZAvaGmQryhiboOu8mmcXap0mQLkp+FIBHhQgrqNmvcbC0BHkw859cxV4OOWRKdVVfSNC6mr6VG8uPxn29uLSonbmTc985pYhUfC0LAoQ+Y+gmeKfmsN_DebwMW72tVjb9EBbiTlXLTO+tj9BkdzCWb2zH1d9IBOqkaExEaw7JcMOzv44PqN1NBUy_SiO6nb7QXgNCGc7KKxoP7CrqLOSsUbxg1UxNhMQXNU0sdz3trAy8QUHP2kExTFbW_0_GVjA4hClx1b25GMt3nBhaJT1tp0aSnQMj47zsgJGz5ybxa6bAsUVcQEipA4dlA3orrkBZ0gKkQofpvBF1xAz8h2j6I5HLqbHj1RhT

Keep login session alive

$ _token_=$(cat token.json)
$ curl -s -k -X 'PATCH' \
'https://192.168.11.82:32215/v1/auth' \
-H 'accept: application/json' \
-H "X-Auth-Token: $_token_"

Logout current logged in user

$ _token_=$(cat token.json)
$ curl -s -k -X 'DELETE' \
'https://192.168.11.82:32215/v1/auth' \
-H 'accept: application/json' \
-H "X-Auth-Token: $_token_"

Process

檢視特定 Group 的 Process Profile Rules

$ _token_=$(cat token.json)
$ curl -s -k -X 'GET' \
'https://192.168.11.82:32215/v1/process_profile/nv.nginx.neuvector' \
-H 'accept: application/json' \
-H "X-Auth-Token: $_token_"

螢幕輸出 :

{
"process_profile": {
"baseline": "basic",
"group": "nv.nginx.neuvector",
"mode": "Protect",
"process_list": [
{
"action": "allow",
"allow_update": false,
"cfg_type": "user_created",
"created_timestamp": 1691398024,
"last_modified_timestamp": 1691398024,
"name": "bash",
"path": "/bin/bash",
"uuid": "505a3ce0-c1c0-44a3-89ee-4bc6ed7ef675"
},
{
"action": "allow",
"allow_update": false,
"cfg_type": "user_created",
"created_timestamp": 1691398143,
"last_modified_timestamp": 1691398143,
"name": "cat",
"path": "/bin/cat",
"uuid": "5ff92222-1818-45ed-9b9c-9edcebdb9125"
},
{
"action": "allow",
"allow_update": false,
"cfg_type": "user_created",
"created_timestamp": 1691398053,
"last_modified_timestamp": 1691398191,
"name": "ls",
"path": "/bin/ls",
"uuid": "55380afb-3164-40cb-a830-ca352b9c0a2a"
},
{
"action": "allow",
"allow_update": false,
"cfg_type": "learned",
"created_timestamp": 1691397615,
"last_modified_timestamp": 1691397615,
"name": "nginx",
"path": "/usr/sbin/nginx",
"user": "root",
"uuid": "3e286470-aa88-4ae6-adcc-26c60d6446c5"
},
{
"action": "allow",
"allow_update": false,
"cfg_type": "user_created",
"created_timestamp": 1691398031,
"last_modified_timestamp": 1691398031,
"name": "sh",
"path": "/bin/sh",
"uuid": "36de28dc-cf30-4795-920e-afd93b6af790"
},
{
"action": "allow",
"allow_update": false,
"cfg_type": "user_created",
"created_timestamp": 1691398216,
"last_modified_timestamp": 1691398216,
"name": "whoami",
"path": "/usr/bin/whoami",
"uuid": "b1aa8a28-1f28-4247-a4f4-f3540cf41634"
}
]
}
}

合規

Sat, 10 Jun 2023 07:37:36 +0800

Common Criteria

目的

幫助買家確信任和產品的規範、實施和評估過程都是以標準化的方式進行。也就是說，已經根據國際公認的規格進行了獨立驗證，認明它是值得被信賴的

Data Center、Internet Service Providers(電信公司)、金融和醫療的組織…等，非常重視安全的非政府公司也會要求使用它

適用對象

Computer systems
- OS
- Databases
- Network Devices
- …等

這份文件定義了一個能在網路環境下運作的通用作業系統所預期要提供的安全功能。

Linux 合規逐字稿

Tue, 06 Jun 2023 17:02:06 +0800

資訊安全的重要性

維護資料的機密性
1. 假設在家目錄，有一個檔案，裡面的內容包含了你的銀行帳號、密碼和其他敏感資料，這個檔案的權限是 777，存放這個檔案的目錄權限也是 777，權限都是 3 個 7 ，是不是代表會很幸運 ? 開玩笑地，下午幫大家提個神，是不是就代表這份資料非常的不機密。再來，
2. 我今天下了一個命令 chmod 000 /etc/shadow，這樣是不是很機密，還沒完，有一台 Container 在啟動的時候，它的 Data Volume 把 Host 主機上的 /etc/shadow 這個檔案掛進來，怎麼辦? 你的資料是不是就不是這麼給他感覺不機密了。
維護資料的完整性
確保系統的可用性
遵守法律法規
保護企業聲譽

Multi Tenancy Security AAA

Sat, 07 Jan 2023 05:56:08 +0800

Kubernetes 一個大樓標準藍圖

把 Kubernetes 想成一棟大樓的標準藍圖，把自己想成要當地產大亨，首先要先確認多少土地，能蓋多少大樓和住戶…等，在 kubernetes 中，土地代表建立 k8s 的資源，以目前來說就是虛擬平台、三大公有雲 or 實體機，接著要開始蓋房子，以大樓來看，就是分住戶、公共建設和管理委員等，公共建設是 K8s 啟動後須先安裝的

Multi Tenancy

Soft multi-tenancy
- which doesn’t have very strict isolation between tenants, is aimed at preventing accidental interference, and is suitable for trusted tenants.
- 多人共用 k8s ，透過內建軟體隔離。
Hard multi-tenancy
- which assumes tenants cannot be trusted, enforces stricter isolation to protect tenants from malicious interference, and is suitable for both trusted and untrusted tenants.
- 一個用戶一個 k8s ，代表會啟動不同的 K8s Cluster

AAA

Authentication 身分認證
- 驗證你是誰
Authorization 授權
- 你有什麼權限，可以做什麼事情
Accounting(Auditing) 統計
- 使用者行為的紀錄

Landlord 角色設計

Gateway
- 提供 SSH 登入的入口，透過 Deployment 分流，將連線轉向用戶專用的資源。
Tenant
- 用戶的專用資源，每個用戶家目錄都有 wk 資料夾，供用戶儲存個人資料，屬於永存資料區塊。
- 透過 K8s 的 namespace 做環境的隔離
Kuser
- 建立 Tenant 對應的 K8S 認證及授權。
Logger
- 接收 Tenant 操作命令及檔案編輯歷程，送至資料庫儲存。
Mariadb
- 儲存 Tenant 操作命令及檔案編輯歷程

Landlord 架構圖

Landlord 資料流

Landlord 監控

Prometheus：住戶社區整體監控中心
Grafana：監控中心裡的超大型 LED 儀表板

問題討論

為甚麼腳色大多使用 Deployment 設計
- 當管理的 pod 毀損或被刪除時，會自動再產生同樣的 pod 出來
為甚麼要有 Gateway 這個腳色
- 當我們透過 ssh -p 20000 tenant-0@Gateway_Headless_Service_ExternalIP 連線時，
- Gateway 裡面的 /etc/profile 會再放一段 sshpass -p \$(whoami) ssh \$(whoami)@\$(whoami).${USERTAG}.${NS} -q 2>/dev/null 連線到 Tenant 的程式，
- Tenant 透過 StatefulSet 建出來的 pod 名字會有流水號，搭配無頭服務和 CoreDNS 來達到名稱解析
- 當 Tenant 數量增加時，Gateway 也能知道連到哪一個 Tenant
- ${USERTAG} 是 StatefulSet 的名字
為甚麼 Tenant 要用 StatefulSets 設計
- pod 的名字是流水號
- 當所在主機掛點時，可在另一台主機生出來
- 多個 pod 可共用 PVC
Tenant 是透過何種方式達到永存資料
- 使用 Local Path Provisioner 來達到永存
- 當 PV 所在的 node 掛了，資料還是救不回來，因為 PVC 建不出來 pod 也會無法建立
- 可使用 NFS Storage 把資料備份到其他主機
- 最佳解決方案 Ceph
可不可以沒有 Logger ，將資料直接放入 Mariadb
- 不行，要保護 Mariadb 不被 Tanent 隨意破壞
- Tanent 傳給 Logger 資料，再統一由 Logger 做資料傳送到 Mariadb

tags: `系統工程`

`/etc/ssh/sshd_config` 的說明

Sun, 07 Aug 2022 14:47:52 +0800

一、修改nano /etc/ssh/sshd_config 文件

1、修改默認端口：默認Port為22,並且已經註釋掉了；修改是把註釋去掉，並修改成其它的端口。 2、禁止root用戶遠程登陸：修改PermitRootLogin，默認為yes且註釋掉了；修改是把註釋去掉，並改成no。 3、PermitEmptyPasswords no不允許空密碼用戶login

二、ssh的公鑰認證配置：修改vi /etc/ssh/sshd_config 文件 RSAAuthentication yes # 啟用RSA 認證（默認是註釋掉的，將註釋去掉，如果不是yes，改為yes） PubkeyAuthentication yes # 啟用公鑰認證（默認是註釋掉的，將註釋去掉，如果不是yes，改為yes） PasswordAuthentication no # 禁止密碼認證(改為no,默認為yes是用密碼認證) StrictModes no #修改為no,默認為yes.如果不修改用key登陸是出現server refused our key(如果StrictModes為yes必需保證存放公鑰的文件夾的擁有與登陸用戶名是相同的.“StrictModes”設置ssh在接收登錄請求之前是否檢查用戶家目錄和rhosts文件的權限和所有權。這通常是必要的，因為新手經常會把自己的目錄和文件設成任何人都有寫權限。)

之後重新啟動ssh服務:/etc/init.d/ssh restart

生成登陸公鑰與私鑰， [root@usousou192_168_0_21 ssh]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /home/linden.guo/.ssh/id_rsa (生成私鑰與公鑰存放位置) Enter passphrase (empty for no passphrase): 輸入密碼 Enter same passphrase again:再次輸入密碼 Your identification has been saved in /home/linden.guo/.ssh/id_rsa. (生成的私鑰) Your public key has been saved in /home/linden.guo/.ssh/id_rsa.pub. (生成的公鑰) The key fingerprint is: 76:04:4d:44:25:37:0f:b1:a5:b7:6e:63:d4:97:22:6b root@usousou192_168_0_21

將生成的公鑰匙id_rsa.pub傳到要登陸的服務器上並追加到authorized_keys文件中，放到用戶目錄的.ssh中cat id_rsa.pub » .ssh/authorized_keys (如果沒有authorized_keys，可直接將id_rsa.pub重命名為authorized_keys,自己認為，沒有測試過)

使用putty連接ssh服務器。為了使用公鑰認證，我們需要同時下載puttygen 這個工具來生成putty 所使用的密鑰如果你按照上面的介紹，在Linux下生成了公鑰和密鑰的話，那麼需要利用puttygen 將密鑰轉換成putty 使用的格式。將Linux下生成的密鑰id_rsa 複製到Windows 下。啟動puttygen，然後單擊Load按鈕，選擇文件類型為所有文件，然後選擇id_rsa，打開。若在生成密鑰時輸入了密碼，則打開時需要輸入該密碼(用linux生成密鑰時輸入的密碼)。之後就可以在puttygen 的主界面上單擊Save private key，保存成putty 格式的密鑰。

最好確定用戶目錄下的.ssh文件夾對於擁有者有讀寫執行的權限，最低要有執行權限，如700或者100；authorized_keys文件中有讀的權限

注：AuthorizedKeysFile .ssh/authorized_keys（認證文件的目錄與公鑰文件名稱，可以修改，並且相應目錄也要修改，如AuthorizedKeysFile .sshd/linden.guo_keys，需要在用戶目錄下建立.sshd文件夾，將linden.guo_keys文件放到下面）

三、為什麼要使用公鑰認證通常，通過ssh登錄遠程服務器時，使用密碼認證，分別輸入用戶名和密碼，兩者滿足一定規則就可以登錄。但是密碼認證有以下的缺點：

用戶無法設置空密碼（即使系統允許空密碼，也會十分危險）密碼容易被人偷窺或猜到服務器上的一個帳戶若要給多人使用，則必須讓所有使用者都知道密碼，導緻密碼容易洩露，而且修改密碼時必須通知所有人而使用公鑰認證則可以解決上述問題。

公鑰認證允許使用空密碼，省去每次登錄都需要輸入密碼的麻煩多個使用者可以通過各自的密鑰登錄到系統上的同一個用戶公鑰認證的原理所謂的公鑰認證，實際上是使用一對加密字符串，一個稱為公鑰(public key)，任何人都可以看到其內容，用於加密；另一個稱為密鑰(private key)，只有擁有者才能看到，用於解密。通過公鑰加密過的密文使用密鑰可以輕鬆解密，但根據公鑰來猜測密鑰卻十分困難。

ssh 的公鑰認證就是使用了這一特性。服務器和客戶端都各自擁有自己的公鑰和密鑰。為了說明方便，以下將使用這些符號。

Ac 客戶端公鑰 Bc 客戶端密鑰 As 服務器公鑰 Bs 服務器密鑰

在認證之前，客戶端需要通過某種方法將公鑰Ac 登錄到服務器上。

認證過程分為兩個步驟。

會話密鑰(session key)生成客戶端請求連接服務器，服務器將As 發送給客戶端。服務器生成會話ID(session id)，設為p，發送給客戶端。客戶端生成會話密鑰(session key)，設為q，併計算r = p xor q。客戶端將r 用As 進行加密，結果發送給服務器。服務器用Bs 進行解密，獲得r。服務器進行r xor p 的運算，獲得q。至此服務器和客戶端都知道了會話密鑰q，以後的傳輸都將被q 加密。認證服務器生成隨機數x，並用Ac 加密後生成結果S(x)，發送給客戶端客戶端使用Bc 解密S(x) 得到x 客戶端計算q + x 的md5 值n(q+x)，q為上一步得到的會話密鑰服務器計算q + x 的md5 值m(q+x) 客戶端將n(q+x) 發送給服務器服務器比較m(q+x) 和n(q+x)，兩者相同則認證成功服務器端設置使用公鑰認證需要對服務器進行一些設置。修改/etc/sshd_config 的以下配置。

RSAAuthentication yes # 啟用RSA 認證 PubkeyAuthentication yes # 啟用公鑰認證 PasswordAuthentication no # 禁止密碼認證StrictModes no #修改為no,默認為yes.如果不修改用key登陸是出現server refused our key然後重新啟動sshd。

/etc/init.d/ssh restart客戶端設置 Linux 假設客戶端的用戶charlee 要以guest 用戶登錄到服務器上。首先在客戶端執行下面的命令。

[charlee@client:~]$ ssh-keygen -t rsa Generating public/private rsa1 key pair. Enter file in which to save the key (/home/charlee/.ssh/id_rsa): Enterpassphrase (empty for no passphrase): 輸入密碼 Enter same passphrase again: 再次輸入密碼 Your identification has been sabed in /home/charlee/.ssh/id_rsa Your public key has been saved in /home/charlee/.ssh/id_rsa.pub生成的文件保存在主目錄的.ssh 目錄下，id_rsa為客戶端密鑰，id_rsa.pub 為客戶端公鑰。

之後，通過U 盤等方式將公鑰id_rsa.pub 複製到服務器上，並執行下列命令。

[guest@server:~]$ cat id_rsa.pub » .ssh/authorized_keys其中id_rsa.pub 是客戶端的用戶charlee 的公鑰。

這樣在客戶端即可通過以下的命令連接服務器。

[charlee@client:~]$ ssh -l guest server若不想每次登錄服務器時都輸入密碼，可以先執行下列命令：

[charlee@client:~]$ ssh-add Enter passphrase for /home/charlee/.ssh/id_rsa: 輸入密碼 Identity added: /home/charlee/.ssh/id_rsa (/home/charlee/.ssh/id_rsa)以後登錄服務器就不需要輸入密碼了。

Windows 假設我們使用putty連接ssh服務器。為了使用公鑰認證，我們需要同時下載puttygen 這個工具來生成putty 所使用的密鑰。

使用Linux下生成的公鑰和密鑰的情況如果你按照上面的介紹，在Linux下生成了公鑰和密鑰的話，那麼需要利用puttygen 將密鑰轉換成putty 使用的格式。

將Linux下生成的密鑰id_rsa 複製到Windows 下。啟動puttygen，然後單擊Load按鈕，選擇文件類型為所有文件，然後選擇id_rsa，打開。若在生成密鑰時輸入了密碼，則打開時需要輸入該密碼。之後就可以在puttygen 的主界面上單擊Save private key，保存成putty 格式的密鑰。

使用puttygen生成公鑰和密鑰你也可以利用puttygen 直接生成公鑰和密鑰。打開puttygen，然後在畫面下方的Parameters 欄選擇加密算法和加密長度（一般取默認值即可），最後單擊Generate。畫面上會出現一個進度條，在界面上隨意移動鼠標以生成隨機數。最後提示生成結束，單擊Save private key 按鈕保存密鑰。最後將上方的Public key for pasting into OpenSSH authorized_keys file欄中的內容複製到Linux 下用戶主目錄下的~/.ssh/authorized_keys 文件中。

設置putty 使用公鑰認證先在putty中設置好連接的主機名、端口等信息（說明省略）。使用公鑰認證需要設置兩個地方：一個是Connection 中的Auto-login username，輸入在服務器上的用戶名；另一個是Connection->SSH->Auth中的Private key file for authentication，選擇剛才轉換或生成的.ppk 格式的密鑰。之後即可使用公鑰認證進行連接了。