316 lines
10 KiB
Markdown
316 lines
10 KiB
Markdown
# Working with Kyverno
|
|
|
|
[Kyverno](https://github.com/kyverno/kyverno) , a [Cloud Native Computing Foundation ](https://cncf.io/) project, is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations using admission controls and background scans. Kyverno policies are Kubernetes resources and do not require learning a new language. Kyverno is designed to work nicely with tools you already use like kubectl, kustomize, and Git.
|
|
|
|
This document gives an example to demonstrate how to use the `Kyverno` to manage policy.
|
|
|
|
## Prerequisites
|
|
## Setup Karmada
|
|
|
|
You just need to clone Karmada repo, and run the following script in Karmada directory.
|
|
|
|
```
|
|
hack/local-up-karmada.sh
|
|
```
|
|
|
|
## Kyverno Installations
|
|
|
|
In this case, we will use Kyverno v1.6.2. Related deployment files are from [here](https://github.com/kyverno/kyverno/blob/main/config/install.yaml).
|
|
|
|
### Install Kyverno APIs on Karmada
|
|
|
|
1. Create resource objects of Kyverno in karmada controller plane, the content is as follows.
|
|
|
|
```console
|
|
kubectl config use-context karmada-apiserver
|
|
```
|
|
|
|
Deploy namespace: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L1-L12
|
|
|
|
Deploy configmap: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L12144-L12176
|
|
|
|
Deploy Kyverno CRDs: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L12-L11656
|
|
|
|
### Install Kyverno components on host cluster
|
|
|
|
1. Create resource objects of Kyverno in karmada-host context, the content is as follows.
|
|
|
|
```console
|
|
kubectl config use-context karmada-host
|
|
```
|
|
|
|
Deploy namespace: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L1-L12
|
|
|
|
Deploy RBAC resources: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L11657-L12143
|
|
|
|
Deploy Kyverno controllers and service:
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
labels:
|
|
app: kyverno
|
|
app.kubernetes.io/component: kyverno
|
|
app.kubernetes.io/instance: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
app.kubernetes.io/part-of: kyverno
|
|
app.kubernetes.io/version: latest
|
|
name: kyverno-svc
|
|
namespace: kyverno
|
|
spec:
|
|
type: NodePort
|
|
ports:
|
|
- name: https
|
|
port: 443
|
|
targetPort: https
|
|
nodePort: {{nodePort}}
|
|
selector:
|
|
app: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
labels:
|
|
app: kyverno
|
|
app.kubernetes.io/component: kyverno
|
|
app.kubernetes.io/instance: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
app.kubernetes.io/part-of: kyverno
|
|
app.kubernetes.io/version: latest
|
|
name: kyverno-svc-metrics
|
|
namespace: kyverno
|
|
spec:
|
|
ports:
|
|
- name: metrics-port
|
|
port: 8000
|
|
targetPort: metrics-port
|
|
selector:
|
|
app: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
labels:
|
|
app: kyverno
|
|
app.kubernetes.io/component: kyverno
|
|
app.kubernetes.io/instance: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
app.kubernetes.io/part-of: kyverno
|
|
app.kubernetes.io/version: latest
|
|
name: kyverno
|
|
namespace: kyverno
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
strategy:
|
|
rollingUpdate:
|
|
maxSurge: 1
|
|
maxUnavailable: 40%
|
|
type: RollingUpdate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: kyverno
|
|
app.kubernetes.io/component: kyverno
|
|
app.kubernetes.io/instance: kyverno
|
|
app.kubernetes.io/name: kyverno
|
|
app.kubernetes.io/part-of: kyverno
|
|
app.kubernetes.io/version: latest
|
|
spec:
|
|
affinity:
|
|
podAntiAffinity:
|
|
preferredDuringSchedulingIgnoredDuringExecution:
|
|
- podAffinityTerm:
|
|
labelSelector:
|
|
matchExpressions:
|
|
- key: app.kubernetes.io/name
|
|
operator: In
|
|
values:
|
|
- kyverno
|
|
topologyKey: kubernetes.io/hostname
|
|
weight: 1
|
|
containers:
|
|
- args:
|
|
- --filterK8sResources=[Event,*,*][*,kube-system,*][*,kube-public,*][*,kube-node-lease,*][Node,*,*][APIService,*,*][TokenReview,*,*][SubjectAccessReview,*,*][*,kyverno,kyverno*][Binding,*,*][ReplicaSet,*,*][ReportChangeRequest,*,*][ClusterReportChangeRequest,*,*][PolicyReport,*,*][ClusterPolicyReport,*,*]
|
|
- -v=2
|
|
- --autogenInternals=false
|
|
- --kubeconfig=/etc/kubeconfig
|
|
- --serverIP={{nodeIP}}:{{nodePort}}
|
|
env:
|
|
- name: INIT_CONFIG
|
|
value: kyverno
|
|
- name: METRICS_CONFIG
|
|
value: kyverno-metrics
|
|
- name: KYVERNO_NAMESPACE
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: metadata.namespace
|
|
- name: KYVERNO_SVC
|
|
value: kyverno-svc
|
|
- name: TUF_ROOT
|
|
value: /.sigstore
|
|
image: ghcr.io/kyverno/kyverno:latest
|
|
imagePullPolicy: Always
|
|
livenessProbe:
|
|
failureThreshold: 2
|
|
httpGet:
|
|
path: /health/liveness
|
|
port: 9443
|
|
scheme: HTTPS
|
|
initialDelaySeconds: 15
|
|
periodSeconds: 30
|
|
successThreshold: 1
|
|
timeoutSeconds: 5
|
|
name: kyverno
|
|
ports:
|
|
- containerPort: 9443
|
|
name: https
|
|
protocol: TCP
|
|
- containerPort: 8000
|
|
name: metrics-port
|
|
protocol: TCP
|
|
readinessProbe:
|
|
failureThreshold: 4
|
|
httpGet:
|
|
path: /health/readiness
|
|
port: 9443
|
|
scheme: HTTPS
|
|
initialDelaySeconds: 5
|
|
periodSeconds: 10
|
|
successThreshold: 1
|
|
timeoutSeconds: 5
|
|
resources:
|
|
limits:
|
|
memory: 384Mi
|
|
requests:
|
|
cpu: 100m
|
|
memory: 128Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
privileged: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
volumeMounts:
|
|
- mountPath: /.sigstore
|
|
name: sigstore
|
|
- mountPath: /etc/kubeconfig
|
|
name: kubeconfig
|
|
subPath: kubeconfig
|
|
initContainers:
|
|
- args:
|
|
- env:
|
|
- name: METRICS_CONFIG
|
|
value: kyverno-metrics
|
|
- name: KYVERNO_NAMESPACE
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: metadata.namespace
|
|
image: ghcr.io/kyverno/kyvernopre:latest
|
|
imagePullPolicy: Always
|
|
name: kyverno-pre
|
|
resources:
|
|
limits:
|
|
cpu: 100m
|
|
memory: 256Mi
|
|
requests:
|
|
cpu: 10m
|
|
memory: 64Mi
|
|
securityContext:
|
|
allowPrivilegeEscalation: false
|
|
capabilities:
|
|
drop:
|
|
- ALL
|
|
privileged: false
|
|
readOnlyRootFilesystem: true
|
|
runAsNonRoot: true
|
|
securityContext:
|
|
runAsNonRoot: true
|
|
serviceAccountName: kyverno-service-account
|
|
volumes:
|
|
- emptyDir: {}
|
|
name: sigstore
|
|
- name: kubeconfig
|
|
secret:
|
|
defaultMode: 420
|
|
secretName: kubeconfig
|
|
---
|
|
apiVersion: v1
|
|
stringData:
|
|
kubeconfig: |-
|
|
apiVersion: v1
|
|
clusters:
|
|
- cluster:
|
|
certificate-authority-data: {{ca_crt}}
|
|
server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443
|
|
name: kind-karmada
|
|
contexts:
|
|
- context:
|
|
cluster: kind-karmada
|
|
user: kind-karmada
|
|
name: karmada
|
|
current-context: karmada
|
|
kind: Config
|
|
preferences: {}
|
|
users:
|
|
- name: kind-karmada
|
|
user:
|
|
client-certificate-data: {{client_cer}}
|
|
client-key-data: {{client_key}}
|
|
kind: Secret
|
|
metadata:
|
|
name: kubeconfig
|
|
namespace: kyverno
|
|
```
|
|
|
|
For multi-cluster deployment, We need to add the config of `--serverIP` which is the address of the webhook server. So you need to ensure that the network from node in karmada control plane to those in karmada-host cluster is connected and expose kyverno controller pods to control plane, for example, using `nodePort` above. Then, fill in the secret which represents kubeconfig pointing to karmada-apiserver, such as **ca_crt, client_cer and client_key** above.
|
|
|
|
## Run demo
|
|
### Create require-labels ClusterPolicy
|
|
|
|
ClusterPolicy is a CRD which `kyverno` offers to support different kinds of rules. Here is an example ClusterPolicy which means that you must create pod with `app.kubernetes.io/name` label.
|
|
|
|
```console
|
|
kubectl config use-context karmada-apiserver
|
|
```
|
|
```console
|
|
kubectl create -f- << EOF
|
|
apiVersion: kyverno.io/v1
|
|
kind: ClusterPolicy
|
|
metadata:
|
|
name: require-labels
|
|
spec:
|
|
validationFailureAction: enforce
|
|
rules:
|
|
- name: check-for-labels
|
|
match:
|
|
any:
|
|
- resources:
|
|
kinds:
|
|
- Pod
|
|
validate:
|
|
message: "label 'app.kubernetes.io/name' is required"
|
|
pattern:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/name: "?*"
|
|
EOF
|
|
```
|
|
|
|
### Create a bad deployment without labels
|
|
|
|
```console
|
|
kubectl create deployment nginx --image=nginx
|
|
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request
|
|
```
|
|
|
|
## Reference
|
|
|
|
- https://github.com/kyverno/kyverno |