Merge pull request #1675 from Poor12/add-karmada-kyverno-guide
[Docs]Add karmada-kyverno-guide for users
This commit is contained in:
karmada-bot
GitHub
commit
9de4a16f0e
|
|
@ -411,7 +411,7 @@ In this case, you will use Gatekeeper v3.7.2. Related deployment files are from
|
||||||
|
|
||||||
### Extra steps
|
### Extra steps
|
||||||
|
|
||||||
After all, we need to copy the secret `gatekeeper-webhook-server-cert` in karmada-apiserver context to that in karmada-host context to keep secrets stored in `etcd` and volumes mounted in controller the same.
|
After all, we need to copy the secret `gatekeeper-webhook-server-cert` in karmada-apiserver context to that in karmada-host context to keep secrets stored in `etcd` and volumes mounted in controller the same.
|
||||||
|
|
||||||
## Run demo
|
## Run demo
|
||||||
### Create k8srequiredlabels template
|
### Create k8srequiredlabels template
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,316 @@
|
||||||
|
# Working with Kyverno
|
||||||
|
|
||||||
|
[Kyverno](https://github.com/kyverno/kyverno) , a [Cloud Native Computing Foundation ](https://cncf.io/) project, is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations using admission controls and background scans. Kyverno policies are Kubernetes resources and do not require learning a new language. Kyverno is designed to work nicely with tools you already use like kubectl, kustomize, and Git.
|
||||||
|
|
||||||
|
This document gives an example to demonstrate how to use the `Kyverno` to manage policy.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
## Setup Karmada
|
||||||
|
|
||||||
|
You just need to clone Karmada repo, and run the following script in Karmada directory.
|
||||||
|
|
||||||
|
```
|
||||||
|
hack/local-up-karmada.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Kyverno Installations
|
||||||
|
|
||||||
|
In this case, we will use Kyverno v1.6.2. Related deployment files are from [here](https://github.com/kyverno/kyverno/blob/main/config/install.yaml).
|
||||||
|
|
||||||
|
### Install Kyverno APIs on Karmada
|
||||||
|
|
||||||
|
1. Create resource objects of Kyverno in karmada controller plane, the content is as follows.
|
||||||
|
|
||||||
|
```console
|
||||||
|
kubectl config use-context karmada-apiserver
|
||||||
|
```
|
||||||
|
|
||||||
|
Deploy namespace: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L1-L12
|
||||||
|
|
||||||
|
Deploy configmap: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L12144-L12176
|
||||||
|
|
||||||
|
Deploy Kyverno CRDs: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L12-L11656
|
||||||
|
|
||||||
|
### Install Kyverno components on host cluster
|
||||||
|
|
||||||
|
1. Create resource objects of Kyverno in karmada-host context, the content is as follows.
|
||||||
|
|
||||||
|
```console
|
||||||
|
kubectl config use-context karmada-host
|
||||||
|
```
|
||||||
|
|
||||||
|
Deploy namespace: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L1-L12
|
||||||
|
|
||||||
|
Deploy RBAC resources: https://github.com/kyverno/kyverno/blob/61a1d40e5ea5ff4875a084b6dc3ef1fdcca1ee27/config/install.yaml#L11657-L12143
|
||||||
|
|
||||||
|
Deploy Kyverno controllers and service:
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/component: kyverno
|
||||||
|
app.kubernetes.io/instance: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
app.kubernetes.io/part-of: kyverno
|
||||||
|
app.kubernetes.io/version: latest
|
||||||
|
name: kyverno-svc
|
||||||
|
namespace: kyverno
|
||||||
|
spec:
|
||||||
|
type: NodePort
|
||||||
|
ports:
|
||||||
|
- name: https
|
||||||
|
port: 443
|
||||||
|
targetPort: https
|
||||||
|
nodePort: {{nodePort}}
|
||||||
|
selector:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/component: kyverno
|
||||||
|
app.kubernetes.io/instance: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
app.kubernetes.io/part-of: kyverno
|
||||||
|
app.kubernetes.io/version: latest
|
||||||
|
name: kyverno-svc-metrics
|
||||||
|
namespace: kyverno
|
||||||
|
spec:
|
||||||
|
ports:
|
||||||
|
- name: metrics-port
|
||||||
|
port: 8000
|
||||||
|
targetPort: metrics-port
|
||||||
|
selector:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
---
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/component: kyverno
|
||||||
|
app.kubernetes.io/instance: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
app.kubernetes.io/part-of: kyverno
|
||||||
|
app.kubernetes.io/version: latest
|
||||||
|
name: kyverno
|
||||||
|
namespace: kyverno
|
||||||
|
spec:
|
||||||
|
replicas: 1
|
||||||
|
selector:
|
||||||
|
matchLabels:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
strategy:
|
||||||
|
rollingUpdate:
|
||||||
|
maxSurge: 1
|
||||||
|
maxUnavailable: 40%
|
||||||
|
type: RollingUpdate
|
||||||
|
template:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app: kyverno
|
||||||
|
app.kubernetes.io/component: kyverno
|
||||||
|
app.kubernetes.io/instance: kyverno
|
||||||
|
app.kubernetes.io/name: kyverno
|
||||||
|
app.kubernetes.io/part-of: kyverno
|
||||||
|
app.kubernetes.io/version: latest
|
||||||
|
spec:
|
||||||
|
affinity:
|
||||||
|
podAntiAffinity:
|
||||||
|
preferredDuringSchedulingIgnoredDuringExecution:
|
||||||
|
- podAffinityTerm:
|
||||||
|
labelSelector:
|
||||||
|
matchExpressions:
|
||||||
|
- key: app.kubernetes.io/name
|
||||||
|
operator: In
|
||||||
|
values:
|
||||||
|
- kyverno
|
||||||
|
topologyKey: kubernetes.io/hostname
|
||||||
|
weight: 1
|
||||||
|
containers:
|
||||||
|
- args:
|
||||||
|
- --filterK8sResources=[Event,*,*][*,kube-system,*][*,kube-public,*][*,kube-node-lease,*][Node,*,*][APIService,*,*][TokenReview,*,*][SubjectAccessReview,*,*][*,kyverno,kyverno*][Binding,*,*][ReplicaSet,*,*][ReportChangeRequest,*,*][ClusterReportChangeRequest,*,*][PolicyReport,*,*][ClusterPolicyReport,*,*]
|
||||||
|
- -v=2
|
||||||
|
- --autogenInternals=false
|
||||||
|
- --kubeconfig=/etc/kubeconfig
|
||||||
|
- --serverIP={{nodeIP}}:{{nodePort}}
|
||||||
|
env:
|
||||||
|
- name: INIT_CONFIG
|
||||||
|
value: kyverno
|
||||||
|
- name: METRICS_CONFIG
|
||||||
|
value: kyverno-metrics
|
||||||
|
- name: KYVERNO_NAMESPACE
|
||||||
|
valueFrom:
|
||||||
|
fieldRef:
|
||||||
|
fieldPath: metadata.namespace
|
||||||
|
- name: KYVERNO_SVC
|
||||||
|
value: kyverno-svc
|
||||||
|
- name: TUF_ROOT
|
||||||
|
value: /.sigstore
|
||||||
|
image: ghcr.io/kyverno/kyverno:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
livenessProbe:
|
||||||
|
failureThreshold: 2
|
||||||
|
httpGet:
|
||||||
|
path: /health/liveness
|
||||||
|
port: 9443
|
||||||
|
scheme: HTTPS
|
||||||
|
initialDelaySeconds: 15
|
||||||
|
periodSeconds: 30
|
||||||
|
successThreshold: 1
|
||||||
|
timeoutSeconds: 5
|
||||||
|
name: kyverno
|
||||||
|
ports:
|
||||||
|
- containerPort: 9443
|
||||||
|
name: https
|
||||||
|
protocol: TCP
|
||||||
|
- containerPort: 8000
|
||||||
|
name: metrics-port
|
||||||
|
protocol: TCP
|
||||||
|
readinessProbe:
|
||||||
|
failureThreshold: 4
|
||||||
|
httpGet:
|
||||||
|
path: /health/readiness
|
||||||
|
port: 9443
|
||||||
|
scheme: HTTPS
|
||||||
|
initialDelaySeconds: 5
|
||||||
|
periodSeconds: 10
|
||||||
|
successThreshold: 1
|
||||||
|
timeoutSeconds: 5
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
memory: 384Mi
|
||||||
|
requests:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 128Mi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
privileged: false
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
volumeMounts:
|
||||||
|
- mountPath: /.sigstore
|
||||||
|
name: sigstore
|
||||||
|
- mountPath: /etc/kubeconfig
|
||||||
|
name: kubeconfig
|
||||||
|
subPath: kubeconfig
|
||||||
|
initContainers:
|
||||||
|
- args:
|
||||||
|
- env:
|
||||||
|
- name: METRICS_CONFIG
|
||||||
|
value: kyverno-metrics
|
||||||
|
- name: KYVERNO_NAMESPACE
|
||||||
|
valueFrom:
|
||||||
|
fieldRef:
|
||||||
|
fieldPath: metadata.namespace
|
||||||
|
image: ghcr.io/kyverno/kyvernopre:latest
|
||||||
|
imagePullPolicy: Always
|
||||||
|
name: kyverno-pre
|
||||||
|
resources:
|
||||||
|
limits:
|
||||||
|
cpu: 100m
|
||||||
|
memory: 256Mi
|
||||||
|
requests:
|
||||||
|
cpu: 10m
|
||||||
|
memory: 64Mi
|
||||||
|
securityContext:
|
||||||
|
allowPrivilegeEscalation: false
|
||||||
|
capabilities:
|
||||||
|
drop:
|
||||||
|
- ALL
|
||||||
|
privileged: false
|
||||||
|
readOnlyRootFilesystem: true
|
||||||
|
runAsNonRoot: true
|
||||||
|
securityContext:
|
||||||
|
runAsNonRoot: true
|
||||||
|
serviceAccountName: kyverno-service-account
|
||||||
|
volumes:
|
||||||
|
- emptyDir: {}
|
||||||
|
name: sigstore
|
||||||
|
- name: kubeconfig
|
||||||
|
secret:
|
||||||
|
defaultMode: 420
|
||||||
|
secretName: kubeconfig
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
stringData:
|
||||||
|
kubeconfig: |-
|
||||||
|
apiVersion: v1
|
||||||
|
clusters:
|
||||||
|
- cluster:
|
||||||
|
certificate-authority-data: {{ca_crt}}
|
||||||
|
server: https://karmada-apiserver.karmada-system.svc.cluster.local:5443
|
||||||
|
name: kind-karmada
|
||||||
|
contexts:
|
||||||
|
- context:
|
||||||
|
cluster: kind-karmada
|
||||||
|
user: kind-karmada
|
||||||
|
name: karmada
|
||||||
|
current-context: karmada
|
||||||
|
kind: Config
|
||||||
|
preferences: {}
|
||||||
|
users:
|
||||||
|
- name: kind-karmada
|
||||||
|
user:
|
||||||
|
client-certificate-data: {{client_cer}}
|
||||||
|
client-key-data: {{client_key}}
|
||||||
|
kind: Secret
|
||||||
|
metadata:
|
||||||
|
name: kubeconfig
|
||||||
|
namespace: kyverno
|
||||||
|
```
|
||||||
|
|
||||||
|
For multi-cluster deployment, We need to add the config of `--serverIP` which is the address of the webhook server. So you need to ensure that the network from node in karmada control plane to those in karmada-host cluster is connected and expose kyverno controller pods to control plane, for example, using `nodePort` above. Then, fill in the secret which represents kubeconfig pointing to karmada-apiserver, such as **ca_crt, client_cer and client_key** above.
|
||||||
|
|
||||||
|
## Run demo
|
||||||
|
### Create require-labels ClusterPolicy
|
||||||
|
|
||||||
|
ClusterPolicy is a CRD which `kyverno` offers to support different kinds of rules. Here is an example ClusterPolicy which means that you must create pod with `app.kubernetes.io/name` label.
|
||||||
|
|
||||||
|
```console
|
||||||
|
kubectl config use-context karmada-apiserver
|
||||||
|
```
|
||||||
|
```console
|
||||||
|
kubectl create -f- << EOF
|
||||||
|
apiVersion: kyverno.io/v1
|
||||||
|
kind: ClusterPolicy
|
||||||
|
metadata:
|
||||||
|
name: require-labels
|
||||||
|
spec:
|
||||||
|
validationFailureAction: enforce
|
||||||
|
rules:
|
||||||
|
- name: check-for-labels
|
||||||
|
match:
|
||||||
|
any:
|
||||||
|
- resources:
|
||||||
|
kinds:
|
||||||
|
- Pod
|
||||||
|
validate:
|
||||||
|
message: "label 'app.kubernetes.io/name' is required"
|
||||||
|
pattern:
|
||||||
|
metadata:
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: "?*"
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
### Create a bad deployment without labels
|
||||||
|
|
||||||
|
```console
|
||||||
|
kubectl create deployment nginx --image=nginx
|
||||||
|
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reference
|
||||||
|
|
||||||
|
- https://github.com/kyverno/kyverno
|
||||||
Loading…
Reference in New Issue