mirror of https://github.com/istio/istio.io.git
124 lines
5.3 KiB
Markdown
124 lines
5.3 KiB
Markdown
---
|
|
title: Citadel Health Checking
|
|
description: Shows how to enable Citadel health checking with Kubernetes.
|
|
weight: 70
|
|
keywords: [security,health-check]
|
|
---
|
|
|
|
You can enable Citadel's health checking feature
|
|
to detect the failures of the Citadel CSR (Certificate Signing Request) service.
|
|
When a failure is detected, Kubelet automatically restarts the Citadel container.
|
|
|
|
When the health checking feature is enabled,
|
|
the **prober client** module in Citadel periodically checks the health status of Citadel's CSR gRPC server.
|
|
It does this by sending CSRs to the gRPC server and verifies the responses.
|
|
If Citadel is healthy, the _prober client_ updates the _modification time_ of the _health status file_.
|
|
Otherwise, it does nothing. Citadel relies on a
|
|
[Kubernetes liveness and readiness probe](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)
|
|
with command line to check the _modification time_ of the _health status file_ on the pod.
|
|
If the file is not updated for a period, Kubelet will restart the Citadel container.
|
|
|
|
Note: because Citadel health checking currently only monitors the health status of CSR service API,
|
|
this feature is not needed if the production setup is not using the
|
|
[SDS](/docs/tasks/security/auth-sds/) or [Mesh Expansion](/docs/examples/mesh-expansion/).
|
|
|
|
## Before you begin
|
|
|
|
To complete this task, you can install Istio using one of the following paths:
|
|
|
|
* To setup Istio without using Helm, follow the instructions in the
|
|
[Kubernetes installation guide](/docs/setup/kubernetes/install/kubernetes/). Remember to enable global mutual TLS with:
|
|
|
|
{{< text bash >}}
|
|
$ kubectl apply -f install/kubernetes/istio-demo-auth.yaml
|
|
{{< /text >}}
|
|
|
|
* Use [Helm](/docs/setup/kubernetes/install/helm/) to setup Istio and set the `global.mtls.enabled` flag to `true`.
|
|
|
|
{{< tip >}}
|
|
Use an [authentication policy](/docs/concepts/security/#authentication-policies) to configure mutual TLS for
|
|
all or only selected services in a namespace. You must repeat the policy for all namespaces to configure the setting globally.
|
|
See the [authentication policy task](/docs/tasks/security/authn-policy/) for details.
|
|
{{< /tip >}}
|
|
|
|
## Deploying Citadel with health checking
|
|
|
|
To enable health checking, redeploy Citadel with helm:
|
|
|
|
{{< text bash >}}
|
|
$ helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
|
|
-x charts/security/templates/deployment.yaml \
|
|
--set global.mtls.enabled=true --set security.citadelHealthCheck=true > citadel-health-check.yaml
|
|
$ kubectl apply -f citadel-health-check.yaml
|
|
{{< /text >}}
|
|
|
|
## Verify that health checking works
|
|
|
|
Citadel will log the health checking results. Run the following in command line:
|
|
|
|
{{< text bash >}}
|
|
$ kubectl logs `kubectl get po -n istio-system | grep istio-citadel | awk '{print $1}'` -n istio-system | grep "CSR signing service"
|
|
{{< /text >}}
|
|
|
|
You will see the output similar to:
|
|
|
|
{{< text plain >}}
|
|
... CSR signing service is healthy (logged every 100 times).
|
|
{{< /text >}}
|
|
|
|
The log above indicates the periodic health checking is working.
|
|
The default health checking interval is 15 seconds and is logged once every 100 checks.
|
|
|
|
## (Optional) Configuring the health checking
|
|
|
|
This section talks about how to modify the health checking configuration. Open the file
|
|
`citadel-health-check.yaml`, and locate the following lines.
|
|
|
|
{{< text plain >}}
|
|
...
|
|
- --liveness-probe-path=/tmp/ca.liveness # path to the liveness health checking status file
|
|
- --liveness-probe-interval=60s # interval for health checking file update
|
|
- --probe-check-interval=15s # interval for health status check
|
|
livenessProbe:
|
|
exec:
|
|
command:
|
|
- /usr/local/bin/istio_ca
|
|
- probe
|
|
- --probe-path=/tmp/ca.liveness # path to the liveness health checking status file
|
|
- --interval=125s # the maximum time gap allowed between the file mtime and the current sys clock.
|
|
initialDelaySeconds: 60
|
|
periodSeconds: 60
|
|
...
|
|
{{< /text >}}
|
|
|
|
The paths to the health status files are `liveness-probe-path` and `probe-path`.
|
|
You should update the paths in Citadel and in the `livenessProbe` at the same time.
|
|
If Citadel is healthy, the value of the `liveness-probe-interval` entry determines the interval used to update the
|
|
health status file.
|
|
The Citadel health checking controller uses the value of the `probe-check-interval` entry to determine the interval to
|
|
call the Citadel CSR service.
|
|
The `interval` is the maximum time elapsed since the last update of the health status file, for the prober to consider
|
|
Citadel as healthy.
|
|
The values in the `initialDelaySeconds` and `periodSeconds`entries determine the initial delay and the interval between
|
|
each activation of the `livenessProbe`.
|
|
|
|
Prolonging `probe-check-interval` will reduce the health checking overhead, but there will be a greater lagging for the
|
|
prober to get notified on the unhealthy status.
|
|
To avoid the prober restarting Citadel due to temporary unavailability, the `interval` on the prober can be
|
|
configured to be more than `N` times of the `liveness-probe-interval`. This will allow the prober to tolerate `N-1`
|
|
continuously failed health checks.
|
|
|
|
## Cleanup
|
|
|
|
* To disable health checking on Citadel:
|
|
|
|
{{< text bash >}}
|
|
$ kubectl apply -f install/kubernetes/istio-demo-auth.yaml
|
|
{{< /text >}}
|
|
|
|
* To remove Citadel:
|
|
|
|
{{< text bash >}}
|
|
$ kubectl delete -f install/kubernetes/istio-citadel-with-health-check.yaml
|
|
{{< /text >}}
|