mirror of https://github.com/istio/istio.io.git
Tao Li
Martin Taillefer
parent
56e87f1568
commit
521a2fe7d9
|
|
@ -0,0 +1,21 @@
|
||||||
|
---
|
||||||
|
title: Extending Istio Self-Signed Root Certificate Lifetime
|
||||||
|
description: Learn how to extend the lifetime of Istio self-signed root certificate.
|
||||||
|
publishdate: 2019-06-07
|
||||||
|
attribution: Oliver Liu
|
||||||
|
keywords: [security, PKI, certificate, Citadel]
|
||||||
|
---
|
||||||
|
|
||||||
|
Istio self-signed certificates have historically had a 1 year default lifetime.
|
||||||
|
If you are using Istio self-signed certificates,
|
||||||
|
you need to schedule regular root transitions before they expire.
|
||||||
|
An expiration of a root certificate may lead to an unexpected cluster-wide outage.
|
||||||
|
The issue affects all versions up to 1.0.7 and 1.1.7.
|
||||||
|
|
||||||
|
See [Extending Self-Signed Certificate Lifetime](/help/ops/security/root-transition/) for
|
||||||
|
information on how to gauge the age of your certificates and how to perform rotation.
|
||||||
|
|
||||||
|
{{< tip >}}
|
||||||
|
We strongly recommend you rotate root keys and root certificates annually as a security best practice.
|
||||||
|
We will send out instructions for root key/cert rotation as a follow-up.
|
||||||
|
{{< /tip >}}
|
||||||
|
|
@ -0,0 +1,199 @@
|
||||||
|
---
|
||||||
|
title: Extending Self-Signed Certificate Lifetime
|
||||||
|
description: Learn how to extend the lifetime of the Istio self-signed root certificate.
|
||||||
|
weight: 90
|
||||||
|
keywords: [security, PKI, certificate, Citadel]
|
||||||
|
---
|
||||||
|
|
||||||
|
Istio self-signed certificates have historically had a 1 year default lifetime.
|
||||||
|
If you are using Istio self-signed certificates,
|
||||||
|
you need to schedule regular root transitions before they expire.
|
||||||
|
An expiration of a root certificate may lead to an unexpected cluster-wide outage.
|
||||||
|
|
||||||
|
{{< tip >}}
|
||||||
|
We strongly recommend you rotate root keys and root certificates annually as a security best practice.
|
||||||
|
We will send out instructions for root key/cert rotation as a follow-up.
|
||||||
|
{{< /tip >}}
|
||||||
|
|
||||||
|
To evaluate the lifetime remaining for your root certificate, please refer to the first step in the
|
||||||
|
[procedure below](#root-transition-procedure).
|
||||||
|
|
||||||
|
We provide the following procedure for you to do the root transition.
|
||||||
|
Note that the Envoy instances will be hot restarted to reload the new root certificates, which may impact long-lived connections.
|
||||||
|
For details about the impacts and how Envoy hot restart works, please refer to
|
||||||
|
[here](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/hot_restart#arch-overview-hot-restart) and
|
||||||
|
[here](https://blog.envoyproxy.io/envoy-hot-restart-1d16b14555b5).
|
||||||
|
|
||||||
|
## Scenarios
|
||||||
|
|
||||||
|
If you are not currently using the mutual TLS feature in Istio and will not use it in the future,
|
||||||
|
you are not affected and no action is required.
|
||||||
|
You may choose to upgrade to 1.0.8, 1.1.8 or later versions to avoid this problem in the future.
|
||||||
|
|
||||||
|
If you are not currently using the mutual TLS feature in Istio and may use it in the future,
|
||||||
|
you are recommended to follow the procedure listed below to upgrade.
|
||||||
|
|
||||||
|
If you are currently using the mutual TLS feature in Istio with self-signed certificates,
|
||||||
|
please follow the procedure and check whether you will be affected.
|
||||||
|
|
||||||
|
## Root transition procedure
|
||||||
|
|
||||||
|
1. Check when the root certificate expires:
|
||||||
|
|
||||||
|
Download this [script](https://raw.githubusercontent.com/istio/tools/master/bin/root-transition.sh)
|
||||||
|
on a machine that has `kubectl` access to the cluster.
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ wget https://raw.githubusercontent.com/istio/tools/master/bin/root-transition.sh
|
||||||
|
$ chmod +x root-transition.sh
|
||||||
|
$ ./root-transition.sh check
|
||||||
|
...
|
||||||
|
===YOU HAVE 30 DAYS BEFORE THE ROOT CERT EXPIRES!=====
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
Execute the remainder of the steps prior to root certificate expiration to avoid system outages.
|
||||||
|
|
||||||
|
1. Execute a root certificate transition:
|
||||||
|
|
||||||
|
During the transition, the Envoy sidecars may be hot-restarted to reload the new certificates.
|
||||||
|
This may have some impact on your traffic. Please refer to
|
||||||
|
[Envoy hot restart](https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/hot_restart#arch-overview-hot-restart)
|
||||||
|
and read [this](https://blog.envoyproxy.io/envoy-hot-restart-1d16b14555b5)
|
||||||
|
blog post for more details.
|
||||||
|
|
||||||
|
{{< warning >}}
|
||||||
|
If your Pilot does not have an Envoy sidecar, consider installing Envoy sidecar for your Pilot.
|
||||||
|
Because the Pilot has issue using the old root certificate to verify the new workload certificates.
|
||||||
|
This may cause disconnection between Pilot and Envoy.
|
||||||
|
Please see the [here](#how-to-check-if-pilot-has-an-envoy-sidecar) for how to check.
|
||||||
|
The [Istio upgrade guide](/docs/setup/kubernetes/upgrade/steps/)
|
||||||
|
by default installs Pilot with Envoy sidecar.
|
||||||
|
{{< /warning >}}
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ ./root-transition.sh transition
|
||||||
|
Create new ca cert, with trust domain as cluster.local
|
||||||
|
Wed Jun 5 19:11:15 PDT 2019 delete old ca secret
|
||||||
|
secret "istio-ca-secret" deleted
|
||||||
|
Wed Jun 5 19:11:15 PDT 2019 create new ca secret
|
||||||
|
secret/istio-ca-secret created
|
||||||
|
pod "istio-citadel-8574b88bcd-j7v2d" deleted
|
||||||
|
Wed Jun 5 19:11:18 PDT 2019 restarted Citadel, checking status
|
||||||
|
NAME READY STATUS RESTARTS AGE
|
||||||
|
istio-citadel-8574b88bcd-l2g74 1/1 Running 0 3s
|
||||||
|
New root certificate:
|
||||||
|
Certificate:
|
||||||
|
Data:
|
||||||
|
...
|
||||||
|
Validity
|
||||||
|
Not Before: Jun 6 03:24:43 2019 GMT
|
||||||
|
Not After : Jun 3 03:24:43 2029 GMT
|
||||||
|
Subject: O = cluster.local
|
||||||
|
...
|
||||||
|
Your old certificate is stored as old-ca-cert.pem, and your private key is stored as ca-key.pem
|
||||||
|
Please save them safely and privately.
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
1. Verify the new workload certificates are generated:
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ ./root-transition.sh verify
|
||||||
|
...
|
||||||
|
Checking the current root CA certificate is propagated to all the Istio-managed workload secrets in the cluster.
|
||||||
|
Root cert MD5 is 8fa8229ab89122edba73706e49a55e4c
|
||||||
|
Checking namespace: default
|
||||||
|
Secret default.istio.default is updated.
|
||||||
|
Secret default.istio.sleep is updated.
|
||||||
|
Checking namespace: istio-system
|
||||||
|
Secret istio-system.istio.default is updated.
|
||||||
|
...
|
||||||
|
------All Istio keys and certificates are updated in secret!
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
If this command fails, wait a minute and run the command again.
|
||||||
|
It takes some time for Citadel to propagate the certificates.
|
||||||
|
|
||||||
|
1. Upgrade to Istio 1.0.8, 1.1.8 or later:
|
||||||
|
|
||||||
|
{{< warning >}}
|
||||||
|
To ensure the control plane components and Envoy sidecars all load the new certificates and keys, this step is mandatory.
|
||||||
|
{{< /warning >}}
|
||||||
|
|
||||||
|
Upgrade your control plane and `istio-proxy` sidecars to 1.0.8, 1.1.8 or later.
|
||||||
|
Please follow the Istio [upgrade procedure](/docs/setup/kubernetes/upgrade/steps/).
|
||||||
|
|
||||||
|
1. Verify the new workload certificates are loaded by Envoy:
|
||||||
|
|
||||||
|
You can verify whether an Envoy has received the new certificates.
|
||||||
|
The following command shows an example to check the Envoy’s certificate for pod _foo_ running in namespace _bar_.
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ kubectl exec -it foo -c istio-proxy -n bar -- curl http://localhost:15000/certs | head -c 1000
|
||||||
|
{
|
||||||
|
"certificates": [
|
||||||
|
{
|
||||||
|
"ca_cert": [
|
||||||
|
...
|
||||||
|
"valid_from": "2019-06-06T03:24:43Z",
|
||||||
|
"expiration_time": ...
|
||||||
|
],
|
||||||
|
"cert_chain": [
|
||||||
|
{
|
||||||
|
...
|
||||||
|
}
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
Please inspect the `valid\_from` value of the `ca\_cert`.
|
||||||
|
If it matches the `_Not_ _Before_` value in the new certificate as shown in Step 2,
|
||||||
|
your Envoy has loaded the new root certificate.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Can I upgrade to 1.0.8, 1.1.8 or later first, and then do the root transition?
|
||||||
|
|
||||||
|
Yes, you can. You can upgrade to 1.0.8, 1.1.8 or later as normal.
|
||||||
|
After that, follow the root transition steps and in Step 4,
|
||||||
|
manually restart Galley, Pilot and sidecar-injector to ensure they load the new root certificates.
|
||||||
|
|
||||||
|
### Why my workloads do not pick up the new certificates (in Step 5)?
|
||||||
|
|
||||||
|
Please make sure you have updated to 1.0.8, 1.1.8 or later for the `istio-proxy` sidecars in Step 4.
|
||||||
|
|
||||||
|
{{< warning >}}
|
||||||
|
If you are using Istio releases 1.1.3 - 1.1.7, the Envoy may not be hot-restarted
|
||||||
|
after the new certificates are generated.
|
||||||
|
{{< /warning >}}
|
||||||
|
|
||||||
|
### Why my Pilot does not work and logs "handshake error"?
|
||||||
|
|
||||||
|
This may because the Pilot is
|
||||||
|
[not using an Envoy sidecar](#how-to-check-if-pilot-has-an-envoy-sidecar),
|
||||||
|
while the `controlPlaneSecurity` is enabled.
|
||||||
|
In this case, restart both Galley and Pilot to ensure they load the new certificates.
|
||||||
|
As an example, the following commands redeploy a pod for Galley / Pilot by removing a pod.
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ kubectl delete po <galley-pod> -n istio-system
|
||||||
|
$ kubectl delete po <pilot-pod> -n istio-system
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
### How to check if Pilot has an Envoy sidecar
|
||||||
|
|
||||||
|
If the following command shows `1/1`, that means your Pilot does not have an Envoy sidecar,
|
||||||
|
otherwise, if it is showing `2/2`, your Pilot is using an Envoy sidecar.
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ kubectl get po -l istio=pilot -n istio-system
|
||||||
|
istio-pilot-569bc6d9c-tfwjr 1/1 Running 0 11m
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
### I can't deploy new workloads with the sidecar-injector
|
||||||
|
|
||||||
|
This may happen if you did not upgrade to 1.0.8, 1.1.8 or later.
|
||||||
|
Try to restart the sidecar injector.
|
||||||
|
The sidecar injector will reload the certificate after the restart:
|
||||||
|
|
||||||
|
{{< text bash>}}
|
||||||
|
$ kubectl delete po -l istio=sidecar-injector -n istio-system
|
||||||
|
pod "istio-sidecar-injector-788bd8fc48-x9gdc" deleted
|
||||||
|
{{< /text >}}
|
||||||
Loading…
Reference in New Issue