Add debugging authorization guide. (#1999)

This commit is contained in:
Yangmin 2018-07-30 20:45:35 -07:00 committed by Martin Taillefer
parent 1bb6e5a4ec
commit 649c42a8c0
5 changed files with 341 additions and 21 deletions

View File

@ -4,14 +4,17 @@ description: Authorization is enabled, but requests make it through anyway.
weight: 50
---
If authorization checks are enabled for a service and yet requests to the service aren't being blocked, then
its likely that authorization was not actually enabled successfully. You can do the following to verify:
authorization was likely not enabled successfully. To verify, follow these steps:
1. Check [this page](/docs/concepts/security/#authorization) to find out how to correctly enable Istio authorization policy. Use `istioctl` instead of `kubectl`, because `istioctl` enforces an additional validation check to prevent errors caused by typos.
1. Check the [enable authorization docs](/docs/concepts/security/#enabling-authorization) to correctly enable
Istio authorization.
1. Avoid enabling authorization for Istio Control Planes Components, including Mixer, Pilot, Ingress. Istio authorization policy is designed for authorizing access to services in Istio Mesh. Enabling it for Istio Control Planes Components may cause unexpected behavior.
1. Avoid enabling authorization for Istio Control Planes Components, including Mixer, Pilot and Ingress.
The Istio authorization features are designed for authorizing access to services in an Istio Mesh.
Enabling the authorization features for the Istio Control Planes components can cause unexpected behavior.
1. In Kubernetes environment, check deployments in all namespaces to make sure there is no legacy deployment left that could cause error in Pilot. Authorization plugin in Pilot could be disabled if there is error pushing authorization policy to Envoy.
1. In your Kubernetes environment, check deployments in all namespaces to make sure there is no legacy
deployment left that can cause an error in Pilot. You can disable Pilot's authorization plug-in if
there is an error pushing authorization policy to Envoy.
1. Check Pilot log (or collect Pilot logs and file a GitHub issue) to find out the exact cause.
1. For Mixer based authorization, check Mixer logs (or collect Mixer logs and file a GitHub issue) to make sure that the RBAC adapter is enabled successfully.
1. Follow the [Debugging Authorization docs](/help/ops/security/debugging-authorization/) to find out the exact cause.

View File

@ -7,7 +7,7 @@ weight: 60
When you first enable authorization for a service, all requests are denied by default. After you add one or more authorization policies, then
matching requests should flow through. If all requests continue to be denied, you can try the following:
1. Make sure there is no typo in your policy yaml file.
1. Make sure there is no typo in your policy YAML file.
1. Avoid enabling authorization for Istio Control Planes Components, including Mixer, Pilot, Ingress. Istio authorization policy is designed for authorizing access to services in Istio Mesh. Enabling it for Istio Control Planes Components may cause unexpected behavior.
@ -18,7 +18,4 @@ matching requests should flow through. If all requests continue to be denied, yo
`default` namespace (`metadata/namespace` line should be `default`). For non-Kubernetes environments, all `ServiceRoles` and `ServiceRoleBindings`
for a mesh should be in the same namespace.
1. Check Pilot and proxy log (or collect Pilot and proxy log and file a GitHub issue) to find out the exact cause. You could collect the LDS configuration generated by Pilot by running
[`dump_kubernetes.sh`]({{< github_file >}}/tools/dump_kubernetes.sh).
1. For Mixer based authorization, it is often the case that Istio Proxy to Mixer connection is down, Mixer attribute manifest is not setup correctly. Check Mixer Log (or collect Mixer log and file a GitHub issue) to find out the exact cause.
1. Follow the [Debugging Authorization docs](/help/ops/security/debugging-authorization/) to find out the exact cause.

View File

@ -0,0 +1,254 @@
---
title: Debugging Authorization
description: Demonstrates how to debug authorization.
weight: 5
keywords: [debug,security,authorization,RBAC]
---
This page demonstrates how to debug Istio authorization. If you still cannot find solutions after following
this page, feel free to send an email to `istio-security@googlegroups.com` for help.
> {{< idea_icon >}}
It would be very helpful to also include a cluster state archive in your email by following instructions in
[reporting bugs](/help/bugs).
## Ensure Authorization is Enabled Correctly
The `rbacConfig` default cluster level singleton custom resource controls the authorization functionality globally.
1. Run the following command to list existing `RbacConfig`:
{{< text bash >}}
$ kubectl get rbacconfigs.rbac.istio.io --all-namespaces
{{< /text >}}
1. Verify there is only **one** instance of `RbacConfig` with name `default`. Otherwise, Istio disables the
authorization functionality and ignores all policies.
{{< text plain >}}
NAMESPACE NAME AGE
default default 1d
{{< /text >}}
1. If there is more than one `RbacConfig` instance, remove any additional `RbacConfig` instances and
ensure **only one** instance is named `default`.
## Ensure Pilot Accepts the Policies
Pilot converts and distributes your authorization policies to the proxies. The following steps help
you ensure Pilot is working as expected:
1. Run the following command to export the Pilot `ControlZ`:
{{< text bash >}}
$ kubectl port-forward $(kubectl -n istio-system get pods -l istio=pilot -o jsonpath='{.items[0].metadata.name}') -n istio-system 9876:9876
{{< /text >}}
1. Verify you see the following output:
{{< text plain >}}
Forwarding from 127.0.0.1:9876 -> 9876
{{< /text >}}
1. Start your browser and open the `ControlZ` page at `http://127.0.0.1:9876/scopez/`.
1. Change the `rbac` Output Level to `debug`.
1. Use `Ctrl+C` in the terminal you started in step 1 to stop the port-forward command.
1. Print the log of Pilot and search for `rbac` with the following command:
> Note: You probably need to first delete and then re-apply your authorization policies so that
the debug output is generated for these policies.
{{< text bash >}}
$ kubectl logs $(kubectl -n istio-system get pods -l istio=pilot -o jsonpath='{.items[0].metadata.name}') -c discovery -n istio-system | grep rbac
{{< /text >}}
1. Check the output and verify:
* There are no errors.
* There is a `"built filter config for ..."` message which means the filter is generated
for the target service.
1. For example, you might see something similar to the following:
{{< text plain >}}
2018-07-26T22:25:41.009838Z debug rbac building filter config for {sleep.foo.svc.cluster.local map[app:sleep pod-template-hash:3326367878] map[destination.name:sleep destination.namespace:foo destination.user:default]}
2018-07-26T22:25:41.009915Z info rbac no service role in namespace foo
2018-07-26T22:25:41.009957Z info rbac no service role binding in namespace foo
2018-07-26T22:25:41.010000Z debug rbac generated filter config: { }
2018-07-26T22:25:41.010114Z info rbac built filter config for sleep.foo.svc.cluster.local
2018-07-26T22:25:41.182400Z debug rbac building filter config for {productpage.default.svc.cluster.local map[pod-template-hash:2600844901 version:v1 app:productpage] map[destination.name:productpage destination.namespace:default destination.user:bookinfo-productpage]}
2018-07-26T22:25:41.183131Z debug rbac checking role app2-grpc-viewer
2018-07-26T22:25:41.183214Z debug rbac role skipped for no AccessRule matched
2018-07-26T22:25:41.183255Z debug rbac checking role productpage-viewer
2018-07-26T22:25:41.183281Z debug rbac matched AccessRule[0]
2018-07-26T22:25:41.183390Z debug rbac generated filter config: {policies:<key:"productpage-viewer" value:<permissions:<and_rules:<rules:<or_rules:<rules:<header:<name:":method" exact_match:"GET" > > > > > > principals:<and_ids:<ids:<any:true > > > > > }
2018-07-26T22:25:41.184407Z info rbac built filter config for productpage.default.svc.cluster.local
{{< /text >}}
It means Pilot generated:
* An empty config for `sleep.foo.svc.cluster.local` as there is no authorization policies matched
and Istio denies all requests sent to this service by default.
* An config for `productpage.default.svc.cluster.local` and Istio will allow anyone to access it
with GET method.
## Ensure Pilot Distributes Policies to Proxies Correctly
Pilot distributes the authorization policies to proxies. The following steps help you ensure Pilot
is working as expected:
> Note: The command used in this section assumes you have deployed [Bookinfo application](/docs/examples/bookinfo/),
otherwise you should replace `"-l app=productpage"` with your actual pod.
1. Run the following command to get the proxy configuration dump for the `productpage` service:
{{< text bash >}}
$ kubectl exec $(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl localhost:15000/config_dump -s
{{< /text >}}
1. Check the log and verify:
* The log includes an `envoy.filters.http.rbac` filter to enforce the authorization policy
on each incoming request.
* Istio updates the filter accordingly after you update your authorization policy.
1. The following output means the proxy of `productpage` has enabled the `envoy.filters.http.rbac` filter
with rules that allows anyone to access it via `GET` method. The `shadow_rules` are not used and you can ignored them safely.
{{< text plain >}}
{
"name": "envoy.filters.http.rbac",
"config": {
"rules": {
"policies": {
"productpage-viewer": {
"permissions": [
{
"and_rules": {
"rules": [
{
"or_rules": {
"rules": [
{
"header": {
"exact_match": "GET",
"name": ":method"
}
}
]
}
}
]
}
}
],
"principals": [
{
"and_ids": {
"ids": [
{
"any": true
}
]
}
}
]
}
}
},
"shadow_rules": {
"policies": {}
}
}
},
{{< /text >}}
## Ensure Proxies Enforce Policies Correctly
Proxies eventually enforce the authorization policies. The following steps help you ensure the proxy
is working as expected:
> Note: The command used in this section assumes you have deployed [Bookinfo application](/docs/examples/bookinfo/).
otherwise you should replace `"-l app=productpage"` with your actual pod.
1. Turn on the authorization debug logging in proxy with the following command:
{{< text bash >}}
$ kubectl exec $(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl localhost:15000/logging?rbac=debug -s
{{< /text >}}
1. Verify you see the following output:
{{< text plain >}}
active loggers:
... ...
rbac: debug
... ...
{{< /text >}}
1. Visit the `productpage` in your browser to generate some logs.
1. Print the proxy logs with the following command:
{{< text bash >}}
$ kubectl logs $(kubectl get pods -l app=productpage -o jsonpath='{.items[0].metadata.name}') -c istio-proxy
{{< /text >}}
1. Check the output and verify:
* The output log shows either `enforced allowed` or `enforced denied` depending on whether the request
was allowed or denied respectively.
* Your authorization policy expects the data extracted from the request.
1. The following output means there is a `GET` request at path `/productpage` and the policy allows the request.
The `shadow denied` has no effect and you can ignore it safely.
{{< text plain >}}
...
[2018-07-26 20:39:18.060][152][debug][rbac] external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:79] checking request: remoteAddress: 10.60.0.139:51158, localAddress: 10.60.0.93:9080, ssl: uriSanPeerCertificate: spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account, subjectPeerCertificate: O=, headers: ':authority', '35.238.0.62'
':path', '/productpage'
':method', 'GET'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
'dnt', '1'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'
'x-forwarded-for', '10.60.0.1'
'x-forwarded-proto', 'http'
'x-request-id', 'e23ea62d-b25d-91be-857c-80a058d746d4'
'x-b3-traceid', '5983108bf6d05603'
'x-b3-spanid', '5983108bf6d05603'
'x-b3-sampled', '1'
'x-istio-attributes', 'CikKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRINEgtwcm9kdWN0cGFnZQoqCh1kZXN0aW5hdGlvbi5zZXJ2aWNlLm5hbWVzcGFjZRIJEgdkZWZhdWx0Ck8KCnNvdXJjZS51aWQSQRI/a3ViZXJuZXRlczovL2lzdGlvLWluZ3Jlc3NnYXRld2F5LTc2NjY0Y2NmY2Ytd3hjcjQuaXN0aW8tc3lzdGVtCj4KE2Rlc3RpbmF0aW9uLnNlcnZpY2USJxIlcHJvZHVjdHBhZ2UuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbApDChhkZXN0aW5hdGlvbi5zZXJ2aWNlLmhvc3QSJxIlcHJvZHVjdHBhZ2UuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbApBChdkZXN0aW5hdGlvbi5zZXJ2aWNlLnVpZBImEiRpc3RpbzovL2RlZmF1bHQvc2VydmljZXMvcHJvZHVjdHBhZ2U='
'content-length', '0'
'x-envoy-internal', 'true'
'sec-istio-authn-payload', 'CkVjbHVzdGVyLmxvY2FsL25zL2lzdGlvLXN5c3RlbS9zYS9pc3Rpby1pbmdyZXNzZ2F0ZXdheS1zZXJ2aWNlLWFjY291bnQSRWNsdXN0ZXIubG9jYWwvbnMvaXN0aW8tc3lzdGVtL3NhL2lzdGlvLWluZ3Jlc3NnYXRld2F5LXNlcnZpY2UtYWNjb3VudA=='
, dynamicMetadata: filter_metadata {
key: "istio_authn"
value {
fields {
key: "request.auth.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
}
}
fields {
key: "source.principal"
value {
string_value: "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
}
}
}
}
[2018-07-26 20:39:18.060][152][debug][rbac] external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:88] shadow denied
[2018-07-26 20:39:18.060][152][debug][rbac] external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:98] enforced allowed
...
{{< /text >}}

View File

@ -11,8 +11,8 @@ You can then verify that Citadel is actually generating keys and certificates:
{{< text bash >}}
$ kubectl get secret istio.my-sa -n my-ns
NAME TYPE DATA AGE
istio.my-sa istio.io/key-and-cert 3 24d
NAME TYPE DATA AGE
istio.my-sa istio.io/key-and-cert 3 24d
{{< /text >}}
Where `my-ns` and `my-sa` are the namespace and service account your pod is running as.
@ -109,6 +109,71 @@ Finally, you can verify that the key and certificate are correctly mounted by yo
can use this command to check:
{{< text bash >}}
$ kubectl exec -it my-pod-id -c istio-proxy ls /etc/certs
$ kubectl exec -it my-pod-id -c istio-proxy -- ls /etc/certs
cert-chain.pem key.pem root-cert.pem
{{< /text >}}
Optionally, you could use the following command to check its contents:
{{< text bash >}}
$ kubectl exec -it my-pod-id -c istio-proxy -- cat /etc/certs/cert-chain.pem | openssl x509 -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
7e:b4:44:fe:d0:46:ba:27:47:5a:50:c8:f0:8e:8b:da
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = k8s.cluster.local
Validity
Not Before: Jul 13 01:23:13 2018 GMT
Not After : Oct 11 01:23:13 2018 GMT
Subject: O =
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:bb:c9:cd:f4:b8:b5:e4:3b:f2:35:aa:4c:67:cc:
1b:a9:30:c4:b7:fd:0a:f5:ac:94:05:b5:82:96:b2:
c8:98:85:f9:fc:09:b3:28:34:5e:79:7e:a9:3c:58:
0a:14:43:c1:f4:d7:b8:76:ab:4e:1c:89:26:e8:55:
cd:13:6b:45:e9:f1:67:e1:9b:69:46:b4:7e:8c:aa:
fd:70:de:21:15:4f:f5:f3:0f:b7:d4:c6:b5:9d:56:
ef:8a:91:d7:16:fa:db:6e:4c:24:71:1c:9c:f3:d9:
4b:83:f1:dd:98:5b:63:5c:98:5e:2f:15:29:0f:78:
31:04:bc:1d:c8:78:c3:53:4f:26:b2:61:86:53:39:
0a:3b:72:3e:3d:0d:22:61:d6:16:72:5d:64:e3:78:
c8:23:9d:73:17:07:5a:6b:79:75:91:ce:71:4b:77:
c5:1f:60:f1:da:ca:aa:85:56:5c:13:90:23:02:20:
12:66:3f:8f:58:b8:aa:72:9d:36:f1:f3:b7:2b:2d:
3e:bb:7c:f9:b5:44:b9:57:cf:fc:2f:4b:3c:e6:ee:
51:ba:23:be:09:7b:e2:02:6a:6e:e7:83:06:cd:6c:
be:7a:90:f1:1f:2c:6d:12:9e:2f:0f:e4:8c:5f:31:
b1:a2:fa:0b:71:fa:e1:6a:4a:0f:52:16:b4:11:73:
65:d9
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
URI:spiffe://cluster.local/ns/default/sa/bookinfo-productpage
Signature Algorithm: sha256WithRSAEncryption
8f:be:af:a4:ee:f7:be:21:e9:c8:c9:e2:3b:d3:ac:41:18:5d:
f8:9a:85:0f:98:f3:35:af:b7:e1:2d:58:5a:e0:50:70:98:cc:
75:f6:2e:55:25:ed:66:e7:a4:b9:4a:aa:23:3b:a6:ee:86:63:
9f:d8:f9:97:73:07:10:25:59:cc:d9:01:09:12:f9:ab:9e:54:
24:8a:29:38:74:3a:98:40:87:67:e4:96:d0:e6:c7:2d:59:3d:
d3:ea:dd:6e:40:5f:63:bf:30:60:c1:85:16:83:66:66:0b:6a:
f5:ab:60:7e:f5:3b:44:c6:11:5b:a1:99:0c:bd:53:b3:a7:cc:
e2:4b:bd:10:eb:fb:f0:b0:e5:42:a4:b2:ab:0c:27:c8:c1:4c:
5b:b5:1b:93:25:9a:09:45:7c:28:31:13:a3:57:1c:63:86:5a:
55:ed:14:29:db:81:e3:34:47:14:ba:52:d6:3c:3d:3b:51:50:
89:a9:db:17:e4:c4:57:ec:f8:22:98:b7:e7:aa:8a:72:28:9a:
a7:27:75:60:85:20:17:1d:30:df:78:40:74:ea:bc:ce:7b:e5:
a5:57:32:da:6d:f2:64:fb:28:94:7d:28:37:6f:3c:97:0e:9c:
0c:33:42:f0:b6:f5:1c:0d:fb:70:65:aa:93:3e:ca:0e:58:ec:
8e:d5:d0:1e
{{< /text >}}

View File

@ -8,16 +8,17 @@ keywords: [security,citadel,ops]
If you suspect Citadel isn't working properly, verify the status of the `istio-citadel` pod:
{{< text bash >}}
$ kubectl get pod -n istio-system
$ kubectl get pod -l istio=citadel -n istio-system
NAME READY STATUS RESTARTS AGE
istio-citadel-ff5696f6f-ht4gq 1/1 Running 0 25d
istio-citadel-ff5696f6f-ht4gq 1/1 Running 0 25d
{{< /text >}}
If the `istio-citadel` pod does not exist, you should try to re-run `istio-demo.yaml`.
If the `istio-citadel` pod doesn't exist, try to re-deploy the pod.
If `istio-citadel` is present, but its status is not `Running`, try the commands below to get more debugging information:
If the `istio-citadel` pod is present but its status is not `Running`, run the commands below to get more
debugging information and check if there are any errors:
{{< text bash >}}
$ kubectl logs istio-citadel-ff5696f6f-ht4gq -n istio-system
$ kubectl describe po istio-citadel-ff5696f6f-ht4gq -n istio-system
$ kubectl logs -l istio=citadel -n istio-system
$ kubectl describe pod -l istio=citadel -n istio-system
{{< /text >}}