Istio Blog and News

Support for Istio 1.8 has ended

Wed, 12 May 2021 00:00:00 +0000

As previously announced, support for Istio 1.8 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.8, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

ISTIO-SECURITY-2021-006

Tue, 11 May 2021 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2021-31921
CVSS Impact Score	10 AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H
Affected Releases	All releases prior to 1.8.6 1.9.0 to 1.9.4

Issue

Istio contains a remotely exploitable vulnerability where an external client can access unexpected services in the cluster, bypassing authorization checks, when a gateway is configured with AUTO_PASSTHROUGH routing configuration.

Am I impacted?

This vulnerability impacts only usage of the AUTO_PASSTHROUGH Gateway type, which is typically only used in multi-network multi-cluster deployments.

The TLS mode of all Gateways in the cluster can be detected with the following command:

$ kubectl get gateways.networking.istio.io -A -o "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,TLS_MODE:.spec.servers[*].tls.mode"

If the output shows any AUTO_PASSTHROUGH Gateways, you may be impacted.

Mitigation

Update your cluster to the latest supported version:

Istio 1.8.6, if using 1.8.x
Istio 1.9.5 or up
The patch version specified by your cloud provider

Credit

We would like to thank John Howard (Google) for reporting this issue.

ISTIO-SECURITY-2021-005

Tue, 11 May 2021 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2021-31920
CVSS Impact Score	8.1 AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N
Affected Releases	All releases prior to 1.8.6 1.9.0 to 1.9.4

Issue

Istio contains a remotely exploitable vulnerability where an HTTP request path with multiple slashes or escaped slash characters (%2F or %5C) could potentially bypass an Istio authorization policy when path based authorization rules are used. Related Envoy CVE: CVE-2021-29492.

For example, assume an Istio cluster administrator defines an authorization DENY policy to reject the request at path /admin. A request sent to the URL path //admin will NOT be rejected by the authorization policy.

According to the RFC 3986, the path //admin with multiple slashes should technically be treated as a different path from the /admin. However, some backend services choose to normalize the URL paths by merging multiple slashes to a single slash. This can result in a bypass of the authorization policy (//admin does not match /admin) and a user can access the resource at path /admin in the backend; this would represent a security incident.

Am I impacted?

Your cluster is impacted by this vulnerability if you have authorization policies using ALLOW action + notPaths field or DENY action + paths field patterns. These patterns are vulnerable to unexpected policy bypasses and you should upgrade to fix the security issue as soon as possible.

The following is an example of vulnerable policy that uses DENY action + paths field pattern:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: deny-path-admin
spec:
action: DENY
rules:
- to:
- operation:
paths: ["/admin"]

The following is another example of vulnerable policy that uses ALLOW action + notPaths field pattern:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: allow-path-not-admin
spec:
action: ALLOW
rules:
- to:
- operation:
notPaths: ["/admin"]

Your cluster is NOT impacted by this vulnerability if:

You don’t have authorization policies
Your authorization policies don’t define paths or notPaths fields.
Your authorization policies use ALLOW action + paths field or DENY action + notPaths field patterns. These patterns could only cause unexpected rejection instead of policy bypasses. The upgrade is optional for these cases.

Mitigation

Update your cluster to the latest supported version. These versions support configuring the Envoy proxies in the system with more normalization options:
- Istio 1.8.6, if using 1.8.x
- Istio 1.9.5 or up
- The patch version specified by your cloud provider
Follow the security best practices to configure your authorization policies.

Credit

We would like to thank Ruilin and Test123 for discovering this issue.

Announcing Istio 1.9.5

Tue, 11 May 2021 00:00:00 +0000

This release fixes the security vulnerabilities described in our May 11th posts, ISTIO-SECURITY-2021-005 and ISTIO-SECURITY-2021-006.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2021-31920: Istio contains a remotely exploitable vulnerability where an HTTP request path with multiple slashes or escaped slash characters (%2F or %5C) could potentially bypass an Istio authorization policy when path based authorization rules are used. See the ISTIO-SECURITY-2021-005 bulletin for more details.
- CVSS Score: 8.1 AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N
CVE-2021-29492: Envoy contains a remotely exploitable vulnerability where an HTTP request with escaped slash characters can bypass Envoy’s authorization mechanisms.
- CVSS Score: 8.3 AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:L
CVE-2021-31921: Istio contains a remotely exploitable vulnerability where an external client can access unexpected services in the cluster, bypassing authorization checks, when a gateway is configured with AUTO_PASSTHROUGH routing configuration. See the ISTIO-SECURITY-2021-006 bulletin for more details.
- CVSS Score: 10.0 AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H

Changes

Added security best practice for authorization policies

Announcing Istio 1.8.6

Tue, 11 May 2021 00:00:00 +0000

This release fixes the security vulnerabilities described in our May 11th posts, ISTIO-SECURITY-2021-005 and ISTIO-SECURITY-2021-006.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2021-31920: Istio contains a remotely exploitable vulnerability where an HTTP request path with multiple slashes or escaped slash characters (%2F or %5C) could potentially bypass an Istio authorization policy when path based authorization rules are used. See the ISTIO-SECURITY-2021-005 bulletin for more details.
- CVSS Score: 8.1 AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:N
CVE-2021-29492: Envoy contains a remotely exploitable vulnerability where an HTTP request with escaped slash characters can bypass Envoy’s authorization mechanisms.
- CVSS Score: 8.3 AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:L
CVE-2021-31921: Istio contains a remotely exploitable vulnerability where an external client can access unexpected services in the cluster, bypassing authorization checks, when a gateway is configured with AUTO_PASSTHROUGH routing configuration. See the ISTIO-SECURITY-2021-006 bulletin for more details.
- CVSS Score: 10.0 AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H

Changes

Added security best practice for authorization policies
Fixed istiod so it will no longer generate listeners for privileged gateway ports (<1024) if the gateway Pod does not have sufficient permissions. Issue 27566
Fixed an issue where transport socket parameters are now taken into account when configured in EnvoyFilter. Issue 28996
Fixed PeerAuthentication to not turn off mTLS while using multi-network, non-mTLS endpoints from the cross-network load-balancing endpoints to prevent 500 errors. Issue 28798
Fixed a bug causing runaway logs in istiod after disabling the default ingress controller. Issue 31336
Fixed the Kubernetes API server so it is now considered to be cluster-local by default . This means that any pod attempting to reach kubernetes.default.svc will always be directed to the in-cluster server. Issue 31340
Fixed Istio operator to prune resources that do not belong to the specific Istio operator CR. Issue 30833

Announcing Istio 1.9.4

Tue, 27 Apr 2021 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.9.3 and Istio 1.9.4

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue where the Istio operator prunes all resources created by the operator, including itself. Now the operator will only remove resources belonging to the custom resource. (Issue #30833)
Fixed an issue ensuring lease duration is always greater than the user configured RENEW_DEADLINE for Istio operator manager. (Issue #27509)
Fixed an issue where a certificate provisioned by sidecar proxy cannot be used by Prometheus. (Issue #29919)
Fixed an issue that creates an IOP under istio-system when installing Istio in another namespace. (Issue #31517)
Fixed an issue when using PeerAuthentication to turn off mTLS while using multi-network. Now non-mTLS endpoints will be removed from cross-network load-balancing endpoints to prevent 500 errors. (Issue #28798)
Fixed istiod never becoming ready when it fails to read resources from clusters configured via remote secrets. After a timeout configured by PILOT_REMOTE_CLUSTER_TIMEOUT (default 30s), istiod will become ready without syncing remote clusters. The stat remote_cluster_sync_timeouts will be incremented when this occurs. (Issue #30838)
Fixed an issue where istiod will not create a self-signed root CA and istio-ca-root-cert configmap when values.global.pilotCertProvider is kubernetes. (Issue #32023)
Improved the istioctl x workload command to configure VMs to disable inbound iptables capture for admin ports, matching the behavior of Kubernetes Pods. (Issue #29412)
Improved performance of istiod when running on clusters with thousands of namespaces. (Issue #32269
Improved detection of Server Side Apply in Kubernetes. (Issue #32101)

ISTIO-SECURITY-2021-004

Thu, 15 Apr 2021 00:00:00 +0000

Disclosure Details
CVE(s)	N/A
CVSS Impact Score	N/A
Affected Releases	All releases 1.5 and later

This is a security advisory for customers to check the authorization policy to make sure mTLS (STRICT mode) is enabled when using mTLS-only fields in the authorization policy.

You can stop reading if:

Your authorization policy does not use mTLS-only fields; or
Your authorization policy uses mTLS-only fields and you have also enabled mTLS with STRICT mode or your authorization policy is configured to reject plain text traffic explicitly.

Issue

In authorization policy, the following are mTLS-only fields:

the principals and notPrincipals field under the source section
the namespaces and notNamespaces field under the source section
the source.principal custom condition
the source.namespace custom condition

These mTLS-only fields will never match when the traffic is plain text (non mTLS) and the request might be allowed unexpectedly.

The following is an example ALLOW policy that uses mTLS-only fields to allow requests if it is not from the namespace foo:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: allow-ns-not-foo
spec:
action: ALLOW
rules:
- from:
- source:
notNamespaces: ["foo"]

A plain text request from the namespace foo will actually be allowed. The mTLS-only field notNamespaces will be compared to an empty value when mTLS is not used, resulting in a policy that allows the plain text request even if the source namespace is foo.

The following is an example DENY policy that uses mTLS-only fields to reject a request if it is from the namespace foo:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: reject-ns-foo
spec:
action: DENY
rules:
- from:
- source:
namespaces: ["foo"]

A plain text request from the namespace foo will not be rejected. The mTLS-only field namespaces will be compared to an empty value when mTLS is not used, resulting in a policy that does not reject the plain text request even if the source namespace is foo.

Solution

To solve this problem, it’s recommended to always enable mTLS with STRICT mode on the workloads before using any mTLS-only fields in the authorization policy on the same workload.

If you are unable to enable mTLS with STRICT mode for the workload, the alternative solution is to update the authorization policy to explicitly allow traffic with non-empty namespaces or reject traffic with empty namespaces (* implies non-empty and not * implies empty). As namespace can only be extracted when mTLS is STRICT. The policies below effectively also reject any plain text traffic.

If you are unable to enable mTLS with STRICT mode for the workload, the alternative solution is to update the authorization policy to explicitly allow traffic with non-empty namespaces or reject traffic with empty namespaces, as namespace can only be extracted when mTLS is STRICT.

* implies non-empty namespaces and not * implies empty namespaces. The policies below also reject any plain text traffic.

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: allow-ns-not-foo
spec:
action: ALLOW
rules:
- from:
- source:
notNamespaces: ["foo"]
# Add the following to explicitly only allow mTLS traffic.
namespaces: ["*"]

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: reject-ns-foo
spec:
action: DENY
rules:
- from:
- source:
namespaces: ["foo"]
# Add the following rule to explicitly reject plain text traffic.
- from:
- source:
notNamespaces: ["*"]

Also check the security policy examples for more details about the above alternative solution.

Credit

We’d like to thank John Howard for reporting this issue.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

ISTIO-SECURITY-2021-003

Thu, 15 Apr 2021 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2021-28683 CVE-2021-28682 CVE-2021-29258
CVSS Impact Score	7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	All releases prior to 1.8.5 1.9.0 to 1.9.2

Envoy, and subsequently Istio, is vulnerable to several newly discovered vulnerabilities:

CVE-2021-28683: Envoy contains a remotely exploitable NULL pointer dereference and crash in TLS when an unknown TLS alert code is received.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2021-28682: Envoy contains a remotely exploitable integer overflow in which a very large grpc-timeout value leads to unexpected timeout calculations.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2021-29258: Envoy contains a remotely exploitable vulnerability where an HTTP2 request with an empty metadata map can cause Envoy to crash.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.9.3

Thu, 15 Apr 2021 00:00:00 +0000

This release fixes the security vulnerability described in our April 15th post.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2021-28683: Envoy contains a remotely exploitable NULL pointer dereference and crash in TLS when an unknown TLS alert code is received.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2021-28682: Envoy contains a remotely exploitable integer overflow in which a very large grpc-timeout value leads to unexpected timeout calculations.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2021-29258: Envoy contains a remotely exploitable vulnerability where an HTTP2 request with an empty metadata map can cause Envoy to crash.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Announcing Istio 1.8.5

Thu, 15 Apr 2021 00:00:00 +0000

This release fixes the security vulnerability described in our April 15th post.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2021-28683: Envoy contains a remotely exploitable NULL pointer dereference and crash in TLS when an unknown TLS alert code is received.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2021-28682: Envoy contains a remotely exploitable integer overflow in which a very large grpc-timeout value leads to unexpected timeout calculations.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2021-29258: Envoy contains a remotely exploitable vulnerability where an HTTP2 request with an empty metadata map can cause Envoy to crash.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Support for Istio 1.8 ends on May 12th, 2021

Mon, 12 Apr 2021 00:00:00 +0000

According to Istio’s support policy, minor releases like 1.8 are supported for three months after the next minor release. Since 1.9 was released on February 9th, support for 1.8 will end on May 12th, 2021.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.8, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

ISTIO-SECURITY-2021-002

Wed, 07 Apr 2021 00:00:00 +0000

Disclosure Details
CVE(s)	N/A
CVSS Impact Score	N/A
Affected Releases	All releases 1.6 and later

Upgrading from Istio versions 1.5 and prior, to 1.6 and later, may result in access control bypass:

Incorrect gateway ports on authorization policies on upgrades: In Istio versions 1.6 and later, the default container ports for Istio ingress gateways are updated from port “80” to “8080” and “443” to “8443” to allow gateways to run as non-root by default. With this change, any existing authorization policies targeting an Istio ingress gateway on ports 80 and 443 need to be migrated to use the new container ports 8080 and 8443, before upgrading to the listed versions. Failure to migrate may result in traffic reaching ingress gateway service ports 80 and 443 to be incorrectly allowed or blocked, thereby causing policy violations.

Example of an authorization policy resource that needs to be updated:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: block-admin-access
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: DENY
rules:
- to:
- operation:
paths: ["/admin"]
ports: [ "80" ]
- to:
- operation:
paths: ["/admin"]
ports: [ "443" ]

The above policy in Istio versions 1.5 and prior will block all access to path /admin for traffic reaching an Istio ingress gateway on container ports 80 and 443. On upgrading to Istio version 1.6 and later, this policy should be updated to the following to have the same effect:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: block-admin-access
namespace: istio-system
spec:
selector:
matchLabels:
istio: ingressgateway
action: DENY
rules:
- to:
- operation:
paths: ["/admin"]
ports: [ "8080" ]
- to:
- operation:
paths: ["/admin"]
ports: [ "8443"

Mitigation

Update your authorization policies before upgrading to the affected Istio versions. You can use this script to check if any of the existing authorization policies attached to the default Istio ingress gateway in the istio-system namespace need to be updated. If you’re using a custom gateway installation, you can customize the script to run with parameters applicable to your environment.

It is recommended to create a copy of your existing authorization policies, update the copied version to use new gateway workload ports, and apply both existing and updated policies in your cluster, before initiating the upgrade process. You should only delete the old policies after a successful upgrade, to ensure no policy violations occur on upgrade failures or rollbacks.

Credit

We’d like to thank Neeraj Poddar for reporting this issue.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.9.2

Thu, 25 Mar 2021 00:00:00 +0000

This release note describes what’s different between Istio 1.9.1 and Istio 1.9.2.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue so transport socket parameters are now taken into account when configured in EnvoyFilter (Issue #28996)
Fixed a bug causing runaway logs in istiod after disabling the default ingress controller. (Issue #31336)
Fixed an issue so the Kubernetes API server is now considered to be cluster-local by default. This means that any pod attempting to reach kubernetes.default.svc will always be directed to the in-cluster server. (Issue #31340)
Fixed an issue with metadata handling for the Azure platform, allowing tagsList serialization of tags on instance metadata. (Issue #31176)
Fixed an issue with DNS proxying causing StatefulSets addresses to not be load balanced. (Issue #31064)

Announcing Istio 1.8.4

Wed, 10 Mar 2021 00:00:00 +0000

This release contains bug fixes to improve stability. This release note describes what’s different between Istio 1.8.3 and Istio 1.8.4

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed issue with metadata handling for Azure platform. Support added for tagsList serialization of tags on instance metadata. (Issue #31176)
Fixed an issue causing an alternative Envoy binary to be included in the docker image. The binaries are functionally equivalent. (Issue #31038)
Fixed an issue causing HTTP headers to be duplicated when using Istio probe rewrite. (Issue #28466)

ISTIO-SECURITY-2021-001

Mon, 01 Mar 2021 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2021-21378
CVSS Impact Score	8.2 AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:L/A:N
Affected Releases	1.9.0

Envoy, and subsequently Istio, is vulnerable to a newly discovered vulnerability:

CVE-2021-21378: JWT authentication bypass with unknown issuer token
- CVSS Score: 8.2 AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:L/A:N

You are subject to the vulnerability if you are using RequestAuthentication alone for JWT validation.

You are not subject to the vulnerability if you use both RequestAuthentication and AuthorizationPolicy for JWT validation.

For Istio, this vulnerability only exists if your service: * Accepts JWT tokens (with RequestAuthentication) * Has some service paths without AuthorizationPolicy applied.

For the service paths that both conditions are met, an incoming request with a JWT token, and the token issuer is not in RequestAuthentication will bypass the JWT validation, instead of getting rejected.

Mitigation

For proper JWT validation, you should always use the AuthorizationPolicy as documented on istio.io for specifying a valid token. To do this you will have to audit all of your RequestAuthentication and subsequent AuthorizationPolicy resources to make sure they align with the documented practice.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.9.1

Mon, 01 Mar 2021 00:00:00 +0000

This release fixes the security vulnerability described in our March 1st, 2021 news post as well as bug fixes to improve robustness.

This release note describes what’s different between Istio 1.9.0 and Istio 1.9.1.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

A zero-day security vulnerability was fixed in the version of Envoy shipped with Istio 1.9.0. This vulnerability was fixed on February 26th, 2021. 1.9.0 is the only version of Istio that includes the vulnerable version of Envoy. This vulnerability can only be exploited on misconfigured systems.

Changes

Improved sidecar injection to automatically specify the kubectl.kubernetes.io/default-logs-container. This ensures kubectl logs defaults to reading the application container’s logs, rather than requiring explicitly setting the container. (Issue #26764)
Improved the sidecar injector to better utilize pod labels to determine if injection is required. This is not enabled by default in this release, but can be tested using --set values.sidecarInjectorWebhook.useLegacySelectors=false. (Issue #30013)
Updated Prometheus metrics to include source_cluster and destination_cluster labels by default for all scenarios. Previously, this was only enabled for multi-cluster scenarios. (Issue #30036)
Updated default access log to include RESPONSE_CODE_DETAILS and CONNECTION_TERMINATION_DETAILS for proxy version >= 1.9. (Issue #27903)
Updated Kiali addon to the latest version v1.29. (Issue #30438)
Added enableIstioConfigCRDs to base to allow users to specify whether the Istio CRDs will be installed. (Issue #28346)
Added support for DestinationRule inheritance for mesh/namespace level rules. Enable feature with the PILOT_ENABLE_DESTINATION_RULE_INHERITANCE environment variable. (Issue #29525)
Added support for applications that bind to their pod IP address, rather than wildcard or localhost address, through the Sidecar API. (Issue #28178)
Added flag to enable capture of DNS traffic to the istio-iptables script. (Issue #29908)
Added canonical service tags to Envoy-generated trace spans. (Issue #28801)
Fixed an issue causing the timeout header x-envoy-upstream-rq-timeout-ms to not be honored. (Issue #30885)
Fixed an issue where access log service causes Istio proxy to reject configuration. (Issue #30939)
Fixed an issue causing an alternative Envoy binary to be included in the Docker image. The binaries are functionally equivalent. (Issue #31038)
Fixed an issue where the TLS v2 version was enforced only on HTTP ports. This option is now applied to all ports.
Fixed an issue where Wasm plugin configuration update will cause requests to fail. (Issue #29843)
Removed support for reading Istio configuration over the Mesh Configuration Protocol (MCP). (Issue #28634)

Support for Istio 1.7 has ended

Thu, 25 Feb 2021 00:00:00 +0000

As previously announced, support for Istio 1.7 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.7, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

Announcing Istio 1.7.8

Thu, 25 Feb 2021 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.7.7 and Istio 1.7.8

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue where dashboard controlz would not port forward to istiod pod. (Issue #30208)
Fixed an issue where namespace isn’t resolved correctly in VirtualService delegation’s short destination host. (Issue #30387)
Fixed an issue causing HTTP headers to be duplicated when using Istio probe rewrite. (Issue #28466)

Announcing Istio 1.8.3

Mon, 08 Feb 2021 00:00:00 +0000

This release contains bug fixes to improve stability. This release note describes what’s different between Istio 1.8.2 and Istio 1.8.3

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security

Istio 1.8.3 will not contain a security fix as previously announced on discuss.istio.io. There is no currently planned date at this time. Be assured that this is a top priority for the Istio Product Security Working Group, but due to the details we cannot release more information at this time. An announcement regarding the delay can be found here.

Changes

Fixed an issue with aggregate cluster during TLS init in Envoy (Issue #28620)
Fixed an issue causing Istio 1.8 to configure Istio 1.7 proxies incorrectly when using the Sidecar ingress configuration. (Issue #30437)
Fixed a bug where DNS agent preview produces malformed DNS responses. (Issue #28970)
Fixed a bug where the env K8S settings are overridden by the env settings in the helm values. (Issue #30079)
Fixed a bug where istioctl dashboard controlz could not port forward to istiod pod. (Issue #30208)
Fixed a bug that prevented Ingress resources created with IngressClass from having their status field updated (Issue #25308)
Fixed an issue where the TLSv2 version was enforced only on HTTP ports. This option is now applied to all ports. (PR #30590)
Fixed issues resulting in missing routes when using httpsRedirect in a Gateway. (Issue #27315),(Issue #27157)

Announcing Istio 1.7.7

Fri, 29 Jan 2021 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.7.6 and Istio 1.7.7

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue of using explicitly empty revision flag on install. (Issue #26940)
Fixed the CA’s certificate signature algorithm to be the default algorithm corresponding to the CA’s signing key type. (Issue #27238)
Fixed an issue showing unnecessary warnings when downgrading to a lower version of Istio. (Issue #29183)
Fixed an issue causing older control planes relying on the rbac.istio.io CRD group to hang on restart due to the fact that newer control plane installations remove those permissions from istiod. (Issue #29364)
Fixed a memory leak in WASM NullPlugin onNetworkNewConnection. (Issue #24720)

Support for Istio 1.7 ends on February 19th, 2021

Tue, 19 Jan 2021 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.7 are supported for three months after the next LTS release. Since 1.8 was released on November 19th, support for 1.7 will end on February 19th, 2021.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.7, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

Fixed an issue showing unnecessary warnings when downgrading to a lower version of Istio. (Issue #29183)
Fixed an issue where a delegate VirtualService change would not trigger an xDS push. (Issue #29123)
Fixed a regression in Istio 1.8.0 causing workloads with multiple Services with overlapping ports to send traffic to the wrong port. (Issue #29199)
Fixed a bug causing Istio to attempt to validate resource types it no longer supports.

Support for Istio 1.6 has ended

Mon, 23 Nov 2020 00:00:00 +0000

As previously announced, support for Istio 1.6 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.6, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

Announcing Istio 1.6.14

Mon, 23 Nov 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.13 and Istio 1.6.14

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed HPA settings for telemetry being overridden by the inline replicas. (Issue #28916)
Fixed an issue that caused very high memory usage with a large number of ServiceEntries. (Issue #25531)
Fixed an issue that caused the user agent header to be missing in the Stackdriver access log. (PR #3083)

ISTIO-SECURITY-2020-011

Sat, 21 Nov 2020 00:00:00 +0000

Disclosure Details
CVE(s)	N/A
CVSS Impact Score	N/A
Affected Releases	1.8.0

Envoy, and subsequently Istio, is vulnerable to a newly discovered vulnerability:

Incorrect proxy protocol downstream address for non-HTTP connections: Envoy incorrectly restores the proxy protocol downstream address for non-HTTP connections. Instead of restoring the address supplied by the proxy protocol filter, Envoy restores the address of the directly connected peer and passes it to subsequent filters. This will affect logging (%DOWNSTREAM_REMOTE_ADDRESS%) and authorization policy (remoteIpBlocks and remote_ip) for non-HTTP network connections because they will use the incorrect proxy protocol downstream address.

This issue does not affect HTTP connections. The address from X-Forwarded-For is also not affected.

Istio does not support proxy protocol, and the only way to enable it is to use a custom EnvoyFilter resource. It is not tested in Istio and should be used at your own risk.

Mitigation

For Istio 1.8.0 deployments: do not use the proxy protocol for non-HTTP connections.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.7.5

Thu, 19 Nov 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.7.4 and Istio 1.7.5

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed pilot agent app probe connection leak. (Issue #27726)
Fixed how install-cni applies istio-cni plugin configuration. Previously, new configurations would be appended to the list. This has been changed to remove existing istio-cni plugins from the CNI config before inserting new plugins. (Issue #27771)
Fixed when a node has multiple IP addresses (e.g., a VM in the mesh expansion scenario). Istio Proxy will now bind inbound listeners to the first applicable address in the list rather than to the last one. (Issue #28269)
Fixed Istio to not run gateway secret fetcher when proxy is configured with FILE_MOUNTED_CERTS.
Fixed multicluster EnvoyFilter to have valid configuration following the underlying changes in Envoy’s API. (Issue #27909)
Fixed an issue causing a short spike in errors during in place upgrades from Istio 1.6 to 1.7. Previously, the xDS version would be upgraded automatically from xDS v2 to xDS v3. This caused downtime with upgrades from Istio 1.6 to Istio 1.7. This has been fixed so that these upgrades no longer cause downtime. Note that, as a trade off, upgrading from Istio 1.7.x to Istio 1.7.5 still causes downtime in any existing 1.6 proxies; if you are in this scenario you may set the PILOT_ENABLE_TLS_XDS_DYNAMIC_TYPES environment variable to false in Istiod to retain the previous behavior. (Issue #28120)
Fixed missing listeners on a VM when the VM sidecar is connected to istiod but a WorkloadEntry is registered later. (Issue #28743)

Upgrade Notice

When upgrading your Istio data plane from 1.7.x (where x < 5) to 1.7.5 or newer, you may observe connectivity issues between your gateway and your sidecars or among your sidecars with 503 errors in the log. This happens when 1.7.5+ proxies send HTTP 1xx or 204 response codes with headers that 1.7.x proxies reject. To fix this, upgrade all your proxies (gateways and sidecars) to 1.7.5+ as soon as possible. (Issue 29427, More information)

Announcing Istio 1.7.4

Tue, 27 Oct 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.7.3 and Istio 1.7.4

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Improved TLS configuration on sidecar server-side inbound paths to enforce TLS 2.0 version along with recommended cipher suites. This is disabled by default and can enabled by setting the environment variable PILOT_SIDECAR_ENABLE_INBOUND_TLS_V2 to true.
Added ability to configure domain suffix for multicluster installation. (Issue #27300)
Added istioctl proxy-status and other commands will attempt to contact the control plane using both port-forwarding and exec before giving up, restoring functionality on clusters that do not offer port-forwarding to the control plane. (Issue #27421)
Added support for securityContext in the Kubernetes settings for the operator API. (Issue #26275)
Added support for revision based istiod to istioctl version. (Issue #27756)
Fixed deleting the remote-secret for multicluster installation removes remote endpoints.
Fixed an issue that Istiod’s cacert.pem is under the testdata directory. (Issue #27574)
Fixed PodDisruptionBudget of istio-egressgateway does not match any pods. (Issue #27730)
Fixed an issue preventing calls to wildcard (such as *.example.com) domains when a port is set in the Host header.
Fixed an issue periodically causing a deadlock in Pilot’s syncz debug endpoint.
Removed deprecated outboundTrafficPolicy from global values. (Issue #27494)

Announcing Istio 1.6.13

Tue, 27 Oct 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.12 and Istio 1.6.13

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue where Istiod’s cacert.pem was under the testdata directory (Issue #27574)
Fixed Pilot agent app probe connection leak. (Issue #27726)

Support for Istio 1.6 ends on November 21st, 2020

Tue, 20 Oct 2020 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.6 are supported for three months after the next LTS release. Since 1.7 was released on August 21st, support for 1.6 will end on November 21st, 2020.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.6, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

Announcing Istio 1.6.12

Tue, 06 Oct 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.11 and Istio 1.6.12

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Added ability to configure domain suffix for multicluster installation (Issue #27300)
Added support for securityContext in the Kubernetes settings for the operator API. (Issue #26275)
Fixed an issue preventing calls to wildcard (such as *.example.com) domains when a port is set in the Host header. (Issue #25350)

ISTIO-SECURITY-2020-010

Tue, 29 Sep 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-25017
CVSS Impact Score	8.3 AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:L
Affected Releases	1.6 to 1.6.10 1.7 to 1.7.2

Envoy, and subsequently Istio, is vulnerable to a newly discovered vulnerability:

CVE-2020-25017: In some cases, Envoy only considers the first value when multiple headers are present. Also, Envoy does not replace all existing occurrences of a non-inline header.
- CVSS Score: 8.3 AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:L

Mitigation

For Istio 1.6.x deployments: update to Istio 1.6.11 or later.
For Istio 1.7.x deployments: update to Istio 1.7.3 or later.

Changes

Added istioctl analyzer to detect when Destination Rules do not specify caCertificates (Istio #25652)
Added missing telemetry.loadshedding.* options to mixer container arguments
Fixed HTTP match request without headers conflict
Fixed Istio operator to watch multiple namespaces (Istio #26317)
Fixed EDS cache when an endpoint appears after its service resource (Istio #26983)
Fixed istioctl remove-from-mesh not removing init containers on CNI installations.
Fixed istioctl add-to-mesh and remove-from-mesh commands from affecting OwnerReferences (Istio #26720)
Fixed cleaning up of service information when the cluster secret is deleted
Fixed egress gateway ports binding to ⁸⁰⁄₄₄₃ due to user permissions
Fixed gateway listeners created with traffic direction outbound to be drained properly on exit
Fixed headless services not updating listeners (Istio #26617)
Fixed inaccurate endpointsPendingPodUpdate metric
Fixed ingress SDS from not getting secret update (Istio #18912)
Fixed ledger capacity size
Fixed operator to update service monitor due to invalid permissions (Istio #26961)
Fixed regression in gateway name resolution (Istio 26264)
Fixed rotated certificates not being stored to /etc/istio-certs VolumeMount (Istio #26821)
Fixed trust domain validation in transport socket level (Istio #26435)
Improved specifying network for a cluster without meshNetworks also being configured
Improved the cache readiness state with TTL (Istio #26418)
Updated SDS timeout to fetch workload certificates to 0s
Updated app_containers to use comma separated values for container specification
Updated default protocol sniffing timeout to 5s (Istio #24379)

Support for Istio 1.5 has ended

Mon, 24 Aug 2020 00:00:00 +0000

As previously announced, support for Istio 1.5 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.5, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

Announcing Istio 1.5.10

Mon, 24 Aug 2020 00:00:00 +0000

This release includes bug fixes to improve robustness. These release notes describe what’s different between Istio 1.5.9 and Istio 1.5.10.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed container name as app_container in telemetry v2.
Fixed ingress SDS not getting secret updates. (Issue 23715).

ISTIO-SECURITY-2020-009

Tue, 11 Aug 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-16844
CVSS Impact Score	6.8 AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:N
Affected Releases	1.5 to 1.5.8 1.6 to 1.6.7

Istio is vulnerable to a newly discovered vulnerability:

CVE-2020-16844: Callers to TCP services that have a defined Authorization Policies with DENY actions using wildcard suffixes (e.g. *-some-suffix) for source principals or namespace fields will never be denied access.
- CVSS Score: 6.8 AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:N

Istio users are exposed to this vulnerability in the following ways:

If the user has an Authorization similar to

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: foo
namespace: foo
spec:
action: DENY
rules:
- from:
- source:
principals:
- */ns/ns1/sa/foo # indicating any trust domain, ns1 namespace, foo svc account

Istio translates the principal (and source.principal) field to an Envoy level string match

stringMatch:
suffix: spiffe:///ns/ns1/sa/foo

which will not match any legitimate caller as it included the spiffe:// string incorrectly. The correct string match should be

stringMatch:
regex: spiffe://.*/ns/ns1/sa/foo

Prefix and exact matches in AuthorizationPolicy is unaffected, as are ALLOW actions in them; HTTP is also unaffected.

Mitigation

For Istio 1.5.x deployments: update to Istio 1.5.9 or later.
For Istio 1.6.x deployments: update to Istio 1.6.8 or later.
Do not use suffix matching in DENY policies in the source principal or namespace field for TCP services and use Prefix and Exact matching where applicable. Where possible change TCP to HTTP for port name suffixes in your Services.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.6.8

Tue, 11 Aug 2020 00:00:00 +0000

This release fixes the security vulnerability described in our August 11th, 2020 news post.

This release contains bug fixes to improve robustness. These release notes describe what’s different between Istio 1.6.7 and Istio 1.6.8.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2020-16844: Callers to TCP services that have a defined Authorization Policies with DENY actions using wildcard suffixes (e.g. *-some-suffix) for source principals or namespace fields will never be denied access.
- CVSS Score: 6.8 AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:N

Announcing Istio 1.5.9

Tue, 11 Aug 2020 00:00:00 +0000

This release fixes the security vulnerability described in our August 11th, 2020 news post.

These release notes describe what’s different between Istio 1.5.8 and Istio 1.5.9.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2020-16844: Callers to TCP services that have a defined Authorization Policies with DENY actions using wildcard suffixes (e.g. *-some-suffix) for source principals or namespace fields will never be denied access.
- CVSS Score: 6.8 AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:H/A:N

Announcing Istio 1.6.7

Thu, 30 Jul 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.6 and Istio 1.6.7.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue which prevented endpoints not associated with pods from working. (Issue #25974)

Announcing Istio 1.6.6

Wed, 29 Jul 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.5 and Istio 1.6.6.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Optimized performance in scenarios with large numbers of gateways. (Issue 25116)
Fixed an issue where out of order events may cause the Istiod update queue to get stuck. This resulted in proxies with stale configuration.
Fixed istioctl upgrade so that it no longer checks remote component versions when using --dry-run. (Issue 24865)
Fixed long log messages for clusters with many gateways.
Fixed outlier detection to only fire on user configured errors and not depend on success rate. (Issue 25220)
Fixed demo profile to use port 15021 as the status port. (Issue #25626)
Fixed Galley to properly handle errors from Kubernetes tombstones.
Fixed an issue where manually enabling TLS/mTLS for communication between a sidecar and an egress gateway did not work. (Issue 23910)
Fixed Bookinfo demo application to verify if a specified namespace exists and if not, use the default namespace.
Added a label to the pilot_xds metric in order to give more information on data plane versions without scraping the data plane.
Added CA_ADDR field to allow configuring the certificate authority address on the egress gateway configuration and fixed the istio-certs mount secret name.
Updated Bookinfo demo application to latest versions of libraries.
Updated Istio to disable auto mTLS when sending traffic to headless services without a sidecar.

Support for Istio 1.5 ends on August 21st, 2020

Wed, 22 Jul 2020 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.5 are supported for three months after the next LTS release. Since 1.6 was released on May 21st, support for 1.5 will end on August 21st, 2020.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.5, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

ISTIO-SECURITY-2020-008

Thu, 09 Jul 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-15104
CVSS Impact Score	6.6 AV:N/AC:H/PR:H/UI:N/S:C/C:H/I:L/A:N/E:F/RL:O/RC:C
Affected Releases	1.5 to 1.5.7 1.6 to 1.6.4 All releases prior to 1.5

Istio is vulnerable to a newly discovered vulnerability:

CVE-2020-15104: When validating TLS certificates, Envoy incorrectly allows a wildcard DNS Subject Alternative Name apply to multiple subdomains. For example, with a SAN of *.example.com, Envoy incorrectly allows nested.subdomain.example.com, when it should only allow subdomain.example.com.
- CVSS Score: 6.6 AV:N/AC:H/PR:H/UI:N/S:C/C:H/I:L/A:N/E:F/RL:O/RC:C

Istio users are exposed to this vulnerability in the following ways:

Direct use of Envoy’s verify_subject_alt_name and match_subject_alt_names configuration via Envoy Filter.
Use of Istio’s subjectAltNames field in destination rules with client TLS settings. A destination rule with a subjectAltNames field containing nested.subdomain.example.com incorrectly accepts a certificate from an upstream peer with a Subject Alternative Name (SAN) of *.example.com. Instead a SAN of *.subdomain.example.com or nested.subdomain.example.com should be present.
Use of Istio’s subjectAltNames in service entries. A service entry with a subjectAltNames field with a value similar to nested.subdomain.example.com incorrectly accepts a certificate from an upstream peer with a SAN of *.example.com.

The Istio CA, which was formerly known as Citadel, does not issue certificates with DNS wildcard SANs. The vulnerability only impacts configurations that validate externally issued certificates.

Mitigation

For Istio 1.5.x deployments: update to Istio 1.5.8 or later.
For Istio 1.6.x deployments: update to Istio 1.6.5 or later.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.6.5

Thu, 09 Jul 2020 00:00:00 +0000

This release fixes the security vulnerability described in our July 9th, 2020 news post.

This release contains bug fixes to improve robustness. These release notes describe what’s different between Istio 1.6.5 and Istio 1.6.4.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2020-15104: When validating TLS certificates, Envoy incorrectly allows a wildcard DNS Subject Alternative Name to apply to multiple subdomains. For example, with a SAN of *.example.com, Envoy incorrectly allows nested.subdomain.example.com, when it should only allow subdomain.example.com.
- CVSS Score: 6.6 AV:N/AC:H/PR:H/UI:N/S:C/C:H/I:L/A:N/E:F/RL:O/RC:C

Changes

Fixed return the proper source name after Mixer does a lookup by IP if multiple pods have the same IP.
Improved the sidecar injection control based on revision at a per-pod level (Issue 24801)
Improved istioctl validate to disallow unknown fields not included in the Open API specification (Issue 24860)
Changed stsPort to sts_port in Envoy’s bootstrap file.
Preserved existing WASM state schema for state objects to reference it later as needed.
Added targetUri to stackdriver_grpc_service.
Updated WASM state to log for Access Log Service.
Increased default protocol detection timeout from 100 ms to 5 s (Issue 24379)
Removed UDP port 53 from Istiod.
Allowed setting status.sidecar.istio.io/port to zero (Issue 24722)
Fixed EDS endpoint selection for subsets with no or empty label selector. (Issue 24969)
Allowed k8s.overlays on BaseComponentSpec. (Issue 24476)
Fixed istio-agent to create elliptical curve CSRs when ECC_SIGNATURE_ALGORITHM is set.
Improved mapping of gRPC status codes into HTTP domain for telemetry.
Fixed scaleTargetRef naming in HorizontalPodAutoscaler for Istiod (Issue 24809)

Announcing Istio 1.5.8

Thu, 09 Jul 2020 00:00:00 +0000

This release fixes the security vulnerability described in our July 9th, 2020 news post.

These release notes describe what’s different between Istio 1.5.8 and Istio 1.5.7.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2020-15104: When validating TLS certificates, Envoy incorrectly allows wildcards in DNS Subject Alternative Name (SAN) to apply to multiple subdomains. For example, with a SAN of *.example.com, Envoy incorrectly allows nested.subdomain.example.com, when it should only allow subdomain.example.com.
- CVSS Score: 6.6 AV:N/AC:H/PR:H/UI:N/S:C/C:H/I:L/A:N/E:F/RL:O/RC:C

Changes

Allowed setting status.sidecar.istio.io/port to zero (Issue 24722)
Improved istioctl validate to disallow unknown fields not included in the Open API specification (Issue 24860)
Fixed a bug in Mixer where it would incorrectly return source names when it did lookup by IP.

ISTIO-SECURITY-2020-007

Tue, 30 Jun 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-12603 CVE-2020-12605 CVE-2020-8663 CVE-2020-12604
CVSS Impact Score	7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	1.5 to 1.5.6 1.6 to 1.6.3

Envoy, and subsequently Istio, are vulnerable to four newly discovered vulnerabilities:

CVE-2020-12603: By sending a specially crafted packet, an attacker could cause Envoy to consume excessive amounts of memory when proxying HTTP/2 requests or responses.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-12605: An attacker could cause Envoy to consume excessive amounts of memory when processing specially crafted HTTP/1.1 packets.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-8663: An attacker could cause Envoy to exhaust file descriptors when accepting too many connections.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-12604: An attacker could cause increased memory usage when processing specially crafted packets.
- CVSS Score: 5.3 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Mitigation

For Istio 1.5.x deployments: update to Istio 1.5.7 or later.
For Istio 1.6.x deployments: update to Istio 1.6.4 or later.

CVE-2020-8663 is addressed in Envoy by adding a configurable limit on downstream connections. The limit must be configured to mitigate this vulnerability. Perform the following steps to configure limits at the ingress gateway.

Create a config map by downloading custom-bootstrap-runtime.yaml. Update global_downstream_max_connections in the config map according to the number of concurrent connections needed by individual gateway instances in your deployment. Once the limit is reached, Envoy will start rejecting tcp connections.
```
$ kubectl -n istio-system apply -f custom-bootstrap-runtime.yaml
```
Patch the ingress gateway deployment to use the above configuration. Download gateway-patch.yaml and apply it using the following command.
```
$ kubectl --namespace istio-system patch deployment istio-ingressgateway --patch "$(cat gateway-patch.yaml)"
```

Confirm that the new limits are in place.

$ ISTIO_INGRESS_PODNAME=$(kubectl get pods -l app=istio-ingressgateway -n istio-system -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace istio-system exec -i -t ${ISTIO_INGRESS_PODNAME} -c istio-proxy -- curl -sS http://localhost:15000/runtime
{
"entries": {
"overload.global_downstream_max_connections": {
"layer_values": [
"",
"250000",
""
],
"final_value": "250000"
}
},
"layers": [
"static_layer_0",
"admin"
]
}

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.6.4

Tue, 30 Jun 2020 00:00:00 +0000

This release fixes the security vulnerability described in our June 30th, 2020 news post.

This release note describes what’s different between Istio 1.6.4 and Istio 1.6.3.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2020-12603: By sending a specially crafted packet, an attacker could cause Envoy to consume excessive amounts of memory when proxying HTTP/2 requests or responses.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-12605: An attacker could cause Envoy to consume excessive amounts of memory when processing specially crafted HTTP/1.1 packets.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-8663: An attacker could cause Envoy to exhaust file descriptors when accepting too many connections.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-12604: An attacker could cause increased memory usage when processing specially crafted packets.
- CVSS Score: 5.3 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Announcing Istio 1.5.7

Tue, 30 Jun 2020 00:00:00 +0000

This release fixes the security vulnerability described in our June 30th, 2020 news post.

This release note describes what’s different between Istio 1.5.7 and Istio 1.5.6.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

CVE-2020-12603: By sending a specially crafted packet, an attacker could cause Envoy to consume excessive amounts of memory when proxying HTTP/2 requests or responses.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-12605: An attacker could cause Envoy to consume excessive amounts of memory when processing specially crafted HTTP/1.1 packets.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-8663: An attacker could cause Envoy to exhaust file descriptors when accepting too many connections.
- CVSS Score: 7.0 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
CVE-2020-12604: An attacker could cause increased memory usage when processing specially crafted packets.
- CVSS Score: 5.3 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:L

Announcing Istio 1.4.10

Mon, 22 Jun 2020 00:00:00 +0000

This is the final release for Istio 1.4.

This release fixes the security vulnerability described in our June 11th, 2020 news post as well as bug fixes to improve robustness.

This release note describes what’s different between Istio 1.4.9 and Istio 1.4.10.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-006 Excessive CPU usage when processing HTTP/2 SETTINGS frames with too many parameters, potentially leading to a denial of service.

CVE-2020-11080: By sending a specially crafted packet, an attacker could cause the CPU to spike at 100%. This could be sent to the ingress gateway or a sidecar.

Bug fixes

Fixed istio-cni-node crash when COS_CONTAINERD and Istio CNI are enabled when running on Google Kubernetes Engine (Issue 23643)
Fixed Istio CNI causes pod initialization to experience a 30-40 second delay on startup when DNS is unreachable (Issue 23770)

Bookinfo sample application security fixes

We’ve updated the versions of Node.js and jQuery used in the Bookinfo sample application. Node.js has been upgraded from version 12.9 to 12.18. jQuery has been updated from version 2.1.4 to version 3.5.0. The highest rated vulnerability fixed: HTTP request smuggling using malformed Transfer-Encoding header (Critical) (CVE-2019-15605)

Announcing Istio 1.6.3

Thu, 18 Jun 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.2 and Istio 1.6.3.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed an issue preventing the operator from recreating watched resources if they are deleted (Issue 23238).
Fixed an issue where Istio crashed with the message: proto.Message is *client.QuotaSpecBinding, not *client.QuotaSpecBinding(Issue 24624).
Fixed an issue preventing operator reconciliation due to improper labels on watched resources (Issue 23603).
Added support for the k8s.v1.cni.cncf.io/networks annotation (Issue 24425).
Updated the SidecarInjectionSpec CRD to read the imagePullSecret from .Values.global (Pull 24365).
Updated split horizon to skip gateways that resolve hostnames.
Fixed istioctl experimental metrics to only flag error response codes as errors (Issue 24322)
Updated istioctl analyze to sort output formats.
Updated gateways to use proxyMetadata
Updated the Prometheus sidecar to use proxyMetadata(Issue 24415).
Removed invalid configuration from PodSecurityContext when gateway.runAsRoot is enabled (Issue 24469).

Grafana addon security fixes

We’ve updated the version of Grafana shipped with Istio from 6.5.2 to 6.7.4. This addresses a Grafana security issue, rated high, that can allow access to internal cluster resources using the Grafana avatar feature. (CVE-2020-13379)

Announcing Istio 1.5.6

Wed, 17 Jun 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.5.5 and Istio 1.5.6.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security

Updated Node.js and jQuery versions used in bookinfo.

Changes

Fixed Transfer-Encoding value case-sensitivity in Envoy (Envoy’s issue 10041)
Fixed handling of user defined ingress gateway configuration (Issue 23303)
Fixed Add TCP MX ALPN in UpstreamTlsContext for clusters that specify http2_protocol_options (Issue 23907)
Fixed election lock for namespace configmap controller.
Fixed istioctl validate -f for networking.istio.io/v1beta1 rules (Issue 24064)
Fixed aggregate clusters configuration (Issue 23909)
Fixed Prometheus mTLS poods scraping (Issue 22391)
Fixed ingress crash for overlapping hosts without match (Issue 22910)
Fixed Istio telemetry Pod crashes (Issue 23813)
Removed hard-coded operator namespace (Issue 24073)

ISTIO-SECURITY-2020-006

Thu, 11 Jun 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-11080
CVSS Impact Score	7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	1.4 to 1.4.9 1.5 to 1.5.4 1.6 to 1.6.1

A vulnerability affecting the HTTP2 library used by Envoy has been fixed and publicly disclosed (c.f. Denial of service: Overly large SETTINGS frames ). Unfortunately Istio did not benefit from a responsible disclosure process.

CVE-2020-11080: By sending a specially crafted packet, an attacker could cause the CPU to spike at 100%. This could be sent to the ingress gateway or a sidecar.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Mitigation

HTTP2 support could be disabled on the Ingress Gateway as a temporary workaround using the following configuration for example (Note that HTTP2 support at ingress can be disabled if you are not exposing gRPC services through ingress):


apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: disable-ingress-h2
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: NETWORK_FILTER # http connection manager is a filter in Envoy
match:
context: GATEWAY
listener:
filterChain:
filter:
name: "envoy.http_connection_manager"
patch:
operation: MERGE
value:
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
codec_type: HTTP1

For Istio 1.4.x deployments: update to Istio 1.4.10 or later.
For Istio 1.5.x deployments: update to Istio 1.5.5 or later.
For Istio 1.6.x deployments: update to Istio 1.6.2 or later.

Credit

We’d like to thank Michael Barton for bringing this publicly disclosed vulnerability to our attention.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.6.2

Thu, 11 Jun 2020 00:00:00 +0000

This release fixes the security vulnerability described in our June 11th, 2020 news post.

This release note describes what’s different between Istio 1.6.2 and Istio 1.6.1.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-006 Excessive CPU usage when processing HTTP/2 SETTINGS frames with too many parameters, potentially leading to a denial of service.

CVE-2020-11080: By sending a specially crafted packet, an attacker could cause the CPU to spike at 100%. This could be sent to the ingress gateway or a sidecar.

Announcing Istio 1.5.5

Thu, 11 Jun 2020 00:00:00 +0000

This release fixes the security vulnerability described in our June 11th, 2020 news post.

This release note describes what’s different between Istio 1.5.5 and Istio 1.5.4.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-006 Excessive CPU usage when processing HTTP/2 SETTINGS frames with too many parameters, potentially leading to a denial of service.

CVE-2020-11080: By sending a specially crafted packet, an attacker could cause the CPU to spike at 100%. This could be sent to the ingress gateway or a sidecar.

Support for Istio 1.4 has ended

Fri, 05 Jun 2020 00:00:00 +0000

As previously announced, support for Istio 1.4 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.4, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

Announcing Istio 1.6.1

Thu, 04 Jun 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.6.0 and Istio 1.6.1.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed support for pod annotations to override mesh-wide proxy settings
Updated EnvoyFilter to register all filter types in order to support typed_config attributes (Issue 23909)
Fixed handling of custom resource names for Gateways (Issue 23303)
Fixed an issue where istiod fails to issue certificates to a remote cluster. Istiod now has support for the cluster name and certificate to generate the injectionURL (Issue 23879)
Fixed remote cluster’s validation controller to check istiod’s ready status endpoint (Issue 23945)
Improved regexp fields validation to match Envoy’s validation (Issue 23436)
Fixed istioctl analyze to validate networking.istio.io/v1beta1 resources (Issue 24064)
Fixed typo of istio in ControlZ dashboard log (Issue 24039)
Fixed tar name to directory translation (Issue 23635)
Improved certificate management for multi-cluster and virtual machine setup from samples/certs directory to install/tools/certs directory
Improved pilot-agent’s handling of client certificates when only a CA client certificate is present
Improved istiocl upgrade to direct users to the istio.io website to migrate from v1alpha1 security policies to v1beta1 security policies
Fixed release URL name for istioctl upgrade
Fixed k8s.overlays for cluster resources
Fixed HTTP/HTTP2 conflict at Gateway (Issue 24061 and Issue 19690)
Fixed Istio operator to respect the --operatorNamespace argument (Issue 24073)
Fixed Istio operator hanging when uninstalling Istio (Issue 24038)
Fixed TCP metadata exchange for upstream clusters that specify http2_protocol_options (Issue 23907)
Added sideEffects field to MutatingWebhookConfiguration for istio-sidecar-injector (Issue 23485)
Improved installation for replicated control planes (Issue 23871)
Fixed istioctl experimental precheck to report compatible versions of Kubernetes (1.14-1.18) (Issue 24132)
Fixed Istio operator namespace mismatches that caused a resource leak when pruning resources (Issue 24222)
Fixed SDS Agent failing to start when proxy uses file mounted certs for Gateways (Issue 23646)
Fixed TCP over HTTP conflicts that caused invalid configuration to be generated (Issue 24084)
Fixed the use of external name when remote Pilot address is a hostname (Issue 24155)
Fixed Istio CNI node DaemonSet starting when Istio CNI and cos_containerd are enabled on Google Kubernetes Engine (GKE) (Issue 23643)
Fixed Istio CNI causing pod initialization to experience a 30-40 second delay on startup when DNS unreachable (Issue 23770)
Improved Google Stackdriver telemetry use of UIDs with GCE VMs
Improved telemetry plugins to not crash due invalid configuration (Issue 23865)
Fixed a proxy sidecar segfault when the response to HTTP calls by WASM filters are empty (Issue 23890)
Fixed a proxy sidecar segfault while parsing CEL expressions (Issue 497)

Bookinfo sample application security fixes

Announcing Istio 1.5.4

Wed, 13 May 2020 00:00:00 +0000

This release fixes the security vulnerability described in our May 12th, 2020 news post.

This release note describes what’s different between Istio 1.5.4 and Istio 1.5.3.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-005 Denial of Service with Telemetry V2 enabled.

CVE-2020-10739: By sending a specially crafted packet, an attacker could trigger a Null Pointer Exception resulting in a Denial of Service. This could be sent to the ingress gateway or a sidecar.

ISTIO-SECURITY-2020-005

Tue, 12 May 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-10739
CVSS Impact Score	7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	1.4 to 1.4.8 1.5 to 1.5.3

Istio 1.4 with telemetry v2 enabled and Istio 1.5 contain the following vulnerability when telemetry v2 is enabled:

CVE-2020-10739: By sending a specially crafted packet, an attacker could trigger a Null Pointer Exception resulting in a Denial of Service. This could be sent to the ingress gateway or a sidecar.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H

Mitigation

For Istio 1.4.x deployments: update to Istio 1.4.9 or later.
For Istio 1.5.x deployments: update to Istio 1.5.4 or later.
Workaround: Alternatively, you can disable telemetry v2 by running the following:

$ istioctl manifest apply --set values.telemetry.v2.enabled=false

Credit

We’d like to thank Joren Zandstra for the original bug report.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.5.3

Tue, 12 May 2020 00:00:00 +0000

Due to a publishing error, the 1.5.3 images do not contain the fix for CVE-2020-10739 as claimed in the original announcement.

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.5.3 and Istio 1.5.2.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed the Helm installer to install Kiali using a dynamically generated signing key.
Fixed overlaying the generated Kubernetes resources for addon components with user-defined overlays (Issue 23048)
Fixed istio-sidecar.deb failing to start on Debian buster with iptables default nftables setting (Issue 23279)
Fixed the corresponding hash policy not being updated after the header name specified in DestinationRule.trafficPolicy.loadBalancer.consistentHash.httpHeaderName is changed (Issue 23434)
Fixed traffic routing when deployed in a namespace other than istio-system (Issue 23401)

Announcing Istio 1.4.9

Tue, 12 May 2020 00:00:00 +0000

This release contains bug fixes to improve robustness and fixes for the security vulnerabilities described in our May 12th, 2020 news post. This release note describes what’s different between Istio 1.4.9 and Istio 1.4.8.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-005 Denial of Service with Telemetry V2 enabled.

CVE-2020-10739: By sending a specially crafted packet, an attacker could trigger a Null Pointer Exception resulting in a Denial of Service. This could be sent to the ingress gateway or a sidecar.

Bug Fixes

Fixed the Helm installer to install Kiali using an dynamically generated signing key.
Fixed Citadel to ignore namespaces that are not part of the mesh.
Fixed the Istio operator installer to print the name of any resources that are not ready when an installation timeout occurs.

Support for Istio 1.4 ends on June 5th, 2020

Tue, 05 May 2020 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.4 are supported for three months after the next LTS release. Since 1.5 was released on March 5th, support for 1.4 will end on June 5th, 2020.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.4, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

Announcing Istio 1.5.2

Fri, 24 Apr 2020 00:00:00 +0000

This release contains bug fixes to improve robustness. This release note describes what’s different between Istio 1.5.1 and Istio 1.5.2.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Changes

Fixed Istiod deployment lacking label used by the matching PodDisruptionBudget (Issue 22267)
Fixed Custom Istio installation with istioctl not working using external charts (Issue 22368)
Fixed Panic in istio-init with GKE+COS and interceptionMode: TPROXY (Issue 22500)
Fixed Logging for validation by sending warnings to stdErr (Issue 22496)
Fixed Kiali not working when external Prometheus link used for the IstioOperator API (Issue 22510)
Fixed Istio agent should calculate grace period based on the cert TTL, not client-side settings (Issue 22226]
Fixed Incorrect error message referring to incorrect CLI option for the istioctl kube-inject command (Issue 22501)
Fixed IstioOperator validation of slice (Issue 21915)
Fixed Race condition caused by read/write of rootCert and rootCertExpireTime not always being protected (Issue 22627)
Fixed BlackHoleCluster HTTP metrics broken with Telemetry v2 (Issue 21385)
Fixed istio-init container failing when Istio CNI is enabled (Issue 22695)
Fixed istioctl does not set gateway name for multiple gateways (Issue 22703)
Fixed Unstable inbound bind address when configuring a sidecar ingress listener without bind address (Issue 22830)
Fixed Proxy pods for Istio 1.4 not showing up when upgrading from Istio 1.4 to 1.5 using default profile (Issue 22841)
Fixed PersistentVolumeClaim for Grafana not being created in the namespace specified in the IstioOperator spec (Issue 22835)
Fixed istio-sidecar-injector and istiod related pods crashing when applying new manifest through istioctl because alwaysInjectSelector and neverInjectSelector are not correctly indented in the istio-sidecar-injector config map (Issue 23027)
Fixed Prometheus scraping failing in CNI injected pods because the default excludeInboundPort configuration does not include port 15090 (Issue 23038)
Fixed Lightstep secret volume issue causing the bundled Prometheus to not install correctly with Istio operator (Issue 23078)
Fixed Avoid using host header to extract destination service name at gateway in default Telemetry V2 configuration.
Fixed Zipkin: Fix wrongly rendered timestamp value (Issue 22968)
Improved Add annotations for setting CPU/memory limits on sidecar (Issue 16126)
Improved Enable rewriteAppHTTPProbe annotation for liveness probe rewrite by default(Issue 10357)

Announcing Istio 1.4.8

Thu, 23 Apr 2020 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.4.7 and Istio 1.4.8.

The fixes below focus on various issues related to installing Istio on OpenShift with CNI. Instructions for installing Istio on OpenShift with CNI can be found here.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed Fixed CNI installation on OpenShift (Issue 21421) (Issue 22449).
Fixed Not all inbound ports are redirected when CNI is enabled (Issue 22448).
Fixed Syntax errors in gateway templates with GoLang 1.14 (Issue 22366).
Fixed Remove namespace from clusterrole and clusterrolebinding (PR 297).

ISTIO-SECURITY-2020-004

Wed, 25 Mar 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-1764
CVSS Impact Score	8.7 AV:A/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:N
Affected Releases	1.4 to 1.4.6 1.5

Istio 1.4 to 1.4.6 and Istio 1.5 contain the following vulnerability:

CVE-2020-1764: Istio uses a default signing_key for Kiali. This can allow an attacker to view and modify the Istio configuration.
- CVSS Score: 8.7 AV:A/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:N

In addition, another CVE is fixed in this release, described by this Kiali security bulletin.

Detection

Your installation is vulnerable in the following configuration:

The Kiali version is 1.15 or earlier.
The Kiali login token and signing key is unset.

To check your Kiali version, run this command:

$ kubectl get pods -n istio-system -l app=kiali -o yaml | grep image:

To determine if your login token is unset, run this command and check for blank output:

$ kubectl get deploy kiali -n istio-system -o yaml | grep LOGIN_TOKEN_SIGNING_KEY

To determine if your signing key is unset, run this command and check for blank output:

$ kubectl get cm kiali -n istio-system -o yaml | grep signing_key

Mitigation

For Istio 1.4.x deployments: update to Istio 1.4.7 or later.
For Istio 1.5.x deployments: update to Istio 1.5.1 or later.

Workaround: You can manually update the signing key to a random token using the following command:

$ kubectl get cm kiali -n istio-system -o yaml | sed "s/server:/login_token:\\\n \
signing_key: $(tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 20 | head -n 1)\\\nserver:/" \
| kubectl apply -f - ; kubectl delete pod -l app=kiali -n istio-system

Announcing Istio 1.5.1

Wed, 25 Mar 2020 00:00:00 +0000

This release contains bug fixes to improve robustness and fixes for the security vulnerabilities described in our March 25th, 2020 news post. This release note describes what’s different between Istio 1.5.0 and Istio 1.5.1.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-004 Istio uses a hard coded signing_key for Kiali.

CVE-2020-1764: Istio uses a default signing key to install Kiali. This can allow an attacker with access to Kiali to bypass authentication and gain administrative privileges over Istio. In addition, another CVE is fixed in this release, described in the Kiali 1.15.1 release.

Changes

Fixed an issue where Istio Operator instance deletion hangs for in-cluster operator (Issue 22280)
Fixed istioctl proxy-status should not list differences if just the order of the routes have changed (Issue 21709)
Fixed Incomplete support for array notation in “istioctl manifest apply —set” (Issue 20950)
Fixed Add possibility to add annotations to services in Kubernetes service spec (Issue 21995)
Fixed Enable setting ILB Gateway using istioctl (Issue 20033)
Fixed istioctl does not correctly set names on gateways (Issue 21938)
Fixed OpenID discovery does not work with beta request authentication policy (Issue 21954)
Fixed Issues related to shared control plane multicluster (Issue 22173)
Fixed Ingress port displaying target port instead of actual port (Issue 22125)
Fixed Issue where endpoints were being pruned automatically when installing the Istio Controller (Issue 21495)
Fixed Add istiod port to gateways for mesh expansion(Issue 22027)
Fixed Multicluster secret controller silently ignoring updates to secrets (Issue 18708)
Fixed Autoscaler for mixer-telemetry always being generated when deploying with istioctl or Helm (Issue 20935)
Fixed Prometheus certificate provisioning is broken (Issue 21843)
Fixed Segmentation fault in Pilot with beta mutual TLS (Issue 21816)
Fixed Operator status enumeration not being rendered as a string (Issue 21554)
Fixed in-cluster operator fails to install control plane after having deleted a prior control plane (Issue 21467)
Fixed TCP metrics for BlackHole clusters does not work with Telemetry v2 (Issue 21566)
Improved Add option to enable V8 runtime for telemetry V2 (Issue 21846)
Improved compatibility of Helm gateway chart (Issue 22295)
Improved operator by adding a Helm installation chart (Issue 21861)
Improved Support custom CA on istio-agent (Issue 22113)
Improved Add a flag that supports passing GCP metadata to STS (Issue 21904)

Announcing Istio 1.4.7

Wed, 25 Mar 2020 00:00:00 +0000

This release contains fixes for the security vulnerabilities described in our March 25th, 2020 news post. This release note describes what’s different between Istio 1.4.6 and Istio 1.4.7.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security Update

ISTIO-SECURITY-2020-004 Istio uses a hard coded signing_key for Kiali.

Changes

Fixed an issue causing protocol detection to break HTTP2 traffic to gateways (Issue 21230).

ISTIO-SECURITY-2020-003

Tue, 03 Mar 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-8659 CVE-2020-8660 CVE-2020-8661 CVE-2020-8664
CVSS Impact Score	7.5
Affected Releases	1.4 to 1.4.5

Envoy, and subsequently Istio are vulnerable to four newly discovered vulnerabilities:

CVE-2020-8659: The Envoy proxy may consume excessive memory when proxying HTTP/1.1 requests or responses with many small (i.e. 1 byte) chunks. Envoy allocates a separate buffer fragment for each incoming or outgoing chunk with the size rounded to the nearest 4Kb and does not release empty chunks after committing data. Processing requests or responses with a lot of small chunks may result in extremely high memory overhead if the peer is slow or unable to read proxied data. The memory overhead could be two to three orders of magnitude more than configured buffer limits.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:F
CVE-2020-8660: The Envoy proxy contains a TLS inspector that can be bypassed (not recognized as a TLS client) by a client using only TLS 1.3. Because TLS extensions (SNI, ALPN) are not inspected, those connections may be matched to a wrong filter chain, possibly bypassing some security restrictions.
- CVSS Score: 5.3 AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
CVE-2020-8661: The Envoy proxy may consume excessive amounts of memory when responding to pipelined HTTP/1.1 requests. In the case of illegally formed requests, Envoy sends an internally generated 400 error, which is sent to the Network::Connection buffer. If the client reads these responses slowly, it is possible to build up a large number of responses, and consume functionally unlimited memory. This bypasses Envoy’s overload manager, which will itself send an internally generated response when Envoy approaches configured memory thresholds, exacerbating the problem.
- CVSS Score: 7.5 AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:F
CVE-2020-8664: For the SDS TLS validation context in the Envoy proxy, the update callback is called only when the secret is received for the first time or when its value changes. This leads to a race condition where other resources referencing the same secret (e.g,. trusted CA) remain unconfigured until the secret’s value changes, creating a potentially sizable window where a complete bypass of security checks from the static (“default”) section can occur.
- CVSS Score: 5.3 AV:N/AC:L/PR:N/UI:N/S:U/C:L/I:N/A:N
This vulnerability only affects the SDS implementation of Istio’s certificate rotation mechanism for Istio 1.4.5 and earlier which is only when SDS and mutual TLS are enabled. SDS is off by default and must be explicitly enabled by the operator in all versions of Istio prior to Istio 1.5. Istio’s default secret distribution implementation based on Kubernetes secret mounts is not affected by this vulnerability.

Detection

To determine if SDS is enabled in your system, run:
```
$ kubectl get pod -l app=pilot -o yaml | grep SDS_ENABLED -A 1
```
If the output contains:
```
- name: SDS_ENABLED
value: "true"
```
your system has SDS enabled.

To determine if mutual TLS is enabled in your system, run:
```
$ kubectl get destinationrule --all-namespaces -o yaml | grep trafficPolicy -A 2
```
If the output contains:
```
--
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
```
your system has mutual TLS enabled.

Mitigation

For Istio 1.4.x deployments: update to Istio 1.4.6 or later.
For Istio 1.5.x deployments: Istio 1.5.0 will contain the equivalent security fixes.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.4.6

Tue, 03 Mar 2020 00:00:00 +0000

This release contains fixes for the security vulnerabilities described in our March 3rd, 2020 news post. This release note describes what’s different between Istio 1.4.5 and Istio 1.4.6.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-003 Two Uncontrolled Resource Consumption and Two Incorrect Access Control Vulnerabilities in Envoy.

CVE-2020-8659: The Envoy proxy may consume excessive memory when proxying HTTP/1.1 requests or responses with many small (i.e. 1 byte) chunks. Envoy allocates a separate buffer fragment for each incoming or outgoing chunk with the size rounded to the nearest 4Kb and does not release empty chunks after committing data. Processing requests or responses with a lot of small chunks may result in extremely high memory overhead if the peer is slow or unable to read proxied data. The memory overhead could be two to three orders of magnitude more than configured buffer limits.

CVE-2020-8660: The Envoy proxy contains a TLS inspector that can be bypassed (not recognized as a TLS client) by a client using only TLS 1.3. Because TLS extensions (SNI, ALPN) are not inspected, those connections may be matched to a wrong filter chain, possibly bypassing some security restrictions.

CVE-2020-8661: The Envoy proxy may consume excessive amounts of memory when responding to pipelined HTTP/1.1 requests. In the case of illegally formed requests, Envoy sends an internally generated 400 error, which is sent to the Network::Connection buffer. If the client reads these responses slowly, it is possible to build up a large number of responses, and consume functionally unlimited memory. This bypasses Envoy’s overload manager, which will itself send an internally generated response when Envoy approaches configured memory thresholds, exacerbating the problem.

CVE-2020-8664: For the SDS TLS validation context in the Envoy proxy, the update callback is called only when the secret is received for the first time or when its value changes. This leads to a race condition where other resources referencing the same secret (e.g,. trusted CA) remain unconfigured until the secret’s value changes, creating a potentially sizable window where a complete bypass of security checks from the static (“default”) section can occur.

This vulnerability only affects the SDS implementation of Istio’s certificate rotation mechanism for Istio 1.4.5 and earlier which is only when SDS and mutual TLS are enabled. SDS is off by default and must be explicitly enabled by the operator in all versions of Istio prior to Istio 1.5. Istio’s default secret distribution implementation based on Kubernetes secret mounts is not affected by this vulnerability.

Announcing Istio 1.4.5

Tue, 18 Feb 2020 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.4.4 and Istio 1.4.5.

The fixes below focus on various bugs occurring during node restarts. If you use Istio CNI, or have nodes that restart, you are highly encouraged to upgrade.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Improvements

Fixed a bug triggered by node restart causing Pods to receive incorrect configuration (Issue 20676).
Improved Istio CNI robustness. Previously, when a node restarted, new pods may be created before the CNI was setup, causing pods to be created without iptables rules configured (Issue 14327).
Fixed MCP metrics to include the size of the MCP responses, rather than just requests (Issue 21049).

Support for Istio 1.3 has ended

Fri, 14 Feb 2020 00:00:00 +0000

As previously announced, support for Istio 1.3 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.3, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

ISTIO-SECURITY-2020-002

Tue, 11 Feb 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-8843
CVSS Impact Score	7.4 AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:N
Affected Releases	1.3 to 1.3.6

Istio 1.3 to 1.3.6 contain a vulnerability affecting Mixer policy checks.

Note: We regret that the vulnerability was silently fixed in Istio 1.4.0 and Istio 1.3.7. An issue was raised and fixed in Istio 1.4.0 as a non-security issue. We reclassified the issue as a vulnerability in Dec 2019.

CVE-2020-8843: Under certain circumstances it is possible to bypass a specifically configured Mixer policy. Istio-proxy accepts x-istio-attributes header at ingress that can be used to affect policy decisions when Mixer policy selectively applies to source equal to ingress. To be vulnerable, Istio must have Mixer Policy enabled and used in the specified way. This feature is disabled by default in Istio 1.3 and 1.4.

Mitigation

For Istio 1.3.x deployments: update to Istio 1.3.7 or later.

Credit

The Istio team would like to thank Krishnan Anantheswaran and Eric Zhang of Splunk for the private bug report.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

ISTIO-SECURITY-2020-001

Tue, 11 Feb 2020 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2020-8595
CVSS Impact Score	9.0 AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H
Affected Releases	1.3 to 1.3.7 1.4 to 1.4.3

Istio 1.3 to 1.3.7 and 1.4 to 1.4.3 are vulnerable to a newly discovered vulnerability affecting Authentication Policy:

CVE-2020-8595: A bug in Istio’s Authentication Policy exact path matching logic allows unauthorized access to resources without a valid JWT token. This bug affects all versions of Istio that support JWT Authentication Policy with path based trigger rules. The logic for the exact path match in the Istio JWT filter includes query strings or fragments instead of stripping them off before matching. This means attackers can bypass the JWT validation by appending ? or # characters after the protected paths.

Mitigation

For Istio 1.3.x deployments: update to Istio 1.3.8 or later.
For Istio 1.4.x deployments: update to Istio 1.4.4 or later.

Credit

The Istio team would like to thank Aspen Mesh for the original bug report and code fix of CVE-2020-8595.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.4.4

Tue, 11 Feb 2020 00:00:00 +0000

This release includes bug fixes to improve robustness and user experience as well as a fix for the security vulnerability described in our February 11th, 2020 news post. This release note describes what’s different between Istio 1.4.3 and Istio 1.4.4.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-001 An improper input validation has been discovered in AuthenticationPolicy.

CVE-2020-8595: A bug in Istio’s Authentication Policy exact path matching logic allows unauthorized access to resources without a valid JWT token.

Bug fixes

Fixed Debian packaging of iptables scripts (Issue 19615).
Fixed an issue where Pilot generated a wrong Envoy configuration when the same port was used more than once (Issue 19935).
Fixed an issue where running multiple instances of Pilot could lead to a crash (Issue 20047).
Fixed a potential flood of configuration pushes from Pilot to Envoy when scaling the deployment to zero (Issue 17957).
Fixed an issue where Mixer could not fetch the correct information from the request/response when pod contains a dot in its name (Issue 20028).
Fixed an issue where Pilot sometimes would not send a correct pod configuration to Envoy (Issue 19025).
Fixed an issue where sidecar injector with SDS enabled was overwriting pod securityContext section, instead of just patching it (Issue 20409).

Improvements

Improved Better compatibility with Google CA. (Issues 20530, 20560).
Improved Added analyzer error message when Policies using JWT are not configured properly (Issues 20884, 20767).

Announcing Istio 1.3.8

Tue, 11 Feb 2020 00:00:00 +0000

This release contains a fix for the security vulnerability described in our February 11th, 2020 news post. This release note describes what’s different between Istio 1.3.7 and Istio 1.3.8.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2020-001 Improper input validation have been discovered in AuthenticationPolicy.

CVE-2020-8595: A bug in Istio’s Authentication Policy exact path matching logic allows unauthorized access to resources without a valid JWT token.

Announcing Istio 1.3.7

Tue, 04 Feb 2020 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.3.6 and Istio 1.3.7.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed root certificate rotation in Citadel to reuse values from the expiring root certificate into the new root certificate (Issue 19644).
Fixed telemetry to ignore forwarded attributes at the gateway.
Fixed sidecar injection into pods with containers that export no port (Issue 18594).
Added telemetry support for pod names containing periods (Issue 19015).
Added support for generating PKCS#8 private keys in Citadel agent (Issue 19948).

Minor enhancements

Improved injection template to fully specify securityContext, allowing PodSecurityPolicies to properly validate injected deployments (Issue 17318).
Added support for setting the lifecycle for proxy containers.
Added support for setting the Mesh UID in the Stackdriver Mixer adapter (Issue 17952).

Security update

ISTIO-SECURITY-2020-002 Mixer policy check bypass caused by improperly accepting certain request headers.

CVE-2020-8843: Under certain circumstances it is possible to bypass a specifically configured Mixer policy. Istio-proxy accepts x-istio-attributes header at ingress that can be used to affect policy decisions when Mixer policy selectively applies to source equal to ingress. Istio 1.3 to 1.3.6 is vulnerable.

Support for Istio 1.3 ends on February 14th, 2020

Wed, 15 Jan 2020 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.3 are supported for three months after the next LTS release. Since 1.4 was released on November 14th, support for 1.3 will end on February 14th, 2020.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.3, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

Announcing Istio 1.4.3

Wed, 08 Jan 2020 00:00:00 +0000

This release includes bug fixes to improve robustness and user experience. This release note describes what’s different between Istio 1.4.2 and Istio 1.4.3.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed an issue where Mixer creates too many watches, overloading kube-apiserver (Issue 19481).
Fixed an issue with injection when pod has multiple containers without exposed ports (Issue 18594).
Fixed overly restrictive validation of regex field (Issue 19212).
Fixed an upgrade issue with regex field (Issue 19665).
Fixed istioctl install to properly send logs to stderr (Issue 17743).
Fixed an issue where a file and profile could not be specified for istioctl installs (Issue 19503).
Fixed an issue preventing certain objects from being installed for istioctl installs (Issue 19371).
Fixed an issue preventing using certain JWKS with EC keys in JWT policy (Issue 19424).

Improvements

Improved injection template to fully specify securityContext, allowing PodSecurityPolicies to properly validate injected deployments (Issue 17318).
Improved telemetry v2 configuration to support Stackdriver and forward compatibility (Issue 591).
Improved output of istioctl installation (Issue 19451).
Improved istioctl installation to set exit code upon failure (Issue 19747).

Support for Istio 1.2 has ended

Fri, 13 Dec 2019 00:00:00 +0000

As previously announced, support for Istio 1.2 has now officially ended.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.2, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

ISTIO-SECURITY-2019-007

Tue, 10 Dec 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-18801 CVE-2019-18802
CVSS Impact Score	9.0 CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H
Affected Releases	1.2 to 1.2.9 1.3 to 1.3.5 1.4 to 1.4.1

Envoy, and subsequently Istio are vulnerable to two newly discovered vulnerabilities:

CVE-2019-18801: This vulnerability affects Envoy’s HTTP/1 codec in its way it processes downstream’s requests with large HTTP/2 headers. A successful exploitation of this vulnerability could lead to a denial of Service, escalation of privileges, or information disclosure.
CVE-2019-18802: HTTP/1 codec incorrectly fails to trim whitespace after header values. This could allow an attacker to bypass Istio’s policy either for information disclosure or escalation of privileges.
CVE-2019-18838: Upon receipt of a malformed HTTP request without the “Host” header, an encoder filter invoking Envoy’s route manager APIs that access request’s “Host” header will cause a NULL pointer to be dereferenced and result in abnormal termination of the Envoy process.

Impact and detection

Both Istio gateways and sidecars are vulnerable to this issue. If you are running one of the affected releases where downstream’s requests are HTTP/2 while upstream’s are HTTP/1, then your cluster is vulnerable. We expect this to be true of most clusters.

Mitigation

For Istio 1.2.x deployments: update to Istio 1.2.10 or later.
For Istio 1.3.x deployments: update to Istio 1.3.6 or later.
For Istio 1.4.x deployments: update to Istio 1.4.2 or later.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.4.2

Tue, 10 Dec 2019 00:00:00 +0000

This release contains fixes for the security vulnerability described in our December 10th, 2019 news post. This release note describes what’s different between Istio 1.4.1 and Istio 1.4.2.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2019-007 A heap overflow and improper input validation have been discovered in Envoy.

CVE-2019-18801: Fix a vulnerability affecting Envoy’s processing of large HTTP/2 request headers. A successful exploitation of this vulnerability could lead to a denial of service, escalation of privileges, or information disclosure. CVE-2019-18802: Fix a vulnerability resulting from whitespace after HTTP/1 header values which could allow an attacker to bypass Istio’s policy checks, potentially resulting in information disclosure or escalation of privileges. CVE-2019-18838: Fix a vulnerability resulting from malformed HTTP request missing the “Host” header. An encoder filter that invokes Envoy’s route manager APIs that access request’s “Host” header will cause a NULL pointer to be dereferenced and result in abnormal termination of the Envoy process.

Announcing Istio 1.3.6

Tue, 10 Dec 2019 00:00:00 +0000

This release contains fixes for the security vulnerability described in our December 10th, 2019 news post as well as bug fixes to improve robustness. This release note describes what’s different between Istio 1.3.5 and Istio 1.3.6.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2019-007 A heap overflow and improper input validation have been discovered in Envoy.

Bug fixes

Fixed an issue where a duplicate listener was generated for a proxy’s IP address when using a headless TCP service. (Issue 17748)
Fixed an issue with the destination_service label in HTTP related metrics incorrectly falling back to request.host which can cause a metric cardinality explosion for ingress traffic. (Issue 18818)

Minor enhancements

Improved load-shedding options for Mixer. Added support for a requests-per-second threshold for load-shedding enforcement. This allows operators to turn off load-shedding for Mixer in low traffic scenarios.

Announcing Istio 1.2.10

Tue, 10 Dec 2019 00:00:00 +0000

This release contains fixes for the security vulnerability described in our December 10th, 2019 news post. This release note describes what’s different between Istio 1.2.9 and Istio 1.2.10.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2019-007 A heap overflow and improper input validation have been discovered in Envoy.

Bug fix

Add support for Citadel to automatically rotate root cert. (Issue 17059)

Announcing Istio 1.4.1

Thu, 05 Dec 2019 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.4.0 and Istio 1.4.1.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed istioctl installation on Windows (Issue 19020).
Fixed an issue with route matching order when using cert-manager with Kubernetes Ingress (Issue 19000).
Fixed Mixer source namespace attribute when the pod name contains a period (Issue 19015).
Fixed excessive metrics generated by Galley (Issue 19165).
Fixed tracing Service port to correctly listen on port 80 (Issue 19227).
Fixed missing istioctl auto-completion files (Issue 19297).

Support for Istio 1.2 ends on December 13th, 2019

Mon, 11 Nov 2019 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.2 are supported for three months after the next LTS release. Since 1.3 was released on September 12th, support for 1.2 will end on December 13th, 2019.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.2, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

Announcing Istio 1.3.5

Mon, 11 Nov 2019 00:00:00 +0000

This release contains fixes for the security vulnerability described in our November 11, 2019 news post as well as bug fixes to improve robustness. This release note describes what’s different between Istio 1.3.4 and Istio 1.3.5.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

ISTIO-SECURITY-2019-006 A DoS vulnerability has been discovered in Envoy.

CVE-2019-18817: An infinite loop can be triggered in Envoy if the option continue_on_listener_filters_timeout is set to True, which is the case in Istio. This vulnerability could be leveraged for a DoS attack. If you applied the mitigation mentioned in our November 11, 2019 news post, you can remove the mitigation once you upgrade to Istio 1.3.5 or newer.

Bug fixes

Fixed Envoy listener configuration for TCP headless services. (Issue #17748)
Fixed an issue which caused stale endpoints to remain even when a deployment was scaled to 0 replicas. (Issue #14436)
Fixed Pilot to no longer crash when an invalid Envoy configuration is generated. (Issue 17266)
Fixed an issue with the destination_service_name label not getting populated for TCP metrics related to BlackHole/Passthrough clusters. (Issue 17271)
Fixed an issue with telemetry not reporting metrics for BlackHole/Passthrough clusters when fall through filter chains were invoked. This occurred when explicit ServiceEntries were configured for external services. (Issue 17759)

Minor enhancements

Added support for Citadel to periodically check the root certificate remaining lifetime and rotate expiring root certificates. (Issue 17059)
Added PILOT_BLOCK_HTTP_ON_443 boolean environment variable to Pilot. If enabled, this flag prevents HTTP services from running on port 443 in order to prevent conflicts with external HTTP services. This is disabled by default. (Issue 16458)

ISTIO-SECURITY-2019-006

Thu, 07 Nov 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-18817
CVSS Impact Score	7.5 CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:H/RL:O/RC:C
Affected Releases	1.3 to 1.3.4

Envoy, and subsequently Istio, are vulnerable to the following DoS attack. An infinite loop can be triggered in Envoy if the option continue_on_listener_filters_timeout is set to True. This has been the case for Istio since the introduction of the Protocol Detection feature in Istio 1.3 A remote attacker may trivially trigger that vulnerability, effectively exhausting Envoy’s CPU resources and causing a denial-of-service attack.

Impact and detection

Both Istio gateways and sidecars are vulnerable to this issue. If you are running one of the affected releases, your cluster is vulnerable.

Mitigation

Workaround: The exploitation of that vulnerability can be prevented by customizing Istio installation (as described in installation options ), using Helm to override the following options:
```
--set pilot.env.PILOT_INBOUND_PROTOCOL_DETECTION_TIMEOUT=0s --set global.proxy.protocolDetectionTimeout=0s
```
For Istio 1.3.x deployments: update to Istio 1.3.5 or later.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.2.9

Wed, 06 Nov 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.9. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix a proxy startup race condition.

Features

Adding support for Citadel automatic root certificate rotation (Issue 17059).

Announcing Istio 1.3.4

Fri, 01 Nov 2019 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.3.3 and Istio 1.3.4.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed a crashing bug in the Google node agent provider. (Pull Request #18296)
Fixed Prometheus annotations and updated Jaeger to 1.14. (Pull Request #18274)
Fixed in-bound listener reloads that occur on 5 minute intervals. (Issue #18138)
Fixed validation of key and certificate rotation. (Issue #17718)
Fixed invalid internal resource garbage collection. (Issue #16818)
Fixed webhooks that were not updated on a failure. (Pull Request #17820
Improved performance of OpenCensus tracing adapter. (Issue #18042)

Minor enhancements

Improved reliability of the SDS service. (Issue #17409, Issue #17905)
Added stable versions of failure domain labels. (Pull Request #17755)
Added update of the global mesh policy on upgrades. (Pull Request #17033)

Announcing Istio 1.2.8

Wed, 23 Oct 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.8. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix a bug introduced by our October 8th security release which incorrectly calculated HTTP header and body sizes (Issue 17735).
Fix a minor bug where endpoints still remained in /clusters while scaling a deployment to 0 replica (Issue 14336).
Fix Helm upgrade process to correctly update mesh policy for mutual TLS (Issue 16170).
Fix inconsistencies in the destination service label for TCP connection opened/closed metrics (Issue 17234).
Fix the Istio secret cleanup mechanism (Issue 17122).
Fix the Mixer Stackdriver adapter encoding process to handle invalid UTF-8 (Issue 16966).

Features

Add pilot support for the new failure domain labels: zone and region.

Support for Istio 1.1 has ended

Mon, 21 Oct 2019 00:00:00 +0000

As previously announced, support for Istio 1.1 has now officially ended.

Since we learned of the security vulnerability behind our October 8th security release while still barely within the 1.1 support period, we decided to extend the 1.1 support period beyond the original announcement and release 1.1.16. Then we discovered a bug in HTTP header size calculation was introduced by the security release, so we decided to release a fix in one last 1.1.17 release before closing out the 1.1 series for good.

At this point we will no longer back-port fixes for security issues and critical bugs to 1.1, so we heartily encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

Announcing Istio 1.1.17

Mon, 21 Oct 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.17. This will be the last 1.1.x patch release. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix a bug introduced by our October 8th security release which incorrectly calculated HTTP header and body sizes (Issue 17735.

Announcing Istio 1.3.3

Mon, 14 Oct 2019 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.3.2 and Istio 1.3.3.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed an issue which caused Prometheus to install improperly when using istioctl x manifest apply. (Issue 16970)
Fixed a bug where locality load balancing can not read locality information from the node. (Issue 17337)
Fixed a bug where long-lived connections were getting dropped by the Envoy proxy as the listeners were getting reconfigured without any user configuration changes. (Issue 17383, Issue 17139)
Fixed a crash in istioctl x analyze command. (Issue 17449)
Fixed istioctl x manifest diff to diff text blocks in ConfigMaps. (Issue 16828)
Fixed a segmentation fault crash in the Envoy proxy. (Issue 17699)

ISTIO-SECURITY-2019-005

Tue, 08 Oct 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-15226
CVSS Impact Score	7.5 CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	1.1 to 1.1.15 1.2 to 1.2.6 1.3 to 1.3.1

Envoy, and subsequently Istio, are vulnerable to the following DoS attack. Upon receiving each incoming request, Envoy will iterate over the request headers to verify that the total size of the headers stays below a maximum limit. A remote attacker may craft a request that stays below the maximum request header size but consists of many thousands of small headers to consume CPU and result in a denial-of-service attack.

Impact and detection

Both Istio gateways and sidecars are vulnerable to this issue. If you are running one of the affected releases, your cluster is vulnerable.

Mitigation

For Istio 1.1.x deployments: update all control plane components (Pilot, Mixer, Citadel, and Galley) and then upgrade the data plane to Istio 1.1.16 or later.
For Istio 1.2.x deployments: update all control plane components (Pilot, Mixer, Citadel, and Galley) and then upgrade the data plane to Istio 1.2.7 or later.
For Istio 1.3.x deployments: update all control plane components (Pilot, Mixer, Citadel, and Galley) and then upgrade the data plane to Istio 1.3.2 or later.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.3.2

Tue, 08 Oct 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.3.2. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

This release contains fixes for the security vulnerability described in our October 8th, 2019 news post. Specifically:

ISTIO-SECURITY-2019-005: A DoS vulnerability has been discovered by the Envoy community. * CVE-2019-15226: After investigation, the Istio team has found that this issue could be leveraged for a DoS attack in Istio if an attacker uses a high quantity of very small headers.

Nothing else is included in this release except for the above security fix. Distroless images will be available in a few days.

Announcing Istio 1.2.7

Tue, 08 Oct 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.7. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

This release contains fixes for the security vulnerability described in our October 8th, 2019 news post. Specifically:

Bug fix

Fix a bug where nodeagent was failing to start when using citadel (Issue 15876)

Announcing Istio 1.1.16

Tue, 08 Oct 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.16. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

This release contains fixes for the security vulnerability described in our October 8th, 2019 news post. Specifically:

Nothing else is included in this release except for the above security fix.

Announcing Istio 1.3.1

Fri, 27 Sep 2019 00:00:00 +0000

This release includes bug fixes to improve robustness. This release note describes what’s different between Istio 1.3.0 and Istio 1.3.1.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fixed an issue which caused the secret cleanup job to erroneously run during upgrades (Issue 16873).
Fixed an issue where the default configuration disabled Kubernetes Ingress support (Issue 17148)
Fixed an issue with handling invalid UTF-8 characters in the Stackdriver logging adapter (Issue 16966).
Fixed an issue which caused the destination_service label in HTTP metrics not to be set for BlackHoleCluster and PassThroughCluster (Issue 16629).
Fixed an issue with the destination_service label in the istio_tcp_connections_closed_total and istio_tcp_connections_opened_total metrics which caused them to not be set correctly (Issue 17234).
Fixed an Envoy crash introduced in Istio 1.2.4 (Issue 16357).
Fixed Istio CNI sidecar initialization when IPv6 is disabled on the node (Issue 15895).
Fixed a regression affecting support of RS384 and RS512 algorithms in JWTs (Issue 15380).

Minor enhancements

Added support for .Values.global.priorityClassName to the telemetry deployment.
Added annotations for Datadog tracing that controls extra features in sidecars.
Added the pilot_xds_push_time metric to report Pilot xDS push time.
Added istioctl experimental analyze to support multi-resource analysis and validation.
Added support for running metadata exchange and stats extensions in a WebAssembly sandbox.
Removed time diff info in the proxy-status command.

Announcing Istio 1.2.6

Tue, 17 Sep 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.6. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix redisquota inconsistency in regards to memquota counting (Issue 15543).
Fix an Envoy crash introduced in Istio 1.2.5 (Issue 16357).
Fix Citadel health check broken in the context of plugin certs (with intermediate certs) (Issue 16593).
Fix Stackdriver Mixer Adapter error log verbosity (Issue 16782).
Fix a bug where the service account map would be erased for service hostnames with more than one port.
Fix incorrect filterChainMatch wildcard hosts duplication produced by Pilot (Issue 16573).

Small enhancements

Expose sidecarToTelemetrySessionAffinity (required for Mixer V1) when it talks to services like Stackdriver. (Issue 16862).
Expose HTTP/2 window size settings as Pilot environment variables (Issue 17117).

Announcing Istio 1.1.15

Mon, 16 Sep 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.15. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix an Envoy crash introduced in Istio 1.1.14 (Issue 16357).

Small enhancements

Expose HTTP/2 window size settings as Pilot environment variables (Issue 17117).

Istio 1.2.4 sidecar image vulnerability

Tue, 10 Sep 2019 00:00:00 +0000

To the Istio’s user community,

For the period between Aug 23rd 2019 09:16PM PST and Sep 6th 2019 09:26AM PST a Docker image shipped as Istio proxyv2 1.2.4 (c.f. https://hub.docker.com/r/istio/proxyv2 ) contained a faulty version of the proxy against the vulnerabilities ISTIO-SECURITY-2019-003 and ISTIO-SECURITY-2019-004.

If you have installed Istio 1.2.4 during that time, please consider upgrading to Istio 1.2.5 that also contains additional security fixes.

Detailed explanation

Because of the communication embargo that we have exercised when fixing the recent HTTP2 DoS vulnerabilities, as it is usual for this type of release, we have built, in advance, a fixed image of the sidecar privately. At the moment of the public disclosure, we pushed that image manually on Docker hub.

For any release that isn’t fixing a privately disclosed security vulnerability, this Docker image is usually pushed through our release pipeline job, entirely automatically.

Our automated release process does not work correctly with the manual interactions required by the vulnerability disclosure embargo: the release pipeline code kept a reference to an outdated version of the Istio repository.

For a problem to occur, an automated build needed to be launched on an old version, this is what happened during the release of Istio 1.2.5: we have experienced a problem that required a revert commit which triggered a rebuild of 1.2.4 against an outdated version of Istio’s code.

This revert commit happened on Aug 23rd 2019 09:16PM PST. We have noticed this problem and pushed back the fixed image on Sep 6th 2019 09:26AM PST.

We are sorry for any inconvenience you may have experienced due to this incident, and are working towards a better release system, as well as a more efficient way to deal with vulnerability reports.

The release managers for 1.2

Announcing Istio 1.2.5

Mon, 26 Aug 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.5. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

Following the previous fixes for the security vulnerabilities described in ISTIO-SECURITY-2019-003 and ISTIO-SECURITY-2019-004, we are now addressing the internal control plane communication surface. These security fixes were not available at the time of our previous security release, and we considered the control plane gRPC surface to be harder to exploit.

You can find the gRPC vulnerability fix description on their mailing list (c.f. HTTP/2 Security Vulnerabilities).

Bug fixes

Fix an Envoy bug that breaks java.net.http.HttpClient and other clients that attempt to upgrade from HTTP/1.1 to HTTP/2 using the Upgrade: h2c header (Issue 16391).
Fix a memory leak on send timeout (Issue 15876).

Announcing Istio 1.1.14

Mon, 26 Aug 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.14. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

You can find the gRPC vulnerability fix description on their mailing list (c.f. HTTP/2 Security Vulnerabilities).

Bug fix

Fix an Envoy bug that breaks java.net.http.HttpClient and other clients that attempt to upgrade from HTTP/1.1 to HTTP/2 using the Upgrade: h2c header (Issue 16391).

Support for Istio 1.1 ends on September 19th, 2019

Thu, 15 Aug 2019 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.1 are supported for three months after the next LTS release. Since 1.2 was released on June 18th, support for 1.1 will end on September 19th, 2019.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.1, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

ISTIO-SECURITY-2019-004

Tue, 13 Aug 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-9512 CVE-2019-9513 CVE-2019-9514 CVE-2019-9515 CVE-2019-9518
CVSS Impact Score	7.5 CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	1.1 to 1.1.12 1.2 to 1.2.3

Envoy, and subsequently Istio are vulnerable to a series of trivial HTTP/2-based DoS attacks:

HTTP/2 flood using PING frames and queuing of response PING ACK frames that results in unbounded memory growth (which can lead to out of memory conditions).
HTTP/2 flood using PRIORITY frames that results in excessive CPU usage and starvation of other clients.
HTTP/2 flood using HEADERS frames with invalid HTTP headers and queuing of response RST_STREAM frames that results in unbounded memory growth (which can lead to out of memory conditions).
HTTP/2 flood using SETTINGS frames and queuing of SETTINGS ACK frames that results in unbounded memory growth (which can lead to out of memory conditions).
HTTP/2 flood using frames with an empty payload that results in excessive CPU usage and starvation of other clients.

Those vulnerabilities were reported externally and affect multiple proxy implementations. See this security bulletin for more information.

Impact and detection

If Istio terminates externally originated HTTP then it is vulnerable. If Istio is instead fronted by an intermediary that terminates HTTP (e.g., a HTTP load balancer), then that intermediary would protect Istio, assuming the intermediary is not itself vulnerable to the same HTTP/2 exploits.

Mitigation

For Istio 1.1.x deployments: update to a Istio 1.1.13 or later.
For Istio 1.2.x deployments: update to a Istio 1.2.4 or later.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

ISTIO-SECURITY-2019-003

Tue, 13 Aug 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-14993
CVSS Impact Score	7.5 CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
Affected Releases	1.1 to 1.1.12 1.2 to 1.2.3

An Envoy user reported publicly an issue (c.f. Envoy Issue 7728) about regular expressions (or regex) matching that crashes Envoy with very large URIs. After investigation, the Istio team has found that this issue could be leveraged for a DoS attack in Istio, if users are employing regular expressions in some of the Istio APIs: JWT, VirtualService, HTTPAPISpecBinding, QuotaSpecBinding.

Impact and detection

To detect if there is any regular expressions used in Istio APIs in your cluster, run the following command which prints either of the following output:

YOU ARE AFFECTED: found regex used in AuthenticationPolicy or VirtualService
YOU ARE NOT AFFECTED: did not find regex usage

$ cat <<'EOF' | bash -
set -e
set -u
set -o pipefail
red=`tput setaf 1`
green=`tput setaf 2`
reset=`tput sgr0`
echo "Checking regex usage in Istio API ..."
AFFECTED=()
JWT_REGEX=()
JWT_REGEX+=($(kubectl get Policy --all-namespaces -o jsonpath='{..regex}'))
JWT_REGEX+=($(kubectl get MeshPolicy --all-namespaces -o jsonpath='{..regex}'))
if [ "${#JWT_REGEX[@]}" != 0 ]; then
AFFECTED+=("AuthenticationPolicy")
fi
VS_REGEX=()
VS_REGEX+=($(kubectl get VirtualService --all-namespaces -o jsonpath='{..regex}'))
if [ "${#VS_REGEX[@]}" != 0 ]; then
AFFECTED+=("VirtualService")
fi
HTTPAPI_REGEX=()
HTTPAPI_REGEX+=($(kubectl get HTTPAPISpec --all-namespaces -o jsonpath='{..regex}'))
if [ "${#HTTPAPI_REGEX[@]}" != 0 ]; then
AFFECTED+=("HTTPAPISpec")
fi
QUOTA_REGEX=()
QUOTA_REGEX+=($(kubectl get QuotaSpec --all-namespaces -o jsonpath='{..regex}'))
if [ "${#QUOTA_REGEX[@]}" != 0 ]; then
AFFECTED+=("QuotaSpec")
fi
if [ "${#AFFECTED[@]}" != 0 ]; then
echo "${red}YOU ARE AFFECTED: found regex used in ${AFFECTED[@]}${reset}"
exit 1
fi
echo "${green}YOU ARE NOT AFFECTED: did not find regex usage${reset}"
EOF

Mitigation

For Istio 1.1.x deployments: update to Istio 1.1.13 or later
For Istio 1.2.x deployments: update to Istio 1.2.4 or later.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Announcing Istio 1.2.4

Tue, 13 Aug 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.4. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

This release contains fixes for the security vulnerabilities described in ISTIO-SECURITY-2019-003] ISTIO-SECURITY-2019-004. Specifically:

ISTIO-SECURITY-2019-003: An Envoy user reported publicly an issue (c.f. Envoy Issue 7728) about regular expressions matching that crashes Envoy with very large URIs. * CVE-2019-14993: After investigation, the Istio team has found that this issue could be leveraged for a DoS attack in Istio, if users are employing regular expressions in some of the Istio APIs: JWT, VirtualService, HTTPAPISpecBinding, QuotaSpecBinding.

ISTIO-SECURITY-2019-004: Envoy, and subsequently Istio are vulnerable to a series of trivial HTTP/2-based DoS attacks: * CVE-2019-9512: HTTP/2 flood using PING frames and queuing of response PING ACK frames that results in unbounded memory growth (which can lead to out of memory conditions). * CVE-2019-9513: HTTP/2 flood using PRIORITY frames that results in excessive CPU usage and starvation of other clients. * CVE-2019-9514: HTTP/2 flood using HEADERS frames with invalid HTTP headers and queuing of response RST_STREAM frames that results in unbounded memory growth (which can lead to out of memory conditions). * CVE-2019-9515: HTTP/2 flood using SETTINGS frames and queuing of SETTINGS ACK frames that results in unbounded memory growth (which can lead to out of memory conditions). * CVE-2019-9518: HTTP/2 flood using frames with an empty payload that results in excessive CPU usage and starvation of other clients.

Nothing else is included in this release except for the above security fixes.

Announcing Istio 1.1.13

Tue, 13 Aug 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.13. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

This release contains fixes for the security vulnerabilities described in ISTIO-SECURITY-2019-003 and ISTIO-SECURITY-2019-004. Specifically:

Nothing else is included in this release except for the above security fixes.

Announcing Istio 1.2.3

Fri, 02 Aug 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.3. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix a bug where the sidecar could infinitely forward requests to itself when pod defines a port undefined for service (Issue 14443) and (Issue 14242)
Fix a bug where Stackdriver adapter shuts down after telemetry is started.
Fix Redis connectivity issues.
Fix case-sensitivity in regex-based HTTP URI matching for Virtual Service (Issue 14983)
Fix HPA and CPU settings for demo profile (Issue 15338)
Relax Keep-Alive enforcement policy to avoid dropping connections under load (Issue 15088)
When SDS is not used, skip Kubernetes JWT authentication to mitigate the risk of compromised (untrustworthy) JWTs being used.

Tests upgrade

Update base image version for Bookinfo reviews sample app (Issue 15477)
Bookinfo samples image qualification (Issue 14237)

Announcing Istio 1.1.12

Fri, 02 Aug 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.12. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix a bug where the sidecar could infinitely forward requests to itself when a Pod resource defines a port that isn’t defined for a service (Issue 14443) and (Issue 14242)

Announcing Istio 1.1.11

Wed, 03 Jul 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.11. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Small enhancements

Add ability to enable HTTP/1.0 support in ingress gateway (Issue 13085).

ISTIO-SECURITY-2019-002

Fri, 28 Jun 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-12995
CVSS Impact Score	7.5 CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H/E:F/RL:O/RC:C
Affected Releases	1.0 to 1.0.8 1.1 to 1.1.9 1.2 to 1.2.1

A bug in Istio’s JWT validation filter causes Envoy to crash in certain cases when the request contains a malformed JWT token. The bug was discovered and reported by a user on GitHub on June 23, 2019.

This bug affects all versions of Istio that are using the JWT authentication policy.

The symptoms of the bug are an HTTP 503 error seen by the client, and

Epoch 0 terminated with an error: signal: segmentation fault (core dumped)

in the Envoy logs.

The Envoy crash can be triggered using a malformed JWT without a valid signature, and on any URI being accessed regardless of the trigger_rules in the JWT specification. Thus, this bug makes Envoy vulnerable to a potential DoS attack.

Impact and detection

Envoy is vulnerable if the following two conditions are satisfied:

A JWT authentication policy is applied to it.
The JWT issuer (specified by jwksUri) uses the RSA algorithm for signature verification

If JWT policy is applied to the Istio ingress gateway, please be aware that any external user who has access to the ingress gateway could crash it with a single HTTP request.

If JWT policy is applied to the sidecar only, please keep in mind it might still be vulnerable. For example, the Istio ingress gateway might forward the JWT token to the sidecar which could be a malformed JWT token that crashes the sidecar.

A vulnerable Envoy will crash on an HTTP request with a malformed JWT token. When Envoy crashes, all existing connections will be disconnected immediately. The pilot-agent will restart the crashed Envoy automatically and it may take a few seconds to a few minutes for the restart. pilot-agent will stop restarting Envoy after it crashed more than ten times. In this case, Kubernetes will redeploy the pod, including the workload behind Envoy.

To detect if there is any JWT authentication policy applied in your cluster, run the following command which print either of the following output:

Found JWT in authentication policy, YOU ARE AFFECTED
Did NOT find JWT in authentication policy, YOU ARE NOT AFFECTED

$ cat <<'EOF' | bash -
set -e
set -u
set -o pipefail
red=`tput setaf 1`
green=`tput setaf 2`
reset=`tput sgr0`
echo "Checking authentication policy..."
JWKS_URI=()
JWKS_URI+=($(kubectl get policy --all-namespaces -o jsonpath='{range .items[*]}{.spec.origins[*].jwt.jwksUri}{" "}{end}'))
JWKS_URI+=($(kubectl get meshpolicy --all-namespaces -o jsonpath='{range .items[*]}{.spec.origins[*].jwt.jwksUri}{" "}{end}'))
if [ "${#JWKS_URI[@]}" != 0 ]; then
echo "${red}Found JWT in authentication policy, YOU ARE AFFECTED${reset}"
exit 1
fi
echo "${green}Did NOT find JWT in authentication policy, YOU ARE NOT AFFECTED${reset}"
EOF

Mitigation

This bug is fixed in the following Istio releases:

For Istio 1.0.x deployments: update to Istio 1.0.9 or later.
For Istio 1.1.x deployments: update to Istio 1.1.10 or later.
For Istio 1.2.x deployments: update to Istio 1.2.2 or later.

If you cannot immediately upgrade to one of these releases, you have the additional option of injecting a Lua filter into older releases of Istio. This filter has been verified to work with Istio 1.1.9, 1.0.8, 1.0.6, and 1.1.3.

The Lua filter is injected before the Istio jwt-auth filter. If a JWT token is presented on an http request, the Lua filter will check if the JWT token header contains alg:ES256. If the filter finds such a JWT token, the request is rejected.

To install the Lua filter, please invoke the following commands:

$ git clone git@github.com:istio/tools.git
$ cd tools/examples/luacheck/
$ ./setup.sh

Inspect the full set of source code changes.

Bug fixes

Fix crash in Istio’s JWT Envoy filter caused by malformed JWT (Issue 15084).

Announcing Istio 1.2.1

Thu, 27 Jun 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.2.1. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix duplicate CRD being generated in the install (Issue 14976)
Fix Mixer unable to start when Galley is disabled (Issue 14841)
Fix environment variable shadowing (NAMESPACE is used for listened namespaces and overwrites Citadel storage namespace (istio-system))
Fix cause of ‘TLS error: Secret is not supplied by SDS’ errors during upgrade (Issue 15020)

Minor enhancements

Allow users to disable Istio default retries by setting retries to 0 (Issue 14900)
Introduction of a Redis filter (this feature is guarded with the environment feature flag PILOT_ENABLE_REDIS_FILTER, disabled by default)
Add HTTP/1.0 support to gateway configuration generation (Issue 13085)
Add toleration for Istio components (Pull Request 15081)

Support for Istio 1.0 has ended

Wed, 19 Jun 2019 00:00:00 +0000

As previously announced, support for Istio 1.0 has now officially ended.

We will no longer back-port fixes for security issues and critical bugs to 1.0, so we encourage you to upgrade to the latest version of Istio (1.9.5) if you haven’t already.

Announcing Istio 1.1.9

Mon, 17 Jun 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.9. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Prevent overly large strings from being sent to Prometheus (Issue 14642).
Reuse previously cached JWT public keys if transport errors are encountered during renewal (Issue 14638).
Bypass JWT authentication for HTTP OPTIONS methods to support CORS requests.
Fix Envoy crash caused by the Mixer filter (Issue 14707).

Small enhancements

Expose cryptographic signature verification functions to Lua Envoy filters (Envoy Issue 7009).

Announcing Istio 1.0.8

Fri, 07 Jun 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.8. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix issue where Citadel could generate a new root CA if it cannot contact the Kubernetes API server, causing mutual TLS verification to incorrectly fail (Issue 14512).

Small enhancements

Update Citadel’s default root CA certificate TTL from 1 year to 10 years.

Announcing Istio 1.1.8

Thu, 06 Jun 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.8. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix PASSTHROUGH DestinationRules for CDS clusters (Issue 13744).
Make the appVersion and version fields in the Helm charts display the correct Istio version (Issue 14290).
Fix Mixer crash affecting both policy and telemetry servers (Issue 14235).
Fix multicluster issue where two pods in different clusters could not share the same IP address (Issue 14066).
Fix issue where Citadel could generate a new root CA if it cannot contact the Kubernetes API server, causing mutual TLS verification to incorrectly fail (Issue 14512).
Improve Pilot validation to reject different VirtualServices with the same domain since Envoy will not accept them (Issue 13267).
Fix locality load balancing issue where only one replica in a locality would receive traffic (13994).
Fix issue where Pilot Agent might not notice a TLS certificate rotation (Issue 14539).
Fix a LuaJIT panic in Envoy (Envoy Issue 6994).
Fix a race condition where Envoy might reuse a HTTP/1.1 connection after the downstream peer had already closed the TCP connection, causing 503 errors and retries (Issue 14037).
Fix a tracing issue in Mixer’s Zipkin adapter causing missing spans (Issue 13391).

Small enhancements

Reduce Pilot log spam by logging the the endpoints within network ... will be ignored for no network configured message at DEBUG.
Make it easier to rollback by making pilot-agent ignore unknown flags.
Update Citadel’s default root CA certificate TTL from 1 year to 10 years.

ISTIO-SECURITY-2019-001

Tue, 28 May 2019 00:00:00 +0000

Disclosure Details
CVE(s)	CVE-2019-12243
CVSS Impact Score	8.9 CVSS:3.0/AV:A/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:N/E:H/RL:O/RC:C
Affected Releases	1.1 to 1.1.6

During review of the Istio 1.1.7 release notes, we realized that issue 13868, which is fixed in the release, actually represents a security vulnerability.

Initially we thought the bug was impacting the TCP Authorization feature advertised as alpha stability, which would not have required invoking this security advisory process, but we later realized that the Deny Checker and List Checker feature were affected and those are considered stable features. We are revisiting our processes to flag vulnerabilities that are initially reported as bugs instead of through the private disclosure process.

We tracked the bug to a code change introduced in Istio 1.1 and affecting all releases up to 1.1.6.

Impact and detection

Since Istio 1.1, In the default Istio installation profile, policy enforcement is disabled by default.

You can check the status of policy enforcement for your mesh with the following command:

$ kubectl -n istio-system get cm istio -o jsonpath="{@.data.mesh}" | grep disablePolicyChecks
disablePolicyChecks: true

You are not impacted by this vulnerability if disablePolicyChecks is set to true.

You are impacted by the vulnerability issue if the following conditions are all true:

You are running one of the affected Istio releases.
disablePolicyChecks is set to false (follow the steps mentioned above to check)
Your workload is NOT using HTTP, HTTP/2, or gRPC protocols
A mixer adapter (e.g., Deny Checker, List Checker) is used to provide authorization for your backend TCP service.

Mitigation

Users of Istio 1.0.x are not affected.
For Istio 1.1.x deployments: update to Istio 1.1.7 or later.

Credit

The Istio team would like to thank Haim Helman for the original bug report.

Reporting vulnerabilities

We’d like to remind our community to follow the vulnerability reporting process to report any bug that can result in a security vulnerability.

Support for Istio 1.0 ends on June 19th, 2019

Thu, 23 May 2019 00:00:00 +0000

According to Istio’s support policy, LTS releases like 1.0 are supported for three months after the next LTS release. Since 1.1 was released on March 19th, support for 1.0 will end on June 19th, 2019.

At that point we will stop back-porting fixes for security issues and critical bugs to 1.0, so we encourage you to upgrade to the latest version of Istio (1.9.5). If you don’t do this you may put yourself in the position of having to do a major upgrade on a short timeframe to pick up a critical fix.

We care about you and your clusters, so please be kind to yourself and upgrade.

Announcing Istio 1.1.7

Fri, 17 May 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.7. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

This release fixes CVE 2019-12243.

Bug fixes

Fix issue where two gateways with overlapping hosts, created at the same second, can cause Pilot to fail to generate routes correctly and lead to Envoy listeners stuck indefinitely at startup in a warming state.
Improve the robustness of the SDS node agent: if Envoy sends a SDS request with an empty ResourceNames, ignore it and wait for the next request instead of closing the connection (Issue 13853).
In prior releases Pilot automatically injected the experimental envoy.filters.network.mysql_proxy filter into the outbound filter chain if the service port name is mysql. This was surprising and caused issues for some operators, so Pilot will now automatically inject the envoy.filters.network.mysql_proxy filter only if the PILOT_ENABLE_MYSQL_FILTER environment variable is set to 1 (Issue 13998).
Fix issue where Mixer policy checks were incorrectly disabled for TCP (Issue 13868).

Small enhancements

Add --applicationPorts option to the ingressgateway Helm charts. When set to a comma-delimited list of ports, readiness checks will fail until all the ports become active. When configured, traffic will not be sent to Envoys stuck in the warming state.
Increase memory limit in the ingressgateway Helm chart to 1GB and add resource request and limits to the SDS node agent container to support HPA autoscaling.

Announcing Istio 1.1.6

Sat, 11 May 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.6. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix Galley Helm charts so that the validatingwebhookconfiguration object can now deployed to a namespace other than istio-system (Issue 13625).
Additional Helm chart fixes for anti-affinity support: fix gatewaypodAntiAffinityRequiredDuringScheduling and podAntiAffinityLabelSelector match expressions and fix the default value for podAntiAffinityLabelSelector (Issue 13892).
Make Pilot handle a condition where Envoy continues to request routes for a deleted gateway while listeners are still draining (Issue 13739).

Small enhancements

If access logs are enabled, passthrough listener requests will be logged.
Make Pilot tolerate unknown JSON fields to make it easier to rollback to older versions during upgrade.
Add support for fallback secrets to SDS which Envoy can use instead of waiting indefinitely for late or non-existent secrets during startup (Issue 13853).

Announcing Istio 1.1.5

Fri, 03 May 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.5. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Add additional validation to Pilot to reject gateway configuration with overlapping hosts matches (Issue 13717).
Build against the latest stable version of istio-cni instead of the latest daily build (Issue 13171).

Small enhancements

Add additional logging to help diagnose hostname resolution failures (Issue 13581).
Improve ease of installing prometheus by removing unnecessary use of busybox image (Issue 13501).
Make Pilot Agent’s certificate paths configurable (Issue 11984).

Announcing Istio 1.1.4

Wed, 24 Apr 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.4. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Behavior change

Changed the default behavior for Pilot to allow traffic to outside the mesh, even if it is on the same port as an internal service. This behavior can be controlled by the PILOT_ENABLE_FALLTHROUGH_ROUTE environment variable.

Bug fixes

Fixed egress route generation for services of type ExternalName.
Added support for configuring Envoy’s idle connection timeout, which prevents running out of memory or IP ports over time (Issue 13355).
Fixed a crashing bug in Pilot in failover handling of locality-based load balancing.
Fixed a crashing bug in Pilot when it was given custom certificate paths.
Fixed a bug in Pilot where it was ignoring short names used as service entry hosts (Issue 13436).
Added missing https_protocol_options to the envoy-metrics-service cluster configuration.
Fixed a bug in Pilot where it didn’t handle https traffic correctly in the fall through route case (Issue 13386).
Fixed a bug where Pilot didn’t remove endpoints from Envoy after they were removed from Kubernetes (Issue 13402).
Fixed a crashing bug in the node agent (Issue 13325).
Added missing validation to prevent gateway names from containing dots (Issue 13211).
Fixed bug where ConsistentHashLB.minimumRingSize was defaulting to 0 instead of the documented 1024 (Issue 13261).

Small enhancements

Updated to the latest version of the Kiali add-on.
Updated to the latest version of Grafana.
Added validation to ensure Citadel is only deployed with a single replica (Issue 13383).
Added support to configure the logging level of the proxy and Istio control plane ((Issue 11847).
Allow sidecars to bind to any loopback address and not just 127.0.0.1 (Issue 13201).

Announcing Istio 1.1.3

Mon, 15 Apr 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.3. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Known issues with 1.1.3

A panic in the Node Agent was discovered late in the 1.1.3 qualification process. The panic only occurs in clusters with the alpha-quality SDS certificate rotation feature enabled. Since this is the first time we have included SDS certificate rotation in our long-running release tests, we don’t know whether this is a latent bug or a new regression. Considering SDS certificate rotation is in alpha, we have decided to release 1.1.3 with this issue and target a fix for the 1.1.4 release.

Bug fixes

Istio-specific back-ports of Envoy patches for CVE-2019-9900 and CVE-2019-9901 included in Istio 1.1.2 have been dropped in favor of an Envoy update which contains the final version of the patches.
Fix load balancer weight setting for split horizon EDS.
Fix typo in the default Envoy JSON log format (Issue 12232).
Correctly reload out-of-process adapter address upon configuration change (Issue 12488).
Restore Kiali settings that were accidentally deleted (Issue 3660).
Prevent services with same target port resulting in duplicate inbound listeners (Issue 9504).
Fix issue with configuring Sidecar egress ports for namespaces other than istio-system resulting in a envoy.tcp_proxy filter of BlackHoleCluster by auto binding to services for Sidecar listeners (Issue 12536).
Fix gateway vhost configuration generation issue by favoring more specific host matches (Issue 12655).
Fix ALLOW_ANY so it now allows external traffic if there is already an http service present on a port.
Fix validation logic so that port.name is no longer a valid PortSelection.
Fix istioctl proxy-config cluster cluster type column rendering (Issue 12455).
Fix SDS secret mount configuration.
Fix incorrect Istio version in the Helm charts.
Fix partial DNS failures in the presence of overlapping ports (Issue 11658).
Fix Helm podAntiAffinity template error (Issue 12790).
Fix bug with the original destination service discovery not using the original destination load balancer.
Fix SDS memory leak in the presence of invalid or missing keying materials (Issue 13197).

Small enhancements

Hide ServiceAccounts from PushContext log to reduce log volume.
Configure localityLbSetting in values.yaml by passing it through to the mesh configuration.
Remove the soon-to-be deprecated critical-pod annotation from Helm charts (Issue 12650).
Support pod anti-affinity annotations to improve control plane availability (Issue 11333).
Pretty print IP addresses in access logs.
Remove redundant write header to further reduce log volume.
Improve destination host validation in Pilot.
Explicitly configure istio-init to run as root so use of pod-level securityContext.runAsUser doesn’t break it (Issue 5453).
Add configuration samples for Vault integration.
Respect locality load balancing weight settings from ServiceEntry.
Make the TLS certificate location watched by Pilot Agent configurable (Issue 11984).
Add support for Datadog tracing.
Add alias to istioctl so ‘x’ can be used instead of ‘experimental’.
Provide improved distribution of sidecar certificate by adding jitter to their CSR requests.
Allow weighted load balancing registry locality to be configured.
Add support for standard CRDs for compiled-in Mixer adapters.
Reduce Pilot resource requirements for demo configuration.
Fully populate Galley dashboard by adding data source (Issue 13040).
Propagate Istio 1.1 sidecar performance tuning to the istio-gateway.
Improve destination host validation by rejecting * hosts (Issue 12794).
Expose upstream idle_timeout in cluster definition so dead connections can sometimes be removed from connection pools before they are used (Issue 9113).
When registering a Sidecar resource to restrict what a pod can see, the restrictions are now applied if the spec contains a workloadSelector (Issue 11818).
Update the Bookinfo example to use port 80 for TLS origination.
Add liveness probe for Citadel.
Improve AWS ELB interoperability by making 15020 the first port listed in the ingressgateway service (Issue 12502).
Use outlier detection for failover mode but not for distribute mode for locality weighted load balancing (Issues 12965).
Replace generation of Envoy’s deprecated enabled field in CorsPolicy with the replacement filter_enabled field for 1.1.0+ sidecars only.
Standardize labels on Mixer’s Helm charts.

Announcing Istio 1.1.2 with Important Security Update

Fri, 05 Apr 2019 00:00:00 +0000

We’re announcing immediate availability of Istio 1.1.2 which contains some important security updates. Please see below for details.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

Two security vulnerabilities have recently been identified in the Envoy proxy (CVE 2019-9900 and CVE 2019-9901). The vulnerabilities have now been patched in Envoy version 1.9.1, and correspondingly in the Envoy builds embedded in Istio 1.1.2 and Istio 1.0.7. Since Envoy is an integral part of Istio, users are advised to update Istio immediately to mitigate security risks arising from these vulnerabilities.

The vulnerabilities are centered on the fact that Envoy did not normalize HTTP URI paths and did not fully validate HTTP/1.1 header values. These vulnerabilities impact Istio features that rely on Envoy to enforce any of authorization, routing, or rate limiting.

Affected Istio releases

The following Istio releases are vulnerable:

1.1, 1.1.1
- These releases can be patched to Istio 1.1.2.
- 1.1.2 is built from the same source as 1.1.1 with the addition of Envoy patches minimally sufficient to address the CVEs.
1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6
- These releases can be patched to Istio 1.0.7
- 1.0.7 is built from the same source as 1.0.6 with the addition of Envoy patches minimally sufficient to address the CVEs.
0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8
- These releases are no longer supported and will not be patched. Please upgrade to a supported release with the necessary fixes.

Vulnerability impact

CVE 2019-9900 and CVE 2019-9901 allow remote attackers access to unauthorized resources by using specially crafted request URI paths (9901) and NUL bytes in HTTP/1.1 headers (9900), potentially circumventing DoS prevention systems such as rate limiting, or routing to a unexposed upstream system. Refer to issue 6434 and issue 6435 for more information.

As Istio is based on Envoy, Istio customers can be affected by these vulnerabilities based on whether paths and request headers are used within Istio policies or routing rules and how the backend HTTP implementation resolves them. If prefix path matching rules are used by Mixer or by Istio authorization policies or the routing rules, an attacker could exploit these vulnerabilities to gain access to unauthorized paths on certain HTTP backends.

Mitigation

Eliminating the vulnerabilities requires updating to a corrected version of Envoy. We’ve incorporated the necessary updates in the latest Istio patch releases.

For Istio 1.1.x deployments: update to a minimum of Istio 1.1.2

For Istio 1.0.x deployments: update to a minimum of Istio 1.0.7

While Envoy 1.9.1 requires opting in to path normalization to address CVE 2019-9901, the version of Envoy embedded in Istio 1.1.2 and 1.0.7 enables path normalization by default.

Detection of NUL header exploit

Based on current information, this only affects HTTP/1.1 traffic. If this is not structurally possible in your network or configuration, then it is unlikely that this vulnerability applies.

File-based access logging uses the c_str() representation for header values, as does gRPC access logging, so there will be no trivial detection via Envoy’s access logs by scanning for NUL. Instead, operators might look for inconsistencies in logs between the routing that Envoy performs and the logic intended in the RouteConfiguration.

External authorization and rate limit services can check for NULs in headers. Backend servers might have sufficient logging to detect NULs or unintended access; it’s likely that many will simply reject NULs in this scenario via 400 Bad Request, as per RFC 7230.

Detection of path traversal exploit

Envoy’s access logs (whether file-based or gRPC) will contain the unnormalized path, so it is possible to examine these logs to detect suspicious patterns and requests that are incongruous with the intended operator configuration intent. In addition, unnormalized paths are available at ext_authz, rate limiting and backend servers for log inspection.

Announcing Istio 1.0.7 with Important Security Update

Fri, 05 Apr 2019 00:00:00 +0000

We’re announcing immediate availability of Istio 1.0.7 which contains some important security updates. Please see below for details.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Security update

Affected Istio releases

The following Istio releases are vulnerable:

1.1, 1.1.1
- These releases can be patched to Istio 1.1.2.
- 1.1.2 is built from the same source as 1.1.1 with the addition of Envoy patches minimally sufficient to address the CVEs.
1.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6
- These releases can be patched to Istio 1.0.7
- 1.0.7 is built from the same source as 1.0.6 with the addition of Envoy patches minimally sufficient to address the CVEs.
0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8
- These releases are no longer supported and will not be patched. Please upgrade to a supported release with the necessary fixes.

Vulnerability impact

Mitigation

Eliminating the vulnerabilities requires updating to a corrected version of Envoy. We’ve incorporated the necessary updates in the latest Istio patch releases.

For Istio 1.1.x deployments: update to a minimum of Istio 1.1.2

For Istio 1.0.x deployments: update to a minimum of Istio 1.0.7

While Envoy 1.9.1 requires opting in to path normalization to address CVE 2019-9901, the version of Envoy embedded in Istio 1.1.2 and 1.0.7 enables path normalization by default.

Detection of NUL header exploit

Based on current information, this only affects HTTP/1.1 traffic. If this is not structurally possible in your network or configuration, then it is unlikely that this vulnerability applies.

Detection of path traversal exploit

Announcing Istio 1.1.1

Mon, 25 Mar 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.1.1. Please see below for what’s changed.

BEFORE YOU UPGRADE

Things to know and prepare before upgrading.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes and minor enhancements

Configure Prometheus to monitor Citadel (Issue 12175)
Improve output of istioctl verify-install command (Issue 12174)
Reduce log level for missing service account messages for a SPIFFE URI (Issue 12108)
Fix broken path on the opt-in SDS feature’s Unix domain socket (Issue 12688)
Fix Envoy tracing that was preventing a child span from being created if the parent span was propagated with an empty string (Envoy Issue 6263)
Add namespace scoping to the Gateway ‘port’ names. This fixes two issues:
- IngressGateway only respects first port 443 Gateway definition (Issue 11509)
- Istio IngressGateway routing broken with two different gateways with same port name (SDS) (Issue 12500)
Five bug fixes for locality weighted load balancing:
- Fix bug causing empty endpoints per locality (Issue 12610)
- Apply locality weighted load balancing configuration correctly (Issue 12587)
- Locality label istio-locality in Kubernetes should not contain /, use . (Issue 12582)
- Fix crash in locality load balancing (Issue 12649)
- Fix bug in locality load balancing normalization (Issue 12579)
Propagate Envoy Metrics Service configuration (Issue 12569)
Do not apply VirtualService rule to the wrong gateway (Issue 10313)

Announcing Istio 1.0.6

Tue, 12 Feb 2019 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.6. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Bug fixes

Fix Galley Helm charts so that the validatingwebhookconfiguration object can now deployed to a namespace other than istio-system (Issue 13625).
Additional Helm chart fixes for anti-affinity support: fix gatewaypodAntiAffinityRequiredDuringScheduling and podAntiAffinityLabelSelector match expressions and fix the default value for podAntiAffinityLabelSelector (Issue 13892).
Make Pilot handle a condition where Envoy continues to request routes for a deleted gateway while listeners are still draining (Issue 13739).

Small enhancements

If access logs are enabled, passthrough listener requests will be logged.
Make Pilot tolerate unknown JSON fields to make it easier to rollback to older versions during upgrade.
Add support for fallback secrets to SDS which Envoy can use instead of waiting indefinitely for late or non-existent secrets during startup (Issue 13853).

Announcing Istio 1.0.5

Thu, 20 Dec 2018 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.5. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

General

Disabled the precondition cache in the istio-policy service as it lead to invalid results. The cache will be reintroduced in a later release.
Mixer now only generates spans when there is an enabled tracespan adapter, resulting in lower CPU overhead in normal cases.
Fixed a problem that could lead Pilot to hang.

Announcing Istio 1.0.4

Wed, 21 Nov 2018 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.4. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Known issues

Pilot may deadlock when using istioctl proxy-status to get proxy synchronization status. The work around is to not use istioctl proxy-status. Once Pilot enters a deadlock, it exhibits continuous memory growth eventually running out of memory.

Networking

Fixed the lack of removal of stale endpoints causing 503 errors.
Fixed sidecar injection when a pod label contains a /.

Policy and telemetry

Fixed occasional data corruption problem with out-of-process Mixer adapters leading to incorrect behavior.
Fixed excessive CPU usage by Mixer when waiting for missing CRDs.

Announcing Istio 1.0.3

Tue, 30 Oct 2018 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.3. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Behavior changes

Validating webhook is now mandatory. Disabling it may result in Pilot crashes.
Service entry validation now rejects the wildcard hostname (*) when configuring DNS resolution. The API has never allowed this, however ServiceEntry was erroneously excluded from validation in the previous release. Use of wildcards as part of a hostname, e.g. *.bar.com, remains unchanged.
The core dump path for istio-proxy has changed to /var/lib/istio.

Networking

Mutual TLS Permissive mode is enabled by default.
Pilot performance and scalability has been greatly enhanced. Pilot now delivers endpoint updates to 500 sidecars in under 1 second.
Default trace sampling is set to 1%.

Policy and telemetry

Mixer (istio-telemetry) now supports load shedding based on request rate and expected latency.
Mixer client (istio-policy) now supports FAIL_OPEN setting.
Istio Performance dashboard added to Grafana.
Reduced istio-telemetry CPU usage by 10%.
Eliminated statsd-to-prometheus deployment. Prometheus now directly scrapes from istio-proxy.

Announcing Istio 1.0.2

Thu, 06 Sep 2018 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.2. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

General

Fixed bug in Envoy where the sidecar would crash if receiving normal traffic on the mutual TLS port.
Fixed bug with Pilot propagating incomplete updates to Envoy in a multicluster environment.
Added a few more Helm options for Grafana.
Improved Kubernetes service registry queue performance.
Fixed bug where istioctl proxy-status was not showing the patch version.
Add validation of virtual service SNI hosts.

Announcing Istio 1.0.1

Wed, 29 Aug 2018 00:00:00 +0000

We’re pleased to announce the availability of Istio 1.0.1. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

SOURCE CHANGES

Inspect the full set of source code changes.

Networking

Improved Pilot scalability and Envoy startup time.
Fixed virtual service host mismatch issue when adding a port.
Added limited support for merging multiple virtual service or destination rule definitions for the same host.
Allow outlier consecutive gateway failures when using HTTP.

Environment

Made it possible to use Pilot standalone, for those users who want to only leverage Istio’s traffic management functionality.
Introduced the convenient values-istio-gateway.yaml configuration that enables users to run standalone gateways.
Fixed a variety of Helm installation issues, including an issue with the istio-sidecar-injector configmap not being found.
Fixed the Istio installation error with Galley not being ready.
Fixed a variety of issues around mesh expansion.

Policy and telemetry

Added an experimental metrics expiration configuration to the Mixer Prometheus adapter.
Updated Grafana to 5.2.2.

Adapters

Ability to specify sink options for the Stackdriver adapter.

Galley

Improved configuration validation for health checks.

Announcing Istio 1.0

Tue, 31 Jul 2018 00:00:00 +0000

Today, we’re excited to announce Istio 1.0! It’s been a little over a year since our initial 0.1 release. Since then, Istio has evolved significantly with the help of a thriving and growing community of contributors and users. We’ve now reached the point where many companies have successfully adopted Istio in production and have gotten real value from the insight and control it provides over their deployments. We’ve helped large enterprises and fast-moving startups like eBay, Auto Trader UK, Descartes Labs, HP FitStation, JUSPAY, Namely, PubNub and Trulia use Istio to connect, manage and secure their services from the ground up. Shipping this release as 1.0 is recognition that we’ve built a core set of functionality that our users can rely on for production use.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

Ecosystem

We’ve seen substantial growth in Istio’s ecosystem in the last year. Envoy continues its impressive growth and added numerous features that are crucial for a production quality service mesh. Observability providers like Datadog, SolarWinds, Sysdig, Google Stackdriver and Amazon CloudWatch have written plugins to integrate Istio with their products. Tigera, Aporeto, Cilium and Styra built extensions to our policy enforcement and networking capabilities. Red Hat built Kiali to wrap a nice user-experience around mesh management and observability. Cloud Foundry is building on Istio for it’s next generation traffic routing stack, the recently announced Knative serverless project is doing the same and Apigee announced that they plan to use it in their API management solution. These are just some of the integrations the community has added in the last year.

Features

Since the 0.8 release we’ve added some important new features and more importantly marked many of our existing features as Beta signaling that they’re ready for production use. Here are some highlights:

Multiple Kubernetes clusters can now be added to a single mesh and enabling cross-cluster communication and consistent policy enforcement. Multicluster support is now Beta.
Networking APIs that enable fine grained control over the flow of traffic through a mesh are now Beta. Explicitly modeling ingress and egress concerns using Gateways allows operators to control the network topology and meet access security requirements at the edge.
Mutual TLS can now be rolled out incrementally without requiring all clients of a service to be updated. This is a critical feature that unblocks adoption in-place by existing production deployments.
Mixer now has support for developing out-of-process adapters. This will become the default way to extend Mixer over the coming releases and makes building adapters much simpler.
Authorization policies which control access to services are now entirely evaluated locally in Envoy increasing their performance and reliability.
Helm chart installation is now the recommended install method offering rich customization options to adopt Istio on your terms.
We’ve put a lot of effort into performance including continuous regression testing, large scale environment simulation and targeted fixes. We’re very happy with the results and will share more on this in detail in the coming weeks.

What’s next?

While this is a significant milestone for the project there’s lots more to do. In working with adopters we’ve gotten a lot of great feedback about what to focus next. We’ve heard consistent themes around support for hybrid-cloud, install modularity, richer networking features and scalability for massive deployments. We’ve already taken some of this feedback into account in the 1.0 release and we’ll continue to aggressively tackle this work in the coming months.

Getting started

If you’re new to Istio and looking to use it for your deployment we’d love to hear from you. Take a look at our docs or stop by our chat forum. If you’d like to go deeper and contribute to the project come to one of our community meetings and say hello.

Thanks

The Istio team would like to give huge thanks to everyone who has made a contribution to the project. It wouldn’t be where it is today without your help. The last year has been pretty amazing and we look forward to the next one with excitement about what we can achieve together as a community.

Release notes

Networking

SNI Routing using Virtual Services. Newly introduced TLS sections in VirtualService can be used to route TLS traffic based on SNI values. Service ports named as TLS/HTTPS can be used in conjunction with virtual service TLS routes. TLS/HTTPS ports without an accompanying virtual service will be treated as opaque TCP.
Streaming gRPC Restored. Istio 0.8 caused periodic termination of long running streaming gRPC connections. This has been fixed in 1.0.
Old (v1alpha1) Networking APIs Removed. Support for the old v1alpha1 traffic management model has been removed.
Istio Ingress Deprecated. The old Istio ingress is deprecated and disabled by default. We encourage users to use gateways instead.

Policy and telemetry

Updated Attributes. The set of attributes used to describe the source and destination of traffic have been completely revamped in order to be more precise and comprehensive.
Policy Check Cache. Mixer now features a large level 2 cache for policy checks, complementing the level 1 cache present in the sidecar proxy. This further reduces the average latency of externally-enforced policy checks.
Telemetry Buffering. Mixer now buffers report calls before dispatching to adapters, which gives an opportunity for adapters to process telemetry data in bigger chunks, reducing overall computational overhead in Mixer and its adapters.
Out of Process Adapters. Mixer now includes initial support for out-of-process adapters. This will be the recommended approach moving forward for integrating with Mixer. Initial documentation on how to build an out-of-process adapter is provided by the Out Of Process Adapter Dev Guide and the Out Of Process Adapter Walk-through.
Client-Side Telemetry. It’s now possible to collect telemetry from the client of an interaction, in addition to the server-side telemetry.

Adapters

SignalFX. There is a new signalfx adapter.
Stackdriver. The stackdriver adapter has been substantially enhanced in this release to add new features and improve performance.

Security

Authorization. We’ve reimplemented our authorization functionality. RPC-level authorization policies can now be implemented without the need for Mixer and Mixer adapters.
Improved Mutual TLS Authentication Control. It’s now easier to control mutual TLS authentication between services. We provide ‘PERMISSIVE’ mode so that you can incrementally turn on mutual TLS for your services. We removed service annotations and have a unique approach to turn on mutual TLS, coupled with client-side destination rules.
JWT Authentication. We now support JWT authentication which can be configured using authentication policies.

`istioctl`

Added the istioctl authn tls-check command.
Added the istioctl proxy-status command.
Added the istioctl experimental convert-ingress command.
Removed the istioctl experimental convert-networking-config command.
Enhancements and bug fixes:
- Align kubeconfig handling with kubectl
- istioctl get all returns all types of networking and authentication configuration.
- Added the --all-namespaces flag to istioctl get to retrieve resources across all namespaces.

Known issues with 1.0

Amazon’s EKS service does not implement automatic sidecar injection. Istio can be used in Amazon’s EKS by using manual injection for sidecars and turning off galley using the Helm parameter --set galley.enabled=false.
In a multicluster deployment the mixer-telemetry and mixer-policy components do not connect to the Kubernetes API endpoints of any of the remote clusters. This results in a loss of telemetry fidelity as some of the metadata associated with workloads on remote clusters is incomplete.
There are Kubernetes manifests available for using Citadel standalone or with Citadel health checking enabled. There is not a Helm implementation of these modes. See Issue 6922 for more details.
Mesh expansion functionality, which lets you add raw VMs to a mesh is broken in 1.0. We’re expecting to produce a patch that fixes this problem within a few days.

Announcing Istio 0.8

Fri, 01 Jun 2018 00:00:00 +0000

This is a major release for Istio on the road to 1.0. There are a great many new features and architectural improvements in addition to the usual pile of bug fixes and performance improvements.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

Networking

Revamped Traffic Management Model. We’re finally ready to take the wraps off our new traffic management APIs. We believe this new model is easier to understand while covering more real world deployment use-cases. For folks upgrading from earlier releases there is a migration guide and a conversion tool built into istioctl to help convert your configuration from the old model.
Streaming Envoy configuration. By default Pilot now streams configuration to Envoy using its ADS API. This new approach increases effective scalability, reduces rollout delay and should eliminate spurious 404 errors.
Gateway for Ingress/Egress. We no longer support combining Kubernetes Ingress specs with Istio routing rules as it has led to several bugs and reliability issues. Istio now supports a platform independent Gateway model for ingress & egress proxies that works across Kubernetes and Cloud Foundry and works seamlessly with routing. The Gateway supports Server Name Indication based routing, as well as serving a certificate based on the server name presented by the client.
Constrained Inbound Ports. We now restrict the inbound ports in a pod to the ones declared by the apps running inside that pod.

Security

Introducing Citadel. We’ve finally given a name to our security component. What was formerly known as Istio-Auth or Istio-CA is now called Citadel.
Multicluster Support. We support per-cluster Citadel in multicluster deployments such that all Citadels share the same root certificate and workloads can authenticate each other across the mesh.
Authentication Policy. We’ve created a unified API for authentication policy that controls whether service-to-service communication uses mutual TLS as well as end user authentication. This is now the recommended way to control these behaviors.

Telemetry

Self-Reporting. Mixer and Pilot now produce telemetry that flows through the normal Istio telemetry pipeline, just like services in the mesh.

Setup

A la Carte Istio. Istio has a rich set of features, however you don’t need to install or consume them all together. By using Helm or istioctl gen-deploy, users can install only the features they want. For example, users can install Pilot only and enjoy traffic management functionality without dealing with Mixer or Citadel.

Mixer adapters

CloudWatch. Mixer can now report metrics to AWS CloudWatch. Learn more

Known issues with 0.8

A gateway with virtual services pointing to a headless service won’t work (Issue #5005).
There is a problem with Google Kubernetes Engine 1.10.2. The workaround is to use Kubernetes 1.9 or switch the node image to Ubuntu. A fix is expected in GKE 1.10.4.
There is a known namespace issue with istioctl experimental convert-networking-config tool where the desired namespace may be changed to the istio-system namespace, please manually adjust to use the desired namespace after running the conversation tool. Learn more

Announcing Istio 0.7

Wed, 28 Mar 2018 00:00:00 +0000

For this release, we focused on improving our build and test infrastructures and increasing the quality of our tests. As a result, there are no new features for this month.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

Please note that this release includes preliminary support for the new v1alpha3 traffic management functionality. This functionality is still in a great deal of flux and there may be some breaking changes in 0.8. So if you feel like exploring, please go right ahead, but expect that this may change in 0.8 and beyond.

Known Issues:

Our Helm chart currently requires some workaround to apply the chart correctly, see 4701 for details.

Announcing Istio 0.6

Thu, 08 Mar 2018 00:00:00 +0000

In addition to the usual pile of bug fixes and performance improvements, this release includes the new or updated features detailed below.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

Networking

Custom Envoy Configuration. Pilot now supports ferrying custom Envoy configuration to the proxy. Learn more

Mixer adapters

SolarWinds. Mixer can now interface to AppOptics and Papertrail. Learn more
Redis Quota. Mixer now supports a Redis-based adapter for rate limit tracking. Learn more
Datadog. Mixer now provides an adapter to deliver metric data to a Datadog agent. Learn more

Other

Separate Check & Report Clusters. When configuring Envoy, it’s now possible to use different clusters for Mixer instances that are used for Mixer’s Check functionality from those used for Mixer’s Report functionality. This may be useful in large deployments for better scaling of Mixer instances.
Monitoring Dashboards. There are now preliminary Mixer & Pilot monitoring dashboard in Grafana.
Liveness and Readiness Probes. Istio components now provide canonical liveness and readiness probe support to help ensure mesh infrastructure health.
Egress Policy and Telemetry. Istio can monitor traffic to external services defined by EgressRule or External Service. Istio can also apply Mixer policies on this traffic.

Announcing Istio 0.5

Fri, 02 Feb 2018 00:00:00 +0000

In addition to the usual pile of bug fixes and performance improvements, this release includes the new or updated features detailed below.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

Networking

Incremental Istio Deployment. (Preview) You can now adopt Istio incrementally, more easily than before, by installing only the components you want (e.g, Pilot+Ingress only as the minimal Istio install). Refer to the istioctl CLI tool for generating a information on customized Istio deployments.
Automatic Proxy Injection. We leverage Kubernetes 1.9’s new mutating webhook feature to provide automatic pod-level proxy injection. Automatic injection requires Kubernetes 1.9 or beyond and therefore doesn’t work on older versions. The alpha initializer mechanism is no longer supported. Learn more
Revised Traffic Rules. Based on user feedback, we have made significant changes to Istio’s traffic management (routing rules, destination rules, etc.). We would love your continuing feedback while we polish this in the coming weeks.

Mixer adapters

Open Policy Agent. Mixer now has an authorization adapter implementing the open policy agent model, providing a flexible fine-grained access control mechanism. Learn more
Istio RBAC. Mixer now has a role-based access control adapter. Learn more
Fluentd. Mixer now has an adapter for log collection through Fluentd.
Stdio. The stdio adapter now lets you log to files with support for log rotation & backup, along with a host of controls.

Security

Bring Your Own CA. There have been many enhancements to the ‘bring your own CA’ feature. Learn more
PKCS8. Add support for PKCS8 keys to Istio PKI.
Istio RBAC Istio RBAC provides access control for services in Istio mesh. Learn more.

Other

Release-Mode Binaries. We switched release and installation default to release for improved performance and security.
Component Logging. Istio components now offer a rich set of command-line options to control local logging, including common support for log rotation.
Consistent Version Reporting. Istio components now offer a consistent command-line interface to report their version information.
Optional Instance Fields. Within configuration, definitions of Mixer instances no longer need to include every field of the associated template. Omitted fields get a zero or empty value.

Known issues

Installing with Helm charts is currently broken.
Automatic sidecar injection only works with Kubernetes 1.9 or later.

Announcing Istio 0.4

Mon, 18 Dec 2017 00:00:00 +0000

This release has only got a few weeks’ worth of changes, as we stabilize our monthly release process. In addition to the usual pile of bug fixes and performance improvements, this release includes the items below.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

General

Cloud Foundry. Added minimum Pilot support for the Cloud Foundry platform, making it possible for Pilot to discover CF services and service instances.
Circonus. Mixer now includes an adapter for the Circonus analytics and monitoring platform.
Pilot Metrics. Pilot now collects metrics for diagnostics.
Helm Charts. We now provide Helm charts to install Istio.
Enhanced Attribute Expressions. Mixer’s expression language gained a few new functions to make it easier to write policy rules. Learn more

If you’re into the nitty-gritty details, you can see our more detailed low-level release notes here.

Announcing Istio 0.3

Wed, 29 Nov 2017 00:00:00 +0000

We’re pleased to announce the availability of Istio 0.3. Please see below for what’s changed.

DOWNLOAD

Download and install this release.

DOCS

Visit the documentation for this release.

General

Starting with 0.3, Istio is switching to a monthly release cadence. We hope this will help accelerate our ability to deliver timely improvements. See here for information on the state of individual features for this release.

This is a fairly modest release in terms of new features as the team put emphasis on internal infrastructure work to improve our velocity. Many bugs and smaller issues have been addressed and overall performance has been improved in a number of areas.

Security

Secure Control Plane Communication. Mixer and Pilot are now secured with mutual TLS, just like all services in a mesh.
Selective Authentication. You can now control authentication on a per-service basis via service annotations, which helps with incremental migration to Istio.

Networking

Egress rules for TCP. You can now specify egress rules that affect TCP-level traffic.

Policy enforcement and telemetry

Improved Caching. Caching between Envoy and Mixer has gotten substantially better, resulting in a significant drop in average latency for authorization checks.
Improved list Adapter. The Mixer ‘list’ adapter now supports regular expression matching. See the adapter’s configuration options for details.
Configuration Validation. Mixer does more extensive validation of configuration state in order to catch problems earlier. We expect to invest more in this area in coming releases.

If you’re into the nitty-gritty details, you can see our more detailed low-level release notes here.

Announcing Istio 0.2

Tue, 10 Oct 2017 00:00:00 +0000

We launched Istio; an open platform to connect, manage, monitor, and secure microservices, on May 24, 2017. We have been humbled by the incredible interest, and rapid community growth of developers, operators, and partners. Our 0.1 release was focused on showing all the concepts of Istio in Kubernetes.

Today we are happy to announce the 0.2 release which improves stability and performance, allows for cluster wide deployment and automated injection of sidecars in Kubernetes, adds policy and authentication for TCP services, and enables expansion of the mesh to include services deployed in virtual machines. In addition, Istio can now run outside Kubernetes, leveraging Consul/Nomad or Eureka. Beyond core features, Istio is now ready for extensions to be written by third party companies and developers.

Highlights for the 0.2 release

Usability improvements

Multiple namespace support: Istio now works cluster-wide, across multiple namespaces and this was one of the top requests from community from 0.1 release.
Policy and security for TCP services: In addition to HTTP, we have added transparent mutual TLS authentication and policy enforcement for TCP services as well. This will allow you to secure more of your Kubernetes deployment, and get Istio features like telemetry, policy and security.
Automated sidecar injection: By leveraging the alpha initializer feature provided by Kubernetes 1.7, envoy sidecars can now be automatically injected into application deployments when your cluster has the initializer enabled. This enables you to deploy microservices using kubectl, the exact same command that you normally use for deploying the microservices without Istio.
Extending Istio: An improved Mixer design that lets vendors write Mixer adapters to implement support for their own systems, such as application management or policy enforcement. The Mixer Adapter Developer’s Guide can help you easily integrate your solution with Istio.
Bring your own CA certificates: Allows users to provide their own key and certificate for Istio CA and persistent CA key/certificate Storage. Enables storing signing key/certificates in persistent storage to facilitate CA restarts.
Improved routing & metrics: Support for WebSocket, MongoDB and Redis protocols. You can apply resilience features like circuit breakers on traffic to third party services. In addition to Mixer’s metrics, hundreds of metrics from Envoy are now visible inside Prometheus for all traffic entering, leaving and within Istio mesh.

Cross environment support

Mesh expansion: Istio mesh can now span services running outside of Kubernetes - like those running in virtual machines while enjoying benefits such as automatic mutual TLS authentication, traffic management, telemetry, and policy enforcement across the mesh.
Running outside Kubernetes: We know many customers use other service registry and orchestration solutions like Consul/Nomad and Eureka. Istio Pilot can now run standalone outside Kubernetes, consuming information from these systems, and manage the Envoy fleet in VMs or containers.

Get involved in shaping the future of Istio

We have a growing roadmap ahead of us, full of great features to implement. Our focus next release is going to be on stability, reliability, integration with third party tools and multicluster use cases.

To learn how to get involved and contribute to Istio’s future, check out our community GitHub repository which will introduce you to our working groups, our mailing lists, our various community meetings, our general procedures and our guidelines.

We want to thank our fantastic community for field testing new versions, filing bug reports, contributing code, helping out other community members, and shaping Istio by participating in countless productive discussions. This has enabled the project to accrue 3000 stars on GitHub since launch and hundreds of active community members on Istio mailing lists.

Thank you

Release notes

General

Updated Configuration Model. Istio now uses the Kubernetes Custom Resource model to describe and store its configuration. When running in Kubernetes, configuration can now be optionally managed using the kubectl command.
Multiple Namespace Support. Istio control plane components are now in the dedicated istio-system namespace. Istio can manage services in other non-system namespaces.
Mesh Expansion. Initial support for adding non-Kubernetes services (in the form of VMs and/or physical machines) to a mesh. This is an early version of this feature and has some limitations (such as requiring a flat network across containers and VMs).
Multi-Environment Support. Initial support for using Istio in conjunction with other service registries including Consul and Eureka.
Automatic injection of sidecars. Istio sidecar can automatically be injected into a pod upon deployment using the Initializers alpha feature in Kubernetes.

Performance and quality

There have been many performance and reliability improvements throughout the system. We don’t consider Istio 0.2 ready for production yet, but we’ve made excellent progress in that direction. Here are a few items of note:

Caching Client. The Mixer client library used by Envoy now provides caching for Check calls and batching for Report calls, considerably reducing end-to-end overhead.
Avoid Hot Restarts. The need to hot-restart Envoy has been mostly eliminated through effective use of LDS/RDS/CDS/EDS.
Reduced Memory Use. Significantly reduced the size of the sidecar helper agent, from 50Mb to 7Mb.
Improved Mixer Latency. Mixer now clearly delineates configuration-time vs. request-time computations, which avoids doing extra setup work at request-time for initial requests and thus delivers a smoother average latency. Better resource caching also contributes to better end-to-end performance.
Reduced Latency for Egress Traffic. We now forward traffic to external services directly from the sidecar.

Traffic management

Egress Rules. It’s now possible to specify routing rules for egress traffic.
New Protocols. Mesh-wide support for WebSocket connections, MongoDB proxying, and Kubernetes headless services.
Other Improvements. Ingress properly supports gRPC services, better support for health checks, and Jaeger tracing.

Policy enforcement & telemetry

Ingress Policies. In addition to east-west traffic supported in 0.1. policies can now be applied to north-south traffic.
Support for TCP Services. In addition to the HTTP-level policy controls available in 0.1, 0.2 introduces policy controls for TCP services.
New Mixer API. The API that Envoy uses to interact with Mixer has been completely redesigned for increased robustness, flexibility, and to support rich proxy-side caching and batching for increased performance.
New Mixer Adapter Model. A new adapter composition model makes it easier to extend Mixer by adding whole new classes of adapters via templates. This new model will serve as the foundational building block for many features in the future. See the Adapter Developer’s Guide to learn how to write adapters.
Improved Mixer Build Model. It’s now easier to build a Mixer binary that includes custom adapters.
Mixer Adapter Updates. The built-in adapters have all been rewritten to fit into the new adapter model. The stackdriver adapter has been added for this release. The experimental redisquota adapter has been removed in the 0.2 release, but is expected to come back in production quality for the 0.3 release.
Mixer Call Tracing. Calls between Envoy and Mixer can now be traced and analyzed in the Zipkin dashboard.

Security

Mutual TLS for TCP Traffic. In addition to HTTP traffic, mutual TLS is now supported for TCP traffic as well.
Identity Provisioning for VMs and Physical Machines. Auth supports a new mechanism using a per-node agent for identity provisioning. This agent runs on each node (VM / physical machine) and is responsible for generating and sending out the CSR (Certificate Signing Request) to get certificates from Istio CA.
Bring Your Own CA Certificates. Allows users to provide their own key and certificate for Istio CA.
Persistent CA Key/Certificate Storage. Istio CA now stores signing key/certificates in persistent storage to facilitate CA restarts.

Known issues

User may get periodical 404 when accessing the application: We have noticed that Envoy doesn’t get routes properly occasionally thus a 404 is returned to the user. We are actively working on this issue.
Istio Ingress or Egress reports ready before Pilot is actually ready: You can check the istio-ingress and istio-egress pods status in the istio-system namespace and wait a few seconds after all the Istio pods reach ready status. We are actively working on this issue.
A service with Istio Auth enabled can’t communicate with a service without Istio: This limitation will be removed in the near future.

Introducing Istio

Wed, 24 May 2017 00:00:00 +0000

Google, IBM, and Lyft are proud to announce the first public release of Istio: an open source project that provides a uniform way to connect, secure, manage and monitor microservices. Our current release is targeted at the Kubernetes environment; we intend to add support for other environments such as virtual machines and Cloud Foundry in the coming months. Istio adds traffic management to microservices and creates a basis for value-add capabilities like security, monitoring, routing, connectivity management and policy. The software is built using the battle-tested Envoy proxy from Lyft, and gives visibility and control over traffic without requiring any changes to application code. Istio gives CIOs a powerful tool to enforce security, policy and compliance requirements across the enterprise.

Background

Writing reliable, loosely coupled, production-grade applications based on microservices can be challenging. As monolithic applications are decomposed into microservices, software teams have to worry about the challenges inherent in integrating services in distributed systems: they must account for service discovery, load balancing, fault tolerance, end-to-end monitoring, dynamic routing for feature experimentation, and perhaps most important of all, compliance and security.

Inconsistent attempts at solving these challenges, cobbled together from libraries, scripts and Stack Overflow snippets leads to solutions that vary wildly across languages and runtimes, have poor observability characteristics and can often end up compromising security.

One solution is to standardize implementations on a common RPC library like gRPC, but this can be costly for organizations to adopt wholesale and leaves out brownfield applications which may be practically impossible to change. Operators need a flexible toolkit to make their microservices secure, compliant, trackable and highly available, and developers need the ability to experiment with different features in production, or deploy canary releases, without impacting the system as a whole.

Solution: service mesh

Imagine if we could transparently inject a layer of infrastructure between a service and the network that gives operators the controls they need while freeing developers from having to bake solutions to distributed system problems into their code. This uniform layer of infrastructure combined with service deployments is commonly referred to as a service mesh. Just as microservices help to decouple feature teams from each other, a service mesh helps to decouple operators from application feature development and release processes. Istio turns disparate microservices into an integrated service mesh by systemically injecting a proxy into the network paths among them.

Google, IBM and Lyft joined forces to create Istio from a desire to provide a reliable substrate for microservice development and maintenance, based on our common experiences building and operating massive scale microservices for internal and enterprise customers. Google and IBM have extensive experience with these large scale microservices in their own applications and with their enterprise customers in sensitive/regulated environments, while Lyft developed Envoy to address their internal operability challenges. Lyft open sourced Envoy after successfully using it in production for over a year to manage more than 100 services spanning 10,000 VMs, processing 2M requests/second.

Benefits of Istio

Fleet-wide Visibility: Failures happen, and operators need tools to stay on top of the health of clusters and their graphs of microservices. Istio produces detailed monitoring data about application and network behaviors that is rendered using Prometheus & Grafana, and can be easily extended to send metrics and logs to any collection, aggregation and querying system. Istio enables analysis of performance hotspots and diagnosis of distributed failure modes with Zipkin tracing.

Grafana Dashboard with Response Size

Zipkin Dashboard

Resiliency and efficiency: When developing microservices, operators need to assume that the network will be unreliable. Operators can use retries, load balancing, flow-control (HTTP/2), and circuit-breaking to compensate for some of the common failure modes due to an unreliable network. Istio provides a uniform approach to configuring these features, making it easier to operate a highly resilient service mesh.

Developer productivity: Istio provides a significant boost to developer productivity by letting them focus on building service features in their language of choice, while Istio handles resiliency and networking challenges in a uniform way. Developers are freed from having to bake solutions to distributed systems problems into their code. Istio further improves productivity by providing common functionality supporting A/B testing, canarying, and fault injection.

Policy Driven Ops: Istio empowers teams with different areas of concern to operate independently. It decouples cluster operators from the feature development cycle, allowing improvements to security, monitoring, scaling, and service topology to be rolled out without code changes. Operators can route a precise subset of production traffic to qualify a new service release. They can inject failures or delays into traffic to test the resilience of the service mesh, and set up rate limits to prevent services from being overloaded. Istio can also be used to enforce compliance rules, defining ACLs between services to allow only authorized services to talk to each other.

Secure by default: It is a common fallacy of distributed computing that the network is secure. Istio enables operators to authenticate and secure all communication between services using a mutual TLS connection, without burdening the developer or the operator with cumbersome certificate management tasks. Our security framework is aligned with the emerging SPIFFE specification, and is based on similar systems that have been tested extensively inside Google.

Incremental Adoption: We designed Istio to be completely transparent to the services running in the mesh, allowing teams to incrementally adopt features of Istio over time. Adopters can start with enabling fleet-wide visibility and once they’re comfortable with Istio in their environment they can switch on other features as needed.

Join us in this journey

Istio is a completely open development project. Today we are releasing version 0.1, which works in a Kubernetes cluster, and we plan to have major new releases every 3 months, including support for additional environments. Our goal is to enable developers and operators to rollout and operate microservices with agility, complete visibility of the underlying network, and uniform control and security in all environments. We look forward to working with the Istio community and our partners towards these goals, following our roadmap.

Visit here to get the latest released bits.

View the presentation from GlueCon 2017, where Istio was unveiled.

Community

We are excited to see early commitment to support the project from many companies in the community: Red Hat with Red Hat OpenShift and OpenShift Application Runtimes, Pivotal with Pivotal Cloud Foundry, WeaveWorks with Weave Cloud and Weave Net 2.0, Tigera with the Project Calico Network Policy Engine and Datawire with the Ambassador project. We hope to see many more companies join us in this journey.

To get involved, connect with us via any of these channels:

[istio.io]() for documentation and examples.
The Istio discussion board general discussions,
Stack Overflow for curated questions and answers
GitHub for filing issues
@IstioMesh on Twitter

From everyone working on Istio, welcome aboard!

Release notes

Installation of Istio into a Kubernetes namespace with a single command.
Semi-automated injection of Envoy proxies into Kubernetes pods.
Automatic traffic capture for Kubernetes pods using iptables.
In-cluster load balancing for HTTP, gRPC, and TCP traffic.
Support for timeouts, retries with budgets, and circuit breakers.
Istio-integrated Kubernetes Ingress support (Istio acts as an Ingress Controller).
Fine-grained traffic routing controls, including A/B testing, canarying, red/black deployments.
Flexible in-memory rate limiting.
L7 telemetry and logging for HTTP and gRPC using Prometheus.
Grafana dashboards showing per-service L7 metrics.
Request tracing through Envoy with Zipkin.
Service-to-service authentication using mutual TLS.
Simple service-to-service authorization using deny expressions.

Helm Changes

Mon, 01 Jan 0001 00:00:00 +0000

The tables below show changes made to the installation options used to customize Istio install using Helm between Istio 1.2 and Istio 1.3. The tables are grouped in to three different categories:

The installation options already in the previous release but whose values have been modified in the new release.
The new installation options added in the new release.
The installation options removed from the new release.

Modified configuration options

Modified `kiali` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`kiali.tag`	`v0.20`	`v1.1.0`

Modified `global` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`global.tag`	`1.2.0-rc.3`	`release-1.3-latest-daily`	`Default tag for Istio images.`	`Default tag for Istio images.`

Modified `gateways` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`gateways.istio-egressgateway.resources.limits.memory`	`256Mi`	`1024Mi`

Modified `tracing` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`tracing.jaeger.tag`	`1.9`	`1.12`
`tracing.zipkin.tag`	`2`	`2.14.2`

New configuration options

New `tracing` key/value pairs

Key	Default Value	Description
`tracing.tolerations`	`[]`
`tracing.jaeger.image`	`all-in-one`
`tracing.jaeger.spanStorageType`	`badger`	`spanStorageType value can be "memory" and "badger" for all-in-one image`
`tracing.jaeger.persist`	`false`
`tracing.jaeger.storageClassName`	`""`
`tracing.jaeger.accessMode`	`ReadWriteMany`
`tracing.zipkin.image`	`zipkin`

New `sidecarInjectorWebhook` key/value pairs

Key	Default Value	Description
`sidecarInjectorWebhook.rollingMaxSurge`	`100%`
`sidecarInjectorWebhook.rollingMaxUnavailable`	`25%`
`sidecarInjectorWebhook.tolerations`	`[]`

New `global` key/value pairs

Key	Default Value	Description
`global.proxy.init.resources.limits.cpu`	`100m`
`global.proxy.init.resources.limits.memory`	`50Mi`
`global.proxy.init.resources.requests.cpu`	`10m`
`global.proxy.init.resources.requests.memory`	`10Mi`
`global.proxy.envoyAccessLogService.enabled`	`false`
`global.proxy.envoyAccessLogService.host`	``	`example: accesslog-service.istio-system`
`global.proxy.envoyAccessLogService.port`	``	`example: 15000`
`global.proxy.envoyAccessLogService.tlsSettings.mode`	`DISABLE`	`DISABLE, SIMPLE, MUTUAL, ISTIO_MUTUAL`
`global.proxy.envoyAccessLogService.tlsSettings.clientCertificate`	``	`example: /etc/istio/als/cert-chain.pem`
`global.proxy.envoyAccessLogService.tlsSettings.privateKey`	``	`example: /etc/istio/als/key.pem`
`global.proxy.envoyAccessLogService.tlsSettings.caCertificates`	``	`example: /etc/istio/als/root-cert.pem`
`global.proxy.envoyAccessLogService.tlsSettings.sni`	``	`example: als.somedomain`
`global.proxy.envoyAccessLogService.tlsSettings.subjectAltNames`	`[]`
`global.proxy.envoyAccessLogService.tcpKeepalive.probes`	`3`
`global.proxy.envoyAccessLogService.tcpKeepalive.time`	`10s`
`global.proxy.envoyAccessLogService.tcpKeepalive.interval`	`10s`
`global.proxy.protocolDetectionTimeout`	`10ms`	Automatic protocol detection uses a set of heuristics to determine whether the connection is using TLS or not (on the server side), as well as the application protocol being used (e.g., http vs tcp). These heuristics rely on the client sending the first bits of data. For server first protocols like MySQL, MongoDB, etc., Envoy will timeout on the protocol detection after the specified period, defaulting to non mTLS plain TCP traffic. Set this field to tweak the period that Envoy will wait for the client to send the first bits of data. (MUST BE >=1ms)
`global.proxy.enableCoreDumpImage`	`ubuntu:xenial`	`Image used to enable core dumps. This is only used, when "enableCoreDump" is set to true.`
`global.defaultTolerations`	`[]`	`Default node tolerations to be applied to all deployments so that all pods can be scheduled to a particular nodes with matching taints. Each component can overwrite these default values by adding its tolerations block in the relevant section below and setting the desired values. Configure this field in case that all pods of Istio control plane are expected to be scheduled to particular nodes with specified taints.`
`global.meshID`	`""`	Mesh ID means Mesh Identifier. It should be unique within the scope where meshes will interact with each other, but it is not required to be globally/universally unique. For example, if any of the following are true, then two meshes must have different Mesh IDs: - Meshes will have their telemetry aggregated in one place - Meshes will be federated together - Policy will be written referencing one mesh from the other If an administrator expects that any of these conditions may become true in the future, they should ensure their meshes have different Mesh IDs assigned. Within a multicluster mesh, each cluster must be (manually or auto) configured to have the same Mesh ID value. If an existing cluster 'joins' a multicluster mesh, it will need to be migrated to the new mesh ID. Details of migration TBD, and it may be a disruptive operation to change the Mesh ID post-install. If the mesh admin does not specify a value, Istio will use the value of the mesh's Trust Domain. The best practice is to select a proper Trust Domain value.
`global.localityLbSetting.enabled`	`true`

New `galley` key/value pairs

Key	Default Value	Description
`galley.rollingMaxSurge`	`100%`
`galley.rollingMaxUnavailable`	`25%`

New `mixer` key/value pairs

Key	Default Value	Description
`mixer.policy.rollingMaxSurge`	`100%`
`mixer.policy.rollingMaxUnavailable`	`25%`
`mixer.telemetry.rollingMaxSurge`	`100%`
`mixer.telemetry.rollingMaxUnavailable`	`25%`
`mixer.telemetry.reportBatchMaxEntries`	`100`	`Set reportBatchMaxEntries to 0 to use the default batching behavior (i.e., every 100 requests). A positive value indicates the number of requests that are batched before telemetry data is sent to the mixer server`
`mixer.telemetry.reportBatchMaxTime`	`1s`	`Set reportBatchMaxTime to 0 to use the default batching behavior (i.e., every 1 second). A positive time value indicates the maximum wait time since the last request will telemetry data be batched before being sent to the mixer server`

New `grafana` key/value pairs

Key	Default Value	Description
`grafana.env`	`{}`
`grafana.envSecrets`	`{}`
`grafana.datasources.datasources.datasources.type.orgId`	`1`
`grafana.datasources.datasources.datasources.type.url`	`http://prometheus:9090`
`grafana.datasources.datasources.datasources.type.access`	`proxy`
`grafana.datasources.datasources.datasources.type.isDefault`	`true`
`grafana.datasources.datasources.datasources.type.jsonData.timeInterval`	`5s`
`grafana.datasources.datasources.datasources.type.editable`	`true`
`grafana.dashboardProviders.dashboardproviders.providers.orgId.folder`	`'istio'`
`grafana.dashboardProviders.dashboardproviders.providers.orgId.type`	`file`
`grafana.dashboardProviders.dashboardproviders.providers.orgId.disableDeletion`	`false`
`grafana.dashboardProviders.dashboardproviders.providers.orgId.options.path`	`/var/lib/grafana/dashboards/istio`

New `prometheus` key/value pairs

Key	Default Value	Description
`prometheus.image`	`prometheus`

New `gateways` key/value pairs

Key	Default Value	Description
`gateways.istio-ingressgateway.rollingMaxSurge`	`100%`
`gateways.istio-ingressgateway.rollingMaxUnavailable`	`25%`
`gateways.istio-egressgateway.rollingMaxSurge`	`100%`
`gateways.istio-egressgateway.rollingMaxUnavailable`	`25%`
`gateways.istio-ilbgateway.rollingMaxSurge`	`100%`
`gateways.istio-ilbgateway.rollingMaxUnavailable`	`25%`

New `certmanager` key/value pairs

Key	Default Value	Description
`certmanager.image`	`cert-manager-controller`

New `kiali` key/value pairs

Key	Default Value	Description
`kiali.image`	`kiali`
`kiali.tolerations`	`[]`
`kiali.dashboard.auth.strategy`	`login`	`Can be anonymous, login, or openshift`
`kiali.security.enabled`	`true`
`kiali.security.cert_file`	`/kiali-cert/cert-chain.pem`
`kiali.security.private_key_file`	`/kiali-cert/key.pem`

New `istiocoredns` key/value pairs

Key	Default Value	Description
`istiocoredns.rollingMaxSurge`	`100%`
`istiocoredns.rollingMaxUnavailable`	`25%`

New `security` key/value pairs

Key	Default Value	Description
`security.replicaCount`	`1`
`security.rollingMaxSurge`	`100%`
`security.rollingMaxUnavailable`	`25%`
`security.workloadCertTtl`	`2160h`	`90*24hour = 2160h`
`security.enableNamespacesByDefault`	`true`	Determines Citadel default behavior if the ca.istio.io/env or ca.istio.io/override labels are not found on a given namespace. For example: consider a namespace called "target", which has neither the "ca.istio.io/env" nor the "ca.istio.io/override" namespace labels. To decide whether or not to generate secrets for service accounts created in this "target" namespace, Citadel will defer to this option. If the value of this option is "true" in this case, secrets will be generated for the "target" namespace. If the value of this option is "false" Citadel will not generate secrets upon service account creation.

New `pilot` key/value pairs

Key	Default Value	Description
`pilot.rollingMaxSurge`	`100%`
`pilot.rollingMaxUnavailable`	`25%`
`pilot.enableProtocolSniffing`	`false`	`if protocol sniffing is enabled. Default to false.`

Removed configuration options

Removed `global` key/value pairs

Key	Default Value	Description
`global.sds.useTrustworthyJwt`	`false`
`global.sds.useNormalJwt`	`false`
`global.localityLbSetting`	`{}`

Removed `mixer` key/value pairs

Key	Default Value	Description
`mixer.templates.useTemplateCRDs`	`false`

Removed `grafana` key/value pairs

Key	Default Value	Description
`grafana.dashboardProviders.dashboardproviders.providers.disableDeletion`	`false`
`grafana.dashboardProviders.dashboardproviders.providers.type`	`file`
`grafana.dashboardProviders.dashboardproviders.providers.folder`	`'istio'`
`grafana.datasources.datasources.datasources.isDefault`	`true`
`grafana.datasources.datasources.datasources.url`	`http://prometheus:9090`
`grafana.datasources.datasources.datasources.access`	`proxy`
`grafana.datasources.datasources.datasources.jsonData.timeInterval`	`5s`
`grafana.dashboardProviders.dashboardproviders.providers.options.path`	`/var/lib/grafana/dashboards/istio`
`grafana.datasources.datasources.datasources.editable`	`true`
`grafana.datasources.datasources.datasources.orgId`	`1`

Helm Changes

Mon, 01 Jan 0001 00:00:00 +0000

The tables below show changes made to the installation options used to customize Istio install using Helm between Istio 1.1 and Istio 1.2. The tables are grouped in to three different categories:

The installation options already in the previous release but whose values have been modified in the new release.
The new installation options added in the new release.
The installation options removed from the new release.

Modified configuration options

Modified `kiali` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`kiali.hub`	`docker.io/kiali`	`quay.io/kiali`
`kiali.tag`	`v0.14`	`v0.20`

Modified `prometheus` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`prometheus.tag`	`v2.3.1`	`v2.8.0`

Modified `global` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`global.tag`	`release-1.1-latest-daily`	`1.2.0-rc.3`	`Default tag for Istio images.`	`Default tag for Istio images.`
`global.proxy.resources.limits.memory`	`128Mi`	`1024Mi`
`global.proxy.dnsRefreshRate`	`5s`	`300s`	`Configure the DNS refresh rate for Envoy cluster of type STRICT_DNS 5 seconds is the default refresh rate used by Envoy`	`Configure the DNS refresh rate for Envoy cluster of type STRICT_DNS This must be given it terms of seconds. For example, 300s is valid but 5m is invalid.`

Modified `mixer` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`mixer.adapters.useAdapterCRDs`	`true`	`false`	`Setting this to false sets the useAdapterCRDs mixer startup argument to false`	`Setting this to false sets the useAdapterCRDs mixer startup argument to false`

Modified `grafana` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`grafana.image.tag`	`5.4.0`	`6.1.6`

New configuration options

New `tracing` key/value pairs

Key	Default Value	Description
`tracing.podAntiAffinityLabelSelector`	`[]`
`tracing.podAntiAffinityTermLabelSelector`	`[]`

New `sidecarInjectorWebhook` key/value pairs

Key	Default Value	Description
`sidecarInjectorWebhook.podAntiAffinityLabelSelector`	`[]`
`sidecarInjectorWebhook.podAntiAffinityTermLabelSelector`	`[]`
`sidecarInjectorWebhook.neverInjectSelector`	`[]`	`You can use the field called alwaysInjectSelector and neverInjectSelector which will always inject the sidecar or always skip the injection on pods that match that label selector, regardless of the global policy. See https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/more-control-adding-exceptions`
`sidecarInjectorWebhook.alwaysInjectSelector`	`[]`

New `global` key/value pairs

Key	Default Value	Description
`global.logging.level`	`"default:info"`
`global.proxy.logLevel`	`""`	`Log level for proxy, applies to gateways and sidecars. If left empty, "warning" is used. Expected values are: trace\\|debug\\|info\\|warning\\|error\\|critical\\|off`
`global.proxy.componentLogLevel`	`""`	`Per Component log level for proxy, applies to gateways and sidecars. If a component level is not set, then the global "logLevel" will be used. If left empty, "misc:error" is used.`
`global.proxy.excludeOutboundPorts`	`""`
`global.tracer.datadog.address`	`"$(HOST_IP):8126"`
`global.imagePullSecrets`	`[]`	`Lists the secrets you need to use to pull Istio images from a secure registry.`
`global.localityLbSetting`	`{}`

New `galley` key/value pairs

Key	Default Value	Description
`galley.nodeSelector`	`{}`
`galley.tolerations`	`[]`
`galley.podAntiAffinityLabelSelector`	`[]`
`galley.podAntiAffinityTermLabelSelector`	`[]`

New `mixer` key/value pairs

Key	Default Value	Description
`mixer.tolerations`	`[]`
`mixer.podAntiAffinityLabelSelector`	`[]`
`mixer.podAntiAffinityTermLabelSelector`	`[]`
`mixer.templates.useTemplateCRDs`	`false`

New `grafana` key/value pairs

Key	Default Value	Description
`grafana.tolerations`	`[]`
`grafana.podAntiAffinityLabelSelector`	`[]`
`grafana.podAntiAffinityTermLabelSelector`	`[]`

New `prometheus` key/value pairs

Key	Default Value	Description
`prometheus.tolerations`	`[]`
`prometheus.podAntiAffinityLabelSelector`	`[]`
`prometheus.podAntiAffinityTermLabelSelector`	`[]`

New `gateways` key/value pairs

Key	Default Value	Description
`gateways.istio-ingressgateway.sds.resources.requests.cpu`	`100m`
`gateways.istio-ingressgateway.sds.resources.requests.memory`	`128Mi`
`gateways.istio-ingressgateway.sds.resources.limits.cpu`	`2000m`
`gateways.istio-ingressgateway.sds.resources.limits.memory`	`1024Mi`
`gateways.istio-ingressgateway.resources.requests.cpu`	`100m`
`gateways.istio-ingressgateway.resources.requests.memory`	`128Mi`
`gateways.istio-ingressgateway.resources.limits.cpu`	`2000m`
`gateways.istio-ingressgateway.resources.limits.memory`	`1024Mi`
`gateways.istio-ingressgateway.applicationPorts`	`""`
`gateways.istio-ingressgateway.tolerations`	`[]`
`gateways.istio-ingressgateway.podAntiAffinityLabelSelector`	`[]`
`gateways.istio-ingressgateway.podAntiAffinityTermLabelSelector`	`[]`
`gateways.istio-egressgateway.resources.requests.cpu`	`100m`
`gateways.istio-egressgateway.resources.requests.memory`	`128Mi`
`gateways.istio-egressgateway.resources.limits.cpu`	`2000m`
`gateways.istio-egressgateway.resources.limits.memory`	`256Mi`
`gateways.istio-egressgateway.tolerations`	`[]`
`gateways.istio-egressgateway.podAntiAffinityLabelSelector`	`[]`
`gateways.istio-egressgateway.podAntiAffinityTermLabelSelector`	`[]`
`gateways.istio-ilbgateway.tolerations`	`[]`

New `certmanager` key/value pairs

Key	Default Value	Description
`certmanager.replicaCount`	`1`
`certmanager.nodeSelector`	`{}`
`certmanager.tolerations`	`[]`
`certmanager.podAntiAffinityLabelSelector`	`[]`
`certmanager.podAntiAffinityTermLabelSelector`	`[]`

New `kiali` key/value pairs

Key	Default Value	Description
`kiali.podAntiAffinityLabelSelector`	`[]`
`kiali.podAntiAffinityTermLabelSelector`	`[]`
`kiali.dashboard.viewOnlyMode`	`false`	`Bind the service account to a role with only read access`

New `istiocoredns` key/value pairs

Key	Default Value	Description
`istiocoredns.tolerations`	`[]`
`istiocoredns.podAntiAffinityLabelSelector`	`[]`
`istiocoredns.podAntiAffinityTermLabelSelector`	`[]`

New `security` key/value pairs

Key	Default Value	Description
`security.tolerations`	`[]`
`security.citadelHealthCheck`	`false`
`security.podAntiAffinityLabelSelector`	`[]`
`security.podAntiAffinityTermLabelSelector`	`[]`

New `nodeagent` key/value pairs

Key	Default Value	Description
`nodeagent.tolerations`	`[]`
`nodeagent.podAntiAffinityLabelSelector`	`[]`
`nodeagent.podAntiAffinityTermLabelSelector`	`[]`

New `pilot` key/value pairs

Key	Default Value	Description
`pilot.tolerations`	`[]`
`pilot.podAntiAffinityLabelSelector`	`[]`
`pilot.podAntiAffinityTermLabelSelector`	`[]`

Removed configuration options

Removed `kiali` key/value pairs

Key	Default Value	Description
`kiali.dashboard.usernameKey`	`username`	`This is the key name within the secret whose value is the actual username.`
`kiali.dashboard.passphraseKey`	`passphrase`	`This is the key name within the secret whose value is the actual passphrase.`

Removed `security` key/value pairs

Key	Default Value	Description
`security.replicaCount`	`1`

Removed `gateways` key/value pairs

Key	Default Value	Description
`gateways.istio-ingressgateway.resources`	`{}`

Removed `mixer` key/value pairs

Key	Default Value	Description
`mixer.enabled`	`true`

Removed `servicegraph` key/value pairs

Key	Default Value	Description
`servicegraph.ingress.enabled`	`false`
`servicegraph.service.name`	`http`
`servicegraph.replicaCount`	`1`
`servicegraph.service.type`	`ClusterIP`
`servicegraph.service.annotations`	`{}`
`servicegraph.enabled`	`false`
`servicegraph.image`	`servicegraph`
`servicegraph.service.externalPort`	`8088`
`servicegraph.ingress.hosts`	`servicegraph.local`	`Used to create an Ingress record.`
`servicegraph.nodeSelector`	`{}`
`servicegraph.prometheusAddr`	`http://prometheus:9090`

Helm Changes

Mon, 01 Jan 0001 00:00:00 +0000

The tables below show changes made to the installation options used to customize Istio install using Helm between Istio 1.0 and Istio 1.1. The tables are grouped in to three different categories:

The installation options already in the previous release but whose values or descriptions have been modified in the new release.
The new installation options added in the new release.
The installation options removed from the new release.

Modified configuration options

Modified `servicegraph` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`servicegraph.ingress.hosts`	`servicegraph.local`	`servicegraph.local`		`Used to create an Ingress record.`

Modified `tracing` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`tracing.jaeger.tag`	`1.5`	`1.9`

Modified `global` key/value pairs

Key	Old Default Value	New Default Value	New Description
`global.hub`	`gcr.io/istio-release`	`gcr.io/istio-release`	`Default hub for Istio images.Releases are published to docker hub under 'istio' project.Daily builds from prow are on gcr.io, and nightly builds from circle on docker.io/istionightly`
`global.tag`	`release-1.0-latest-daily`	`release-1.1-latest-daily`	`Default tag for Istio images.`
`global.proxy.resources.requests.cpu`	`10m`	`100m`
`global.proxy.accessLogFile`	`"/dev/stdout"`	`""`
`global.proxy.enableCoreDump`	`false`	`false`	`If set, newly injected sidecars will have core dumps enabled.`
`global.proxy.autoInject`	`enabled`	`enabled`	`This controls the 'policy' in the sidecar injector.`
`global.proxy.envoyStatsd.enabled`	`true`	`false`	`If enabled is set to true, host and port must also be provided. Istio no longer provides a statsd collector.`
`global.proxy.envoyStatsd.host`	`istio-statsd-prom-bridge`	``	`example: statsd-svc.istio-system`
`global.proxy.envoyStatsd.port`	`9125`	``	`example: 9125`
`global.proxy_init.image`	`proxy_init`	`proxy_init`	`Base name for the proxy_init container, used to configure iptables.`
`global.controlPlaneSecurityEnabled`	`false`	`false`	`controlPlaneMtls enabled. Will result in delays starting the pods while secrets arepropagated, not recommended for tests.`
`global.disablePolicyChecks`	`false`	`true`	`disablePolicyChecks disables mixer policy checks.if mixer.policy.enabled==true then disablePolicyChecks has affect.Will set the value with same name in istio config map - pilot needs to be restarted to take effect.`
`global.enableTracing`	`true`	`true`	`EnableTracing sets the value with same name in istio config map, requires pilot restart to take effect.`
`global.mtls.enabled`	`false`	`false`	`Default setting for service-to-service mtls. Can be set explicitly usingdestination rules or service annotations.`
`global.oneNamespace`	`false`	`false`	`Whether to restrict the applications namespace the controller manages;If not set, controller watches all namespaces`
`global.configValidation`	`true`	`true`	`Whether to perform server-side validation of configuration.`

Modified `gateways` key/value pairs

Key	Old Default Value	New Default Value	New Description
`gateways.istio-ingressgateway.type`	`LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be`	`LoadBalancer`	`change to NodePort, ClusterIP or LoadBalancer if need be`
`gateways.istio-egressgateway.enabled`	`true`	`false`
`gateways.istio-egressgateway.type`	`ClusterIP #change to NodePort or LoadBalancer if need be`	`ClusterIP`	`change to NodePort or LoadBalancer if need be`

Modified `certmanager` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`certmanager.tag`	`v0.3.1`	`v0.6.2`

Modified `kiali` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`kiali.tag`	`istio-release-1.0`	`v0.14`

Modified `security` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`security.selfSigned`	`true # indicate if self-signed CA is used.`	`true`		`indicate if self-signed CA is used.`

Modified `pilot` key/value pairs

Key	Old Default Value	New Default Value	Old Description	New Description
`pilot.autoscaleMax`	`1`	`5`
`pilot.traceSampling`	`100.0`	`1.0`

New configuration options

New `istio_cni` key/value pairs

Key	Default Value	Description
`istio_cni.enabled`	`false`

New `servicegraph` key/value pairs

Key	Default Value	Description
`servicegraph.nodeSelector`	`{}`

New `tracing` key/value pairs

Key	Default Value	Description
`tracing.nodeSelector`	`{}`
`tracing.zipkin.hub`	`docker.io/openzipkin`
`tracing.zipkin.tag`	`2`
`tracing.zipkin.probeStartupDelay`	`200`
`tracing.zipkin.queryPort`	`9411`
`tracing.zipkin.resources.limits.cpu`	`300m`
`tracing.zipkin.resources.limits.memory`	`900Mi`
`tracing.zipkin.resources.requests.cpu`	`150m`
`tracing.zipkin.resources.requests.memory`	`900Mi`
`tracing.zipkin.javaOptsHeap`	`700`
`tracing.zipkin.maxSpans`	`500000`
`tracing.zipkin.node.cpus`	`2`

New `sidecarInjectorWebhook` key/value pairs

Key	Default Value	Description
`sidecarInjectorWebhook.nodeSelector`	`{}`
`sidecarInjectorWebhook.rewriteAppHTTPProbe`	`false`	`If true, webhook or istioctl injector will rewrite PodSpec for livenesshealth check to redirect request to sidecar. This makes liveness check workeven when mTLS is enabled.`

New `global` key/value pairs

Key	Default Value	Description
`global.monitoringPort`	`15014`	`monitoring port used by mixer, pilot, galley`
`global.k8sIngress.enabled`	`false`
`global.k8sIngress.gatewayName`	`ingressgateway`	`Gateway used for k8s Ingress resources. By default it isusing 'istio:ingressgateway' that will be installed by setting'gateways.enabled' and 'gateways.istio-ingressgateway.enabled'flags to true.`
`global.k8sIngress.enableHttps`	`false`	`enableHttps will add port 443 on the ingress.It REQUIRES that the certificates are installed in theexpected secrets - enabling this option without certificateswill result in LDS rejection and the ingress will not work.`
`global.proxy.clusterDomain`	`"cluster.local"`	`cluster domain. Default value is "cluster.local".`
`global.proxy.resources.requests.memory`	`128Mi`
`global.proxy.resources.limits.cpu`	`2000m`
`global.proxy.resources.limits.memory`	`128Mi`
`global.proxy.concurrency`	`2`	`Controls number of Proxy worker threads.If set to 0 (default), then start worker thread for each CPU thread/core.`
`global.proxy.accessLogFormat`	`""`	`Configure how and what fields are displayed in sidecar access log. Setting toempty string will result in default log format`
`global.proxy.accessLogEncoding`	`TEXT`	`Configure the access log for sidecar to JSON or TEXT.`
`global.proxy.dnsRefreshRate`	`5s`	`Configure the DNS refresh rate for Envoy cluster of type STRICT_DNS5 seconds is the default refresh rate used by Envoy`
`global.proxy.privileged`	`false`	`If set to true, istio-proxy container will have privileged securityContext`
`global.proxy.statusPort`	`15020`	`Default port for Pilot agent health checks. A value of 0 will disable health checking.`
`global.proxy.readinessInitialDelaySeconds`	`1`	`The initial delay for readiness probes in seconds.`
`global.proxy.readinessPeriodSeconds`	`2`	`The period between readiness probes.`
`global.proxy.readinessFailureThreshold`	`30`	`The number of successive failed probes before indicating readiness failure.`
`global.proxy.kubevirtInterfaces`	`""`	`pod internal interfaces`
`global.proxy.envoyMetricsService.enabled`	`false`
`global.proxy.envoyMetricsService.host`	``	`example: metrics-service.istio-system`
`global.proxy.envoyMetricsService.port`	``	`example: 15000`
`global.proxy.tracer`	`"zipkin"`	`Specify which tracer to use. One of: lightstep, zipkin`
`global.policyCheckFailOpen`	`false`	`policyCheckFailOpen allows traffic in cases when the mixer policy service cannot be reached.Default is false which means the traffic is denied when the client is unable to connect to Mixer.`
`global.tracer.lightstep.address`	`""`	`example: lightstep-satellite:443`
`global.tracer.lightstep.accessToken`	`""`	`example: abcdefg1234567`
`global.tracer.lightstep.secure`	`true`	`example: true\\|false`
`global.tracer.lightstep.cacertPath`	`""`	`example: /etc/lightstep/cacert.pem`
`global.tracer.zipkin.address`	`""`
`global.defaultNodeSelector`	`{}`	`Default node selector to be applied to all deployments so that all pods can beconstrained to run a particular nodes. Each component can overwrite these defaultvalues by adding its node selector block in the relevant section below and settingthe desired values.`
`global.meshExpansion.enabled`	`false`
`global.meshExpansion.useILB`	`false`	`If set to true, the pilot and citadel mtls and the plain text pilot portswill be exposed on an internal gateway`
`global.multiCluster.enabled`	`false`	`Set to true to connect two kubernetes clusters via their respectiveingressgateway services when pods in each cluster cannot directlytalk to one another. All clusters should be using Istio mTLS and musthave a shared root CA for this model to work.`
`global.defaultPodDisruptionBudget.enabled`	`true`
`global.useMCP`	`true`	`Use the Mesh Control Protocol (MCP) for configuring Mixer andPilot. Requires galley (--set galley.enabled=true).`
`global.trustDomain`	`""`
`global.outboundTrafficPolicy.mode`	`ALLOW_ANY`
`global.sds.enabled`	`false`	`SDS enabled. IF set to true, mTLS certificates for the sidecars will bedistributed through the SecretDiscoveryService instead of using K8S secrets to mount the certificates.`
`global.sds.udsPath`	`""`
`global.sds.useTrustworthyJwt`	`false`
`global.sds.useNormalJwt`	`false`
`global.meshNetworks`	`{}`
`global.enableHelmTest`	`false`	`Specifies whether helm test is enabled or not.This field is set to false by default, so 'helm template ...'will ignore the helm test yaml files when generating the template`

New `mixer` key/value pairs

Key	Default Value	Description
`mixer.env.GODEBUG`	`gctrace=1`
`mixer.env.GOMAXPROCS`	`"6"`	`max procs should be ceil(cpu limit + 1)`
`mixer.policy.enabled`	`false`	`if policy is enabled, global.disablePolicyChecks has affect.`
`mixer.policy.replicaCount`	`1`
`mixer.policy.autoscaleEnabled`	`true`
`mixer.policy.autoscaleMin`	`1`
`mixer.policy.autoscaleMax`	`5`
`mixer.policy.cpu.targetAverageUtilization`	`80`
`mixer.telemetry.enabled`	`true`
`mixer.telemetry.replicaCount`	`1`
`mixer.telemetry.autoscaleEnabled`	`true`
`mixer.telemetry.autoscaleMin`	`1`
`mixer.telemetry.autoscaleMax`	`5`
`mixer.telemetry.cpu.targetAverageUtilization`	`80`
`mixer.telemetry.sessionAffinityEnabled`	`false`
`mixer.telemetry.loadshedding.mode`	`enforce`	`disabled, logonly or enforce`
`mixer.telemetry.loadshedding.latencyThreshold`	`100ms`	`based on measurements 100ms p50 translates to p99 of under 1s. This is ok for telemetry which is inherently async.`
`mixer.telemetry.resources.requests.cpu`	`1000m`
`mixer.telemetry.resources.requests.memory`	`1G`
`mixer.telemetry.resources.limits.cpu`	`4800m`	`It is best to do horizontal scaling of mixer using moderate cpu allocation.We have experimentally found that these values work well.`
`mixer.telemetry.resources.limits.memory`	`4G`
`mixer.podAnnotations`	`{}`
`mixer.nodeSelector`	`{}`
`mixer.adapters.kubernetesenv.enabled`	`true`
`mixer.adapters.stdio.enabled`	`false`
`mixer.adapters.stdio.outputAsJson`	`true`
`mixer.adapters.prometheus.enabled`	`true`
`mixer.adapters.prometheus.metricsExpiryDuration`	`10m`
`mixer.adapters.useAdapterCRDs`	`true`	`Setting this to false sets the useAdapterCRDs mixer startup argument to false`

New `grafana` key/value pairs

Key	Default Value	Description
`grafana.image.repository`	`grafana/grafana`
`grafana.image.tag`	`5.4.0`
`grafana.ingress.enabled`	`false`
`grafana.ingress.hosts`	`grafana.local`	`Used to create an Ingress record.`
`grafana.persist`	`false`
`grafana.storageClassName`	`""`
`grafana.accessMode`	`ReadWriteMany`
`grafana.security.secretName`	`grafana`
`grafana.security.usernameKey`	`username`
`grafana.security.passphraseKey`	`passphrase`
`grafana.nodeSelector`	`{}`
`grafana.contextPath`	`/grafana`
`grafana.datasources.datasources.apiVersion`	`1`
`grafana.datasources.datasources.datasources.type`	`prometheus`
`grafana.datasources.datasources.datasources.orgId`	`1`
`grafana.datasources.datasources.datasources.url`	`http://prometheus:9090`
`grafana.datasources.datasources.datasources.access`	`proxy`
`grafana.datasources.datasources.datasources.isDefault`	`true`
`grafana.datasources.datasources.datasources.jsonData.timeInterval`	`5s`
`grafana.datasources.datasources.datasources.editable`	`true`
`grafana.dashboardProviders.dashboardproviders.apiVersion`	`1`
`grafana.dashboardProviders.dashboardproviders.providers.orgId`	`1`
`grafana.dashboardProviders.dashboardproviders.providers.folder`	`'istio'`
`grafana.dashboardProviders.dashboardproviders.providers.type`	`file`
`grafana.dashboardProviders.dashboardproviders.providers.disableDeletion`	`false`
`grafana.dashboardProviders.dashboardproviders.providers.options.path`	`/var/lib/grafana/dashboards/istio`

New `prometheus` key/value pairs

Key	Default Value	Description
`prometheus.retention`	`6h`
`prometheus.nodeSelector`	`{}`
`prometheus.scrapeInterval`	`15s`	`Controls the frequency of prometheus scraping`
`prometheus.contextPath`	`/prometheus`
`prometheus.ingress.enabled`	`false`
`prometheus.ingress.hosts`	`prometheus.local`	`Used to create an Ingress record.`
`prometheus.security.enabled`	`true`

New `gateways` key/value pairs

Key	Default Value	Description
`gateways.istio-ingressgateway.sds.enabled`	`false`	`If true, ingress gateway fetches credentials from SDS server to handle TLS connections.`
`gateways.istio-ingressgateway.sds.image`	`node-agent-k8s`	`SDS server that watches kubernetes secrets and provisions credentials to ingress gateway.This server runs in the same pod as ingress gateway.`
`gateways.istio-ingressgateway.autoscaleEnabled`	`true`
`gateways.istio-ingressgateway.cpu.targetAverageUtilization`	`80`
`gateways.istio-ingressgateway.loadBalancerSourceRanges`	`[]`
`gateways.istio-ingressgateway.externalIPs`	`[]`
`gateways.istio-ingressgateway.podAnnotations`	`{}`
`gateways.istio-ingressgateway.ports.targetPort`	`15029`
`gateways.istio-ingressgateway.ports.name`	`https-kiali`
`gateways.istio-ingressgateway.ports.name`	`https-prometheus`
`gateways.istio-ingressgateway.ports.name`	`https-grafana`
`gateways.istio-ingressgateway.ports.targetPort`	`15032`
`gateways.istio-ingressgateway.ports.name`	`https-tracing`
`gateways.istio-ingressgateway.ports.targetPort`	`15443`
`gateways.istio-ingressgateway.ports.name`	`tls`
`gateways.istio-ingressgateway.ports.targetPort`	`15020`
`gateways.istio-ingressgateway.ports.name`	`status-port`
`gateways.istio-ingressgateway.meshExpansionPorts.targetPort`	`15011`
`gateways.istio-ingressgateway.meshExpansionPorts.name`	`tcp-pilot-grpc-tls`
`gateways.istio-ingressgateway.meshExpansionPorts.targetPort`	`15004`
`gateways.istio-ingressgateway.meshExpansionPorts.name`	`tcp-mixer-grpc-tls`
`gateways.istio-ingressgateway.meshExpansionPorts.targetPort`	`8060`
`gateways.istio-ingressgateway.meshExpansionPorts.name`	`tcp-citadel-grpc-tls`
`gateways.istio-ingressgateway.meshExpansionPorts.targetPort`	`853`
`gateways.istio-ingressgateway.meshExpansionPorts.name`	`tcp-dns-tls`
`gateways.istio-ingressgateway.env.ISTIO_META_ROUTER_MODE`	`"sni-dnat"`	`A gateway with this mode ensures that pilot generates an additionalset of clusters for internal services but without Istio mTLS, toenable cross cluster routing.`
`gateways.istio-ingressgateway.nodeSelector`	`{}`
`gateways.istio-egressgateway.autoscaleEnabled`	`true`
`gateways.istio-egressgateway.cpu.targetAverageUtilization`	`80`
`gateways.istio-egressgateway.podAnnotations`	`{}`
`gateways.istio-egressgateway.ports.targetPort`	`15443`
`gateways.istio-egressgateway.ports.name`	`tls`
`gateways.istio-egressgateway.env.ISTIO_META_ROUTER_MODE`	`"sni-dnat"`
`gateways.istio-egressgateway.nodeSelector`	`{}`
`gateways.istio-ilbgateway.autoscaleEnabled`	`true`
`gateways.istio-ilbgateway.cpu.targetAverageUtilization`	`80`
`gateways.istio-ilbgateway.podAnnotations`	`{}`
`gateways.istio-ilbgateway.nodeSelector`	`{}`

New `kiali` key/value pairs

Key	Default Value	Description
`kiali.contextPath`	`/kiali`
`kiali.nodeSelector`	`{}`
`kiali.ingress.hosts`	`kiali.local`	`Used to create an Ingress record.`
`kiali.dashboard.secretName`	`kiali`
`kiali.dashboard.usernameKey`	`username`
`kiali.dashboard.passphraseKey`	`passphrase`
`kiali.prometheusAddr`	`http://prometheus:9090`
`kiali.createDemoSecret`	`false`	`When true, a secret will be created with a default username and password. Useful for demos.`

New `istiocoredns` key/value pairs

Key	Default Value	Description
`istiocoredns.enabled`	`false`
`istiocoredns.replicaCount`	`1`
`istiocoredns.coreDNSImage`	`coredns/coredns:1.1.2`
`istiocoredns.coreDNSPluginImage`	`istio/coredns-plugin:0.2-istio-1.1`
`istiocoredns.nodeSelector`	`{}`

New `security` key/value pairs

Key	Default Value	Description
`security.enabled`	`true`
`security.createMeshPolicy`	`true`
`security.nodeSelector`	`{}`

New `nodeagent` key/value pairs

Key	Default Value	Description
`nodeagent.enabled`	`false`
`nodeagent.image`	`node-agent-k8s`
`nodeagent.env.CA_PROVIDER`	`""`	`name of authentication provider.`
`nodeagent.env.CA_ADDR`	`""`	`CA endpoint.`
`nodeagent.env.Plugins`	`""`	`names of authentication provider's plugins.`
`nodeagent.nodeSelector`	`{}`

New `pilot` key/value pairs

Key	Default Value	Description
`pilot.autoscaleEnabled`	`true`
`pilot.env.PILOT_PUSH_THROTTLE`	`100`
`pilot.env.GODEBUG`	`gctrace=1`
`pilot.cpu.targetAverageUtilization`	`80`
`pilot.nodeSelector`	`{}`
`pilot.keepaliveMaxServerConnectionAge`	`30m`	`The following is used to limit how long a sidecar can be connectedto a pilot. It balances out load across pilot instances at the cost ofincreasing system churn.`

Removed configuration options

Removed `ingress` key/value pairs

Key	Default Value	Description
`ingress.service.ports.nodePort`	`32000`
`ingress.service.selector.istio`	`ingress`
`ingress.autoscaleMin`	`1`
`ingress.service.loadBalancerIP`	`""`
`ingress.enabled`	`false`
`ingress.service.annotations`	`{}`
`ingress.service.ports.name`	`http`
`ingress.service.ports.name`	`https`
`ingress.autoscaleMax`	`5`
`ingress.replicaCount`	`1`
`ingress.service.type`	`LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be`

Removed `servicegraph` key/value pairs

Key	Default Value	Description
`servicegraph`	`servicegraph.local`
`servicegraph.ingress`	`servicegraph.local`
`servicegraph.service.internalPort`	`8088`

Removed `telemetry-gateway` key/value pairs

Key	Default Value	Description
`telemetry-gateway.prometheusEnabled`	`false`
`telemetry-gateway.gatewayName`	`ingressgateway`
`telemetry-gateway.grafanaEnabled`	`false`

Removed `global` key/value pairs

Key	Default Value	Description
`global.hyperkube.tag`	`v1.7.6_coreos.0`
`global.k8sIngressHttps`	`false`
`global.crds`	`true`
`global.hyperkube.hub`	`quay.io/coreos`
`global.meshExpansion`	`false`
`global.k8sIngressSelector`	`ingress`
`global.meshExpansionILB`	`false`

Removed `mixer` key/value pairs

Key	Default Value	Description
`mixer.autoscaleMin`	`1`
`mixer.istio-policy.cpu.targetAverageUtilization`	`80`
`mixer.autoscaleMax`	`5`
`mixer.istio-telemetry.autoscaleMin`	`1`
`mixer.prometheusStatsdExporter.tag`	`v0.6.0`
`mixer.istio-telemetry.autoscaleMax`	`5`
`mixer.istio-telemetry.cpu.targetAverageUtilization`	`80`
`mixer.istio-policy.autoscaleEnabled`	`true`
`mixer.istio-telemetry.autoscaleEnabled`	`true`
`mixer.replicaCount`	`1`
`mixer.prometheusStatsdExporter.hub`	`docker.io/prom`
`mixer.istio-policy.autoscaleMin`	`1`
`mixer.istio-policy.autoscaleMax`	`5`

Removed `grafana` key/value pairs

Key	Default Value	Description
`grafana.image`	`grafana`
`grafana.service.internalPort`	`3000`
`grafana.security.adminPassword`	`admin`
`grafana.security.adminUser`	`admin`

Removed `gateways` key/value pairs

Key	Default Value	Description
`gateways.istio-ilbgateway.replicaCount`	`1`
`gateways.istio-egressgateway.replicaCount`	`1`
`gateways.istio-ingressgateway.replicaCount`	`1`
`gateways.istio-ingressgateway.ports.name`	`tcp-pilot-grpc-tls`
`gateways.istio-ingressgateway.ports.name`	`tcp-citadel-grpc-tls`
`gateways.istio-ingressgateway.ports.name`	`http2-prometheus`
`gateways.istio-ingressgateway.ports.name`	`http2-grafana`
`gateways.istio-ingressgateway.ports.targetPort`	`15011`
`gateways.istio-ingressgateway.ports.targetPort`	`8060`

Removed `tracing` key/value pairs

Key	Default Value	Description
`tracing.service.internalPort`	`9411`
`tracing.replicaCount`	`1`
`tracing.jaeger.ingress`	`jaeger.local`
`tracing.ingress`	`tracing.local`
`tracing.jaeger`	`jaeger.local`
`tracing`	`jaeger.local tracing.local`
`tracing.jaeger.ingress.hosts`	`jaeger.local`
`tracing.jaeger.ingress.enabled`	`false`
`tracing.ingress.hosts`	`tracing.local`
`tracing.jaeger.ui.port`	`16686`

Removed `kiali` key/value pairs

Key	Default Value	Description
`kiali.dashboard.username`	`admin`
`kiali.dashboard.passphrase`	`admin`

Removed `pilot` key/value pairs

Key	Default Value	Description
`pilot.replicaCount`	`1`

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

When you upgrade from Istio 1.8 to Istio 1.9.x, you need to consider the changes on this page. These notes detail the changes which purposefully break backwards compatibility with Istio 1.8. The notes also mention changes which preserve backwards compatibility while introducing new behavior. Changes are only included if the new behavior would be unexpected to a user of Istio 1.8.

PeerAuthentication per-port-level configuration will now also apply to pass through filter chains

Previously the PeerAuthentication per-port-level configuration is ignored if the port number is not defined in a service and the traffic will be handled by a pass through filter chain. Now the per-port-level setting will be supported even if the port number is not defined in a service, a special pass through filter chain will be added to respect the corresponding per-port-level mTLS specification. Please check your PeerAuthentication to make sure you are not using the per-port-level configuration on pass through filter chains, it was not a supported feature and you should update your PeerAuthentication accordingly if you are currently relying on the unsupported behavior before the upgrade. You don’t need to do anything if you are not using per-port-level PeerAuthentication on pass through filter chains.

Service Tags added to trace spans

Istio now configures Envoy to include tags identifying the canonical service for a workload in generated trace spans.

This will lead to a small increase in storage per span for tracing backends.

To disable these additional tags, modify the ‘istiod’ deployment to set an environment variable of PILOT_ENABLE_ISTIO_TAGS=false.

`EnvoyFilter` XDS v2 removal

Envoy has removed support for the XDS v2 API. EnvoyFilters depending on these APIs must be updated before upgrading.

For example:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: add-header
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_OUTBOUND
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"@type": type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
inlineCode: |
function envoy_on_request(handle)
handle:headers():add("foo", "bar")
end

Should be updated to:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: add-header
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_OUTBOUND
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(handle)
handle:headers():add("foo", "bar")
end

Both istioctl analyze and the validating webhook (run during kubectl apply) will warn about deprecated usage:

$ kubectl apply -f envoyfilter.yaml
Warning: using deprecated filter name "envoy.http_connection_manager"; use "envoy.filters.network.http_connection_manager" instead
Warning: using deprecated filter name "envoy.router"; use "envoy.filters.http.router" instead
Warning: using deprecated type_url(s); type.googleapis.com/envoy.config.filter.http.lua.v2.Lua
envoyfilter.networking.istio.io/add-header configured

If these filters are applied, the Envoy proxy will reject the configuration (The v2 xDS major version is deprecated and disabled by default.) and be unable to receive updated configurations.

In general, we recommend that EnvoyFilters are applied to a specific version to ensure Envoy changes do not break them during upgrade. This can be done with a match clause:

match:
proxy:
proxyVersion: ^1\.9.*

However, since Istio 1.8 supports both v2 and v3 XDS versions, your EnvoyFilters may also be updated before upgrading Istio.

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

When you upgrade from Istio 1.7.x to Istio 1.8.x, you need to consider the changes on this page. These notes detail the changes which purposefully break backwards compatibility with Istio 1.7.x. The notes also mention changes which preserve backwards compatibility while introducing new behavior. Changes are only included if the new behavior would be unexpected to a user of Istio 1.7.x.

Multicluster `.global` Stub Domain Deprecation

As part of this release, Istio has switched to a new configuration for multi-primary (formerly “replicated control planes”). The new configuration is simpler, has fewer limitations, and has been thoroughly tested in a variety of environments. As a result, the .global stub domain is now deprecated and no longer guaranteed to work going forward.

Mixer is no longer supported in Istio

If you are using the istio-policy or istio-telemetry services, or any related Mixer configuration, you will not be able to upgrade without taking action to either (a) convert your existing configuration and code to the new extension model for Istio or (b) use the gRPC shim developed to bridge transition to the new model. For more details, please refer to the developer wiki.

The semantics of revision for gateways in `IstioOperator` has changed from 1.7 to 1.8

In 1.7, revision means you are creating a new gateway with a different revision so it would not conflict with the default gateway. In 1.8, it means the revision of istiod the gateway is configuring with. If you are using revision for gateways in IstioOperator in 1.7, before moving to 1.8, you must upgrade it to the revision of the Istiod (or delete the revision if you don’t use revision). See Issue #28849.

Istio CoreDNS Plugin Deprecation

The Istio sidecar now provides native support for DNS resolution with ServiceEntries using meshConfig.defaultConfig.proxyMetadata.ISTIO_META_DNS_CAPTURE="true". Previously, this support was provided by the third party Istio CoreDNS plugin. As a result, the istio-coredns-plugin is now deprecated and will be removed in a future release.

Use the new filter names for `EnvoyFilter`

If you are using EnvoyFilter API, it is recommended to change to the new filter names as described in Envoy’s deprecation notice The deprecated filter names will be supported in this release for backward compatibility but will be removed in future releases.

Inbound Cluster Name Format

The format of inbound Envoy cluster names has changed. Previously, they included the Service hostname and port name, such as inbound|80|http|httpbin.default.svc.cluster.local. This lead to issues when multiple Services select the same pod. As a result, we have removed the port name and hostname - the new format will instead resemble inbound|80||.

For most users, this is an implementation detail, and will only impact debugging or tooling that directly interacts with Envoy configuration.

Avoid use of mesh expansion installation flags

To ease setup for multicluster and virtual machines while giving more control to users, the meshExpansion and meshExpansionPorts installation flags have been deprecated, and port 15012 has been added to the default list of ports for the istio-ingressgateway Service.

For users with values.global.meshExpansion.enabled=true, perform the following steps before upgrading Istio:

Apply the code sample for exposing Istiod through ingress.

$ kubectl apply -f @samples/multicluster/expose-istiod.yaml@

This removes operator.istio.io/managed labels from the associated Istio networking resources so that the Istio installer won’t delete them. After this step, you can modify these resources freely.

If components.ingressGateways[name=istio-ingressgateway].k8s.service.ports is overridden, add port 15012 to the list of ports:

- port: 15012
targetPort: 15012
name: tcp-istiod

If values.gateways.istio-ingressgateway.meshExpansionPorts is set, move all ports to components.ingressGateways[name=istio-ingressgateway].k8s.service.ports if they’re not already present. Then, unset this value.
Unset values.global.meshExpansion.enabled.

Protocol Detection Timeout Changes

In order to support permissive mTLS traffic as well as automatic protocol detection, the proxy will sniff the first few bytes of traffic to determine the protocol used. For certain “server first” protocols, such as the protocol used by MySQL, there will be no initial bytes to sniff. To mitigate this issue in the past, Istio introduced a detection timeout. However, we found this caused frequent telemetry and traffic failures during slow connections, while increasing latency for misconfigured server first protocols rather than failing fast.

This timeout has been disabled by default. This has the following impacts:

Non “server first” protocols will no longer have a risk of telemetry or traffic failures during slow connections
Properly configured “server first” protocols will no longer have an extra 5 seconds latency on each connection
Improperly configured “server first” protocols will experience connection timeouts. Please ensure you follow the steps listed in Server First Protocols to ensure you do not run into traffic issues.

Update AuthorizationPolicy resources to use `remoteIpBlocks`/`notRemoteIpBlocks` instead of `ipBlocks`/`notIpBlocks` if using the Proxy Protocol

If using the Proxy Protocol on a load balancer in front an ingress gateway in conjunction with ipBlocks/notIpBlocks on an AuthorizationPolicy to perform IP-based access control, then please update the AuthorizationPolicy to use remoteIpBlocks/notRemoteIpBlocks instead after upgrading. The ipBlocks/notIpBlocks fields now strictly refer to the source IP address of the packet that arrives at the sidecar.

`AUTO_PASSTHROUGH` Gateway mode

Previously, gateways were configured with multiple Envoy cluster configurations for each Service in the cluster, even those not referenced by any Gateway or VirtualService. This was added to support the AUTO_PASSTHROUGH mode on Gateway, generally used for exposing Services across networks.

However, this came at an increased CPU and memory cost in the gateway and Istiod. As a result, we have disabled these by default on the istio-ingressgateway and istio-egressgateway.

If you are relying on this feature for multi-network support, please ensure you apply one of the following changes:

Follow our new Multicluster Installation documentation.

This documentation will guide you through running a dedicate gateway deployment for this type of traffic (generally referred to as the eastwest-gateway). This eastwest-gateway will automatically be configured to support AUTO_PASSTHROUGH.

Modify your installation of the gateway deployment to include this configuration. This is controlled by the ISTIO_META_ROUTER_MODE environment variable. Setting this to sni-dnat enables these clusters, while standard (the new default) disables them.

ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: "sni-dnat"

Connectivity issues among your proxies when updating from 1.7.x (where x < 5)

When upgrading your Istio data plane from 1.7.x (where x < 5) to 1.8, you may observe connectivity issues between your gateway and your sidecars or among your sidecars with 503 errors in the log. This happens when 1.7.5+ proxies send HTTP 1xx or 204 response codes with headers that 1.7.x proxies reject. To fix this, upgrade all your proxies (gateways and sidecars) to 1.7.5+ as soon as possible. (Issue 29427, More information)

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

When you upgrade from Istio 1.6.x to Istio 1.7.x, you need to consider the changes on this page. These notes detail the changes which purposefully break backwards compatibility with Istio 1.6.x. The notes also mention changes which preserve backwards compatibility while introducing new behavior. Changes are only included if the new behavior would be unexpected to a user of Istio 1.6.x.

Require Kubernetes 1.16+

Kubernetes 1.16+ is now required for installation.

Installation

istioctl manifest apply is removed, please use istioctl install instead.
Installation of telemetry addons by istioctl is deprecated, please use these addons integration instructions.

Gateways run as non-root

Gateways will now run without root permissions by default. As a result, they will no longer be able to bind to ports below 1024. By default, we will bind to valid ports. However, if you are explicitly declaring ports on the gateway, you may need to modify your installation. For example, if you previously had the following configuration:

ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
service:
ports:
- port: 15021
targetPort: 15021
name: status-port
- port: 80
name: http2
- port: 443
name: https

It should be changed to specify a valid targetPort that can be bound to:

ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
service:
ports:
- port: 15021
targetPort: 15021
name: status-port
- port: 80
name: http2
targetPort: 8080
- port: 443
name: https
targetPort: 8443

Note: the targetPort only modifies which port the gateway binds to. Clients will still connect to the port defined by port (generally 80 and 443), so this change should be transparent.

If you need to run as root, this option can be enabled with --set values.gateways.istio-ingressgateway.runAsRoot=true.

`EnvoyFilter` syntax change

EnvoyFilters using the legacy config syntax will need to be migrated to the new typed_config. This is due to underlying changes in Envoy’s API.

As EnvoyFilter is a break glass API without backwards compatibility guarantees, we recommend users explicitly bind EnvoyFilters to specific versions and appropriately test them prior to upgrading.

For example, a configuration for Istio 1.6, using the legacy config syntax:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: lua-1.6
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: envoy.http_connection_manager
proxy:
proxyVersion: ^1\.6.*
patch:
operation: INSERT_BEFORE
value:
name: envoy.lua
config:
inlineCode: |
function envoy_on_request(handle)
request_handle:headers():add("foo", "bar")
end

When upgrading to Istio 1.7, a new filter should be added:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: lua-1.7
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: ANY
listener:
filterChain:
filter:
name: envoy.http_connection_manager
proxy:
proxyVersion: ^1\.7.*
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.lua
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
inlineCode: |
function envoy_on_request(handle)
request_handle:headers():add("foo", "bar")
end

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

When you upgrade from Istio 1.5.x to Istio 1.6.x, you need to consider the changes on this page. These notes detail the changes which purposefully break backwards compatibility with Istio 1.5.x. The notes also mention changes which preserve backwards compatibility while introducing new behavior. Changes are only included if the new behavior would be unexpected to a user of Istio 1.5.x.

Currently, Istio doesn’t support skip-level upgrades. If you are using Istio 1.4, you must upgrade to Istio 1.5 first, and then upgrade to Istio 1.6. If you upgrade from versions earlier than Istio 1.4, you should first disable Galley’s configuration validation.

Update the Galley deployment using the following steps:

To edit the Galley deployment configuration, run the following command:
```
$ kubectl edit deployment -n istio-system istio-galley
```

Add the --enable-validation=false option to the command: section as shown below:

apiVersion: extensions/v1beta1
kind: Deployment
...
spec:
...
template:
...
spec:
...
containers:
- command:
...
- --log_output_level=default:info
- --enable-validation=false

Save and quit the editor to update the deployment configuration in the cluster.

Remove the ValidatingWebhookConfiguration Custom Resource (CR) with the following command:

$ kubectl delete ValidatingWebhookConfiguration istio-galley -n istio-system

Change the readiness port of gateways

If you are using the 15020 port to check the health of your Istio ingress gateway with your Kubernetes network load balancer, change the port from 15020 to 15021.

Removal of legacy Helm charts

Istio 1.4 introduced a new way to install Istio using the in-cluster Operator or istioctl install command. Part of this change meant deprecating the old Helm charts in 1.5. Many new Istio features rely on the new installation method. As a result, Istio 1.6 doesn’t include the old Helm installation charts.

Go to the Istio 1.5 Upgrade Notes before you continue because Istio 1.5 introduced several changes not present in the legacy installation method, such as Istiod and telemetry v2.

To safely upgrade from the legacy installation method that uses Helm charts, perform a control plane revision. Upgrading in-place is not supported. Upgrading could result in downtime unless you perform a canary upgrade.

Support ended for `v1alpha1` security policy

Istio 1.6 no longer supports the following security policy APIs:

Starting in Istio 1.6, Istio ignores these v1alpha1 security policy APIs.

Istio 1.6 replaced the v1alpha1 authentication policy with the following APIs:

Istio 1.6 replaces the v1alpha1 RBAC policy APIs with the v1beta1 authorization policy APIs.

Verify that there are no v1alpha1 security policies in your clusters the following commands:

$ kubectl get policies.authentication.istio.io --all-namespaces
$ kubectl get meshpolicies.authentication.istio.io --all-namespaces
$ kubectl get rbacconfigs.rbac.istio.io --all-namespaces
$ kubectl get clusterrbacconfigs.rbac.istio.io --all-namespaces
$ kubectl get serviceroles.rbac.istio.io --all-namespaces
$ kubectl get servicerolebindings.rbac.istio.io --all-namespaces

If there are any v1alpha1 security policies in your clusters, migrate to the new APIs before upgrading.

Istio configuration during installation

Past Istio releases deployed configuration objects during installation. The presence of those objects caused the following issues:

Problems with upgrades
A confusing user experience
A less flexible installation

To address these issues, Istio 1.6 minimized the configuration objects deployed during installation.

The following configurations are impacted:

global.mtls.enabled: Configuration removed to avoid confusion. Configure a peer authentication policy to enable strict mTLS instead.
No default Gateway and associated Certificate custom resources are deployed during installation. Go to the Ingress task to configure a gateway for your mesh.
Istio no longer creates Ingress custom resources for telemetry addons. Visit remotely accessing telemetry addons to learn how to reach the addons externally.
The default sidecar configuration is no longer defined through the automatically generated Sidecar custom resource. The default configuration is implemented internally and the change should have no impact on deployments.

Reach Istiod through external workloads

In Istio 1.6, Istiod is configured to be cluster-local by default. With cluster-local enabled, only workloads running on the same cluster can reach Istiod. Workloads on another cluster can only access the Istiod instance through the Istio gateway. This configuration prevents the ingress gateway of the master cluster from incorrectly forwarding service discovery requests to Istiod instances in remote clusters. The Istio team is actively investigating alternatives to no longer require cluster-local.

To override the default cluster-local behavior, modify the configuration in the MeshConfig section as shown below:

values:
meshConfig:
serviceSettings:
- settings:
clusterLocal: false
hosts:
- "istiod.istio-system.svc.cluster.local"

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes changes you need to be aware of when upgrading from Istio 1.4.x to 1.5.x. Here, we detail cases where we intentionally broke backwards compatibility. We also mention cases where backwards compatibility was preserved but new behavior was introduced that would be surprising to someone familiar with the use and operation of Istio 1.4.

Control Plane Restructuring

In Istio 1.5, we have moved towards a new deployment model for the control plane, with many components consolidated. The following describes where various functionality has been moved to.

Istiod

In Istio 1.5, there will be a new deployment, istiod. This component is the core of the control plane, and will handle configuration and certificate distribution, sidecar injection, and more.

Sidecar injection

Previously, sidecar injection was handled by a mutating webhook that was processed by a deployment named istio-sidecar-injector. In Istio 1.5, the same mutating webhook remains, but it will now point to the istiod deployment. All injection logic remains the same.

Galley

Configuration Validation - this functionality remains the same, but is now handled by the istiod deployment.
MCP Server - the MCP server has been disabled by default. For most users, this is an implementation detail. If you depend on this functionality, you will need to run the istio-galley deployment.
Experimental features (such as configuration analysis) - These features will require the istio-galley deployment.

Citadel

Previously, Citadel served two functions: writing certificates to secrets in each namespace, and serving secrets to the nodeagent over gRPC when SDS is used. In Istio 1.5, secrets are no longer written to each namespace. Instead, they are only served over gRPC. This functionality has been moved to the istiod deployment.

SDS Node Agent

The nodeagent deployment has been removed. This functionality now exists in the Envoy sidecar.

Sidecar

Previously, the sidecar was able to access certificates in two ways: through secrets mounted as files, or over SDS (through the nodeagent deployment). In Istio 1.5, this has been simplified. All secrets will be served over a locally run SDS server. For most users, these secrets will be fetched from the istiod deployment. For users with a custom CA, file mounted secrets can still be used, however, these will still be served by the local SDS server. This means that certificate rotations will no longer require Envoy to restart.

CNI

There have been no changes to the deployment of istio-cni.

Pilot

The istio-pilot deployment has been removed in favor of the istiod deployment, which contains all functionality that Pilot once had. For backwards compatibility, there are still some references to Pilot.

Mixer deprecation

Mixer, the process behind the istio-telemetry and istio-policy deployments, has been deprecated with the 1.5 release. istio-policy was disabled by default since Istio 1.3 and istio-telemetry is disabled by default in Istio 1.5.

Telemetry is collected using an in-proxy extension mechanism (Telemetry V2) that does not require Mixer.

If you depend on specific Mixer features like out of process adapters, you may re-enable Mixer. Mixer will continue receiving bug fixes and security fixes until Istio 1.7. Many features supported by Mixer have alternatives as specified in the Mixer Deprecation document including the in-proxy extensions based on the WebAssembly sandbox API.

If you rely on a Mixer feature that does not have an equivalent, we encourage you to open issues and discuss in the community.

Please check Mixer Deprecation notice for details.

Feature gaps between Telemetry V2 and Mixer Telemetry

Out of mesh telemetry is not supported. Some telemetry is missing if the traffic source or destination is not sidecar injected.
Egress gateway telemetry is not supported.
TCP telemetry is only supported with mtls.
Black Hole telemetry for TCP and HTTP protocols is not supported.
Histogram buckets are significantly different than Mixer Telemetry and cannot be changed.

Traffic management resource visibility changes

In Istio 1.5 proxy configuration for hosts is determined by VirtualService visibility in addition to that of any relevant ServiceEntry.

If in previous versions you relied on Sidecar resources to restrict the visibility of hosts (mesh internal or external) to a target set of sidecar proxies, you now also need to consider the hosts implied by any VirtualService.

Depending on your use of Sidecar resources in your mesh, this may require you to review the namespaces that your VirtualServices are in to ensure only the intended workloads can see them.

More details on this change can be found at 24251 and 20408.

Authentication policy

Istio 1.5 introduces PeerAuthentication and RequestAuthentication, which are replacing the alpha version of the Authentication API. For more information about how to use the new API, see the authentication policy tutorial.

After you upgrade Istio, your alpha authentication policies remain in place and being used. You can gradually replace them with the equivalent PeerAuthentication and RequestAuthentication. The new policy will take over the old policy in the scope it is defined. We recommend starting with workload-wide (the most specific scope), then namespace-wide, and finally mesh-wide.
After you replace policies for workload, namespace, and mesh, you can safely remove the alpha authentication policies. To delete the alpha policies, use this command:

$ kubectl delete policies.authentication.istio.io --all-namespaces --all
$ kubectl delete meshpolicies.authentication.istio.io --all

Istio workload key and certificate provisioning

We have stabilized the SDS certificate and key provisioning flow. Now the Istio workloads are using SDS to provision certificates. The secret volume mount approach is deprecated.
Please note when mutual TLS is enabled, Prometheus deployment needs to be manually modified to monitor the workloads. The details are described in this issue. This is not required in 1.5.1.

Automatic mutual TLS

Automatic mutual TLS is now enabled by default. Traffic between sidecars is automatically configured as mutual TLS. You can disable this explicitly if you worry about the encryption overhead by adding the option -- set values.global.mtls.auto=false during install. For more details, refer to automatic mutual TLS.

Control plane security

As part of the Istiod effort, we have changed how proxies securely communicate with the control plane. In previous versions, proxies would connect to the control plane securely when the setting values.global.controlPlaneSecurityEnabled=true was configured, which was the default for Istio 1.4. Each control plane component ran a sidecar with Citadel certificates, and proxies connected to Pilot over port 15011.

In Istio 1.5, this is no longer the recommended or default way to connect the proxies with the control plane; instead, DNS certificates, which can be signed by Kubernetes or Istiod, will be used to connect to Istiod over port 15012.

Note: despite the naming, in Istio 1.5 when controlPlaneSecurityEnabled is set to false, communication between the control plane will be secure by default.

Multicluster setup

Helm upgrade

If you used helm upgrade to update your cluster to newer Istio versions, we recommend you to switch to use istioctl upgrade or follow the helm template steps.

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes changes you need to be aware of when upgrading from Istio 1.3.x to 1.4.x. Here, we detail cases where we intentionally broke backwards compatibility. We also mention cases where backwards compatibility was preserved but new behavior was introduced that would be surprising to someone familiar with the use and operation of Istio 1.3.

Traffic management

HTTP services on port 443

Services of type http are no longer allowed on port 443. This change was made to prevent protocol conflicts with external HTTPS services.

If you depend on this behavior, there are a few options:

Move the application to another port.
Change the protocol from type http to type tcp
Specify the environment variable PILOT_BLOCK_HTTP_ON_443=false to the Pilot deployment. Note: this may be removed in future releases.

See Protocol Selection for more information about specifying the protocol of a port

Regex Engine Changes

To prevent excessive resource consumption from large regular expressions, Envoy has moved to a new regular expression engine based on re2. Previously, std::regex was used. These two engines may have slightly different syntax; in particular, the regex fields are now limited to 100 bytes.

If you depend on specific behavior of the old regex engine, you can opt out of this change by adding the environment variable PILOT_ENABLE_UNSAFE_REGEX=true to the Pilot deployment. Note: this will be removed in future releases.

Configuration management

We introduced OpenAPI v3 schemas in the Kubernetes Custom Resource Definitions (CRD) of Istio resources. The schemas describe the Istio resources and help ensure the Istio resources you create and modify are structurally correct.

If one or more fields in your configurations are unknown or have wrong types, they will be rejected by the Kubernetes API server when you create or modify Istio resources. This feature, CustomResourceValidation, is on by default for Kubernetes 1.9+ clusters. Please note that existing configurations already in Kubernetes are NOT affected if they stay unchanged.

To help with your upgrade, here are some steps you could take:

After upgrading Istio, run your Istio configurations with kubectl apply --dry-run so that you are able to know if the configurations can be accepted by the API server as well as any possible unknown and/or invalid fields to the API server. (DryRun feature is on by default for Kubernetes 1.13+ clusters.)
Use the reference documentation to confirm and correct the field names and data types.
In addition to structural validation, you can also use istioctl x analyze to help you detect other potential issues with your Istio configurations. Refer to here for more details.

If you choose to ignore the validation errors, add --validate=false to your kubectl command when you create or modify Istio resources. We strongly discourage doing so however, since it is willingly introducing incorrect configuration.

Leftover CRD

Istio 1.4 introduces a new CRD authorizationpolicies.security.istio.io for the authorization policy. Your cluster may have an interim leftover CRD authorizationpolicies.rbac.istio.io due to an internal implementation detail before Istio 1.4.

The leftover CRD is unused and you can safely remove it from the cluster using this command:

$ kubectl delete crd authorizationpolicies.rbac.istio.io --ignore-not-found=true

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes changes you need to be aware of when upgrading from Istio 1.2.x to 1.3.x. Here, we detail cases where we intentionally broke backwards compatibility. We also mention cases where backwards compatibility was preserved but new behavior was introduced that would be surprising to someone familiar with the use and operation of Istio 1.2.

Installation and upgrade

We simplified the configuration model for Mixer and removed support for adapter-specific and template-specific Custom Resource Definitions (CRDs) entirely in 1.3. Please move to the new configuration model.

We removed the Mixer CRDs from the system to simplify the configuration model, improve Mixer’s performance in Kubernetes deployments, and improve reliability in various Kubernetes environments.

Traffic management

Istio now captures all ports by default. If you don’t specify container ports to intentionally bypass Envoy, you must opt out of port capturing with the traffic.sidecar.istio.io/excludeInboundPorts option.

Protocol sniffing is now enabled by default. Disable protocol sniffing with the --set pilot.enableProtocolSniffing=false option when you upgrade to get the previous behavior. To learn more see our protocol selection page.

To specify a hostname in multiple namespaces, you must select a single host using a Sidecar resource.

Trust domain validation

Trust domain validation is new in Istio 1.3. If you only have one trust domain or you don’t enable mutual TLS through authentication policies, there is nothing you must do.

To opt-out the trust domain validation, include the following flag in your Helm template before upgrading to Istio 1.3: --set pilot.env.PILOT_SKIP_VALIDATE_TRUST_DOMAIN=true

Secret discovery service

In Istio 1.3, we are taking advantage of improvements in Kubernetes to issue certificates for workload instances more securely.

Kubernetes 1.12 introduces trustworthy JWTs to solve these issues. Kubernetes 1.13 introduced the ability to change the value of the aud field to a value other than the API server. The aud field represents the audience in Kubernetes. To better secure the mesh, Istio 1.3 only supports trustworthy JWTs and requires the audience, the value of the aud field, to be istio-ca when you enable SDS.

Before upgrading to Istio 1.3 with SDS enabled, see our blog post on trustworthy JWTs and SDS.

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes changes you need to be aware of when upgrading from Istio 1.1.x to 1.2.x. Here, we detail cases where we intentionally broke backwards compatibility. We also mention cases where backwards compatibility was preserved but new behavior was introduced that would be surprising to someone familiar with the use and operation of Istio 1.1.

Installation and upgrade

Most Mixer CRDs were removed from the system to simplify the configuration model, improve performance of Mixer when used with Kubernetes, and improve reliability in a variety of Kubernetes environments.

The following CRDs remain:

Custom Resource Definition name	Purpose
`adapter`	Specification of Istio extension declarations
`attributemanifest`	Specification of Istio extension declarations
`template`	Specification of Istio extension declarations
`handler`	Specification of extension invocations
`rule`	Specification of extension invocations
`instance`	Specification of extension invocations

In the event you are using the removed mixer configuration schemas, set the following Helm flags during upgrade of the main Helm chart: --set mixer.templates.useTemplateCRDs=true --set mixer.adapters.useAdapterCRDs=true

Upgrade Notes

Mon, 01 Jan 0001 00:00:00 +0000

This page describes changes you need to be aware of when upgrading from Istio 1.0.x to 1.1.x. Here we detail cases where we intentionally broke backwards compatibility. We also mention cases where backwards compatibility was preserved but new behavior was introduced that would be surprising to someone familiar with the use and operation of Istio 1.0.

For an overview of new features introduced with Istio 1.1, please refer to the 1.1 change notes.

Installation

We have increased the control plane and envoy sidecar’s required CPU and memory. It is critical to ensure your cluster have enough resource before proceeding the update.
Istio’s CRDs have been placed into their own Helm chart istio-init. This prevents loss of custom resource data, facilitates the upgrade process, and enables Istio to evolve beyond a Helm-based installation. The upgrade documentation provides the proper procedures for upgrading from Istio 1.0.6 to Istio 1.1. Please follow these instructions carefully when upgrading. If certmanager is desired, use the --set certmanager=true flag when installing both istio-init and Istio charts with either template or tiller installation modes.
Many installation options have been added, removed, or changed. Refer to Installation Options Changes for a detailed summary of the changes.
The 1.0 istio-remote chart used for multicluster VPN and multicluster split horizon remote cluster installation has been consolidated into the Istio chart. To generate an equivalent istio-remote chart, use the --set global.istioRemote=true flag.
Addons are no longer exposed via separate load balancers. Instead addons can now be optionally exposed via the Ingress Gateway. To expose an addon via the Ingress Gateway, please follow the Remotely Accessing Telemetry Addons guide.
The built-in Istio Statsd collector has been removed. Istio retains the capability of integrating with your own Statsd collector, using the --set global.envoyStatsd.enabled=true flag.
The ingress series of options for configuring a Kubernetes Ingress have been removed. Kubernetes Ingress is still functional and can be enabled using the --set global.k8sIngress.enabled=true flag. Check out Securing Kubernetes Ingress with Cert-Manager to learn how to secure your Kubernetes ingress resources.

Traffic management

Outbound traffic policy now defaults to ALLOW_ANY. Traffic to unknown ports will be forwarded as-is. Traffic to known ports (e.g., port 80) will be matched with one of the services in the system and forwarded accordingly.
During sidecar routing to a service, destination rules for the target service in the same namespace as the sidecar will take precedence, followed by destination rules in the service’s namespace, and finally followed by destination rules in other namespaces if applicable.
We recommend storing gateway resources in the same namespace as the gateway workload (e.g., istio-system in case of istio-ingressgateway). When referring to gateway resources in virtual services, use the namespace/name format instead of using name.namespace.svc.cluster.local.
The optional egress gateway is now disabled by default. It is enabled in the demo profile for users to explore but disabled in all other profiles by default. If you need to control and secure your outbound traffic through the egress gateway, you will need to enable gateways.istio-egressgateway.enabled=true manually in any of the non-demo profiles.

Policy & telemetry

istio-policy check is now disabled by default. It is enabled in the demo profile for users to explore but disabled in all other profiles. This change is only for istio-policy and not for istio-telemetry. In order to re-enable policy checking, run helm template with --set global.disablePolicyChecks=false and re-apply the configuration.
The Service Graph component has now been deprecated in favor of Kiali.

Security

RBAC configuration has been modified to implement cluster scoping. The RbacConfig resource has been replaced with the ClusterRbacConfig resource. Refer to Migrating RbacConfig to ClusterRbacConfig for migration instructions.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Known Issues

Wasm extension configuration updates can be disruptive (see Issue #29843).

Traffic Management

Added Add pprof endpoint to pilot-agent. (Issue #28040)
Added Allow enabling gRPC logging with –log_output_level for pilot. (Issue #28482)
Added a new experimental proxy option DNS_AUTO_ALLOCATE, to control auto allocation of ServiceEntry addresses. Previously, this option was tied to DNS_CAPTURE. Now, DNS_CAPTURE can be enabled without auto allocation. See Smart DNS Proxying for more info. (Issue #29324)
Fixed istiod will no longer generate listeners for privileged gateway ports (<1024) if the gateway Pod does not have sufficient permissions. (Issue #27566)
Fixed an issue that caused very high memory usage with a large number of ServiceEntries. (Issue #25531)
Removed support for reading Istio configuration over the Mesh Configuration Protocol (MCP). (Pull Request #28634)

Security

Added option to allow users to enable token exchange for their XDS flows, which exchanges a k8s token for a token that can be authenticated by their XDS servers. (Issue #29943)
Added OIDC JWT authenticator that supports both JWKS-URI and OIDC discovery. The OIDC JWT authenticator will be used when configured through the JWT_RULE env variable. (Issue #30295)
Added support of PeerAuthentication per-port-level configuration on pass through filter chains. (Issue #27994)
Added an experimental CUSTOM action in AuthorizationPolicy for integration with external authorization systems like OPA, OAuth2 and more. See the blog on this feature for more info. (Issue #27790)

Telemetry

Added Istio Grafana Dashboards Query Reporter Dropdown. (Issue #27595)
Added canonical service tags to Envoy-generated trace spans. (Pull Request #28801)
Fixed an issue to allow nested JSON structure in meshConfig.accessLogFormat. (Issue #28597)
Updated Prometheus metrics to include source_cluster and destination_cluster labels by default for all scenarios. Previously, this was only enabled for multi-cluster scenarios. (Pull Request #30036)
Updated default access log to include RESPONSE_CODE_DETAILS and CONNECTION_TERMINATION_DETAILS for proxy version >= 1.9. (Pull Request #27903)

Extensibility

Added Reliable Wasm module remote load with Istio agent. (Issue #29989)

Networking

Added Correctly iptables rules and listener filters setting to support original src ip preserve in TPROXY mode within a cluster. (Issue #23369)
Fixed a bug where locality weights are only applied when outlier detection is enabled. (Issue #28942)

Installation

Added post-install/in-place upgrade verification of control plane health. Use --verify flag with istioctl install or istioctl upgrade. (Issue #21715)
Added Add pprof endpoint to pilot-agent. (Issue #28040)
Added enableIstioConfigCRDs to base to allow user specify whether the Istio CRDs will be installed. (Pull Request #28346)
Added Istio 1.9 supports Kubernetes versions 1.17 to 1.20. (Issue #30176)
Added support for applications that bind to their pod IP address, rather than wildcard or localhost address, through the Sidecar API. (Pull Request #28178)
Fixed revision is not applied to the scale target reference of HorizontalPodAutoscaler when helm values for hpa are specified explicitly. (Issue #30203)
Improved the sidecar injector to better utilize pod labels to determine if injection is required. This is not enabled by default in this release, but can be tested using --set values.sidecarInjectorWebhook.useLegacySelectors=false. (Pull Request #30013)
Updated Kiali addon to the latest version v1.29 . (Pull Request #30438)

istioctl

Added istioctl install will detect different Istio version installed (istioctl, control plan version) and display warning. (Issue #18487)
Added istioctl apply as an alias for istioctl install. (Issue #28753)
Added --browser flag to istioctl dashboard, which controls whether you want to open a browser to view the dashboard. (Issue #29022)
Added istioctl verify-install will indicate errors in red and expected configuration in green. (Issue #29336)
Added the severity level for each analysis message in the validationMessages field within the status field. (Issue #29445)
Added WorkloadEntry resources will be read from all clusters in multi-cluster installations and do not need to be duplicated. Makes Virtual Machine auto-registration compatible with multi-primary multi-cluster. This feature is disabled by default and can be enabled by setting the PILOT_ENABLE_CROSS_CLUSTER_WORKLOAD_ENTRY environment variable in istiod. (Issue #29026)
Added istioctl analyze now informs if deprecated or alpha-level annotations are present. (These checks can be disabled using --suppress "IST0135=*" and --suppress "IST0136=*" respectively.) (Issue #29154)
Added istioctl x injector list command to show which namespaces have Istio sidecar injection and, for control plane canaries, show all Istio injectors and the namespaces they control. (Issue #23892)
Fixed istioctl wait now tracks resource’s metadata.generation field, rather than metadata.resourceVersion. Command line arguments have been updated to reflect this. (Issue #28797)
Fixed namespace shorthand flag missing in dashboard subcommand. (Issue #28970)
Fixed istioctl dashboard controlz could not port forward to istiod pod. (Issue #30208)
Fixed installation issue in which --readiness-timeout flag is not honored. (Issue #30221)
Improved verify-install detects Istio injector without control plane. (Issue #29607)
Removed istioctl convert-ingress command. (Issue #29153)
Removed istioctl experimental multicluster command. (Issue #29153)
Removed istioctl experimental post-install webhook command. (Issue #29153)
Removed istioctl register and deregister commands. (Issue #29153)
Updated istioctl proxy-config log to allow filtering logs based on label. (Issue #27490)

Documentation

Added The locality load balancing docs have been re-written into a formal traffic management task. The new docs describe in more detail how locality load balancing works as well as how to configure both failover and weighted distribution. In addition, the new docs are now automatically verified for correctness. (Pull Request #29651)

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Known Issues

Memory leak in TCP Wasm extensions affecting TCP telemetry (see Issue #24720). The leak occurs when upstream connections are interrupted mid-stream.
Wasm extension configuration updates are disruptive (see Issue #13690). The configuration is immediately applied for existing requests and connections, and is not reverted if the outer xDS is rejected.
Race condition with Envoy aggregate cluster when creating an EnvoyFilter and ServiceEntry for the same service. Istio-injected pods are unable to start up due to istio-proxy crashing with a segfault. See Issue #28620 for more information.

Traffic Management

Deprecated the use of the .global stub domain for multi-primary (formerly “replicated control planes”) multicluster configurations. The new onboarding flow uses a simpler configuration which allows services across the mesh to be accessed via *.cluster.local. There were several limitations with .global, such as poor load balancing when using gateways, which are no longer an issue with the new configuration.
Added DNS capture in istio-agent by default for VMs installed using istioctl x workload entry configure.
Added holdApplicationUntilProxyStarts field to ProxyConfig, allowing it to be configured at the pod level. Should not be used in conjunction with the deprecated values.global.proxy.holdApplicationUntilProxyStarts value. (Issue #27696)

Added support for INSERT_FIRST, INSERT_BEFORE, INSERT_AFTER insert operations for HTTP_ROUTE in EnvoyFilter (Issue #26692)
Added REPLACE operation for EnvoyFilter. REPLACE operation can replace the contents of a named filter with new contents. It is only valid for HTTP_FILTER and NETWORK_FILTER. (Issue #27425)
Added Istio resource status now includes observed generation (Issue #28003)
Fixed remove endpoints when the new labels in WorkloadEntry do not match the workloadSelector in ServiceEntry. (Issue #25678)
Fixed when a node has multiple IP addresses (e.g., a VM in the mesh expansion scenario), Istio Proxy will now bind inbound listeners to the first applicable address in the list (new behavior) rather than to the last one (former behavior). (Issue #28269)

Security

Improved Gateway certificates to be read and distributed from Istiod, rather than in the gateway pods. This reduces the permissions required in the gateways, improves performance, and makes certificate reading more extensible. This change is fully backwards compatible with the old mechanism, and requires no changes to your cluster. If required, it can be disabled by setting the ISTIOD_ENABLE_SDS_SERVER=false environment variable in Istiod. (Pull Request #27744)
Improved TLS configuration on sidecar server side inbound paths to enforce TLSv2 version along with recommended cipher suites. If this is not needed or creates problems with non Envoy clients, it can disabled by setting Istiod env variable PILOT_SIDECAR_ENABLE_INBOUND_TLS_V2 to false. (Pull Request #27500)
Updated The ipBlocks/notIpBlocks fields of an AuthorizationPolicy now strictly refer to the source IP address of the IP packet as it arrives at the sidecar. Prior to this release, if using the Proxy Protocol, then the ipBlocks/notIpBlocks would refer to the IP address determined by the Proxy Protocol. Now the remoteIpBlocks/notRemoteIpBlocks fields must be used to refer to the client IP address from the Proxy Protocol. (reference)(usage)(usage) (Issue #22341)
Added AuthorizationPolicy now supports nested JWT claims. (Issue #21340)
Added support for client side Envoy secure naming config when trust domain alias is used. This fixes the multi-cluster service discovery client SAN generation to use all endpoints’ service accounts rather than the first found service registry. (Pull Request #26185)
Added Experimental feature support allowing Istiod to integrate with external certificate authorities using Kubernetes CSR API (>=1.18 only). (Issue #27606)(usage)
Added Enable user to set the custom VM identity provider for credential authentication (Issue #27947)
Added action ‘AUDIT’ to Authorization Policy that can be used to determine which requests should be audited. (Issue #25591)
Added support for migration and concurrent use of regular K8S tokens as well as new K8S tokens with audience. This feature is enabled by default, can be disabled by REQUIRE_3P_TOKEN environment variable in Istiod, which will require new tokens with audience. The TOKEN_AUDIENCES environment variable allows customizing the checked audience, default remains istio-ca. (Pull Request #26482)
Added AuthorizationPolicy now supports a Source of type remoteIpBlocks/notRemoteIpBlocks that map to a new Condition attribute called remote.ip that can also be used in the “when” clause. If using an http/https load balancer in front of the ingress gateway, the remote.ip attribute is set to the original client IP address determined by the X-Forwarded-For http header from the trusted proxy configured through the numTrustedProxies field of the gatewayTopology under the meshConfig when you install Istio or set it via an annotation on the ingress gateway. See the documentation here: Configuring Gateway Network Topology. If using a TCP load balancer with the Proxy Protocol in front of the ingress gateway, the remote.ip is set to the original client IP address as given by the Proxy Protocol. (reference)(usage)(usage) (Issue #22341)

Telemetry

Updated the “Control Plane Dashboard” and the “Performance Dashboard” to use the container_memory_working_set_bytes metric to display memory. This metric only counts memory that cannot be reclaimed by the kernel even under memory pressure, and therefore more relevant for tracking. It is also consistent with kubectl top. The reported values are lower than the previous values.
Updated the Istio Workload and Istio Service dashboards resulting in faster load time. (Issue #22408)
Added datasource parameter to Grafana dashboards (Issue #22408)
Added Listener Access Logs when ResponseFlag from Envoy is set. (Issue #26851)
Added support for OpenCensusAgent formatted trace export with configurable trace context headers.
Added Proxy config to control Envoy native stats generation. (Issue #26546)
Added Istio Wasm Extension Grafana Dashboard. (Issue #25843)
Added gRPC streaming message count proxy Prometheus metrics istio_request_messages_total and istio_response_messages_total (Pull Request #3048)
Added support for properly labeling traffic in client metrics for cases when the destination is not reached or is not behind a proxy. (Issue #20538)
Fixed interpretation of $(HOST_IP) in Zipkin and Datadog tracer address. (Issue #27911)
Removed all Mixer-related features and functionality. This is a scheduled removal of a deprecated Istio services and deployments, as well as Mixer-focused CRDs and component and related functionality. (Issue #25333),(Issue #24300)

Installation

Promoted external control plane to alpha. (Issue #11)
Updated Kiali addon to version 1.26.
Added support for installing and upgrading Istio using Helm 3
Improved multi-network configuration so that labeling a service with topology.istio.io/network=network-name can configure cross-network gateways without using mesh networks.
Improved sidecar injection to not modify the pod securityPolicy.fsGroup which could conflict with existing settings and secret mounts. This option is enabled automatically on Kubernetes 1.19+ and is not supported on older versions. (Issue #26882)
Improved Generated operator manifests for use with kustomize are available in the manifests directory. (Issue #27139)
Updated install script to bypass GitHub API Rate Limiting. (Pull Request #23469)
Added port 15012 to the default list of ports for the istio-ingressgateway Service. (Issue #25933)
Added support for Kubernetes versions 1.16 to 1.19 to Istio 1.8. (Issue #28814)
Added the ability to specify the network for a Pod using the label topology.istio.io/network. This overrides the setting for the cluster’s installation values (values.globalnetwork). If the label isn’t set, it is injected based on the global value for the cluster. (Issue #25500)
Deprecated installation flags values.global.meshExpansion.enabled in favor of user-managed config and values.gateways.istio-ingressgateway.meshExpansionPorts in favor of components.ingressGateways[name=istio-ingressgateway].k8s.service.ports (Issue #25933)
Fixed Istio operator manager to allow configuring RENEW_DEADLINE. (Issue #27509)
Fixed an issue preventing NodePort services from being used as the registryServiceName in meshNetworks.
Removed support for installing third-party telemetry applications with istioctl. These applications (Prometheus, Grafana, Zipkin, Jaeger, and Kiali), often referred to as the Istio addons, must now be installed separately. This does not impact Istio’s ability to produce telemetry for those use in the addons. See Reworking our Addon Integrations for more info. (Issue #23868),(Issue #23583)
Removed istio-telemetry and istio-policy services and deployments from installation by istioctl. (Issue #23868),(Issue #23583)
Fixed Istio Grafana Dashboards queries which have reporter field. (Issue #27595)

istioctl

Improved istioctl analyze to find the exact line number with configuration errors when analyzing yaml files. Before, it would return the first line of the resource with the error. (Issue #22872)
Updated istioctl experimental version and proxy-status to use token security. A new option, --plaintext, has been created for testing without tokens. (Issue #24905)
Added istioctl commands may now refer to pods indirectly, for example istioctl dashboard envoy deployment/httpbin (Issue #26080)
Added io as short name for Istio Operator resources in addition to iop. (Issue #27159)
Added --type for istioctl experimental create-remote-secret to allow user specify type for the created secret.
Added an experimental OpenShift Kubernetes platform profile to istioctl. To install with the OpenShift profile, use istioctl install --set profile=openshift. (OpenShift Platform Setup)(Install OpenShift using istioctl)
Added istioctl bug-report command to generate an archive of Istio and cluster information to assist with debugging. (Issue #26045)
Added new command istioctl experimental istiod log to enable managing logging levels of istiod components. (Issue #25276),(Issue #27797)
Deprecated centralIstiod flag in favor of externalIstiod to better support external control plane model. (Issue #24471)
Fixed an issue which allowed an empty revision flag on install. (Issue #26940)

Documentation

Improved Multicluster install docs to include current best practices, incorporating recent updates to onboarding tooling. In particular, the multi-primary configuration (formerly known as “replicated control planes”) no longer relies on manually configuring the .global stub domain, preferring instead to use *.svc.cluster.local for accessing services throughout the mesh.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Traffic Management

Added config option values.global.proxy.holdApplicationUntilProxyStarts, which causes the sidecar injector to inject the sidecar at the start of the pod’s container list and configures it to block the start of all other containers until the proxy is ready. This option is disabled by default. (Issue #11130)
Added SDS support for Client Certificate and CA certificate used for TLS/mTLS Origination from Egress Gateway using DestinationRule. (Issue #14039)

Security

Improved Trust Domain Validation to validate TCP traffic as well, previously only HTTP traffic was validated. (Issue #26224)
Improved Istio Gateways to allow use of source principal based authorization when the Server’s TLS mode is ISTIO_MUTUAL. (Issue #25818)
Improved VM security. VM identity is now bootstrapped from a short-lived Kubernetes service account token. And VM’s workload certificate is automatically rotated. (Issue #24554)

Telemetry

Added Prometheus metrics to istio-agent. (Issue #22825)
Added Metric customization with istioctl. (Issue #25963)
Added TCP Metrics and Access Logs to Stackdriver. (Issue #23134)
Deprecated installation of telemetry addons by istioctl. These will be disabled by default, and in a future release removed entirely. More information on installing these addons can be found in the Integrations page. (Issue #22762)
Enabled Prometheus metrics merging by default. (Issue #21366)
Fixed Prometheus metrics merging to not drop Envoy metrics during application failures. (Issue #22825)
Fixed Fix unexplained telemetry which affects Kiali graph. This fix increases default outbound protocol sniffing timeout to 5s, which has impact on server first protocol like mysql. (Issue #24379)
Removed the pilot_xds_eds_instances and pilot_xds_eds_all_locality_endpoints Istiod metrics, which were not accurate. (Issue #25154)

Installation

Added RPM packages for running the Istio sidecar on a VM to the release. (Issue #9117)
Added experimental external Istiod support.
Fixed an issue preventing NodePort services from being used as the registryServiceName in meshNetworks.
Improved gateway deployments to run as non-root by default. (Issue #23379)
Improved the operator to run as non-root by default. (Issue #24960)
Improved the operator by specifying a rigorous security context. (Issue #24963)
Improved Istiod to run as non-root by default. (Issue #24961)
Improved Kubernetes strategic merge is used to overlay IstioOperator user files, which improves how list items are handled. (Issue #24432)
Upgraded the CRD and Webhook versions to v1. (Issue #18771),(Issue #18838)

istioctl

Added Allow proxy-status <pod> command for non Kubernetes workloads with proxy config passed in from the --file parameter.
Added a configuration file to hold istioctl default flags. Its default location ($HOME/.istioctl/config.yaml) can be changed using the environment variable ISTIOCONFIG. The new command istioctl experimental config list shows the default flags. (Issue #23868)
Added --revision flag to istioctl operator init and istioctl operator remove commands to support multiple control plane upgrade. (Issue #23479)
Added istioctl x uninstall command to uninstall Istio control plane. (Issue #24360)
Improved istioctl analyze to warn if deprecated mixer resources are present (Issue #24471)
Improved istioctl analyze to warn if DestinationRule is not using CaCertificates to validate server identity.
Improved istioctl validate to check for unknown fields in resources. (Issue #24861)
Improved istioctl install to emit a warning when attempting to install Istio in an old, non supported Kubernetes version. (Issue #26141)
Removed istioctl manifest apply. The simpler install command replaces manifest apply. (Issue #25737)

Documentation changes

Added visual indication if an istio.io page has been tested by istio.io automated tests. (Issue #7672)

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Traffic Management

Added VirtualService delegation. This allows mesh routing configurations to be specified in multiple composable VirtualServices.
Added the new Workload Entry resource. This allows easier configuration for non-Kubernetes workloads to join the mesh.
Added configuration for gateway topology. This addresses providing correct X-Forwarded-For headers and X-Forwarded-Client-Cert headers based on gateway deployment topology .
Added experimental support for the Kubernetes Service APIs.
Added support for using appProtocol to select the protocol for a port introduced in Kubernetes 1.18.
Changed Gateway SDS to be enabled by default. File mounted gateway continues to be available to help users to transition to secure gateway SDS.
Added support for reading certificates from Secrets, pathType, and IngressClass, which provides better support for Kubernetes ingress.
Added a new proxy.istio.io/config annotation to override proxy configuration per pod.
Removed most configuration flags and environment variables for the proxy. These now read directly from the mesh configuration.
Changed the proxy readiness probe to port 15021.
Fixed a bug, which blocked external HTTPS/TCP traffic in some cases.

Security

Added JSON Web Token (JWT) caching to the Istio-agent, which provides better Istio Agent SDS performance.
Fixed the Istio Agent certificate provisioning grace period calculation.
Removed Security alpha API. Security beta API, which was introduced in Istio 1.5, is the only supported security API in Istio 1.6.

Telemetry

Added experimental support for request classification filters. This enables operators to configure new attributes for use in telemetry, based on request information. A primary use case for this feature is labeling of traffic by API method.
Added an experimental mesh-wide tracing configuration API. This API provides control of trace sampling rates, the maximum tag lengths for URL tags, and custom tags extraction for all traces within the mesh.
Added standard Prometheus scrape annotations to proxies and the control plane workloads, which improves the Prometheus integration experience. This removes the need for specialized configuration to discover and consume Istio metrics. More details are available in the operations guide for Prometheus.
Added the ability for mesh operators to add and remove labels used in Istio metrics, based on expressions over the set of available request and response attributes. This improves Istio’s support for customizing v2 metrics generation.
Updated default telemetry v2 configuration to avoid using host header to extract destination service name at the gateway. This prevents unbound cardinality due to an untrusted host header, and implies that destination service labels are going to be omitted for requests that hit blackhole and passthrough at the gateway.
Added automated publishing of Grafana dashboards to grafana.com as part of the Istio release process. Please see the Istio org page for more information.
Updated Grafana dashboards to adapt to the new Istiod deployment model.

Installation

Added support for Istio canary upgrades. See the Upgrade guide for more information.
Removed the legacy Helm charts. For migration from them please see the Upgrade guide.
Added the ability for users to add a custom hostname for istiod.
Changed gateway readiness port used from 15020 to 15021. If you check health on your Istio ingressgateway from your Kubernetes network load balancer you will need to update the port.
Added functionality to save installation state in a CustomResource in the cluster.
Changed the Istio installation to no longer manage the installation namespace, allowing more flexibility.
Removed the separate Citadel, Sidecar Injector, and Galley deployments. These were disabled by default in 1.5, and all functionality has moved into Istiod.
Removed the legacy istio-pilot configurations, such as Service.
Removed ports 15029-15032 from the default ingressgateway. It is recommended to expose telemetry addons by host routing instead.
Removed built in Istio configurations from the installation, including the Gateway, VirtualServices, and mTLS settings.
Added a new profile, called preview, allowing users to try out new experimental features that include WASM enabled telemetry v2.
Added istioctl install command as a replacement for istioctl manifest apply.
Added istiod-remote chart to allow users to experiment with a central Istiod managing a remote data plane.

istioctl

Added better display characteristics for the istioctl command.
Added support for key:value list selection when using –set flag paths.
Added support for deletes and setting non-scalar values when using the Kubernetes overlays patching mechanism.

Documentation changes

Added new and improved Istio documentation. For more information, see Website content changes.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Traffic management

Improved performance of the ServiceEntry resource by avoiding unnecessary full pushes #19305
Improved Envoy sidecar readiness probe to more accurate determine readiness #18164.
Improved performance of Envoy proxy configuration updates via xDS by sending partial updates where possible #18354.
Added an option to configure locality load balancing settings for each targeted service via destination rule#18406.
Fixed an issue where pods crashing would trigger excessive Envoy proxy configuration pushes #18574.
Fixed issues with applications such as headless services to call themselves directly without going through Envoy proxy #19308.
Added detection of iptables failure when using Istio CNI #19534
Added consecutiveGatewayErrors and consecutive5xxErrors as outlier detection options within destination rule #19771.
Improved EnvoyFilter matching performance #19786
Added support for HTTP_PROXY protocol #19919.
Improved iptables setup to use iptables-restore by default #18847.
Improved Gateway performance by filtering unused clusters. This setting is disabled by default #20124.

Security

Graduated SDS to stable and enabled by default. It provides identity provisioning for Istio Envoy proxies.
Added Beta authentication API. The new API separates peer (i.e mutual TLS) and origin (JWT) authentication into PeerAuthentication and RequestAuthentication respectively. Both new APIs are workload-oriented, as opposed to service-oriented in alpha AuthenticationPolicy.
Added deny semantics and exclusion matching to Authorization Policy.
Graduated auto mutual TLS from alpha to beta. This feature is now enabled by default.
Improved SDS security by merging Node Agent with Pilot Agent as Istio Agent and removing cross-pod UDS, which no longer requires users to deploy Kubernetes pod security policies for UDS connections.
Improved Istio by including certificate provisioning functionality within Istiod.
Added Support Kubernetes first-party-jwt as a fallback token for CSR authentication in clusters where third-party-jwt is not supported.
Added Support Istio CA and Kubernetes CA to provision certificates for the control plane, configurable via values.global.pilotCertProvider.
Added Istio Agent provisions a key and certificates for Prometheus.

Telemetry

Added TCP protocol support for v2 telemetry.
Added gRPC response status code support in metrics/logs.
Added support for Istio Canonical Service.
Improved stability of v2 telemetry pipeline.
Added alpha-level support for configurability in v2 telemetry.
Added support for populating AWS platform metadata in Envoy node metadata.
Improved Stackdriver adapter for Mixer to support configurable flush intervals for tracing data.
Added support for a headless collector service to the Jaeger addon.
Fixed kubernetesenv adapter to provide proper support for pods that contain a dot in their name.
Improved the Fluentd adapter for Mixer to provide millisecond-resolution in exported timestamps.

Configuration management

Operator

Replaced the alpha IstioControlPlane API with the new IstioOperator API to align with existing MeshConfig API.
Added istioctl operator init and istioctl operator remove commands.
Improved reconciliation speed with caching operator#667.

`istioctl`

Graduated Istioctl Analyze out of experimental.
Added various analyzers: mutual TLS, JWT, ServiceAssociation, Secret, sidecar image, port name and policy deprecated analyzers.
Updated more validation rules for RequestAuthentication.
Added a new flag -A|--all-namespaces to istioctl analyze to analyze the entire cluster.
Added support for analyzing content passed via stdin to istioctl analyze.
Added istioctl analyze -L to show a list of all analyzers available.
Added the ability to suppress messages from istioctl analyze.
Added structured format options to istioctl analyze.
Added links to relevant documentation to istioctl analyze output.
Updated annotation methods provided by Istio API in Istioctl Analyze.
Updated istioctl analyze now loads files from a directory.
Updated istioctl analyze to try to associate message with their source filename.
Updated istioctl analyze to print the namespace that is being analyzed.
Updated istioctl analyze to analyze in-cluster resources by default.
Fixed bug where istioctl analyze suppressed cluster-level resource messages.
Added support for multiple input files to istioctl manifest.
Replaced the IstioControlPlane API with the IstioOperator API.
Added selector for istioctl dashboard.
Added support for slices and lists in istioctl manifest --set flag.
Added support for istioctl manifest to read profiles from stdin.
Added a docker/istioctl image #19079.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Traffic management

Added support for mirroring a percentage of traffic.
Improved the Envoy sidecar. The Envoy sidecar now exits when it crashes. This change makes it easier to see whether or not the Envoy sidecar is healthy.
Improved Pilot to skip sending redundant configuration to Envoy when no changes are required.
Improved headless services to avoid conflicts with different services on the same port.
Disabled default circuit breakers.
Updated the default regex engine to re2. Please see the Upgrade Notes for details.

Security

Added the v1beta1 authorization policy model for enforcing access control. This will eventually replace the v1alpha1 RBAC policy.
Added experimental support for automatic mutual TLS to enable mutual TLS without destination rule configuration.
Added experimental support for authorization policy trust domain migration.
Added experimental DNS certificate management to securely provision and manage DNS certificates signed by the Kubernetes CA.
Improved Citadel to periodically check and rotate the expired root certificate when running in self-sign CA mode.
Updated JWT authentication to treat space-delimited claim as a list of claims.

Telemetry

Added experimental in-proxy telemetry reporting to Stackdriver.
Improved support for in-proxy Prometheus generation of HTTP service metrics (from experimental to alpha).
Improved telemetry collection for blocked and passthrough external service traffic.
Added the option to configure stat patterns for Envoy stats.
Added the inbound and outbound prefixes to the Envoy HTTP stats to specify traffic direction.
Improved reporting of telemetry for traffic that goes through an egress gateway.

Configuration management

Added multiple validation checks to the istioctl analyze sub-command.
Added the experimental option to enable validation messages for Istio resource statuses.
Added OpenAPI v3 schema validation of Custom Resource Definitions (CRDs). Please see the Upgrade Notes for details.
Added client-go libraries to access Istio APIs.

Installation

Added the experimental operator controller for dynamic updates to an Istio installation.
Removed the proxy_init Docker image. Instead, the istio-init container reuses the proxyv2 image.
Updated the base image to ubuntu:bionic.

`istioctl`

Added the istioctl proxy-config logs sub-command retrieve and update Envoy logging levels.
Updated the istioctl authn tls-check sub-command to display which policy is in use.
Added the experimental istioctl experimental wait sub-command to have Istio wait until it has pushed a configuration to all Envoy sidecars.
Added the experimental istioctl experimental multicluster sub-command to help manage Istio across multiple clusters.
Added the experimental istioctl experimental post-install webhook sub-command to securely manage webhook configurations.
Added the experimental istioctl experimental upgrade sub-command to perform upgrades of Istio.
Improved the istioctl version sub-command. It now shows the Envoy proxy versions.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Installation

Added experimental manifest and profile commands to install and manage the Istio control plane for evaluation.

Traffic management

Added automatic protocol determination of HTTP or TCP for outbound traffic when ports are not named according to Istio’s conventions.
Added a mode to the Gateway API for mutual TLS operation.
Fixed issues present when a service communicates over the network first in permissive mutual TLS mode for protocols like MySQL and MongoDB.
Improved Envoy proxy readiness checks. They now check Envoy’s readiness status.
Improved container ports are no longer required in the pod spec. All ports are captured by default.
Improved the EnvoyFilter API. You can now add or update all configurations.
Improved the Redis load balancer to now default to MAGLEV when using the Redis proxy.
Improved load balancing to direct traffic to the same region and zone by default.
Improved Pilot by reducing CPU utilization. The reduction approaches 90% depending on the specific deployment.
Improved the ServiceEntry API to allow for the same hostname in different namespaces.
Improved the Sidecar API to customize the OutboundTrafficPolicy policy.

Security

Added trust domain validation for services using mutual TLS. By default, the server only authenticates the requests from the same trust domain.
Added [labels]((/docs/ops/configuration/mesh/secret-creation/) to control service account secret generation by namespace.
Added SDS support to deliver the private key and certificates to each Istio control plane service.
Added support for introspection to Citadel.
Added metrics to the /metrics endpoint of Citadel Agent on port 15014 to monitor the SDS service.
Added diagnostics to the Citadel Agent using the /debug/sds/workload and /debug/sds/gateway on port 8080.
Improved the ingress gateway to load the trusted CA certificate from a separate secret when using SDS.
Improved SDS security by enforcing the usage of Kubernetes Trustworthy JWTs.
Improved Citadel Agent logs by unifying the logging pattern.
Removed support for Istio SDS when using Kubernetes versions earlier than 1.13.
Removed integration with Vault CA temporarily. SDS requirements caused the temporary removal but we will reintroduce Vault CA integration in a future release.
Enabled the Envoy JWT filter by default to improve security and reliability.

Telemetry

Added Access Log Service ALS support for Envoy gRPC.
Added a Grafana dashboard for Citadel monitoring.
Added metrics for monitoring the sidecar injector webhook.
Added control plane metrics to monitor Istio’s configuration state.
Added telemetry reporting for traffic destined to the Passthrough and BlackHole clusters.
Added alpha support for in-proxy generation of service metrics using Prometheus.
Added alpha support for environmental metadata in Envoy node metadata.
Added alpha support for Proxy Metadata Exchange.
Added alpha support for the OpenCensus trace driver.
Improved reporting for external services by removing requirements to add a service entry.
Improved the mesh dashboard to provide monitoring of Istio’s configuration state.
Improved the Pilot dashboard to expose additional key metrics to more clearly identify errors.
Removed deprecated Adapter and Template custom resource definitions (CRDs).
Deprecated the HTTP API spec used to produce API attributes. We will remove support for producing API attributes in Istio 1.4.

Policy

Improved rate limit enforcement to allow communication when the quota backend is unavailable.

Configuration management

Fixed Galley to stop too many gRPC pings from closing connections.
Improved Galley to avoid control plane upgrade failures.

`istioctl`

Added istioctl experimental manifest to manage the new experimental install manifests.
Added istioctl experimental profile to manage the new experimental install profiles.
Added istioctl experimental metrics
Added istioctl experimental describe pod to describe an Istio pod’s configuration.
Added istioctl experimental add-to-mesh to add Kubernetes services or virtual machines to an existing Istio service mesh.
Added istioctl experimental remove-from-mesh to remove Kubernetes services or virtual machines from an existing Istio service mesh.
Promoted the istioctl experimental convert-ingress command to istioctl convert-ingress.
Promoted the istioctl experimental dashboard command to istioctl dashboard.

Miscellaneous

Added new images based on distroless base images.
Improved the Istio CNI Helm chart to have consistent versions with Istio.
Improved Kubernetes Jobs behavior. Kubernetes Jobs now exit correctly when the job manually calls the /quitquitquit endpoint.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

General

Added traffic.sidecar.istio.io/includeInboundPorts annotation to eliminate the need for service owner to declare containerPort in the deployment yaml file. This will become the default in a future release.
Added IPv6 experimental support for Kubernetes clusters.

Traffic management

Improved locality based routing in multicluster environments.
Improved outbound traffic policy in ALLOW_ANY mode. Traffic for unknown HTTP/HTTPS hosts on an existing port will be forwarded as is. Unknown traffic will be logged in Envoy access logs.
Added support for setting HTTP idle timeouts to upstream services.
Improved Sidecar support for NONE mode (without iptables) .
Added ability to configure the DNS refresh rate for sidecar Envoys, to reduce the load on the DNS servers.
Graduated Sidecar API from Alpha to Alpha API and Beta runtime.

Security

Improved extend the default lifetime of self-signed Citadel root certificates to 10 years.
Added Kubernetes health check prober rewrite per deployment via sidecar.istio.io/rewriteAppHTTPProbers: "true" in the PodSpec annotation.
Added support for configuring the secret paths for Istio mutual TLS certificates. Refer here for more details.
Added support for PKCS 8 private keys for workloads, enabled by the flag pkcs8-keys on Citadel.
Improved JWT public key fetching logic to be more resilient to network failure.
Fixed SAN field in workload certificates is set as critical. This fixes the issue that some custom certificate verifiers cannot verify Istio certificates.
Fixed mutual TLS probe rewrite for HTTPS probes.
Graduated SNI with multiple certificates support at ingress gateway from Alpha to Stable.
Graduated certification management on Ingress Gateway from Alpha to Beta.

Telemetry

Added Full support for control over Envoy stats generation, based on stats prefixes, suffixes, and regular expressions through the use of annotations.
Changed Prometheus generated traffic is excluded from metrics.
Added support for sending traces to Datadog.
Graduated distributed tracing from Beta to Stable.

Policy

Fixed Mixer basedTCP Policy enforcement.
Graduated Authorization (RBAC) from Alpha to Alpha API and Beta runtime.

Configuration management

Improved validation of Policy & Telemetry CRDs.
Graduated basic configuration resource validation from Alpha to Beta.

Installation and upgrade

Updated default proxy memory limit size(global.proxy.resources.limits.memory) from 128Mi to 1024Mi to ensure proxy has sufficient memory.
Added pod anti-affinity and toleration support to all of our control plane components.
Added sidecarInjectorWebhook.neverInjectSelector and sidecarInjectorWebhook.alwaysInjectSelector to allow users to further refine whether workloads should have sidecar automatically injected or not, based on label selectors.
Added global.logging.level and global.proxy.logLevel to allow users to easily configure logs for control plane and data plane components globally.
Added support to configure the Datadog location via global.tracer.datadog.address.
Removed Previously deprecated Adapter and Template CRDs are disabled by default. Use mixer.templates.useTemplateCRDs=true and mixer.adapters.useAdapterCRDs=true install options to re-enable them.

Refer to the installation option change page to view the complete list of changes.

`istioctl` and `kubectl`

Graduated istioctl verify-install out of experimental.
Improved istioctl verify-install to validate if a given Kubernetes environment meets Istio’s prerequisites.
Added auto-completion support to istioctl.
Added istioctl experimental dashboard to allow users to easily open the web UI of any Istio addons.
Added istioctl x alias to conveniently run istioctl experimental command.
Improved istioctl version to report both Istio control plane and istioctl version info by default.
Improved istioctl validate to validate Mixer configuration and supports deep validation with referential integrity.

Miscellaneous

Added Istio CNI support to setup sidecar network redirection and remove the use of istio-init containers requiring NET_ADMIN capability.
Added a new experimental ‘a-la-carte’ Istio installer to enable users to install and upgrade Istio with desired isolation and security.
Added environment variable and configuration file support for configuring Galley, in addition to command-line flags.
Added ControlZ support to visualize the state of the MCP Server in Galley.
Added the enableServiceDiscovery command-line flag to control the service discovery module in Galley.
Added InitialWindowSize and InitialConnWindowSize parameters to Galley and Pilot to allow fine-tuning of MCP (gRPC) connection settings.
Graduated configuration processing with Galley from Alpha to Beta.

Change Notes

Mon, 01 Jan 0001 00:00:00 +0000

Incompatible changes from 1.0

In addition to the new features and improvements listed below, Istio 1.1 has introduced a number of significant changes from 1.0 that can alter the behavior of applications. A concise list of these changes can be found in the upgrade notice.

Upgrades

We recommend a manual upgrade of the control plane and data plane to 1.1. See the upgrades documents for more information.

Installation

CRD Install Separated from Istio Install. Placed Istio’s Custom Resource Definitions (CRDs) into the istio-init Helm chart. Placing the CRDs in their own Helm chart preserves the data continuity of the custom resource content during the upgrade process and further enables Istio to evolve beyond a Helm-based installation.
Installation Configuration Profiles. Added several installation configuration profiles to simplify the installation process using well-known and well-tested patterns. Learn more about the better user experience afforded by the installation profile feature.
Improved Multicluster Integration. Consolidated the 1.0 istio-remote chart previously used for multicluster VPN and multicluster split horizon remote cluster installation into the Istio Helm chart simplifying the operational experience.

Traffic management

New Sidecar Resource. The new sidecar resource enables more fine-grained control over the behavior of the sidecar proxies attached to workloads within a namespace. In particular it adds support to limit the set of services a sidecar will send traffic to. This reduces the amount of configuration computed and transmitted to the proxy, improving startup time, resource consumption and control-plane scalability. For large deployments, we recommend adding a sidecar resource per namespace. Controls are also provided for ports, protocols and traffic capture for advanced use-cases.
Restrict Visibility of Services. Added the new exportTo feature which allows service owners to control which namespaces can reference their services. This feature is added to ServiceEntry, VirtualService and is also supported on a Kubernetes Service via the networking.istio.io/exportTo annotation.
Namespace Scoping. When referring to a VirtualService in a Gateway we use DNS based name matching in our configuration model. This can be ambiguous when more than one namespace defines a virtual service for the same host name. To resolve ambiguity it is now possible to explicitly scope these references by namespace using a syntax of the form [{namespace-name}]/{hostname-match} in the hosts field. The equivalent capability is also available in Sidecar for egress.
Updates to ServiceEntry Resources. Added support to specify the locality of a service and the associated SAN to use with mutual TLS. Service entries with HTTPS ports no longer need an additional virtual service to enable SNI-based routing.
Locality-Aware Routing. Added full support for routing to services in the same locality before picking services in other localities. See Locality Load Balancer Settings
Refined Multicluster Routing. Simplified the multicluster setup and enabled additional deployment modes. You can now connect multiple clusters simply using their ingress gateways without needing pod-level VPNs, deploy control planes in each cluster for high-availability cases, and span a namespace across several clusters to create global namespaces. Locality-aware routing is enabled by default in the high-availability control plane solution.
Istio Ingress Deprecated. Removed the previously deprecated Istio ingress. Refer to the Securing Kubernetes Ingress with Cert-Manager example for more details on how to use Kubernetes Ingress resources with gateways.
Performance and Scalability Improvements. Tuned the performance and scalability of Istio and Envoy. Read more about Performance and Scalability enhancements.
Access Logging Off by Default. Disabled the access logs for all Envoy sidecars by default to improve performance.

Security

Readiness and Liveness Probes. Added support for Kubernetes’ HTTP readiness and liveness probes when mutual TLS is enabled.
Cluster RBAC Configuration. Replaced the RbacConfig resource with the ClusterRbacConfig resource to implement the correct cluster scope. See Migrating RbacConfig to ClusterRbacConfig. for migration instructions.
Identity Provisioning Through SDS. Added SDS support to provide stronger security with on-node key generation and dynamic certificate rotation without restarting Envoy.
Authorization for TCP Services. Added support of authorization for TCP services in addition to HTTP and gRPC services. See Authorization for TCP Services for more information.
Authorization for End-User Groups. Added authorization based on groups claim or any list-typed claims in JWT. See Authorization for JWT for more information.
External Certificate Management on Ingress Gateway Controller. Added a controller to dynamically load and rotate external certificates.
Custom PKI Integration. Added Vault PKI integration with support for Vault-protected signing keys and ability to integrate with existing Vault PKIs.
Customized (non cluster.local) Trust Domains. Added support for organization- or cluster-specific trust domains in the identities.

Policies and telemetry

Policy Checks Off By Default. Changed policy checks to be turned off by default to improve performance for most customer scenarios. Enabling Policy Enforcement details how to turn on Istio policy checks, if needed.
Kiali. Replaced the Service Graph addon with Kiali to provide a richer visualization experience. See the Kiali task for more details.
Reduced Overhead. Added several performance and scale improvements including:
- Significant reduction in default collection of Envoy-generated statistics.
- Added load-shedding functionality to Mixer workloads.
- Improved the protocol between Envoy and Mixer.
Control Headers and Routing. Added the option to create adapters to influence the headers and routing of an incoming request. See the Control Headers and Routing task for more information.
Out of Process Adapters. Added the out-of-process adapter functionality for production use. As a result, we deprecated the in-process adapter model in this release. All new adapter development should use the out-of-process model moving forward.
Tracing Improvements. Performed many improvements in our overall tracing story:
- Trace ids are now 128 bit wide.
- Added support for sending trace data to Lightstep
- Added the option to disable tracing for Mixer-backed services entirely.
- Added policy decision-aware tracing.
Default TCP Metrics. Added default metrics for tracking TCP connections.
Reduced Load Balancer Requirements for Addons. Stopped exposing addons via separate load balancers. Instead, addons are exposed via the Istio gateway. To expose addons externally using either HTTP or HTTPS protocols, please use the Addon Gateway documentation.
Secure Addon Credentials. Changed storage of the addon credentials. Grafana, Kiali, and Jaeger passwords and username are now stored in Kubernetes secrets for improved security and compliance.
More Flexibility with statsd Collector. Removed the built-in statsd collector. Istio now supports bring your own statsd for improved flexibility with existing Kubernetes deployments.

Configuration management

Galley. Added Galley as the primary configuration ingestion and distribution mechanism within Istio. It provides a robust model to validate, transform, and distribute configuration states to Istio components insulating the Istio components from Kubernetes details. Galley uses the Mesh Configuration Protocol to interact with components.
Monitoring Port. Changed Galley’s default monitoring port from 9093 to 15014.

`istioctl` and `kubectl`

Validate Command. Added the istioctl validate command for offline validation of Istio Kubernetes resources.
Verify-Install Command. Added the istioctl verify-install command to verify the status of an Istio installation given a specified installation YAML file.
Deprecated Commands. Deprecated the istioctl create, istioctl replace, istioctl get, and istioctl delete commands. Use the kubectl equivalents instead. Deprecated the istioctl gen-deploy command too. Use a helm template instead. Release 1.2 will remove these commands.
Short Commands. Included short commands in kubectl for gateways, virtual services, destination rules and service entries.

Updates to how Istio security releases are handled: Patch Tuesday, embargoes, and 0-days

Jacob Delgado (Aspen Mesh) — Tue, 11 May 2021 00:00:00 +0000

While most of the work in the Istio Product Security Working Group is done behind the scenes, we are listening to the community in setting expectations for security releases. We understand that it is difficult for mesh administrators, operators and vendors to be aware of security bulletins and security releases.

We currently disclose vulnerabilities and security releases via numerous channels:

istio.io via our Release Announcements and Security Bulletins
Discuss
announcements channel on Slack
Twitter
RSS

When operating any software, it is preferable to plan for possible downtime when upgrading. Given the work that the Istio community is doing around Day 2 operations in 2021, the Environments working group has done a good job to streamline many upgrade issues users have seen. The Product Security Working Group intends to help Day 2 operations by having routine security release days so that upgrade operations can be planned in advance for our users.

Patch Tuesdays

The Product Security working group is intending to ship a security release the 2nd Tuesday of each month. These security releases may contain fixes for multiple CVEs. It is the intent of the Product Security working group to have these security releases not contain any other fixes, although that may not always be possible.

When the Product Security working group intends to ship an upcoming security patch, an announcement will be made on the Istio discussion board 2 weeks prior to release. If you’re running Istio in production, we suggest you watch the Announcements category to be notified of such a release. If no such announcement is made there will not be a security release for that month, barring some exceptions listed below.

First Patch Tuesday

We are pleased to announce that Istio 1.9.5, and the final release of Istio 1.8, 1.8.6, are the first security releases to fit this pattern. As Istio 1.10 will be shipping soon we are intending to continue this new tradition in June.

These releases fix 3 CVEs. Please see the release pages for information regarding the specific CVEs fixed.

Unscheduled security releases

0-day vulnerabilities

Unfortunately, 0-day vulnerabilities cannot be planned. Upon disclosure, the Product Security Working Group will need to issue an out-of-band security release. The above methods will be used to disclose such issues, so please use at least one of them to be notified of such disclosures.

Third party embargoes

Similar to 0-day vulnerabilities, security releases can be dictated by third party embargoes, namely Envoy. When this occurs, Istio will release a same-day patch once the embargo is lifted.

Security Best Practices

The Istio Security Best Practices has seen many improvements over the past few months. We recommend you check it regularly, as many of our recent security bulletins can be mitigated by utilizing methods discussed in the Security Best Practices page.

Early Disclosure List

If you meet the criteria to be a part of the Istio Early Disclosure list, please apply for membership. Patches for upcoming security releases will be made available to the early disclosure list ~2 weeks prior to Istio’s Patch Tuesday.

There will be times when an upcoming Istio security release will also need patches from Envoy. We cannot redistribute Envoy patches due to their embargo. Please refer to Envoy’s guidance on how to join their early disclosure list.

Security Feedback

The Product Security Working Group holds bi-weekly meetings on Tuesdays from 9:00-9:30 Pacific. For more information see the Istio Working Group Calendar.

Our next public meeting will be held on May 25, 2021. Please join us!

Use discovery selectors to configure namespaces for your Istio service mesh

Lin Sun (Solo.io), Christian Posta (Solo.io), Harvey Xia (Solo.io) — Fri, 30 Apr 2021 00:00:00 +0000

As users move their services to run in the Istio service mesh, they are often surprised that the control plane watches and processes all of the Kubernetes resources, from all namespaces in the cluster, by default. This can be an issue for very large clusters with lots of namespaces and deployments, or even for a moderately sized cluster with rapidly churning resources (for example, Spark jobs).

Both in the community as well as for our large-scale customers at Solo.io, we need a way to dynamically restrict the set of namespaces that are part of the mesh so that the Istio control plane only processes resources in those namespaces. The ability to restrict the namespaces enables Istiod to watch and push fewer resources and associated changes to the sidecars, thus improving the overall performance on the control plane and data plane.

Background

By default, Istio watches all Namespaces, Services, Endpoints and Pods in a cluster. For example, in my Kubernetes cluster, I deployed the sleep service in the default namespace, and the httpbin service in the ns-x namespace. I’ve added the sleep service to the mesh, but I have no plan to add the httpbin service to the mesh, or have any service in the mesh interact with the httpbin service.

Use istioctl proxy-config endpoint command to display all the endpoints for the sleep deployment:

Endpoints for Sleep Deployment

Note that the httpbin service endpoint in the ns-x namespace is in the list of discovered endpoints. This may not be an issue when you only have a few services. However, when you have hundreds of services that don’t interact with any of the services running in the Istio service mesh, you probably don’t want your Istio control plane to watch these services and send their information to the sidecars of your services in the mesh.

Introducing Discovery Selectors

Starting with Istio 1.10, we are introducing the new discoverySelectors option to MeshConfig, which is an array of Kubernetes selectors. The exact type is []LabelSelector, as defined here, allowing both simple selectors and set-based selectors. These selectors apply to labels on namespaces.

You can configure each label selector for expressing a variety of use cases, including but not limited to:

Arbitrary label names/values, for example, all namespaces with label istio-discovery=enabled
A list of namespace labels using set-based selectors which carries OR semantics, for example, all namespaces with label istio-discovery=enabled OR region=us-east1
Inclusion and/or exclusion of namespaces, for example, all namespaces with label istio-discovery=enabled AND label key app equal to helloworld

Note: discoverySelectors is not a security boundary. Istiod will continue to have access to all namespaces even when you have configured your discoverySelectors.

Discovery Selectors in Action

Assuming you know which namespaces to include as part of the service mesh, as a mesh administrator, you can configure discoverySelectors at installation time or post-installation by adding your desired discovery selectors to Istio’s MeshConfig resource. For example, you can configure Istio to discover only the namespaces that have the label istio-discovery=enabled.

Using our examples earlier, let’s label the default namespace with label istio-discovery=enabled.
```
$ kubectl label namespace default istio-discovery=enabled
```

Use istioctl to apply the yaml with discoverySelectors to update your Istio installation. Note, to avoid any impact to your stable environment, we recommend that you use a different revision for your Istio installation:

$ istioctl install --skip-confirmation -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
spec:
# You may override parts of meshconfig by uncommenting the following lines.
meshConfig:
discoverySelectors:
- matchLabels:
istio-discovery: enabled
EOF

Display the endpoint configuration for the sleep deployment:

Endpoints for Sleep Deployment With Discovery Selectors

Note this time the httpbin service in the ns-x namespace is NOT in the list of discovered endpoints, along with many other services that are not in the default namespace. If you display routes (or cluster or listeners) information for the sleep deployment, you will also notice much less configuration is returned:

Routes for Sleep Deployment With Discovery Selectors

You can use matchLabels to configure multiple labels with AND semantics or use matchLabels sets to configure OR semantics among multiple labels. Whether you deploy services or pods to namespaces with different sets of labels or multiple application teams in your organization use different labeling conventions, discoverySelectors provides the flexibility you need. Furthermore, you could use matchLabels and matchExpressions together per our documentation. Refer to the Kubernetes selector docs for additional detail on selector semantics.

Discovery Selectors vs Sidecar Resource

The discoverySelectors configuration enables users to dynamically restrict the set of namespaces that are part of the mesh. A Sidecar resource also controls the visibility of sidecar configurations and what gets pushed to the sidecar proxy. What are the differences between them?

The discoverySelectors configuration declares what Istio control plane watches and processes. Without discoverySelectors configuration, the Istio control plane watches and processes all namespaces/services/endpoints/pods in the cluster regardless of the sidecar resources you have.
discoverySelectors is configured globally for the mesh by the mesh administrators. While Sidecar resources can also be configured for the mesh globally by the mesh administrators in the MeshConfig root namespace, they are commonly configured by service owners for their namespaces.

You can use discoverySelectors with Sidecar resources. You can use discoverySelectors to configure at the mesh-wide level what namespaces the Istio control plane should watch and process. For these namespaces in the Istio service mesh, you can create Sidecar resources globally or per namespace to further control what gets pushed to the sidecar proxies. Let us add Bookinfo services to the ns-y namespace in the mesh as shown in the diagram below. discoverySelectors enables us to define the default and ns-y namespaces are part of the mesh. How can we configure the sleep service not to see anything other than the default namespace? Adding a Sidecar resource for the default namespace, we can effectively configure the sleep sidecar to only have visibility to the clusters/routes/listeners/endpoints associated with its current namespace plus any other required namespaces.

Discovery Selectors vs Sidecar Resource

Wrapping up

Discovery selectors are powerful configurations to tune the Istio control plane to only watch and process specific namespaces. If you don’t want all namespaces in your Kubernetes cluster to be part of the service mesh or you have multiple Istio service meshes within your Kubernetes cluster, we highly recommend that you explore this configuration and reach out to us for feedback on our Istio slack or GitHub.

Upcoming networking changes in Istio 1.10

John Howard (Google) — Thu, 15 Apr 2021 00:00:00 +0000

Background

While Kubernetes networking is customizable, a typical pod’s network will look like this:

A pod's network

An application may choose to bind to either the loopback interface lo (typically binding to 127.0.0.1), or the pods network interface eth0 (typically to the pod’s IP), or both (typically binding to 0.0.0.0).

Binding to lo allows calls such as curl localhost to work from within the pod. Binding to eth0 allows calls to the pod from other pods.

Typically, an application will bind to both. However, applications which have internal logic, such as an admin interface may choose to bind to only lo to avoid access from other pods. Additionally, some applications, typically stateful applications, choose to bind only to eth0.

Current behavior

In Istio prior to release 1.10, the Envoy proxy, running in the same pod as the application, binds to the eth0 interface and redirects all inbound traffic to the lo interface.

A pod's network with Istio today

This has two important side effects that cause the behavior to differ from standard Kubernetes:

Applications binding only to lo will receive traffic from other pods, when otherwise this is not allowed.
Applications binding only to eth0 will not receive traffic.

Applications that bind to both interfaces (which is typical) will not be impacted.

Future behavior

Starting with Istio 1.10, the networking behavior is changed to align with the standard behavior present in Kubernetes.

A pod's network with Istio in the future

Here we can see that the proxy no longer redirects the traffic to the lo interface, but instead forwards it to the application on eth0. As a result, the standard behavior of Kubernetes is retained, but we still get all the benefits of Istio. This change allows Istio to get closer to its goal of being a drop-in transparent proxy that works with existing workloads with zero configuration. Additionally, it avoids unintended exposure of applications binding only to lo.

Am I impacted?

For new users, this change should only be an improvement. However, if you are an existing user, you may have come to depend on the old behavior, intentionally or accidentally.

To help detect these situations, we have added a check to find pods that will be impacted. You can run the istioctl experimental precheck command to get a report of any pods binding to lo on a port exposed in a Service. This command is available in Istio 1.10+. Without action, these ports will no longer be accessible upon upgrade.

$ istioctl experimental precheck
Error [IST0143] (Pod echo-local-849647c5bd-g9wxf.default) Port 443 is exposed in a Service but listens on localhost. It will not be exposed to other pods.
Error [IST0143] (Pod echo-local-849647c5bd-g9wxf.default) Port 7070 is exposed in a Service but listens on localhost. It will not be exposed to other pods.
Error: Issues found when checking the cluster. Istio may not be safe to install or upgrade.
See https://istio.io/latest/docs/reference/config/analysis for more information about causes and resolutions.

Migration

If you are currently binding to lo, you have a few options:

Switch your application to bind to all interfaces (0.0.0.0 or ::).

Explicitly configure the port using the Sidecar ingress configuration to send to lo, preserving the old behavior.

For example, to configure request to be sent to localhost for the ratings application:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
name: ratings
spec:
workloadSelector:
labels:
app: ratings
ingress:
- port:
number: 8080
protocol: HTTP
name: http
defaultEndpoint: 127.0.0.1:8080

Disable the change entirely with the PILOT_ENABLE_INBOUND_PASSTHROUGH=false environment variable in Istiod, to enable the same behavior as prior to Istio 1.10. This option will be removed in the future.

Istio and Envoy WebAssembly Extensibility, One Year On

Pengyuan Bian (Google) — Fri, 05 Mar 2021 00:00:00 +0000

One year ago today, in the 1.5 release, we introduced WebAssembly-based extensibility to Istio. Over the course of the year, the Istio, Envoy, and Proxy-Wasm communities have continued our joint efforts to make WebAssembly (Wasm) extensibility stable, reliable, and easy to adopt. Let’s walk through the updates to Wasm support through the Istio 1.9 release, and our plans for the future.

WebAssembly support merged in upstream Envoy

After adding experimental support for Wasm and the WebAssembly for Proxies (Proxy-Wasm) ABI to Istio’s fork of Envoy, we collected some great feedback from our community of early adopters. This, combined with the experience gained from developing core Istio Wasm extensions, helped us mature and stabilize the runtime. These improvements unblocked merging Wasm support directly into Envoy upstream in October 2020, allowing it to become part of all official Envoy releases. This was a significant milestone, since it indicates that:

The runtime is ready for wider adoption.
The programming ABI/API, extension configuration API, and runtime behavior, are becoming stable.
You can expect a larger community of adoption and support moving forward.

`wasm-extensions` Ecosystem Repository

As an early adopter of the Envoy Wasm runtime, the Istio Extensions and Telemetry working group gained a lot of experience in developing extensions. We built several first-class extensions, including metadata exchange, Prometheus stats, and attribute generation. In order to share our learning more broadly, we created a wasm-extensions repository in the istio-ecosystem organization. This repository serves two purposes:

It provides canonical example extensions, covering several highly demanded features (such as basic authentication).
It provides a guide for Wasm extension development, testing, and release. The guide is based on the same build tool chains and test frameworks that are used, maintained and tested by the Istio extensibility team.

The guide currently covers WebAssembly extension development and unit testing with C++, as well as integration testing with a Go test framework, which simulates a real runtime by running a Wasm module with the Istio proxy binary. In the future, we will also add several more canonical extensions, such as an integration with Open Policy Agent, and header manipulation based on JWT tokens.

Wasm module distribution via the Istio Agent

Prior to Istio 1.9, Envoy remote data sources were needed to distribute remote Wasm modules to the proxy. In this example, you can see two EnvoyFilter resources are defined: one to add a remote fetch Envoy cluster, and the other one to inject a Wasm filter into the HTTP filter chain. This method has a drawback: if remote fetch fails, either due to bad configuration or transient error, Envoy will be stuck with the bad configuration. If a Wasm extension is configured as fail closed, a bad remote fetch will stop Envoy from serving. To fix this issue, a fundamental change is needed to the Envoy xDS protocol to make it allow asynchronous xDS responses.

Istio 1.9 provides a reliable distribution mechanism out of the box by leveraging the xDS proxy inside istio-agent and Envoy’s Extension Configuration Discovery Service (ECDS).

istio-agent intercepts the extension config resource update from istiod, reads the remote fetch hint from it, downloads the Wasm module, and rewrites the ECDS configuration with the path of the downloaded Wasm module. If the download fails, istio-agent will reject the ECDS update and prevent a bad configuration reaching Envoy. For more detail, please see our docs on Wasm module distribution.

Remote Wasm module fetch flow

Istio Wasm SIG and Future Work

Although we have made a lot of progress on Wasm extensibility, there are still many aspects of the project that remain to be completed. In order to consolidate the efforts from various parties and better tackle the challenges ahead, we have formed an Istio WebAssembly SIG, with aim of providing a standard and reliable way for Istio to consume Wasm extensions. Here are some of the things we are working on:

A first-class extension API: Currently Wasm extensions needs to be injected via Istio’s EnvoyFilter API. A first-class extension API will make using Wasm with Istio easier, and we expect this to be introduced in Istio 1.10.
Distribution artifacts interoperability: Built on top of Solo.io’s WebAssembly OCI image spec effort, a standard Wasm artifacts format will make it easy to build, pull, publish, and execute.
Container Storage Interface (CSI) based artifacts distribution: Using istio-agent to distribute modules is easy for adoption, but may not be efficient as each proxy will keep a copy of the Wasm module. As a more efficient solution, with Ephemeral CSI, a DaemonSet will be provided which could configure storage for pods. Working similarly to a CNI plugin, a CSI driver would fetch the Wasm module out-of-band from the xDS flow and mount it inside the rootfs when the pod starts up.

If you would like to join us, the group will meet every other week Tuesdays at 2PM PT. You can find the meeting on the Istio working group calendar.

We look forward to seeing how you will use Wasm to extend Istio!

Migrate pre-Istio 1.4 Alpha security policy to the current APIs

Yangmin Zhu (Google), Craig Box (Google) — Wed, 03 Mar 2021 00:00:00 +0000

In versions of Istio prior to 1.4, security policy was configured using v1alpha1 APIs (MeshPolicy, Policy, ClusterRbacConfig, ServiceRole and ServiceRoleBinding). After consulting with our early adopters, we made major improvements to the policy system and released v1beta1 APIs along with Istio 1.4. These refreshed APIs (PeerAuthentication, RequestAuthentication and AuthorizationPolicy) helped standardize how we define policy targets in Istio, helped users understand where policies were applied, and cut the number of configuration objects required.

The old APIs were deprecated in Istio 1.4. Two releases after the v1beta1 APIs were introduced, Istio 1.6 removed support for the v1alpha1 APIs.

If you are using a version of Istio prior to 1.6 and you want to upgrade, you will have to migrate your alpha security policy objects to the beta API. This tutorial will help you make that move.

Overview

Your control plane must first be upgraded to a version that supports the v1beta1 security policy.

It is recommended to first upgrade to Istio 1.5 as a transitive version, because it is the only version that supports both v1alpha1 and v1beta1 security policies. You will complete the security policy migration in Istio 1.5, remove the v1alpha1 security policy, and then continue to upgrade to later Istio versions. For a given workload, the v1beta1 version will take precedence over the v1alpha1 version.

Alternatively, if you want to do a skip-level upgrade directly from Istio 1.4 to 1.6 or later, you should use the canary upgrade method to install a new Istio version as a separate control plane, and gradually migrate your workloads to the new control plane completing the security policy migration at the same time.

In either case, it is recommended to migrate using namespace granularity: for each namespace, find all the v1alpha1 policies that have an effect on workloads in the namespace and migrate all the policies to v1beta1 at the same time. This allows a safer migration as you can make sure everything is working as expected, and then move forward to the next namespace.

Major differences

Before starting the migration, read through the v1beta1 authentication and authorization documentation to understand the v1beta1 policy.

You should examine all of your existing v1alpha1 security policies, find out what fields are used and which policies need migration, compare the findings with the major differences listed below and confirm there are no blocking issues (e.g., using an alpha feature that is no longer supported in beta):

Major Differences	`v1alpha1`	`v1beta1`
API stability	not backward compatible	backward compatible
mTLS	`MeshPolicy` and `Policy`	`PeerAuthentication`
JWT	`MeshPolicy` and `Policy`	`RequestAuthentication`
Authorization	`ClusterRbacConfig`, `ServiceRole` and `ServiceRoleBinding`	`AuthorizationPolicy`
Policy target	service name based	workload selector based
Port number	service ports	workload ports

Although RequestAuthentication in v1beta1 security policy is similar to the v1alpha1 JWT policy, there is a notable semantics change. The v1alpha1 JWT policy needs to be migrated to two v1beta1 resources: RequestAuthentication and AuthorizationPolicy. This will change the JWT deny message due to the use of AuthorizationPolicy. In the alpha version, the HTTP code 401 is returned with the body Origin authentication failed. In the beta version, the HTTP code 403 is returned with the body RBAC: access denied.

The v1alpha1 JWT policy triggerRule field is replaced by the AuthorizationPolicy with the exception that the regex field is no longer supported.

Migration flow

This section describes in detail how to migrate a v1alpha1 security policy.

For each namespace, find all v1alpha1 security policies that have an effect on workloads in the namespace. The result could include:

a single MeshPolicy that applies to all services in the mesh;
a single namespace-level Policy that applies to all workloads in the namespace;
multiple service-level Policy objects that apply to the selected services in the namespace;
a single ClusterRbacConfig that enables the RBAC on the whole namespace or some services in the namespace;
multiple namespace-level ServiceRole and ServiceRoleBinding objects that apply to all services in the namespace;
multiple service-level ServiceRole and ServiceRoleBinding objects that apply to the selected services in the namespace;

Step 2: Convert service name to workload selector

The v1alpha1 policy selects targets using their service name. You should refer to the corresponding service definition to decide the workload selector that should be used in the v1beta1 policy.

A single v1alpha1 policy may include multiple services. It will need to be migrated to multiple v1beta1 policies because the v1beta1 policy currently only supports at most one workload selector per policy.

Also note the v1alpha1 policy uses service port but the v1beta1 policy uses the workload port. This means the port number might be different in the migrated v1beta1 policy.

Step 3: Migrate authentication policy

For each v1alpha1 authentication policy, migrate with the following rules:

If the whole namespace is enabled with mTLS or JWT, create the PeerAuthentication, RequestAuthentication and AuthorizationPolicy without a workload selector for the whole namespace. Fill out the policy based on the semantics of the corresponding MeshPolicy or Policy for the namespace.
If a workload is enabled with mTLS or JWT, create the PeerAuthentication, RequestAuthentication and AuthorizationPolicy with a corresponding workload selector for the workload. Fill out the policy based on the semantics of the corresponding MeshPolicy or Policy for the workload.
For mTLS related configuration, use STRICT mode if the alpha policy is using STRICT, or use PERMISSIVE in all other cases.
For JWT related configuration, refer to the end-user authentication documentation to learn how to migrate to RequestAuthentication and AuthorizationPolicy.

A security policy migration tool is provided to automatically migrate authentication policy automatically. Please refer to the tool’s README for its usage.

Step 4: Migrate RBAC policy

For each v1alpha1 RBAC policy, migrate with the following rules:

If the whole namespace is enabled with RBAC, create an AuthorizationPolicy without a workload selector for the whole namespace. Add an empty rule so that it will deny all requests to the namespace by default.
If a workload is enabled with RBAC, create an AuthorizationPolicy with a corresponding workload selector for the workload. Add rules based on the semantics of the corresponding ServiceRole and ServiceRoleBinding for the workload.

Step 5: Verify migrated policy

Double check the migrated v1beta1 policies: make sure there are no policies with duplicate names, the namespace is specified correctly and all v1alpha1 policies for the given namespace are migrated.
Dry-run the v1beta1 policy with the command kubectl apply --dry-run=server -f beta-policy.yaml to make sure it is valid.
Apply the v1beta1 policy to the given namespace and closely monitor the effect. Make sure to test both allow and deny scenarios if JWT or authorization are used.
Migrate the next namespace. Only remove the v1alpha1 policy after completing migration for all namespaces successfully.

Example

`v1alpha1` policy

This section gives a full example showing the migration for namespace foo. Assume the namespace foo has the following v1alpha1 policies that affect the workloads in it:

# A MeshPolicy that enables mTLS globally, including the whole foo namespace
apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
metadata:
name: "default"
spec:
peers:
- mtls: {}
---
# A Policy that enables mTLS permissive mode and enables JWT for the httpbin service on port 8000
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: httpbin
namespace: foo
spec:
targets:
- name: httpbin
ports:
- number: 8000
peers:
- mtls:
mode: PERMISSIVE
origins:
- jwt:
issuer: testing@example.com
jwksUri: https://www.example.com/jwks.json
triggerRules:
- includedPaths:
- prefix: /admin/
excludedPaths:
- exact: /admin/status
principalBinding: USE_ORIGIN
---
# A ClusterRbacConfig that enables RBAC globally, including the foo namespace
apiVersion: "rbac.istio.io/v1alpha1"
kind: ClusterRbacConfig
metadata:
name: default
spec:
mode: 'ON'
---
# A ServiceRole that enables RBAC for the httpbin service
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
name: httpbin
namespace: foo
spec:
rules:
- services: ["httpbin.foo.svc.cluster.local"]
methods: ["GET"]
---
# A ServiceRoleBinding for the above ServiceRole
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: httpbin
namespace: foo
spec:
subjects:
- user: cluster.local/ns/foo/sa/sleep
roleRef:
kind: ServiceRole
name: httpbin

`httpbin` service

The httpbin service has the following definition:

apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: foo
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: httpbin

This means the service name httpbin should be replaced by the workload selector app: httpbin, and the service port 8000 should be replaced by the workload port 80.

`v1beta1` authentication policy

The migrated v1beta1 policies for the v1alpha1 authentication policies in foo namespace are listed below:

# A PeerAuthentication that enables mTLS for the foo namespace, migrated from the MeshPolicy
# Alternatively the MeshPolicy could also be migrated to a PeerAuthentication at mesh level
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: foo
spec:
mtls:
mode: STRICT
---
# A PeerAuthentication that enables mTLS for the httpbin workload, migrated from the Policy
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: httpbin
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
# port level mtls set for the workload port 80 corresponding to the service port 8000
portLevelMtls:
80:
mode: PERMISSIVE
--
# A RequestAuthentication that enables JWT for the httpbin workload, migrated from the Policy
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: httpbin
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
jwtRules:
- issuer: testing@example.com
jwksUri: https://www.example.com/jwks.json
---
# An AuthorizationPolicy that enforces to require JWT validation for the httpbin workload, migrated from the Policy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin-jwt
namespace: foo
spec:
# Use DENY action to explicitly deny requests without JWT token
action: DENY
selector:
matchLabels:
app: httpbin
rules:
- from:
- source:
# This makes sure requests without JWT token will be denied
notRequestPrincipals: ["*"]
to:
- operation:
# This should be the workload port 80, not the service port 8000
ports: ["80"]
# The path and notPath is converted from the trigger rule in the Policy
paths: ["/admin/*"]
notPaths: ["/admin/status"]

`v1beta1` authorization policy

The migrated v1beta1 policies for the v1alpha1 RBAC policies in foo namespace are listed below:

# An AuthorizationPolicy that denies by default, migrated from the ClusterRbacConfig
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: default
namespace: foo
spec:
# An empty rule that allows nothing
{}
---
# An AuthorizationPolicy that enforces to authorization for the httpbin workload, migrated from the ServiceRole and ServiceRoleBinding
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
version: v1
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/foo/sa/sleep"]
to:
- operation:
methods: ["GET"]

Finish the upgrade

Congratulations; having reached this point, you should only have v1beta1 policy objects, and you will be able to continue upgrading Istio to 1.6 and beyond.

Zero Configuration Istio

John Howard (Google) — Thu, 25 Feb 2021 00:00:00 +0000

When a new user encounters Istio for the first time, they are sometimes overwhelmed by the vast feature set it exposes. Unfortunately, this can give the impression that Istio is needlessly complex and not fit for small teams or clusters.

One great part about Istio, however, is that it aims to bring as much value to users out of the box without any configuration at all. This enables users to get most of the benefits of Istio with minimal efforts. For some users with simple requirements, custom configurations may never be required at all. Others will be able to incrementally add Istio configurations once they are more comfortable and as they need them, such as to add ingress routing, fine-tune networking settings, or lock down security policies.

Getting started

To get started, check out our getting started documentation, where you will learn how to install Istio. If you are already familiar, you can simply run istioctl install.

Next, we will explore all the benefits Istio provides us, without any configuration or changes to application code.

Security

Istio automatically enables mutual TLS for traffic between pods in the mesh. This enables applications to forgo complex TLS configuration and certificate management, and offload all transport layer security to the sidecar.

Once comfortable with automatic TLS, you may choose to allow only mTLS traffic, or configure custom authorization policies for your needs.

Observability

Istio automatically generates detailed telemetry for all service communications within a mesh. This telemetry provides observability of service behavior, empowering operators to troubleshoot, maintain, and optimize their applications – without imposing any additional burdens on service developers. Through Istio, operators gain a thorough understanding of how monitored services are interacting, both with other services and with the Istio components themselves.

All of this functionality is added by Istio without any configuration. Integrations with tools such as Prometheus, Grafana, Jaeger, Zipkin, and Kiali are also available.

For more information about the observability Istio provides, check out the observability overview.

Traffic Management

While Kubernetes provides a lot of networking functionality, such as service discovery and DNS, this is done at Layer 4, which can have unintended inefficiencies. For example, in a simple HTTP application sending traffic to a service with 3 replicas, we can see unbalanced load:

$ curl http://echo/{0..5} -s | grep Hostname
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
$ curl http://echo/{0..5} -s | grep Hostname
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn

The problem here is Kubernetes will determine the backend to send to when the connection is established, and all future requests on the same connection will be sent to the same backend. In our example here, our first 5 requests are all sent to echo-cb96f8d94-2ssll, while our next set (using a new connection) are all sent to echo-cb96f8d94-879sn. Our third instance never receives any requests.

With Istio, HTTP traffic (including HTTP/2 and gRPC) is automatically detected, and our services will automatically be load balanced per request, rather than per connection:

$ curl http://echo/{0..5} -s | grep Hostname
Hostname=echo-cb96f8d94-wf4xk
Hostname=echo-cb96f8d94-rpfqz
Hostname=echo-cb96f8d94-cgmxr
Hostname=echo-cb96f8d94-wf4xk
Hostname=echo-cb96f8d94-rpfqz
Hostname=echo-cb96f8d94-cgmxr

Here we can see our requests are round-robin load balanced between all backends.

In addition to these better defaults, Istio offers customization of a variety of traffic management settings, including timeouts, retries, and much more.

IstioCon 2021: Schedule Is Live!

Istio Steering Committee — Tue, 16 Feb 2021 00:00:00 +0000

IstioCon 2021 is a week-long, community-led, virtual conference starting on February 22. This event provides an opportunity to hear the lessons learned from companies like Atlassian, Airbnb, FICO, eBay, T-Mobile and Salesforce running Istio in production, hands-on experiences from the Istio community, and will feature maintainers from across the Istio ecosystem.

You can now find the full schedule of events which includes a series of English sessions and Chinese sessions.

By attending the conference, you’ll connect with community members from across the globe. Each day you will find keynotes, technical talks, lightning talks, panel discussions, workshops and roadmap sessions led by diverse speakers representing the Istio community. You can also connect with other Istio and Open Source ecosystem community members through social hour events that include activities on the social platform Gather.town, a live cartoonist, virtual swag bags, raffles, live music and games.

Don’t miss it! Registration is free. We look forward to seeing you at the first IstioCon!

Better External Authorization

Yangmin Zhu (Google) — Tue, 09 Feb 2021 00:00:00 +0000

Background

Istio’s authorization policy provides access control for services in the mesh. It is fast, powerful and a widely used feature. We have made continuous improvements to make policy more flexible since its first release in Istio 1.4, including the DENY action, exclusion semantics, X-Forwarded-For header support, nested JWT claim support and more. These features improve the flexibility of the authorization policy, but there are still many use cases that cannot be supported with this model, for example:

You have your own in-house authorization system that cannot be easily migrated to, or cannot be easily replaced by, the authorization policy.
You want to integrate with a 3rd-party solution (e.g. Open Policy Agent or oauth2 proxy) which may require use of the low-level Envoy configuration APIs in Istio, or may not be possible at all.
Authorization policy lacks necessary semantics for your use case.

Solution

In Istio 1.9, we have implemented extensibility into authorization policy by introducing a CUSTOM action, which allows you to delegate the access control decision to an external authorization service.

The CUSTOM action allows you to integrate Istio with an external authorization system that implements its own custom authorization logic. The following diagram shows the high level architecture of this integration:

External Authorization Architecture

At configuration time, the mesh admin configures an authorization policy with a CUSTOM action to enable the external authorization on a proxy (either gateway or sidecar). The admin should verify the external auth service is up and running.

At runtime,

A request is intercepted by the proxy, and the proxy will send check requests to the external auth service, as configured by the user in the authorization policy.
The external auth service will make the decision whether to allow it or not.
If allowed, the request will continue and will be enforced by any local authorization defined by ALLOW/DENY action.
If denied, the request will be rejected immediately.

Let’s look at an example authorization policy with the CUSTOM action:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ext-authz
namespace: istio-system
spec:
# The selector applies to the ingress gateway in the istio-system namespace.
selector:
matchLabels:
app: istio-ingressgateway
# The action "CUSTOM" delegates the access control to an external authorizer, this is different from
# the ALLOW/DENY action that enforces the access control right inside the proxy.
action: CUSTOM
# The provider specifies the name of the external authorizer defined in the meshconfig, which tells where and how to
# talk to the external auth service. We will cover this more later.
provider:
name: "my-ext-authz-service"
# The rule specifies that the access control is triggered only if the request path has the prefix "/admin/".
# This allows you to easily enable or disable the external authorization based on the requests, avoiding the external
# check request if it is not needed.
rules:
- to:
- operation:
paths: ["/admin/*"]

It refers to a provider called my-ext-authz-service which is defined in the mesh config:

extensionProviders:
# The name "my-ext-authz-service" is referred to by the authorization policy in its provider field.
- name: "my-ext-authz-service"
# The "envoyExtAuthzGrpc" field specifies the type of the external authorization service is implemented by the Envoy
# ext-authz filter gRPC API. The other supported type is the Envoy ext-authz filter HTTP API.
# See more in https://www.envoyproxy.io/docs/envoy/v1.16.2/intro/arch_overview/security/ext_authz_filter.
envoyExtAuthzGrpc:
# The service and port specifies the address of the external auth service, "ext-authz.istio-system.svc.cluster.local"
# means the service is deployed in the mesh. It can also be defined out of the mesh or even inside the pod as a separate
# container.
service: "ext-authz.istio-system.svc.cluster.local"
port: 9000

The authorization policy of CUSTOM action enables the external authorization in runtime, it could be configured to trigger the external authorization conditionally based on the request using the same rule that you have already been using with other actions.

The external authorization service is currently defined in the meshconfig API and referred to by its name. It could be deployed in the mesh with or without proxy. If with the proxy, you could further use PeerAuthentication to enable mTLS between the proxy and your external authorization service.

The CUSTOM action is currently in the experimental stage; the API might change in a non-backward compatible way based on user feedback. The rule currently does not support authentication related fields (e.g. source principal or JWT claim) and only one provider is allowed for a given workload, but you can still use different providers on different workloads.

For more information, please see the Better External Authorization design doc.

Example with OPA

In this section, we will demonstrate using the CUSTOM action with the Open Policy Agent as the external authorizer on the ingress gateway. We will conditionally enable the external authorization on all paths except /ip.

You can also refer to the external authorization task for a more basic introduction that uses a sample ext-authz server.

Create the example OPA policy

Run the following command create an OPA policy that allows the request if the prefix of the path is matched with the claim “path” (base64 encoded) in the JWT token:

$ cat > policy.rego <<EOF
package envoy.authz
import input.attributes.request.http as http_request
default allow = false
token = {"valid": valid, "payload": payload} {
[_, encoded] := split(http_request.headers.authorization, " ")
[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}
allow {
is_token_valid
action_allowed
}
is_token_valid {
token.valid
now := time.now_ns() / 1000000000
token.payload.nbf <= now
now < token.payload.exp
}
action_allowed {
startswith(http_request.path, base64url.decode(token.payload.path))
}
EOF
$ kubectl create secret generic opa-policy --from-file policy.rego

Deploy httpbin and OPA

Enable the sidecar injection:

$ kubectl label ns default istio-injection=enabled

Run the following command to deploy the example application httpbin and OPA. The OPA could be deployed either as a separate container in the httpbin pod or completely in a separate pod:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: httpbin-with-opa
labels:
app: httpbin-with-opa
service: httpbin-with-opa
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: httpbin-with-opa
---
# Define the service entry for the local OPA service on port 9191.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: local-opa-grpc
spec:
hosts:
- "local-opa-grpc.local"
endpoints:
- address: "127.0.0.1"
ports:
- name: grpc
number: 9191
protocol: GRPC
resolution: STATIC
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: httpbin-with-opa
labels:
app: httpbin-with-opa
spec:
replicas: 1
selector:
matchLabels:
app: httpbin-with-opa
template:
metadata:
labels:
app: httpbin-with-opa
spec:
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: httpbin
ports:
- containerPort: 80
- name: opa
image: openpolicyagent/opa:latest-envoy
securityContext:
runAsUser: 1111
volumeMounts:
- readOnly: true
mountPath: /policy
name: opa-policy
args:
- "run"
- "--server"
- "--addr=localhost:8181"
- "--diagnostic-addr=0.0.0.0:8282"
- "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
- "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
- "--set=decision_logs.console=true"
- "--ignore=.*"
- "/policy/policy.rego"
livenessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: proxy-config
configMap:
name: proxy-config
- name: opa-policy
secret:
secretName: opa-policy
EOF

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: opa
labels:
app: opa
spec:
ports:
- name: grpc
port: 9191
targetPort: 9191
selector:
app: opa
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: opa
labels:
app: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
spec:
containers:
- name: opa
image: openpolicyagent/opa:latest-envoy
securityContext:
runAsUser: 1111
volumeMounts:
- readOnly: true
mountPath: /policy
name: opa-policy
args:
- "run"
- "--server"
- "--addr=localhost:8181"
- "--diagnostic-addr=0.0.0.0:8282"
- "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
- "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
- "--set=decision_logs.console=true"
- "--ignore=.*"
- "/policy/policy.rego"
ports:
- containerPort: 9191
livenessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
volumes:
- name: proxy-config
configMap:
name: proxy-config
- name: opa-policy
secret:
secretName: opa-policy
EOF

Deploy the httpbin as well:

$ kubectl apply -f @samples/httpbin/httpbin.yaml@

Define external authorizer

Run the following command to edit the meshconfig:

$ kubectl edit configmap istio -n istio-system

Add the following extensionProviders to the meshconfig:

apiVersion: v1
data:
mesh: |-
# Add the following contents:
extensionProviders:
- name: "opa.local"
envoyExtAuthzGrpc:
service: "local-opa-grpc.local"
port: "9191"

apiVersion: v1
data:
mesh: |-
# Add the following contents:
extensionProviders:
- name: "opa.default"
envoyExtAuthzGrpc:
service: "opa.default.svc.cluster.local"
port: "9191"

Create an AuthorizationPolicy with a CUSTOM action

Run the following command to create the authorization policy that enables the external authorization on all paths except /ip:

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin-opa
spec:
selector:
matchLabels:
app: httpbin-with-opa
action: CUSTOM
provider:
name: "opa.local"
rules:
- to:
- operation:
notPaths: ["/ip"]
EOF

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin-opa
spec:
selector:
matchLabels:
app: httpbin
action: CUSTOM
provider:
name: "opa.default"
rules:
- to:
- operation:
notPaths: ["/ip"]
EOF

Test the OPA policy

Create a client pod to send the request:

$ kubectl apply -f @samples/sleep/sleep.yaml@
$ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

Use a test JWT token signed by the OPA:

$ export TOKEN_PATH_HEADERS="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXRoIjoiTDJobFlXUmxjbk09IiwibmJmIjoxNTAwMDAwMDAwLCJleHAiOjE5MDAwMDAwMDB9.9yl8LcZdq-5UpNLm0Hn0nnoBHXXAnK4e8RSl9vn6l98"

The test JWT token has the following claims:

{
"path": "L2hlYWRlcnM=",
"nbf": 1500000000,
"exp": 1900000000
}

The path claim has value L2hlYWRlcnM= which is the base64 encode of /headers.

Send a request to path /headers without a token. This should be rejected with 403 because there is no JWT token:

$ kubectl exec ${SLEEP_POD} -c sleep -- curl http://httpbin-with-opa:8000/headers -s -o /dev/null -w "%{http_code}\n"
403

Send a request to path /get with a valid token. This should be rejected with 403 because the path /get is not matched with the token /headers:

$ kubectl exec ${SLEEP_POD} -c sleep -- curl http://httpbin-with-opa:8000/get -H "Authorization: Bearer $TOKEN_PATH_HEADERS" -s -o /dev/null -w "%{http_code}\n"
403

Send a request to path /headers with valid token. This should be allowed with 200 because the path is matched with the token:

$ kubectl exec ${SLEEP_POD} -c sleep -- curl http://httpbin-with-opa:8000/headers -H "Authorization: Bearer $TOKEN_PATH_HEADERS" -s -o /dev/null -w "%{http_code}\n"
200

Send request to path /ip without token. This should be allowed with 200 because the path /ip is excluded from authorization:
```
$ kubectl exec ${SLEEP_POD} -c sleep -- curl http://httpbin-with-opa:8000/ip -s -o /dev/null -w "%{http_code}\n"
200
```
Check the proxy and OPA logs to confirm the result.

Summary

In Istio 1.9, the CUSTOM action in the authorization policy allows you to easily integrate Istio with any external authorization system with the following benefits:

First-class support in the authorization policy API
Ease of usage: define the external authorizer simply with a URL and enable with the authorization policy, no more hassle with the EnvoyFilter API
Conditional triggering, allowing improved performance
Support for various deployment type of the external authorizer:
- A normal service and pod with or without proxy
- Inside the workload pod as a separate container
- Outside the mesh

We’re working to promote this feature to a more stable stage in following versions and welcome your feedback at discuss.istio.io.

Acknowledgements

Thanks to Craig Box, Christian Posta and Limin Wang for reviewing drafts of this blog.

Proxying legacy services using Istio egress gateways

Antonio Berben (Deutsche Telekom - PAN-NET) — Wed, 16 Dec 2020 00:00:00 +0000

At Deutsche Telekom Pan-Net, we have embraced Istio as the umbrella to cover our services. Unfortunately, there are services which have not yet been migrated to Kubernetes, or cannot be.

We can set Istio up as a proxy service for these upstream services. This allows us to benefit from capabilities like authorization/authentication, traceability and observability, even while legacy services stand as they are.

At the end of this article there is a hands-on exercise where you can simulate the scenario. In the exercise, an upstream service hosted at https://httpbin.org will be proxied by an Istio egress gateway.

If you are familiar with Istio, one of the methods offered to connect to upstream services is through an egress gateway.

You can deploy one to control all the upstream traffic or you can deploy multiple in order to have fine-grained control and satisfy the single-responsibility principle as this picture shows:

Overview multiple Egress Gateways

With this model, one egress gateway is in charge of exactly one upstream service.

Although the Operator spec allows you to deploy multiple egress gateways, the manifest can become unmanageable:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
[...]
spec:
egressGateways:
- name: egressgateway-1
enabled: true
- name: egressgateway-2
enabled: true
[egressgateway-3, egressgateway-4, ...]
- name: egressgateway-N
enabled: true
[...]

As a benefit of decoupling egress getaways from the Operator manifest, you have enabled the possibility of setting up custom readiness probes to have both services (Gateway and upstream Service) aligned.

You can also inject OPA as a sidecar into the pod to perform authorization with complex rules (OPA envoy plugin).

Authorization with OPA and `healthcheck` to external

As you can see, your possibilities increase and Istio becomes very extensible.

Let’s look at how you can implement this pattern.

Solution

There are several ways to perform this task, but here you will find how to define multiple Operators and deploy the generated resources.

In the following section you will deploy an egress gateway to connect to an upstream service: httpbin (https://httpbin.org/)

At the end, you will have:

Communication

Hands on

Prerequisites

kind (Kubernetes-in-Docker - perfect for local development)
istioctl

Kind

Save this as config.yaml.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
- |
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
metadata:
name: config
apiServer:
extraArgs:
"service-account-issuer": "kubernetes.default.svc"
"service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"

$ kind create cluster --name <my-cluster-name> --config config.yaml

Where <my-cluster-name> is the name for the cluster.

Istio Operator with Istioctl

Install the Operator

$ istioctl operator init --watchedNamespaces=istio-operator

$ kubectl create ns istio-system

Save this as operator.yaml:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-operator
namespace: istio-operator
spec:
profile: default
tag: 1.8.0
meshConfig:
accessLogFile: /dev/stdout
outboundTrafficPolicy:
mode: REGISTRY_ONLY

$ kubectl apply -f operator.yaml

Deploy Egress Gateway

The steps for this task assume:

The service is installed under the namespace: httpbin.
The service name is: http-egress.

Istio 1.8 introduced the possibility to apply overlay configuration, to give fine-grain control over the created resources.

Save this as egress.yaml:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: empty
tag: 1.8.0
namespace: httpbin
components:
egressGateways:
- name: httpbin-egress
enabled: true
label:
app: istio-egressgateway
istio: egressgateway
custom-egress: httpbin-egress
k8s:
overlays:
- kind: Deployment
name: httpbin-egress
patches:
- path: spec.template.spec.containers[0].readinessProbe
value:
failureThreshold: 30
exec:
command:
- /bin/sh
- -c
- curl http://localhost:15021/healthz/ready && curl https://httpbin.org/status/200
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 1
values:
gateways:
istio-egressgateway:
runAsRoot: true

Create the namespace where you will install the egress gateway:

$ kubectl create ns httpbin

As it is described in the documentation, you can deploy several Operator resources. However, they have to be pre-parsed and then applied to the cluster.

$ istioctl manifest generate -f egress.yaml | kubectl apply -f -

Istio configuration

Now you will configure Istio to allow connections to the upstream service at https://httpbin.org.

Certificate for TLS

You need a certificate to make a secure connection from outside the cluster to your egress service.

How to generate a certificate is explained in the Istio ingress documentation.

Create and apply one to be used at the end of this article to access the service from outside the cluster (<my-proxied-service-hostname>):

$ kubectl create -n istio-system secret tls <my-secret-name> --key=<key> --cert=<cert>

Where <my-secret-name> is the name used later for the Gateway resource. <key> and <cert> are the files for the certificate. <cert>.

Ingress Gateway

Create a Gateway resource to operate ingress gateway to accept requests.

An example:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: my-ingressgateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- "<my-proxied-service-hostname>"
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: true
- port:
number: 443
name: https
protocol: https
hosts:
- "<my-proxied-service-hostname>"
tls:
mode: SIMPLE
credentialName: <my-secret-name>

Where <my-proxied-service-hostname> is the hostname to access the service through the my-ingressgateway and <my-secret-name> is the secret which contains the certificate.

Egress Gateway

Create another Gateway object, but this time to operate the egress gateway you have already installed:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
selector:
istio: egressgateway
service.istio.io/canonical-name: "httpbin-egress"
servers:
- hosts:
- "<my-proxied-service-hostname>"
port:
number: 80
name: http
protocol: HTTP

Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.

Virtual Service

Create a VirtualService for three use cases:

Mesh gateway for service-to-service communications within the mesh
Ingress Gateway for the communication from outside the mesh
Egress Gateway for the communication to the upstream service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
hosts:
- "<my-proxied-service-hostname>"
gateways:
- mesh
- "istio-system/my-ingressgateway"
- "httpbin/httpbin-egress"
http:
- match:
- gateways:
- "istio-system/my-ingressgateway"
- mesh
uri:
prefix: "/"
route:
- destination:
host: "httpbin-egress.httpbin.svc.cluster.local"
port:
number: 80
- match:
- gateways:
- "httpbin/httpbin-egress"
uri:
prefix: "/"
route:
- destination:
host: "httpbin.org"
subset: "http-egress-subset"
port:
number: 443

Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.

Service Entry

Create a ServiceEntry to allow the communication to the upstream service:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
hosts:
- "httpbin.org"
location: MESH_EXTERNAL
ports:
- number: 443
name: https
protocol: TLS
resolution: DNS

Destination Rule

Create a DestinationRule to allow TLS origination for egress traffic as explained in the documentation

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
host: "httpbin.org"
subsets:
- name: "http-egress-subset"
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: SIMPLE

Peer Authentication

To secure the service-to-service, you need to enforce mTLS:

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
name: "httpbin-egress"
namespace: "httpbin"
spec:
mtls:
mode: STRICT

Test

Verify that your objects were all specified correctly:

$ istioctl analyze --all-namespaces

External access

Test the egress gateway from outside the cluster forwarding the ingressgateway service’s port and calling the service

$ kubectl -n istio-system port-forward svc/istio-ingressgateway 15443:443

$ curl -vvv -k -HHost:<my-proxied-service-hostname> --resolve "<my-proxied-service-hostname>:15443:127.0.0.1" --cacert <cert> "https://<my-proxied-service-hostname>:15443/status/200"

Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway and <cert> is the certificate defined for the ingressgateway object. This is due to tls.mode: SIMPLE which does not terminate TLS

Service-to-service access

Test the egress gateway from inside the cluster deploying the sleep service. This is useful when you design failover.

$ kubectl label namespace httpbin istio-injection=enabled --overwrite

$ kubectl apply -n httpbin -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/sleep/sleep.yaml

$ kubectl -n httpbin "$(kubectl get pod -n httpbin -l app=sleep -o jsonpath={.items..metadata.name})" -- curl -vvv http://<my-proxied-service-hostname>/status/200

Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.

Now it is time to create a second, third and fourth egress gateway pointing to other upstream services.

Final thoughts

Istio might seem complex to configure. But it is definitely worthwhile, due to the huge set of benefits it brings to your services (with an extra Olé! for Kiali).

The way Istio is developed allows us, with minimal effort, to satisfy uncommon requirements like the one presented in this article.

To finish, I just wanted to point out that Istio, as a good cloud native technology, does not require a large team to maintain. For example, our current team is composed of 3 engineers.

To discuss more about Istio and its possibilities, please contact one of us:

Proxy protocol on AWS NLB and Istio ingress gateway

Xinhui Li (Salesforce) — Fri, 11 Dec 2020 00:00:00 +0000

This blog presents my latest experience about how to configure and enable proxy protocol with stack of AWS NLB and Istio Ingress gateway. The Proxy Protocol was designed to chain proxies and reverse-proxies without losing the client information. The proxy protocol prevents the need for infrastructure changes or NATing firewalls, and offers the benefits of being protocol agnostic and providing good scalability. Additionally, we also enable the X-Forwarded-For HTTP header in the deployment to make the client IP address easy to read. In this blog, traffic management of Istio ingress is shown with an httpbin service on ports 80 and 443 to demonstrate the use of proxy protocol. Note that both v1 and v2 of the proxy protocol work for the purpose of this example, but because the AWS NLB currently only supports v2, proxy protocol v2 is used in the rest of this blog by default. The following image shows the use of proxy protocol v2 with an AWS NLB.

AWS NLB portal to enable proxy protocol

Separate setups for 80 and 443

Before going through the following steps, an AWS environment that is configured with the proper VPC, IAM, and Kubernetes setup is assumed.

Step 1: Install Istio with AWS NLB

The blog Configuring Istio Ingress with AWS NLB provides detailed steps to set up AWS IAM roles and enable the usage of AWS NLB by Helm. You can also use other automation tools, such as Terraform, to achieve the same goal. In the following example, more complete configurations are shown in order to enable proxy protocol and X-Forwarded-For at the same time.

apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
proxy.istio.io/config: '{"gatewayTopology" : { "numTrustedProxies": 2 } }'
labels:
app: istio-ingressgateway
istio: ingressgateway
release: istio
name: istio-ingressgateway

Step 2: Create proxy-protocol Envoy Filter

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: proxy-protocol
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: LISTENER
patch:
operation: MERGE
value:
listener_filters:
- name: envoy.filters.listener.proxy_protocol
- name: envoy.filters.listener.tls_inspector

Step 3: Enable `X-Forwarded-For` header

This blog includes several samples of configuring Gateway Network Topology. In the following example, the configurations are tuned to enable X-Forwarded-For without any middle proxy.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: ingressgateway-settings
namespace: istio-system
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
listener:
filterChain:
filter:
name: envoy.http_connection_manager
patch:
operation: MERGE
value:
name: envoy.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager
skip_xff_append: false
use_remote_address: true
xff_num_trusted_hops: 1

Step 4: Deploy ingress gateway for httpbin on port 80 and 443

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: httpbin-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "a25fa0b4835b.elb.us-west-2.amazonaws.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "a25fa0b4835b.elb.us-west-2.amazonaws.com"
gateways:
- httpbin-gateway
http:
- match:
- uri:
prefix: /headers
route:
- destination:
port:
number: 8000
host: httpbin

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: mygateway2
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: SIMPLE
credentialName: httpbin-credential # must be the same as secret
hosts:
- "a25fa0b4835b.elb.us-west-2.amazonaws.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- "a25fa0b4835b.elb.us-west-2.amazonaws.com"
gateways:
- mygateway2
http:
- match:
- uri:
prefix: /headers
route:
- destination:
port:
number: 8000
host: httpbin

Step 5: Check header output of httpbin

Check port 443 (80 will be similar) and compare the cases with and without proxy protocol.

//////with proxy_protocal enabled in the stack
* Trying YY.XXX.141.26...
* TCP_NODELAY set
* Connection failed
* connect to YY.XXX.141.26 port 443 failed: Operation timed out
* Trying YY.XXX.205.117...
* TCP_NODELAY set
* Connected to a25fa0b4835b.elb.us-west-2.amazonaws.com (XX.YYY.205.117) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: new_certificates/example.com.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=a25fa0b4835b.elb.us-west-2.amazonaws.com; O=httpbin organization
* start date: Oct 29 20:39:12 2020 GMT
* expire date: Oct 29 20:39:12 2021 GMT
* common name: a25fa0b4835b.elb.us-west-2.amazonaws.com (matched)
* issuer: O=example Inc.; CN=example.com
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc6c8810800)
> GET /headers?show_env=1 HTTP/2
> Host: a25fa0b4835b.elb.us-west-2.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
< server: istio-envoy
< date: Thu, 29 Oct 2020 21:39:46 GMT
< content-type: application/json
< content-length: 629
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 2
<
{
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "a25fa0b4835b.elb.us-west-2.amazonaws.com",
"User-Agent": "curl/7.64.1",
"X-B3-Sampled": "0",
"X-B3-Spanid": "74f99a1c6fc29975",
"X-B3-Traceid": "85db86fe6aa322a074f99a1c6fc29975",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Decorator-Operation": "httpbin.default.svc.cluster.local:8000/headers*",
"X-Envoy-External-Address": "XX.110.54.41",
"X-Forwarded-For": "XX.110.54.41",
"X-Forwarded-Proto": "https",
"X-Request-Id": "5c3bc236-0c49-4401-b2fd-2dbfbce506fc"
}
}
* Connection #0 to host a25fa0b4835b.elb.us-west-2.amazonaws.com left intact
* Closing connection 0

//////////without proxy_protocal
* Trying YY.XXX.141.26...
* TCP_NODELAY set
* Connection failed
* connect to YY.XXX.141.26 port 443 failed: Operation timed out
* Trying YY.XXX.205.117...
* TCP_NODELAY set
* Connected to a25fa0b4835b.elb.us-west-2.amazonaws.com (YY.XXX.205.117) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: new_certificates/example.com.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=a25fa0b4835b.elb.us-west-2.amazonaws.com; O=httpbin organization
* start date: Oct 29 20:39:12 2020 GMT
* expire date: Oct 29 20:39:12 2021 GMT
* common name: a25fa0b4835b.elb.us-west-2.amazonaws.com (matched)
* issuer: O=example Inc.; CN=example.com
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbf8c808200)
> GET /headers?show_env=1 HTTP/2
> Host: a25fa0b4835b.elb.us-west-2.amazonaws.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
< server: istio-envoy
< date: Thu, 29 Oct 2020 20:44:01 GMT
< content-type: application/json
< content-length: 612
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 1
<
{
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "a25fa0b4835b.elb.us-west-2.amazonaws.com",
"User-Agent": "curl/7.64.1",
"X-B3-Sampled": "0",
"X-B3-Spanid": "69913a6e6e949334",
"X-B3-Traceid": "729d5da3618545da69913a6e6e949334",
"X-Envoy-Attempt-Count": "1",
"X-Envoy-Decorator-Operation": "httpbin.default.svc.cluster.local:8000/headers*",
"X-Envoy-Internal": "true",
"X-Forwarded-For": "172.16.5.30",
"X-Forwarded-Proto": "https",
"X-Request-Id": "299c7f8a-5f89-480a-82c9-028c76d45d84"
}
}
* Connection #0 to host a25fa0b4835b.elb.us-west-2.amazonaws.com left intact
* Closing connection 0

Conclusion

This blog presents the deployment of a stack that consists of an AWS NLB and Istio ingress gateway that are enabled with proxy-protocol. We hope it is useful to you if you are interested in protocol enabling in an anecdotal, experiential, and more informal way. However, note that the X-Forwarded-For header should be used only for the convenience of reading in test, as dealing with fake X-Forwarded-For attacks is not within the scope of this blog.

References

Join us for the first IstioCon in 2021!

Istio Steering Committee — Tue, 08 Dec 2020 00:00:00 +0000

IstioCon 2021 will be the inaugural conference for Istio, the industry’s most popular service mesh. In its inaugural year, IstioCon will be 100% virtual, connecting community members across the globe with Istio’s ecosystem. This conference will take place at the end of February.

All the information related to IstioCon will be published on the conference website. IstioCon provides an opportunity to showcase the lessons learned from running Istio in production, hands-on experiences from the Istio community, and will feature maintainers from across the Istio ecosystem. At this time, we encourage Istio users, developers, partners, and advocates to submit a session proposal through the conference’s CFP portal. The conference offers a mix of keynotes, technical talks, lightning talks, workshops, and roadmap sessions. Choose from the following formats to submit a session proposal for IstioCon:

Presentation: 40 minute presentation, maximum of 2 speakers
Panel: 40 minutes of discussion among 3 to 5 speakers
Workshop: 160 minute (2h 40m), in-depth, hands-on presentation with 1–4 speakers
Lighting Talk: 10 minute presentation, limited to 1 speaker

This community-led event also has in store two social hours to take the load off and mesh with the Istio community, vendors, and maintainers. Participation in the event is free of charge, and will only require participants to register in order to join.

Stay tuned to hear more about this conference, and we hope you can join us at the first IstioCon in 2021!

Handling Docker Hub rate limiting

John Howard (Google) — Mon, 07 Dec 2020 00:00:00 +0000

Since November 20th, 2020, Docker Hub has introduced rate limits on image pulls.

Because Istio uses Docker Hub as the default registry, usage on a large cluster may lead to pods failing to startup due to exceeding rate limits. This can be especially problematic for Istio, as there is typically the Istio sidecar image alongside most pods in the cluster.

Mitigations

Istio allows you to specify a custom docker registry which you can use to make container images be fetched from your private registry. This can be configured by passing --set hub=<some-custom-registry> at installation time.

Istio provides official mirrors to Google Container Registry. This can be configured with --set hub=gcr.io/istio-release. This is available for Istio 1.5+.

Alternatively, you can copy the official Istio images to your own registry. This is especially useful if your cluster runs in an environment with a registry tailored for your use case (for example, on AWS you may want to mirror images to Amazon ECR) or you have air gapped security requirements where access to public registries is restricted. This can be done with the following script:

$ SOURCE_HUB=istio
$ DEST_HUB=my-registry # Replace this with the destination hub
$ IMAGES=( install-cni operator pilot proxyv2 ) # Images to mirror.
$ VERSIONS=( 1.7.5 1.8.0 ) # Versions to copy
$ VARIANTS=( "" "-distroless" ) # Variants to copy
$ for image in $IMAGES; do
$ for version in $VERSIONS; do
$ for variant in $VARIANTS; do
$ name=$image:$version$variant
$ docker pull $SOURCE_HUB/$name
$ docker tag $SOURCE_HUB/$name $DEST_HUB/$name
$ docker push $DEST_HUB/$name
$ docker rmi $SOURCE_HUB/$name
$ docker rmi $DEST_HUB/$name
$ done
$ done
$ done

Expanding into New Frontiers - Smart DNS Proxying in Istio

Shriram Rajagopalan (Tetrate.io) on behalf of Istio Networking WG — Thu, 12 Nov 2020 00:00:00 +0000

DNS resolution is a vital component of any application infrastructure on Kubernetes. When your application code attempts to access another service in the Kubernetes cluster or even a service on the internet, it has to first lookup the IP address corresponding to the hostname of the service, before initiating a connection to the service. This name lookup process is often referred to as service discovery. In Kubernetes, the cluster DNS server, be it kube-dns or CoreDNS, resolves the service’s hostname to a unique non-routable virtual IP (VIP), if it is a service of type clusterIP. The kube-proxy on each node maps this VIP to a set of pods of the service, and forwards the traffic to one of them selected at random. When using a service mesh, the sidecar works similarly to the kube-proxy as far as traffic forwarding is concerned.

The following diagram depicts the role of DNS today:

Role of DNS in Istio, today

Problems posed by DNS

While the role of DNS within the service mesh may seem insignificant, it has consistently stood in the way of expanding the mesh to VMs and enabling seamless multicluster access.

VM access to Kubernetes services

Consider the case of a VM with a sidecar. As shown in the illustration below, applications on the VM look up the IP addresses of services inside the Kubernetes cluster as they typically have no access to the cluster’s DNS server.

DNS resolution issues on VMs accessing Kubernetes services

It is technically possible to use kube-dns as a name server on the VM if one is willing to engage in some convoluted workarounds involving dnsmasq and external exposure of kube-dns using NodePort services: assuming you manage to convince your cluster administrator to do so. Even so, you are opening the door to a host of security issues. At the end of the day, these are point solutions that are typically out of scope for those with limited organizational capability and domain expertise.

External TCP services without VIPs

It is not just the VMs in the mesh that suffer from the DNS issue. For the sidecar to accurately distinguish traffic between two different TCP services that are outside the mesh, the services must be on different ports or they need to have a globally unique VIP, much like the clusterIP assigned to Kubernetes services. But what if there is no VIP? Cloud hosted services like hosted databases, typically do not have a VIP. Instead, the provider’s DNS server returns one of the instance IPs that can then be directly accessed by the application. For example, consider the two service entries below, pointing to two different AWS RDS services:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: db1
namespace: ns1
spec:
hosts:
- mysql-instance1.us-east-1.rds.amazonaws.com
ports:
- name: mysql
number: 3306
protocol: TCP
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: db2
namespace: ns1
spec:
hosts:
- mysql-instance2.us-east-1.rds.amazonaws.com
ports:
- name: mysql
number: 3306
protocol: TCP
resolution: DNS

The sidecar has a single listener on 0.0.0.0:3306 that looks up the IP address of mysql-instance1.us-east1.rds.amazonaws.com from public DNS servers and forwards traffic to it. It cannot route traffic to db2 as it has no way of distinguishing whether traffic arriving at 0.0.0.0:3306 is bound for db1 or db2. The only way to accomplish this is to set the resolution to NONE causing the sidecar to blindly forward any traffic on port 3306 to the original IP requested by the application. This is akin to punching a hole in the firewall allowing all traffic to port 3306 irrespective of the destination IP. To get traffic flowing, you are now forced to compromise on the security posture of your system.

Resolving DNS for services in remote clusters

The DNS limitations of a multicluster mesh are well known. Services in one cluster cannot lookup the IP addresses of services in other clusters, without clunky workarounds such as creating stub services in the caller namespace.

Taking control of DNS

All in all, DNS has been a thorny issue in Istio for a while. It was time to slay the beast. We (the Istio networking team) decided to tackle the problem once and for all in a way that is completely transparent to you, the end user. Our first attempt involved utilizing Envoy’s DNS proxy. It turned out to be very unreliable, and disappointing overall due to the general lack of sophistication in the c-ares DNS library used by Envoy. Determined to solve the problem, we decided to implement the DNS proxy in the Istio sidecar agent, written in Go. We were able to optimize the implementation to handle all the scenarios that we wanted to tackle without compromising on scale and stability. The Go DNS library we use is the same one used by scalable DNS implementations such as CoreDNS, Consul, Mesos, etc. It has been battle tested in production for scale and stability.

Starting with Istio 1.8, the Istio agent on the sidecar will ship with a caching DNS proxy, programmed dynamically by Istiod. Istiod pushes the hostname-to-IP-address mappings for all the services that the application may access based on the Kubernetes services and service entries in the cluster. DNS lookup queries from the application are transparently intercepted and served by the Istio agent in the pod or VM. If the query is for a service within the mesh, irrespective of the cluster that the service is in, the agent responds directly to the application. If not, it forwards the query to the upstream name servers defined in /etc/resolv.conf. The following diagram depicts the interactions that occur when an application tries to access a service using its hostname.

Smart DNS proxying in Istio sidecar agent

As you will see in the following sections, the DNS proxying feature has had an enormous impact across many aspects of Istio.

Reduced load on your DNS servers w/ faster resolution

The load on your cluster’s Kubernetes DNS server drops drastically as almost all DNS queries are resolved within the pod by Istio. The bigger the footprint of mesh on a cluster, the lesser the load on your DNS servers. Implementing our own DNS proxy in the Istio agent has allowed us to implement cool optimizations such as CoreDNS auto-path without the correctness issues that CoreDNS currently faces.

To understand the impact of this optimization, lets take a simple DNS lookup scenario, in a standard Kubernetes cluster without any custom DNS setup for pods - i.e., with the default setting of ndots:5 in /etc/resolv.conf. When your application starts a DNS lookup for productpage.ns1.svc.cluster.local, it appends the DNS search namespaces in /etc/resolv.conf (e.g., ns1.svc.cluster.local) as part of the DNS query, before querying the host as-is. As a result, the first DNS query that is actually sent out will look like productpage.ns1.svc.cluster.local.ns1.svc.cluster.local, which will inevitably fail DNS resolution when Istio is not involved. If your /etc/resolv.conf has 5 search namespaces, the application will send two DNS queries for each search namespace, one for the IPv4 A record and another for the IPv6 AAAA record, and then a final pair of queries with the exact hostname used in the code. Before establishing the connection, the application performs 12 DNS lookup queries for each host!

With Istio’s implementation of the CoreDNS style auto-path technique, the sidecar agent will detect the real hostname being queried within the first query and return a cname record to productpage.ns1.svc.cluster.local as part of this DNS response, as well as the A/AAAA record for productpage.ns1.svc.cluster.local. The application receiving this response can now extract the IP address immediately and proceed to establishing a TCP connection to that IP. The smart DNS proxy in the Istio agent dramatically cuts down the number of DNS queries from 12 to just 2!

VMs to Kubernetes integration

Since the Istio agent performs local DNS resolution for services within the mesh, DNS lookup queries for Kubernetes services from VMs will now succeed without requiring clunky workarounds for exposing kube-dns outside the cluster. The ability to seamlessly resolve internal services in a cluster will now simplify your monolith to microservice journey, as the monolith on VMs can now access microservices on Kubernetes without additional levels of indirection via API gateways.

Automatic VIP allocation where possible

You may ask, how does this DNS functionality in the agent solve the problem of distinguishing between multiple external TCP services without VIPs on the same port?

Taking inspiration from Kubernetes, Istio will now automatically allocate non-routable VIPs (from the Class E subnet) to such services as long as they do not use a wildcard host. The Istio agent on the sidecar will use the VIPs as responses to the DNS lookup queries from the application. Envoy can now clearly distinguish traffic bound for each external TCP service and forward it to the right target. With the introduction of the DNS proxying, you will no longer need to use resolution: NONE for non-wildcard TCP services, improving your overall security posture. Istio cannot help much with wildcard external services (e.g., *.us-east1.rds.amazonaws.com). You will have to resort to NONE resolution mode to handle such services.

Multicluster DNS lookup

For the adventurous lot, attempting to weave a multicluster mesh where applications directly call internal services of a namespace in a remote cluster, the DNS proxy functionality comes in quite handy. Your applications can resolve Kubernetes services on any cluster in any namespace, without the need to create stub Kubernetes services in every cluster.

The benefits of the DNS proxy extend beyond the multicluster models that are currently described in Istio today. At Tetrate, we use this mechanism extensively in our customers’ multicluster deployments to enable sidecars to resolve DNS for hosts exposed at ingress gateways of all the clusters in a mesh, and access them over mutual TLS.

Concluding thoughts

The problems caused by lack of control over DNS have often been overlooked and ignored in its entirety when it comes to weaving a mesh across many clusters, different environments, and integrating external services. The introduction of a caching DNS proxy in the Istio sidecar agent solves these issues. Exercising control over the application’s DNS resolution allows Istio to accurately identify the target service to which traffic is bound, and enhance the overall security, routing, and telemetry posture in Istio within and across clusters.

Smart DNS proxying is enabled in the preview profile in Istio 1.8. Please try it out!

2020 Steering Committee Election Results

Istio Steering Committee — Tue, 29 Sep 2020 00:00:00 +0000

Last month, we announced a revision to our Steering Committee charter, opening up governance roles to more contributors and community members. The Steering Committee now consists of 9 proportionally-allocated Contribution Seats, and 4 elected Community Seats.

We have now concluded our inaugural election for the Community Seats, and we’re excited to welcome the following new members to the Committee:

Neeraj Poddar (Aspen Mesh)
Zack Butcher (Tetrate)
Christian Posta (Solo.io)
Zhonghu Xu (Huawei)

They join Contribution Seat holders from Google, IBM/Red Hat and Salesforce. We now have representation from 7 organizations on Steering, reflecting the breadth of our contributor ecosystem.

Thank you to everyone who participated in the election process. The next election will be in July 2021.

Large Scale Security Policy Performance Tests

Michael Eizaguirre (Google), Yangmin Zhu (Google), Carolyn Hu (Google) — Tue, 15 Sep 2020 00:00:00 +0000

Overview

Istio has a wide range of security policies which can be easily configured into systems of services. As the number of applied policies increases, it is important to understand the relationship of latency, memory usage, and CPU usage of the system.

This blog post goes over common security policies use cases and how the number of security policies or the number of specific rules in a security policy can affect the overall latency of requests.

Setup

There are a wide range of security policies and many more combinations of those policies. We will go over 6 of the most commonly used test cases.

The following test cases are run in an environment which consists of a Fortio client sending requests to a Fortio server, with a baseline of no Envoy sidecars deployed. The following data was gathered by using the Istio performance benchmarking tool.

In these test cases, requests either do not match any rules or match only the very last rule in the security policies. This ensures that the RBAC filter is applied to all policy rules, and never matches a policy rule before before viewing all the policies. Even though this is not necessarily what will happen in your own system, this policy setup provides data for the worst possible performance of each test case.

Test cases

Mutual TLS STRICT vs plaintext.
A single authorization policy with a variable number of principal rules as well as a PeerAuthentication policy. The principal rule is dependent on the PeerAuthentication policy being applied to the system.
A single authorization policy with a variable number of requestPrincipal rules as well as a RequestAuthentication policy. The requestPrincipal is dependent on the RequestAuthentication policy being applied to the system.
A single authorization policy with a variable number of paths vs sourceIP rules.
A variable number of authorization policies consisting of a single path or sourceIP rule.
A single RequestAuthentication policy with variable number of JWTRules rules.

Data

The y-axis of each test is the latency in milliseconds, and the x-axis is the number of concurrent connections. The x-axis of each graph consists of 3 data points that represent a small load (qps=100, conn=8), medium load (qps=500, conn=32), and large load (qps=1000, conn=64).

The difference of latency between MTLS mode STRICT and plaintext is very small in lower loads. As the qps and conn increase, the latency of requests with MTLS STRICT increases. The additional latency increased in larger loads is minimal compared to that of the increase from having no sidecars to having sidecars in the plaintext.

For Authorization policies with a variable number of requestPrincipal rules, the latency increase of 10 requestPrincipal rules compared to no policies is nearly the same as the latency increase of 1000 requestPrincipal rules compared to 10 requestPrincipal rules.

The latency increase of a single AuthZ policy with 10 sourceIP rules is not proportional to the latency increase of a single AuthZ policy with 1000 sourceIP rules compared to the system with sidecar and no policies.

The latency increase of a variable number of sourceIP rules is marginally greater than that of path rules.

The latency increase of a single AuthZ policy with 10 path rules is not proportional to the latency increase of a single AuthZ policy with 1000 path rules compared to the system with sidecar and no policies. This trend is similar to that of sourceIP rules.

The latency of a variable number of paths rules is marginally lesser than that of sourceIP rules.

To test how the number of Authorization policies affect runtime, the tests can be broken into two cases:

Every Authorization policy has a single sourceIP rule.
Every Authorization policy has a single path rule.

The overall trends of both graphs are similar. This is consistent to the paths vs sourceIP data, which showed that the latency is marginally greater for sourceIP rules than that of path rules.

Conclusion

In general, adding security policies does not add relatively high overhead to the system. The policies that add the most latency include:
1. Authorization policy with JWTRules rules.
2. Authorization policy with requestPrincipal rules.
3. Authorization policy with principals rules.
In lower loads (requests with lower qps and conn) the difference in latency for most policies is minimal.
Envoy proxy sidecars increase latency more than most policies, even if the policies are large.
The latency increase of extremely large policies is relatively similar to the latency increase of adding Envoy proxy sidecars compared to that of no sidecars.
Two different tests determined that the sourceIP rule is marginally slower than a path rule.

If you are interested in creating your own large scale security policies and running performance tests with them, see the performance benchmarking tool README.

If you are interested in reading more about the security policies tests, see our design doc. If you don’t already have access, you can join the Istio team drive.

Deploying Istio Control Planes Outside the Mesh

Lin Sun (IBM), Iris Ding (IBM) — Thu, 27 Aug 2020 00:00:00 +0000

Overview

From experience working with various service mesh users and vendors, we believe there are 3 key personas for a typical service mesh:

Mesh Operator, who manages the service mesh control plane installation and upgrade.
Mesh Admin, often referred as Platform Owner, who owns the service mesh platform and defines the overall strategy and implementation for service owners to adopt service mesh.
Mesh User, often referred as Service Owner, who owns one or more services in the mesh.

Prior to version 1.7, Istio required the control plane to run in one of the primary clusters in the mesh, leading to a lack of separation between the mesh operator and the mesh admin. Istio 1.7 introduces a new external control plane deployment model which enables mesh operators to install and manage mesh control planes on separate external clusters. This deployment model allows a clear separation between mesh operators and mesh admins. Istio mesh operators can now run Istio control planes for mesh admins while mesh admins can still control the configuration of the control plane without worrying about installing or managing the control plane. This model is transparent to mesh users.

External control plane deployment model

After installing Istio using the default installation profile, you will have an Istiod control plane installed in a single cluster like the diagram below:

Istio mesh in a single cluster

With the new deployment model in Istio 1.7, it’s possible to run Istiod on an external cluster, separate from the mesh services as shown in the diagram below. The external control plane cluster is owned by the mesh operator while the mesh admin owns the cluster running services deployed in the mesh. The mesh admin has no access to the external control plane cluster. Mesh operators can follow the external istiod single cluster step by step guide to explore more on this. (Note: In some internal discussions among Istio maintainers, this model was previously referred to as “central istiod”.)

Single cluster Istio mesh with Istiod in an external control plane cluster

Mesh admins can expand the service mesh to multiple clusters, which are managed by the same Istiod running in the external cluster. None of the mesh clusters are primary clusters, in this case. They are all remote clusters. However, one of them also serves as the Istio configuration cluster, in addition to running services. The external control plane reads Istio configurations from the config cluster and Istiod pushes configuration to the data plane running in both the config cluster and other remote clusters as shown in the diagram below.

Multicluster Istio mesh with Istiod in an external control plane cluster

Mesh operators can further expand this deployment model to manage multiple Istio control planes from an external cluster running multiple Istiod control planes:

Multiple single clusters with multiple Istiod control planes in an external control plane cluster

In this case, each Istiod manages its own remote cluster(s). Mesh operators can even install their own Istio mesh in the external control plane cluster and configure its istio-ingress gateway to route traffic from remote clusters to their corresponding Istiod control planes. To learn more about this, check out these steps.

Conclusion

The external control plane deployment model enables the Istio control plane to be run and managed by mesh operators who have operational expertise in Istio, and provides a clean separation between service mesh control and data planes. Mesh operators can run the control plane in their own clusters or other environments, providing the control plane as a service to mesh admins. Mesh operators can run multiple Istiod control planes in a single cluster, deploying their own Istio mesh and using istio-ingress gateways to control access to these Istiod control planes. Through the examples provided here, mesh operators can explore different implementation choices and choose what works best for them.

This new model reduces complexity for mesh admins by allowing them to focus on mesh configurations without operating the control plane themselves. Mesh admins can continue to configure mesh-wide settings and Istio resources without any access to external control plane clusters. Mesh users can continue to interact with the service mesh without any changes.

Introducing the new Istio steering committee

Istio Steering Committee — Mon, 24 Aug 2020 00:00:00 +0000

Today, the Istio project is pleased to announce a new revision to its steering charter, which opens up governance roles to more contributors and community members. This revision solidifies our commitment to open governance, ensuring that the community around the project will always be able to steer its direction, and that no one company has majority voting control over the project.

The Istio Steering Committee oversees the administrative aspects of the project and sets the marketing direction. From the earliest days of the project, it was bootstrapped with members from Google and IBM, the two founders and largest contributors, with the explicit intention that other seats would be added. We are very happy to deliver on that promise today, with a new charter designed to reward contribution and community.

The new Steering Committee consists of 13 seats: 9 proportionally allocated Contribution Seats, and 4 elected Community Seats.

Contribution Seats

The direction of a project is set by the people who contribute to it. We’ve designed our committee to reflect that, with 9 seats to be attributed in proportion to contributions made to Istio in the previous 12 months. In Kubernetes, the mantra was “chop wood, carry water,” and we similarly want to reward companies who are fueling the growth of the project with contributions.

This year, we’ve chosen to use merged pull requests as our proxy for proportional contribution. We know that no measure of contribution is perfect, and as such we will explicitly reconsider the formula every year. (Other measures we considered, including commits, comments, and actions, gave the same results for this period.)

In order to ensure corporate diversity, there will always be a minimum of three companies represented in Contribution Seats.

Community Seats

There are many wonderful contributors to the Istio community, including developers, SREs and mesh admins, working for companies large and small. We wanted to ensure that their voices were included, both in terms of representation and selection.

We have added 4 seats for representatives from 4 different organizations, who are not represented in the Contribution Seat allocation. These seats will be voted on by the Istio community in an annual election.

Any project member can stand for election; all Istio members who have been active in the last 12 months are eligible to vote.

Corporate diversification is the goal

Our goal is that the governance of Istio reflects the diverse set of contributors. Both Google and IBM/Red Hat will have fewer seats than previously, and the new model is designed to ensure representation from at least 7 different organizations.

We also want to make it clear that no single vendor, no matter how large their contribution, has majority voting control over the Istio project. We’ve implemented a cap on the number of seats a company can hold, such that they can neither unanimously win a vote, or veto a decision of the rest of the committee.

The 2020 committee and election

According to our seat allocation process, this year Google will be allocated 5 seats and IBM/Red Hat will be allocated 3. As the third largest contributor to Istio in the last 12 months, we are pleased to announce that Salesforce has earned a Contribution Seat.

The first election for Community Seats begins today. Members have two weeks to nominate themselves, and voting will run from 14 to 27 September. You can learn all about the election in the istio/community repository on GitHub. We’re also hosting a special community meeting this Thursday at 10:00 Pacific to discuss the changes and the election process. We’d love to see you there!

Using MOSN with Istio: an alternative data plane

Wang Fakang (mosn.io) — Tue, 28 Jul 2020 00:00:00 +0000

MOSN (Modular Open Smart Network) is a network proxy server written in GoLang. It was built at Ant Group as a sidecar/API Gateway/cloud-native Ingress/Layer 4 or Layer 7 load balancer etc. Over time, we’ve added extra features, like a multi-protocol framework, multi-process plug-in mechanism, a DSL, and support for the xDS APIs. Supporting xDS means we are now able to use MOSN as the network proxy for Istio. This configuration is not supported by the Istio project; for help, please see Learn More below.

Background

In the service mesh world, using Istio as the control plane has become the mainstream. Because Istio was built on Envoy, it uses Envoy’s data plane APIs (collectively known as the xDS APIs). These APIs have been standardized separately from Envoy, and so by implementing them in MOSN, we are able to drop in MOSN as a replacement for Envoy. Istio’s integration of third-party data planes can be implemented in three steps, as follows.

Implement xDS protocols to fulfill the capabilities for data plane related services.
Build proxyv2 images using Istio’s script and set the relevant SIDECAR and other parameters.
Specify a specific data plane via the istioctl tool and set the proxy-related configuration.

Architecture

MOSN has a layered architecture with four layers, NET/IO, Protocol, Stream, and Proxy, as shown in the following figure.

The architecture of MOSN

NET/IO acts as the network layer, monitoring connections and incoming packets, and as a mount point for the listener filter and network filter.
Protocol is the multi-protocol engine layer that examines packets and uses the corresponding protocol for decode/encode processing.
Stream does a secondary encapsulation of the decode packet into stream, which acts as a mount for the stream filter.
Proxy acts as a forwarding framework for MOSN, and does proxy processing on the encapsulated streams.

Why use MOSN?

Before the service mesh transformation, we have expected that as the next generation of Ant Group’s infrastructure, service mesh will inevitably bring revolutionary changes and evolution costs. We have a very ambitious blueprint: ready to integrate the original network and middleware various capabilities have been re-precipitated and polished to create a low-level platform for the next-generation architecture of the future, which will carry the responsibility of various service communications.

This is a long-term planning project that takes many years to build and meets the needs of the next five or even ten years, and cooperates to build a team that spans business, SRE, middleware, and infrastructure departments. We must have a network proxy forwarding plane with flexible expansion, high performance, and long-term evolution. Nginx and Envoy have a very long-term capacity accumulation and active community in the field of network agents. We have also borrowed from other excellent open source network agents such as Nginx and Envoy. At the same time, we have enhanced research and development efficiency and flexible expansion. Mesh transformation involves a large number of departments and R & D personnel. We must consider the landing cost of cross-team cooperation. Therefore, we have developed a new network proxy MOSN based on GoLang in the cloud-native scenario. For GoLang’s performance, we also did a full investigation and test in the early stage to meet the performance requirements of Ant Group’s services.

At the same time, we received a lot of feedback and needs from the end user community. Everyone has the same needs and thoughts. So we combined the actual situation of the community and ourselves to conduct the research and development of MOSN from the perspective of satisfying the community and users. We believe that the open source competition is mainly competition between standards and specifications. We need to make the most suitable implementation choice based on open source standards.

What is the difference between MOSN and Istio’s default proxy?

Differences in language stacks

MOSN is written in GoLang. GoLang has strong guarantees in terms of production efficiency and memory security. At the same time, GoLang has an extensive library ecosystem in the cloud-native era. The performance is acceptable and usable in the service mesh scenario. Therefore, MOSN has a lower intellectual cost for companies and individuals using languages such as GoLang and Java.

Differentiation of core competence

MOSN supports a multi-protocol framework, and users can easily access private protocols with a unified routing framework.
Multi-process plug-in mechanism, which can easily extend the plug-ins of independent MOSN processes through the plug-in framework, and do some other management, bypass and other functional module extensions.
Transport layer national secret algorithm support with Chinese encryption compliance, etc.

What are the drawbacks of MOSN

Because MOSN is written in GoLang, it doesn’t have as good performance as Istio default proxy, but the performance is acceptable and usable in the service mesh scenario.
Compared with Istio default proxy, some features are not fully supported, such as WASM, HTTP3, Lua, etc. However, these are all in the roadmap of MOSN, and the goal is to be fully compatible with Istio.

MOSN with Istio

The following describes how to set up MOSN as the data plane for Istio.

Setup Istio

You can download a zip file for your operating system from the Istio release page. This file contains: the installation file, examples and the istioctl command line tool. To download Istio (this example uses Istio 1.5.2) uses the following command.

$ export ISTIO_VERSION=1.5.2
$ curl -L https://istio.io/downloadIstio | sh -

The downloaded Istio package is named istio-1.5.2 and contains: - install/kubernetes: Contains YAML installation files related to Kubernetes. - examples/: Contains example applications. - bin/: Contains the istioctl client files.

Switch to the folder where Istio is located.

$ cd istio-$ISTIO_VERSION/

Add the istioctl client path to $PATH with the following command.

$ export PATH=$PATH:$(pwd)/bin

Setting MOSN as the Data Plane

It is possible to flexibly customize the Istio control plane and data plane configuration parameters using the istioctl command line tool. MOSN can be specified as the data plane for Istio using the following command.

$ istioctl manifest apply --set .values.global.proxy.image="mosnio/proxyv2:1.5.2-mosn" --set meshConfig.defaultConfig.binaryPath="/usr/local/bin/mosn"

Check that Istio-related pods and services are deployed successfully.

$ kubectl get svc -n istio-system

If the service STATUS is Running, then Istio has been successfully installed using MOSN and you can now deploy the Bookinfo sample.

Bookinfo Examples

You can run the Bookinfo sample by following the MOSN with Istio tutorial where you can find instructions for using MOSN and Istio. You can install MOSN and get to the same point you would have using the default Istio instructions with Envoy.

Moving forward

Next, MOSN will not only be compatible with the features of the latest version of Istio, but also evolve in the following aspects.

As a microservices runtime, MOSN oriented programming makes services lighter, smaller and faster.
Programmable, support WASM.
More scenario support, Cache Mesh/Message Mesh/Block-chain Mesh etc.

MOSN is an open source project that anyone in the community can use, improve, and enjoy. We’d love you to join us! Here are a few ways to find out what’s happening and get involved.

Learn More

Open and neutral: transferring our trademarks to the Open Usage Commons

Sean Suchter (Google) — Wed, 08 Jul 2020 00:00:00 +0000

Since day one, the Istio project has believed in the importance of being contributor-run, open, transparent and available to all. In that spirit, Google is pleased to announce that it will be transferring ownership of the project’s trademarks to the new Open Usage Commons.

Istio is an open source project, released under the Apache 2.0 license. That means people can copy, modify, distribute, make, use and sell the source code. The only freedom people don’t have under the Apache 2.0 license is to use the name Istio, or its logo, in a way that would confuse consumers.

As one of the founders of the project, Google is the current owner of the Istio trademark. While anyone who is using the software in accordance with the license can use the trademarks, the historic ownership has caused some confusion and uncertainty about who can use the name and how, and at times this confusion has been a barrier to community growth. So today, as part of Istio’s continued commitment to openness, Google is announcing that the Istio trademarks will be transferred to a new organization, the Open Usage Commons, to provide neutral, independent oversight of the marks.

A neutral home for Istio’s trademarks

The Open Usage Commons is a new organization that is focused solely on providing management and guidance of open source project trademarks in a way that is aligned with the Open Source Definition. For projects, particularly projects with robust ecosystems like Istio, ensuring that the trademark is available to anyone who is using the software in accordance with the license is important. The trademark allows maintainers to grow a community and use the name to do so. It also lets ecosystem partners create services on top of the project, and it enables developers to create tooling and integrations that reference the project. Maintainers, ecosystem partners, and developers alike must feel confident in their investments in Istio - for the long term. Google thinks having the Istio trademarks in the Open Usage Commons is the right way to give that clarity and provide that confidence.

The Open Usage Commons will work with the Istio Steering Committee to generate trademark usage guidelines. There will be no immediate changes to the Istio usage guidelines, and if you are currently using the Istio marks in a way that follows the existing brand guide, you can continue to do so.

You can learn more about open source project IP and the Open Usage Commons at openusage.org.

A continued commitment to open

The Open Usage Commons is focused on project trademarks; it does not address other facets of an open project, like rules around who gets decision-making votes. Similar to many projects in their early days, Istio’s committees started as small groups that stemmed from the founding companies. But Istio has grown and matured (last year Istio was #4 on GitHub’s list of fastest growing open source projects!), and it is time for the next evolution of Istio’s governance.

Recently, we were proud to appoint Neeraj Poddar, Co-founder & Chief Architect of Aspen Mesh, to the Technical Oversight Committee — the group responsible for all technical decision-making in the project. Neeraj is a long-time contributor to the project and served as a Working Group lead. The TOC is now made up of 7 members from 4 different companies - Tetrate, IBM, Google & now Aspen Mesh.

Our community is currently discussing how the Steering Committee, which oversees marketing and community activities, should be governed, to reflect the expanding community and ecosystem. If you have ideas for this new governance, visit the pull request on GitHub where an active discussion is taking place.

In the last 12 months, Istio has had commits from more than 100 organizations and currently has 70 maintainers from 14 different companies. This trend is the kind of contributor diversity the project’s founders intended, and nurturing it remains a priority. Google is excited about what the future holds for Istio, and hopes you’ll be a part of it.

Reworking our Addon Integrations

John Howard (Google) — Thu, 04 Jun 2020 00:00:00 +0000

Starting with Istio 1.6, we are introducing a new method for integration with telemetry addons, such as Grafana, Prometheus, Zipkin, Jaeger, and Kiali.

In previous releases, these addons were bundled as part of the Istio installation. This allowed users to quickly get started with Istio without any complicated configurations to install and integrate these addons. However, it came with some issues:

The Istio addon installations were not as up to date or feature rich as upstream installation methods. Users were left missing out on some of the great features provided by these applications, such as:
- Persistent storage
- Features like Alertmanager for Prometheus
- Advanced security settings
Integration with existing deployments that were using these features was more challenging than it should be.

Changes

In order to address these gaps, we have made a number of changes:

Added a new Integrations documentation section to explain which applications Istio can integrate with, how to use them, and best practices.
Reduced the amount of configuration required to set up telemetry addons
- Grafana dashboards are now published to grafana.com.
- Prometheus can now scrape all Istio pods using standard prometheus.io annotations. This allows most Prometheus deployments to work with Istio without any special configuration.
Removed the bundled addon installations from istioctl and the operator. Istio does not install components that are not delivered by the Istio project. As a result, Istio will stop shipping installation artifacts related to addons. However, Istio will guarantee version compatibility where necessary. It is the user’s responsibility to install these components by using the official Integrations documentation and artifacts provided by the respective projects. For demos, users can deploy simple YAML files from the samples/addons/ directory.

We hope these changes allow users to make the most of these addons so as to fully experience what Istio can offer.

Timeline

Istio 1.6: The new demo deployments for telemetry addons are available under samples/addons/ directory.
Istio 1.7: Upstream installation methods or the new samples deployment are the recommended installation methods. Installation by istioctl is deprecated.
Istio 1.8: Installation of addons by istioctl is removed.

Introducing Workload Entries

Thu, 21 May 2020 00:00:00 +0000

Introducing Workload Entries: Bridging Kubernetes and VMs

Historically, Istio has provided great experience to workloads that run on Kubernetes, but it has been less smooth for other types of workloads, such as Virtual Machines (VMs) and bare metal. The gaps included the inability to declaratively specify the properties of a sidecar on a VM, inability to properly respond to the lifecycle changes of the workload (e.g., booting to not ready to ready, or health checks), and cumbersome DNS workarounds as the workloads are migrated into Kubernetes to name a few.

Istio 1.6 has introduced a few changes in how you manage non-Kubernetes workloads, driven by a desire to make it easier to gain Istio’s benefits for use cases beyond containers, such as running traditional databases on a platform outside of Kubernetes, or adopting Istio’s features for existing applications without rewriting them.

Background

Prior to Istio 1.6, non-containerized workloads were configurable simply as an IP address in a ServiceEntry, which meant that they only existed as part of a service. Istio lacked a first-class abstraction for these non-containerized workloads, something similar to how Kubernetes treats Pods as the fundamental unit of compute - a named object that serves as the collection point for all things related to a workload - name, labels, security properties, lifecycle status events, etc. Enter WorkloadEntry.

Consider the following ServiceEntry describing a service implemented by a few tens of VMs with IP addresses:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: svc1
spec:
hosts:
- svc1.internal.com
ports:
- number: 80
name: http
protocol: HTTP
resolution: STATIC
endpoints:
- address: 1.1.1.1
- address: 2.2.2.2
....

If you wanted to migrate this service into Kubernetes in an active-active manner - i.e. launch a bunch of Pods, send a portion of the traffic to the Pods over Istio mutual TLS (mTLS) and send the rest to the VMs without sidecars - how would you do it? You would have needed to use a combination of a Kubernetes service, a virtual service, and a destination rule to achieve the behavior. Now, let’s say you decided to add sidecars to these VMs, one by one, such that you want only the traffic to the VMs with sidecars to use Istio mTLS. If any other Service Entry happens to include the same VM in its addresses, things start to get very complicated and error prone.

The primary source of these complications is that Istio lacked a first-class definition of a non-containerized workload, whose properties can be described independently of the service(s) it is part of.

The Internal of Service Entries Pointing to Workload Entries

Workload Entry: A Non-Kubernetes Endpoint

WorkloadEntry was created specifically to solve this problem. WorkloadEntry allows you to describe non-Pod endpoints that should still be part of the mesh, and treat them the same as a Pod. From here everything becomes easier, like enabling MUTUAL_TLS between workloads, whether they are containerized or not.

To create a WorkloadEntry and attach it to a ServiceEntry you can do something like this:

---
apiVersion: networking.istio.io/v1alpha3
kind: WorkloadEntry
metadata:
name: vm1
namespace: ns1
spec:
address: 1.1.1.1
labels:
app: foo
instance-id: vm-78ad2
class: vm
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: svc1
namespace: ns1
spec:
hosts:
- svc1.internal.com
ports:
- number: 80
name: http
protocol: HTTP
resolution: STATIC
workloadSelector:
labels:
app: foo

This creates a new WorkloadEntry with a set of labels and an address, and a ServiceEntry that uses a WorkloadSelector to select all endpoints with the desired labels, in this case including the WorkloadEntry that are created for the VM.

The Internal of Service Entries Pointing to Workload Entries

Notice that the ServiceEntry can reference both Pods and WorkloadEntries, using the same selector. VMs and Pods can now be treated identically by Istio, rather than being kept separate.

If you were to migrate some of your workloads to Kubernetes, and you choose to keep a substantial number of your VMs, the WorkloadSelector can select both Pods and VMs, and Istio will automatically load balance between them. The 1.6 changes also mean that WorkloadSelector syncs configurations between the Pods and VMs and removes the manual requirement to target both infrastructures with duplicate policies like mTLS and authorization. The Istio 1.6 release provides a great starting point for what will be possible for the future of Istio. The ability to describe what exists outside of the mesh the same way you do with a Pod leads to added benefits like improved bootstrapping experience. However, these benefits are merely side effects. The core benefit is you can now have VMs, and Pods co-exist without any configuration needed to bridge the two together.

Safely Upgrade Istio using a Canary Control Plane Deployment

John Howard (Google) — Tue, 19 May 2020 00:00:00 +0000

Canary deployments are a core feature of Istio. Users rely on Istio’s traffic management features to safely control the rollout of new versions of their applications, while making use of Istio’s rich telemetry to compare the performance of canaries. However, when it came to upgrading Istio, there was not an easy way to canary the upgrade, and due to the in-place nature of the upgrade, issues or changes found affect the entire mesh at once.

Istio 1.6 will support a new upgrade model to safely canary-deploy new versions of Istio. In this new model, proxies will associate with a specific control plane that they use. This allows a new version to deploy to the cluster with less risk - no proxies connect to the new version until the user explicitly chooses to. This allows gradually migrating workloads to the new control plane, while monitoring changes using Istio telemetry to investigate any issues, just like using VirtualService for workloads. Each independent control plane is referred to as a “revision” and has an istio.io/rev label.

Understanding upgrades

Upgrading Istio is a complicated process. During the transition period between two versions, which might take a long time for large clusters, there are version differences between proxies and the control plane. In the old model the old and new control planes use the same Service, traffic is randomly distributed between the two, offering no control to the user. However, in the new model, there is not cross-version communication. Look at how the upgrade changes:

Configuring

Control plane selection is done based on the sidecar injection webhook. Each control plane is configured to select objects with a matching istio.io/rev label on the namespace. Then, the upgrade process configures the pods to connect to a control plane specific to that revision. Unlike in the current model, this means that a given proxy connects to the same revision during its lifetime. This avoids subtle issues that might arise when a proxy switches which control plane it is connected to.

The new istio.io/rev label will replace the istio-injection=enabled label when using revisions. For example, if we had a revision named canary, we would label our namespaces that we want to use this revision with istio.io/rev=canary. See the upgrade guide for more information.

Direct encrypted traffic from IBM Cloud Kubernetes Service Ingress to Istio Ingress Gateway

Vadim Eisenberg (IBM) — Fri, 15 May 2020 00:00:00 +0000

In this blog post I show how to configure the Ingress Application Load Balancer (ALB) on IBM Cloud Kubernetes Service (IKS) to direct traffic to the Istio ingress gateway, while securing the traffic between them using mutual TLS authentication.

When you use IKS without Istio, you may control your ingress traffic using the provided ALB. This ingress-traffic routing is configured using a Kubernetes Ingress resource with ALB-specific annotations. IKS provides a DNS domain name, a TLS certificate that matches the domain, and a private key for the certificate. IKS stores the certificates and the private key in a Kubernetes secret.

When you start using Istio in your IKS cluster, the recommended method to send traffic to your Istio enabled workloads is by using the Istio Ingress Gateway instead of using the Kubernetes Ingress. One of the main reasons to use the Istio ingress gateway is the fact the ALB provided by IKS will not be able to communicate directly with the services inside the mesh when you enable STRICT mutual TLS. During your transition to having only Istio ingress gateway as your main entry point, you can continue to use the traditional Ingress for non-Istio services while using the Istio ingress gateway for services that are part of the mesh.

IKS provides a convenient way for clients to access Istio ingress gateway by letting you register a new DNS subdomain for the Istio gateway’s IP with an IKS command. The domain is in the following format: <cluster_name>-<globally_unique_account_HASH>-0001.<region>.containers.appdomain.cloud, for example mycluster-a1b2cdef345678g9hi012j3kl4567890-0001.us-south.containers.appdomain.cloud. In the same way as for the ALB domain, IKS provides a certificate and a private key, storing them in another Kubernetes secret.

This blog describes how you can chain together the IKS Ingress ALB and the Istio ingress gateway to send traffic to your Istio enabled workloads while being able to continue using the ALB specific features and the ALB subdomain name. You configure the IKS Ingress ALB to direct traffic to the services inside an Istio service mesh through the Istio ingress gateway, while using mutual TLS authentication between the ALB and the gateway. For the mutual TLS authentication, you will configure the ALB and the Istio ingress gateway to use the certificates and keys provided by IKS for the ALB and NLB subdomains. Using certificates provided by IKS saves you the overhead of managing your own certificates for the connection between the ALB and the Istio ingress gateway.

You will use the NLB subdomain certificate as the server certificate for the Istio ingress gateway as intended. The NLB subdomain certificate represents the identity of the server that serves a particular NLB subdomain, in this case, the ingress gateway.

You will use the ALB subdomain certificate as the client certificate in mutual TLS authentication between the ALB and the Istio Ingress. When ALB acts as a server it presents the ALB certificate to the clients so the clients can authenticate the ALB. When ALB acts as a client of the Istio ingress gateway, it presents the same certificate to the Istio ingress gateway, so the Istio ingress gateway could authenticate the ALB.

Traffic to the services without an Istio sidecar can continue to flow as before directly from the ALB.

The diagram below exemplifies the described setting. It shows two services in the cluster, service A and service B. service A has an Istio sidecar injected and requires mutual TLS. service B has no Istio sidecar. service B can be accessed by clients through the ALB, which directly communicates with service B. service A can be also accessed by clients through the ALB, but in this case the traffic must pass through the Istio ingress gateway. Mutual TLS authentication between the ALB and the gateway is based on the certificates provided by IKS. The clients can also access the Istio ingress gateway directly. IKS registers different DNS domains for the ALB and for the ingress gateway.

A cluster with the ALB and the Istio ingress gateway

Initial setting

Create the httptools namespace and enable Istio sidecar injection:

$ kubectl create namespace httptools
$ kubectl label namespace httptools istio-injection=enabled
namespace/httptools created
namespace/httptools labeled

Deploy the httpbin sample to httptools:

$ kubectl apply -f @samples/httpbin/httpbin.yaml@ -n httptools
service/httpbin created
deployment.apps/httpbin created

Create secrets for the ALB and the Istio ingress gateway

IKS generates a TLS certificate and a private key and stores them as a secret in the default namespace when you register a DNS domain for an external IP by using the ibmcloud ks nlb-dns-create command. IKS stores the ALB’s certificate and private key also as a secret in the default namespace. You need these credentials to establish the identities that the ALB and the Istio ingress gateway will present during the mutual TLS authentication between them. You will configure the ALB and the Istio ingress gateway to exchange these certificates, to trust the certificates of one another, and to use their private keys to encrypt and sign the traffic.

Store the name of your cluster in the CLUSTER_NAME environment variable:
```
$ export CLUSTER_NAME=<your cluster name>
```

Store the domain name of your ALB in the ALB_INGRESS_DOMAIN environment variable:

$ ibmcloud ks cluster get --cluster $CLUSTER_NAME | grep Ingress
Ingress Subdomain: <your ALB ingress domain>
Ingress Secret: <your ALB secret>

$ export ALB_INGRESS_DOMAIN=<your ALB ingress domain>
$ export ALB_SECRET=<your ALB secret>

Store the external IP of your istio-ingressgateway service in an environment variable.

$ export INGRESS_GATEWAY_IP=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo INGRESS_GATEWAY_IP = $INGRESS_GATEWAY_IP

Create a DNS domain and certificates for the IP of the Istio Ingress Gateway service:

$ ibmcloud ks nlb-dns create classic --cluster $CLUSTER_NAME --ip $INGRESS_GATEWAY_IP --secret-namespace istio-system
Host name subdomain is created as <some domain>

Store the domain name from the previous command in an environment variable:
```
$ export INGRESS_GATEWAY_DOMAIN=<the domain from the previous command>
```

List the registered domain names:

$ ibmcloud ks nlb-dnss --cluster $CLUSTER_NAME
Retrieving host names, certificates, IPs, and health check monitors for network load balancer (NLB) pods in cluster <your cluster>...
OK
Hostname IP(s) Health Monitor SSL Cert Status SSL Cert Secret Name Secret Namespace
<your ingress gateway hostname> <your ingress gateway IP> None created <the matching secret name> istio-system
...

Wait until the status of the certificate (the fourth field) of the new domain name becomes enabled (initially it is pending).

Store the name of the secret of the new domain name:

$ export INGRESS_GATEWAY_SECRET=<the secret's name as shown in the SSL Cert Secret Name column>

Extract the certificate and the key from the secret provided for the ALB:

$ mkdir alb_certs
$ kubectl get secret $ALB_SECRET --namespace=default -o yaml | grep 'tls.key:' | cut -f2 -d: | base64 --decode > alb_certs/client.key
$ kubectl get secret $ALB_SECRET --namespace=default -o yaml | grep 'tls.crt:' | cut -f2 -d: | base64 --decode > alb_certs/client.crt
$ ls -al alb_certs
-rw-r--r-- 1 user staff 3738 Sep 11 07:57 client.crt
-rw-r--r-- 1 user staff 1675 Sep 11 07:57 client.key

Download the issuer certificate of the Let’s Encrypt certificate, which is the issuer of the certificates provided by IKS. You specify this certificate as the certificate of a certificate authority to trust, for both the ALB and the Istio ingress gateway.
```
$ curl https://letsencrypt.org/certs/trustid-x3-root.pem --output trusted.crt
```
Create a Kubernetes secret to be used by the ALB to establish mutual TLS connection.

The certificates provided by IKS expire every 90 days and are automatically renewed by IKS 37 days before they expire. You will have to recreate the secrets by rerunning the instructions of this section every time the secrets provided by IKS are updated. You may want to use scripts or operators to automate this and keep the secrets in sync.
```
$ kubectl create secret generic alb-certs -n istio-system --from-file=trusted.crt --from-file=alb_certs/client.crt --from-file=alb_certs/client.key
secret "alb-certs" created
```

For mutual TLS, a separate Secret named <tls-cert-secret>-cacert with a cacert key is needed for the ingress gateway.

$ kubectl create -n istio-system secret generic $INGRESS_GATEWAY_SECRET-cacert --from-file=ca.crt=trusted.crt
secret/cluster_name-hash-XXXX-cacert created

Configure a mutual TLS ingress gateway

In this section you configure the Istio ingress gateway to perform mutual TLS between external clients and the gateway. You use the certificates and the keys provided to you for the ingress gateway and the ALB.

Define a Gateway to allow access on port 443 only, with mutual TLS:

$ kubectl apply -n httptools -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: default-ingress-gateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: MUTUAL
credentialName: $INGRESS_GATEWAY_SECRET
hosts:
- "$INGRESS_GATEWAY_DOMAIN"
- "httpbin.$ALB_INGRESS_DOMAIN"
EOF

Configure routes for traffic entering via the Gateway:

$ kubectl apply -n httptools -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: default-ingress
spec:
hosts:
- "$INGRESS_GATEWAY_DOMAIN"
- "httpbin.$ALB_INGRESS_DOMAIN"
gateways:
- default-ingress-gateway
http:
- match:
- uri:
prefix: /status
route:
- destination:
port:
number: 8000
host: httpbin.httptools.svc.cluster.local
EOF

Send a request to httpbin by curl, passing as parameters the client certificate (the --cert option) and the private key (the --key option):

$ curl https://$INGRESS_GATEWAY_DOMAIN/status/418 --cert alb_certs/client.crt --key alb_certs/client.key
-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`

Remove the directories with the ALB and ingress gateway certificates and keys.
```
$ rm -r alb_certs trusted.crt
```

Configure the ALB

You need to configure your Ingress resource to direct traffic to the Istio ingress gateway while using the certificate stored in the alb-certs secret. Normally, the ALB decrypts HTTPS requests before forwarding traffic to your apps. You can configure the ALB to re-encrypt the traffic before it is forwarded to the Istio ingress gateway by using the ssl-services annotation on the Ingress resource. This annotation also allows you to specify the certificate stored in the alb-certs secret, required for mutual TLS.

Configure the Ingress resource for the ALB. You must create the Ingress resource in the istio-system namespace in order to forward the traffic to the Istio ingress gateway.

$ kubectl apply -f - <<EOF
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: alb-ingress
namespace: istio-system
annotations:
ingress.bluemix.net/ssl-services: "ssl-service=istio-ingressgateway ssl-secret=alb-certs proxy-ssl-name=$INGRESS_GATEWAY_DOMAIN"
spec:
tls:
- hosts:
- httpbin.$ALB_INGRESS_DOMAIN
secretName: $ALB_SECRET
rules:
- host: httpbin.$ALB_INGRESS_DOMAIN
http:
paths:
- path: /status
backend:
serviceName: istio-ingressgateway
servicePort: 443
EOF

Test the ALB ingress:

$ curl https://httpbin.$ALB_INGRESS_DOMAIN/status/418
-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`

Congratulations! You configured the IKS Ingress ALB to send encrypted traffic to the Istio ingress gateway. You allocated a host name and certificate for your Istio ingress gateway and used that certificate as the server certificate for Istio ingress gateway. As the client certificate of the ALB you used the certificate provided by IKS for the ALB. Once you had the certificates deployed as Kubernetes secrets, you directed the ingress traffic from the ALB to the Istio ingress gateway for some specific paths and used the certificates for mutual TLS authentication between the ALB and the Istio ingress gateway.

Cleanup

Delete the Gateway configuration, the VirtualService, and the secrets:

$ kubectl delete ingress alb-ingress -n istio-system
$ kubectl delete virtualservice default-ingress -n httptools
$ kubectl delete gateway default-ingress-gateway -n httptools
$ kubectl delete secrets alb-certs -n istio-system
$ rm -rf alb_certs trusted.crt
$ unset CLUSTER_NAME ALB_INGRESS_DOMAIN ALB_SECRET INGRESS_GATEWAY_DOMAIN INGRESS_GATEWAY_SECRET

Shutdown the httpbin service:

$ kubectl delete -f @samples/httpbin/httpbin.yaml@ -n httptools

Delete the httptools namespace:
```
$ kubectl delete namespace httptools
```

Provision a certificate and key for an application without sidecars

Lei Tang (Google) — Wed, 25 Mar 2020 00:00:00 +0000

Istio sidecars obtain their certificates using the secret discovery service. A service in the service mesh may not need (or want) an Envoy sidecar to handle its traffic. In this case, the service will need to obtain a certificate itself if it wants to connect to other TLS or mutual TLS secured services.

For a service with no need of a sidecar to manage its traffic, a sidecar can nevertheless still be deployed only to provision the private key and certificates through the CSR flow from the CA and then share the certificate with the service through a mounted file in tmpfs. We have used Prometheus as our example application for provisioning a certificate using this mechanism.

In the example application (i.e., Prometheus), a sidecar is added to the Prometheus deployment by setting the flag .Values.prometheus.provisionPrometheusCert to true (this flag is set to true by default in an Istio installation). This deployed sidecar will then request and share a certificate with Prometheus.

The key and certificate provisioned for the example application are mounted in the directory /etc/istio-certs/. We can list the key and certificate provisioned for the application by running the following command:

$ kubectl exec -it `kubectl get pod -l app=prometheus -n istio-system -o jsonpath='{.items[0].metadata.name}'` -c prometheus -n istio-system -- ls -la /etc/istio-certs/

The output from the above command should include non-empty key and certificate files, similar to the following:

-rwxr-xr-x 1 root root 2209 Feb 25 13:06 cert-chain.pem
-rwxr-xr-x 1 root root 1679 Feb 25 13:06 key.pem
-rwxr-xr-x 1 root root 1054 Feb 25 13:06 root-cert.pem

If you want to use this mechanism to provision a certificate for your own application, take a look at our Prometheus example application and simply follow the same pattern.

Extended and Improved WebAssemblyHub to Bring the Power of WebAssembly to Envoy and Istio

Idit Levine (Solo.io) — Wed, 25 Mar 2020 00:00:00 +0000

Originally posted on the Solo.io blog

As organizations adopt Envoy-based infrastructure like Istio to help solve challenges with microservices communication, they inevitably find themselves needing to customize some part of that infrastructure to fit within their organization’s constraints. WebAssembly (Wasm) has emerged as a safe, secure, and dynamic environment for platform extension.

In the recent announcement of Istio 1.5, the Istio project lays the foundation for bringing WebAssembly to the popular Envoy proxy. Solo.io is collaborating with Google and the Istio community to simplify the overall experience of creating, sharing, and deploying WebAssembly extensions to Envoy and Istio. It wasn’t that long ago that Google and others laid the foundation for containers, and Docker built a great user experience to make it consumable. Similarly, this effort makes Wasm consumable by building the best user experience for WebAssembly on Istio.

Back in December 2019, Solo.io began an effort to provide a great developer experience for WebAssembly with the announcement of WebAssembly Hub. The WebAssembly Hub allows developers to very quickly spin up a new WebAssembly project in C++ (we’re expanding this language choice, see below), build it using Bazel in Docker, and push it to an OCI-compliant registry. From there, operators had to pull the module, and configure Envoy proxies themselves to load it from disk. Beta support in Gloo, an API Gateway built on Envoy allows you to declaratively and dynamically load the module, the Solo.io team wanted to bring the same effortless and secure experience to other Envoy-based frameworks as well - like Istio.

There has been a lot of interest in the innovation in this area, and the Solo.io team has been working hard to further the capabilities of WebAssembly Hub and the workflows it supports. In conjunction with Istio 1.5, Solo.io is thrilled to announce new enhancements to WebAssembly Hub that evolve the viability of WebAssembly with Envoy for production, improve the developer experience, and streamline using Wasm with Envoy in Istio.

Evolving toward production

The Envoy community is working hard to bring Wasm support into the upstream project (right now it lives on a working development fork), with Istio declaring Wasm support an Alpha feature. In Gloo 1.0, we also announced early, non-production support for Wasm. What is Gloo? Gloo is a modern API Gateway and Ingress Controller (built on Envoy Proxy) that supports routing and securing incoming traffic to legacy monoliths, microservices / Kubernetes and serverless functions. Dev and ops teams are able to shape and control traffic patterns from external end users/clients to backend application services. Gloo is a Kubernetes and Istio native ingress gateway.

Although it’s still maturing in each individual project, there are things that we, as a community, can do to improve the foundation for production support.

The first area is standardizing what a WebAssembly extension for Envoy looks like. Solo.io, Google, and the Istio community have defined an open specification for bundling and distributing WebAssembly modules as OCI images. This specification provides a powerful model for distributing any type of Wasm module including Envoy extensions.

This is open to the community - Join in the effort

The next area is improving the experience of deploying Wasm extensions into an Envoy-based framework running in production. In the Kubernetes ecosystem, it is considered a best practice in production to use declarative CRD-based configuration to manage cluster configuration. The new WebAssembly Hub Operator adds a single, declarative CRD which automatically deploys and configures Wasm filters to Envoy proxies running inside of a Kubernetes cluster. This operator enables GitOps workflows and cluster automation to manage Wasm filters without human intervention or imperative workflows. We will provide more information about the Operator in an upcoming blog post.

Lastly, the interactions between developers of Wasm extensions and the teams that deploy them need some kind of role-based access, organization management, and facilities to share, discover, and consume these extensions. The WebAssembly Hub adds team management features like permissions, organizations, user management, sharing, and more.

Improving the developer experience

As developers want to target more languages and runtimes, the experience must be kept as simple and as productive as possible. Multi-language support and runtime ABI (Application Binary Interface) targets should be handled automatically in tooling.

One of the benefits of Wasm is the ability to write modules in many languages. The collaboration between Solo.io and Google provides out-of-the-box support for Envoy filters written in C++, Rust, and AssemblyScript. We will continue to add support for more languages.

Wasm extensions use the Application Binary Interface (ABI) within the Envoy proxy to which they are deployed. The WebAssembly Hub provides strong ABI versioning guarantees between Envoy, Istio, and Gloo to prevent unpredictable behavior and bugs. All you have to worry about is writing your extension code.

Lastly, like Docker, the WebAssembly Hub stores and distributes Wasm extensions as OCI images. This makes pushing, pulling, and running extensions as easy as Docker containers. Wasm extension images are versioned and cryptographically secure, making it safe to run extensions locally the same way you would in production. This allows you to build and push as well as trust the source when they pull down and deploy images.

WebAssembly Hub with Istio

The WebAssembly Hub now fully automates the process of deploying Wasm extensions to Istio, (as well as other Envoy-based frameworks like Gloo API Gateway) installed in Kubernetes. With this deployment feature, the WebAssembly Hub relieves the operator or end user from having to manually configure the Envoy proxy in their Istio service mesh to use their WebAssembly modules.

Take a look at the following video to see just how easy it is to get started with WebAssembly and Istio:

Get Started

We hope that the WebAssembly Hub will become a meeting place for the community to share, discover, and distribute Wasm extensions. By providing a great user experience, we hope to make developing, installing, and running Wasm easier and more rewarding. Join us at the WebAssembly Hub, share your extensions and ideas, and join an upcoming webinar.

Introducing istiod: simplifying the control plane

Craig Box (Google) — Thu, 19 Mar 2020 00:00:00 +0000

Microservices are a great pattern when they map services to disparate teams that deliver them, or when the value of independent rollout and the value of independent scale are greater than the cost of orchestration. We regularly talk to customers and teams running Istio in the real world, and they told us that none of these were the case for the Istio control plane. So, in Istio 1.5, we’ve changed how Istio is packaged, consolidating the control plane functionality into a single binary called istiod.

History of the Istio control plane

Istio implements a pattern that has been in use at both Google and IBM for many years, which later became known as “service mesh”. By pairing client and server processes with proxy servers, they act as an application-aware data plane that’s not simply moving packets around hosts, or pulses over wires.

This pattern helps the world come to terms with microservices: fine-grained, loosely-coupled services connected via lightweight protocols. The common cross-platform and cross-language standards like HTTP and gRPC that replace proprietary transports, and the widespread presence of the needed libraries, empower different teams to write different parts of an overall architecture in whatever language makes the most sense. Furthermore, each service can scale independently as needed. A desire to implement security, observability and traffic control for such a network powers Istio’s popularity.

Istio’s control plane is, itself, a modern, cloud-native application. Thus, it was built from the start as a set of microservices. Individual Istio components like service discovery (Pilot), configuration (Galley), certificate generation (Citadel) and extensibility (Mixer) were all written and deployed as separate microservices. The need for these components to communicate securely and be observable, provided opportunities for Istio to eat its own dogfood (or “drink its own champagne”, to use a more French version of the metaphor!).

The cost of complexity

Good teams look back upon their choices and, with the benefit of hindsight, revisit them. Generally, when a team adopts microservices and their inherent complexity, they look for improvements in other areas to justify the tradeoffs. Let’s look at the Istio control plane through that lens.

Microservices empower you to write in different languages. The data plane (the Envoy proxy) is written in C++, and this boundary benefits from a clean separation in terms of the xDS APIs. However, all of the Istio control plane components are written in Go. We were able to choose the appropriate language for the appropriate job: highly performant C++ for the proxy, but accessible and speedy-development for everything else.
Microservices empower you to allow different teams to manage services individually.. In the vast majority of Istio installations, all the components are installed and operated by a single team or individual. The componentization done within Istio is aligned along the boundaries of the development teams who build it. This would make sense if the Istio components were delivered as a managed service by the people who wrote them, but this is not the case! Making life simpler for the development teams had an outsized impact of the usability for the orders-of-magnitude more users.
Microservices empower you to decouple versions, and release different components at different times. All the components of the control plane have always been released at the same version, at the same time. We have never tested or supported running different versions of (for example) Citadel and Pilot.
Microservices empower you to scale components independently. In Istio 1.5, control plane costs are dominated by a single feature: serving the Envoy xDS APIs that program the data plane. Every other feature has a marginal cost, which means there is very little value to having those features in separately-scalable microservices.
Microservices empower you to maintain security boundaries. Another good reason to separate an application into different microservices is if they have different security roles. Multiple Istio microservices like the sidecar injector, the Envoy bootstrap, Citadel, and Pilot hold nearly equivalent permissions to change the proxy configuration. Therefore, exploiting any of these services would cause near equivalent damage. When you deploy Istio, all the components are installed by default into the same Kubernetes namespace, offering limited security isolation.

The benefit of consolidation: introducing istiod

Having established that many of the common benefits of microservices didn’t apply to the Istio control plane, we decided to unify them into a single binary: istiod (the ’d’ is for daemon).

Let’s look at the benefits of the new packaging:

Installation becomes easier. Fewer Kubernetes deployments and associated configurations are required, so the set of configuration options and flags for Istio is reduced significantly. In the simplest case, you can start the Istio control plane, with all features enabled, by starting a single Pod.
Configuration becomes easier. Many of the configuration options that Istio has today are ways to orchestrate the control plane components, and so are no longer needed. You also no longer need to change cluster-wide PodSecurityPolicy to deploy Istio.
Using VMs becomes easier. To add a workload to a mesh, you now just need to install one agent and the generated certificates. That agent connects back to only a single service.
Maintenance becomes easier. Installing, upgrading, and removing Istio no longer require a complicated dance of version dependencies and startup orders. For example: To upgrade, you only need to start a new istiod version alongside your existing control plane, canary it, and then move all traffic over to it.
Scalability becomes easier. There is now only one component to scale.
Debugging becomes easier. Fewer components means less cross-component environmental debugging.
Startup time goes down. Components no longer need to wait for each other to start in a defined order.
Resource usage goes down and responsiveness goes up. Communication between components becomes guaranteed, and not subject to gRPC size limits. Caches can be shared safely, which decreases the resource footprint as a result.

istiod unifies functionality that Pilot, Galley, Citadel and the sidecar injector previously performed, into a single binary.

A separate component, the istio-agent, helps each sidecar connect to the mesh by securely passing configuration and secrets to the Envoy proxies. While the agent, strictly speaking, is still part of the control plane, it runs on a per-pod basis. We’ve further simplified by rolling per-node functionality that used to run as a DaemonSet, into that per-pod agent.

Extra for experts

There will still be some cases where you might want to run Istio components independently, or replace certain components.

Some users might want to use a Certificate Authority (CA) outside the mesh, and we have documentation on how to do that. If you do your certificate provisioning using a different tool, we can use that instead of the built-in CA.

Moving forward

At its heart, istiod is just a packaging and optimization change. It’s built on the same code and API contracts as the separate components, and remains covered by our comprehensive test suite. This gives us confidence in making it the default in Istio 1.5. The service is now called istiod - you’ll see an istio-pilot for existing proxies as the upgrade process completes.

While the move to istiod may seem like a big change, and is a huge improvement for the people who administer and maintain the mesh, it won’t make the day-to-day life of using Istio any different. istiod is not changing any of the APIs used to configure your mesh, so your existing processes will all stay the same.

Does this change imply that microservice are a mistake for all workloads and architectures? Of course not. They are a tool in a toolbelt, and they work best when they are reflected in your organizational reality. Instead, this change shows a willingness in the project to change based on user feedback, and a continued focus on simplification for all users. Microservices have to be right sized, and we believe we have found the right size for Istio.

Declarative WebAssembly deployment for Istio

Christian Posta (Solo.io) — Mon, 16 Mar 2020 00:00:00 +0000

As outlined in the Istio 2020 trade winds blog and more recently announced with Istio 1.5, WebAssembly (Wasm) is now an (alpha) option for extending the functionality of the Istio service proxy (Envoy proxy). With Wasm, users can build support for new protocols, custom metrics, loggers, and other filters. Working closely with Google, we in the community (Solo.io) have focused on the user experience of building, socializing, and deploying Wasm extensions to Istio. We’ve announced WebAssembly Hub and associated tooling to build a “docker-like” experience for working with Wasm.

Background

With the WebAssembly Hub tooling, we can use the wasme CLI to easily bootstrap a Wasm project for Envoy, push it to a repository, and then pull/deploy it to Istio. For example, to deploy a Wasm extension to Istio with wasme we can run the following:

$ wasme deploy istio webassemblyhub.io/ceposta/demo-add-header:v0.2 \
--id=myfilter \
--namespace=bookinfo \
--config 'tomorrow'

This will add the demo-add-header extension to all workloads running in the bookinfo namespace. We can get more fine-grained control over which workloads get the extension by using the --labels parameter:

$ wasme deploy istio webassemblyhub.io/ceposta/demo-add-header:v0.2 \
--id=myfilter \
--namespace=bookinfo \
--config 'tomorrow' \
--labels app=details

This is a much easier experience than manually creating EnvoyFilter resources and trying to get the Wasm module to each of the pods that are part of the workload you’re trying to target. However, this is a very imperative approach to interacting with Istio. Just like users typically don’t use kubectl directly in production and prefer a declarative, resource-based workflow, we want the same for making customizations to our Istio proxies.

A declarative approach

The WebAssembly Hub tooling also includes an operator for deploying Wasm extensions to Istio workloads. The operator allows users to define their WebAssembly extensions using a declarative format and leave it to the operator to rectify the deployment. For example, we use a FilterDeployment resource to define what image and workloads need the extension:

apiVersion: wasme.io/v1
kind: FilterDeployment
metadata:
name: bookinfo-custom-filter
namespace: bookinfo
spec:
deployment:
istio:
kind: Deployment
labels:
app: details
filter:
config: 'world'
image: webassemblyhub.io/ceposta/demo-add-header:v0.2

We could then take this FilterDeployment document and version it with the rest of our Istio resources. You may be wondering why we need this Custom Resource to configure Istio’s service proxy to use a Wasm extension when Istio already has the EnvoyFilter resource.

Let’s take a look at exactly how all of this works under the covers.

How it works

Under the covers the operator is doing a few things that aid in deploying and configuring a Wasm extension into the Istio service proxy (Envoy Proxy).

Set up local cache of Wasm extensions
Pull desired Wasm extension into the local cache
Mount the wasm-cache into appropriate workloads
Configure Envoy with EnvoyFilter CRD to use the Wasm filter

Understanding how wasme operator works

At the moment, the Wasm image needs to be published into a registry for the operator to correctly cache it. The cache pods run as DaemonSet on each node so that the cache can be mounted into the Envoy container. This is being improved, as it’s not the ideal mechanism. Ideally we wouldn’t have to deal with mounting anything and could stream the module to the proxy directly over HTTP, so stay tuned for updates (should land within next few days). The mount is established by using the sidecar.istio.io/userVolume and sidecar.istio.io/userVolumeMount annotations. See the docs on Istio Resource Annotations for more about how that works.

Once the Wasm module is cached correctly and mounted into the workload’s service proxy, the operator then configures the EnvoyFilter resources.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: details-v1-myfilter
namespace: bookinfo
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: envoy.http_connection_manager
subFilter:
name: envoy.router
patch:
operation: INSERT_BEFORE
value:
config:
config:
configuration: tomorrow
name: myfilter
rootId: add_header
vmConfig:
code:
local:
filename: /var/local/lib/wasme-cache/44bf95b368e78fafb663020b43cf099b23fc6032814653f2f47e4d20643e7267
runtime: envoy.wasm.runtime.v8
vmId: myfilter
name: envoy.filters.http.wasm
workloadSelector:
labels:
app: details
version: v1

You can see the EnvoyFilter resource configures the proxy to add the envoy.filter.http.wasm filter and load the Wasm module from the wasme-cache.

Once the Wasm extension is loaded into the Istio service proxy, it will extend the capabilities of the proxy with whatever custom code you introduced.

Next Steps

In this blog we explored options for installing Wasm extensions into Istio workloads. The easiest way to get started with WebAssembly on Istio is to use the wasme tool to bootstrap a new Wasm project with C++, AssemblyScript [or Rust coming really soon!]. For example, to set up a C++ Wasm module, you can run:

$ wasme init ./filter --language cpp --platform istio --platform-version 1.5.x

If we didn’t have the extra flags, wasme init would enter an interactive mode walking you through the correct values to choose.

Take a look at the WebAssembly Hub wasme tooling to get started with Wasm on Istio.

Learn more

Redefine Extensibility with WebAssembly on Envoy and Istio
WebAssembly SF talk (video): Extensions for network proxies, by John Plevyak
Solo blog
Proxy-Wasm ABI specification
Proxy-Wasm C++ SDK and its developer documentation
Proxy-Wasm Rust SDK
Proxy-Wasm AssemblyScript SDK
Tutorials
Videos on the Solo.io Youtube Channel

Redefining extensibility in proxies - introducing WebAssembly to Envoy and Istio

Thu, 05 Mar 2020 00:00:00 +0000

Since adopting Envoy in 2016, the Istio project has always wanted to provide a platform on top of which a rich set of extensions could be built, to meet the diverse needs of our users. There are many reasons to add capability to the data plane of a service mesh — to support newer protocols, integrate with proprietary security controls, or enhance observability with custom metrics, to name a few.

Over the last year and a half our team here at Google has been working on adding dynamic extensibility to the Envoy proxy using WebAssembly. We are delighted to share that work with the world today, as well as unveiling WebAssembly (Wasm) for Proxies (Proxy-Wasm): an ABI, which we intend to standardize; SDKs; and its first major implementation, the new, lower-latency Istio telemetry system.

We have also worked closely with the community to ensure that there is a great developer experience for users to get started quickly. The Google team has been working closely with the team at Solo.io who have built the WebAssembly Hub, a service for building, sharing, discovering and deploying Wasm extensions. With the WebAssembly Hub, Wasm extensions are as easy to manage, install and and run as containers.

This work is being released today in Alpha and there is still lots of work to be done, but we are excited to get this into the hands of developers so they can start experimenting with the tremendous possibilities this opens up.

Background

The need for extensibility has been a founding tenet of both the Istio and Envoy projects, but the two projects took different approaches. Istio project focused on enabling a generic out-of-process extension model called Mixer with a lightweight developer experience, while Envoy focused on in-proxy extensions.

Each approach has its share of pros and cons. The Istio model led to significant resource inefficiencies that impacted tail latencies and resource utilization. This model was also intrinsically limited - for example, it was never going to provide support for implementing custom protocol handling.

The Envoy model imposed a monolithic build process, and required extensions to be written in C++, limiting the developer ecosystem. Rolling out a new extension to the fleet required pushing new binaries and rolling restarts, which can be difficult to coordinate, and risk downtime. This also incentivized developers to upstream extensions into Envoy that were used by only a small percentage of deployments, just to piggyback on its release mechanisms.

Over time some of the most performance-sensitive features of Istio have been upstreamed into Envoy - policy checks on traffic, and JWT authentication, for example. Still, we have always wanted to converge on a single stack for extensibility that imposes fewer tradeoffs: something that decouples Envoy releases from its extension ecosystem, enables developers to work in their languages of choice, and enables Istio to reliably roll out new capability without downtime risk. Enter WebAssembly.

What is WebAssembly?

WebAssembly (Wasm) is a portable bytecode format for executing code written in multiple languages at near-native speed. Its initial design goals align well with the challenges outlined above, and it has sizable industry support behind it. Wasm is the fourth standard language (following HTML, CSS and JavaScript) to run natively in all the major browsers, having become a W3C Recommendation in December 2019. That gives us confidence in making a strategic bet on it.

While WebAssembly started life as a client-side technology, there are a number of advantages to using it on the server. The runtime is memory-safe and sandboxed for security. There is a large tooling ecosystem for compiling and debugging Wasm in its textual or binary format. The W3C and BytecodeAlliance have become active hubs for other server-side efforts. For example, the Wasm community is standardizing a “WebAssembly System Interface” (WASI) at the W3C, with a sample implementation, which provides an OS-like abstraction to Wasm ‘programs’.

Bringing WebAssembly to Envoy

Over the past 18 months, we have been working with the Envoy community to build Wasm extensibility into Envoy and contribute it upstream. We’re pleased to announce it is available as Alpha in the Envoy build shipped with Istio 1.5, with source in the envoy-wasm development fork and work ongoing to merge it into the main Envoy tree. The implementation uses the WebAssembly runtime built into Google’s high performance V8 engine.

In addition to the underlying runtime, we have also built:

A generic Application Binary Interface (ABI) for embedding Wasm in proxies, which means compiled extensions will work across different versions of Envoy - or even other proxies, should they choose to implement the ABI
SDKs for easy extension development in C++, Rust and AssemblyScript, with more to follow
Comprehensive samples and instructions on how to deploy in Istio and standalone Envoy
Abstractions to allow for other Wasm runtimes to be used, including a ‘null’ runtime which simply compiles the extension natively into Envoy — very useful for testing and debugging

Using Wasm for extending Envoy brings us several key benefits:

Agility: Extensions can be delivered and reloaded at runtime using the Istio control plane. This enables a fast develop → test → release cycle for extensions without requiring Envoy rollouts.
Stock releases: Once merging into the main tree is complete, Istio and others will be able to use stock releases of Envoy, instead of custom builds. This will also free the Envoy community to move some of the built-in extensions to this model, thereby reducing their supported footprint.
Reliability and isolation: Extensions are deployed inside a sandbox with resource constraints, which means they can now crash, or leak memory, without bringing the whole Envoy process down. CPU and memory usage can also be constrained.
Security: The sandbox has a clearly defined API for communicating with Envoy, so extensions only have access to, and can modify, a limited number of properties of a connection or request. Furthermore, because Envoy mediates this interaction, it can hide or sanitize sensitive information from the extension (e.g. “Authorization” and “Cookie” HTTP headers, or the client’s IP address).
Flexibility: over 30 programming languages can be compiled to WebAssembly, allowing developers from all backgrounds - C++, Go, Rust, Java, TypeScript, etc. - to write Envoy extensions in their language of choice.

“I am extremely excited to see WASM support land in Envoy; this is the future of Envoy extensibility, full stop. Envoy’s WASM support coupled with a community driven hub will unlock an incredible amount of innovation in the networking space across both service mesh and API gateway use cases. I can’t wait to see what the community builds moving forward.” – Matt Klein, Envoy creator.

For technical details of the implementation, look out for an upcoming post to the Envoy blog.

The Proxy-Wasm interface between host environment and extensions is deliberately proxy agnostic. We’ve built it into Envoy, but it was designed to be adopted by other proxy vendors. We want to see a world where you can take an extension written for Istio and Envoy and run it in other infrastructure; you’ll hear more about that soon.

Building on WebAssembly in Istio

Istio moved several of its extensions into its build of Envoy as part of the 1.5 release, in order to significantly improve performance. While doing that work we have been testing to ensure those same extensions can compile and run as Proxy-Wasm modules with no variation in behavior. We’re not quite ready to make this setup the default, given that we consider Wasm support to be Alpha; however, this has given us a lot of confidence in our general approach and in the host environment, ABI and SDKs that have been developed.

We have also been careful to ensure that the Istio control plane and its Envoy configuration APIs are Wasm-ready. We have samples to show how several common customizations such as custom header decoding or programmatic routing can be performed which are common asks from users. As we move this support to Beta, you will see documentation showing best practices for using Wasm with Istio.

Finally, we are working with the many vendors who have written Mixer adapters, to help them with a migration to Wasm — if that is the best path forward. Mixer will move to a community project in a future release, where it will remain available for legacy use cases.

Developer Experience

Powerful tooling is nothing without a great developer experience. Solo.io recently announced the release of WebAssembly Hub, a set of tools and repository for building, deploying, sharing and discovering Envoy Proxy Wasm extensions for Envoy and Istio.

The WebAssembly Hub fully automates many of the steps required for developing and deploying Wasm extensions. Using WebAssembly Hub tooling, users can easily compile their code - in any supported language - into Wasm extensions. The extensions can then be uploaded to the Hub registry, and be deployed and undeployed to Istio with a single command.

Behind the scenes the Hub takes care of much of the nitty-gritty, such as pulling in the correct toolchain, ABI version verification, permission control, and more. The workflow also eliminates toil with configuration changes across Istio service proxies by automating the deployment of your extensions. This tooling helps users and operators avoid unexpected behaviors due to misconfiguration or version mismatches.

The WebAssembly Hub tools provide a powerful CLI as well as an elegant and easy-to-use graphical user interface. An important goal of the WebAssembly Hub is to simplify the experience around building Wasm modules and provide a place of collaboration for developers to share and discover useful extensions.

Check out the getting started guide to create your first Proxy-Wasm extension.

Next Steps

In addition to working towards a beta release, we are committed to making sure that there is a durable community around Proxy-Wasm. The ABI needs to be finalized, and turning it into a standard will be done with broader feedback within the appropriate standards body. Completing upstreaming support into the Envoy mainline is still in progress. We are also seeking an appropriate community home for the tooling and the WebAssembly Hub

Learn more

WebAssembly SF talk (video): Extensions for network proxies, by John Plevyak
Solo blog
Proxy-Wasm ABI specification
Proxy-Wasm C++ SDK and its developer documentation
Proxy-Wasm Rust SDK
Proxy-Wasm AssemblyScript SDK
Tutorials
Videos on the Solo.io Youtube Channel

Istio in 2020 - Following the Trade Winds

Istio Team — Tue, 03 Mar 2020 00:00:00 +0000

Istio solves real problems that people encounter running microservices. Even very early pre-release versions helped users debug the latency in their architecture, increase the reliability of services, and transparently secure traffic behind the firewall.

Last year, the Istio project experienced major growth. After a 9-month gestation before the 1.1 release in Q1, we set a goal of having a quarterly release cadence. We knew it was important to deliver value consistently and predictably. With three releases landing in the successive quarters as planned, we are proud to have reached that goal.

During that time, we improved our build and test infrastructure, resulting in higher quality and easier release cycles. We doubled down on user experience, adding many commands to make operating and debugging the mesh easier. We also saw tremendous growth in the number of developers and companies contributing to the product - culminating in us being #4 on GitHub’s top ten list of fastest growing projects!

We have ambitious goals for Istio in 2020 and there are many major efforts underway, but at the same time we strongly believe that good infrastructure should be “boring.” Using Istio in production should be a seamless experience; performance should not be a concern, upgrades should be a non-event and complex tasks should be automated away. With our investment in a more powerful extensibility story we think the pace of innovation in the service mesh space can increase while Istio focuses on being gloriously dull. More details on our major efforts in 2020 below.

Sleeker, smoother and faster

Istio provided for extensibility from day one, implemented by a component called Mixer. Mixer is a platform that allows custom adapters to act as an intermediary between the data plane and the backends you use for policy or telemetry. Mixer necessarily added overhead to requests because it required extensions to be out-of-process. So, we’re moving to a model that enables extension directly in the proxies instead.

Most of Mixer’s use cases for policy enforcement are already addressed with Istio’s authentication and authorization policies, which allow you to control workload-to-workload and end-user-to-workload authorization directly in the proxy. Common monitoring use cases have already moved into the proxy too - we have introduced in-proxy support for sending telemetry to Prometheus and Stackdriver.

Our benchmarking shows that the new telemetry model reduces our latency dramatically and gives us industry-leading performance, with 50% reductions in both latency and CPU consumption.

A new model for Istio extensibility

The model that replaces Mixer uses extensions in Envoy to provide even more capability. The Istio community is leading the implementation of a WebAssembly (Wasm) runtime in Envoy, which lets us implement extensions that are modular, sandboxed, and developed in one of over 20 languages. Extensions can be dynamically loaded and reloaded while the proxy continues serving traffic. Wasm extensions will also be able to extend the platform in ways that Mixer simply couldn’t. They can act as custom protocol handlers and transform payloads as they pass through Envoy — in short they can do the same things as modules built into Envoy.

We’re working with the Envoy community on ways to discover and distribute these extensions. We want to make WebAssembly extensions as easy to install and run as containers. Many of our partners have written Mixer adapters, and together we are getting them ported to Wasm. We are also developing guides and codelabs on how to write your own extensions for custom integrations.

By changing the extension model, we were also able to remove dozens of CRDs. You no longer need a unique CRD for every piece of software you integrate with Istio.

Installing Istio 1.5 with the ‘preview’ configuration profile won’t install Mixer. If you upgrade from a previous release, or install the ‘default’ profile, we still keep Mixer around, to be safe. When using Prometheus or Stackdriver for metrics, we recommend you try out the new mode and see how much your performance improves.

You can keep Mixer installed and enabled if you need it. Eventually Mixer will become a separately released add-on to Istio that is part of the istio-ecosystem.

Fewer moving parts

We are also simplifying the deployment of the rest of the control plane. To that end, we combined several of the control plane components into a single component: Istiod. This binary includes the features of Pilot, Citadel, Galley, and the sidecar injector. This approach improves many aspects of installing and managing Istio – reducing installation and configuration complexity, maintenance effort, and issue diagnosis time while increasing responsiveness. Read more about Istiod in this post from Christian Posta.

We are shipping Istiod as the default for all profiles in 1.5.

To reduce the per-node footprint, we are getting rid of the node-agent, used to distribute certificates, and moving its functionality to the istio-agent, which already runs in each Pod. For those of you who like pictures we are moving from this …

The Istio architecture today

to this…

The Istio architecture in 2020

In 2020, we will continue to invest in onboarding to achieve our goal of a “zero config” default that doesn’t require you to change any of your application’s configuration to take advantage of most Istio features.

Improved lifecycle management

To improve Istio’s life-cycle management, we moved to an operator-based installation. We introduced the IstioOperator CRD and two installation modes:

Human-triggered: use istioctl to apply the settings to the cluster.
Machine-triggered: use a controller that is continually watching for changes in that CRD and affecting those in real time.

In 2020 upgrades will getting easier too. We will add support for “canarying” new versions of the Istio control plane, which allows you to run a new version alongside the existing version and gradually switch your data plane over to use the new one.

Secure By Default

Istio already provides the fundamentals for strong service security: reliable workload identity, robust access policies and comprehensive audit logging. We’re stabilizing APIs for these features; many Alpha APIs are moving to Beta in 1.5, and we expect them to all be v1 by the end of 2020. To learn more about the status of our APIs, see our features page.

Network traffic is also becoming more secure by default. After many users enabled it in preview, automated rollout of mutual TLS is becoming the recommended practice in Istio 1.5.

In addition we will make Istio require fewer privileges and simplify its dependencies which in turn make it a more robust system. Historically, you had to mount certificates into Envoy using Kubernetes Secrets, which were mounted as files into each proxy. By leveraging the Secret Discovery Service we can distribute these certificates securely without concern of them being intercepted by other workloads on the machine. This mode will become the default in 1.5.

Getting rid of the node-agent not only simplifies the deployment, but also removes the requirement for a cluster-wide PodSecurityPolicy, further improving the security posture of your cluster.

Other features

Here’s a snapshot of some more exciting things you can expect from Istio in 2020:

Integration with more hosted Kubernetes environments - service meshes powered by Istio are currently available from 15 vendors, including Google, IBM, Red Hat, VMware, Alibaba and Huawei
More investment in istioctl and its ability to help diagnose problems
Better integration of VM-based workloads into meshes
Continued work towards making multi-cluster and multi-network meshes easier to configure, maintain, and run
Integration with more service discovery systems, including Functions-as-a-Service
Implementation of the new Kubernetes service APIs, which are currently in development
An enhancement repository, to track feature development
Making it easier to run Istio without needing Kubernetes!

From the seas to the skies, we’re excited to see where you take Istio next.

Remove cross-pod unix domain sockets

Lei Tang (Google) — Thu, 20 Feb 2020 00:00:00 +0000

In Istio versions before 1.5, during secret discovery service (SDS) execution, the SDS client and the SDS server communicate through a cross-pod Unix domain socket (UDS), which needs to be protected by Kubernetes pod security policies.

With Istio 1.5, Pilot Agent, Envoy, and Citadel Agent will be running in the same container (the architecture is shown in the following diagram). To defend against attackers eavesdropping on the cross-pod UDS between Envoy (SDS client) and Citadel Agent (SDS server), Istio 1.5 merges Pilot Agent and Citadel Agent into a single Istio Agent and makes the UDS between Envoy and Citadel Agent private to the Istio Agent container. The Istio Agent container is deployed as the sidecar of the application service container.

The architecture of Istio Agent

Multicluster Istio configuration and service discovery using Admiral

Anil Attuluri (Intuit), Jason Webb (Intuit) — Sun, 05 Jan 2020 00:00:00 +0000

At Intuit, we read the blog post Multi-Mesh Deployments for Isolation and Boundary Protection and immediately related to some of the problems mentioned. We realized that even though we wanted to configure a single multi-cluster mesh, instead of a federation of multiple meshes as described in the blog post, the same non-uniform naming issues also applied in our environment. This blog post explains how we solved these problems using Admiral, an open source project under istio-ecosystem in GitHub.

Background

Using Istio, we realized the configuration for multi-cluster was complex and challenging to maintain over time. As a result, we chose the model described in Multi-Cluster Istio Service Mesh with replicated control planes for scalability and other operational reasons. Following this model, we had to solve these key requirements before widely adopting an Istio service mesh:

Creation of service DNS entries decoupled from the namespace, as described in Features of multi-mesh deployments.
Service discovery across many clusters.
Supporting active-active & HA/DR deployments. We also had to support these crucial resiliency patterns with services being deployed in globally unique namespaces across discrete clusters.

We have over 160 Kubernetes clusters with a globally unique namespace name across all clusters. In this configuration, we can have the same service workload deployed in different regions running in namespaces with different names. As a result, following the routing strategy mentioned in Multicluster version routing, the example name foo.namespace.global wouldn’t work across clusters. We needed a globally unique and discoverable service DNS that resolves service instances in multiple clusters, each instance running/addressable with its own unique Kubernetes FQDN. For example, foo.global should resolve to both foo.uswest2.svc.cluster.local & foo.useast2.svc.cluster.local if foo is running in two Kubernetes clusters with different names. Also, our services need additional DNS names with different resolution and global routing properties. For example, foo.global should resolve locally first, then route to a remote instance using topology routing, while foo-west.global and foo-east.global (names used for testing) should always resolve to the respective regions.

Contextual Configuration

After further investigation, it was apparent that configuration needed to be contextual: each cluster needs a configuration specifically tailored for its view of the world.

For example, we have a payments service consumed by orders and reports. The payments service has a HA/DR deployment across us-east (cluster 3) and us-west (cluster 2). The payments service is deployed in namespaces with different names in each region. The orders service is deployed in a different cluster as payments in us-west (cluster 1). The reports service is deployed in the same cluster as payments in us-west (cluster 2).

Cross cluster workload communication with Istio

Istio ServiceEntry yaml for payments service in Cluster 1 and Cluster 2 below illustrates the contextual configuration that other services need to use the payments service:

Cluster 1 Service Entry

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: payments.global-se
spec:
addresses:
- 240.0.0.10
endpoints:
- address: ef394f...us-east-2.elb.amazonaws.com
locality: us-east-2
ports:
http: 15443
- address: ad38bc...us-west-2.elb.amazonaws.com
locality: us-west-2
ports:
http: 15443
hosts:
- payments.global
location: MESH_INTERNAL
ports:
- name: http
number: 80
protocol: http
resolution: DNS

Cluster 2 Service Entry

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: payments.global-se
spec:
addresses:
- 240.0.0.10
endpoints:
- address: ef39xf...us-east-2.elb.amazonaws.com
locality: us-east-2
ports:
http: 15443
- address: payments.default.svc.cluster.local
locality: us-west-2
ports:
http: 80
hosts:
- payments.global
location: MESH_INTERNAL
ports:
- name: http
number: 80
protocol: http
resolution: DNS

The payments ServiceEntry (Istio CRD) from the point of view of the reports service in Cluster 2, would set the locality us-west pointing to the local Kubernetes FQDN and locality us-east pointing to the istio-ingressgateway (load balancer) for Cluster 3. The payments ServiceEntry from the point of view of the orders service in Cluster 1, will set the locality us-west pointing to Cluster 2 istio-ingressgateway and locality us-east pointing to the istio-ingressgateway for Cluster 3.

But wait, there’s even more complexity: What if the payment services want to move traffic to the us-east region for a planned maintenance in us-west? This would require the payments service to change the Istio configuration in all of their clients’ clusters. This would be nearly impossible to do without automation.

Admiral to the Rescue: Admiral is that Automation

Admiral is a controller of Istio control planes.

Cross cluster workload communication with Istio and Admiral

Admiral provides automatic configuration for an Istio mesh spanning multiple clusters to work as a single mesh based on a unique service identifier that associates workloads running on multiple clusters to a service. It also provides automatic provisioning and syncing of Istio configuration across clusters. This removes the burden on developers and mesh operators, which helps scale beyond a few clusters.

Admiral CRDs

Global Traffic Routing

With Admiral’s global traffic policy CRD, the payments service can update regional traffic weights and Admiral updates the Istio configuration in all clusters that consume the payments service.

apiVersion: admiral.io/v1alpha1
kind: GlobalTrafficPolicy
metadata:
name: payments-gtp
spec:
selector:
identity: payments
policy:
- dns: default.payments.global
lbType: 1
target:
- region: us-west-2/*
weight: 10
- region: us-east-2/*
weight: 90

In the example above, 90% of the payments service traffic is routed to the us-east region. This Global Traffic Configuration is automatically converted into Istio configuration and contextually mapped into Kubernetes clusters to enable multi-cluster global routing for the payments service for its clients within the Mesh.

This Global Traffic Routing feature relies on Istio’s locality load-balancing per service available in Istio 1.5 or later.

Dependency

The Admiral Dependency CRD allows us to specify a service’s dependencies based on a service identifier. This optimizes the delivery of Admiral generated configuration only to the required clusters where the dependent clients of a service are running (instead of writing it to all clusters). Admiral also configures and/or updates the Sidecar Istio CRD in the client’s workload namespace to limit the Istio configuration to only its dependencies. We use service-to-service authorization information recorded elsewhere to generate this dependency records for Admiral to use.

An example dependency for the orders service:

apiVersion: admiral.io/v1alpha1
kind: Dependency
metadata:
name: dependency
namespace: admiral
spec:
source: orders
identityLabel: identity
destinations:
- payments

Dependency is optional and a missing dependency for a service will result in an Istio configuration for that service pushed to all clusters.

Summary

Admiral provides a new Global Traffic Routing and unique service naming functionality to address some challenges posed by the Istio model described in multi-cluster deployment with replicated control planes. It removes the need for manual configuration synchronization between clusters and generates contextual configuration for each cluster. This makes it possible to operate a Service Mesh composed of many Kubernetes clusters.

We think Istio/Service Mesh community would benefit from this approach, so we open sourced Admiral and would love your feedback and support!

Secure Webhook Management

Lei Tang (Google) — Thu, 14 Nov 2019 00:00:00 +0000

Istio has two webhooks: Galley and the sidecar injector. Galley validates Kubernetes resources and the sidecar injector injects sidecar containers into Istio.

By default, Galley and the sidecar injector manage their own webhook configurations. This can pose a security risk if they are compromised, for example, through buffer overflow attacks. Configuring a webhook is a highly privileged operation as a webhook may monitor and mutate all Kubernetes resources.

In the following example, the attacker compromises Galley and modifies the webhook configuration of Galley to eavesdrop on all Kubernetes secrets (the clientConfig is modified by the attacker to direct the secrets resources to a service owned by the attacker).

An example attack

To protect against this kind of attack, Istio 1.4 introduces a new feature to securely manage webhooks using istioctl:

istioctl, instead of Galley and the sidecar injector, manage the webhook configurations. Galley and the sidecar injector are de-privileged so even if they are compromised, they will not be able to alter the webhook configurations.
Before configuring a webhook, istioctl will verify the webhook server is up and that the certificate chain used by the webhook server is valid. This reduces the errors that can occur before a server is ready or if a server has invalid certificates.

To try this new feature, refer to the Istio webhook management task.

Introducing the Istio v1beta1 Authorization Policy

Yangmin Zhu (Google) — Thu, 14 Nov 2019 00:00:00 +0000

Istio 1.4 introduces the v1beta1 authorization policy, which is a major update to the previous v1alpha1 role-based access control (RBAC) policy. The new policy provides these improvements:

Aligns with Istio configuration model.
Improves the user experience by simplifying the API.
Supports more use cases (e.g. Ingress/Egress gateway support) without added complexity.

The v1beta1 policy is not backward compatible and requires a one time conversion. A tool is provided to automate this process. The previous configuration resources ClusterRbacConfig, ServiceRole, and ServiceRoleBinding will not be supported from Istio 1.6 onwards.

This post describes the new v1beta1 authorization policy model, its design goals and the migration from v1alpha1 RBAC policies. See the authorization concept page for a detailed in-depth explanation of the v1beta1 authorization policy.

We welcome your feedback about the v1beta1 authorization policy at discuss.istio.io.

Background

To date, Istio provided RBAC policies to enforce access control on services using three configuration resources: ClusterRbacConfig, ServiceRole and ServiceRoleBinding. With this API, users have been able to enforce control access at mesh-level, namespace-level and service-level. Like other RBAC policies, Istio RBAC uses the same concept of role and binding for granting permissions to identities.

Although Istio RBAC has been working reliably, we’ve found that many improvements were possible.

For example, users have mistakenly assumed that access control enforcement happens at service-level because ServiceRole uses service to specify where to apply the policy, however, the policy is actually applied on workloads, the service is only used to find the corresponding workload. This nuance is significant when multiple services are referring to the same workload. A ServiceRole for service A will also affect service B if the two services are referring to the same workload, which can cause confusion and incorrect configuration.

An other example is that it’s proven difficult for users to maintain and manage the Istio RBAC configurations because of the need to deeply understand three related resources.

Design goals

The new v1beta1 authorization policy had several design goals:

Align with Istio Configuration Model for better clarity on the policy target. The configuration model provides a unified configuration hierarchy, resolution and target selection.
Improve the user experience by simplifying the API. It’s easier to manage one custom resource definition (CRD) that includes all access control specifications, instead of multiple CRDs.
Support more use cases without added complexity. For example, allow the policy to be applied on Ingress/Egress gateway to enforce access control for traffic entering/exiting the mesh.

`AuthorizationPolicy`

An AuthorizationPolicy custom resource enables access control on workloads. This section gives an overview of the changes in the v1beta1 authorization policy.

An AuthorizationPolicy includes a selector and a list of rule. The selector specifies on which workload to apply the policy and the list of rule specifies the detailed access control rule for the workload.

The rule is additive, which means a request is allowed if any rule allows the request. Each rule includes a list of from, to and when, which specifies who is allowed to do what under which conditions.

The selector replaces the functionality provided by ClusterRbacConfig and the services field in ServiceRole. The rule replaces the other fields in the ServiceRole and ServiceRoleBinding.

Example

The following authorization policy applies to workloads with app: httpbin and version: v1 label in the foo namespace:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
version: v1
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/sleep"]
to:
- operation:
methods: ["GET"]
when:
- key: request.headers[version]
values: ["v1", "v2"]

The policy allows principal cluster.local/ns/default/sa/sleep to access the workload using the GET method when the request includes a version header of value v1 or v2. Any requests not matched with the policy will be denied by default.

Assuming the httpbin service is defined as:

apiVersion: v1
kind: Service
metadata:
name: httpbin
namespace: foo
spec:
selector:
app: httpbin
version: v1
ports:
# omitted

You would need to configure three resources to achieve the same result in v1alpha1:

apiVersion: "rbac.istio.io/v1alpha1"
kind: ClusterRbacConfig
metadata:
name: default
spec:
mode: 'ON_WITH_INCLUSION'
inclusion:
services: ["httpbin.foo.svc.cluster.local"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
name: httpbin
namespace: foo
spec:
rules:
- services: ["httpbin.foo.svc.cluster.local"]
methods: ["GET"]
constraints:
- key: request.headers[version]
values: ["v1", "v2"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: httpbin
namespace: foo
spec:
subjects:
- user: "cluster.local/ns/default/sa/sleep"
roleRef:
kind: ServiceRole
name: "httpbin"

Workload selector

A major change in the v1beta1 authorization policy is that it now uses workload selector to specify where to apply the policy. This is the same workload selector used in the Gateway, Sidecar and EnvoyFilter configurations.

The workload selector makes it clear that the policy is applied and enforced on workloads instead of services. If a policy applies to a workload that is used by multiple different services, the same policy will affect the traffic to all the different services.

You can simply leave the selector empty to apply the policy to all workloads in a namespace. The following policy applies to all workloads in the namespace bar:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: policy
namespace: bar
spec:
rules:
# omitted

Root namespace

A policy in the root namespace applies to all workloads in the mesh in every namespaces. The root namespace is configurable in the MeshConfig and has the default value of istio-system.

For example, you installed Istio in istio-system namespace and deployed workloads in default and bookinfo namespace. The root namespace is changed to istio-config from the default value. The following policy will apply to workloads in every namespace including default, bookinfo and the istio-system:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: policy
namespace: istio-config
spec:
rules:
# omitted

Ingress/Egress Gateway support

The v1beta1 authorization policy can also be applied on ingress/egress gateway to enforce access control on traffic entering/leaving the mesh, you only need to change the selector to make select the ingress/egress workload.

The following policy applies to workloads with the app: istio-ingressgateway label:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: ingress
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
rules:
# omitted

Remember the authorization policy only applies to workloads in the same namespace as the policy, unless the policy is applied in the root namespace:

If you don’t change the default root namespace value (i.e. istio-system), the above policy will apply to workloads with the app: istio-ingressgateway label in every namespace.
If you have changed the root namespace to a different value, the above policy will only apply to workloads with the app: istio-ingressgateway label only in the istio-system namespace.

Comparison

The following table highlights the key differences between the old v1alpha1 RBAC policies and the new v1beta1 authorization policy.

Feature

Feature	`v1alpha1` RBAC policy	`v1beta1` Authorization Policy
API stability	`alpha`: No backward compatible	`beta`: backward compatible guaranteed
Number of CRDs	Three: `ClusterRbacConfig`, `ServiceRole` and `ServiceRoleBinding`	Only One: `AuthorizationPolicy`
Policy target	service	workload
Deny-by-default behavior	Enabled explicitly by configuring `ClusterRbacConfig`	Enabled implicitly with `AuthorizationPolicy`
Ingress/Egress gateway support	Not supported	Supported
The `"*"` value in policy	Match all contents (empty and non-empty)	Match non-empty contents only

The following tables show the relationship between the v1alpha1 and v1beta1 API.

`ClusterRbacConfig`

`ClusterRbacConfig.Mode`	`AuthorizationPolicy`
`OFF`	No policy applied
`ON`	A deny-all policy applied in root namespace
`ON_WITH_INCLUSION`	policies should be applied to namespaces or workloads included by `ClusterRbacConfig`
`ON_WITH_EXCLUSION`	policies should be applied to namespaces or workloads excluded by `ClusterRbacConfig`

`ServiceRole`

`ServiceRole`	`AuthorizationPolicy`
`services`	`selector`
`paths`	`paths` in `to`
`methods`	`methods` in `to`
`destination.ip` in constraint	Not supported
`destination.port` in constraint	`ports` in `to`
`destination.labels` in constraint	`selector`
`destination.namespace` in constraint	Replaced by the namespace of the policy, i.e. the `namespace` in metadata
`destination.user` in constraint	Not supported
`experimental.envoy.filters` in constraint	`experimental.envoy.filters` in `when`
`request.headers` in constraint	`request.headers` in `when`

`ServiceRoleBinding`

`ServiceRoleBinding`	`AuthorizationPolicy`
`user`	`principals` in `from`
`group`	`request.auth.claims[group]` in `when`
`source.ip` in property	`ipBlocks` in `from`
`source.namespace` in property	`namespaces` in `from`
`source.principal` in property	`principals` in `from`
`request.headers` in property	`request.headers` in `when`
`request.auth.principal` in property	`requestPrincipals` in `from` or `request.auth.principal` in `when`
`request.auth.audiences` in property	`request.auth.audiences` in `when`
`request.auth.presenter` in property	`request.auth.presenter` in `when`
`request.auth.claims` in property	`request.auth.claims` in `when`

Beyond all the differences, the v1beta1 policy is enforced by the same engine in Envoy and supports the same authenticated identity (mutual TLS or JWT), condition and other primitives (e.g. IP, port and etc.) as the v1alpha1 policy.

Future of the `v1alpha1` policy

The v1alpha1 RBAC policy (ClusterRbacConfig, ServiceRole, and ServiceRoleBinding) is deprecated by the v1beta1 authorization policy.

Istio 1.4 continues to support the v1alpha1 RBAC policy to give you enough time to move away from the alpha policies.

Migration from the `v1alpha1` policy

Istio only supports one of the two versions for a given workload:

If there is only v1beta1 policy for a workload, the v1beta1 policy will be used.
If there is only v1alpha1 policy for a workload, the v1alpha1 policy will be used.
If there are both v1beta1 and v1alpha1 policies for a workload, only the v1beta1 policy will be used and the the v1alpha1 policy will be ignored.

General Guideline

The typical flow of migrating to v1beta1 policy is to start by checking the ClusterRbacConfig to decide which namespace or service is enabled with RBAC.

For each service enabled with RBAC:

Get the workload selector from the service definition.
Create a v1beta1 policy with the workload selector.
Update the v1beta1 policy for each ServiceRole and ServiceRoleBinding applied to the service.
Apply the v1beta1 policy and monitor the traffic to make sure the policy is working as expected.
Repeat the process for the next service enabled with RBAC.

For each namespace enabled with RBAC:

Apply a v1beta1 policy that denies all traffic to the given namespace.

Migration Example

Assume you have the following v1alpha1 policies for the httpbin service in the foo namespace:

apiVersion: "rbac.istio.io/v1alpha1"
kind: ClusterRbacConfig
metadata:
name: default
spec:
mode: 'ON_WITH_INCLUSION'
inclusion:
namespaces: ["foo"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
name: httpbin
namespace: foo
spec:
rules:
- services: ["httpbin.foo.svc.cluster.local"]
methods: ["GET"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: httpbin
namespace: foo
spec:
subjects:
- user: "cluster.local/ns/default/sa/sleep"
roleRef:
kind: ServiceRole
name: "httpbin"

Migrate the above policies to v1beta1 in the following ways:

Assume the httpbin service has the following workload selector:
```
selector:
app: httpbin
version: v1
```

Create a v1beta1 policy with the workload selector:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
version: v1

Update the v1beta1 policy with each ServiceRole and ServiceRoleBinding applied to the service:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin
namespace: foo
spec:
selector:
matchLabels:
app: httpbin
version: v1
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/sleep"]
to:
- operation:
methods: ["GET"]

Apply the v1beta1 policy and monitor the traffic to make sure it works as expected.
Apply the following v1beta1 policy that denies all traffic to the foo namespace because the foo namespace is enabled with RBAC:
```
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: deny-all
namespace: foo
spec:
{}
```

Make sure the v1beta1 policy is working as expected and then you can delete the v1alpha1 policies from the cluster.

Automation of the Migration

To help ease the migration, the istioctl experimental authz convert command is provided to automatically convert the v1alpha1 policies to the v1beta1 policy.

You can evaluate the command but it is experimental in Istio 1.4 and doesn’t support the full v1alpha1 semantics as of the date of this blog post.

The command to support the full v1alpha1 semantics is expected in a patch release following Istio 1.4.

Introducing the Istio Operator

Martin Ostrowski (Google), Frank Budinsky (IBM) — Thu, 14 Nov 2019 00:00:00 +0000

Kubernetes operators provide a pattern for encoding human operational knowledge in software and are a popular way to simplify the administration of software infrastructure components. Istio is a natural candidate for an automated operator as it is challenging to administer.

Up until now, Helm has been the primary tool to install and upgrade Istio. Istio 1.4 introduces a new method of installation using istioctl. This new installation method builds on the strengths of Helm with the addition of the following:

Users only need to install one tool: istioctl
All API fields are validated
Small customizations not in the API don’t require chart or API changes
Version specific upgrade hooks can be easily and robustly implemented

The Helm installation method is in the process of deprecation. Upgrading from Istio 1.4 with a version not initially installed with Helm will also be replaced by a new istioctl upgrade feature.

The new istioctl installation commands use a custom resource to configure the installation. The custom resource is part of a new Istio operator implementation intended to simplify the common administrative tasks of installation, upgrade, and complex configuration changes for Istio. Validation and checking for installation and upgrade is tightly integrated with the tools to prevent common errors and simplify troubleshooting.

The Operator API

Every operator implementation requires a custom resource definition (CRD) to define its custom resource, that is, its API. Istio’s operator API is defined by the IstioControlPlane CRD, which is generated from an IstioControlPlane proto. The API supports all of Istio’s current configuration profiles using a single field to select the profile. For example, the following IstioControlPlane resource configures Istio using the demo profile:

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
metadata:
namespace: istio-operator
name: example-istiocontrolplane
spec:
profile: demo

You can then customize the configuration with additional settings. For example, to disable telemetry:

apiVersion: install.istio.io/v1alpha2
kind: IstioControlPlane
metadata:
namespace: istio-operator
name: example-istiocontrolplane
spec:
profile: demo
telemetry:
enabled: false

Installing with istioctl

The recommended way to use the Istio operator API is through a new set of istioctl commands. For example, to install Istio into a cluster:

$ istioctl manifest apply -f <your-istiocontrolplane-customresource>

Make changes to the installation configuration by editing the configuration file and executing istioctl manifest apply again.

To upgrade to a new version of Istio:

$ istioctl x upgrade -f <your-istiocontrolplane-config-changes>

In addition to specifying the complete configuration in an IstioControlPlane resource, the istioctl commands can also be passed individual settings using a --set flag:

$ istioctl manifest apply --set telemetry.enabled=false

There are also a number of other istioctl commands that, for example, help you list, display, and compare configuration profiles and manifests.

Refer to the Istio install instructions for more details.

Istio Controller (alpha)

Operator implementations use a Kubernetes controller to continuously monitor their custom resource and apply the corresponding configuration changes. The Istio controller monitors an IstioControlPlane resource and reacts to changes by updating the Istio installation configuration in the corresponding cluster.

In the 1.4 release, the Istio controller is in the alpha phase of development and not fully integrated with istioctl. It is, however, available for experimentation using kubectl commands. For example, to install the controller and a default version of Istio into your cluster, run the following command:

$ kubectl apply -f https://<repo URL>/operator.yaml
$ kubectl apply -f https://<repo URL>/default-cr.yaml

You can then make changes to the Istio installation configuration:

$ kubectl edit istiocontrolplane example-istiocontrolplane -n istio-system

As soon as the resource is updated, the controller will detect the changes and respond by updating the Istio installation correspondingly.

Both the operator controller and istioctl commands share the same implementation. The significant difference is the execution context. In the istioctl case, the operation runs in the admin user’s command execution and security context. In the controller case, a pod in the cluster runs the code in its security context. In both cases, configuration is validated against a schema and the same correctness checks are performed.

Migration from Helm

To help ease the transition from previous configurations using Helm, istioctl and the controller support pass-through access for the full Helm installation API.

You can pass Helm configuration options using istioctl --set by prepending the string values. to the option name. For example, instead of this Helm command:

$ helm template ... --set global.mtls.enabled=true

You can use this istioctl command:

$ istioctl manifest generate ... --set values.global.mtls.enabled=true

You can also set Helm configuration values in an IstioControlPlane custom resource. See Customize Istio settings using Helm for details.

Another feature to help with the transition from Helm is the alpha istioctl manifest migrate command. This command can be used to automatically convert a Helm values.yaml file to a corresponding IstioControlPlane configuration.

Implementation

Several frameworks have been created to help implement operators by generating stubs for some or all of the components. The Istio operator was created with the help of a combination of kubebuilder and operator framework. Istio’s installation now uses a proto to describe the API such that runtime validation can be executed against a schema.

More information about the implementation can be found in the README and ARCHITECTURE documents in the Istio operator repository.

Summary

Starting in Istio 1.4, Helm installation is being replaced by new istioctl commands using a new operator custom resource definition, IstioControlPlane, for the configuration API. An alpha controller is also available for early experimentation with the operator.

The new istioctl commands and operator controller both validate configuration schemas and perform a range of checks for installation change or upgrade. These checks are tightly integrated with the tools to prevent common errors and simplify troubleshooting.

The Istio maintainers expect that this new approach will improve the user experience during Istio installation and upgrade, better stabilize the installation API, and help users better manage and monitor their Istio installations.

We welcome your feedback about the new installation approach at discuss.istio.io.

Introducing istioctl analyze

David Ebbo (Google) — Thu, 14 Nov 2019 00:00:00 +0000

Istio 1.4 introduces an experimental new tool to help you analyze and debug your clusters running Istio.

istioctl analyze is a diagnostic tool that detects potential issues with your Istio configuration, as well as gives general insights to improve your configuration. It can run against a live cluster or a set of local configuration files. It can also run against a combination of the two, allowing you to catch problems before you apply changes to a cluster.

To get started with it in just minutes, head over to the documentation.

Designed to be approachable for novice users

One of the key design goals that we followed for this feature is to make it extremely approachable. This is achieved by making the command useful without having to pass any required complex parameters.

In practice, here are some of the scenarios that it goes after:

“There is some problem with my cluster, but I have no idea where to start”
“Things are generally working, but I’m wondering if there is anything I could improve”

In that sense, it is very different from some of the more advanced diagnostic tools, which go after scenarios along the lines of (taking istioctl proxy-config as an example):

“Show me the Envoy configuration for this specific pod so I can see if anything looks wrong”

This can be very useful for advanced debugging, but it requires a lot of expertize before you can figure out that you need to run this specific command, and which pod to run it on.

So really, the one-line pitch for analyze is: just run it! It’s completely safe, it takes no thinking, it might help you, and at worst, you’ll have wasted a minute!

Improving this tool over time

In Istio 1.4, analyze comes with a nice set of analyzers that can detect a number of common issues. But this is just the beginning, and we are planning to keep growing and fine tuning the analyzers with each release.

In fact, we would welcome suggestions from Istio users. Specifically, if you encounter a situation where you think an issue could be detected via configuration analysis, but is not currently flagged by analyze, please do let us know. The best way to do this is to open an issue on GitHub.

DNS Certificate Management

Lei Tang (Google) — Thu, 14 Nov 2019 00:00:00 +0000

By default, Citadel manages the DNS certificates of the Istio control plane. Citadel is a large component that maintains its own private signing key, and acts as a Certificate Authority (CA).

New in Istio 1.4, we introduce a feature to securely provision and manage DNS certificates signed by the Kubernetes CA, which has the following advantages.

Lighter weight DNS certificate management with no dependency on Citadel.
Unlike Citadel, this feature doesn’t maintain a private signing key, which enhances security.
Simplified root certificate distribution to TLS clients. Clients no longer need to wait for Citadel to generate and distribute its CA certificate.

The following diagram shows the architecture of provisioning and managing DNS certificates in Istio. Chiron is the component provisioning and managing DNS certificates in Istio.

The architecture of provisioning and managing DNS certificates in Istio

To try this new feature, refer to the DNS certificate management task.

Announcing Istio client-go

Neeraj Poddar (Aspen Mesh) — Thu, 14 Nov 2019 00:00:00 +0000

We are pleased to announce the initial release of the Istio client go repository which enables developers to gain programmatic access to Istio APIs in a Kubernetes environment. The generated Kubernetes informers and client set in this repository makes it easy for developers to create controllers and perform Create, Read, Update and Delete (CRUD) operations for all Istio Custom Resource Definitions (CRDs).

This was a highly requested functionality by many Istio users, as is evident from the feature requests on the clients generated by Aspen Mesh and the Knative project. If you’re currently using one of the above mentioned clients, you can easily switch to using Istio client go like this:

import (
...
- versionedclient "github.com/aspenmesh/istio-client-go/pkg/client/clientset/versioned"
+ versionedclient "istio.io/client-go/pkg/clientset/versioned"
)

As the generated client sets are functionally equivalent, switching the imported client libraries should be sufficient in order to consume the newly generated library.

How to use client-go

The Istio client go repository follows the same branching strategy as the Istio API repository, as the client repository depends on the API definitions. If you want to use a stable client set, you can use the release branches or tagged versions in the client go repository. Using the client set is very similar to using the Kubernetes client go, here’s a quick example of using the client to list all Istio virtual services in the passed namespace:

package main
import (
"log"
"os"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/clientcmd"
versionedclient "istio.io/client-go/pkg/clientset/versioned"
)
func main() {
kubeconfig := os.Getenv("KUBECONFIG")
namespace := os.Getenv("NAMESPACE")
if len(kubeconfig) == 0 || len(namespace) == 0 {
log.Fatalf("Environment variables KUBECONFIG and NAMESPACE need to be set")
}
restConfig, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
log.Fatalf("Failed to create k8s rest client: %s", err)
}
ic, err := versionedclient.NewForConfig(restConfig)
if err != nil {
log.Fatalf("Failed to create istio client: %s", err)
}
// Print all VirtualServices
vsList, err := ic.NetworkingV1alpha3().VirtualServices(namespace).List(metav1.ListOptions{})
if err != nil {
log.Fatalf("Failed to get VirtualService in %s namespace: %s", namespace, err)
}
for i := range vsList.Items {
vs := vsList.Items[i]
log.Printf("Index: %d VirtualService Hosts: %+v\n", i, vs.Spec.GetHosts())
}
}

You can find a more in-depth example here.

Useful tools created for generating Istio client-go

If you’re wondering why it took so long or why was it difficult to generate this client set, this section is for you. In Istio, we use protobuf specifications to write APIs which are then converted to Go definitions using the protobuf tool chain. There are three major challenges which you might face if you’re trying to generate Kubernetes client set from a protobuf-generated API:

Creating Kubernetes Wrapper Types - Kubernetes client generation library only works for Go objects which follow the Kubernetes object specification for e.g. Authentication Policy Kubernetes Wrappers. This means for every API which needs programmatic access, you need to create these wrappers. Additionally, there is a fair amount of boilerplate needed for every CRD group, version and kind that needs client code generation. To automate this process, we created a Kubernetes type generator tool which can automatically create the Kubernetes types based on annotations. The annotations parsed by this tool and the various available options are explained in the README. Note that if you’re using protobuf tools to generate Go types, you would need to add these annotations as comments in the proto files, so that the comments are present in the generated Go files which are then used by this tool.
Generating deep copy methods - In Kubernetes client machinery, if you want to mutate any object returned from the client set, you are required to make a copy of the object to prevent modifying the object in-place in the cache store. The canonical way to do this is to create a deepcopy method on all nested types. We created a tool protoc deep copy generator which is a protoc plugin and can automatically create deepcopy method based on annotations using the Proto library utility Proto Clone. Here’s an example of the generated deepcopy method.
Marshaling and Unmarshaling types to/from JSON - For the types generated from proto definitions, it is often problematic to use the default Go JSON encoder/decoder as there are various fields like protobuf’s oneof which requires special handling. Additionally, any Proto fields with underscores in their name might serialize/deserialize to different field names depending on the encoder/decoder as the Go struct tag are generated differently. It is always recommended to use protobuf primitives for serializing/deserializing to JSON instead of relying on default Go library. We created a tool protoc JSON shim which is a protoc plugin and can automatically create Marshalers/Unmarshalers for all Go type generated from Proto definitions. Here’s an example of the code generated by this tool.

I’m hoping that the newly released client library enables users to create more integrations and controllers for the Istio APIs, and the tools mentioned above can be used by developers to generate Kubernetes client set from Proto APIs.

Istio as a Proxy for External Services

Vadim Eisenberg (IBM) — Tue, 15 Oct 2019 00:00:00 +0000

The Control Ingress Traffic and the Ingress Gateway without TLS Termination tasks describe how to configure an ingress gateway to expose services inside the mesh to external traffic. The services can be HTTP or HTTPS. In the case of HTTPS, the gateway passes the traffic through, without terminating TLS.

This blog post describes how to use the same ingress gateway mechanism of Istio to enable access to external services and not to applications inside the mesh. This way Istio as a whole can serve just as a proxy server, with the added value of observability, traffic management and policy enforcement.

The blog post shows configuring access to an HTTP and an HTTPS external service, namely httpbin.org and edition.cnn.com.

Configure an ingress gateway

Define an ingress gateway with a servers: section configuring the 80 and 443 ports. Ensure mode: is set to PASSTHROUGH for tls: in the port 443, which instructs the gateway to pass the ingress traffic AS IS, without terminating TLS.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: proxy
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- httpbin.org
- port:
number: 443
name: tls
protocol: TLS
tls:
mode: PASSTHROUGH
hosts:
- edition.cnn.com
EOF

Create service entries for the httpbin.org and edition.cnn.com services to make them accessible from the ingress gateway:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: httpbin-ext
spec:
hosts:
- httpbin.org
ports:
- number: 80
name: http
protocol: HTTP
resolution: DNS
location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: cnn
spec:
hosts:
- edition.cnn.com
ports:
- number: 443
name: tls
protocol: TLS
resolution: DNS
location: MESH_EXTERNAL
EOF

Create a service entry and configure a destination rule for the localhost service. You need this service entry in the next step as a destination for traffic to the external services from applications inside the mesh to block traffic from inside the mesh. In this example you use Istio as a proxy between external applications and external services.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: localhost
spec:
hosts:
- localhost.local
location: MESH_EXTERNAL
ports:
- number: 80
name: http
protocol: HTTP
- number: 443
name: tls
protocol: TLS
resolution: STATIC
endpoints:
- address: 127.0.0.1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: localhost
spec:
host: localhost.local
trafficPolicy:
tls:
mode: DISABLE
sni: localhost.local
EOF

Create a virtual service for each external service to configure routing to it. Both virtual services include the proxy gateway in the gateways: section and in the match: section for HTTP and HTTPS traffic accordingly.

Notice the route: section for the mesh gateway, the gateway that represents the applications inside the mesh. The route: for the mesh gateway shows how the traffic is directed to the localhost.local service, effectively blocking the traffic.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
spec:
hosts:
- httpbin.org
gateways:
- proxy
- mesh
http:
- match:
- gateways:
- proxy
port: 80
uri:
prefix: /status
route:
- destination:
host: httpbin.org
port:
number: 80
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: localhost.local
port:
number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: cnn
spec:
hosts:
- edition.cnn.com
gateways:
- proxy
- mesh
tls:
- match:
- gateways:
- proxy
port: 443
sni_hosts:
- edition.cnn.com
route:
- destination:
host: edition.cnn.com
port:
number: 443
- match:
- gateways:
- mesh
port: 443
sni_hosts:
- edition.cnn.com
route:
- destination:
host: localhost.local
port:
number: 443
EOF

Enable Envoy’s access logging.
Follow the instructions in Determining the ingress IP and ports to define the SECURE_INGRESS_PORT and INGRESS_HOST environment variables.
Access the httbin.org service through your ingress IP and port which you stored in the $INGRESS_HOST and $INGRESS_PORT environment variables, respectively, during the previous step. Access the /status/418 path of the httpbin.org service that returns the HTTP status 418 I’m a teapot.
```
$ curl $INGRESS_HOST:$INGRESS_PORT/status/418 -Hhost:httpbin.org
-=[ teapot ]=-
_...._
.' _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
| ;/
\_ _/
`"""`
```
If the Istio ingress gateway is deployed in the istio-system namespace, print the gateway’s log with the following command:
```
$ kubectl logs -l istio=ingressgateway -c istio-proxy -n istio-system | grep 'httpbin.org'
```

Search the log for an entry similar to:

[2019-01-31T14:40:18.645Z] "GET /status/418 HTTP/1.1" 418 - 0 135 187 186 "10.127.220.75" "curl/7.54.0" "28255618-6ca5-9d91-9634-c562694a3625" "httpbin.org" "34.232.181.106:80" outbound|80||httpbin.org - 172.30.230.33:80 10.127.220.75:52077 -

Access the edition.cnn.com service through your ingress gateway:

$ curl -s --resolve edition.cnn.com:$SECURE_INGRESS_PORT:$INGRESS_HOST https://edition.cnn.com:$SECURE_INGRESS_PORT | grep -o "<title>.*</title>"
<title>CNN International - Breaking News, US News, World News and Video</title>

If the Istio ingress gateway is deployed in the istio-system namespace, print the gateway’s log with the following command:
```
$ kubectl logs -l istio=ingressgateway -c istio-proxy -n istio-system | grep 'edition.cnn.com'
```

Search the log for an entry similar to:

[2019-01-31T13:40:11.076Z] "- - -" 0 - 589 17798 1644 - "-" "-" "-" "-" "172.217.31.132:443" outbound|443||edition.cnn.com 172.30.230.33:54508 172.30.230.33:443 10.127.220.75:49467 edition.cnn.com

Cleanup

Remove the gateway, the virtual services and the service entries:

$ kubectl delete gateway proxy
$ kubectl delete virtualservice cnn httpbin
$ kubectl delete serviceentry cnn httpbin-ext localhost
$ kubectl delete destinationrule localhost

Multi-Mesh Deployments for Isolation and Boundary Protection

Vadim Eisenberg (IBM) — Wed, 02 Oct 2019 00:00:00 +0000

Various compliance standards require protection of sensitive data environments. Some of the important standards and the types of sensitive data they protect appear in the following table:

Standard	Sensitive data
PCI DSS	payment card data
FedRAMP	federal information, data and metadata
HIPAA	personal health data
GDPR	personal data

PCI DSS, for example, recommends putting cardholder data environment on a network, separate from the rest of the system. It also requires using a DMZ, and setting firewalls between the public Internet and the DMZ, and between the DMZ and the internal network.

Isolation of sensitive data environments from other information systems can reduce the scope of the compliance checks and improve the security of the sensitive data. Reducing the scope reduces the risks of failing a compliance check and reduces the costs of compliance since there are less components to check and secure, according to compliance requirements.

You can achieve isolation of sensitive data by separating the parts of the application that process that data into a separate service mesh, preferably on a separate network, and then connect the meshes with different compliance requirements together in a multi-mesh deployment. The process of connecting inter-mesh applications is called mesh federation.

Note that using mesh federation to create a multi-mesh deployment is very different than creating a multicluster deployment, which defines a single service mesh composed from services spanning more than one cluster. Unlike multi-mesh, a multicluster deployment is not suitable for applications that require isolation and boundary protection.

In this blog post I describe the requirements for isolation and boundary protection, and outline the principles of multi-mesh deployments. Finally, I touch on the current state of mesh-federation support and automation work under way for Istio.

Isolation and boundary protection

Isolation and boundary protection mechanisms are explained in the NIST Special Publication 800-53, Revision 4, Security and Privacy Controls for Federal Information Systems and Organizations, Appendix F, Security Control Catalog, SC-7 Boundary Protection.

In particular, the Boundary protection, isolation of information system components control enhancement:

Organizations can isolate information system components performing different missions and/or business functions. Such isolation limits unauthorized information flows among system components and also provides the opportunity to deploy greater levels of protection for selected components. Separating system components with boundary protection mechanisms provides the capability for increased protection of individual components and to more effectively control information flows between those components. This type of enhanced protection limits the potential harm from cyber attacks and errors. The degree of separation provided varies depending upon the mechanisms chosen. Boundary protection mechanisms include, for example, routers, gateways, and firewalls separating system components into physically separate networks or subnetworks, cross-domain devices separating subnetworks, virtualization techniques, and encrypting information flows among system components using distinct encryption keys.

Various compliance standards recommend isolating environments that process sensitive data from the rest of the organization. The Payment Card Industry (PCI) Data Security Standard recommends implementing network isolation for cardholder data environment and requires isolating this environment from the DMZ. FedRAMP Authorization Boundary Guidance describes authorization boundary for federal information and data, while NIST Special Publication 800-37, Revision 2, Risk Management Framework for Information Systems and Organizations: A System Life Cycle Approach for Security and Privacy recommends protecting of such a boundary in Appendix G, Authorization Boundary Considerations:

Dividing a system into subsystems (i.e., divide and conquer) facilitates a targeted application of controls to achieve adequate security, protection of individual privacy, and a cost-effective risk management process. Dividing complex systems into subsystems also supports the important security concepts of domain separation and network segmentation, which can be significant when dealing with high value assets. When systems are divided into subsystems, organizations may choose to develop individual subsystem security and privacy plans or address the system and subsystems in the same security and privacy plans. Information security and privacy architectures play a key part in the process of dividing complex systems into subsystems. This includes monitoring and controlling communications at internal boundaries among subsystems and selecting, allocating, and implementing controls that meet or exceed the security and privacy requirements of the constituent subsystems.

Boundary protection, in particular, means:

put an access control mechanism at the boundary (firewall, gateway, etc.)
monitor the incoming/outgoing traffic at the boundary
all the access control mechanisms must be deny-all by default
do not expose private IP addresses from the boundary
do not let components from outside the boundary to impact security inside the boundary

Multi-mesh deployments facilitate division of a system into subsystems with different security and compliance requirements, and facilitate the boundary protection. You put each subsystem into a separate service mesh, preferably on a separate network. You connect the Istio meshes using gateways. The gateways monitor and control cross-mesh traffic at the boundary of each mesh.

Features of multi-mesh deployments

non-uniform naming. The withdraw service in the accounts namespace in one mesh might have different functionality and API than the withdraw services in the accounts namespace in other meshes. Such situation could happen in an organization where there is no uniform policy on naming of namespaces and services, or when the meshes belong to different organizations.
expose-nothing by default. None of the services in a mesh are exposed by default, the mesh owners must explicitly specify which services are exposed.
boundary protection. The access control of the traffic must be enforced at the ingress gateway, which stops forbidden traffic from entering the mesh. This requirement implements Defense-in-depth principle and is part of some compliance standards, such as the Payment Card Industry (PCI) Data Security Standard.
common trust may not exist. The Istio sidecars in one mesh may not trust the Citadel certificates in other meshes, due to some security requirement or due to the fact that the mesh owners did not initially plan to federate the meshes.

While expose-nothing by default and boundary protection are required to facilitate compliance and improve security, non-uniform naming and common trust may not exist are required when connecting meshes of different organizations, or of an organization that cannot enforce uniform naming or cannot or may not establish common trust between the meshes.

An optional feature that you may want to use is service location transparency: consuming services send requests to the exposed services in remote meshes using local service names. The consuming services are oblivious to the fact that some of the destinations are in remote meshes and some are local services. The access is uniform, using the local service names, for example, in Kubernetes, reviews.default.svc.cluster.local. Service location transparency is useful in the cases when you want to be able to change the location of the consumed services, for example when some service is migrated from private cloud to public cloud, without changing the code of your applications.

The current mesh-federation work

While you can perform mesh federation using standard Istio configurations already today, it requires writing a lot of boilerplate YAML files and is error-prone. There is an effort under way to automate the mesh federation process. In the meantime, you can look at these multi-mesh deployment examples to get an idea of what a generated federation might include.

Summary

In this blog post I described the requirements for isolation and boundary protection of sensitive data environments by using Istio multi-mesh deployments. I outlined the principles of Istio multi-mesh deployments and reported the current work on mesh federation in Istio.

I will be happy to hear your opinion about multi-mesh and multicluster at discuss.istio.io.

Monitoring Blocked and Passthrough External Service Traffic

Neeraj Poddar (Aspen Mesh) — Sat, 28 Sep 2019 00:00:00 +0000

Understanding, controlling and securing your external service access is one of the key benefits that you get from a service mesh like Istio. From a security and operations point of view, it is critical to monitor what external service traffic is getting blocked as they might surface possible misconfigurations or a security vulnerability if an application is attempting to communicate with a service that it should not be allowed to. Similarly, if you currently have a policy of allowing any external service access, it is beneficial to monitor the traffic so you can incrementally add explicit Istio configuration to allow access and better secure your cluster. In either case, having visibility into this traffic via telemetry is quite helpful as it enables you to create alerts and dashboards, and better reason about your security posture. This was a highly requested feature by production users of Istio and we are excited that the support for this was added in release 1.3.

To implement this, the Istio default metrics are augmented with explicit labels to capture blocked and passthrough external service traffic. This blog will cover how you can use these augmented metrics to monitor all external service traffic.

The Istio control plane configures the sidecar proxy with predefined clusters called BlackHoleCluster and Passthrough which block or allow all traffic respectively. To understand these clusters, let’s start with what external and internal services mean in the context of Istio service mesh.

External and internal services

Internal services are defined as services which are part of your platform and are considered to be in the mesh. For internal services, Istio control plane provides all the required configuration to the sidecars by default. For example, in Kubernetes clusters, Istio configures the sidecars for all Kubernetes services to preserve the default Kubernetes behavior of all services being able to communicate with other.

External services are services which are not part of your platform i.e. services which are outside of the mesh. For external services, Istio provides two options, first to block all external service access (enabled by setting global.outboundTrafficPolicy.mode to REGISTRY_ONLY) and second to allow all access to external service (enabled by setting global.outboundTrafficPolicy.mode to ALLOW_ANY). The default option for this setting (as of Istio 1.3) is to allow all external service access. This option can be configured via mesh configuration.

This is where the BlackHole and Passthrough clusters are used.

What are BlackHole and Passthrough clusters?

BlackHoleCluster - The BlackHoleCluster is a virtual cluster created in the Envoy configuration when global.outboundTrafficPolicy.mode is set to REGISTRY_ONLY. In this mode, all traffic to external service is blocked unless service entries are explicitly added for each service. To implement this, the default virtual outbound listener at 0.0.0.0:15001 which uses original destination is setup as a TCP Proxy with the BlackHoleCluster as the static cluster. The configuration for the BlackHoleCluster looks like this:

{
"name": "BlackHoleCluster",
"type": "STATIC",
"connectTimeout": "10s"
}

As you can see, this cluster is static with no endpoints so all the traffic will be dropped. Additionally, Istio creates unique listeners for every port/protocol combination of platform services which gets hit instead of the virtual listener if the request is made to an external service on the same port. In that case, the route configuration of every virtual route in Envoy is augmented to add the BlackHoleCluster like this:

{
"name": "block_all",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"directResponse": {
"status": 502
}
}
]
}

The route is setup as direct response with 502 response code which means if no other routes match the Envoy proxy will directly return a 502 HTTP status code.

PassthroughCluster - The PassthroughCluster is a virtual cluster created in the Envoy configuration when global.outboundTrafficPolicy.mode is set to ALLOW_ANY. In this mode, all traffic to any external service external is allowed. To implement this, the default virtual outbound listener at 0.0.0.0:15001 which uses SO_ORIGINAL_DST, is setup as a TCP Proxy with the PassthroughCluster as the static cluster. The configuration for the PassthroughCluster looks like this:

{
"name": "PassthroughCluster",
"type": "ORIGINAL_DST",
"connectTimeout": "10s",
"lbPolicy": "ORIGINAL_DST_LB",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 102400,
"maxRetries": 1024
}
]
}
}

This cluster uses the original destination load balancing policy which configures Envoy to send the traffic to the original destination i.e. passthrough.

Similar to the BlackHoleCluster, for every port/protocol based listener the virtual route configuration is augmented to add the PassthroughCluster as the default route:

{
"name": "allow_any",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "PassthroughCluster"
}
}
]
}

Prior to Istio 1.3, there were no metrics reported or if metrics were reported there were no explicit labels set when traffic hit these clusters, resulting in lack of visibility in traffic flowing through the mesh.

The next section covers how to take advantage of this enhancement as the metrics and labels emitted are conditional on whether the virtual outbound or explicit port/protocol listener is being hit.

Using the augmented metrics

To capture all external service traffic in either of the cases (BlackHole or Passthrough), you will need to monitor istio_requests_total and istio_tcp_connections_closed_total metrics. Depending upon the Envoy listener type i.e. TCP proxy or HTTP proxy that gets invoked, one of these metrics will be incremented.

Additionally, in case of a TCP proxy listener in order to see the IP address of the external service that is blocked or allowed via BlackHole or Passthrough cluster, you will need to add the destination_ip label to the istio_tcp_connections_closed_total metric. In this scenario, the host name of the external service is not captured. This label is not added by default and can be easily added by augmenting the Istio configuration for attribute generation and Prometheus handler. You should be careful about cardinality explosion in time series if you have many services with non-stable IP addresses.

PassthroughCluster metrics

This section explains the metrics and the labels emitted based on the listener type invoked in Envoy.

HTTP proxy listener: This happens when the port of the external service is same as one of the service ports defined in the cluster. In this scenario, when the PassthroughCluster is hit, istio_requests_total will get increased like this:

{
"metric": {
"__name__": "istio_requests_total",
"connection_security_policy": "unknown",
"destination_app": "unknown",
"destination_principal": "unknown",
"destination_service": "httpbin.org",
"destination_service_name": "PassthroughCluster",
"destination_service_namespace": "unknown",
"destination_version": "unknown",
"destination_workload": "unknown",
"destination_workload_namespace": "unknown",
"instance": "100.96.2.183:42422",
"job": "istio-mesh",
"permissive_response_code": "none",
"permissive_response_policyid": "none",
"reporter": "source",
"request_protocol": "http",
"response_code": "200",
"response_flags": "-",
"source_app": "sleep",
"source_principal": "unknown",
"source_version": "unknown",
"source_workload": "sleep",
"source_workload_namespace": "default"
},
"value": [
1567033080.282,
"1"
]
}

Note that the destination_service_name label is set to PassthroughCluster to indicate that this cluster was hit and the destination_service is set to the host of the external service.

TCP proxy virtual listener - If the external service port doesn’t map to any HTTP based service ports within the cluster, this listener is invoked and istio_tcp_connections_closed_total is the metric that will be increased:

{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"__name__": "istio_tcp_connections_closed_total",
"connection_security_policy": "unknown",
"destination_app": "unknown",
"destination_ip": "52.22.188.80",
"destination_principal": "unknown",
"destination_service": "unknown",
"destination_service_name": "PassthroughCluster",
"destination_service_namespace": "unknown",
"destination_version": "unknown",
"destination_workload": "unknown",
"destination_workload_namespace": "unknown",
"instance": "100.96.2.183:42422",
"job": "istio-mesh",
"reporter": "source",
"response_flags": "-",
"source_app": "sleep",
"source_principal": "unknown",
"source_version": "unknown",
"source_workload": "sleep",
"source_workload_namespace": "default"
},
"value": [
1567033761.879,
"1"
]
}
]
}
}

In this case, destination_service_name is set to PassthroughCluster and the destination_ip is set to the IP address of the external service. The destination_ip label can be used to do a reverse DNS lookup and get the host name of the external service. As this cluster is passthrough, other TCP related metrics like istio_tcp_connections_opened_total, istio_tcp_received_bytes_total and istio_tcp_sent_bytes_total are also updated.

BlackHoleCluster metrics

Similar to the PassthroughCluster, this section explains the metrics and the labels emitted based on the listener type invoked in Envoy.

HTTP proxy listener: This happens when the port of the external service is same as one of the service ports defined in the cluster. In this scenario, when the BlackHoleCluster is hit, istio_requests_total will get increased like this:

{
"metric": {
"__name__": "istio_requests_total",
"connection_security_policy": "unknown",
"destination_app": "unknown",
"destination_principal": "unknown",
"destination_service": "httpbin.org",
"destination_service_name": "BlackHoleCluster",
"destination_service_namespace": "unknown",
"destination_version": "unknown",
"destination_workload": "unknown",
"destination_workload_namespace": "unknown",
"instance": "100.96.2.183:42422",
"job": "istio-mesh",
"permissive_response_code": "none",
"permissive_response_policyid": "none",
"reporter": "source",
"request_protocol": "http",
"response_code": "502",
"response_flags": "-",
"source_app": "sleep",
"source_principal": "unknown",
"source_version": "unknown",
"source_workload": "sleep",
"source_workload_namespace": "default"
},
"value": [
1567034251.717,
"1"
]
}

Note the destination_service_name label is set to BlackHoleCluster and the destination_service to the host name of the external service. The response code should always be 502 in this case.

TCP proxy virtual listener - If the external service port doesn’t map to any HTTP based service ports within the cluster, this listener is invoked and istio_tcp_connections_closed_total is the metric that will be increased:

{
"metric": {
"__name__": "istio_tcp_connections_closed_total",
"connection_security_policy": "unknown",
"destination_app": "unknown",
"destination_ip": "52.22.188.80",
"destination_principal": "unknown",
"destination_service": "unknown",
"destination_service_name": "BlackHoleCluster",
"destination_service_namespace": "unknown",
"destination_version": "unknown",
"destination_workload": "unknown",
"destination_workload_namespace": "unknown",
"instance": "100.96.2.183:42422",
"job": "istio-mesh",
"reporter": "source",
"response_flags": "-",
"source_app": "sleep",
"source_principal": "unknown",
"source_version": "unknown",
"source_workload": "sleep",
"source_workload_namespace": "default"
},
"value": [
1567034481.03,
"1"
]
}

Note the destination_ip label represents the IP address of the external service and the destination_service_name is set to BlackHoleCluster to indicate that this traffic was blocked by the mesh. Is is interesting to note that for the BlackHole cluster case, other TCP related metrics like istio_tcp_connections_opened_total are not increased as there’s no connection that is ever established.

Monitoring these metrics can help operators easily understand all the external services consumed by the applications in their cluster.

Mixer Adapter for Knative

Idan Zach (IBM) — Wed, 18 Sep 2019 00:00:00 +0000

This post demonstrates how you can use Mixer to push application logic into Istio. It describes a Mixer adapter which implements the Knative scale-from-zero logic with simple code and similar performance to the original implementation.

Knative serving

Knative Serving builds on Kubernetes to support deploying and serving of serverless applications. A core capability of serverless platforms is scale-to-zero functionality which reduces resource usage and cost of inactive workloads. A new mechanism is required to scale from zero when an idle application receives a new request.

The following diagram represents the current Knative architecture for scale-from-zero.

Knative scale-from-zero

The traffic for an idle application is redirected to Activator component by programming Istio with VirtualServices and DestinationRules. When Activator receives a new request, it:

buffers incoming requests
triggers the Autoscaler
redirects requests to the application after it has been scaled up, including retries and load-balancing (if needed)

Once the application is up and running again, Knative restores the routing from Activator to the running application.

Mixer adapter

Mixer provides a rich intermediation layer between the Istio components and infrastructure backends. It is designed as a stand-alone component, separate from Envoy, and has a simple extensibility model to enable Istio to interoperate with a wide breadth of backends. Mixer is inherently easier to extend than Envoy is.

Mixer is an attribute processing engine that uses operator-supplied configuration to map request attributes from the Istio proxy into calls to the infrastructure backends systems via a pluggable set of adapters. Adapters enable Mixer to expose a single consistent API, independent of the infrastructure backends in use. The exact set of adapters used at runtime is determined through operator configuration and can easily be extended to target new or custom infrastructure backends.

In order to achieve Knative scale-from-zero, we use a Mixer out-of-process adapter to call the Autoscaler. Out-of-process adapters for Mixer allow developers to use any programming language and to build and maintain your extension as a stand-alone program without the need to build the Istio proxy.

The following diagram represents the Knative design using the Mixer adapter.

Knative scale-from-zero

In this design, there is no need to change the routing from/to Activator for an idle application as in the original Knative setup. When the Istio proxy represented by the ingress gateway component receives a new request for an idle application, it informs Mixer, including all the relevant metadata information. Mixer then calls your adapter which triggers the Knative Autoscaler using the original Knative protocol.

Istio’s use of Mixer adapters makes it possible to replace otherwise complex networking-based application logic with a more straightforward implementation, as demonstrated in the Knative adapter.

When the adapter receives a message from Mixer, it sends a StatMessage directly to Autoscaler component using the Knative protocol. The metadata information (namespace and service name) required by Autoscaler are transferred by Istio proxy to Mixer and from there to the adapter.

Summary

I compared the cold-start time of the original Knative reference architecture to the new Istio Mixer adapter reference architecture. The results show similar cold-start times. The implementation using the Mixer adapter has greater simplicity. It is not necessary to handle low-level network-based mechanisms as these are handled by Envoy.

The next step is converting this Mixer adapter into an Envoy-specific filter running inside an ingress gateway. This will allow to further improve the latency overhead (no more calls to Mixer and the adapter) and to remove the dependency on the Istio Mixer.

App Identity and Access Adapter

Anton Aleksandrov (IBM) — Wed, 18 Sep 2019 00:00:00 +0000

If you are running your containerized applications on Kubernetes, you can benefit from using the App Identity and Access Adapter for an abstracted level of security with zero code changes or redeploys.

Whether your computing environment is based on a single cloud provider, a combination of multiple cloud providers, or following a hybrid cloud approach, having a centralized identity management can help you to preserve existing infrastructure and avoid vendor lock-in.

With the App Identity and Access Adapter, you can use any OAuth2/OIDC provider: IBM Cloud App ID, Auth0, Okta, Ping Identity, AWS Cognito, Azure AD B2C and more. Authentication and authorization policies can be applied in a streamlined way in all environments — including frontend and backend applications — all without code changes or redeploys.

Understanding Istio and the adapter

Istio is an open source service mesh that transparently layers onto distributed applications and seamlessly integrates with Kubernetes. To reduce the complexity of deployments Istio provides behavioral insights and operational control over the service mesh as a whole. See the Istio Architecture for more details.

Istio uses Envoy proxy sidecars to mediate inbound and outbound traffic for all pods in the service mesh. Istio extracts telemetry from the Envoy sidecars and sends it to Mixer, the Istio component responsible for collecting telemetry and enforcing policy.

The App Identity and Access adapter extends the Mixer functionality by analyzing the telemetry (attributes) against various access control policies across the service mesh. The access control policies can be linked to a particular Kubernetes services and can be finely tuned to specific service endpoints. For more information about policies and telemetry, see the Istio documentation.

When App Identity and Access Adapter is combined with Istio, it provides a scalable, integrated identity and access solution for multicloud architectures that does not require any custom application code changes.

Installation

App Identity and Access adapter can be installed using Helm directly from the github.com repository

$ helm repo add appidentityandaccessadapter https://raw.githubusercontent.com/ibm-cloud-security/app-identity-and-access-adapter/master/helm/appidentityandaccessadapter
$ helm install --name appidentityandaccessadapter appidentityandaccessadapter/appidentityandaccessadapter

Alternatively, you can clone the repository and install the Helm chart locally

$ git clone git@github.com:ibm-cloud-security/app-identity-and-access-adapter.git
$ helm install ./helm/appidentityandaccessadapter --name appidentityandaccessadapter.

Protecting web applications

Web applications are most commonly protected by the OpenID Connect (OIDC) workflow called authorization_code. When an unauthenticated/unauthorized user is detected, they are automatically redirected to the identity service of your choice and presented with the authentication page. When authentication completes, the browser is redirected back to an implicit /oidc/callback endpoint intercepted by the adapter. At this point, the adapter obtains access and identity tokens from the identity service and then redirects users back to their originally requested URL in the web app.

Authentication state and tokens are maintained by the adapter. Each request processed by the adapter will include the Authorization header bearing both access and identity tokens in the following format Authorization: Bearer <access_token> <id_token>

Developers can read leverage the tokens for application experience adjustments, e.g. displaying user name, adjusting UI based on user role etc.

In order to terminate the authenticated session and wipe tokens, aka user logout, simply redirect browser to the /oidc/logout endpoint under the protected service, e.g. if you’re serving your app from https://example.com/myapp, redirect users to https://example.com/myapp/oidc/logout

Whenever access token expires, a refresh token is used to automatically acquire new access and identity tokens without your user’s needing to re-authenticate. If the configured identity provider returns a refresh token, it is persisted by the adapter and used to retrieve new access and identity tokens when the old ones expire.

Applying web application protection

Protecting web applications requires creating two types of resources - use OidcConfig resources to define various OIDC providers, and Policy resources to define the web app protection policies.

apiVersion: "security.cloud.ibm.com/v1"
kind: OidcConfig
metadata:
name: my-oidc-provider-config
namespace: sample-namespace
spec:
discoveryUrl: <discovery-url-from-oidc-provider>
clientId: <client-id-from-oidc-provider>
clientSecretRef:
name: <kubernetes-secret-name>
key: <kubernetes-secret-key>

apiVersion: "security.cloud.ibm.com/v1"
kind: Policy
metadata:
name: my-sample-web-policy
namespace: sample-namespace
spec:
targets:
- serviceName: <kubernetes-service-name-to-protect>
paths:
- prefix: /webapp
method: ALL
policies:
- policyType: oidc
config: my-oidc-provider-config
rules: // optional
- claim: iss
match: ALL
source: access_token
values:
- <expected-issuer-id>
- claim: scope
match: ALL
source: access_token
values:
- openid

Protecting backend application and APIs

Backend applications and APIs are protected using the Bearer Token flow, where an incoming token is validated against a particular policy. The Bearer Token authorization flow expects a request to contain the Authorization header with a valid access token in JWT format. The expected header structure is Authorization: Bearer {access_token}. In case token is successfully validated request will be forwarded to the requested service. In case token validation fails the HTTP 401 will be returned back to the client with a list of scopes that are required to access the API.

Applying backend application and APIs protection

Protecting backend applications and APIs requires creating two types of resources - use JwtConfig resources to define various JWT providers, and Policy resources to define the backend app protection policies.

apiVersion: "security.cloud.ibm.com/v1"
kind: JwtConfig
metadata:
name: my-jwt-config
namespace: sample-namespace
spec:
jwksUrl: <the-jwks-url>

apiVersion: "security.cloud.ibm.com/v1"
kind: Policy
metadata:
name: my-sample-backend-policy
namespace: sample-namespace
spec:
targets:
- serviceName: <kubernetes-service-name-to-protect>
paths:
- prefix: /api/files
method: ALL
policies:
- policyType: jwt
config: my-oidc-provider-config
rules: // optional
- claim: iss
match: ALL
source: access_token
values:
- <expected-issuer-id>
- claim: scope
match: ALL
source: access_token
values:
- files.read
- files.write

Known limitations

At the time of writing this blog there are two known limitations of the App Identity and Access adapter:

If you use the App Identity and Access adapter for Web Applications you should not create more than a single replica of the adapter. Due to the way Envoy Proxy was handling HTTP headers it was impossible to return multiple Set-Cookie headers from Mixer back to Envoy. Therefore we couldn’t set all the cookies required for handling Web Application scenarios. The issue was recently addressed in Envoy and Mixer and we’re planning to address this in future versions of our adapter. Note that this only affects Web Applications, and doesn’t affect Backend Apps and APIs in any way.
As a general best practice you should always consider using mutual-tls for any in-cluster communications. At the moment the communications channel between Mixer and App Identity and Access adapter currently does not use mutual-tls. In future we plan to address this by implementing an approach described in the Mixer Adapter developer guide.

Summary

When a multicloud strategy is in place, security can become complicated as the environment grows and diversifies. While cloud providers supply protocols and tools to ensure their offerings are safe, the development teams are still responsible for the application-level security, such as API access control with OAuth2, defending against man-in-the-middle attacks with traffic encryption, and providing mutual TLS for service access control. However, this becomes complex in a multicloud environment since you might need to define those security details for each service separately. With proper security protocols in place, those external and internal threats can be mitigated.

Development teams have spent time making their services portable to different cloud providers, and in the same regard, the security in place should be flexible and not infrastructure-dependent.

Istio and App Identity and Access Adapter allow you to secure your Kubernetes apps with absolutely zero code changes or redeployments regardless of which programming language and which frameworks you use. Following this approach ensures maximum portability of your apps, and ability to easily enforce same security policies across multiple environments.

You can read more about the App Identity and Access Adapter in the release blog.

Change in Secret Discovery Service in Istio 1.3

Phillip Quy Le (Google) — Tue, 10 Sep 2019 00:00:00 +0000

In Istio 1.3, we are taking advantage of improvements in Kubernetes to issue certificates for workload instances more securely.

When a Citadel Agent sends a certificate signing request to Citadel to get a certificate for a workload instance, it includes the JWT that the Kubernetes API server issued representing the service account of the workload instance. If Citadel can authenticate the JWT, it extracts the service account name needed to issue the certificate for the workload instance.

Before Kubernetes 1.12, the Kubernetes API server issues JWTs with the following issues:

The tokens don’t have important fields to limit their scope of usage, such as aud or exp. See Bound Service Tokens for more info.
The tokens are mounted onto all the pods without a way to opt-out. See Service Account Token Volumes for motivation.

Kubernetes 1.12 introduces trustworthy JWTs to solve these issues. However, support for the aud field to have a different value than the API server audience didn’t become available until Kubernetes 1.13. To better secure the mesh, Istio 1.3 only supports trustworthy JWTs and requires the value of the aud field to be istio-ca when you enable SDS. Before upgrading your Istio deployment to 1.3 with SDS enabled, verify that you use Kubernetes 1.13 or later.

Make the following considerations based on your platform of choice:

GKE: Upgrade your cluster version to at least 1.13.
On-prem Kubernetes and GKE on-prem: Add extra configurations to your Kubernetes. You may also want to refer to the api-server page for the most up-to-date flag names.
For other platforms, check with your provider. If your vendor does not support trustworthy JWTs, you will need to fall back to the file-mount approach to propagate the workload keys and certificates in Istio 1.3.

The Evolution of Istio's APIs

Louis Ryan (Google), Sandeep Parikh (Google) — Mon, 05 Aug 2019 00:00:00 +0000

One of Istio’s main goals has always been, and continues to be, enabling teams to develop abstractions that work best for their specific organization and workloads. Istio provides robust and powerful building blocks for service-to-service networking. Since Istio 0.1, the Istio team has been learning from production users about how they map their own architectures, workloads, and constraints to Istio’s capabilities, and we’ve been evolving Istio’s APIs to make them work better for you.

Evolving Istio’s APIs

The next step in Istio’s evolution is to sharpen our focus and align with the roles of Istio’s users. A security admin should be able to interact with an API that logically groups and simplifies security operations within an Istio mesh; the same goes for service operators and traffic management operations.

Taking it a step further, there’s an opportunity to provide improved experiences for beginning, intermediate, and advanced use cases for each role. There are many common use cases that can be addressed with obvious default settings and a better defined initial experience that requires little to no configuration. For intermediate use cases, the Istio team wants to leverage contextual cues from the environment and provide you with a simpler configuration experience. Finally, for advanced scenarios, our goal is to make easy things easy and hard things possible.

To provide these sorts of role-centric abstractions, however, the APIs underneath them must be able to describe all of Istio’s power and capabilities. Historically, Istio’s approach to API design followed paths similar to those of other infrastructure APIs. Istio follows these design principles:

The Istio APIs should seek to:
- Properly represent the underlying resources to which they are mapped
- Shouldn’t hide any of the underlying resource’s useful capabilities
The Istio APIs should also be composable, so end users can combine infrastructure APIs in a way that makes sense for their own needs.
The Istio APIs should be flexible: Within an organization, it should be possible to have different representations of the underlying resources and surface the ones that make sense for each individual team.

Over the course of the next several releases we will share our progress as we strengthen the alignment between Istio’s APIs and the roles of Istio users.

Composability and abstractions

Istio and Kubernetes often go together, but Istio is much more than an add-on to Kubernetes – it is as much a platform as Kubernetes is. Istio aims to provide infrastructure, and surface the capabilities you need in a powerful service mesh. For example, there are platform-as-a-service offerings that use Kubernetes as their foundation, and build on Kubernetes’ composability to provide a subset of APIs to application developers.

The number of objects that must be configured to deploy applications is a concrete example of Kubernetes’ composability. By our count, at least 10 objects need to be configured: Namespace, Service, Ingress, Deployment, HorizontalPodAutoscaler, Secret, ConfigMap, RBAC, PodDisruptionBudget, and NetworkPolicy.

It sounds complicated, but not everyone needs to interact with those concepts. Some are the responsibility of different teams like the cluster, network, or security admin teams, and many provide sensible defaults. A great benefit of cloud native platforms and deployment tools is that they can hide that complexity by taking in a small amount of information and configuring those objects for you.

Another example of composability in the networking space can be found in the Google Cloud HTTP(S) Load Balancer (GCLB). To correctly use an instance of the GCLB, six different infrastructure objects need to be created and configured. This design is the result of our 20 years of experience in operating distributed systems and there is a reason why each one is separate from the others. But the steps are simplified when you’re creating an instance via the Google Cloud console. We provide the more common end-user/role-specific configurations, and you can configure less common settings later. Ultimately, the goals of infrastructure APIs are to offer the most flexibility without sacrificing functionality.

Knative is a platform for building, running, and operating serverless workloads that provides a great real-world example of role-centric, higher-level APIs. Knative Serving, a component of Knative that builds on Kubernetes and Istio to support deploying and serving serverless applications and functions, provides an opinionated workflow for application developers to manage routes and revisions of their services. Thanks to that opinionated approach, Knative Serving exposes a subset of Istio’s networking APIs that are most relevant to application developers via a simplified Routes object that supports revisions and traffic routing, abstracting Istio’s VirtualService and DestinationRule resources.

As Istio has matured, we’ve also seen production users develop workload- and organization-specific abstractions on top of Istio’s infrastructure APIs.

AutoTrader UK has one of our favorite examples of a custom platform built on Istio. In an interview with the Kubernetes Podcast from Google, Russel Warman and Karl Stoney describe their Kubernetes-based delivery platform, with cost dashboards using Prometheus and Grafana. With minimal effort, they added configuration options to determine what their developers want configured on the network, and it now manages the Istio objects required to make that happen. There are countless other platforms being built in enterprise and cloud-native companies: some designed to replace a web of company-specific custom scripts, and some aimed to be a general-purpose public tool. As more companies start to talk about their tooling publicly, we’ll bring their stories to this blog.

What’s coming next

Some areas of improvement that we’re working on for upcoming releases include:

Installation profiles to setup standard patterns for ingress and egress, with the Istio operator
Automatic inference of container ports and protocols for telemetry
Support for routing all traffic by default to constrain routing incrementally
Add a single global flag to enable mutual TLS and encrypt all inter-pod traffic

Oh, and if for some reason you judge a toolbox by the list of CRDs it installs, in Istio 1.2 we cut the number from 54 down to 23. Why? It turns out that if you have a bunch of features, you need to have a way to configure them all. With the improvements we’ve made to our installer, you can now install Istio using a configuration that works with your adapters.

All service meshes and, by extension, Istio seeks to automate complex infrastructure operations, like networking and security. That means there will always be complexity in its APIs, but Istio will always aim to solve the needs of operators, while continuing to evolve the API to provide robust building blocks and prioritize flexibility through role-centric abstractions.

We can’t wait for you to join our community to see what you build with Istio next!

Secure Control of Egress Traffic in Istio, part 3

Vadim Eisenberg (IBM) — Mon, 22 Jul 2019 00:00:00 +0000

Welcome to part 3 in our series about secure control of egress traffic in Istio. In the first part in the series, I presented the attacks involving egress traffic and the requirements we collected for a secure control system for egress traffic. In the second part in the series, I presented the Istio way of securing egress traffic and showed how you can prevent the attacks using Istio.

In this installment, I compare secure control of egress traffic in Istio with alternative solutions such as using Kubernetes network policies and legacy egress proxies and firewalls. Finally, I describe the performance considerations regarding the secure control of egress traffic in Istio.

Alternative solutions for egress traffic control

First, let’s remember the requirements for egress traffic control we previously collected:

Support of TLS with SNI or of TLS origination.
Monitor SNI and the source workload of every egress access.
Define and enforce policies per cluster.
Define and enforce policies per source, Kubernetes-aware.
Prevent tampering.
Traffic control is transparent to the applications.

Next, I’m going to cover two alternative solutions for egress traffic control: the Kubernetes network policies and egress proxies and firewalls. I show the requirements they satisfy, and, more importantly, the requirements they can’t satisfy.

Kubernetes provides a native solution for traffic control, and in particular, for control of egress traffic, through the network policies. Using these network policies, cluster operators can configure which pods can access specific external services. Cluster operators can identify pods by pod labels, namespace labels, or by IP ranges. To specify the external services, cluster operators can use IP ranges, but cannot use domain names like cnn.com. This is because Kubernetes network policies are not DNS-aware. Network policies satisfy the first requirement since they can control any TCP traffic. Network policies only partially satisfy the third and the fourth requirements because cluster operators can specify policies per cluster or per pod but operators can’t identify external services by domain names. Network policies only satisfy the fifth requirement if the attackers are not able to break from a malicious container into the Kubernetes node and interfere with the implementation of the policies inside said node. Lastly, network policies do satisfy the sixth requirement: Operators don’t need to change the code or the container environment. In summary, we can say that Kubernetes Network Policies provide transparent, Kubernetes-aware egress traffic control, which is not DNS-aware.

The second alternative predates the Kubernetes network policies. Using a DNS-aware egress proxy or firewall lets you configure applications to direct the traffic to the proxy and use some proxy protocol, for example, SOCKS. Since operators must configure the applications, this solution is not transparent. Moreover, operators can’t use pod labels or pod service accounts to configure the proxies because the egress proxies don’t know about them. Therefore, the egress proxies are not Kubernetes-aware and can’t fulfill the fourth requirement because egress proxies cannot enforce policies by source if a Kubernetes artifact specifies the source. In summary, egress proxies can fulfill the first, second, third and fifth requirements, but can’t satisfy the fourth and the six requirements because they are not transparent and not Kubernetes-aware.

Advantages of Istio egress traffic control

Istio egress traffic control is DNS-aware: you can define policies based on URLs or on wildcard domains like *.ibm.com. In this sense, it is better than Kubernetes network policies which are not DNS-aware.

Istio egress traffic control is transparent with regard to TLS traffic, since Istio is transparent: you don’t need to change the applications or configure their containers. For HTTP traffic with TLS origination, you must configure the applications in the mesh to use HTTP instead of HTTPS.

Istio egress traffic control is Kubernetes-aware: the identity of the source of egress traffic is based on Kubernetes service accounts. Istio egress traffic control is better than the legacy DNS-aware proxies or firewalls which are not transparent and not Kubernetes-aware.

Istio egress traffic control is secure: it is based on the strong identity of Istio and, when you apply additional security measures, Istio’s traffic control is resilient to tampering.

Additionally, Istio’s egress traffic control provides the following advantages:

Define access policies in the same language for ingress, egress, and in-cluster traffic. You need to learn a single policy and configuration language for all types of traffic.
Out-of-the-Box integration of Istio’s egress traffic control with Istio’s policy and observability adapters.
Write the adapters to use external monitoring or access control systems with Istio only once and apply them for all types of traffic: ingress, egress, and in-cluster.
Use Istio’s traffic management features for egress traffic: load balancing, passive and active health checking, circuit breaker, timeouts, retries, fault injection, and others.

We refer to a system with the advantages above as Istio-aware.

The following table summarizes the egress traffic control features that Istio and the alternative solutions provide:

	Istio Egress Traffic Control	Kubernetes Network Policies	Legacy Egress Proxy or Firewall
DNS-aware
Kubernetes-aware
Transparent
Istio-aware

Performance considerations

Controlling egress traffic using Istio has a price: increased latency of calls to external services and increased CPU usage by the cluster’s pods. Traffic passes through two proxies:

The application’s sidecar proxy
The egress gateway’s proxy

If you use TLS egress traffic to wildcard domains, you must add an additional proxy between the application and the external service. Since the traffic between the egress gateway’s proxy and the proxy needed for the configuration of arbitrary domains using wildcards is on the pod’s local network, that traffic shouldn’t have a significant impact on latency.

See a performance evaluation of different Istio configurations set to control egress traffic. I would encourage you to carefully measure different configurations with your own applications and your own external services, before you decide whether you can afford the performance overhead for your use cases. You should weigh the required level of security versus your performance requirements and compare the performance overhead of all alternative solutions.

Let me share my thoughts on the performance overhead that controlling egress traffic using Istio adds: Accessing external services already could have high latency and the overhead added because of two or three proxies inside the cluster could likely not be very significant by comparison. After all, applications with a microservice architecture can have chains of dozens of calls between microservices. Therefore, an additional hop with one or two proxies in the egress gateway should not have a large impact.

Moreover, we continue to work towards reducing Istio’s performance overhead. Possible optimizations include:

Extending Envoy to handle wildcard domains: This would eliminate the need for a third proxy between the application and the external services for that use case.
Using mutual TLS for authentication only without encrypting the TLS traffic, since the traffic is already encrypted.

Summary

I hope that after reading this series you are convinced that controlling egress traffic is very important for the security of your cluster. Hopefully, I also managed to convince you that Istio is an effective tool to control egress traffic securely, and that Istio has multiple advantages over the alternative solutions. Istio is the only solution I’m aware of that lets you:

Control egress traffic in a secure and transparent way
Specify external services as domain names
Use Kubernetes artifacts to specify the traffic source

In my opinion, secure control of egress traffic is a great choice if you are looking for your first Istio use case. In this case, Istio already provides you some benefits even before you start using all other Istio features: traffic management, security, policies and observability, applied to traffic between microservices inside the cluster.

So, if you haven’t had the chance to work with Istio yet, install Istio on your cluster and check our egress traffic control tasks and the tasks for the other Istio features. We also want to hear from you, please join us at discuss.istio.io.

Secure Control of Egress Traffic in Istio, part 2

Vadim Eisenberg (IBM) — Wed, 10 Jul 2019 00:00:00 +0000

Welcome to part 2 in our new series about secure control of egress traffic in Istio. In the first part in the series, I presented the attacks involving egress traffic and the requirements we collected for a secure control system for egress traffic. In this installment, I describe the Istio way to securely control the egress traffic, and show how Istio can help you prevent the attacks.

Secure control of egress traffic in Istio

To implement secure control of egress traffic in Istio, you must direct TLS traffic to external services through an egress gateway. Alternatively, you can direct HTTP traffic through an egress gateway and let the egress gateway perform TLS origination.

Both alternatives have their pros and cons, you should choose between them according to your circumstances. The choice mainly depends on whether your application can send unencrypted HTTP requests and whether your organization’s security policies allow sending unencrypted HTTP requests. For example, if your application uses some client library that encrypts the traffic without a possibility to cancel the encryption, you cannot use the option of sending unencrypted HTTP traffic. The same in the case your organization’s security policies do not allow sending unencrypted HTTP requests inside the pod (outside the pod the traffic is encrypted by Istio).

If the application sends HTTP requests and the egress gateway performs TLS origination, you can monitor HTTP information like HTTP methods, headers, and URL paths. You can also define policies based on said HTTP information. If the application performs TLS origination, you can monitor SNI and the service account of the source pod’s TLS traffic, and define policies based on SNI and service accounts.

You must ensure that traffic from your cluster to the outside cannot bypass the egress gateway. Istio cannot enforce it for you, so you must apply some additional security mechanisms, for example, the Kubernetes network policies or an L3 firewall. See an example of the Kubernetes network policies configuration. According to the Defense in depth concept, the more security mechanisms you apply for the same goal, the better.

You must also ensure that Istio control plane and the egress gateway cannot be compromised. While you may have hundreds or thousands of application pods in your cluster, there are only a dozen of Istio control plane pods and the gateways. You can and should focus on protecting the control plane pods and the gateways, since it is easy (there is a small number of pods to protect) and it is most crucial for the security of your cluster. If attackers compromise the control plane or the egress gateway, they could violate any policy.

You might have multiple tools to protect the control plane pods, depending on your environment. The reasonable security measures are:

Run the control plane pods on nodes separate from the application nodes.
Run the control plane pods in their own separate namespace.
Apply the Kubernetes RBAC and network policies to protect the control plane pods.
Monitor the control plane pods more closely than you do the application pods.

Once you direct egress traffic through an egress gateway and apply the additional security mechanisms, you can securely monitor and enforce security policies for the traffic.

The following diagram shows Istio’s security architecture, augmented with an L3 firewall which is part of the additional security mechanisms that should be provided outside of Istio.

Istio Security Architecture with Egress Gateway and L3 Firewall

You can configure the L3 firewall trivially to only allow incoming traffic through the Istio ingress gateway and only allow outgoing traffic through the Istio egress gateway. The Istio proxies of the gateways enforce policies and report telemetry just as all other proxies in the mesh do.

Now let’s examine possible attacks and let me show you how the secure control of egress traffic in Istio prevents them.

Preventing possible attacks

Consider the following security policies for egress traffic:

Application A is allowed to access *.ibm.com, which includes all the external services with URLs matching *.ibm.com.
Application B is allowed to access mongo1.composedb.com.
All egress traffic is monitored.

Suppose the attackers have the following goals:

Access *.ibm.com from your cluster.
Access *.ibm.com from your cluster, unmonitored. The attackers want their traffic to be unmonitored to prevent a possibility that you will detect the forbidden access.
Access mongo1.composedb.com from your cluster.

Now suppose that the attackers manage to break into one of the pods of application A, and try to use the compromised pod to perform the forbidden access. The attackers may try their luck and access the external services in a straightforward way. You will react to the straightforward attempts as follows:

Initially, there is no way to prevent a compromised application A to access *.ibm.com, because the compromised pod is indistinguishable from the original pod.
Fortunately, you can monitor all access to external services, detect suspicious traffic, and thwart attackers from gaining unmonitored access to *.ibm.com. For example, you could apply anomaly detection tools on the egress traffic logs.
To stop attackers from accessing mongo1.composedb.com from your cluster, Istio will correctly detect the source of the traffic, application A in this case, and verify that it is not allowed to access mongo1.composedb.com according to the security policies mentioned above.

Having failed to achieve their goals in a straightforward way, the malicious actors may resort to advanced attacks:

Bypass the container’s sidecar proxy to be able to access any external service directly, without the sidecar’s policy enforcement and reporting. This attack is prevented by a Kubernetes Network Policy or by an L3 firewall that allow egress traffic to exit the mesh only from the egress gateway.
Compromise the egress gateway to be able to force it to send fake information to the monitoring system or to disable enforcement of the security policies. This attack is prevented by applying the special security measures to the egress gateway pods.
Impersonate as application B since application B is allowed to access mongo1.composedb.com. This attack, fortunately, is prevented by Istio’s strong identity support.

As far as we can see, all the forbidden access is prevented, or at least is monitored and can be prevented later. If you see other attacks that involve egress traffic or security holes in the current design, we would be happy to hear about it.

Summary

Hopefully, I managed to convince you that Istio is an effective tool to prevent attacks involving egress traffic. In the next part of this series, I compare secure control of egress traffic in Istio with alternative solutions such as Kubernetes Network Policies and legacy egress proxies/firewalls.

Best Practices: Benchmarking Service Mesh Performance

Megan O'Keefe (Google), John Howard (Google), Mandar Jog (Google) — Tue, 09 Jul 2019 00:00:00 +0000

Service meshes add a lot of functionality to application deployments, including traffic policies, observability, and secure communication. But adding a service mesh to your environment comes at a cost, whether that’s time (added latency) or resources (CPU cycles). To make an informed decision on whether a service mesh is right for your use case, it’s important to evaluate how your application performs when deployed with a service mesh.

Earlier this year, we published a blog post on Istio’s performance improvements in version 1.1. Following the release of Istio 1.2, we want to provide guidance and tools to help you benchmark Istio’s data plane performance in a production-ready Kubernetes environment.

Overall, we found that Istio’s sidecar proxy latency scales with the number of concurrent connections. At 1000 requests per second (RPS), across 16 connections, Istio adds 3 milliseconds per request in the 50th percentile, and 10 milliseconds in the 99th percentile.

In the Istio Tools repository, you’ll find scripts and instructions for measuring Istio’s data plane performance, with additional instructions on how to run the scripts with Linkerd, another service mesh implementation. Follow along as we detail some best practices for each step of the performance test framework.

1. Use a production-ready Istio installation

To accurately measure the performance of a service mesh at scale, it’s important to use an adequately-sized Kubernetes cluster. We test using three worker nodes, each with at least 4 vCPUs and 15 GB of memory.

Then, it’s important to use a production-ready Istio installation profile on that cluster. This lets us achieve performance-oriented settings such as control plane pod autoscaling, and ensures that resource limits are appropriate for heavy traffic load. The default Istio installation is suitable for most benchmarking use cases. For extensive performance benchmarking, with thousands of proxy-injected services, we also provide a tuned Istio install that allocates extra memory and CPU to the Istio control plane.

Istio’s demo installation is not suitable for performance testing, because it is designed to be deployed on a small trial cluster, and has full tracing and access logs enabled to showcase Istio’s features.

2. Focus on the data plane

Our benchmarking scripts focus on evaluating the Istio data plane: the Envoy proxies that mediate traffic between application containers. Why focus on the data plane? Because at scale, with lots of application containers, the data plane’s memory and CPU usage quickly eclipses that of the Istio control plane. Let’s look at an example of how this happens:

Say you run 2,000 Envoy-injected pods, each handling 1,000 requests per second. Each proxy is using 50 MB of memory, and to configure all these proxies, Pilot is using 1 vCPU and 1.5 GB of memory. All together, the Istio data plane (the sum of all the Envoy proxies) is using 100 GB of memory, compared to Pilot’s 1.5 GB.

It is also important to focus on data plane performance for latency reasons. This is because most application requests move through the Istio data plane, not the control plane. There are two exceptions:

Telemetry reporting: Each proxy sends raw telemetry data to Mixer, which Mixer processes into metrics, traces, and other telemetry. The raw telemetry data is similar to access logs, and therefore comes at a cost. Access log processing consumes CPU and keeps a worker thread from picking up the next unit of work. At higher throughput, it is more likely that the next unit of work is waiting in the queue to be picked up by the worker. This can lead to long-tail (99th percentile) latency for Envoy.
Custom policy checks: When using custom Istio policy adapters, policy checks are on the request path. This means that request headers and metadata on the data path will be sent to the control plane (Mixer), resulting in higher request latency. Note: These policy checks are disabled by default, as the most common policy use case (RBAC) is performed entirely by the Envoy proxies.

Both of these exceptions will go away in a future Istio release, when Mixer V2 moves all policy and telemetry features directly into the proxies.

Next, when testing Istio’s data plane performance at scale, it’s important to test not only at increasing requests per second, but also against an increasing number of concurrent connections. This is because real-world, high-throughput traffic comes from multiple clients. The provided scripts allow you to perform the same load test with any number of concurrent connections, at increasing RPS.

Lastly, our test environment measures requests between two pods, not many. The client pod is Fortio, which sends traffic to the server pod.

Why test with only two pods? Because scaling up throughput (RPS) and connections (threads) has a greater effect on Envoy’s performance than increasing the total size of the service registry — or, the total number of pods and services in the Kubernetes cluster. When the size of the service registry grows, Envoy does have to keep track of more endpoints, and lookup time per request does increase, but by a tiny constant. If you have many services, and this constant becomes a latency concern, Istio provides a Sidecar resource, which allows you to limit which services each Envoy knows about.

3. Measure with and without proxies

While many Istio features, such as mutual TLS authentication, rely on an Envoy proxy next to an application pod, you can selectively disable sidecar proxy injection for some of your mesh services. As you scale up Istio for production, you may want to incrementally add the sidecar proxy to your workloads.

To that end, the test scripts provide three different modes. These modes analyze Istio’s performance when a request goes through both the client and server proxies (both), just the server proxy (serveronly), and neither proxy (baseline).

You can also disable Mixer to stop Istio’s telemetry during the performance tests, which provides results in line with the performance we expect when the Mixer V2 work is completed. Istio also supports Envoy native telemetry, which performs similarly to having Istio’s telemetry disabled.

Istio 1.2 Performance

Let’s see how to use this test environment to analyze the data plane performance of Istio 1.2. We also provide instructions to run the same performance tests for the Linkerd data plane. Currently, only latency benchmarking is supported for Linkerd.

For measuring Istio’s sidecar proxy latency, we look at the 50th, 90th, and 99th percentiles for an increasing number of concurrent connections,keeping request throughput (RPS) constant.

We found that with 16 concurrent connections and 1000 RPS, Istio adds 3ms over the baseline (P50) when a request travels through both a client and server proxy. (Subtract the pink line, base, from the green line, both.) At 64 concurrent connections, Istio adds 12ms over the baseline, but with Mixer disabled (nomixer_both), Istio only adds 7ms.

In the 90th percentile, with 16 concurrent connections, Istio adds 6ms; with 64 connections, Istio adds 20ms.

Finally, in the 99th percentile, with 16 connections, Istio adds 10ms over the baseline. At 64 connections, Istio adds 25ms with Mixer, or 10ms without Mixer.

For CPU usage, we measured with an increasing request throughput (RPS), and a constant number of concurrent connections. We found that Envoy’s maximum CPU usage at 3000 RPS, with Mixer enabled, was 1.2 vCPUs. At 1000 RPS, one Envoy uses approximately half of a CPU.

Summary

In the process of benchmarking Istio’s performance, we learned several key lessons:

Use an environment that mimics production.
Focus on data plane traffic.
Measure against a baseline.
Increase concurrent connections as well as total throughput.

For a mesh with 1000 RPS across 16 connections, Istio 1.2 adds just 3 milliseconds of latency over the baseline, in the 50th percentile.

Also check out the Istio Performance and Scalability guide for the most up-to-date performance data.

Thank you for reading, and happy benchmarking!

Extending Istio Self-Signed Root Certificate Lifetime

Oliver Liu — Fri, 07 Jun 2019 00:00:00 +0000

Istio self-signed certificates have historically had a 1 year default lifetime. If you are using Istio self-signed certificates, you need to schedule regular root transitions before they expire. An expiration of a root certificate may lead to an unexpected cluster-wide outage. The issue affects new clusters created with versions up to 1.0.7 and 1.1.7.

See Extending Self-Signed Certificate Lifetime for information on how to gauge the age of your certificates and how to perform rotation.

Secure Control of Egress Traffic in Istio, part 1

Vadim Eisenberg (IBM) — Wed, 22 May 2019 00:00:00 +0000

This is part 1 in a new series about secure control of egress traffic in Istio that I am going to publish. In this installment, I explain why you should apply egress traffic control to your cluster, the attacks involving egress traffic you want to prevent, and the requirements for a system for egress traffic control to do so. Once you agree that you should control the egress traffic coming from your cluster, the following questions arise: What is required from a system for secure control of egress traffic? Which is the best solution to fulfill these requirements? (spoiler: Istio in my opinion) Future installments will describe the implementation of the secure control of egress traffic in Istio and compare it with other solutions.

The most important security aspect for a service mesh is probably ingress traffic. You definitely must prevent attackers from penetrating the cluster through ingress APIs. Having said that, securing the traffic leaving the mesh is also very important. Once your cluster is compromised, and you must be prepared for that scenario, you want to reduce the damage as much as possible and prevent the attackers from using the cluster for further attacks on external services and legacy systems outside of the cluster. To achieve that goal, you need secure control of egress traffic.

Compliance requirements are another reason to implement secure control of egress traffic. For example, the Payment Card Industry (PCI) Data Security Standard requires that inbound and outbound traffic must be restricted to that which is necessary:

And specifically regarding outbound traffic:

Let’s start with the attacks that involve egress traffic.

The attacks

An IT organization must assume it will be attacked if it hasn’t been attacked already, and that part of its infrastructure could already be compromised or become compromised in the future. Once attackers are able to penetrate an application in a cluster, they can proceed to attack external services: legacy systems, external web services and databases. The attackers may want to steal the data of the application and to transfer it to their external servers. Attackers’ malware may require access to attackers’ servers to download updates. The attackers may use pods in the cluster to perform DDOS attacks or to break into external systems. Even though you cannot know all the possible types of attacks, you want to reduce possibilities for any attacks, both for known and unknown ones.

The external attackers gain access to the application’s container from outside the mesh through a bug in the application but attackers can also be internal, for example, malicious DevOps people inside the organization.

To prevent the attacks described above, some form of egress traffic control must be applied. Let me present egress traffic control in the following section.

The solution: secure control of egress traffic

Secure control of egress traffic means monitoring the egress traffic and enforcing all the security policies regarding the egress traffic. Monitoring the egress traffic, enables you to analyze it, possibly offline, and detect the attacks even if you were unable to prevent them in real time. Another good practice to reduce possibilities of attacks is to specify policies that limit access following the Need to know principle: only the applications that need external services should be allowed to access the external services they need.

Let me now turn to the requirements for egress traffic control we collected.

Requirements for egress traffic control

My colleagues at IBM and I collected requirements for secure control of egress traffic from several customers, and combined them with the egress traffic control requirements from Kubernetes Network Special Interest Group.

Istio 1.1 satisfies all gathered requirements:

Support for TLS with SNI or for TLS origination by Istio.
Monitor SNI and the source workload of every egress access.
Define and enforce policies per cluster, e.g.:
- all applications in the cluster may access service1.foo.com (a specific host)
- all applications in the cluster may access any host of the form *.bar.com (a wildcarded domain)
All unspecified access must be blocked.
Define and enforce policies per source, Kubernetes-aware:
- application A may access *.foo.com.
- application B may access *.bar.com.
All other access must be blocked, in particular access of application A to service1.bar.com.
Prevent tampering. In case an application pod is compromised, prevent the compromised pod from escaping monitoring, from sending fake information to the monitoring system, and from breaking the egress policies.
Nice to have: traffic control is transparent to the applications.

Let me explain each requirement in more detail. The first requirement states that only TLS traffic to the external services must be supported. The requirement emerged upon observation that all the traffic that leaves the cluster must be encrypted. This means that either the applications perform TLS origination or Istio must perform TLS origination for them. Note that in the case an application performs TLS origination, the Istio proxies cannot see the original traffic, only the encrypted one, so the proxies see the TLS protocol only. For the proxies it does not matter if the original protocol is HTTP or MongoDB, all the Istio proxies can see is TLS traffic.

The second requirement states that SNI and the source of the traffic must be monitored. Monitoring is the first step to prevent attacks. Even if attackers would be able to access external services from the cluster, if the access is monitored, there is a chance to discover the suspicious traffic and take a corrective action.

Note that in the case of TLS originated by an application, the Istio sidecar proxies can only see TCP traffic and a TLS handshake that includes SNI. A label of the source pod could identify the source of the traffic but a service account of the pod or some other source identifier could be used. We call this property of an egress control system as being Kubernetes-aware: the system must understand Kubernetes artifacts like pods and service accounts. If the system is not Kubernetes-aware, it can only monitor the IP address as the identifier of the source.

The third requirement states that Istio operators must be able to define policies for egress traffic for the entire cluster. The policies state which external services may be accessed by any pod in the cluster. The external services can be identified either by a Fully qualified domain name of the service, e.g. www.ibm.com or by a wildcarded domain, e.g. *.ibm.com. Only the specified external services may be accessed, all other egress traffic is blocked.

This requirement originates from the need to prevent attackers from accessing malicious sites, for example for downloading updates/instructions for their malware. You also want to limit the number of external sites that the attackers can access and attack. You want to allow access only to the external services that the applications in the cluster need to access and to block access to all the other services, this way you reduce the attack surface. While the external services can have their own security mechanisms, you want to exercise Defense in depth and to have multiple security layers: a security layer in your cluster in addition to the security layers in the external systems.

This requirement means that the external services must be identifiable by domain names. We call this property of an egress control system as being DNS-aware. If the system is not DNS-aware, the external services must be specified by IP addresses. Using IP addresses is not convenient and often is not feasible, since the IP addresses of a service can change. Sometimes all the IP addresses of a service are not even known, for example in the case of CDNs.

The fourth requirement states that the source of the egress traffic must be added to the policies effectively extending the third requirement. Policies can specify which source can access which external service and the source must be identified just as in the second requirement, for example, by a label of the source pod or by service account of the pod. It means that policy enforcement must also be Kubernetes-aware. If policy enforcement is not Kubernetes-aware, the policies must identify the source of traffic by the IP of the pod, which is not convenient, especially since the pods can come and go so their IPs are not static.

The fifth requirement states that even if the cluster is compromised and the attackers control some of the pods, they must not be able to cheat the monitoring or to violate policies of the egress control system. We say that such a system provides secure control of egress traffic.

The sixth requirement states that the traffic control should be provided without changing the application containers, in particular without changing the code of the applications and without changing the environment of the containers. We call such a control of egress traffic transparent.

In the next posts I will show that Istio can function as an example of an egress traffic control system that satisfies all of these requirements, in particular it is transparent, DNS-aware, and Kubernetes-aware.

Summary

I hope that you are convinced that controlling egress traffic is important for the security of your cluster. In the part 2 of this series I describe the Istio way to perform secure control of egress traffic. In the part 3 of this series I compare it with alternative solutions such as Kubernetes Network Policies and legacy egress proxies/firewalls.

Architecting Istio 1.1 for Performance

Surya V Duggirala (IBM), Mandar Jog (Google), Jose Nativio (IBM) — Tue, 19 Mar 2019 00:00:00 +0000

Hyper-scale, microservice-based cloud environments have been exciting to build but challenging to manage. Along came Kubernetes (container orchestration) in 2014, followed by Istio (container service management) in 2017. Both open-source projects enable developers to scale container-based applications without spending too much time on administration tasks.

Now, new enhancements in Istio 1.1 deliver scale-up with improved application performance and service management efficiency. Simulations using our sample commercial airline reservation application show the following improvements, compared to Istio 1.0.

We’ve seen substantial application performance gains:

up to 30% reduction in application average latency
up to 40% faster service startup times in a large mesh

As well as impressive improvements in service management efficiency:

up to 90% reduction in Pilot CPU usage in a large mesh
up to 50% reduction in Pilot memory usage in a large mesh

With Istio 1.1, organizations can be more confident in their ability to scale applications with consistency and control – even in hyper-scale cloud environments.

Congratulations to the Istio experts around the world who contributed to this release. We could not be more pleased with these results.

Istio 1.1 performance enhancements

As members of the Istio Performance and Scalability workgroup, we have done extensive performance evaluations. We introduced many performance design features for Istio 1.1, in collaboration with other Istio contributors. Some of the most visible performance enhancements in 1.1 include:

Significant reduction in default collection of Envoy-generated statistics
Added load-shedding functionality to Mixer workloads
Improved the protocol between Envoy and Mixer
Namespace isolation, to reduce operational overhead
Configurable concurrent worker threads, which can improve overall throughput
Configurable filters that limit telemetry data
Removal of synchronization bottlenecks

Continuous code quality and performance verification

Regression Patrol drives continuous improvement in Istio performance and quality. Behind the scenes, the Regression Patrol helps Istio developers to identify and fix code issues. Daily builds are checked using a customer-centric benchmark, BluePerf. The results are published to the Istio community web portal. Various application configurations are evaluated to help provide insights on Istio component performance.

Another tool that is used to evaluate the performance of Istio’s builds is Fortio, which provides a synthetic end to end load testing benchmark.

Summary

Istio 1.1 was designed for performance and scalability. The Istio Performance and Scalability workgroup measured significant performance improvements over 1.0. Istio 1.1 introduces new features and optimizations to help harden the service mesh for enterprise microservice workloads. The Istio 1.1 Performance and Tuning Guide documents performance simulations, provides sizing and capacity planning guidance, and includes best practices for tuning custom use cases.

Useful links

Istio Service Mesh Performance (34:30), by Surya Duggirala, Laurent Demailly and Fawad Khaliq at KubeCon Europe 2018
Istio Performance and Scalability discussion forum

Disclaimer

The performance data contained herein was obtained in a controlled, isolated environment. Actual results that may be obtained in other operating environments may vary significantly. There is no guarantee that the same or similar results will be obtained elsewhere.

Version Routing in a Multicluster Service Mesh

Frank Budinsky (IBM) — Thu, 07 Feb 2019 00:00:00 +0000

If you’ve spent any time looking at Istio, you’ve probably noticed that it includes a lot of features that can be demonstrated with simple tasks and examples running on a single Kubernetes cluster. Because most, if not all, real-world cloud and microservices-based applications are not that simple and will need to have the services distributed and running in more than one location, you may be wondering if all these things will be just as simple in your real production environment.

Fortunately, Istio provides several ways to configure a service mesh so that applications can, more-or-less transparently, be part of a mesh where the services are running in more than one cluster, i.e., in a multicluster deployment. The simplest way to set up a multicluster mesh, because it has no special networking requirements, is using a replicated control plane model. In this configuration, each Kubernetes cluster contributing to the mesh has its own control plane, but each control plane is synchronized and running under a single administrative control.

In this article we’ll look at how one of the features of Istio, traffic management, works in a multicluster mesh with a dedicated control plane topology. We’ll show how to configure Istio route rules to call remote services in a multicluster service mesh by deploying the Bookinfo sample with version v1 of the reviews service running in one cluster, versions v2 and v3 running in a second cluster.

Set up clusters

To start, you’ll need two Kubernetes clusters, both running a slightly customized configuration of Istio.

Set up a multicluster environment with two Istio clusters by following the replicated control planes instructions.

The kubectl command is used to access both clusters with the --context flag. Use the following command to list your contexts:

$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* cluster1 cluster1 user@foo.com default
cluster2 cluster2 user@foo.com default

Export the following environment variables with the context names of your configuration:

$ export CTX_CLUSTER1=<cluster1 context name>
$ export CTX_CLUSTER2=<cluster2 context name>

Deploy version v1 of the `bookinfo` application in `cluster1`

Run the productpage and details services and version v1 of the reviews service in cluster1:

$ kubectl label --context=$CTX_CLUSTER1 namespace default istio-injection=enabled
$ kubectl apply --context=$CTX_CLUSTER1 -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: productpage
labels:
app: productpage
spec:
ports:
- port: 9080
name: http
selector:
app: productpage
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: productpage-v1
spec:
replicas: 1
template:
metadata:
labels:
app: productpage
version: v1
spec:
containers:
- name: productpage
image: istio/examples-bookinfo-productpage-v1:1.10.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
spec:
ports:
- port: 9080
name: http
selector:
app: details
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: details-v1
spec:
replicas: 1
template:
metadata:
labels:
app: details
version: v1
spec:
containers:
- name: details
image: istio/examples-bookinfo-details-v1:1.10.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v1
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v1
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v1:1.10.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
EOF

Deploy `bookinfo` v2 and v3 services in `cluster2`

Run the ratings service and version v2 and v3 of the reviews service in cluster2:

$ kubectl label --context=$CTX_CLUSTER2 namespace default istio-injection=enabled
$ kubectl apply --context=$CTX_CLUSTER2 -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: ratings
labels:
app: ratings
spec:
ports:
- port: 9080
name: http
selector:
app: ratings
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ratings-v1
spec:
replicas: 1
template:
metadata:
labels:
app: ratings
version: v1
spec:
containers:
- name: ratings
image: istio/examples-bookinfo-ratings-v1:1.10.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: v1
kind: Service
metadata:
name: reviews
labels:
app: reviews
spec:
ports:
- port: 9080
name: http
selector:
app: reviews
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v2
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v2
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v2:1.10.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: reviews-v3
spec:
replicas: 1
template:
metadata:
labels:
app: reviews
version: v3
spec:
containers:
- name: reviews
image: istio/examples-bookinfo-reviews-v3:1.10.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 9080
EOF

Access the `bookinfo` application

Just like any application, we’ll use an Istio gateway to access the bookinfo application.

Create the bookinfo gateway in cluster1:

$ kubectl apply --context=$CTX_CLUSTER1 -f @samples/bookinfo/networking/bookinfo-gateway.yaml@

Follow the Bookinfo sample instructions to determine the ingress IP and port and then point your browser to http://$GATEWAY_URL/productpage.

You should see the productpage with reviews, but without ratings, because only v1 of the reviews service is running on cluster1 and we have not yet configured access to cluster2.

Create a service entry and destination rule on `cluster1` for the remote reviews service

As described in the setup instructions, remote services are accessed with a .global DNS name. In our case, it’s reviews.default.global, so we need to create a service entry and destination rule for that host. The service entry will use the cluster2 gateway as the endpoint address to access the service. You can use the gateway’s DNS name, if it has one, or its public IP, like this:

$ export CLUSTER2_GW_ADDR=$(kubectl get --context=$CTX_CLUSTER2 svc --selector=app=istio-ingressgateway \
-n istio-system -o jsonpath="{.items[0].status.loadBalancer.ingress[0].ip}")

Now create the service entry and destination rule using the following command:

$ kubectl apply --context=$CTX_CLUSTER1 -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: reviews-default
spec:
hosts:
- reviews.default.global
location: MESH_INTERNAL
ports:
- name: http1
number: 9080
protocol: http
resolution: DNS
addresses:
- 240.0.0.3
endpoints:
- address: ${CLUSTER2_GW_ADDR}
labels:
cluster: cluster2
ports:
http1: 15443 # Do not change this port value
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews-global
spec:
host: reviews.default.global
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v2
labels:
cluster: cluster2
- name: v3
labels:
cluster: cluster2
EOF

The address 240.0.0.3 of the service entry can be any arbitrary unallocated IP. Using an IP from the class E addresses range 240.0.0.0/4 is a good choice. Check out the gateway-connected multicluster example for more details.

Note that the labels of the subsets in the destination rule map to the service entry endpoint label (cluster: cluster2) corresponding to the cluster2 gateway. Once the request reaches the destination cluster, a local destination rule will be used to identify the actual pod labels (version: v1 or version: v2) corresponding to the requested subset.

Create a destination rule on both clusters for the local reviews service

Technically, we only need to define the subsets of the local service that are being used in each cluster (i.e., v1 in cluster1, v2 and v3 in cluster2), but for simplicity we’ll just define all three subsets in both clusters, since there’s nothing wrong with defining subsets for versions that are not actually deployed.

$ kubectl apply --context=$CTX_CLUSTER1 -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews.default.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
EOF

$ kubectl apply --context=$CTX_CLUSTER2 -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews.default.svc.cluster.local
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
EOF

Create a virtual service to route reviews service traffic

At this point, all calls to the reviews service will go to the local reviews pods (v1) because if you look at the source code you will see that the productpage implementation is simply making requests to http://reviews:9080 (which expands to host reviews.default.svc.cluster.local), the local version of the service. The corresponding remote service is named reviews.default.global, so route rules are needed to redirect requests to the global host.

Apply the following virtual service to direct traffic for user jason to reviews versions v2 and v3 (⁵⁰⁄₅₀) which are running on cluster2. Traffic for any other user will go to reviews version v1.

$ kubectl apply --context=$CTX_CLUSTER1 -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews.default.svc.cluster.local
http:
- match:
- headers:
end-user:
exact: jason
route:
- destination:
host: reviews.default.global
subset: v2
weight: 50
- destination:
host: reviews.default.global
subset: v3
weight: 50
- route:
- destination:
host: reviews.default.svc.cluster.local
subset: v1
EOF

Return to your browser and login as user jason. If you refresh the page several times, you should see the display alternating between black and red ratings stars (v2 and v3). If you logout, you will only see reviews without ratings (v1).

Summary

In this article, we’ve seen how to use Istio route rules to distribute the versions of a service across clusters in a multicluster service mesh with a replicated control plane model. In this example, we manually configured the .global service entry and destination rules needed to provide connectivity to one remote service, reviews. In general, however, if we wanted to enable any service to run either locally or remotely, we would need to create .global resources for every service. Fortunately, this process could be automated and likely will be in a future Istio release.

Sail the Blog!

Rigs Caballero, Google — Tue, 05 Feb 2019 00:00:00 +0000

Welcome to the Istio blog!

To make it easier to publish your content on our website, we updated the content types guide.

The goal of the updated guide is to make sharing and finding content easier.

We want to make sharing timely information on Istio easy and the Istio blog is a good place to start.

We welcome your posts to the blog if you think your content falls in one of the following four categories:

Your post details your experience using and configuring Istio. Ideally, your post shares a novel experience or perspective.
Your post highlights Istio features.
Your post details how to accomplish a task or fulfill a specific use case using Istio.

Posting your blog is only one PR away and, if you wish, you can request a review.

We look forward to reading about your Istio experience on the blog soon!

Egress Gateway Performance Investigation

Jose Nativio, IBM — Thu, 31 Jan 2019 00:00:00 +0000

The main objective of this investigation was to determine the impact on performance and resource utilization when an egress gateway is added in the service mesh to access an external service (MongoDB, in this case). The steps to configure an egress gateway for an external MongoDB are described in the blog Consuming External MongoDB Services.

The application used for this investigation was the Java version of Acmeair, which simulates an airline reservation system. This application is used in the Performance Regression Patrol of Istio daily builds, but on that setup the microservices have been accessing the external MongoDB directly via their sidecars, without an egress gateway.

The diagram below illustrates how regression patrol currently runs with Acmeair and Istio:

Acmeair benchmark in the Istio performance regression patrol environment

Another difference is that the application communicates with the external DB with plain MongoDB protocol. The first change made for this study was to establish a TLS communication between the MongoDB and its clients running within the application, as this is a more realistic scenario.

Several cases for accessing the external database from the mesh were tested and described next.

Egress traffic cases

Case 1: Bypassing the sidecar

In this case, the sidecar does not intercept the communication between the application and the external DB. This is accomplished by setting the init container argument -x with the CIDR of the MongoDB, which makes the sidecar ignore messages to/from this IP address. For example:

 - -x
- "169.47.232.211/32"

Traffic to external MongoDB by-passing the sidecar

Case 2: Through the sidecar, with service entry

This is the default configuration when the sidecar is injected into the application pod. All messages are intercepted by the sidecar and routed to the destination according to the configured rules, including the communication with external services. The MongoDB was defined as a ServiceEntry.

Sidecar intercepting traffic to external MongoDB

Case 3: Egress gateway

The egress gateway and corresponding destination rule and virtual service resources are defined for accessing MongoDB. All traffic to and from the external DB goes through the egress gateway (envoy).

Introduction of the egress gateway to access MongoDB

Case 4: Mutual TLS between sidecars and the egress gateway

In this case, there is an extra layer of security between the sidecars and the gateway, so some impact in performance is expected.

Enabling mutual TLS between sidecars and the egress gateway

Case 5: Egress gateway with SNI proxy

This scenario is used to evaluate the case where another proxy is required to access wildcarded domains. This may be required due current limitations of envoy. An nginx proxy was created as sidecar in the egress gateway pod.

Egress gateway with additional SNI Proxy

Environment

Istio version: 1.0.2
K8s version: 1.10.5_1517
Acmeair App: 4 services (1 replica of each), inter-services transactions, external Mongo DB, avg payload: 620 bytes.

Results

Jmeter was used to generate the workload which consisted in a sequence of 5-minute runs, each one using a growing number of clients making http requests. The number of clients used were 1, 5, 10, 20, 30, 40, 50 and 60.

Throughput

The chart below shows the throughput obtained for the different cases:

Throughput obtained for the different cases

As you can see, there is no major impact in having sidecars and the egress gateway between the application and the external MongoDB, but enabling mutual TLS and then adding the SNI proxy caused a degradation in the throughput of about 10% and 24%, respectively.

Response time

The average response times for the different requests were collected when traffic was being driven with 20 clients. The chart below shows the average, median, 90%, 95% and 99% average values for each case:

Response times obtained for the different configurations

Likewise, not much difference in the response times for the 3 first cases, but mutual TLS and the extra proxy adds noticeable latency.

CPU utilization

The CPU usage was collected for all Istio components as well as for the sidecars during the runs. For a fair comparison, CPU used by Istio was normalized by the throughput obtained for a given run. The results are shown in the following graph:

CPU usage normalized by TPS

In terms of CPU consumption per transaction, Istio has used significantly more CPU only in the egress gateway + SNI proxy case.

Conclusion

In this investigation, we tried different options to access an external TLS-enabled MongoDB to compare their performance. The introduction of the Egress Gateway did not have a significant impact on the performance nor meaningful additional CPU consumption. Only when enabling mutual TLS between sidecars and egress gateway or using an additional SNI proxy for wildcarded domains we could observe some degradation.

Demystifying Istio's Sidecar Injection Model

Manish Chugtu — Thu, 31 Jan 2019 00:00:00 +0000

A simple overview of an Istio service-mesh architecture always starts with describing the control-plane and data-plane.

From Istio’s documentation:

Istio Architecture

It is important to understand that the sidecar injection into the application pods happens automatically, though manual injection is also possible. Traffic is directed from the application services to and from these sidecars without developers needing to worry about it. Once the applications are connected to the Istio service mesh, developers can start using and reaping the benefits of all that the service mesh has to offer. However, how does the data plane plumbing happen and what is really required to make it work seamlessly? In this post, we will deep-dive into the specifics of the sidecar injection models to gain a very clear understanding of how sidecar injection works.

Sidecar injection

In simple terms, sidecar injection is adding the configuration of additional containers to the pod template. The added containers needed for the Istio service mesh are:

istio-init This init container is used to setup the iptables rules so that inbound/outbound traffic will go through the sidecar proxy. An init container is different than an app container in following ways:

It runs before an app container is started and it always runs to completion.
If there are many init containers, each should complete with success before the next container is started.

So, you can see how this type of container is perfect for a set-up or initialization job which does not need to be a part of the actual application container. In this case, istio-init does just that and sets up the iptables rules.

istio-proxy This is the actual sidecar proxy (based on Envoy).

Manual injection

In the manual injection method, you can use istioctl to modify the pod template and add the configuration of the two containers previously mentioned. For both manual as well as automatic injection, Istio takes the configuration from the istio-sidecar-injector configuration map (configmap) and the mesh’s istio configmap.

Let’s look at the configuration of the istio-sidecar-injector configmap, to get an idea of what actually is going on.

$ kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.config}'
SNIPPET from the output:
policy: enabled
template: |-
initContainers:
- name: istio-init
image: docker.io/istio/proxy_init:1.0.2
args:
- "-p"
- [[ .MeshConfig.ProxyListenPort ]]
- "-u"
- 1337
.....
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
add:
- NET_ADMIN
restartPolicy: Always
containers:
- name: istio-proxy
image: [[ if (isset .ObjectMeta.Annotations "sidecar.istio.io/proxyImage") -]]
"[[ index .ObjectMeta.Annotations "sidecar.istio.io/proxyImage" ]]"
[[ else -]]
docker.io/istio/proxyv2:1.0.2
[[ end -]]
args:
- proxy
- sidecar
.....
env:
.....
- name: ISTIO_META_INTERCEPTION_MODE
value: [[ or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String ]]
imagePullPolicy: IfNotPresent
securityContext:
readOnlyRootFilesystem: true
[[ if eq (or (index .ObjectMeta.Annotations "sidecar.istio.io/interceptionMode") .ProxyConfig.InterceptionMode.String) "TPROXY" -]]
capabilities:
add:
- NET_ADMIN
restartPolicy: Always
.....

As you can see, the configmap contains the configuration for both, the istio-init init container and the istio-proxy proxy container. The configuration includes the name of the container image and arguments like interception mode, capabilities, etc.

From a security point of view, it is important to note that istio-init requires NET_ADMIN capabilities to modify iptables within the pod’s namespace and so does istio-proxy if configured in TPROXY mode. As this is restricted to a pod’s namespace, there should be no problem. However, I have noticed that recent open-shift versions may have some issues with it and a workaround is needed. One such option is mentioned at the end of this post.

To modify the current pod template for sidecar injection, you can:

$ istioctl kube-inject -f demo-red.yaml | kubectl apply -f -

To use modified configmaps or local configmaps:

Create inject-config.yaml and mesh-config.yaml from the configmaps

$ kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.config}' > inject-config.yaml
$ kubectl -n istio-system get configmap istio -o=jsonpath='{.data.mesh}' > mesh-config.yaml

Modify the existing pod template, in my case, demo-red.yaml:

$ istioctl kube-inject --injectConfigFile inject-config.yaml --meshConfigFile mesh-config.yaml --filename demo-red.yaml --output demo-red-injected.yaml

Apply the demo-red-injected.yaml

$ kubectl apply -f demo-red-injected.yaml

As seen above, we create a new template using the sidecar-injector and the mesh configuration to then apply that new template using kubectl. If we look at the injected YAML file, it has the configuration of the Istio-specific containers, as we discussed above. Once we apply the injected YAML file, we see two containers running. One of them is the actual application container, and the other is the istio-proxy sidecar.

$ kubectl get pods | grep demo-red
demo-red-pod-8b5df99cc-pgnl7 2/2 Running 0 3d

The count is not 3 because the istio-init container is an init type container that exits after doing what it supposed to do, which is setting up the iptable rules within the pod. To confirm the init container exit, let’s look at the output of kubectl describe:

$ kubectl describe pod demo-red-pod-8b5df99cc-pgnl7
SNIPPET from the output:
Name: demo-red-pod-8b5df99cc-pgnl7
Namespace: default
.....
Labels: app=demo-red
pod-template-hash=8b5df99cc
version=version-red
Annotations: sidecar.istio.io/status={"version":"3c0b8d11844e85232bc77ad85365487638ee3134c91edda28def191c086dc23e","initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs...
Status: Running
IP: 10.32.0.6
Controlled By: ReplicaSet/demo-red-pod-8b5df99cc
Init Containers:
istio-init:
Container ID: docker://bef731eae1eb3b6c9d926cacb497bb39a7d9796db49cd14a63014fc1a177d95b
Image: docker.io/istio/proxy_init:1.0.2
Image ID: docker-pullable://docker.io/istio/proxy_init@sha256:e16a0746f46cd45a9f63c27b9e09daff5432e33a2d80c8cc0956d7d63e2f9185
.....
State: Terminated
Reason: Completed
.....
Ready: True
Containers:
demo-red:
Container ID: docker://8cd9957955ff7e534376eb6f28b56462099af6dfb8b9bc37aaf06e516175495e
Image: chugtum/blue-green-image:v3
Image ID: docker-pullable://docker.io/chugtum/blue-green-image@sha256:274756dbc215a6b2bd089c10de24fcece296f4c940067ac1a9b4aea67cf815db
State: Running
Started: Sun, 09 Dec 2018 18:12:31 -0800
Ready: True
istio-proxy:
Container ID: docker://ca5d690be8cd6557419cc19ec4e76163c14aed2336eaad7ebf17dd46ca188b4a
Image: docker.io/istio/proxyv2:1.0.2
Image ID: docker-pullable://docker.io/istio/proxyv2@sha256:54e206530ba6ca9b3820254454e01b7592e9f986d27a5640b6c03704b3b68332
Args:
proxy
sidecar
.....
State: Running
Started: Sun, 09 Dec 2018 18:12:31 -0800
Ready: True
.....

As seen in the output, the State of the istio-init container is Terminated with the Reason being Completed. The only two containers running are the main application demo-red container and the istio-proxy container.

Automatic injection

Most of the times, you don’t want to manually inject a sidecar every time you deploy an application, using the istioctl command, but would prefer that Istio automatically inject the sidecar to your pod. This is the recommended approach and for it to work, all you need to do is to label the namespace where you are deploying the app with istio-injection=enabled.

Once labeled, Istio injects the sidecar automatically for any pod you deploy in that namespace. In the following example, the sidecar gets automatically injected in the deployed pods in the istio-dev namespace.

$ kubectl get namespaces --show-labels
NAME STATUS AGE LABELS
default Active 40d <none>
istio-dev Active 19d istio-injection=enabled
istio-system Active 24d <none>
kube-public Active 40d <none>
kube-system Active 40d <none>

But how does this work? To get to the bottom of this, we need to understand Kubernetes admission controllers.

From Kubernetes documentation:

For automatic sidecar injection, Istio relies on Mutating Admission Webhook. Let’s look at the details of the istio-sidecar-injector mutating webhook configuration.

$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml
SNIPPET from the output:
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"admissionregistration.k8s.io/v1beta1","kind":"MutatingWebhookConfiguration","metadata":{"annotations":{},"labels":{"app":"istio-sidecar-injector","chart":"sidecarInjectorWebhook-1.0.1","heritage":"Tiller","release":"istio-remote"},"name":"istio-sidecar-injector","namespace":""},"webhooks":[{"clientConfig":{"caBundle":"","service":{"name":"istio-sidecar-injector","namespace":"istio-system","path":"/inject"}},"failurePolicy":"Fail","name":"sidecar-injector.istio.io","namespaceSelector":{"matchLabels":{"istio-injection":"enabled"}},"rules":[{"apiGroups":[""],"apiVersions":["v1"],"operations":["CREATE"],"resources":["pods"]}]}]}
creationTimestamp: 2018-12-10T08:40:15Z
generation: 2
labels:
app: istio-sidecar-injector
chart: sidecarInjectorWebhook-1.0.1
heritage: Tiller
release: istio-remote
name: istio-sidecar-injector
.....
webhooks:
- clientConfig:
service:
name: istio-sidecar-injector
namespace: istio-system
path: /inject
name: sidecar-injector.istio.io
namespaceSelector:
matchLabels:
istio-injection: enabled
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods

This is where you can see the webhook namespaceSelector label that is matched for sidecar injection with the label istio-injection: enabled. In this case, you also see the operations and resources for which this is done when the pods are created. When an apiserver receives a request that matches one of the rules, the apiserver sends an admission review request to the webhook service as specified in the clientConfig:configuration with the name: istio-sidecar-injector key-value pair. We should be able to see that this service is running in the istio-system namespace.

$ kubectl get svc --namespace=istio-system | grep sidecar-injector
istio-sidecar-injector ClusterIP 10.102.70.184 <none> 443/TCP 24d

This configuration ultimately does pretty much the same as we saw in manual injection. Just that it is done automatically during pod creation, so you won’t see the change in the deployment. You need to use kubectl describe to see the sidecar proxy and the init proxy.

The automatic sidecar injection not only depends on the namespaceSelector mechanism of the webhook, but also on the default injection policy and the per-pod override annotation.

If you look at the istio-sidecar-injector ConfigMap again, it has the default injection policy defined. In our case, it is enabled by default.

$ kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath='{.data.config}'
SNIPPET from the output:
policy: enabled
template: |-
initContainers:
- name: istio-init
image: "gcr.io/istio-release/proxy_init:1.0.2"
args:
- "-p"
- [[ .MeshConfig.ProxyListenPort ]]

You can also use the annotation sidecar.istio.io/inject in the pod template to override the default policy. The following example disables the automatic injection of the sidecar for the pods in a Deployment.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ignored
spec:
template:
metadata:
annotations:
sidecar.istio.io/inject: "false"
spec:
containers:
- name: ignored
image: tutum/curl
command: ["/bin/sleep","infinity"]

This example shows there are many variables, based on whether the automatic sidecar injection is controlled in your namespace, ConfigMap, or pod and they are:

webhooks namespaceSelector (istio-injection: enabled)
default policy (Configured in the ConfigMap istio-sidecar-injector)
per-pod override annotation (sidecar.istio.io/inject)

The injection status table shows a clear picture of the final injection status based on the value of the above variables.

Traffic flow from application container to sidecar proxy

Now that we are clear about how a sidecar container and an init container are injected into an application manifest, how does the sidecar proxy grab the inbound and outbound traffic to and from the container? We did briefly mention that it is done by setting up the iptable rules within the pod namespace, which in turn is done by the istio-init container. Now, it is time to verify what actually gets updated within the namespace.

Let’s get into the application pod namespace we deployed in the previous section and look at the configured iptables. I am going to show an example using nsenter. Alternatively, you can enter the container in a privileged mode to see the same information. For folks without access to the nodes, using exec to get into the sidecar and running iptables is more practical.

$ docker inspect b8de099d3510 --format '{{ .State.Pid }}'
4125

$ nsenter -t 4215 -n iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N ISTIO_INBOUND
-N ISTIO_IN_REDIRECT
-N ISTIO_OUTPUT
-N ISTIO_REDIRECT
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 80 -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15001
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ISTIO_REDIRECT
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001

The output above clearly shows that all the incoming traffic to port 80, which is the port our red-demo application is listening, is now REDIRECTED to port 15001, which is the port that the istio-proxy, an Envoy proxy, is listening. The same holds true for the outgoing traffic.

This brings us to the end of this post. I hope it helped to de-mystify how Istio manages to inject the sidecar proxies into an existing deployment and how Istio routes the traffic to the proxy.

Sidestepping Dependency Ordering with AppSwitch

Dinesh Subhraveti (AppOrbit and Columbia University) — Mon, 14 Jan 2019 00:00:00 +0000

We are going through an interesting cycle of application decomposition and recomposition. While the microservice paradigm is driving monolithic applications to be broken into separate individual services, the service mesh approach is helping them to be connected back together into well-structured applications. As such, microservices are logically separate but not independent. They are usually closely interdependent and taking them apart introduces many new concerns such as need for mutual authentication between services. Istio directly addresses most of those issues.

Dependency ordering problem

An issue that arises due to application decomposition and one that Istio doesn’t address is dependency ordering – bringing up individual services of an application in an order that guarantees that the application as a whole comes up quickly and correctly. In a monolithic application, with all its components built-in, dependency ordering between the components is enforced by internal locking mechanisms. But with individual services potentially scattered across the cluster in a service mesh, starting a service first requires checking that the services it depends on are up and available.

Dependency ordering is deceptively nuanced with a host of interrelated problems. Ordering individual services requires having the dependency graph of the services so that they can be brought up starting from leaf nodes back to the root nodes. It is not easy to construct such a graph and keep it updated over time as interdependencies evolve with the behavior of the application. Even if the dependency graph is somehow provided, enforcing the ordering itself is not easy. Simply starting the services in the specified order obviously won’t do. A service may have started but not be ready to accept connections yet. This is the problem with docker-compose’s depends-on tag, for example.

Apart from introducing sufficiently long sleeps between service startups, a common pattern that is often used is to check for readiness of dependencies before starting a service. In Kubernetes, this could be done with a wait script as part of the init container of the pod. However that means that the entire application would be held up until all its dependencies come alive. Sometimes applications spend several minutes initializing themselves on startup before making their first outbound connection. Not allowing a service to start at all adds substantial overhead to overall startup time of the application. Also, the strategy of waiting on the init container won’t work for the case of multiple interdependent services within the same pod.

Example scenario: IBM WebSphere ND

Let us consider IBM WebSphere ND – a widely deployed application middleware – to grok these problems more closely. It is a fairly complex framework in itself and consists of a central component called deployment manager (dmgr) that manages a set of node instances. It uses UDP to negotiate cluster membership among the nodes and requires that deployment manager is up and operational before any of the node instances can come up and join the cluster.

Why are we talking about a traditional application in the modern cloud-native context? It turns out that there are significant gains to be had by enabling them to run on the Kubernetes and Istio platforms. Essentially it’s a part of the modernization journey that allows running traditional apps alongside green-field apps on the same modern platform to facilitate interoperation between the two. In fact, WebSphere ND is a demanding application. It expects a consistent network environment with specific network interface attributes etc. AppSwitch is equipped to take care of those requirements. For the purpose of this blog however, I’ll focus on the dependency ordering requirement and how AppSwitch addresses it.

Simply deploying dmgr and node instances as pods on a Kubernetes cluster does not work. dmgr and the node instances happen to have a lengthy initialization process that can take several minutes. If they are all co-scheduled, the application typically ends up in a funny state. When a node instance comes up and finds that dmgr is missing, it would take an alternate startup path. Instead, if it had exited immediately, Kubernetes crash-loop would have taken over and perhaps the application would have come up. But even in that case, it turns out that a timely startup is not guaranteed.

One dmgr along with its node instances is a basic deployment configuration for WebSphere ND. Applications like IBM Business Process Manager that are built on top of WebSphere ND running in production environments include several other services. In those configurations, there could be a chain of interdependencies. Depending on the applications hosted by the node instances, there may be an ordering requirement among them as well. With long service initialization times and crash-loop restarts, there is little chance for the application to start in any reasonable length of time.

Sidecar dependency in Istio

Istio itself is affected by a version of the dependency ordering problem. Since connections into and out of a service running under Istio are redirected through its sidecar proxy, an implicit dependency is created between the application service and its sidecar. Unless the sidecar is fully operational, all requests from and to the service get dropped.

Dependency ordering with AppSwitch

So how do we go about addressing these issues? One way is to defer it to the applications and say that they are supposed to be “well behaved” and implement appropriate logic to make themselves immune to startup order issues. However, many applications (especially traditional ones) either timeout or deadlock if misordered. Even for new applications, implementing one off logic for each service is substantial additional burden that is best avoided. Service mesh needs to provide adequate support around these problems. After all, factoring out common patterns into an underlying framework is really the point of service mesh.

AppSwitch explicitly addresses dependency ordering. It sits on the control path of the application’s network interactions between clients and services in a cluster and knows precisely when a service becomes a client by making the connect call and when a particular service becomes ready to accept connections by making the listen call. It’s service router component disseminates information about these events across the cluster and arbitrates interactions among clients and servers. That is how AppSwitch implements functionality such as load balancing and isolation in a simple and efficient manner. Leveraging the same strategic location of the application’s network control path, it is conceivable that the connect and listen calls made by those services can be lined up at a finer granularity rather than coarsely sequencing entire services as per a dependency graph. That would effectively solve the multilevel dependency problem and speedup application startup.

But that still requires a dependency graph. A number of products and tools exist to help with discovering service dependencies. But they are typically based on passive monitoring of network traffic and cannot provide the information beforehand for any arbitrary application. Network level obfuscation due to encryption and tunneling also makes them unreliable. The burden of discovering and specifying the dependencies ultimately falls to the developer or the operator of the application. As it is, even consistency checking a dependency specification is itself quite complex and any way to avoid requiring a dependency graph would be most desirable.

The point of a dependency graph is to know which clients depend on a particular service so that those clients can then be made to wait for the respective service to become live. But does it really matter which specific clients? Ultimately one tautology that always holds is that all clients of a service have an implicit dependency on the service. That’s what AppSwitch leverages to get around the requirement. In fact, that sidesteps dependency ordering altogether. All services of the application can be co-scheduled without regard to any startup order. Interdependencies among them automatically work themselves out at the granularity of individual requests and responses, resulting in quick and correct application startups.

AppSwitch model and constructs

Now that we have a conceptual understanding of AppSwitch’s high-level approach, let’s look at the constructs involved. But first a quick summary of the usage model is in order. Even though it is written for a different context, reviewing my earlier blog on this topic would be useful as well. For completeness, let me also note AppSwitch doesn’t bother with non-network dependencies. For example it may be possible for two services to interact using IPC mechanisms or through the shared file system. Processes with deep ties like that are typically part of the same service anyway and don’t require framework’s intervention for ordering.

At its core, AppSwitch is built on a mechanism that allows instrumenting the BSD socket API and other related calls like fcntl and ioctl that deal with sockets. As interesting as the details of its implementation are, it’s going to distract us from the main topic, so I’d just summarize the key properties that distinguish it from other implementations. (1) It’s fast. It uses a combination of seccomp filtering and binary instrumentation to aggressively limit intervening with application’s normal execution. AppSwitch is particularly suited for service mesh and application networking use cases given that it implements those features without ever having to actually touch the data. In contrast, network level approaches incur per-packet cost. Take a look at this blog for some of the performance measurements. (2) It doesn’t require any kernel support, kernel module or a patch and works on standard distro kernels (3) It can run as regular user (no root). In fact, the mechanism can even make it possible to run Docker daemon without root by removing root requirement to network containers (4) It doesn’t require any changes to the applications whatsoever and works for any type of application – from WebSphere ND and SAP to custom C apps to statically linked Go apps. Only requirement at this point is Linux/x86.

Decoupling services from their references

AppSwitch is built on the fundamental premise that applications should be decoupled from their references. The identity of applications is traditionally derived from the identity of the host on which they run. However, applications and hosts are very different objects that need to be referenced independently. Detailed discussion around this topic along with a conceptual foundation of AppSwitch is presented in this research paper.

The central AppSwitch construct that achieves the decoupling between services objects and their identities is service reference (reference, for short). AppSwitch implements service references based on the API instrumentation mechanism outlined above. A service reference consists of an IP:port pair (and optionally a DNS name) and a label-selector that selects the service represented by the reference and the clients to which this reference applies. A reference supports a few key properties. (1) It can be named independently of the name of the object it refers to. That is, a service may be listening on an IP and port but a reference allows that service to be reached on any other IP and port chosen by the user. This is what allows AppSwitch to run traditional applications captured from their source environments with static IP configurations to run on Kubernetes by providing them with necessary IP addresses and ports regardless of the target network environment. (2) It remains unchanged even if the location of the target service changes. A reference automatically redirects itself as its label-selector now resolves to the new instance of the service (3) Most important for this discussion, a reference remains valid even as the target service is coming up.

To facilitate discovering services that can be accessed through service references, AppSwitch provides an auto-curated service registry. The registry is automatically kept up to date as services come and go across the cluster based on the network API that AppSwitch tracks. Each entry in the registry consists of the IP and port where the respective service is bound. Along with that, it includes a set of labels indicating the application to which this service belongs, the IP and port that the application passed through the socket API when creating the service, the IP and port where AppSwitch actually bound the service on the underlying host on behalf of the application etc. In addition, applications created under AppSwitch carry a set of labels passed by the user that describe the application together with a few default system labels indicating the user that created the application and the host where the application is running etc. These labels are all available to be expressed in the label-selector carried by a service reference. A service in the registry can be made accessible to clients by creating a service reference. A client would then be able to reach the service at the reference’s name (IP:port). Now let’s look at how AppSwitch guarantees that the reference remains valid even when the target service has not yet come up.

Non-blocking requests

AppSwitch leverages the semantics of the BSD socket API to ensure that service references appear valid from the perspective of clients as corresponding services come up. When a client makes a blocking connect call to another service that has not yet come up, AppSwitch blocks the call for a certain time waiting for the target service to become live. Since it is known that the target service is a part of the application and is expected to come up shortly, making the client block rather than returning an error such as ECONNREFUSED prevents the application from failing to start. If the service doesn’t come up within time, an error is returned to the application so that framework-level mechanisms like Kubernetes crash-loop can kick in.

If the client request is marked as non-blocking, AppSwitch handles that by returning EAGAIN to inform the application to retry rather than give up. Once again, that is in-line with the semantics of socket API and prevents failures due to startup races. AppSwitch essentially enables the retry logic already built into applications in support of the BSD socket API to be transparently repurposed for dependency ordering.

Application timeouts

What if the application times out based on its own internal timer? Truth be told, AppSwitch can also fake application’s perception of time if wanted but that would be overstepping and actually unnecessary. Application decides and knows best how long it should wait and it’s not appropriate for AppSwitch to mess with that. Application timeouts are conservatively long and if the target service still hasn’t come up in time, it is unlikely to be a dependency ordering issue. There must be something else going on that should not be masked.

Wildcard service references for sidecar dependency

Service references can be used to address the Istio sidecar dependency issue mentioned earlier. AppSwitch allows the IP:port specified as part of a service reference to be a wildcard. That is, the service reference IP address can be a netmask indicating the IP address range to be captured. If the label selector of the service reference points to the sidecar service, then all outgoing connections of any application for which this service reference is applied, will be transparently redirected to the sidecar. And of course, the service reference remains valid while sidecar is still coming up and the race is removed.

Using service references for sidecar dependency ordering also implicitly redirects application’s connections to the sidecar without requiring iptables and attendant privilege issues. Essentially it works as if the application is directly making connections to the sidecar rather than the target destination, leaving the sidecar in charge of what to do. AppSwitch would interject metadata about the original destination etc. into the data stream of the connection using the proxy protocol that the sidecar could decode before passing the connection through to the application. Some of these details were discussed here. That takes care of outbound connections but what about incoming connections? With all services and their sidecars running under AppSwitch, any incoming connections that would have come from remote nodes would be redirected to their respective remote sidecars. So nothing special to do about incoming connections.

Summary

Dependency ordering is a pesky problem. This is mostly due to lack of access to fine-grain application-level events around inter-service interactions. Addressing this problem would have normally required applications to implement their own internal logic. But AppSwitch makes those internal application events to be instrumented without requiring application changes. AppSwitch then leverages the ubiquitous support for the BSD socket API to sidestep the requirement of ordering dependencies.

Acknowledgements

Thanks to Eric Herness and team for their insights and support with IBM WebSphere and BPM products as we modernized them onto the Kubernetes platform and to Mandar Jog, Martin Taillefer and Shriram Rajagopalan for reviewing early drafts of this blog.

Deploy a Custom Ingress Gateway Using Cert-Manager

Julien Senon — Thu, 10 Jan 2019 00:00:00 +0000

This post provides instructions to manually create a custom ingress gateway with automatic provisioning of certificates based on cert-manager.

The creation of custom ingress gateway could be used in order to have different loadbalancer in order to isolate traffic.

Before you begin

Set up Istio by following the instructions in the Installation guide.
Set up cert-manager with helm chart
We will use demo.mydemo.com for our example, it must be resolved with your DNS

Configuring the custom ingress gateway

Check if cert-manager was installed using Helm with the following command:

$ helm ls

The output should be similar to the example below and show cert-manager with a STATUS of DEPLOYED:

NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
istio 1 Thu Oct 11 13:34:24 2018 DEPLOYED istio-1.0.X 1.0.X istio-system
cert 1 Wed Oct 24 14:08:36 2018 DEPLOYED cert-manager-v0.6.0-dev.2 v0.6.0-dev.2 istio-system

To create the cluster’s issuer, apply the following configuration:

apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-demo
namespace: kube-system
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: <REDACTED>
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-demo
dns01:
# Here we define a list of DNS-01 providers that can solve DNS challenges
providers:
- name: your-dns
route53:
accessKeyID: <REDACTED>
region: eu-central-1
secretAccessKeySecretRef:
name: prod-route53-credentials-secret
key: secret-access-key

If you use the route53 provider, you must provide a secret to perform DNS ACME Validation. To create the secret, apply the following configuration file:
```
apiVersion: v1
kind: Secret
metadata:
name: prod-route53-credentials-secret
type: Opaque
data:
secret-access-key: <REDACTED BASE64>
```

Create your own certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: demo-certificate
namespace: istio-system
spec:
acme:
config:
- dns01:
provider: your-dns
domains:
- '*.mydemo.com'
commonName: '*.mydemo.com'
dnsNames:
- '*.mydemo.com'
issuerRef:
kind: ClusterIssuer
name: letsencrypt-demo
secretName: istio-customingressgateway-certs

Make a note of the value of secretName since a future step requires it.

To scale automatically, declare a new horizontal pod autoscaler with the following configuration:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: my-ingressgateway
namespace: istio-system
spec:
maxReplicas: 5
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: my-ingressgateway
targetCPUUtilizationPercentage: 80
status:
currentCPUUtilizationPercentage: 0
currentReplicas: 1
desiredReplicas: 1

Apply your deployment with declaration provided in the yaml definition

The annotations used, for example aws-load-balancer-type, only apply for AWS.

Create your service:

apiVersion: v1
kind: Service
metadata:
name: my-ingressgateway
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
labels:
app: my-ingressgateway
istio: my-ingressgateway
spec:
type: LoadBalancer
selector:
app: my-ingressgateway
istio: my-ingressgateway
ports:
-
name: http2
nodePort: 32380
port: 80
targetPort: 80
-
name: https
nodePort: 32390
port: 443
-
name: tcp
nodePort: 32400
port: 31400

Create your Istio custom gateway configuration object:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
annotations:
name: istio-custom-gateway
namespace: default
spec:
selector:
istio: my-ingressgateway
servers:
- hosts:
- '*.mydemo.com'
port:
name: http
number: 80
protocol: HTTP
tls:
httpsRedirect: true
- hosts:
- '*.mydemo.com'
port:
name: https
number: 443
protocol: HTTPS
tls:
mode: SIMPLE
privateKey: /etc/istio/ingressgateway-certs/tls.key
serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Link your istio-custom-gateway with your VirtualService:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-virtualservice
spec:
hosts:
- "demo.mydemo.com"
gateways:
- istio-custom-gateway
http:
- route:
- destination:
host: my-demoapp

Correct certificate is returned by the server and it is successfully verified (SSL certificate verify ok is printed):
```
$ curl -v `https://demo.mydemo.com`
Server certificate:
SSL certificate verify ok.
```

Congratulations! You can now use your custom istio-custom-gateway gateway configuration object.

Announcing discuss.istio.io

Thu, 10 Jan 2019 00:00:00 +0000

We in the Istio community have been working to find the right medium for users to engage with other members of the community – to ask questions, to get help from other users, and to engage with developers working on the project.

We’ve tried several different avenues, but each has had some downsides. RocketChat was our most recent endeavor, but the lack of certain features (for example, threading) meant it wasn’t ideal for any longer discussions around a single issue. It also led to a dilemma for some users – when should I email istio-users@googlegroups.com and when should I use RocketChat?

We think we’ve found the right balance of features in a single platform, and we’re happy to announce discuss.istio.io. It’s a full-featured forum where we will have discussions about Istio from here on out. It will allow you to ask a question and get threaded replies! As a real bonus, you can use your GitHub identity.

If you prefer emails, you can configure it to send emails just like Google groups did.

We will be marking our Google groups “read only” so that the content remains, but we ask you to send further questions over to discuss.istio.io. If you have any outstanding questions or discussions in the groups, please move the conversation over.

Happy meshing!

Incremental Istio Part 1, Traffic Management

Sandeep Parikh — Wed, 21 Nov 2018 00:00:00 +0000

Traffic management is one of the critical benefits provided by Istio. At the heart of Istio’s traffic management is the ability to decouple traffic flow and infrastructure scaling. This lets you control your traffic in ways that aren’t possible without a service mesh like Istio.

For example, let’s say you want to execute a canary deployment. With Istio, you can specify that v1 of a service receives 90% of incoming traffic, while v2 of that service only receives 10%. With standard Kubernetes deployments, the only way to achieve this is to manually control the number of available Pods for each version, for example 9 Pods running v1 and 1 Pod running v2. This type of manual control is hard to implement, and over time may have trouble scaling. For more information, check out Canary Deployments using Istio.

The same issue exists when deploying updates to existing services. While you can update deployments with Kubernetes, it requires replacing v1 Pods with v2 Pods. Using Istio, you can deploy v2 of your service and use built-in traffic management mechanisms to shift traffic to your updated services at a network level, then remove the v1 Pods.

In addition to canary deployments and general traffic shifting, Istio also gives you the ability to implement dynamic request routing (based on HTTP headers), failure recovery, retries, circuit breakers, and fault injection. For more information, check out the Traffic Management documentation.

This post walks through a technique that highlights a particularly useful way that you can implement Istio incrementally – in this case, only the traffic management features – without having to individually update each of your Pods.

Setup: why implement Istio traffic management features?

Of course, the first question is: Why would you want to do this?

If you’re part of one of the many organizations out there that have a large cluster with lots of teams deploying, the answer is pretty clear. Let’s say Team A is getting started with Istio and wants to start some canary deployments on Service A, but Team B hasn’t started using Istio, so they don’t have sidecars deployed.

With Istio, Team A can still implement their canaries by having Service B call Service A through Istio’s ingress gateway.

Background: traffic routing in an Istio mesh

But how can you use Istio’s traffic management capabilities without updating each of your applications’ Pods to include the Istio sidecar? Before answering that question, let’s take a quick high-level look at how traffic enters an Istio mesh and how it’s routed.

Pods that are part of the Istio mesh contain a sidecar proxy that is responsible for mediating all inbound and outbound traffic to the Pod. Within an Istio mesh, Pilot is responsible for converting high-level routing rules into configurations and propagating them to the sidecar proxies. That means when services communicate with one another, their routing decisions are determined from the client side.

Let’s say you have two services that are part of the Istio mesh, Service A and Service B. When A wants to communicate with B, the sidecar proxy of Pod A is responsible for directing traffic to Service B. For example, if you wanted to split traffic ⁵⁰⁄₅₀ across Service B v1 and v2, the traffic would flow as follows:

50/50 Traffic Split

If Services A and B are not part of the Istio mesh, there is no sidecar proxy that knows how to route traffic to different versions of Service B. In that case you need to use another approach to get traffic from Service A to Service B, following the ⁵⁰⁄₅₀ rules you’ve setup.

Fortunately, a standard Istio deployment already includes a Gateway that specifically deals with ingress traffic outside of the Istio mesh. This Gateway is used to allow ingress traffic from outside the cluster via an external load balancer, or to allow ingress traffic from within the Kubernetes cluster but outside the service mesh. It can be configured to proxy incoming ingress traffic to the appropriate Pods, even if they don’t have a sidecar proxy. While this approach allows you to leverage Istio’s traffic management features, it does mean that traffic going through the ingress gateway will incur an extra hop.

50/50 Traffic Split using Ingress Gateway

In action: traffic routing with Istio

A simple way to see this type of approach in action is to first setup your Kubernetes environment using the Platform Setup instructions, and then install the minimal Istio profile using Helm, including only the traffic management components (ingress gateway, egress gateway, Pilot). The following example uses Google Kubernetes Engine.

First, setup and configure GKE:

$ gcloud container clusters create istio-inc --zone us-central1-f
$ gcloud container clusters get-credentials istio-inc
$ kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--user=$(gcloud config get-value core/account)

Next, install Helm and generate a minimal Istio install – only traffic management components:

$ helm template install/kubernetes/helm/istio \
--name istio \
--namespace istio-system \
--set security.enabled=false \
--set galley.enabled=false \
--set sidecarInjectorWebhook.enabled=false \
--set mixer.enabled=false \
--set prometheus.enabled=false \
--set pilot.sidecar=false > istio-minimal.yaml

Then create the istio-system namespace and deploy Istio:

$ kubectl create namespace istio-system
$ kubectl apply -f istio-minimal.yaml

Next, deploy the Bookinfo sample without the Istio sidecar containers:

$ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo.yaml@

Now, configure a new Gateway that allows access to the reviews service from outside the Istio mesh, a new VirtualService that splits traffic evenly between v1 and v2 of the reviews service, and a set of new DestinationRule resources that match destination subsets to service versions:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: reviews-gateway
spec:
selector:
istio: ingressgateway # use istio default controller
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- "*"
gateways:
- reviews-gateway
http:
- match:
- uri:
prefix: /reviews
route:
- destination:
host: reviews
subset: v1
weight: 50
- destination:
host: reviews
subset: v2
weight: 50
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
- name: v3
labels:
version: v3
EOF

Finally, deploy a pod that you can use for testing with curl (and without the Istio sidecar container):

$ kubectl apply -f @samples/sleep/sleep.yaml@

Testing your deployment

Now, you can test different behaviors using the curl commands via the sleep Pod.

The first example is to issue requests to the reviews service using standard Kubernetes service DNS behavior (note: jq is used in the examples below to filter the output from curl):

$ export SLEEP_POD=$(kubectl get pod -l app=sleep \
-o jsonpath={.items..metadata.name})
$ for i in `seq 3`; do \
kubectl exec -it $SLEEP_POD curl http://reviews:9080/reviews/0 | \
jq '.reviews|.[]|.rating?'; \
done

{
"stars": 5,
"color": "black"
}
{
"stars": 4,
"color": "black"
}
null
null
{
"stars": 5,
"color": "red"
}
{
"stars": 4,
"color": "red"
}

Notice how we’re getting responses from all three versions of the reviews service (null is from reviews v1 which doesn’t have ratings) and not getting the even split across v1 and v2. This is expected behavior because the curl command is using Kubernetes service load balancing across all three versions of the reviews service. In order to access the reviews ⁵⁰⁄₅₀ split we need to access the service via the ingress Gateway:

$ for i in `seq 4`; do \
kubectl exec -it $SLEEP_POD curl http://istio-ingressgateway.istio-system/reviews/0 | \
jq '.reviews|.[]|.rating?'; \
done

{
"stars": 5,
"color": "black"
}
{
"stars": 4,
"color": "black"
}
null
null
{
"stars": 5,
"color": "black"
}
{
"stars": 4,
"color": "black"
}
null
null

Mission accomplished! This post showed how to deploy a minimal installation of Istio that only contains the traffic management components (Pilot, ingress Gateway), and then use those components to direct traffic to specific versions of the reviews service. And it wasn’t necessary to deploy the Istio sidecar proxy to gain these capabilities, so there was little to no interruption of existing workloads or applications.

Using the built-in ingress gateway (along with some VirtualService and DestinationRule resources) this post showed how you can easily leverage Istio’s traffic management for cluster-external ingress traffic and cluster-internal service-to-service traffic. This technique is a great example of an incremental approach to adopting Istio, and can be especially useful in real-world cases where Pods are owned by different teams or deployed to different namespaces.

Consuming External MongoDB Services

Vadim Eisenberg — Fri, 16 Nov 2018 00:00:00 +0000

In the Consuming External TCP Services blog post, I described how external services can be consumed by in-mesh Istio applications via TCP. In this post, I demonstrate consuming external MongoDB services. You use the Istio Bookinfo sample application, the version in which the book ratings data is persisted in a MongoDB database. You deploy this database outside the cluster and configure the ratings microservice to use it. You will learn multiple options of controlling traffic to external MongoDB services and their pros and cons.

Bookinfo with external ratings database

First, you set up a MongoDB database instance to hold book ratings data outside of your Kubernetes cluster. Then you modify the Bookinfo sample application to use your database.

Setting up the ratings database

For this task you set up an instance of MongoDB. You can use any MongoDB instance; I used Compose for MongoDB.

Set an environment variable for the password of your admin user. To prevent the password from being preserved in the Bash history, remove the command from the history immediately after running the command, using history -d.
```
$ export MONGO_ADMIN_PASSWORD=<your MongoDB admin password>
```
Set an environment variable for the password of the new user you will create, namely bookinfo. Remove the command from the history using history -d.
```
$ export BOOKINFO_PASSWORD=<password>
```
Set environment variables for your MongoDB service, MONGODB_HOST and MONGODB_PORT.

Create the bookinfo user:

$ cat <<EOF | mongo --ssl --sslAllowInvalidCertificates $MONGODB_HOST:$MONGODB_PORT -u admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin
use test
db.createUser(
{
user: "bookinfo",
pwd: "$BOOKINFO_PASSWORD",
roles: [ "read"]
}
);
EOF

Create a collection to hold ratings. The following command sets both ratings to be equal 1 to provide a visual clue when your database is used by the Bookinfo ratings service (the default Bookinfo ratings are 4 and 5).

$ cat <<EOF | mongo --ssl --sslAllowInvalidCertificates $MONGODB_HOST:$MONGODB_PORT -u admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin
use test
db.createCollection("ratings");
db.ratings.insert(
[{rating: 1},
{rating: 1}]
);
EOF

Check that bookinfo user can get ratings:

$ cat <<EOF | mongo --ssl --sslAllowInvalidCertificates $MONGODB_HOST:$MONGODB_PORT -u bookinfo -p $BOOKINFO_PASSWORD --authenticationDatabase test
use test
db.ratings.find({});
EOF

The output should be similar to:

MongoDB server version: 3.4.10
switched to db test
{ "_id" : ObjectId("5b7c29efd7596e65b6ed2572"), "rating" : 1 }
{ "_id" : ObjectId("5b7c29efd7596e65b6ed2573"), "rating" : 1 }
bye

Initial setting of Bookinfo application

To demonstrate the scenario of using an external database, you start with a Kubernetes cluster with Istio installed. Then you deploy the Istio Bookinfo sample application, apply the default destination rules, and change Istio to the blocking-egress-by-default policy.

This application uses the ratings microservice to fetch book ratings, a number between 1 and 5. The ratings are displayed as stars for each review. There are several versions of the ratings microservice. You will deploy the version that uses MongoDB as the ratings database in the next subsection.

The example commands in this blog post work with Istio 1.0.

As a reminder, here is the end-to-end architecture of the application from the Bookinfo sample application.

The original Bookinfo application

Use the external database in Bookinfo application

Deploy the spec of the ratings microservice that uses a MongoDB database (ratings v2):

$ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml@
serviceaccount "bookinfo-ratings-v2" created
deployment "ratings-v2" created

Update the MONGO_DB_URL environment variable to the value of your MongoDB:

$ kubectl set env deployment/ratings-v2 "MONGO_DB_URL=mongodb://bookinfo:$BOOKINFO_PASSWORD@$MONGODB_HOST:$MONGODB_PORT/test?authSource=test&ssl=true"
deployment.extensions/ratings-v2 env updated

Route all the traffic destined to the reviews service to its v3 version. You do this to ensure that the reviews service always calls the ratings service. In addition, route all the traffic destined to the ratings service to ratings v2 that uses your database.

Specify the routing for both services above by adding two virtual services. These virtual services are specified in samples/bookinfo/networking/virtual-service-ratings-mongodb.yaml of an Istio release archive. Important: make sure you applied the default destination rules before running the following command.
Zip
```
$ kubectl apply -f @samples/bookinfo/networking/virtual-service-ratings-db.yaml@
```

The updated architecture appears below. Note that the blue arrows inside the mesh mark the traffic configured according to the virtual services we added. According to the virtual services, the traffic is sent to reviews v3 and ratings v2.

The Bookinfo application with ratings v2 and an external MongoDB database

Note that the MongoDB database is outside the Istio service mesh, or more precisely outside the Kubernetes cluster. The boundary of the service mesh is marked by a dashed line.

Access the webpage

Access the webpage of the application, after determining the ingress IP and port.

Since you did not configure the egress traffic control yet, the access to the MongoDB service is blocked by Istio. This is why instead of the rating stars, the message “Ratings service is currently unavailable” is currently displayed below each review:

The Ratings service error messages

In the following sections you will configure egress access to the external MongoDB service, using different options for egress control in Istio.

Egress control for TCP

Since MongoDB Wire Protocol runs on top of TCP, you can control the egress traffic to your MongoDB as traffic to any other external TCP service. To control TCP traffic, a block of IPs in the CIDR notation that includes the IP address of your MongoDB host must be specified. The caveat here is that sometimes the IP of the MongoDB host is not stable or known in advance.

In the cases when the IP of the MongoDB host is not stable, the egress traffic can either be controlled as TLS traffic, or the traffic can be routed directly, bypassing the Istio sidecar proxies.

Get the IP address of your MongoDB database instance. As an option, you can use the host command:

$ export MONGODB_IP=$(host $MONGODB_HOST | grep " has address " | cut -d" " -f4)

Control TCP egress traffic without a gateway

In case you do not need to direct the traffic through an egress gateway, for example if you do not have a requirement that all the traffic that exists your mesh must exit through the gateway, follow the instructions in this section. Alternatively, if you do want to direct your traffic through an egress gateway, proceed to Direct TCP egress traffic through an egress gateway.

Define a TCP mesh-external service entry:
```
$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: mongo
spec:
hosts:
- my-mongo.tcp.svc
addresses:
- $MONGODB_IP/32
ports:
- number: $MONGODB_PORT
name: tcp
protocol: TCP
location: MESH_EXTERNAL
resolution: STATIC
endpoints:
- address: $MONGODB_IP
EOF
```
Note that the protocol TCP is specified instead of MONGO due to the fact that the traffic can be encrypted in case the MongoDB protocol runs on top of TLS. If the traffic is encrypted, the encrypted MongoDB protocol cannot be parsed by the Istio proxy.

If you know that the plain MongoDB protocol is used, without encryption, you can specify the protocol as MONGO and let the Istio proxy produce MongoDB related statistics. Also note that when the protocol TCP is specified, the configuration is not specific for MongoDB, but is the same for any other database with the protocol on top of TCP.

Note that the host of your MongoDB is not used in TCP routing, so you can use any host, for example my-mongo.tcp.svc. Notice the STATIC resolution and the endpoint with the IP of your MongoDB service. Once you define such an endpoint, you can access MongoDB services that do not have a domain name.
Refresh the web page of the application. Now the application should display the ratings without error:

Book Ratings Displayed Correctly

Note that you see a one-star rating for both displayed reviews, as expected. You set the ratings to be one star to provide yourself with a visual clue that your external database is indeed being used.
If you want to direct the traffic through an egress gateway, proceed to the next section. Otherwise, perform cleanup.

Direct TCP Egress traffic through an egress gateway

In this section you handle the case when you need to direct the traffic through an egress gateway. The sidecar proxy routes TCP connections from the MongoDB client to the egress gateway, by matching the IP of the MongoDB host (a CIDR block of length 32). The egress gateway forwards the traffic to the MongoDB host, by its hostname.

Deploy Istio egress gateway.
If you did not perform the steps in the previous section, perform them now.
You may want to enable mutual TLS Authentication between the sidecar proxies of your MongoDB clients and the egress gateway to let the egress gateway monitor the identity of the source pods and to enable Mixer policy enforcement based on that identity. By enabling mutual TLS you also encrypt the traffic. If you do not want to enable mutual TLS, proceed to the Mutual TLS between the sidecar proxies and the egress gateway section. Otherwise, proceed to the following section.

Configure TCP traffic from sidecars to the egress gateway

Define the EGRESS_GATEWAY_MONGODB_PORT environment variable to hold some port for directing traffic through the egress gateway, e.g. 7777. You must select a port that is not used for any other service in the mesh.
```
$ export EGRESS_GATEWAY_MONGODB_PORT=7777
```

Add the selected port to the istio-egressgateway service. You should use the same values you used for installing Istio, in particular you have to specify all the ports of the istio-egressgateway service that you previously configured.

$ helm template install/kubernetes/helm/istio/ --name istio-egressgateway --namespace istio-system -x charts/gateways/templates/deployment.yaml -x charts/gateways/templates/service.yaml --set gateways.istio-ingressgateway.enabled=false --set gateways.istio-egressgateway.enabled=true --set gateways.istio-egressgateway.ports[0].port=80 --set gateways.istio-egressgateway.ports[0].name=http --set gateways.istio-egressgateway.ports[1].port=443 --set gateways.istio-egressgateway.ports[1].name=https --set gateways.istio-egressgateway.ports[2].port=$EGRESS_GATEWAY_MONGODB_PORT --set gateways.istio-egressgateway.ports[2].name=mongo | kubectl apply -f -

Check that the istio-egressgateway service indeed has the selected port:

$ kubectl get svc istio-egressgateway -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-egressgateway ClusterIP 172.21.202.204 <none> 80/TCP,443/TCP,7777/TCP 34d

Disable mutual TLS authentication for the istio-egressgateway service:

$ kubectl apply -f - <<EOF
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: istio-egressgateway
namespace: istio-system
spec:
targets:
- name: istio-egressgateway
EOF

Create an egress Gateway for your MongoDB service, and destination rules and a virtual service to direct the traffic through the egress gateway and from the egress gateway to the external service.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: $EGRESS_GATEWAY_MONGODB_PORT
name: tcp
protocol: TCP
hosts:
- my-mongo.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-mongo
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: mongo
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: mongo
spec:
host: my-mongo.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-mongo-through-egress-gateway
spec:
hosts:
- my-mongo.tcp.svc
gateways:
- mesh
- istio-egressgateway
tcp:
- match:
- gateways:
- mesh
destinationSubnets:
- $MONGODB_IP/32
port: $MONGODB_PORT
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: mongo
port:
number: $EGRESS_GATEWAY_MONGODB_PORT
- match:
- gateways:
- istio-egressgateway
port: $EGRESS_GATEWAY_MONGODB_PORT
route:
- destination:
host: my-mongo.tcp.svc
port:
number: $MONGODB_PORT
weight: 100
EOF

Verify that egress traffic is directed through the egress gateway.

Mutual TLS between the sidecar proxies and the egress gateway

Delete the previous configuration:

$ kubectl delete gateway istio-egressgateway --ignore-not-found=true
$ kubectl delete virtualservice direct-mongo-through-egress-gateway --ignore-not-found=true
$ kubectl delete destinationrule egressgateway-for-mongo mongo --ignore-not-found=true
$ kubectl delete policy istio-egressgateway -n istio-system --ignore-not-found=true

Enforce mutual TLS authentication for the istio-egressgateway service:

$ kubectl apply -f - <<EOF
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: istio-egressgateway
namespace: istio-system
spec:
targets:
- name: istio-egressgateway
peers:
- mtls: {}
EOF

Create an egress Gateway for your MongoDB service, and destination rules and a virtual service to direct the traffic through the egress gateway and from the egress gateway to the external service.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- my-mongo.tcp.svc
tls:
mode: MUTUAL
serverCertificate: /etc/certs/cert-chain.pem
privateKey: /etc/certs/key.pem
caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-mongo
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: mongo
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: ISTIO_MUTUAL
sni: my-mongo.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: mongo
spec:
host: my-mongo.tcp.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-mongo-through-egress-gateway
spec:
hosts:
- my-mongo.tcp.svc
gateways:
- mesh
- istio-egressgateway
tcp:
- match:
- gateways:
- mesh
destinationSubnets:
- $MONGODB_IP/32
port: $MONGODB_PORT
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: mongo
port:
number: 443
- match:
- gateways:
- istio-egressgateway
port: 443
route:
- destination:
host: my-mongo.tcp.svc
port:
number: $MONGODB_PORT
weight: 100
EOF

Proceed to the next section.

Verify that egress traffic is directed through the egress gateway

Refresh the web page of the application again and verify that the ratings are still displayed correctly.
Enable Envoy’s access logging

Check the log of the egress gateway’s Envoy and see a line that corresponds to your requests to the MongoDB service. If Istio is deployed in the istio-system namespace, the command to print the log is:

$ kubectl logs -l istio=egressgateway -n istio-system
[2019-04-14T06:12:07.636Z] "- - -" 0 - "-" 1591 4393 94 - "-" "-" "-" "-" "<Your MongoDB IP>:<your MongoDB port>" outbound|<your MongoDB port>||my-mongo.tcp.svc 172.30.146.119:59924 172.30.146.119:443 172.30.230.1:59206 -

Cleanup of TCP egress traffic control

$ kubectl delete serviceentry mongo
$ kubectl delete gateway istio-egressgateway --ignore-not-found=true
$ kubectl delete virtualservice direct-mongo-through-egress-gateway --ignore-not-found=true
$ kubectl delete destinationrule egressgateway-for-mongo mongo --ignore-not-found=true
$ kubectl delete policy istio-egressgateway -n istio-system --ignore-not-found=true

Egress control for TLS

In the real life, most of the communication to the external services must be encrypted and the MongoDB protocol runs on top of TLS. Also, the TLS clients usually send Server Name Indication, SNI, as part of their handshake. If your MongoDB server runs TLS and your MongoDB client sends SNI as part of the handshake, you can control your MongoDB egress traffic as any other TLS-with-SNI traffic. With TLS and SNI, you do not need to specify the IP addresses of your MongoDB servers. You specify their host names instead, which is more convenient since you do not have to rely on the stability of the IP addresses. You can also specify wildcards as a prefix of the host names, for example allowing access to any server from the *.com domain.

To check if your MongoDB server supports TLS, run:

$ openssl s_client -connect $MONGODB_HOST:$MONGODB_PORT -servername $MONGODB_HOST

If the command above prints a certificate returned by the server, the server supports TLS. If not, you have to control your MongoDB egress traffic on the TCP level, as described in the previous sections.

Control TLS egress traffic without a gateway

In case you do not need an egress gateway, follow the instructions in this section. If you want to direct your traffic through an egress gateway, proceed to Direct TCP Egress traffic through an egress gateway.

Create a ServiceEntry for the MongoDB service:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: mongo
spec:
hosts:
- $MONGODB_HOST
ports:
- number: $MONGODB_PORT
name: tls
protocol: TLS
resolution: DNS
EOF

Refresh the web page of the application. The application should display the ratings without error.

Cleanup of the egress configuration for TLS

$ kubectl delete serviceentry mongo

Direct TLS Egress traffic through an egress gateway

In this section you handle the case when you need to direct the traffic through an egress gateway. The sidecar proxy routes TLS connections from the MongoDB client to the egress gateway, by matching the SNI of the MongoDB host. The egress gateway forwards the traffic to the MongoDB host. Note that the sidecar proxy rewrites the destination port to be 443. The egress gateway accepts the MongoDB traffic on the port 443, matches the MongoDB host by SNI, and rewrites the port again to be the port of the MongoDB server.

Deploy Istio egress gateway.

Create a ServiceEntry for the MongoDB service:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: mongo
spec:
hosts:
- $MONGODB_HOST
ports:
- number: $MONGODB_PORT
name: tls
protocol: TLS
- number: 443
name: tls-port-for-egress-gateway
protocol: TLS
resolution: DNS
location: MESH_EXTERNAL
EOF

Refresh the web page of the application and verify that the ratings are displayed correctly.

Create an egress Gateway for your MongoDB service, and destination rules and virtual services to direct the traffic through the egress gateway and from the egress gateway to the external service.

If you want to enable mutual TLS Authentication between the sidecar proxies of your application pods and the egress gateway, use the following command. (You may want to enable mutual TLS to let the egress gateway monitor the identity of the source pods and to enable Mixer policy enforcement based on that identity.)

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- $MONGODB_HOST
tls:
mode: MUTUAL
serverCertificate: /etc/certs/cert-chain.pem
privateKey: /etc/certs/key.pem
caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-mongo
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: mongo
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: ISTIO_MUTUAL
sni: $MONGODB_HOST
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-mongo-through-egress-gateway
spec:
hosts:
- $MONGODB_HOST
gateways:
- mesh
- istio-egressgateway
tls:
- match:
- gateways:
- mesh
port: $MONGODB_PORT
sni_hosts:
- $MONGODB_HOST
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: mongo
port:
number: 443
tcp:
- match:
- gateways:
- istio-egressgateway
port: 443
route:
- destination:
host: $MONGODB_HOST
port:
number: $MONGODB_PORT
weight: 100
EOF

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- $MONGODB_HOST
tls:
mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-mongo
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: mongo
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-mongo-through-egress-gateway
spec:
hosts:
- $MONGODB_HOST
gateways:
- mesh
- istio-egressgateway
tls:
- match:
- gateways:
- mesh
port: $MONGODB_PORT
sni_hosts:
- $MONGODB_HOST
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: mongo
port:
number: 443
- match:
- gateways:
- istio-egressgateway
port: 443
sni_hosts:
- $MONGODB_HOST
route:
- destination:
host: $MONGODB_HOST
port:
number: $MONGODB_PORT
weight: 100
EOF

Verify that the traffic is directed though the egress gateway

Cleanup directing TLS egress traffic through an egress gateway

$ kubectl delete serviceentry mongo
$ kubectl delete gateway istio-egressgateway
$ kubectl delete virtualservice direct-mongo-through-egress-gateway
$ kubectl delete destinationrule egressgateway-for-mongo

Enable MongoDB TLS egress traffic to arbitrary wildcarded domains

Sometimes you want to configure egress traffic to multiple hostnames from the same domain, for example traffic to all MongoDB services from *.<your company domain>.com. You do not want to create multiple configuration items, one for each and every MongoDB service in your company. To configure access to all the external services from the same domain by a single configuration, you use wildcarded hosts.

In this section you configure egress traffic for a wildcarded domain. I used a MongoDB instance at composedb.com domain, so configuring egress traffic for *.com worked for me (I could have used *.composedb.com as well). You can pick a wildcarded domain according to your MongoDB host.

To configure egress gateway traffic for a wildcarded domain, you will first need to deploy a custom egress gateway with an additional SNI proxy. This is needed due to current limitations of Envoy, the proxy used by the standard Istio egress gateway.

Prepare a new egress gateway with an SNI proxy

In this subsection you deploy an egress gateway with an SNI proxy, in addition to the standard Istio Envoy proxy. You can use any SNI proxy that is capable of routing traffic according to arbitrary, not-preconfigured SNI values; we used Nginx to achieve this functionality.

Create a configuration file for the Nginx SNI proxy. You may want to edit the file to specify additional Nginx settings, if required.

$ cat <<EOF > ./sni-proxy.conf
user www-data;
events {
}
stream {
log_format log_stream '\$remote_addr [\$time_local] \$protocol [\$ssl_preread_server_name]'
'\$status \$bytes_sent \$bytes_received \$session_time';
access_log /var/log/nginx/access.log log_stream;
error_log /var/log/nginx/error.log;
# tcp forward proxy by SNI
server {
resolver 8.8.8.8 ipv6=off;
listen 127.0.0.1:$MONGODB_PORT;
proxy_pass \$ssl_preread_server_name:$MONGODB_PORT;
ssl_preread on;
}
}
EOF

Create a Kubernetes ConfigMap to hold the configuration of the Nginx SNI proxy:

$ kubectl create configmap egress-sni-proxy-configmap -n istio-system --from-file=nginx.conf=./sni-proxy.conf

The following command will generate istio-egressgateway-with-sni-proxy.yaml to edit and deploy.

$ cat <<EOF | helm template install/kubernetes/helm/istio/ --name istio-egressgateway-with-sni-proxy --namespace istio-system -x charts/gateways/templates/deployment.yaml -x charts/gateways/templates/service.yaml -x charts/gateways/templates/serviceaccount.yaml -x charts/gateways/templates/autoscale.yaml -x charts/gateways/templates/role.yaml -x charts/gateways/templates/rolebindings.yaml --set global.mtls.enabled=true --set global.istioNamespace=istio-system -f - > ./istio-egressgateway-with-sni-proxy.yaml
gateways:
enabled: true
istio-ingressgateway:
enabled: false
istio-egressgateway:
enabled: false
istio-egressgateway-with-sni-proxy:
enabled: true
labels:
app: istio-egressgateway-with-sni-proxy
istio: egressgateway-with-sni-proxy
replicaCount: 1
autoscaleMin: 1
autoscaleMax: 5
cpu:
targetAverageUtilization: 80
serviceAnnotations: {}
type: ClusterIP
ports:
- port: 443
name: https
secretVolumes:
- name: egressgateway-certs
secretName: istio-egressgateway-certs
mountPath: /etc/istio/egressgateway-certs
- name: egressgateway-ca-certs
secretName: istio-egressgateway-ca-certs
mountPath: /etc/istio/egressgateway-ca-certs
configVolumes:
- name: sni-proxy-config
configMapName: egress-sni-proxy-configmap
additionalContainers:
- name: sni-proxy
image: nginx
volumeMounts:
- name: sni-proxy-config
mountPath: /etc/nginx
readOnly: true
EOF

Deploy the new egress gateway:

$ kubectl apply -f ./istio-egressgateway-with-sni-proxy.yaml
serviceaccount "istio-egressgateway-with-sni-proxy-service-account" created
role "istio-egressgateway-with-sni-proxy-istio-system" created
rolebinding "istio-egressgateway-with-sni-proxy-istio-system" created
service "istio-egressgateway-with-sni-proxy" created
deployment "istio-egressgateway-with-sni-proxy" created
horizontalpodautoscaler "istio-egressgateway-with-sni-proxy" created

Verify that the new egress gateway is running. Note that the pod has two containers (one is the Envoy proxy and the second one is the SNI proxy).

$ kubectl get pod -l istio=egressgateway-with-sni-proxy -n istio-system
NAME READY STATUS RESTARTS AGE
istio-egressgateway-with-sni-proxy-79f6744569-pf9t2 2/2 Running 0 17s

Create a service entry with a static address equal to 127.0.0.1 (localhost), and disable mutual TLS on the traffic directed to the new service entry:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: sni-proxy
spec:
hosts:
- sni-proxy.local
location: MESH_EXTERNAL
ports:
- number: $MONGODB_PORT
name: tcp
protocol: TCP
resolution: STATIC
endpoints:
- address: 127.0.0.1
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: disable-mtls-for-sni-proxy
spec:
host: sni-proxy.local
trafficPolicy:
tls:
mode: DISABLE
EOF

Configure access to `*.com` using the new egress gateway

Define a ServiceEntry for *.com:

$ cat <<EOF | kubectl create -f -
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: mongo
spec:
hosts:
- "*.com"
ports:
- number: 443
name: tls
protocol: TLS
- number: $MONGODB_PORT
name: tls-mongodb
protocol: TLS
location: MESH_EXTERNAL
EOF

Create an egress Gateway for *.com, port 443, protocol TLS, a destination rule to set the SNI for the gateway, and Envoy filters to prevent tampering with SNI by a malicious application (the filters verify that the SNI issued by the application is the SNI reported to Mixer).

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway-with-sni-proxy
spec:
selector:
istio: egressgateway-with-sni-proxy
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- "*.com"
tls:
mode: MUTUAL
serverCertificate: /etc/certs/cert-chain.pem
privateKey: /etc/certs/key.pem
caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: mtls-for-egress-gateway
spec:
host: istio-egressgateway-with-sni-proxy.istio-system.svc.cluster.local
subsets:
- name: mongo
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: ISTIO_MUTUAL
---
# The following filter is used to forward the original SNI (sent by the application) as the SNI of the mutual TLS
# connection.
# The forwarded SNI will be reported to Mixer so that policies will be enforced based on the original SNI value.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: forward-downstream-sni
spec:
filters:
- listenerMatch:
portNumber: $MONGODB_PORT
listenerType: SIDECAR_OUTBOUND
filterName: forward_downstream_sni
filterType: NETWORK
filterConfig: {}
---
# The following filter verifies that the SNI of the mutual TLS connection (the SNI reported to Mixer) is
# identical to the original SNI issued by the application (the SNI used for routing by the SNI proxy).
# The filter prevents Mixer from being deceived by a malicious application: routing to one SNI while
# reporting some other value of SNI. If the original SNI does not match the SNI of the mutual TLS connection, the
# filter will block the connection to the external service.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: egress-gateway-sni-verifier
spec:
workloadLabels:
app: istio-egressgateway-with-sni-proxy
filters:
- listenerMatch:
portNumber: 443
listenerType: GATEWAY
filterName: sni_verifier
filterType: NETWORK
filterConfig: {}
EOF

Route the traffic destined for *.com to the egress gateway and from the egress gateway to the SNI proxy.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-mongo-through-egress-gateway
spec:
hosts:
- "*.com"
gateways:
- mesh
- istio-egressgateway-with-sni-proxy
tls:
- match:
- gateways:
- mesh
port: $MONGODB_PORT
sni_hosts:
- "*.com"
route:
- destination:
host: istio-egressgateway-with-sni-proxy.istio-system.svc.cluster.local
subset: mongo
port:
number: 443
weight: 100
tcp:
- match:
- gateways:
- istio-egressgateway-with-sni-proxy
port: 443
route:
- destination:
host: sni-proxy.local
port:
number: $MONGODB_PORT
weight: 100
EOF

Refresh the web page of the application again and verify that the ratings are still displayed correctly.
Enable Envoy’s access logging

Check the log of the egress gateway’s Envoy proxy. If Istio is deployed in the istio-system namespace, the command to print the log is:

$ kubectl logs -l istio=egressgateway-with-sni-proxy -c istio-proxy -n istio-system

You should see lines similar to the following:

[2019-01-02T17:22:04.602Z] "- - -" 0 - 768 1863 88 - "-" "-" "-" "-" "127.0.0.1:28543" outbound|28543||sni-proxy.local 127.0.0.1:49976 172.30.146.115:443 172.30.146.118:58510 <your MongoDB host>
[2019-01-02T17:22:04.713Z] "- - -" 0 - 1534 2590 85 - "-" "-" "-" "-" "127.0.0.1:28543" outbound|28543||sni-proxy.local 127.0.0.1:49988 172.30.146.115:443 172.30.146.118:58522 <your MongoDB host>

Check the logs of the SNI proxy. If Istio is deployed in the istio-system namespace, the command to print the log is:

$ kubectl logs -l istio=egressgateway-with-sni-proxy -n istio-system -c sni-proxy
127.0.0.1 [23/Aug/2018:03:28:18 +0000] TCP [<your MongoDB host>]200 1863 482 0.089
127.0.0.1 [23/Aug/2018:03:28:18 +0000] TCP [<your MongoDB host>]200 2590 1248 0.095

Understanding what happened

In this section you configured egress traffic to your MongoDB host using a wildcarded domain. While for a single MongoDB host there is no gain in using wildcarded domains (an exact hostname can be specified), it could be beneficial for cases when the applications in the cluster access multiple MongoDB hosts that match some wildcarded domain. For example, if the applications access mongodb1.composedb.com, mongodb2.composedb.com and mongodb3.composedb.com, the egress traffic can be configured by a single configuration for the wildcarded domain *.composedb.com.

I will leave it as an exercise for the reader to verify that no additional Istio configuration is required when you configure an app to use another instance of MongoDB with a hostname that matches the wildcarded domain used in this section.

Cleanup of configuration for MongoDB TLS egress traffic to arbitrary wildcarded domains

Delete the configuration items for *.com:

$ kubectl delete serviceentry mongo
$ kubectl delete gateway istio-egressgateway-with-sni-proxy
$ kubectl delete virtualservice direct-mongo-through-egress-gateway
$ kubectl delete destinationrule mtls-for-egress-gateway
$ kubectl delete envoyfilter forward-downstream-sni egress-gateway-sni-verifier

Delete the configuration items for the egressgateway-with-sni-proxy deployment:

$ kubectl delete serviceentry sni-proxy
$ kubectl delete destinationrule disable-mtls-for-sni-proxy
$ kubectl delete -f ./istio-egressgateway-with-sni-proxy.yaml
$ kubectl delete configmap egress-sni-proxy-configmap -n istio-system

Remove the configuration files you created:

$ rm ./istio-egressgateway-with-sni-proxy.yaml
$ rm ./nginx-sni-proxy.conf

Cleanup

Drop the bookinfo user:

$ cat <<EOF | mongo --ssl --sslAllowInvalidCertificates $MONGODB_HOST:$MONGODB_PORT -u admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin
use test
db.dropUser("bookinfo");
EOF

Drop the ratings collection:

$ cat <<EOF | mongo --ssl --sslAllowInvalidCertificates $MONGODB_HOST:$MONGODB_PORT -u admin -p $MONGO_ADMIN_PASSWORD --authenticationDatabase admin
use test
db.ratings.drop();
EOF

Unset the environment variables you used:

$ unset MONGO_ADMIN_PASSWORD BOOKINFO_PASSWORD MONGODB_HOST MONGODB_PORT MONGODB_IP

Remove the virtual services:

$ kubectl delete -f @samples/bookinfo/networking/virtual-service-ratings-db.yaml@
Deleted config: virtual-service/default/reviews
Deleted config: virtual-service/default/ratings

Undeploy ratings v2-mongodb:

$ kubectl delete -f @samples/bookinfo/platform/kube/bookinfo-ratings-v2.yaml@
deployment "ratings-v2" deleted

Conclusion

In this blog post I demonstrated various options for MongoDB egress traffic control. You can control the MongoDB egress traffic on a TCP or TLS level where applicable. In both TCP and TLS cases, you can direct the traffic from the sidecar proxies directly to the external MongoDB host, or direct the traffic through an egress gateway, according to your organization’s security requirements. In the latter case, you can also decide to apply or disable mutual TLS authentication between the sidecar proxies and the egress gateway. If you want to control MongoDB egress traffic on the TLS level by specifying wildcarded domains like *.com and you need to direct the traffic through the egress gateway, you must deploy a custom egress gateway with an SNI proxy.

Note that the configuration and considerations described in this blog post for MongoDB are rather the same for other non-HTTP protocols on top of TCP/TLS.

All Day Istio Twitch Stream

Spencer Krum, IBM — Fri, 03 Aug 2018 00:00:00 +0000

To celebrate the 1.0 release and to promote the software to a wider audience, the Istio community is hosting an all day live stream on Twitch on August 17th.

What is Twitch?

Twitch is a popular video gaming live streaming platform and recently has seen a lot of coding content showing up. The IBM Advocates have been doing live coding and presentations there and it’s been fun. While mostly used for gaming content, there is a growing community sharing and watching programming content on the site.

What does this have to do with Istio?

The stream is going to be a full day of Istio content. Hopefully we’ll have a good mix of deep technical content, beginner content and line-of-business content for our audience. We’ll have developers, users, and evangelists on throughout the day to share their demos and stories. Expect live coding, q and a, and some surprises. We have stellar guests lined up from IBM, Google, Datadog, Pivotal, and more!

Recordings

Recordings are available here.

Schedule

All times are PDT.

Time	Speaker	Affiliation
10:00 - 10:30	`Spencer Krum + Lisa-Marie Namphy`	`IBM / Portworx`
10:30 - 11:00	`Lin Sun / Spencer Krum / Sven Mawson`	`IBM / Google`
11:00 - 11:10	`Lin Sun / Spencer Krum`	`IBM`
11:10 - 11:30	`Jason Yee / Ilan Rabinovich`	`Datadog`
11:30 - 11:50	`April Nassl`	`Google`
11:50 - 12:10	`Spike Curtis`	`Tigera`
12:10 - 12:30	`Shannon Coen`	`Pivotal`
12:30 - 1:00	`Matt Klein`	`Lyft`
1:00 - 1:20	`Zach Jory`	`F5/Aspen Mesh`
1:20 - 1:40	`Dan Ciruli`	`Google`
1:40 - 2:00	`Isaiah Snell-Feikema` / `Greg Hanson`	`IBM`
2:00 - 2:20	`Zach Butcher`	`Tetrate`
2:20 - 2:40	`Ray Hudaihed`	`American Airlines`
2:40 - 3:00	`Christian Posta`	`Red Hat`
3:00 - 3:20	`Google/IBM China`	`Google / IBM`
3:20 - 3:40	`Colby Dyess`	`Tuffin`
3:40 - 4:00	`Rohit Agarwalla`	`Cisco`

Istio a Game Changer for HP's FitStation Platform

Tue, 31 Jul 2018 00:00:00 +0000

The FitStation team at HP strongly believes in the future of Kubernetes, BPF and service-mesh as the next standards in cloud infrastructure. We are also very happy to see Istio coming to its official Istio 1.0 release – thanks to the joint collaboration that started at Google, IBM and Lyft beginning in May 2017.

Throughout the development of FitStation’s large scale and progressive cloud platform, Istio, Cilium and Kubernetes technologies have delivered a multitude of opportunities to make our systems more robust and scalable. Istio was a game changer in creating reliable and dynamic network communication.

FitStation powered by HP is a technology platform that captures 3D biometric data to design personalized footwear to perfectly fit individual foot size and shape as well as gait profile. It uses 3D scanning, pressure sensing, 3D printing and variable density injection molding to create unique footwear. Footwear brands such as Brooks, Steitz Secura or Superfeet are connecting to FitStation to build their next generation of high performance sports, professional and medical shoes.

FitStation is built on the promise of ultimate security and privacy for users’ biometric data. ISTIO is the cornerstone to make that possible for data-at-flight within our cloud. By managing these aspects at the infrastructure level, we focused on solving business problems instead of spending time on individual implementations of secure service communication. Using Istio allowed us to dramatically reduce the complexity of maintaining a multitude of libraries and services to provide secure service communication.

As a bonus benefit of Istio 1.0, we gained network visibility, metrics and tracing out of the box. This radically improved decision-making and response quality for our development and devops teams. The team got in-depth insight in the network communication across the entire platform, both for new as well as legacy applications. The integration of Cilium with Envoy delivered a remarkable performance benefit on Istio service mesh communication, combined with a fine-grained kernel driven L7 network security layer. This was due to the powers of BPF brought to Istio by Cilium. We believe this will drive the future of Linux kernel security.

It has been very exciting to follow Istio’s growth. We have been able to see clear improvements of performance and stability over the different development versions. The improvements between version 0.7 and 0.8 made our teams feel comfortable with version 1.0, we can state that Istio is now ready for real production usage.

We are looking forward to the promising roadmaps of Istio, Envoy, Cilium and CNCF.

Delayering Istio with AppSwitch

Dinesh Subhraveti (AppOrbit and Columbia University) — Mon, 30 Jul 2018 00:00:00 +0000

The sidecar proxy approach enables a lot of awesomeness. Squarely in the datapath between microservices, the sidecar can precisely tell what the application is trying to do. It can monitor and instrument protocol traffic, not in the bowels of the networking layers but at the application level, to enable deep visibility, access controls and traffic management.

If we look closely however, there are many intermediate layers that the data has to pass through before the high-value analysis of application-traffic can be performed. Most of those layers are part of the base plumbing infrastructure that are there just to push the data along. In doing so, they add latency to communication and complexity to the overall system.

Over the years, there has been much collective effort in implementing aggressive fine-grained optimizations within the layers of the network datapath. Each iteration may shave another few microseconds. But then the true necessity of those layers itself has not been questioned.

Don’t optimize layers, remove them

In my belief, optimizing something is a poor fallback to removing its requirement altogether. That was the goal of my initial work (broken link: https://apporbit.com/a-brief-history-of-containers-from-reality-to-hype/) on OS-level virtualization that led to Linux containers which effectively removed virtual machines by running applications directly on the host operating system without requiring an intermediate guest. For a long time the industry was fighting the wrong battle distracted by optimizing VMs rather than removing the additional layer altogether.

I see the same pattern repeat itself with the connectivity of microservices, and networking in general. The network has been going through the changes that physical servers have gone through a decade earlier. New set of layers and constructs are being introduced. They are being baked deep into the protocol stack and even silicon without adequately considering low-touch alternatives. Perhaps there is a way to remove those additional layers altogether.

I have been thinking about these problems for some time and believe that an approach similar in concept to containers can be applied to the network stack that would fundamentally simplify how application endpoints are connected across the complexity of many intermediate layers. I have reapplied the same principles from the original work on containers to create AppSwitch. Similar to the way containers provide an interface that applications can directly consume, AppSwitch plugs directly into well-defined and ubiquitous network API that applications currently use and directly connects application clients to appropriate servers, skipping all intermediate layers. In the end, that’s what networking is all about.

Before going into the details of how AppSwitch promises to remove unnecessary layers from the Istio stack, let me give a very brief introduction to its architecture. Further details are available at the documentation page.

AppSwitch

Not unlike the container runtime, AppSwitch consists of a client and a daemon that speak over HTTP via a REST API. Both the client and the daemon are built as one self-contained binary, ax. The client transparently plugs into the application and tracks its system calls related to network connectivity and notifies the daemon about their occurrences. As an example, let’s say an application makes the connect(2) system call to the service IP of a Kubernetes service. The AppSwitch client intercepts the connect call, nullifies it and notifies the daemon about its occurrence along with some context that includes the system call arguments. The daemon would then handle the system call, potentially by directly connecting to the Pod IP of the upstream server on behalf of the application.

It is important to note that no data is forwarded between AppSwitch client and daemon. They are designed to exchange file descriptors (FDs) over a Unix domain socket to avoid having to copy data. Note also that client is not a separate process. Rather it directly runs in the context of the application itself. There is no data copy between the application and AppSwitch client either.

Delayering the stack

Now that we have an idea about what AppSwitch does, let’s look at the layers that it optimizes away from a standard service mesh.

Network devirtualization

Kubernetes offers simple and well-defined network constructs to the microservice applications it runs. In order to support them however, it imposes specific requirements on the underlying network. Meeting those requirements is often not easy. The go-to solution of adding another layer is typically adopted to satisfy the requirements. In most cases the additional layer consists of a network overlay that sits between Kubernetes and underlying network. Traffic produced by the applications is encapsulated at the source and decapsulated at the target, which not only costs network resources but also takes up compute cores.

Because AppSwitch arbitrates what the application sees through its touchpoints with the platform, it projects a consistent virtual view of the underlying network to the application similar to an overlay but without introducing an additional layer of processing along the datapath. Just to draw a parallel to containers, the inside of a container looks and feels like a VM. However the underlying implementation does not intervene along the high-incidence control paths of low-level interrupts etc.

AppSwitch can be injected into a standard Kubernetes manifest (similar to Istio injection) such that the application’s network is directly handled by AppSwitch bypassing any network overlay underneath. More details to follow in just a bit.

Artifacts of container networking

Extending network connectivity from host into the container has been a major challenge. New layers of network plumbing were invented explicitly for that purpose. As such, an application running in a container is simply a process on the host. However due to a fundamental misalignment between the network abstraction expected by the application and the abstraction exposed by container network namespace, the process cannot directly access the host network. Applications think of networking in terms of sockets or sessions whereas network namespaces expose a device abstraction. Once placed in a network namespace, the process suddenly loses all connectivity. The notion of veth-pair and corresponding tooling were invented just to close that gap. The data would now have to go from a host interface into a virtual switch and then through a veth-pair to the virtual network interface of the container network namespace.

AppSwitch can effectively remove both the virtual switch and veth-pair layers on both ends of the connection. Since the connections are established by the daemon running on the host using the network that’s already available on the host, there is no need for additional plumbing to bridge host network into the container. The socket FDs created on the host are passed to the application running within the pod’s network namespace. By the time the application receives the FD, all control path work (security checks, connection establishment) is already done and the FD is ready for actual IO.

Skip TCP/IP for colocated endpoints

TCP/IP is the universal protocol medium over which pretty much all communication occurs. But if application endpoints happen to be on the same host, is TCP/IP really required? After all, it does do quite a bit of work and it is quite complex. Unix sockets are explicitly designed for intrahost communication and AppSwitch can transparently switch the communication to occur over a Unix socket for colocated endpoints.

For each listening socket of an application, AppSwitch maintains two listening sockets, one each for TCP and Unix. When a client tries to connect to a server that happens to be colocated, AppSwitch daemon would choose to connect to the Unix listening socket of the server. The resulting Unix sockets on each end are passed into respective applications. Once a fully connected FD is returned, the application would simply treat it as a bit pipe. The protocol doesn’t really matter. The application may occasionally make protocol specific calls such as getsockname(2) and AppSwitch would handle them in kind. It would present consistent responses such that the application would continue to run on.

Data pushing proxy

As we continue to look for layers to remove, let us also reconsider the requirement of the proxy layer itself. There are times when the role of the proxy may degenerate into a plain data pusher:

There may not be a need for any protocol decoding
The protocol may not be recognized by the proxy
The communication may be encrypted and the proxy cannot access relevant headers
The application (redis, memcached etc.) may be too latency-sensitive and cannot afford the cost of an intermediate proxy

In all these cases, the proxy is not different from any low-level plumbing layer. In fact, the latency introduced can be far higher because the same level of optimizations won’t be available to a proxy.

To illustrate this with an example, consider the application shown below. It consists of a Python app and a set of memcached servers behind it. An upstream memcached server is selected based on connection time routing. Speed is the primary concern here.

Latency-sensitive application scenario

If we look at the data flow in this setup, the Python app makes a connection to the service IP of memcached. It is redirected to the client-side sidecar. The sidecar routes the connection to one of the memcached servers and copies the data between the two sockets – one connected to the app and another connected to memcached. And the same also occurs on the server side between the server-side sidecar and memcached. The role of proxy at that point is just boring shoveling of bits between the two sockets. However, it ends up adding substantial latency to the end-to-end connection.

Now let us imagine that the app is somehow made to connect directly to memcached, then the two intermediate proxies could be skipped. The data would flow directly between the app and memcached without any intermediate hops. AppSwitch can arrange for that by transparently tweaking the target address passed by the Python app when it makes the connect(2) system call.

Proxyless protocol decoding

Things are going to get a bit strange here. We have seen that the proxy can be bypassed for cases that don’t involve looking into application traffic. But is there anything we can do even for those other cases? It turns out, yes.

In a typical communication between microservices, much of the interesting information is exchanged in the initial headers. Headers are followed by body or payload which typically represents bulk of the communication. And once again the proxy degenerates into a data pusher for this part of communication. AppSwitch provides a nifty mechanism to skip proxy for these cases.

Even though AppSwitch is not a proxy, it does arbitrate connections between application endpoints and it does have access to corresponding socket FDs. Normally, AppSwitch simply passes those FDs to the application. But it can also peek into the initial message received on the connection using the MSG_PEEK option of the recvfrom(2) system call on the socket. It allows AppSwitch to examine application traffic without actually removing it from the socket buffers. When AppSwitch returns the FD to the application and steps out of the datapath, the application would do an actual read on the connection. AppSwitch uses this technique to perform deeper analysis of application-level traffic and implement sophisticated network functions as discussed in the next section, all without getting into the datapath.

Zero-cost load balancer, firewall and network analyzer

Typical implementations of network functions such as load balancers and firewalls require an intermediate layer that needs to tap into data/packet stream. Kubernetes’ implementation of load balancer (kube-proxy) for example introduces a probe into the packet stream through iptables and Istio implements the same at the proxy layer. But if all that is required is to redirect or drop connections based on policy, it is not really necessary to stay in the datapath during the entire course of the connection. AppSwitch can take care of that much more efficiently by simply manipulating the control path at the API level. Given its intimate proximity to the application, AppSwitch also has easy access to various pieces of application level metrics such as dynamics of stack and heap usage, precisely when a service comes alive, attributes of active connections etc., all of which could potentially form a rich signal for monitoring and analytics.

To go a step further, AppSwitch can also perform L7 load balancing and firewall functions based on the protocol data that it obtains from the socket buffers. It can synthesize the protocol data and various other signals with the policy information acquired from Pilot to implement a highly efficient form of routing and access control enforcement. It can essentially “influence” the application to connect to the right backend server without requiring any changes to the application or its configuration. It is as if the application itself is infused with policy and traffic-management intelligence. Except in this case, the application can’t escape the influence.

There is some more black-magic possible that would actually allow modifying the application data stream without getting into the datapath but I am going to save that for a later post. Current implementation of AppSwitch uses a proxy if the use case requires application protocol traffic to be modified. For those cases, AppSwitch provides a highly optimal mechanism to attract traffic to the proxy as discussed in the next section.

Traffic redirection

Before the sidecar proxy can look into application protocol traffic, it needs to first receive the connections. Redirection of connections coming into and going out of the application is currently done by a layer of packet filtering that rewrites packets such that they go to respective sidecars. Creating potentially large number of rules required to represent the redirection policy is tedious. And the process of applying the rules and updating them, as the target subnets to be captured by the sidecar change, is expensive.

While some of the performance concerns are being addressed by the Linux community, there is another concern related to privilege: iptables rules need to be updated whenever the policy changes. Given the current architecture, all privileged operations are performed in an init container that runs just once at the very beginning before privileges are dropped for the actual application. Since updating iptables rules requires root privileges, there is no way to do that without restarting the application.

AppSwitch provides a way to redirect application connections without root privilege. As such, an unprivileged application is already able to connect to any host (modulo firewall rules etc.) and the owner of the application should be allowed to change the host address passed by its application via connect(2) without requiring additional privilege.

Socket delegation

Let’s see how AppSwitch could help redirect connections without using iptables. Imagine that the application somehow voluntarily passes the socket FDs that it uses for its communication to the sidecar, then there would be no need for iptables. AppSwitch provides a feature called socket delegation that does exactly that. It allows the sidecar to transparently gain access to copies of socket FDs that the application uses for its communication without any changes to the application itself.

Here are the sequence of steps that would achieve this in the context of the Python application example.

The application initiates a connection request to the service IP of memcached service.
The connection request from client is forwarded to the daemon.
The daemon creates a pair of pre-connected Unix sockets (using socketpair(2) system call).
It passes one end of the socket pair into the application such that the application would use that socket FD for read/write. It also ensures that the application consistently sees it as a legitimate TCP socket as it expects by interposing all calls that query connection properties.
The other end is passed to sidecar over a different Unix socket where the daemon exposes its API. Information such as the original destination that the application was connecting to is also conveyed over the same interface.

Socket delegation based connection redirection

Once the application and sidecar are connected, the rest happens as usual. Sidecar would initiate a connection to upstream server and proxy data between the socket received from the daemon and the socket connected to upstream server. The main difference here is that sidecar would get the connection, not through the accept(2) system call as it is in the normal case, but from the daemon over the Unix socket. In addition to listening for connections from applications through the normal accept(2) channel, the sidecar proxy would connect to the AppSwitch daemon’s REST endpoint and receive sockets that way.

For completeness, here are the sequence of steps that would occur on the server side:

The application receives a connection
AppSwitch daemon accepts the connection on behalf of the application
It creates a pair of pre-connected Unix sockets using socketpair(2) system call
One end of the socket pair is returned to the application through the accept(2) system call
The other end of the socket pair along with the socket originally accepted by the daemon on behalf of the application is sent to sidecar
Sidecar would extract the two socket FDs – a Unix socket FD connected to the application and a TCP socket FD connected to the remote client
Sidecar would read the metadata supplied by the daemon about the remote client and perform its usual operations

“Sidecar-aware” applications

Socket delegation feature can be very useful for applications that are explicitly aware of the sidecar and wish to take advantage of its features. They can voluntarily delegate their network interactions by passing their sockets to the sidecar using the same feature. In a way, AppSwitch transparently turns every application into a sidecar-aware application.

How does it all come together?

Just to step back, Istio offloads common connectivity concerns from applications to a sidecar proxy that performs those functions on behalf of the application. And AppSwitch simplifies and optimizes the service mesh by sidestepping intermediate layers and invoking the proxy only for cases where it is truly necessary.

In the rest of this section, I outline how AppSwitch may be integrated with Istio based on a very cursory initial implementation. This is not intended to be anything like a design doc – not every possible way of integration is explored and not every detail is worked out. The intent is to discuss high-level aspects of the implementation to present a rough idea of how the two systems may come together. The key is that AppSwitch would act as a cushion between Istio and a real proxy. It would serve as the “fast-path” for cases that can be performed more efficiently without invoking the sidecar proxy. And for the cases where the proxy is used, it would shorten the datapath by cutting through unnecessary layers. Look at this blog for a more detailed walk through of the integration.

AppSwitch client injection

Similar to Istio sidecar-injector, a simple tool called ax-injector injects AppSwitch client into a standard Kubernetes manifest. Injected client transparently monitors the application and intimates AppSwitch daemon of the control path network API events that the application produces.

It is possible to not require the injection and work with standard Kubernetes manifests if AppSwitch CNI plugin is used. In that case, the CNI plugin would perform necessary injection when it gets the initialization callback. Using injector does have some advantages, however: (1) It works in tightly-controlled environments like GKE (2) It can be easily extended to support other frameworks such as Mesos (3) Same cluster would be able to run standard applications alongside “AppSwitch-enabled” applications.

AppSwitch `DaemonSet`

AppSwitch daemon can be configured to run as a DaemonSet or as an extension to the application that is directly injected into application manifest. In either case it handles network events coming in from the applications that it supports.

Agent for policy acquisition

This is the component that conveys policy and configuration dictated by Istio to AppSwitch. It implements xDS API to listen from Pilot and calls appropriate AppSwitch APIs to program the daemon. For example, it allows the load balancing strategy, as specified by istioctl, to be translated into equivalent AppSwitch capability.

Platform adapter for AppSwitch “Auto-Curated” service registry

Given that AppSwitch is in the control path of applications’ network APIs, it has ready access to the topology of services across the cluster. AppSwitch exposes that information in the form of a service registry that is automatically and (almost) synchronously updated as applications and their services come and go. A new platform adapter for AppSwitch alongside Kubernetes, Eureka etc. would provide the details of upstream services to Istio. This is not strictly necessary but it does make it easier to correlate service endpoints received from Pilot by AppSwitch agent above.

Proxy integration and chaining

Connections that do require deep scanning and mutation of application traffic are handed off to an external proxy through the socket delegation mechanism discussed earlier. It uses an extended version of proxy protocol. In addition to the simple parameters supported by the proxy protocol, a variety of other metadata (including the initial protocol headers obtained from the socket buffers) and live socket FDs (representing application connections) are forwarded to the proxy.

The proxy can look at the metadata and decide how to proceed. It could respond by accepting the connection to do the proxying or by directing AppSwitch to allow the connection and use the fast-path or to just drop the connection.

One of the interesting aspects of the mechanism is that, when the proxy accepts a socket from AppSwitch, it can in turn delegate the socket to another proxy. In fact that is how AppSwitch currently works. It uses a simple built-in proxy to examine the metadata and decide whether to handle the connection internally or to hand it off to an external proxy (Envoy). The same mechanism can be potentially extended to allow for a chain of plugins, each looking for a specific signature, with the last one in the chain doing the real proxy work.

It’s not just about performance

Removing intermediate layers along the datapath is not just about improving performance. Performance is a great side effect, but it is a side effect. There are a number of important advantages to an API level approach.

Automatic application onboarding and policy authoring

Before microservices and service mesh, traffic management was done by load balancers and access controls were enforced by firewalls. Applications were identified by IP addresses and DNS names which were relatively static. In fact, that’s still the status quo in most environments. Such environments stand to benefit immensely from service mesh. However a practical and scalable bridge to the new world needs to be provided. The difficulty in transformation is not as much due to lack of features and functionality but the investment required to rethink and reimplement the entire application infrastructure. Currently most of the policy and configuration exists in the form of load balancer and firewall rules. Somehow that existing context needs to be leveraged in providing a scalable path to adopting the service mesh model.

AppSwitch can substantially ease the onboarding process. It can project the same network environment to the application at the target as its current source environment. Not having any assistance here is typically a non-starter in case of traditional applications which have complex configuration files with static IP addresses or specific DNS names hard-coded in them. AppSwitch could help capture those applications along with their existing configuration and connect them over a service mesh without requiring any changes.

Broader application and protocol support

HTTP clearly dominates the modern application landscapes but once we talk about traditional applications and environments, we’d encounter all kinds of protocols and transports. Particularly, support for UDP becomes unavoidable. Traditional application servers such as IBM WebSphere rely extensively on UDP. Most multimedia applications use UDP media streams. Of course DNS is probably the most widely used UDP “application”. AppSwitch supports UDP at the API level much the same way as TCP and when it detects a UDP connection, it can transparently handle it in its “fast-path” rather than delegating it to the proxy.

Client IP preservation and end-to-end principle

The same mechanism that preserves the source network environment can also preserve client IP addresses as seen by the servers. With a sidecar proxy in place, connection requests come from the proxy rather than the client. As a result, the peer address (IP:port) of the connection as seen by the server would be that of the proxy rather than the client. AppSwitch ensures that the server sees correct address of the client, logs it correctly and any decisions made based on the client address remain valid. More generally, AppSwitch preserves the end-to-end principle which is otherwise broken by intermediate layers that obfuscate the true underlying context.

Enhanced application signal with access to encrypted headers

Encrypted traffic completely undermines the ability of the service mesh to analyze application traffic. API level interposition could potentially offer a way around it. Current implementation of AppSwitch gains access to application’s network API at the system call level. However it is possible in principle to influence the application at an API boundary, higher in the stack where application data is not yet encrypted or already decrypted. Ultimately the data is always produced in the clear by the application and then encrypted at some point before it goes out. Since AppSwitch directly runs within the memory context of the application, it is possible to tap into the data higher on the stack where it is still held in clear. Only requirement for this to work is that the API used for encryption should be well-defined and amenable for interposition. Particularly, it requires access to the symbol table of the application binaries. Just to be clear, AppSwitch doesn’t implement this today.

So what’s the net?

AppSwitch removes a number of layers and processing from the standard service mesh stack. What does all that translate to in terms of performance?

We ran some initial experiments to characterize the extent of the opportunity for optimization based on the initial integration of AppSwitch discussed earlier. The experiments were run on GKE using fortio-0.11.0, istio-0.8.0 and appswitch-0.4.0-2. In case of the proxyless test, AppSwitch daemon was run as a DaemonSet on the Kubernetes cluster and the Fortio pod spec was modified to inject AppSwitch client. These were the only two changes made to the setup. The test was configured to measure the latency of GRPC requests across 100 concurrent connections.

Latency with and without AppSwitch

Initial results indicate a difference of over 18x in p50 latency with and without AppSwitch (3.99ms vs 72.96ms). The difference was around 8x when mixer and access logs were disabled. Clearly the difference was due to sidestepping all those intermediate layers along the datapath. Unix socket optimization wasn’t triggered in case of AppSwitch because client and server pods were scheduled to separate hosts. End-to-end latency of AppSwitch case would have been even lower if the client and server happened to be colocated. Essentially the client and server running in their respective pods of the Kubernetes cluster are directly connected over a TCP socket going over the GKE network – no tunneling, bridge or proxies.

Net Net

I started out with David Wheeler’s seemingly reasonable quote that says adding another layer is not a solution for the problem of too many layers. And I argued through most of the blog that current network stack already has too many layers and that they should be removed. But isn’t AppSwitch itself a layer?

Yes, AppSwitch is clearly another layer. However it is one that can remove multiple other layers. In doing so, it seamlessly glues the new service mesh layer with existing layers of traditional network environments. It offsets the cost of sidecar proxy and as Istio graduates to 1.0, it provides a bridge for existing applications and their network environments to transition to the new world of service mesh.

Perhaps Wheeler’s quote should read:

Acknowledgements

Thanks to Mandar Jog (Google) for several discussions about the value of AppSwitch for Istio and to the following individuals (in alphabetical order) for their review of early drafts of this blog.

Frank Budinsky (IBM)
Lin Sun (IBM)
Shriram Rajagopalan (VMware)

Micro-Segmentation with Istio Authorization

Limin Wang — Fri, 20 Jul 2018 00:00:00 +0000

Micro-segmentation is a security technique that creates secure zones in cloud deployments and allows organizations to isolate workloads from one another and secure them individually. Istio’s authorization feature, also known as Istio Role Based Access Control, provides micro-segmentation for services in an Istio mesh. It features:

Authorization at different levels of granularity, including namespace level, service level, and method level.
Service-to-service and end-user-to-service authorization.
High performance, as it is enforced natively on Envoy.
Role-based semantics, which makes it easy to use.
High flexibility as it allows users to define conditions using combinations of attributes.

In this blog post, you’ll learn about the main authorization features and how to use them in different situations.

Characteristics

RPC level authorization

Authorization is performed at the level of individual RPCs. Specifically, it controls “who can access my bookstore service”, or “who can access method getBook in my bookstore service”. It is not designed to control access to application-specific resource instances, like access to “storage bucket X” or access to “3rd book on 2nd shelf”. Today this kind of application specific access control logic needs to be handled by the application itself.

Role-based access control with conditions

Authorization is a role-based access control (RBAC) system, contrast this to an attribute-based access control (ABAC) system. Compared to ABAC, RBAC has the following advantages:

Roles allow grouping of attributes. Roles are groups of permissions, which specifies the actions you are allowed to perform on a system. Users are grouped based on the roles within an organization. You can define the roles and reuse them for different cases.
It is easier to understand and reason about who has access. The RBAC concepts map naturally to business concepts. For example, a DB admin may have all access to DB backend services, while a web client may only be able to view the frontend service.
It reduces unintentional errors. RBAC policies make otherwise complex security changes easier. You won’t have duplicate configurations in multiple places and later forget to update some of them when you need to make changes.

On the other hand, Istio’s authorization system is not a traditional RBAC system. It also allows users to define conditions using combinations of attributes. This gives Istio flexibility to express complex access control policies. In fact, the “RBAC + conditions” model that Istio authorization adopts, has all the benefits an RBAC system has, and supports the level of flexibility that normally an ABAC system provides. You’ll see some examples below.

High performance

Because of its simple semantics, Istio authorization is enforced on Envoy as a native authorization support. At runtime, the authorization decision is completely done locally inside an Envoy filter, without dependency to any external module. This allows Istio authorization to achieve high performance and availability.

Work with/without primary identities

Like any other RBAC system, Istio authorization is identity aware. In Istio authorization policy, there is a primary identity called user, which represents the principal of the client.

In addition to the primary identity, you can also specify any conditions that define the identities. For example, you can specify the client identity as “user Alice calling from Bookstore frontend service”, in which case, you have a combined identity of the calling service (Bookstore frontend) and the end user (Alice).

To improve security, you should enable authentication features, and use authenticated identities in authorization policies. However, strongly authenticated identity is not required for using authorization. Istio authorization works with or without identities. If you are working with a legacy system, you may not have mutual TLS or JWT authentication setup for your mesh. In this case, the only way to identify the client is, for example, through IP. You can still use Istio authorization to control which IP addresses or IP ranges are allowed to access your service.

Examples

The authorization task shows you how to use Istio’s authorization feature to control namespace level and service level access using the Bookinfo application. In this section, you’ll see more examples on how to achieve micro-segmentation with Istio authorization.

Namespace level segmentation via RBAC + conditions

Suppose you have services in the frontend and backend namespaces. You would like to allow all your services in the frontend namespace to access all services that are marked external in the backend namespace.

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
name: external-api-caller
namespace: backend
spec:
rules:
- services: ["*"]
methods: ["*”]
constraints:
- key: "destination.labels[visibility]”
values: ["external"]
---
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: external-api-caller
namespace: backend
spec:
subjects:
- properties:
source.namespace: "frontend”
roleRef:
kind: ServiceRole
name: "external-api-caller"

The ServiceRole and ServiceRoleBinding above expressed “who is allowed to do what under *which conditions*” (RBAC + conditions). Specifically:

“who” are the services in the frontend namespace.
“what” is to call services in backend namespace.
“conditions” is the visibility label of the destination service having the value external.

Service/method level isolation with/without primary identities

Here is another example that demonstrates finer grained access control at service/method level. The first step is to define a book-reader service role that allows READ access to /books/* resource in bookstore service.

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
name: book-reader
namespace: default
spec:
rules:
- services: ["bookstore.default.svc.cluster.local"]
paths: ["/books/*”]
methods: ["GET”]

Using authenticated client identities

Suppose you want to grant this book-reader role to your bookstore-frontend service. If you have enabled mutual TLS authentication for your mesh, you can use a service account to identify your bookstore-frontend service. Granting the book-reader role to the bookstore-frontend service can be done by creating a ServiceRoleBinding as shown below:

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: book-reader
namespace: default
spec:
subjects:
- user: "cluster.local/ns/default/sa/bookstore-frontend”
roleRef:
kind: ServiceRole
name: "book-reader"

You may want to restrict this further by adding a condition that “only users who belong to the qualified-reviewer group are allowed to read books”. The qualified-reviewer group is the end user identity that is authenticated by JWT authentication. In this case, the combination of the client service identity (bookstore-frontend) and the end user identity (qualified-reviewer) is used in the authorization policy.

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: book-reader
namespace: default
spec:
subjects:
- user: "cluster.local/ns/default/sa/bookstore-frontend"
properties:
request.auth.claims[group]: "qualified-reviewer"
roleRef:
kind: ServiceRole
name: "book-reader"

Client does not have identity

Using authenticated identities in authorization policies is strongly recommended for security. However, if you have a legacy system that does not support authentication, you may not have authenticated identities for your services. You can still use Istio authorization to protect your services even without authenticated identities. The example below shows that you can specify allowed source IP range in your authorization policy.

apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
name: book-reader
namespace: default
spec:
subjects:
- properties:
source.ip: 10.20.0.0/9
roleRef:
kind: ServiceRole
name: "book-reader"

Summary

Istio’s authorization feature provides authorization at namespace-level, service-level, and method-level granularity. It adopts “RBAC + conditions” model, which makes it easy to use and understand as an RBAC system, while providing the level of flexibility that an ABAC system normally provides. Istio authorization achieves high performance as it is enforced natively on Envoy. While it provides the best security by working together with Istio authentication features, Istio authorization can also be used to provide access control for legacy systems that do not have authentication.

Exporting Logs to BigQuery, GCS, Pub/Sub through Stackdriver

Nupur Garg and Douglas Reid — Mon, 09 Jul 2018 00:00:00 +0000

This post shows how to direct Istio logs to Stackdriver and export those logs to various configured sinks such as such as BigQuery, Google Cloud Storage or Cloud Pub/Sub. At the end of this post you can perform analytics on Istio data from your favorite places such as BigQuery, GCS or Cloud Pub/Sub.

The Bookinfo sample application is used as the example application throughout this task.

Before you begin

Install Istio in your cluster and deploy an application.

Configuring Istio to export logs

Istio exports logs using the logentry template. This specifies all the variables that are available for analysis. It contains information like source service, destination service, auth metrics (coming..) among others. Following is a diagram of the pipeline:

Exporting logs from Istio to Stackdriver for analysis

Istio supports exporting logs to Stackdriver which can in turn be configured to export logs to your favorite sink like BigQuery, Pub/Sub or GCS. Please follow the steps below to setup your favorite sink for exporting logs first and then Stackdriver in Istio.

Setting up various log sinks

Common setup for all sinks:

Enable Stackdriver Monitoring API for the project.
Make sure principalEmail that would be setting up the sink has write access to the project and Logging Admin role permissions.
Make sure the GOOGLE_APPLICATION_CREDENTIALS environment variable is set. Please follow instructions here to set it up.

BigQuery

Create a BigQuery dataset as a destination for the logs export.
Record the ID of the dataset. It will be needed to configure the Stackdriver handler. It would be of the form bigquery.googleapis.com/projects/[PROJECT_ID]/datasets/[DATASET_ID]
Give sink’s writer identity: cloud-logs@system.gserviceaccount.com BigQuery Data Editor role in IAM.
If using Google Kubernetes Engine, make sure bigquery Scope is enabled on the cluster.

Google Cloud Storage (GCS)

Create a GCS bucket where you would like logs to get exported in GCS.
Recode the ID of the bucket. It will be needed to configure Stackdriver. It would be of the form storage.googleapis.com/[BUCKET_ID]
Give sink’s writer identity: cloud-logs@system.gserviceaccount.com Storage Object Creator role in IAM.

Google Cloud Pub/Sub

Create a topic where you would like logs to get exported in Google Cloud Pub/Sub.
Recode the ID of the topic. It will be needed to configure Stackdriver. It would be of the form pubsub.googleapis.com/projects/[PROJECT_ID]/topics/[TOPIC_ID]
Give sink’s writer identity: cloud-logs@system.gserviceaccount.com Pub/Sub Publisher role in IAM.
If using Google Kubernetes Engine, make sure pubsub Scope is enabled on the cluster.

Setting up Stackdriver

A Stackdriver handler must be created to export data to Stackdriver. The configuration for a Stackdriver handler is described here.

Save the following yaml file as stackdriver.yaml. Replace <project_id>, <sink_id>, <sink_destination>, <log_filter> with their specific values.

apiVersion: "config.istio.io/v1alpha2"
kind: stackdriver
metadata:
name: handler
namespace: istio-system
spec:
# We'll use the default value from the adapter, once per minute, so we don't need to supply a value.
# pushInterval: 1m
# Must be supplied for the Stackdriver adapter to work
project_id: "<project_id>"
# One of the following must be set; the preferred method is `appCredentials`, which corresponds to
# Google Application Default Credentials.
# If none is provided we default to app credentials.
# appCredentials:
# apiKey:
# serviceAccountPath:
# Describes how to map Istio logs into Stackdriver.
logInfo:
accesslog.logentry.istio-system:
payloadTemplate: '{{or (.sourceIp) "-"}} - {{or (.sourceUser) "-"}} [{{or (.timestamp.Format "02/Jan/2006:15:04:05 -0700") "-"}}] "{{or (.method) "-"}} {{or (.url) "-"}} {{or (.protocol) "-"}}" {{or (.responseCode) "-"}} {{or (.responseSize) "-"}}'
httpMapping:
url: url
status: responseCode
requestSize: requestSize
responseSize: responseSize
latency: latency
localIp: sourceIp
remoteIp: destinationIp
method: method
userAgent: userAgent
referer: referer
labelNames:
- sourceIp
- destinationIp
- sourceService
- sourceUser
- sourceNamespace
- destinationIp
- destinationService
- destinationNamespace
- apiClaims
- apiKey
- protocol
- method
- url
- responseCode
- responseSize
- requestSize
- latency
- connectionMtls
- userAgent
- responseTimestamp
- receivedBytes
- sentBytes
- referer
sinkInfo:
id: '<sink_id>'
destination: '<sink_destination>'
filter: '<log_filter>'
---
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: stackdriver
namespace: istio-system
spec:
match: "true" # If omitted match is true.
actions:
- handler: handler.stackdriver
instances:
- accesslog.logentry
---

Push the configuration

$ kubectl apply -f stackdriver.yaml
stackdriver "handler" created
rule "stackdriver" created
logentry "stackdriverglobalmr" created
metric "stackdriverrequestcount" created
metric "stackdriverrequestduration" created
metric "stackdriverrequestsize" created
metric "stackdriverresponsesize" created

Send traffic to the sample application.

For the Bookinfo sample, visit http://$GATEWAY_URL/productpage in your web browser or issue the following command:
```
$ curl http://$GATEWAY_URL/productpage
```
Verify that logs are flowing through Stackdriver to the configured sink.
- Stackdriver: Navigate to the Stackdriver Logs Viewer for your project and look under “GKE Container” -> “Cluster Name” -> “Namespace Id” for Istio Access logs.
- BigQuery: Navigate to the BigQuery Interface for your project and you should find a table with prefix accesslog_logentry_istio in your sink dataset.
- GCS: Navigate to the Storage Browser for your project and you should find a bucket named accesslog.logentry.istio-system in your sink bucket.
- Pub/Sub: Navigate to the Pub/Sub Topic List for your project and you should find a topic for accesslog in your sink topic.

Understanding what happened

Stackdriver.yaml file above configured Istio to send access logs to Stackdriver and then added a sink configuration where these logs could be exported. In detail as follows:

Added a handler of kind stackdriver

apiVersion: "config.istio.io/v1alpha2"
kind: stackdriver
metadata:
name: handler
namespace: <your defined namespace>

Added logInfo in spec
```
spec:
logInfo: accesslog.logentry.istio-system:
labelNames:
- sourceIp
- destinationIp
...
...
sinkInfo:
id: '<sink_id>'
destination: '<sink_destination>'
filter: '<log_filter>'
```
In the above configuration sinkInfo contains information about the sink where you want the logs to get exported to. For more information on how this gets filled for different sinks please refer here.

Added a rule for Stackdriver

apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: stackdriver
namespace: istio-system spec:
match: "true" # If omitted match is true
actions:
- handler: handler.stackdriver
instances:
- accesslog.logentry

Cleanup

Remove the new Stackdriver configuration:
```
$ kubectl delete -f stackdriver.yaml
```
If you are not planning to explore any follow-on tasks, refer to the Bookinfo cleanup instructions to shutdown the application.

Availability of logs in export sinks

Export to BigQuery is within minutes (we see it to be almost instant), GCS can have a delay of 2 to 12 hours and Pub/Sub is almost immediately.

Monitoring and Access Policies for HTTP Egress Traffic

Vadim Eisenberg and Ronen Schaffer (IBM) — Fri, 22 Jun 2018 00:00:00 +0000

While Istio’s main focus is management of traffic between microservices inside a service mesh, Istio can also manage ingress (from outside into the mesh) and egress (from the mesh outwards) traffic. Istio can uniformly enforce access policies and aggregate telemetry data for mesh-internal, ingress and egress traffic.

In this blog post, we show how to apply monitoring and access policies to HTTP egress traffic with Istio.

Use case

Consider an organization that runs applications that process content from cnn.com. The applications are decomposed into microservices deployed in an Istio service mesh. The applications access pages of various topics from cnn.com: edition.cnn.com/politics, edition.cnn.com/sport and edition.cnn.com/health. The organization configures Istio to allow access to edition.cnn.com and everything works fine. However, at some point in time, the organization decides to banish politics. Practically, it means blocking access to edition.cnn.com/politics and allowing access to edition.cnn.com/sport and edition.cnn.com/health only. The organization will grant permissions to individual applications and to particular users to access edition.cnn.com/politics, on a case-by-case basis.

To achieve that goal, the organization’s operations people monitor access to the external services and analyze Istio logs to verify that no unauthorized request was sent to edition.cnn.com/politics. They also configure Istio to prevent access to edition.cnn.com/politics automatically.

The organization is resolved to prevent any tampering with the new policy. It decides to put mechanisms in place that will prevent any possibility for a malicious application to access the forbidden topic.

The Control Egress Traffic task demonstrates how external (outside the Kubernetes cluster) HTTP and HTTPS services can be accessed by applications inside the mesh.
The Configure an Egress Gateway example describes how to configure Istio to direct egress traffic through a dedicated gateway service called egress gateway.
The Egress Gateway with TLS Origination example demonstrates how to allow applications to send HTTP requests to external servers that require HTTPS, while directing traffic through egress gateway.
The Visualizing Metrics with Grafana describes the Istio Dashboard to monitor mesh traffic.
The Basic Access Control task shows how to control access to in-mesh services.
The Denials and White/Black Listing task shows how to configure access policies using black or white list checkers.

As opposed to the observability and security tasks above, this blog post describes Istio’s monitoring and access policies applied exclusively to the egress traffic.

Before you begin

Follow the steps in the Egress Gateway with TLS Origination example, with mutual TLS authentication enabled, without the Cleanup step. After completing that example, you can access edition.cnn.com/politics from an in-mesh container with curl installed. This blog post assumes that the SOURCE_POD environment variable contains the source pod’s name and that the container’s name is sleep.

Configure monitoring and access policies

Since you want to accomplish your tasks in a secure way, you should direct egress traffic through egress gateway, as described in the Egress Gateway with TLS Origination task. The secure way here means that you want to prevent malicious applications from bypassing Istio monitoring and policy enforcement.

According to our scenario, the organization performed the instructions in the Before you begin section, enabled HTTP traffic to edition.cnn.com, and configured that traffic to pass through the egress gateway. The egress gateway performs TLS origination to edition.cnn.com, so the traffic leaves the mesh encrypted. At this point, the organization is ready to configure Istio to monitor and apply access policies for the traffic to edition.cnn.com.

Logging

Configure Istio to log access to *.cnn.com. You create a logentry and two stdio handlers, one for logging forbidden access (error log level) and another one for logging all access to *.cnn.com (info log level). Then you create rules to direct your logentry instances to your handlers. One rule directs access to *.cnn.com/politics to the handler for logging forbidden access, another rule directs log entries to the handler that outputs each access to *.cnn.com as an info log entry. To understand the Istio logentries, rules, and handlers, see Istio Adapter Model. A diagram with the involved entities and dependencies between them appears below:

Instances, rules and handlers for egress monitoring

Create the logentry, rules and handlers. Note that you specify context.reporter.uid as kubernetes://istio-egressgateway in the rules to get logs from the egress gateway only.

$ cat <<EOF | kubectl apply -f -
# Log entry for egress access
apiVersion: "config.istio.io/v1alpha2"
kind: logentry
metadata:
name: egress-access
namespace: istio-system
spec:
severity: '"info"'
timestamp: request.time
variables:
destination: request.host | "unknown"
path: request.path | "unknown"
responseCode: response.code | 0
responseSize: response.size | 0
reporterUID: context.reporter.uid | "unknown"
sourcePrincipal: source.principal | "unknown"
monitored_resource_type: '"UNSPECIFIED"'
---
# Handler for error egress access entries
apiVersion: "config.istio.io/v1alpha2"
kind: stdio
metadata:
name: egress-error-logger
namespace: istio-system
spec:
severity_levels:
info: 2 # output log level as error
outputAsJson: true
---
# Rule to handle access to *.cnn.com/politics
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: handle-politics
namespace: istio-system
spec:
match: request.host.endsWith("cnn.com") && request.path.startsWith("/politics") && context.reporter.uid.startsWith("kubernetes://istio-egressgateway")
actions:
- handler: egress-error-logger.stdio
instances:
- egress-access.logentry
---
# Handler for info egress access entries
apiVersion: "config.istio.io/v1alpha2"
kind: stdio
metadata:
name: egress-access-logger
namespace: istio-system
spec:
severity_levels:
info: 0 # output log level as info
outputAsJson: true
---
# Rule to handle access to *.cnn.com
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: handle-cnn-access
namespace: istio-system
spec:
match: request.host.endsWith(".cnn.com") && context.reporter.uid.startsWith("kubernetes://istio-egressgateway")
actions:
- handler: egress-access-logger.stdio
instances:
- egress-access.logentry
EOF

Send three HTTP requests to cnn.com, to edition.cnn.com/politics, edition.cnn.com/sport and edition.cnn.com/health. All three should return 200 OK.

$ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
200
200
200

Query the Mixer log and see that the information about the requests appears in the log:

$ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep egress-access | grep cnn | tail -4
{"level":"info","time":"2019-01-29T07:43:24.611462Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/politics","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":1883355,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}
{"level":"info","time":"2019-01-29T07:43:24.886316Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/sport","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":2094561,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}
{"level":"info","time":"2019-01-29T07:43:25.369663Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/health","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":2157009,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}
{"level":"error","time":"2019-01-29T07:43:24.611462Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/politics","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":1883355,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}

You see four log entries related to your three requests. Three info entries about the access to edition.cnn.com and one error entry about the access to edition.cnn.com/politics. The service mesh operators can see all the access instances, and can also search the log for error log entries that represent forbidden accesses. This is the first security measure the organization can apply before blocking the forbidden accesses automatically, namely logging all the forbidden access instances as errors. In some settings this can be a sufficient security measure.

Note the attributes:

destination, path, responseCode, responseSize are related to HTTP parameters of the requests
sourcePrincipal:cluster.local/ns/default/sa/sleep - a string that represents the sleep service account in the default namespace
reporterUID: kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system - a UID of the reporting pod, in this case istio-egressgateway-747b6764b8-44rrh in the istio-system namespace

Access control by routing

After enabling logging of access to edition.cnn.com, automatically enforce an access policy, namely allow accessing /health and /sport URL paths only. Such a simple policy control can be implemented with Istio routing.

Redefine your VirtualService for edition.cnn.com:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-cnn-through-egress-gateway
spec:
hosts:
- edition.cnn.com
gateways:
- istio-egressgateway
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: cnn
port:
number: 443
weight: 100
- match:
- gateways:
- istio-egressgateway
port: 443
uri:
regex: "/health|/sport"
route:
- destination:
host: edition.cnn.com
port:
number: 443
weight: 100
EOF

Note that you added a match by uri condition that checks that the URL path is either /health or /sport. Also note that this condition is added to the istio-egressgateway section of the VirtualService, since the egress gateway is a hardened component in terms of security (see egress gateway security considerations). You don’t want any tampering with your policies.

Send the previous three HTTP requests to cnn.com:
```
$ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
404
200
200
```
The request to edition.cnn.com/politics returned 404 Not Found, while requests to edition.cnn.com/sport and edition.cnn.com/health returned 200 OK, as expected.

You may need to wait several seconds for the update of the VirtualService to propagate to the egress gateway.

Query the Mixer log and see that the information about the requests appears again in the log:

$ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep egress-access | grep cnn | tail -4
{"level":"info","time":"2019-01-29T07:55:59.686082Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/politics","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":404,"responseSize":0,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}
{"level":"info","time":"2019-01-29T07:55:59.697565Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/sport","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":2094561,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}
{"level":"info","time":"2019-01-29T07:56:00.264498Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/health","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":2157009,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}
{"level":"error","time":"2019-01-29T07:55:59.686082Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/politics","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":404,"responseSize":0,"sourcePrincipal":"cluster.local/ns/default/sa/sleep"}

You still get info and error messages regarding accesses to edition.cnn.com/politics, however this time the responseCode is 404, as expected.

While implementing access control using Istio routing worked for us in this simple case, it would not suffice for more complex cases. For example, the organization may want to allow access to edition.cnn.com/politics under certain conditions, so more complex policy logic than just filtering by URL paths will be required. You may want to apply Istio Mixer Adapters, for example white lists or black lists of allowed/forbidden URL paths, respectively. Policy Rules allow specifying complex conditions, specified in a rich expression language, which includes AND and OR logical operators. The rules can be reused for both logging and policy checks. More advanced users may want to apply Istio Role-Based Access Control.

An additional aspect is integration with remote access policy systems. If the organization in our use case operates some Identity and Access Management system, you may want to configure Istio to use access policy information from such a system. You implement this integration by applying Istio Mixer Adapters.

Cancel the access control by routing you used in this section and implement access control by Mixer policy checks in the next section.

Replace the VirtualService for edition.cnn.com with your previous version from the Configure an Egress Gateway example:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: direct-cnn-through-egress-gateway
spec:
hosts:
- edition.cnn.com
gateways:
- istio-egressgateway
- mesh
http:
- match:
- gateways:
- mesh
port: 80
route:
- destination:
host: istio-egressgateway.istio-system.svc.cluster.local
subset: cnn
port:
number: 443
weight: 100
- match:
- gateways:
- istio-egressgateway
port: 443
route:
- destination:
host: edition.cnn.com
port:
number: 443
weight: 100
EOF

Send the previous three HTTP requests to cnn.com, this time you should get three 200 OK responses as previously:

$ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
200
200
200

Access control by Mixer policy checks

In this step you use a Mixer Listchecker adapter, its whitelist variety. You define a listentry with the URL path of the request and a listchecker to check the listentry using a static list of allowed URL paths, specified by the overrides field. For an external Identity and Access Management system, use the providerurl field instead. The updated diagram of the instances, rules and handlers appears below. Note that you reuse the same policy rule, handle-cnn-access both for logging and for access policy checks.

Instances, rules and handlers for egress monitoring and access policies

Define path-checker and request-path:

$ cat <<EOF | kubectl create -f -
apiVersion: "config.istio.io/v1alpha2"
kind: listchecker
metadata:
name: path-checker
namespace: istio-system
spec:
overrides: ["/health", "/sport"] # overrides provide a static list
blacklist: false
---
apiVersion: "config.istio.io/v1alpha2"
kind: listentry
metadata:
name: request-path
namespace: istio-system
spec:
value: request.path
EOF

Modify the handle-cnn-access policy rule to send request-path instances to the path-checker:

$ cat <<EOF | kubectl apply -f -
# Rule handle egress access to cnn.com
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: handle-cnn-access
namespace: istio-system
spec:
match: request.host.endsWith(".cnn.com") && context.reporter.uid.startsWith("kubernetes://istio-egressgateway")
actions:
- handler: egress-access-logger.stdio
instances:
- egress-access.logentry
- handler: path-checker.listchecker
instances:
- request-path.listentry
EOF

Perform your usual test by sending HTTP requests to edition.cnn.com/politics, edition.cnn.com/sport and edition.cnn.com/health. As expected, the request to edition.cnn.com/politics returns 403 (Forbidden).

$ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
403
200
200

Access control by Mixer policy checks, part 2

After the organization in our use case managed to configure logging and access control, it decided to extend its access policy by allowing the applications with a special Service Account to access any topic of cnn.com, without being monitored. You’ll see how this requirement can be configured in Istio.

Start the sleep sample with the politics service account.

$ sed 's/: sleep/: politics/g' @samples/sleep/sleep.yaml@ | kubectl create -f -
serviceaccount "politics" created
service "politics" created
deployment "politics" created

Define the SOURCE_POD_POLITICS shell variable to hold the name of the source pod with the politics service account, for sending requests to external services.
```
$ export SOURCE_POD_POLITICS=$(kubectl get pod -l app=politics -o jsonpath={.items..metadata.name})
```

Perform your usual test of sending three HTTP requests this time from SOURCE_POD_POLITICS. The request to edition.cnn.com/politics returns 403, since you did not configure the exception for the politics namespace.

$ kubectl exec -it $SOURCE_POD_POLITICS -c politics -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
403
200
200

Query the Mixer log and see that the information about the requests from the politics namespace appears in the log:

$ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep egress-access | grep cnn | tail -4
{"level":"info","time":"2019-01-29T08:04:42.559812Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/politics","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":403,"responseSize":84,"sourcePrincipal":"cluster.local/ns/default/sa/politics"}
{"level":"info","time":"2019-01-29T08:04:42.568424Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/sport","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":2094561,"sourcePrincipal":"cluster.local/ns/default/sa/politics"}
{"level":"error","time":"2019-01-29T08:04:42.559812Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/politics","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":403,"responseSize":84,"sourcePrincipal":"cluster.local/ns/default/sa/politics"}
{"level":"info","time":"2019-01-29T08:04:42.615641Z","instance":"egress-access.logentry.istio-system","destination":"edition.cnn.com","path":"/health","reporterUID":"kubernetes://istio-egressgateway-747b6764b8-44rrh.istio-system","responseCode":200,"responseSize":2157009,"sourcePrincipal":"cluster.local/ns/default/sa/politics"}

Note that sourcePrincipal is cluster.local/ns/default/sa/politics which represents the politics service account in the default namespace.

Redefine handle-cnn-access and handle-politics policy rules, to make the applications in the politics namespace exempt from monitoring and policy enforcement.

$ cat <<EOF | kubectl apply -f -
# Rule to handle access to *.cnn.com/politics
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: handle-politics
namespace: istio-system
spec:
match: request.host.endsWith("cnn.com") && context.reporter.uid.startsWith("kubernetes://istio-egressgateway") && request.path.startsWith("/politics") && source.principal != "cluster.local/ns/default/sa/politics"
actions:
- handler: egress-error-logger.stdio
instances:
- egress-access.logentry
---
# Rule handle egress access to cnn.com
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
name: handle-cnn-access
namespace: istio-system
spec:
match: request.host.endsWith(".cnn.com") && context.reporter.uid.startsWith("kubernetes://istio-egressgateway") && source.principal != "cluster.local/ns/default/sa/politics"
actions:
- handler: egress-access-logger.stdio
instances:
- egress-access.logentry
- handler: path-checker.listchecker
instances:
- request-path.listentry
EOF

Perform your usual test from SOURCE_POD:

$ kubectl exec -it $SOURCE_POD -c sleep -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
403
200
200

Since SOURCE_POD does not have politics service account, access to edition.cnn.com/politics is forbidden, as previously.

Perform the previous test from SOURCE_POD_POLITICS:

$ kubectl exec -it $SOURCE_POD_POLITICS -c politics -- sh -c 'curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/politics; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/sport; curl -sL -o /dev/null -w "%{http_code}\n" http://edition.cnn.com/health'
200
200
200

Access to all the topics of edition.cnn.com is allowed.

Examine the Mixer log and see that no more requests with sourcePrincipal equal cluster.local/ns/default/sa/politics appear in the log.
```
$ kubectl -n istio-system logs -l istio-mixer-type=telemetry -c mixer | grep egress-access | grep cnn | tail -4
```

Comparison with HTTPS egress traffic control

In this use case the applications use HTTP and Istio Egress Gateway performs TLS origination for them. Alternatively, the applications could originate TLS themselves by issuing HTTPS requests to edition.cnn.com. In this section we describe both approaches and their pros and cons.

In the HTTP approach, the requests are sent unencrypted on the local host, intercepted by the Istio sidecar proxy and forwarded to the egress gateway. Since you configure Istio to use mutual TLS between the sidecar proxy and the egress gateway, the traffic leaves the pod encrypted. The egress gateway decrypts the traffic, inspects the URL path, the HTTP method and headers, reports telemetry and performs policy checks. If the request is not blocked by some policy check, the egress gateway performs TLS origination to the external destination (cnn.com in our case), so the request is encrypted again and sent encrypted to the external destination. The diagram below demonstrates the network flow of this approach. The HTTP protocol inside the gateway designates the protocol as seen by the gateway after decryption.

HTTP egress traffic through an egress gateway

The drawback of this approach is that the requests are sent unencrypted inside the pod, which may be against security policies in some organizations. Also some SDKs have external service URLs hard-coded, including the protocol, so sending HTTP requests could be impossible. The advantage of this approach is the ability to inspect HTTP methods, headers and URL paths, and to apply policies based on them.

In the HTTPS approach, the requests are encrypted end-to-end, from the application to the external destination. The diagram below demonstrates the network flow of this approach. The HTTPS protocol inside the gateway designates the protocol as seen by the gateway.

HTTPS egress traffic through an egress gateway

The end-to-end HTTPS is considered a better approach from the security point of view. However, since the traffic is encrypted the Istio proxies and the egress gateway can only see the source and destination IPs and the SNI of the destination. Since you configure Istio to use mutual TLS between the sidecar proxy and the egress gateway, the identity of the source is also known. The gateway is unable to inspect the URL path, the HTTP method and the headers of the requests, so no monitoring and policies based on the HTTP information can be possible. In our use case, the organization would be able to allow access to edition.cnn.com and to specify which applications are allowed to access edition.cnn.com. However, it will not be possible to allow or block access to specific URL paths of edition.cnn.com. Neither blocking access to edition.cnn.com/politics nor monitoring such access are possible with the HTTPS approach.

We guess that each organization will consider the pros and cons of the two approaches and choose the one most appropriate to its needs.

Summary

In this blog post we showed how different monitoring and policy mechanisms of Istio can be applied to HTTP egress traffic. Monitoring can be implemented by configuring a logging adapter. Access policies can be implemented by configuring VirtualServices or by configuring various policy check adapters. We demonstrated a simple policy that allowed certain URL paths only. We also showed a more complex policy that extended the simple policy by making an exemption to the applications with a certain service account. Finally, we compared HTTP-with-TLS-origination egress traffic with HTTPS egress traffic, in terms of control possibilities by Istio.

Cleanup

Perform the instructions in Cleanup section of the Configure an Egress Gateway example.

Delete the logging and policy checks configuration:

$ kubectl delete logentry egress-access -n istio-system
$ kubectl delete stdio egress-error-logger -n istio-system
$ kubectl delete stdio egress-access-logger -n istio-system
$ kubectl delete rule handle-politics -n istio-system
$ kubectl delete rule handle-cnn-access -n istio-system
$ kubectl delete -n istio-system listchecker path-checker
$ kubectl delete -n istio-system listentry request-path

Delete the politics source pod:

$ sed 's/: sleep/: politics/g' @samples/sleep/sleep.yaml@ | kubectl delete -f -
serviceaccount "politics" deleted
service "politics" deleted
deployment "politics" deleted

Introducing the Istio v1alpha3 routing API

Frank Budinsky (IBM) and Shriram Rajagopalan (VMware) — Wed, 25 Apr 2018 00:00:00 +0000

Up until now, Istio has provided a simple API for traffic management using four configuration resources: RouteRule, DestinationPolicy, EgressRule, and (Kubernetes) Ingress. With this API, users have been able to easily manage the flow of traffic in an Istio service mesh. The API has allowed users to route requests to specific versions of services, inject delays and failures for resilience testing, add timeouts and circuit breakers, and more, all without changing the application code itself.

While this functionality has proven to be a very compelling part of Istio, user feedback has also shown that this API does have some shortcomings, specifically when using it to manage very large applications containing thousands of services, and when working with protocols other than HTTP. Furthermore, the use of Kubernetes Ingress resources to configure external traffic has proven to be woefully insufficient for our needs.

To address these, and other concerns, a new traffic management API, a.k.a. v1alpha3, is being introduced, which will completely replace the previous API going forward. Although the v1alpha3 model is fundamentally the same, it is not backward compatible and will require manual conversion from the old API.

To justify this disruption, the v1alpha3 API has gone through a long and painstaking community review process that has hopefully resulted in a greatly improved API that will stand the test of time. In this article, we will introduce the new configuration model and attempt to explain some of the motivation and design principles that influenced it.

Design principles

A few key design principles played a role in the routing model redesign:

Explicitly model infrastructure as well as intent. For example, in addition to configuring an ingress gateway, the component (controller) implementing it can also be specified.
The authoring model should be “producer oriented” and “host centric” as opposed to compositional. For example, all rules associated with a particular host are configured together, instead of individually.
Clear separation of routing from post-routing behaviors.

Configuration resources in v1alpha3

A typical mesh will have one or more load balancers (we call them gateways) that terminate TLS from external networks and allow traffic into the mesh. Traffic then flows through internal services via sidecar gateways. It is also common for applications to consume external services (e.g., Google Maps API). These may be called directly or, in certain deployments, all traffic exiting the mesh may be forced through dedicated egress gateways. The following diagram depicts this mental model.

Gateways in an Istio service mesh

With the above setup in mind, v1alpha3 introduces the following new configuration resources to control traffic routing into, within, and out of the mesh.

Gateway
VirtualService
DestinationRule
ServiceEntry

VirtualService, DestinationRule, and ServiceEntry replace RouteRule, DestinationPolicy, and EgressRule respectively. The Gateway is a platform independent abstraction to model the traffic flowing into dedicated middleboxes.

The figure below depicts the flow of control across configuration resources.

Relationship between different v1alpha3 elements

`Gateway`

A Gateway configures a load balancer for HTTP/TCP traffic, regardless of where it will be running. Any number of gateways can exist within the mesh and multiple different gateway implementations can co-exist. In fact, a gateway configuration can be bound to a particular workload by specifying the set of workload (pod) labels as part of the configuration, allowing users to reuse off the shelf network appliances by writing a simple gateway controller.

For ingress traffic management, you might ask: Why not reuse Kubernetes Ingress APIs? The Ingress APIs proved to be incapable of expressing Istio’s routing needs. By trying to draw a common denominator across different HTTP proxies, the Ingress is only able to support the most basic HTTP routing and ends up pushing every other feature of modern proxies into non-portable annotations.

Istio Gateway overcomes the Ingress shortcomings by separating the L4-L6 spec from L7. It only configures the L4-L6 functions (e.g., ports to expose, TLS configuration) that are uniformly implemented by all good L7 proxies. Users can then use standard Istio rules to control HTTP requests as well as TCP traffic entering a Gateway by binding a VirtualService to it.

For example, the following simple Gateway configures a load balancer to allow external https traffic for host bookinfo.com into the mesh:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: bookinfo-gateway
spec:
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- bookinfo.com
tls:
mode: SIMPLE
serverCertificate: /tmp/tls.crt
privateKey: /tmp/tls.key

To configure the corresponding routes, a VirtualService (described in the following section) must be defined for the same host and bound to the Gateway using the gateways field in the configuration:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
spec:
hosts:
- bookinfo.com
gateways:
- bookinfo-gateway # <---- bind to gateway
http:
- match:
- uri:
prefix: /reviews
route:
...

The Gateway can be used to model an edge-proxy or a purely internal proxy as shown in the first figure. Irrespective of the location, all gateways can be configured and controlled in the same way.

`VirtualService`

Replacing route rules with something called “virtual services” might seem peculiar at first, but in reality it’s fundamentally a much better name for what is being configured, especially after redesigning the API to address the scalability issues with the previous model.

In effect, what has changed is that instead of configuring routing using a set of individual configuration resources (rules) for a particular destination service, each containing a precedence field to control the order of evaluation, we now configure the (virtual) destination itself, with all of its rules in an ordered list within a corresponding VirtualService resource. For example, where previously we had two RouteRule resources for the Bookinfo application’s reviews service, like this:

apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: reviews-default
spec:
destination:
name: reviews
precedence: 1
route:
- labels:
version: v1
---
apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: reviews-test-v2
spec:
destination:
name: reviews
precedence: 2
match:
request:
headers:
cookie:
regex: "^(.*?;)?(user=jason)(;.*)?$"
route:
- labels:
version: v2

In v1alpha3, we provide the same configuration in a single VirtualService resource:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: reviews
spec:
hosts:
- reviews
http:
- match:
- headers:
cookie:
regex: "^(.*?;)?(user=jason)(;.*)?$"
route:
- destination:
host: reviews
subset: v2
- route:
- destination:
host: reviews
subset: v1

As you can see, both of the rules for the reviews service are consolidated in one place, which at first may or may not seem preferable. However, if you look closer at this new model, you’ll see there are fundamental differences that make v1alpha3 vastly more functional.

First of all, notice that the destination service for the VirtualService is specified using a hosts field (repeated field, in fact) and is then again specified in a destination field of each of the route specifications. This is a very important difference from the previous model.

A VirtualService describes the mapping between one or more user-addressable destinations to the actual destination workloads inside the mesh. In our example, they are the same, however, the user-addressed hosts can be any DNS names with optional wildcard prefix or CIDR prefix that will be used to address the service. This can be particularly useful in facilitating turning monoliths into a composite service built out of distinct microservices without requiring the consumers of the service to adapt to the transition.

For example, the following rule allows users to address both the reviews and ratings services of the Bookinfo application as if they are parts of a bigger (virtual) service at http://bookinfo.com/:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: bookinfo
spec:
hosts:
- bookinfo.com
http:
- match:
- uri:
prefix: /reviews
route:
- destination:
host: reviews
- match:
- uri:
prefix: /ratings
route:
- destination:
host: ratings
...

The hosts of a VirtualService do not actually have to be part of the service registry, they are simply virtual destinations. This allows users to model traffic for virtual hosts that do not have routable entries inside the mesh. These hosts can be exposed outside the mesh by binding the VirtualService to a Gateway configuration for the same host (as described in the previous section).

In addition to this fundamental restructuring, VirtualService includes several other important changes:

Multiple match conditions can be expressed inside the VirtualService configuration, reducing the need for redundant rules.
Each service version has a name (called a service subset). The set of pods/VMs belonging to a subset is defined in a DestinationRule, described in the following section.
VirtualService hosts can be specified using wildcard DNS prefixes to create a single rule for all matching services. For example, in Kubernetes, to apply the same rewrite rule for all services in the foo namespace, the VirtualService would use *.foo.svc.cluster.local as the host.

`DestinationRule`

A DestinationRule configures the set of policies to be applied while forwarding traffic to a service. They are intended to be authored by service owners, describing the circuit breakers, load balancer settings, TLS settings, etc.. DestinationRule is more or less the same as its predecessor, DestinationPolicy, with the following exceptions:

The host of a DestinationRule can include wildcard prefixes, allowing a single rule to be specified for many actual services.
A DestinationRule defines addressable subsets (i.e., named versions) of the corresponding destination host. These subsets are used in VirtualService route specifications when sending traffic to specific versions of the service. Naming versions this way allows us to cleanly refer to them across different virtual services, simplify the stats that Istio proxies emit, and to encode subsets in SNI headers.

A DestinationRule that configures policies and subsets for the reviews service might look something like this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: reviews
spec:
host: reviews
trafficPolicy:
loadBalancer:
simple: RANDOM
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
- name: v3
labels:
version: v3

Notice that, unlike DestinationPolicy, multiple policies (e.g., default and v2-specific) are specified in a single DestinationRule configuration.

`ServiceEntry`

ServiceEntry is used to add additional entries into the service registry that Istio maintains internally. It is most commonly used to allow one to model traffic to external dependencies of the mesh such as APIs consumed from the web or traffic to services in legacy infrastructure.

Everything you could previously configure using an EgressRule can just as easily be done with a ServiceEntry. For example, access to a simple external service from inside the mesh can be enabled using a configuration something like this:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: foo-ext
spec:
hosts:
- foo.com
ports:
- number: 80
name: http
protocol: HTTP

That said, ServiceEntry has significantly more functionality than its predecessor. First of all, a ServiceEntry is not limited to external service configuration, it can be of two types: mesh-internal or mesh-external. Mesh-internal entries are like all other internal services but are used to explicitly add services to the mesh. They can be used to add services as part of expanding the service mesh to include unmanaged infrastructure (e.g., VMs added to a Kubernetes-based service mesh). Mesh-external entries represent services external to the mesh. For them, mutual TLS authentication is disabled and policy enforcement is performed on the client-side, instead of on the usual server-side for internal service requests.

Because a ServiceEntry configuration simply adds a destination to the internal service registry, it can be used in conjunction with a VirtualService and/or DestinationRule, just like any other service in the registry. The following DestinationRule, for example, can be used to initiate mutual TLS connections for an external service:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: foo-ext
spec:
host: foo.com
trafficPolicy:
tls:
mode: MUTUAL
clientCertificate: /etc/certs/myclientcert.pem
privateKey: /etc/certs/client_private_key.pem
caCertificates: /etc/certs/rootcacerts.pem

In addition to its expanded generality, ServiceEntry provides several other improvements over EgressRule including the following:

A single ServiceEntry can configure multiple service endpoints, which previously would have required multiple EgressRules.
The resolution mode for the endpoints is now configurable (NONE, STATIC, or DNS).
Additionally, we are working on addressing another pain point: the need to access secure external services over plain text ports (e.g., http://google.com:443). This should be fixed in the coming weeks, allowing you to directly access https://google.com from your application. Stay tuned for an Istio patch release (0.8.x) that addresses this limitation.

Creating and deleting v1alpha3 route rules

Because all route rules for a given destination are now stored together as an ordered list in a single VirtualService resource, adding a second and subsequent rules for a particular destination is no longer done by creating a new (RouteRule) resource, but instead by updating the one-and-only VirtualService resource for the destination.

old routing rules:

$ kubectl apply -f my-second-rule-for-destination-abc.yaml

v1alpha3 routing rules:

$ kubectl apply -f my-updated-rules-for-destination-abc.yaml

Deleting route rules other than the last one for a particular destination is also done by updating the existing resource using kubectl apply.

When adding or removing routes that refer to service versions, the subsets will need to be updated in the service’s corresponding DestinationRule. As you might have guessed, this is also done using kubectl apply.

Summary

The Istio v1alpha3 routing API has significantly more functionality than its predecessor, but unfortunately is not backwards compatible, requiring a one time manual conversion. The previous configuration resources, RouteRule, DesintationPolicy, and EgressRule, will not be supported from Istio 0.9 onwards. Kubernetes users can continue to use Ingress to configure their edge load balancers for basic routing. However, advanced routing features (e.g., traffic split across two versions) will require use of Gateway, a significantly more functional and highly recommended Ingress replacement.

Acknowledgments

Credit for the routing model redesign and implementation work goes to the following people (in alphabetical order):

Frank Budinsky (IBM)
Zack Butcher (Google)
Greg Hanson (IBM)
Costin Manolache (Google)
Martin Ostrowski (Google)
Shriram Rajagopalan (VMware)
Louis Ryan (Google)
Isaiah Snell-Feikema (IBM)
Kuat Yessenov (Google)

Configuring Istio Ingress with AWS NLB

Julien SENON — Fri, 20 Apr 2018 00:00:00 +0000

This post provides instructions to use and configure ingress Istio with AWS Network Load Balancer.

Network load balancer (NLB) could be used instead of classical load balancer. You can see the comparison between different AWS loadbalancer for more explanation.

Prerequisites

The following instructions require a Kubernetes 1.9.0 or newer cluster.

IAM policy

You need to apply policy on the master role in order to be able to provision network load balancer.

In AWS iam console click on policies and click on create a new one:

Create a new policy
Select json:

Select json

Copy/paste text below:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "kopsK8sNLBMasterPermsRestrictive",
"Effect": "Allow",
"Action": [
"ec2:DescribeVpcs",
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancerPolicies",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:SetLoadBalancerPoliciesOfListener"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeVpcs",
"ec2:DescribeRegions"
],
"Resource": "*"
}
]
}

Click review policy, fill all fields and click create policy:

Validate policy
Click on roles, select you master role nodes, and click attach policy:

Attach policy
Your policy is now attach to your master node.

Generate the Istio manifest

To use an AWS nlb load balancer, it is necessary to add an AWS specific annotation to the Istio installation. These instructions explain how to add the annotation.

Save this as the file override.yaml:

gateways:
istio-ingressgateway:
serviceAnnotations:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"

Generate a manifest with Helm:

$ helm template install/kubernetes/helm/istio --namespace istio -f override.yaml > $HOME/istio.yaml

Istio Soft Multi-Tenancy Support

John Joyce and Rich Curran — Thu, 19 Apr 2018 00:00:00 +0000

Multi-tenancy is commonly used in many environments across many different applications, but the implementation details and functionality provided on a per tenant basis does not follow one model in all environments. The Kubernetes multi-tenancy working group is working to define the multi-tenant use cases and functionality that should be available within Kubernetes. However, from their work so far it is clear that only “soft multi-tenancy” is possible due to the inability to fully protect against malicious containers or workloads gaining access to other tenant’s pods or kernel resources.

Soft multi-tenancy

For this blog, “soft multi-tenancy” is defined as having a single Kubernetes control plane with multiple Istio control planes and multiple meshes, one control plane and one mesh per tenant. The cluster administrator gets control and visibility across all the Istio control planes, while the tenant administrator only gets control of a specific Istio instance. Separation between the tenants is provided by Kubernetes namespaces and RBAC.

One use case for this deployment model is a shared corporate infrastructure where malicious actions are not expected, but a clean separation of the tenants is still required.

Potential future Istio multi-tenant deployment models are described at the bottom of this blog.

Deployment

Multiple Istio control planes

Deploying multiple Istio control planes starts by replacing all namespace references in a manifest file with the desired namespace. Using istio.yaml as an example, if two tenant level Istio control planes are required; the first can use the istio.yaml default name of istio-system and a second control plane can be created by generating a new yaml file with a different namespace. As an example, the following command creates a yaml file with the Istio namespace of istio-system1.

$ cat istio.yaml | sed s/istio-system/istio-system1/g > istio-system1.yaml

The istio.yaml file contains the details of the Istio control plane deployment, including the pods that make up the control plane (Mixer, Pilot, Ingress, Galley, CA). Deploying the two Istio control plane yaml files:

$ kubectl apply -f install/kubernetes/istio.yaml
$ kubectl apply -f install/kubernetes/istio-system1.yaml

Results in two Istio control planes running in two namespaces.

$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-system istio-ca-ffbb75c6f-98w6x 1/1 Running 0 15d
istio-system istio-ingress-68d65fc5c6-dnvfl 1/1 Running 0 15d
istio-system istio-mixer-5b9f8dffb5-8875r 3/3 Running 0 15d
istio-system istio-pilot-678fc976c8-b8tv6 2/2 Running 0 15d
istio-system1 istio-ca-5f496fdbcd-lqhlk 1/1 Running 0 15d
istio-system1 istio-ingress-68d65fc5c6-2vldg 1/1 Running 0 15d
istio-system1 istio-mixer-7d4f7b9968-66z44 3/3 Running 0 15d
istio-system1 istio-pilot-5bb6b7669c-779vb 2/2 Running 0 15d

The Istio sidecar and addons, if required, manifests must also be deployed to match the configured namespace in use by the tenant’s Istio control plane.

The execution of these two yaml files is the responsibility of the cluster administrator, not the tenant level administrator. Additional RBAC restrictions will also need to be configured and applied by the cluster administrator, limiting the tenant administrator to only the assigned namespace.

Split common and namespace specific resources

The manifest files in the Istio repositories create both common resources that would be used by all Istio control planes as well as resources that are replicated per control plane. Although it is a simple matter to deploy multiple control planes by replacing the istio-system namespace references as described above, a better approach is to split the manifests into a common part that is deployed once for all tenants and a tenant specific part. For the Custom Resource Definitions, the roles and the role bindings should be separated out from the provided Istio manifests. Additionally, the roles and role bindings in the provided Istio manifests are probably unsuitable for a multi-tenant environment and should be modified or augmented as described in the next section.

Kubernetes RBAC for Istio control plane resources

To restrict a tenant administrator to a single Istio namespace, the cluster administrator would create a manifest containing, at a minimum, a Role and RoleBinding similar to the one below. In this example, a tenant administrator named sales-admin is limited to the namespace istio-system1. A completed manifest would contain many more apiGroups under the Role providing resource access to the tenant administrator.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: istio-system1
name: ns-access-for-sales-admin-istio-system1
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["*"]
verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: access-all-istio-system1
namespace: istio-system1
subjects:
- kind: User
name: sales-admin
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: ns-access-for-sales-admin-istio-system1
apiGroup: rbac.authorization.k8s.io

Watching specific namespaces for service discovery

In addition to creating RBAC rules limiting the tenant administrator’s access to a specific Istio control plane, the Istio manifest must be updated to specify the application namespace that Pilot should watch for creation of its xDS cache. This is done by starting the Pilot component with the additional command line arguments --appNamespace, ns-1. Where ns-1 is the namespace that the tenant’s application will be deployed in. An example snippet from the istio-system1.yaml file is shown below.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-pilot
namespace: istio-system1
annotations:
sidecar.istio.io/inject: "false"
spec:
replicas: 1
template:
metadata:
labels:
istio: pilot
spec:
serviceAccountName: istio-pilot-service-account
containers:
- name: discovery
image: docker.io/<user ID>/pilot:<tag>
imagePullPolicy: IfNotPresent
args: ["discovery", "-v", "2", "--admission-service", "istio-pilot", "--appNamespace", "ns-1"]
ports:
- containerPort: 8080
- containerPort: 443

Deploying the tenant application in a namespace

Now that the cluster administrator has created the tenant’s namespace (ex. istio-system1) and Pilot’s service discovery has been configured to watch for a specific application namespace (ex. ns-1), create the application manifests to deploy in that tenant’s specific namespace. For example:

apiVersion: v1
kind: Namespace
metadata:
name: ns-1

And add the namespace reference to each resource type included in the application’s manifest file. For example:

apiVersion: v1
kind: Service
metadata:
name: details
labels:
app: details
namespace: ns-1

Although not shown, the application namespaces will also have RBAC settings limiting access to certain resources. These RBAC settings could be set by the cluster administrator and/or the tenant administrator.

Using `kubectl` in a multi-tenant environment

When defining route rules or destination policies, it is necessary to ensure that the kubectl command is scoped to the namespace the Istio control plane is running in to ensure the resource is created in the proper namespace. Additionally, the rule itself must be scoped to the tenant’s namespace so that it will be applied properly to that tenant’s mesh. The -i option is used to create (or get or describe) the rule in the namespace that the Istio control plane is deployed in. The -n option will scope the rule to the tenant’s mesh and should be set to the namespace that the tenant’s app is deployed in. Note that the -n option can be skipped on the command line if the .yaml file for the resource scopes it properly instead.

For example, the following command would be required to add a route rule to the istio-system1 namespace:

$ kubectl –i istio-system1 apply -n ns-1 -f route_rule_v2.yaml

And can be displayed using the command:

$ kubectl -i istio-system1 -n ns-1 get routerule
NAME KIND NAMESPACE
details-Default RouteRule.v1alpha2.config.istio.io ns-1
productpage-default RouteRule.v1alpha2.config.istio.io ns-1
ratings-default RouteRule.v1alpha2.config.istio.io ns-1
reviews-default RouteRule.v1alpha2.config.istio.io ns-1

See the Multiple Istio control planes section of this document for more details on namespace requirements in a multi-tenant environment.

Test results

Following the instructions above, a cluster administrator can create an environment limiting, via RBAC and namespaces, what a tenant administrator can deploy.

After deployment, accessing the Istio control plane pods assigned to a specific tenant administrator is permitted:

$ kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-78d649479f-8pqk9 1/1 Running 0 1d
istio-ca-ffbb75c6f-98w6x 1/1 Running 0 1d
istio-ingress-68d65fc5c6-dnvfl 1/1 Running 0 1d
istio-mixer-5b9f8dffb5-8875r 3/3 Running 0 1d
istio-pilot-678fc976c8-b8tv6 2/2 Running 0 1d
istio-sidecar-injector-7587bd559d-5tgk6 1/1 Running 0 1d
prometheus-cf8456855-hdcq7 1/1 Running 0 1d

However, accessing all the cluster’s pods is not permitted:

$ kubectl get pods --all-namespaces
Error from server (Forbidden): pods is forbidden: User "dev-admin" cannot list pods at the cluster scope

And neither is accessing another tenant’s namespace:

$ kubectl get pods -n istio-system1
Error from server (Forbidden): pods is forbidden: User "dev-admin" cannot list pods in the namespace "istio-system1"

The tenant administrator can deploy applications in the application namespace configured for that tenant. As an example, updating the Bookinfo manifests and then deploying under the tenant’s application namespace of ns-0, listing the pods in use by this tenant’s namespace is permitted:

$ kubectl get pods -n ns-0
NAME READY STATUS RESTARTS AGE
details-v1-64b86cd49-b7rkr 2/2 Running 0 1d
productpage-v1-84f77f8747-rf2mt 2/2 Running 0 1d
ratings-v1-5f46655b57-5b4c5 2/2 Running 0 1d
reviews-v1-ff6bdb95b-pm5lb 2/2 Running 0 1d
reviews-v2-5799558d68-b989t 2/2 Running 0 1d
reviews-v3-58ff7d665b-lw5j9 2/2 Running 0 1d

But accessing another tenant’s application namespace is not:

$ kubectl get pods -n ns-1
Error from server (Forbidden): pods is forbidden: User "dev-admin" cannot list pods in the namespace "ns-1"

If the add-on tools, example Prometheus, are deployed (also limited by an Istio namespace) the statistical results returned would represent only that traffic seen from that tenant’s application namespace.

Conclusion

The evaluation performed indicates Istio has sufficient capabilities and security to meet a small number of multi-tenant use cases. It also shows that Istio and Kubernetes cannot provide sufficient capabilities and security for other use cases, especially those use cases that require complete security and isolation between untrusted tenants. The improvements required to reach a more secure model of security and isolation require work in container technology, ex. Kubernetes, rather than improvements in Istio capabilities.

Issues

The CA (Certificate Authority) and Mixer pod logs from one tenant’s Istio control plane (e.g. istio-system namespace) contained ‘info’ messages from a second tenant’s Istio control plane (e.g. istio-system1 namespace).

Challenges with other multi-tenancy models

Other multi-tenancy deployment models were considered:

A single mesh with multiple applications, one for each tenant on the mesh. The cluster administrator gets control and visibility mesh wide and across all applications, while the tenant administrator only gets control of a specific application.
A single Istio control plane with multiple meshes, one mesh per tenant. The cluster administrator gets control and visibility across the entire Istio control plane and all meshes, while the tenant administrator only gets control of a specific mesh.
A single cloud environment (cluster controlled), but multiple Kubernetes control planes (tenant controlled).

These options either can’t be properly supported without code changes or don’t fully address the use cases.

Current Istio capabilities are poorly suited to support the first model as it lacks sufficient RBAC capabilities to support cluster versus tenant operations. Additionally, having multiple tenants under one mesh is too insecure with the current mesh model and the way Istio drives configuration to the Envoy proxies.

Regarding the second option, the current Istio paradigm assumes a single mesh per Istio control plane. The needed changes to support this model are substantial. They would require finer grained scoping of resources and security domains based on namespaces, as well as, additional Istio RBAC changes. This model will likely be addressed by future work, but not currently possible.

The third model doesn’t satisfy most use cases, as most cluster administrators prefer a common Kubernetes control plane which they provide as a PaaS to their tenants.

Future work

Allowing a single Istio control plane to control multiple meshes would be an obvious next feature. An additional improvement is to provide a single mesh that can host different tenants with some level of isolation and security between the tenants. This could be done by partitioning within a single control plane using the same logical notion of namespace as Kubernetes. A document has been started within the Istio community to define additional use cases and the Istio functionality required to support those use cases.

References

Video on Kubernetes multi-tenancy support, Multi-Tenancy Support & Security Modeling with RBAC and Namespaces, and the supporting slide deck.
KubeCon talk on security that discusses Kubernetes support for “Cooperative soft multi-tenancy”, Building for Trust: How to Secure Your Kubernetes.
Kubernetes documentation on RBAC and namespaces.
KubeCon slide deck on Multi-tenancy Deep Dive.
Google document on Multi-tenancy models for Kubernetes. (Requires permission)
Cloud Foundry WIP document, Multi-cloud and Multi-tenancy
Istio Auto Multi-Tenancy 101

Traffic Mirroring with Istio for Testing in Production

Christian Posta — Thu, 08 Feb 2018 00:00:00 +0000

Trying to enumerate all the possible combinations of test cases for testing services in non-production/test environments can be daunting. In some cases, you’ll find that all of the effort that goes into cataloging these use cases doesn’t match up to real production use cases. Ideally, we could use live production use cases and traffic to help illuminate all of the feature areas of the service under test that we might miss in more contrived testing environments.

Istio can help here. With the release of Istio 0.5, Istio can mirror traffic to help test your services. You can write route rules similar to the following to enable traffic mirroring:

apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
name: mirror-traffic-to-httbin-v2
spec:
destination:
name: httpbin
precedence: 11
route:
- labels:
version: v1
weight: 100
- labels:
version: v2
weight: 0
mirror:
name: httpbin
labels:
version: v2

A few things to note here:

When traffic gets mirrored to a different service, that happens outside the critical path of the request
Responses to any mirrored traffic is ignored; traffic is mirrored as “fire-and-forget”
You’ll need to have the 0-weighted route to hint to Istio to create the proper Envoy cluster under the covers; this should be ironed out in future releases.

Learn more about mirroring by visiting the Mirroring Task and see a more comprehensive treatment of this scenario on my blog.

Consuming External TCP Services

Vadim Eisenberg — Tue, 06 Feb 2018 00:00:00 +0000

In my previous blog post, Consuming External Web Services, I described how external services can be consumed by in-mesh Istio applications via HTTPS. In this post, I demonstrate consuming external services over TCP. You will use the Istio Bookinfo sample application, the version in which the book ratings data is persisted in a MySQL database. You deploy this database outside the cluster and configure the ratings microservice to use it. You define a Service Entry to allow the in-mesh applications to access the external database.

Bookinfo sample application with external ratings database

First, you set up a MySQL database instance to hold book ratings data outside of your Kubernetes cluster. Then you modify the Bookinfo sample application to use your database.

Setting up the database for ratings data

For this task you set up an instance of MySQL. You can use any MySQL instance; I used Compose for MySQL. I used mysqlsh (MySQL Shell) as a MySQL client to feed the ratings data.

Set the MYSQL_DB_HOST and MYSQL_DB_PORT environment variables:
```
$ export MYSQL_DB_HOST=<your MySQL database host>
$ export MYSQL_DB_PORT=<your MySQL database port>
```
In case of a local MySQL database with the default port, the values are localhost and 3306, respectively.

To initialize the database, run the following command entering the password when prompted. The command is performed with the credentials of the admin user, created by default by Compose for MySQL.

$ curl -s https://raw.githubusercontent.com/istio/istio/release-1.9/samples/bookinfo/src/mysql/mysqldb-init.sql | mysqlsh --sql --ssl-mode=REQUIRED -u admin -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT

When using the mysql client and a local MySQL database, run:

$ curl -s https://raw.githubusercontent.com/istio/istio/release-1.9/samples/bookinfo/src/mysql/mysqldb-init.sql | mysql -u root -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT

Create a user with the name bookinfo and grant it SELECT privilege on the test.ratings table:
```
$ mysqlsh --sql --ssl-mode=REQUIRED -u admin -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "CREATE USER 'bookinfo' IDENTIFIED BY '<password you choose>'; GRANT SELECT ON test.ratings to 'bookinfo';"
```
OR

For mysql and the local database, the command is:
```
$ mysql -u root -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "CREATE USER 'bookinfo' IDENTIFIED BY '<password you choose>'; GRANT SELECT ON test.ratings to 'bookinfo';"
```
Here you apply the principle of least privilege. This means that you do not use your admin user in the Bookinfo application. Instead, you create a special user for the Bookinfo application , bookinfo, with minimal privileges. In this case, the bookinfo user only has the SELECT privilege on a single table.

After running the command to create the user, you may want to clean your bash history by checking the number of the last command and running history -d <the number of the command that created the user>. You don’t want the password of the new user to be stored in the bash history. If you’re using mysql, remove the last command from ~/.mysql_history file as well. Read more about password protection of the newly created user in MySQL documentation.

Inspect the created ratings to see that everything worked as expected:

$ mysqlsh --sql --ssl-mode=REQUIRED -u bookinfo -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "select * from test.ratings;"
Enter password:
+----------+--------+
| ReviewID | Rating |
+----------+--------+
| 1 | 5 |
| 2 | 4 |
+----------+--------+

For mysql and the local database:

$ mysql -u bookinfo -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "select * from test.ratings;"
Enter password:
+----------+--------+
| ReviewID | Rating |
+----------+--------+
| 1 | 5 |
| 2 | 4 |
+----------+--------+

Set the ratings temporarily to 1 to provide a visual clue when our database is used by the Bookinfo ratings service:

$ mysqlsh --sql --ssl-mode=REQUIRED -u admin -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "update test.ratings set rating=1; select * from test.ratings;"
Enter password:
Rows matched: 2 Changed: 2 Warnings: 0
+----------+--------+
| ReviewID | Rating |
+----------+--------+
| 1 | 1 |
| 2 | 1 |
+----------+--------+

For mysql and the local database:

$ mysql -u root -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "update test.ratings set rating=1; select * from test.ratings;"
Enter password:
+----------+--------+
| ReviewID | Rating |
+----------+--------+
| 1 | 1 |
| 2 | 1 |
+----------+--------+

You used the admin user (and root for the local database) in the last command since the bookinfo user does not have the UPDATE privilege on the test.ratings table.

Now you are ready to deploy a version of the Bookinfo application that will use your database.

Initial setting of Bookinfo application

This application uses the ratings microservice to fetch book ratings, a number between 1 and 5. The ratings are displayed as stars for each review. There are several versions of the ratings microservice. Some use MongoDB, others use MySQL as their database.

The example commands in this blog post work with Istio 0.8+, with or without mutual TLS enabled.

As a reminder, here is the end-to-end architecture of the application from the Bookinfo sample application.

The original Bookinfo application

Use the database for ratings data in Bookinfo application

Modify the deployment spec of a version of the ratings microservice that uses a MySQL database, to use your database instance. The spec is in samples/bookinfo/platform/kube/bookinfo-ratings-v2-mysql.yaml of an Istio release archive. Edit the following lines:
```
- name: MYSQL_DB_HOST
value: mysqldb
- name: MYSQL_DB_PORT
value: "3306"
- name: MYSQL_DB_USER
value: root
- name: MYSQL_DB_PASSWORD
value: password
```
Replace the values in the snippet above, specifying the database host, port, user, and password. Note that the correct way to work with passwords in container’s environment variables in Kubernetes is to use secrets. For this example task only, you may want to write the password directly in the deployment spec. Do not do it in a real environment! I also assume everyone realizes that "password" should not be used as a password…

Apply the modified spec to deploy the version of the ratings microservice, v2-mysql, that will use your database.

$ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo-ratings-v2-mysql.yaml@
deployment "ratings-v2-mysql" created

Route all the traffic destined to the reviews service to its v3 version. You do this to ensure that the reviews service always calls the ratings service. In addition, route all the traffic destined to the ratings service to ratings v2-mysql that uses your database.

Specify the routing for both services above by adding two virtual services. These virtual services are specified in samples/bookinfo/networking/virtual-service-ratings-mysql.yaml of an Istio release archive. Important: make sure you applied the default destination rules before running the following command.
Zip
```
$ kubectl apply -f @samples/bookinfo/networking/virtual-service-ratings-mysql.yaml@
```

The Bookinfo application with ratings v2-mysql and an external MySQL database

Note that the MySQL database is outside the Istio service mesh, or more precisely outside the Kubernetes cluster. The boundary of the service mesh is marked by a dashed line.

Access the webpage

Access the webpage of the application, after determining the ingress IP and port.

You have a problem… Instead of the rating stars, the message “Ratings service is currently unavailable” is currently displayed below each review:

The Ratings service error messages

As in Consuming External Web Services, you experience graceful service degradation, which is good. The application did not crash due to the error in the ratings microservice. The webpage of the application correctly displayed the book information, the details, and the reviews, just without the rating stars.

You have the same problem as in Consuming External Web Services, namely all the traffic outside the Kubernetes cluster, both TCP and HTTP, is blocked by default by the sidecar proxies. To enable such traffic for TCP, a mesh-external service entry for TCP must be defined.

Mesh-external service entry for an external MySQL instance

TCP mesh-external service entries come to our rescue.

Get the IP address of your MySQL database instance. As an option, you can use the host command:
```
$ export MYSQL_DB_IP=$(host $MYSQL_DB_HOST | grep " has address " | cut -d" " -f4)
```
For a local database, set MYSQL_DB_IP to contain the IP of your machine, accessible from your cluster.

Define a TCP mesh-external service entry:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: mysql-external
spec:
hosts:
- $MYSQL_DB_HOST
addresses:
- $MYSQL_DB_IP/32
ports:
- name: tcp
number: $MYSQL_DB_PORT
protocol: tcp
location: MESH_EXTERNAL
EOF

Review the service entry you just created and check that it contains the correct values:

$ kubectl get serviceentry mysql-external -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
...

Note that for a TCP service entry, you specify tcp as the protocol of a port of the entry. Also note that you have to specify the IP of the external service in the list of addresses, as a CIDR block with suffix 32.

I will talk more about TCP service entries below. For now, verify that the service entry we added fixed the problem. Access the webpage and see if the stars are back.

It worked! Accessing the web page of the application displays the ratings without error:

Book Ratings Displayed Correctly

Note that you see a one-star rating for both displayed reviews, as expected. You changed the ratings to be one star to provide us with a visual clue that our external database is indeed being used.

As with service entries for HTTP/HTTPS, you can delete and create service entries for TCP using kubectl, dynamically.

Motivation for egress TCP traffic control

Some in-mesh Istio applications must access external services, for example legacy systems. In many cases, the access is not performed over HTTP or HTTPS protocols. Other TCP protocols are used, such as database-specific protocols like MongoDB Wire Protocol and MySQL Client/Server Protocol to communicate with external databases.

Next let me provide more details about the service entries for TCP traffic.

Service entries for TCP traffic

The service entries for enabling TCP traffic to a specific port must specify TCP as the protocol of the port. Additionally, for the MongoDB Wire Protocol, the protocol can be specified as MONGO, instead of TCP.

For the addresses field of the entry, a block of IPs in CIDR notation must be used. Note that the hosts field is ignored for TCP service entries.

To enable TCP traffic to an external service by its hostname, all the IPs of the hostname must be specified. Each IP must be specified by a CIDR block.

Note that all the IPs of an external service are not always known. To enable egress TCP traffic, only the IPs that are used by the applications must be specified.

Also note that the IPs of an external service are not always static, for example in the case of CDNs. Sometimes the IPs are static most of the time, but can be changed from time to time, for example due to infrastructure changes. In these cases, if the range of the possible IPs is known, you should specify the range by CIDR blocks. If the range of the possible IPs is not known, service entries for TCP cannot be used and the external services must be called directly, bypassing the sidecar proxies.

Relation to virtual machines support

Note that the scenario described in this post is different from the Bookinfo with Virtual Machines example. In that scenario, a MySQL instance runs on an external (outside the cluster) machine (a bare metal or a VM), integrated with the Istio service mesh. The MySQL service becomes a first-class citizen of the mesh with all the beneficial features of Istio applicable. Among other things, the service becomes addressable by a local cluster domain name, for example by mysqldb.vm.svc.cluster.local, and the communication to it can be secured by mutual TLS authentication. There is no need to create a service entry to access this service; however, the service must be registered with Istio. To enable such integration, Istio components (Envoy proxy, node-agent, _istio-agent_) must be installed on the machine and the Istio control plane (Pilot, Mixer, Citadel) must be accessible from it. See the Istio VM-related tasks for more details.

In our case, the MySQL instance can run on any machine or can be provisioned as a service by a cloud provider. There is no requirement to integrate the machine with Istio. The Istio control plane does not have to be accessible from the machine. In the case of MySQL as a service, the machine which MySQL runs on may be not accessible and installing on it the required components may be impossible. In our case, the MySQL instance is addressable by its global domain name, which could be beneficial if the consuming applications expect to use that domain name. This is especially relevant when that expected domain name cannot be changed in the deployment configuration of the consuming applications.

Cleanup

Drop the test database and the bookinfo user:

$ mysqlsh --sql --ssl-mode=REQUIRED -u admin -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "drop database test; drop user bookinfo;"

For mysql and the local database:

$ mysql -u root -p --host $MYSQL_DB_HOST --port $MYSQL_DB_PORT -e "drop database test; drop user bookinfo;"

Remove the virtual services:

$ kubectl delete -f @samples/bookinfo/networking/virtual-service-ratings-mysql.yaml@
Deleted config: virtual-service/default/reviews
Deleted config: virtual-service/default/ratings

Undeploy ratings v2-mysql:

$ kubectl delete -f @samples/bookinfo/platform/kube/bookinfo-ratings-v2-mysql.yaml@
deployment "ratings-v2-mysql" deleted

Delete the service entry:

$ kubectl delete serviceentry mysql-external -n default
Deleted config: serviceentry mysql-external

Conclusion

In this blog post, I demonstrated how the microservices in an Istio service mesh can consume external services via TCP. By default, Istio blocks all the traffic, TCP and HTTP, to the hosts outside the cluster. To enable such traffic for TCP, TCP mesh-external service entries must be created for the service mesh.

Consuming External Web Services

Vadim Eisenberg — Wed, 31 Jan 2018 00:00:00 +0000

In many cases, not all the parts of a microservices-based application reside in a service mesh. Sometimes, the microservices-based applications use functionality provided by legacy systems that reside outside the mesh. You may want to migrate these systems to the service mesh gradually. Until these systems are migrated, they must be accessed by the applications inside the mesh. In other cases, the applications use web services provided by third parties.

In this blog post, I modify the Istio Bookinfo Sample Application to fetch book details from an external web service (Google Books APIs). I show how to enable egress HTTPS traffic in Istio by using mesh-external service entries. I provide two options for egress HTTPS traffic and describe the pros and cons of each of the options.

Initial setting

To demonstrate the scenario of consuming an external web service, I start with a Kubernetes cluster with Istio installed. Then I deploy Istio Bookinfo Sample Application. This application uses the details microservice to fetch book details, such as the number of pages and the publisher. The original details microservice provides the book details without consulting any external service.

The example commands in this blog post work with Istio 1.0+, with or without mutual TLS enabled. The Bookinfo configuration files reside in the samples/bookinfo directory of the Istio release archive.

Here is a copy of the end-to-end architecture of the application from the original Bookinfo sample application.

The Original Bookinfo Application

Perform the steps in the Deploying the application, Confirm the app is running, Apply default destination rules sections, and change Istio to the blocking-egress-by-default policy.

Bookinfo with HTTPS access to a Google Books web service

Deploy a new version of the details microservice, v2, that fetches the book details from Google Books APIs. Run the following command; it sets the DO_NOT_ENCRYPT environment variable of the service’s container to false. This setting will instruct the deployed service to use HTTPS (instead of HTTP) to access to the external service.

$ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo-details-v2.yaml@ --dry-run -o yaml | kubectl set env --local -f - 'DO_NOT_ENCRYPT=false' -o yaml | kubectl apply -f -

The updated architecture of the application now looks as follows:

The Bookinfo Application with details V2

Note that the Google Books web service is outside the Istio service mesh, the boundary of which is marked by a dashed line.

Now direct all the traffic destined to the details microservice, to details version v2.

$ kubectl apply -f @samples/bookinfo/networking/virtual-service-details-v2.yaml@

Note that the virtual service relies on a destination rule that you created in the Apply default destination rules section.

Access the web page of the application, after determining the ingress IP and port.

Oops… Instead of the book details you have the Error fetching product details message displayed:

The Error Fetching Product Details Message

The good news is that your application did not crash. With a good microservice design, you do not have failure propagation. In your case, the failing details microservice does not cause the productpage microservice to fail. Most of the functionality of the application is still provided, despite the failure in the details microservice. You have graceful service degradation: as you can see, the reviews and the ratings are displayed correctly, and the application is still useful.

So what might have gone wrong? Ah… The answer is that I forgot to tell you to enable traffic from inside the mesh to an external service, in this case to the Google Books web service. By default, the Istio sidecar proxies (Envoy proxies) block all the traffic to destinations outside the cluster. To enable such traffic, you must define a mesh-external service entry.

Enable HTTPS access to a Google Books web service

No worries, define a mesh-external service entry and fix your application. You must also define a virtual service to perform routing by SNI to the external service.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: googleapis
spec:
hosts:
- www.googleapis.com
ports:
- number: 443
name: https
protocol: HTTPS
location: MESH_EXTERNAL
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: googleapis
spec:
hosts:
- www.googleapis.com
tls:
- match:
- port: 443
sni_hosts:
- www.googleapis.com
route:
- destination:
host: www.googleapis.com
port:
number: 443
weight: 100
EOF

Now accessing the web page of the application displays the book details without error:

Book Details Displayed Correctly

You can query your service entries:

$ kubectl get serviceentries
NAME AGE
googleapis 8m

You can delete your service entry:

$ kubectl delete serviceentry googleapis
serviceentry "googleapis" deleted

and see in the output that the service entry is deleted.

Accessing the web page after deleting the service entry produces the same error that you experienced before, namely Error fetching product details. As you can see, the service entries are defined dynamically, as are many other Istio configuration artifacts. The Istio operators can decide dynamically which domains they allow the microservices to access. They can enable and disable traffic to the external domains on the fly, without redeploying the microservices.

Cleanup of HTTPS access to a Google Books web service

$ kubectl delete serviceentry googleapis
$ kubectl delete virtualservice googleapis
$ kubectl delete -f @samples/bookinfo/networking/virtual-service-details-v2.yaml@
$ kubectl delete -f @samples/bookinfo/platform/kube/bookinfo-details-v2.yaml@

TLS origination by Istio

There is a caveat to this story. Suppose you want to monitor which specific set of Google APIs your microservices use (Books, Calendar, Tasks etc.) Suppose you want to enforce a policy that using only Books APIs is allowed. Suppose you want to monitor the book identifiers that your microservices access. For these monitoring and policy tasks you need to know the URL path. Consider for example the URL www.googleapis.com/books/v1/volumes?q=isbn:0486424618. In that URL, Books APIs is specified by the path segment /books, and the ISBN number by the path segment /volumes?q=isbn:0486424618. However, in HTTPS, all the HTTP details (hostname, path, headers etc.) are encrypted and such monitoring and policy enforcement by the sidecar proxies is not possible. Istio can only know the server name of the encrypted requests by the SNI (Server Name Indication) field, in this case www.googleapis.com.

To allow Istio to perform monitoring and policy enforcement of egress requests based on HTTP details, the microservices must issue HTTP requests. Istio then opens an HTTPS connection to the destination (performs TLS origination). The code of the microservices must be written differently or configured differently, according to whether the microservice runs inside or outside an Istio service mesh. This contradicts the Istio design goal of maximizing transparency. Sometimes you need to compromise…

The diagram below shows two options for sending HTTPS traffic to external services. On the top, a microservice sends regular HTTPS requests, encrypted end-to-end. On the bottom, the same microservice sends unencrypted HTTP requests inside a pod, which are intercepted by the sidecar Envoy proxy. The sidecar proxy performs TLS origination, so the traffic between the pod and the external service is encrypted.

HTTPS traffic to external services, with TLS originated by the microservice vs. by the sidecar proxy

Here is how both patterns are supported in the Bookinfo details microservice code, using the Ruby net/http module:

uri = URI.parse('https://www.googleapis.com/books/v1/volumes?q=isbn:' + isbn)
http = Net::HTTP.new(uri.host, ENV['DO_NOT_ENCRYPT'] === 'true' ? 80:443)
...
unless ENV['DO_NOT_ENCRYPT'] === 'true' then
http.use_ssl = true
end

When the DO_NOT_ENCRYPT environment variable is defined, the request is performed without SSL (plain HTTP) to port 80.

You can set the DO_NOT_ENCRYPT environment variable to “true” in the Kubernetes deployment spec of details v2, the container section:

env:
- name: DO_NOT_ENCRYPT
value: "true"

In the next section you will configure TLS origination for accessing an external web service.

Bookinfo with TLS origination to a Google Books web service

Deploy a version of details v2 that sends an HTTP request to Google Books APIs. The DO_NOT_ENCRYPT variable is set to true in bookinfo-details-v2.yaml.
Zip
```
$ kubectl apply -f @samples/bookinfo/platform/kube/bookinfo-details-v2.yaml@
```

Direct the traffic destined to the details microservice, to details version v2.

$ kubectl apply -f @samples/bookinfo/networking/virtual-service-details-v2.yaml@

Create a mesh-external service entry for www.google.apis , a virtual service to rewrite the destination port from 80 to 443, and a destination rule to perform TLS origination.

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: googleapis
spec:
hosts:
- www.googleapis.com
ports:
- number: 80
name: http
protocol: HTTP
- number: 443
name: https
protocol: HTTPS
resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: rewrite-port-for-googleapis
spec:
hosts:
- www.googleapis.com
http:
- match:
- port: 80
route:
- destination:
host: www.googleapis.com
port:
number: 443
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: originate-tls-for-googleapis
spec:
host: www.googleapis.com
trafficPolicy:
loadBalancer:
simple: ROUND_ROBIN
portLevelSettings:
- port:
number: 443
tls:
mode: SIMPLE # initiates HTTPS when accessing www.googleapis.com
EOF

Access the web page of the application and verify that the book details are displayed without errors.
Enable Envoy’s access logging

Check the log of of the sidecar proxy of details v2 and see the HTTP request.

$ kubectl logs $(kubectl get pods -l app=details -l version=v2 -o jsonpath='{.items[0].metadata.name}') istio-proxy | grep googleapis
[2018-08-09T11:32:58.171Z] "GET /books/v1/volumes?q=isbn:0486424618 HTTP/1.1" 200 - 0 1050 264 264 "-" "Ruby" "b993bae7-4288-9241-81a5-4cde93b2e3a6" "www.googleapis.com:80" "172.217.20.74:80"
EOF

Note the URL path in the log, the path can be monitored and access policies can be applied based on it. To read more about monitoring and access policies for HTTP egress traffic, check out this blog post.

Cleanup of TLS origination to a Google Books web service

$ kubectl delete serviceentry googleapis
$ kubectl delete virtualservice rewrite-port-for-googleapis
$ kubectl delete destinationrule originate-tls-for-googleapis
$ kubectl delete -f @samples/bookinfo/networking/virtual-service-details-v2.yaml@
$ kubectl delete -f @samples/bookinfo/platform/kube/bookinfo-details-v2.yaml@

Relation to Istio mutual TLS

Note that the TLS origination in this case is unrelated to the mutual TLS applied by Istio. The TLS origination for the external services will work, whether the Istio mutual TLS is enabled or not. The mutual TLS secures service-to-service communication inside the service mesh and provides each service with a strong identity. The external services in this blog post were accessed using one-way TLS, the same mechanism used to secure communication between a web browser and a web server. TLS is applied to the communication with external services to verify the identity of the external server and to encrypt the traffic.

Conclusion

In this blog post I demonstrated how microservices in an Istio service mesh can consume external web services by HTTPS. By default, Istio blocks all the traffic to the hosts outside the cluster. To enable such traffic, mesh-external service entries must be created for the service mesh. It is possible to access the external sites either by issuing HTTPS requests, or by issuing HTTP requests with Istio performing TLS origination. When the microservices issue HTTPS requests, the traffic is encrypted end-to-end, however Istio cannot monitor HTTP details like the URL paths of the requests. When the microservices issue HTTP requests, Istio can monitor the HTTP details of the requests and enforce HTTP-based access policies. However, in that case the traffic between microservice and the sidecar proxy is unencrypted. Having part of the traffic unencrypted can be forbidden in organizations with very strict security requirements.

Mixer and the SPOF Myth

Martin Taillefer — Thu, 07 Dec 2017 00:00:00 +0000

As Mixer is in the request path, it is natural to question how it impacts overall system availability and latency. A common refrain we hear when people first glance at Istio architecture diagrams is “Isn’t this just introducing a single point of failure?”

In this post, we’ll dig deeper and cover the design principles that underpin Mixer and the surprising fact Mixer actually increases overall mesh availability and reduces average request latency.

Istio’s use of Mixer has two main benefits in terms of overall system availability and latency:

Increased SLO. Mixer insulates proxies and services from infrastructure backend failures, enabling higher effective mesh availability. The mesh as a whole tends to experience a lower rate of failure when interacting with the infrastructure backends than if Mixer were not present.
Reduced Latency. Through aggressive use of shared multi-level caches and sharding, Mixer reduces average observed latencies across the mesh.

We’ll explain this in more detail below.

How we got here

For many years at Google, we’ve been using an internal API & service management system to handle the many APIs exposed by Google. This system has been fronting the world’s biggest services (Google Maps, YouTube, Gmail, etc) and sustains a peak rate of hundreds of millions of QPS. Although this system has served us well, it had problems keeping up with Google’s rapid growth, and it became clear that a new architecture was needed in order to tamp down ballooning operational costs.

In 2014, we started an initiative to create a replacement architecture that would scale better. The result has proven extremely successful and has been gradually deployed throughout Google, saving in the process millions of dollars a month in ops costs.

The older system was built around a centralized fleet of fairly heavy proxies into which all incoming traffic would flow, before being forwarded to the services where the real work was done. The newer architecture jettisons the shared proxy design and instead consists of a very lean and efficient distributed sidecar proxy sitting next to service instances, along with a shared fleet of sharded control plane intermediaries:

Google's API & Service Management System

Look familiar? Of course: it’s just like Istio! Istio was conceived as a second generation of this distributed proxy architecture. We took the core lessons from this internal system, generalized many of the concepts by working with our partners, and created Istio.

Architecture recap

As shown in the diagram below, Mixer sits between the mesh and the infrastructure backends that support it:

Istio Topology

The Envoy sidecar logically calls Mixer before each request to perform precondition checks, and after each request to report telemetry. The sidecar has local caching such that a relatively large percentage of precondition checks can be performed from cache. Additionally, the sidecar buffers outgoing telemetry such that it only actually needs to call Mixer once for every several thousands requests. Whereas precondition checks are synchronous to request processing, telemetry reports are done asynchronously with a fire-and-forget pattern.

At a high level, Mixer provides:

Backend Abstraction. Mixer insulates the Istio components and services within the mesh from the implementation details of individual infrastructure backends.
Intermediation. Mixer allows operators to have fine-grained control over all interactions between the mesh and the infrastructure backends.

However, even beyond these purely functional aspects, Mixer has other characteristics that provide the system with additional benefits.

Mixer: SLO booster

Contrary to the claim that Mixer is a SPOF and can therefore lead to mesh outages, we believe it in fact improves the effective availability of a mesh. How can that be? There are three basic characteristics at play:

Statelessness. Mixer is stateless in that it doesn’t manage any persistent storage of its own.
Hardening. Mixer proper is designed to be a highly reliable component. The design intent is to achieve > 99.999% uptime for any individual Mixer instance.
Caching and Buffering. Mixer is designed to accumulate a large amount of transient ephemeral state.

The sidecar proxies that sit next to each service instance in the mesh must necessarily be frugal in terms of memory consumption, which constrains the possible amount of local caching and buffering. Mixer, however, lives independently and can use considerably larger caches and output buffers. Mixer thus acts as a highly-scaled and highly-available second-level cache for the sidecars.

Mixer’s expected availability is considerably higher than most infrastructure backends (those often have availability of perhaps 99.9%). Its local caches and buffers help mask infrastructure backend failures by being able to continue operating even when a backend has become unresponsive.

Mixer: Latency slasher

As we explained above, the Istio sidecars generally have fairly effective first-level caching. They can serve the majority of their traffic from cache. Mixer provides a much greater shared pool of second-level cache, which helps Mixer contribute to a lower average per-request latency.

While it’s busy cutting down latency, Mixer is also inherently cutting down the number of calls your mesh makes to infrastructure backends. Depending on how you’re paying for these backends, this might end up saving you some cash by cutting down the effective QPS to the backends.

Work ahead

We have opportunities ahead to continue improving the system in many ways.

Configuration canaries

Mixer is highly scaled so it is generally resistant to individual instance failures. However, Mixer is still susceptible to cascading failures in the case when a poison configuration is deployed which causes all Mixer instances to crash basically at the same time (yeah, that would be a bad day). To prevent this from happening, configuration changes can be canaried to a small set of Mixer instances, and then more broadly rolled out.

Mixer doesn’t yet do canarying of configuration changes, but we expect this to come online as part of Istio’s ongoing work on reliable configuration distribution.

Cache tuning

We have yet to fine-tune the sizes of the sidecar and Mixer caches. This work will focus on achieving the highest performance possible using the least amount of resources.

At the moment, each Mixer instance operates independently of all other instances. A request handled by one Mixer instance will not leverage data cached in a different instance. We will eventually experiment with a distributed cache such as memcached or Redis in order to provide a much larger mesh-wide shared cache, and further reduce the number of calls to infrastructure backends.

Sharding

In very large meshes, the load on Mixer can be great. There can be a large number of Mixer instances, each straining to keep caches primed to satisfy incoming traffic. We expect to eventually introduce intelligent sharding such that Mixer instances become slightly specialized in handling particular data streams in order to increase the likelihood of cache hits. In other words, sharding helps improve cache efficiency by routing related traffic to the same Mixer instance over time, rather than randomly dispatching to any available Mixer instance.

Conclusion

Practical experience at Google showed that the model of a slim sidecar proxy and a large shared caching control plane intermediary hits a sweet spot, delivering excellent perceived availability and latency. We’ve taken the lessons learned there and applied them to create more sophisticated and effective caching, prefetching, and buffering strategies in Istio. We’ve also optimized the communication protocols to reduce overhead when a cache miss does occur.

Mixer is still young. As of Istio 0.3, we haven’t really done significant performance work within Mixer itself. This means when a request misses the sidecar cache, we spend more time in Mixer to respond to requests than we should. We’re doing a lot of work to improve this in coming months to reduce the overhead that Mixer imparts in the synchronous precondition check case.

We hope this post makes you appreciate the inherent benefits that Mixer brings to Istio. Don’t hesitate to post comments or questions to istio-policies-and-telemetry@.

Mixer Adapter Model

Martin Taillefer — Fri, 03 Nov 2017 00:00:00 +0000

Istio 0.2 introduced a new Mixer adapter model which is intended to increase Mixer’s flexibility to address a varied set of infrastructure backends. This post intends to put the adapter model in context and explain how it works.

Why adapters?

Infrastructure backends provide support functionality used to build services. They include such things as access control systems, telemetry capturing systems, quota enforcement systems, billing systems, and so forth. Services traditionally directly integrate with these backend systems, creating a hard coupling and baking-in specific semantics and usage options.

Mixer serves as an abstraction layer between Istio and an open-ended set of infrastructure backends. The Istio components and services that run within the mesh can interact with these backends, while not being coupled to the backends’ specific interfaces.

In addition to insulating application-level code from the details of infrastructure backends, Mixer provides an intermediation model that allows operators to inject and control policies between application code and backends. Operators can control which data is reported to which backend, which backend to consult for authorization, and much more.

Given that individual infrastructure backends each have different interfaces and operational models, Mixer needs custom code to deal with each and we call these custom bundles of code adapters.

Adapters are Go packages that are directly linked into the Mixer binary. It’s fairly simple to create custom Mixer binaries linked with specialized sets of adapters, in case the default set of adapters is not sufficient for specific use cases.

Philosophy

Mixer is essentially an attribute processing and routing machine. The proxy sends it attributes as part of doing precondition checks and telemetry reports, which it turns into a series of calls into adapters. The operator supplies configuration which describes how to map incoming attributes to inputs for the adapters.

Attribute Machine

Configuration is a complex task. In fact, evidence shows that the overwhelming majority of service outages are caused by configuration errors. To help combat this, Mixer’s configuration model enforces a number of constraints designed to avoid errors. For example, the configuration model uses strong typing to ensure that only meaningful attributes or attribute expressions are used in any given context.

Handlers: configuring adapters

Each adapter that Mixer uses requires some configuration to operate. Typically, adapters need things like the URL to their backend, credentials, caching options, and so forth. Each adapter defines the exact configuration data it needs via a protobuf message.

You configure each adapter by creating handlers for them. A handler is a configuration resource which represents a fully configured adapter ready for use. There can be any number of handlers for a single adapter, making it possible to reuse an adapter in different scenarios.

Templates: adapter input schema

Mixer is typically invoked twice for every incoming request to a mesh service, once for precondition checks and once for telemetry reporting. For every such call, Mixer invokes one or more adapters. Different adapters need different pieces of data as input in order to do their work. A logging adapter needs a log entry, a metric adapter needs a metric, an authorization adapter needs credentials, etc. Mixer templates are used to describe the exact data that an adapter consumes at request time.

Each template is specified as a protobuf message. A single template describes a bundle of data that is delivered to one or more adapters at runtime. Any given adapter can be designed to support any number of templates, the specific templates the adapter supports is determined by the adapter developer.

metric and logentry are two of the most essential templates used within Istio. They represent respectively the payload to report a single metric and a single log entry to appropriate backends.

Instances: attribute mapping

You control which data is delivered to individual adapters by creating instances. Instances control how Mixer uses the attributes delivered by the proxy into individual bundles of data that can be routed to different adapters.

Creating instances generally requires using attribute expressions. The point of these expressions is to use any attribute or literal value in order to produce a result that can be assigned to an instance’s field.

Every instance field has a type, as defined in the template, every attribute has a type, and every attribute expression has a type. You can only assign type-compatible expressions to any given instance fields. For example, you can’t assign an integer expression to a string field. This kind of strong typing is designed to minimize the risk of creating bogus configurations.

Rules: delivering data to adapters

The last piece to the puzzle is telling Mixer which instances to send to which handler and when. This is done by creating rules. Each rule identifies a specific handler and the set of instances to send to that handler. Whenever Mixer processes an incoming call, it invokes the indicated handler and gives it the specific set of instances for processing.

Rules contain matching predicates. A predicate is an attribute expression which returns a true/false value. A rule only takes effect if its predicate expression returns true. Otherwise, it’s like the rule didn’t exist and the indicated handler isn’t invoked.

Future

We are working to improve the end to end experience of using and developing adapters. For example, several new features are planned to make templates more expressive. Additionally, the expression language is being substantially enhanced to be more powerful and well-rounded.

Longer term, we are evaluating ways to support adapters which aren’t directly linked into the main Mixer binary. This would simplify deployment and composition.

Conclusion

The refreshed Mixer adapter model is designed to provide a flexible framework to support an open-ended set of infrastructure backends.

Handlers provide configuration data for individual adapters, templates determine exactly what kind of data different adapters want to consume at runtime, instances let operators prepare this data, rules direct the data to one or more handlers.

You can learn more about Mixer’s overall architecture here, and learn the specifics of templates, handlers, and rules here. You can find many examples of Mixer configuration resources in the Bookinfo sample here.

Using Network Policy with Istio

Spike Curtis — Thu, 10 Aug 2017 00:00:00 +0000

The use of Network Policy to secure applications running on Kubernetes is a now a widely accepted industry best practice. Given that Istio also supports policy, we want to spend some time explaining how Istio policy and Kubernetes Network Policy interact and support each other to deliver your application securely.

Let’s start with the basics: why might you want to use both Istio and Kubernetes Network Policy? The short answer is that they are good at different things. Consider the main differences between Istio and Network Policy (we will describe “typical” implementations, e.g. Calico, but implementation details can vary with different network providers):

	Istio Policy	Network Policy
Layer	“Service” — L7	“Network” — L3-4
Implementation	User space	Kernel
Enforcement Point	Pod	Node

Layer

Istio policy operates at the “service” layer of your network application. This is Layer 7 (Application) from the perspective of the OSI model, but the de facto model of cloud native applications is that Layer 7 actually consists of at least two layers: a service layer and a content layer. The service layer is typically HTTP, which encapsulates the actual application data (the content layer). It is at this service layer of HTTP that the Istio’s Envoy proxy operates. In contrast, Network Policy operates at Layers 3 (Network) and 4 (Transport) in the OSI model.

Operating at the service layer gives the Envoy proxy a rich set of attributes to base policy decisions on, for protocols it understands, which at present includes HTTP/1.1 & HTTP/2 (gRPC operates over HTTP/2). So, you can apply policy based on virtual host, URL, or other HTTP headers. In the future, Istio will support a wide range of Layer 7 protocols, as well as generic TCP and UDP transport.

In contrast, operating at the network layer has the advantage of being universal, since all network applications use IP. At the network layer you can apply policy regardless of the layer 7 protocol: DNS, SQL databases, real-time streaming, and a plethora of other services that do not use HTTP can be secured. Network Policy isn’t limited to a classic firewall’s tuple of IP addresses, proto, and ports. Both Istio and Network Policy are aware of rich Kubernetes labels to describe pod endpoints.

Implementation

The Istio’s proxy is based on Envoy, which is implemented as a user space daemon in the data plane that interacts with the network layer using standard sockets. This gives it a large amount of flexibility in processing, and allows it to be distributed (and upgraded!) in a container.

Network Policy data plane is typically implemented in kernel space (e.g. using iptables, eBPF filters, or even custom kernel modules). Being in kernel space allows them to be extremely fast, but not as flexible as the Envoy proxy.

Enforcement point

Policy enforcement using the Envoy proxy is implemented inside the pod, as a sidecar container in the same network namespace. This allows a simple deployment model. Some containers are given permission to reconfigure the networking inside their pod (CAP_NET_ADMIN). If such a service instance is compromised, or misbehaves (as in a malicious tenant) the proxy can be bypassed.

While this won’t let an attacker access other Istio-enabled pods, so long as they are correctly configured, it opens several attack vectors:

Attacking unprotected pods
Attempting to deny service to protected pods by sending lots of traffic
Exfiltrating data collected in the pod
Attacking the cluster infrastructure (servers or Kubernetes services)
Attacking services outside the mesh, like databases, storage arrays, or legacy systems.

Network Policy is typically enforced at the host node, outside the network namespace of the guest pods. This means that compromised or misbehaving pods must break into the root namespace to avoid enforcement. With the addition of egress policy due in Kubernetes 1.8, this difference makes Network Policy a key part of protecting your infrastructure from compromised workloads.

Examples

Let’s walk through a few examples of what you might want to do with Kubernetes Network Policy for an Istio-enabled application. Consider the Bookinfo sample application. We’re going to cover the following use cases for Network Policy:

Reduce attack surface of the application ingress
Enforce fine-grained isolation within the application

Reduce attack surface of the application ingress

Our application ingress controller is the main entry-point to our application from the outside world. A quick peek at istio.yaml (used to install Istio) defines the Istio ingress like this:

apiVersion: v1
kind: Service
metadata:
name: istio-ingress
labels:
istio: ingress
spec:
type: LoadBalancer
ports:
- port: 80
name: http
- port: 443
name: https
selector:
istio: ingress

The istio-ingress exposes ports 80 and 443. Let’s limit incoming traffic to just these two ports. Envoy has a built-in administrative interface, and we don’t want a misconfigured istio-ingress image to accidentally expose our admin interface to the outside world. This is an example of defense in depth: a properly configured image should not expose the interface, and a properly configured Network Policy will prevent anyone from connecting to it. Either can fail or be misconfigured and we are still protected.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: istio-ingress-lockdown
namespace: default
spec:
podSelector:
matchLabels:
istio: ingress
ingress:
- ports:
- protocol: TCP
port: 80
- protocol: TCP
port: 443

Enforce fine-grained isolation within the application

Here is the service graph for the Bookinfo application.

Bookinfo Service Graph

This graph shows every connection that a correctly functioning application should be allowed to make. All other connections, say from the Istio Ingress directly to the Rating service, are not part of the application. Let’s lock out those extraneous connections so they cannot be used by an attacker. Imagine, for example, that the Ingress pod is compromised by an exploit that allows an attacker to run arbitrary code. If we only allow connections to the Product Page pods using Network Policy, the attacker has gained no more access to my application backends even though they have compromised a member of the service mesh.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: product-page-ingress
namespace: default
spec:
podSelector:
matchLabels:
app: productpage
ingress:
- ports:
- protocol: TCP
port: 9080
from:
- podSelector:
matchLabels:
istio: ingress

You can and should write a similar policy for each service to enforce which other pods are allowed to access each.

Summary

Our take is that Istio and Network Policy have different strengths in applying policy. Istio is application-protocol aware and highly flexible, making it ideal for applying policy in support of operational goals, like service routing, retries, circuit-breaking, etc, and for security that operates at the application layer, such as token validation. Network Policy is universal, highly efficient, and isolated from the pods, making it ideal for applying policy in support of network security goals. Furthermore, having policy that operates at different layers of the network stack is a really good thing as it gives each layer specific context without commingling of state and allows separation of responsibility.

This post is based on the three part blog series by Spike Curtis, one of the Istio team members at Tigera. The full series can be found here: https://www.projectcalico.org/using-network-policy-in-concert-with-istio/

Canary Deployments using Istio

Frank Budinsky — Wed, 14 Jun 2017 00:00:00 +0000

One of the benefits of the Istio project is that it provides the control needed to deploy canary services. The idea behind canary deployment (or rollout) is to introduce a new version of a service by first testing it using a small percentage of user traffic, and then if all goes well, increase, possibly gradually in increments, the percentage while simultaneously phasing out the old version. If anything goes wrong along the way, we abort and roll back to the previous version. In its simplest form, the traffic sent to the canary version is a randomly selected percentage of requests, but in more sophisticated schemes it can be based on the region, user, or other properties of the request.

Depending on your level of expertise in this area, you may wonder why Istio’s support for canary deployment is even needed, given that platforms like Kubernetes already provide a way to do version rollout and canary deployment. Problem solved, right? Well, not exactly. Although doing a rollout this way works in simple cases, it’s very limited, especially in large scale cloud environments receiving lots of (and especially varying amounts of) traffic, where autoscaling is needed.

Canary deployment in Kubernetes

As an example, let’s say we have a deployed service, helloworld version v1, for which we would like to test (or simply roll out) a new version, v2. Using Kubernetes, you can roll out a new version of the helloworld service by simply updating the image in the service’s corresponding Deployment and letting the rollout happen automatically. If we take particular care to ensure that there are enough v1 replicas running when we start and pause the rollout after only one or two v2 replicas have been started, we can keep the canary’s effect on the system very small. We can then observe the effect before deciding to proceed or, if necessary, roll back. Best of all, we can even attach a horizontal pod autoscaler to the Deployment and it will keep the replica ratios consistent if, during the rollout process, it also needs to scale replicas up or down to handle traffic load.

Although fine for what it does, this approach is only useful when we have a properly tested version that we want to deploy, i.e., more of a blue/green, a.k.a. red/black, kind of upgrade than a “dip your feet in the water” kind of canary deployment. In fact, for the latter (for example, testing a canary version that may not even be ready or intended for wider exposure), the canary deployment in Kubernetes would be done using two Deployments with common pod labels. In this case, we can’t use autoscaling anymore because it’s now being done by two independent autoscalers, one for each Deployment, so the replica ratios (percentages) may vary from the desired ratio, depending purely on load.

Whether we use one deployment or two, canary management using deployment features of container orchestration platforms like Docker, Mesos/Marathon, or Kubernetes has a fundamental problem: the use of instance scaling to manage the traffic; traffic version distribution and replica deployment are not independent in these systems. All replica pods, regardless of version, are treated the same in the kube-proxy round-robin pool, so the only way to manage the amount of traffic that a particular version receives is by controlling the replica ratio. Maintaining canary traffic at small percentages requires many replicas (e.g., 1% would require a minimum of 100 replicas). Even if we ignore this problem, the deployment approach is still very limited in that it only supports the simple (random percentage) canary approach. If, instead, we wanted to limit the visibility of the canary to requests based on some specific criteria, we still need another solution.

Enter Istio

With Istio, traffic routing and replica deployment are two completely independent functions. The number of pods implementing services are free to scale up and down based on traffic load, completely orthogonal to the control of version traffic routing. This makes managing a canary version in the presence of autoscaling a much simpler problem. Autoscalers may, in fact, respond to load variations resulting from traffic routing changes, but they are nevertheless functioning independently and no differently than when loads change for other reasons.

Istio’s routing rules also provide other important advantages; you can easily control fine-grained traffic percentages (e.g., route 1% of traffic without requiring 100 pods) and you can control traffic using other criteria (e.g., route traffic for specific users to the canary version). To illustrate, let’s look at deploying the helloworld service and see how simple the problem becomes.

We begin by defining the helloworld Service, just like any other Kubernetes service, something like this:

apiVersion: v1
kind: Service
metadata:
name: helloworld
labels:
app: helloworld
spec:
selector:
app: helloworld
...

We then add 2 Deployments, one for each version (v1 and v2), both of which include the service selector’s app: helloworld label:

apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v1
spec:
replicas: 1
template:
metadata:
labels:
app: helloworld
version: v1
spec:
containers:
- image: helloworld-v1
...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-v2
spec:
replicas: 1
template:
metadata:
labels:
app: helloworld
version: v2
spec:
containers:
- image: helloworld-v2
...

Note that this is exactly the same way we would do a canary deployment using plain Kubernetes, but in that case we would need to adjust the number of replicas of each Deployment to control the distribution of traffic. For example, to send 10% of the traffic to the canary version (v2), the replicas for v1 and v2 could be set to 9 and 1, respectively.

However, since we are going to deploy the service in an Istio enabled cluster, all we need to do is set a routing rule to control the traffic distribution. For example if we want to send 10% of the traffic to the canary, we could use kubectl to set a routing rule something like this:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld
spec:
hosts:
- helloworld
http:
- route:
- destination:
host: helloworld
subset: v1
weight: 90
- destination:
host: helloworld
subset: v2
weight: 10
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: helloworld
spec:
host: helloworld
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
EOF

After setting this rule, Istio will ensure that only one tenth of the requests will be sent to the canary version, regardless of how many replicas of each version are running.

Autoscaling the deployments

Because we don’t need to maintain replica ratios anymore, we can safely add Kubernetes horizontal pod autoscalers to manage the replicas for both version Deployments:

$ kubectl autoscale deployment helloworld-v1 --cpu-percent=50 --min=1 --max=10
deployment "helloworld-v1" autoscaled

$ kubectl autoscale deployment helloworld-v2 --cpu-percent=50 --min=1 --max=10
deployment "helloworld-v2" autoscaled

$ kubectl get hpa
NAME REFERENCE TARGET CURRENT MINPODS MAXPODS AGE
Helloworld-v1 Deployment/helloworld-v1 50% 47% 1 10 17s
Helloworld-v2 Deployment/helloworld-v2 50% 40% 1 10 15s

If we now generate some load on the helloworld service, we would notice that when scaling begins, the v1 autoscaler will scale up its replicas significantly higher than the v2 autoscaler will for its replicas because v1 pods are handling 90% of the load.

$ kubectl get pods | grep helloworld
helloworld-v1-3523621687-3q5wh 0/2 Pending 0 15m
helloworld-v1-3523621687-73642 2/2 Running 0 11m
helloworld-v1-3523621687-7hs31 2/2 Running 0 19m
helloworld-v1-3523621687-dt7n7 2/2 Running 0 50m
helloworld-v1-3523621687-gdhq9 2/2 Running 0 11m
helloworld-v1-3523621687-jxs4t 0/2 Pending 0 15m
helloworld-v1-3523621687-l8rjn 2/2 Running 0 19m
helloworld-v1-3523621687-wwddw 2/2 Running 0 15m
helloworld-v1-3523621687-xlt26 0/2 Pending 0 19m
helloworld-v2-4095161145-963wt 2/2 Running 0 50m

If we then change the routing rule to send 50% of the traffic to v2, we should, after a short delay, notice that the v1 autoscaler will scale down the replicas of v1 while the v2 autoscaler will perform a corresponding scale up.

$ kubectl get pods | grep helloworld
helloworld-v1-3523621687-73642 2/2 Running 0 35m
helloworld-v1-3523621687-7hs31 2/2 Running 0 43m
helloworld-v1-3523621687-dt7n7 2/2 Running 0 1h
helloworld-v1-3523621687-gdhq9 2/2 Running 0 35m
helloworld-v1-3523621687-l8rjn 2/2 Running 0 43m
helloworld-v2-4095161145-57537 0/2 Pending 0 21m
helloworld-v2-4095161145-9322m 2/2 Running 0 21m
helloworld-v2-4095161145-963wt 2/2 Running 0 1h
helloworld-v2-4095161145-c3dpj 0/2 Pending 0 21m
helloworld-v2-4095161145-t2ccm 0/2 Pending 0 17m
helloworld-v2-4095161145-v3v9n 0/2 Pending 0 13m

The end result is very similar to the simple Kubernetes Deployment rollout, only now the whole process is not being orchestrated and managed in one place. Instead, we’re seeing several components doing their jobs independently, albeit in a cause and effect manner. What’s different, however, is that if we now stop generating load, the replicas of both versions will eventually scale down to their minimum (1), regardless of what routing rule we set.

$ kubectl get pods | grep helloworld
helloworld-v1-3523621687-dt7n7 2/2 Running 0 1h
helloworld-v2-4095161145-963wt 2/2 Running 0 1h

Focused canary testing

As mentioned above, the Istio routing rules can be used to route traffic based on specific criteria, allowing more sophisticated canary deployment scenarios. Say, for example, instead of exposing the canary to an arbitrary percentage of users, we want to try it out on internal users, maybe even just a percentage of them. The following command could be used to send 50% of traffic from users at some-company-name.com to the canary version, leaving all other users unaffected:

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: helloworld
spec:
hosts:
- helloworld
http:
- match:
- headers:
cookie:
regex: "^(.*?;)?(email=[^;]*@some-company-name.com)(;.*)?$"
route:
- destination:
host: helloworld
subset: v1
weight: 50
- destination:
host: helloworld
subset: v2
weight: 50
- route:
- destination:
host: helloworld
subset: v1
EOF

As before, the autoscalers bound to the 2 version Deployments will automatically scale the replicas accordingly, but that will have no affect on the traffic distribution.

Summary

In this article we’ve seen how Istio supports general scalable canary deployments, and how this differs from the basic deployment support in Kubernetes. Istio’s service mesh provides the control necessary to manage traffic distribution with complete independence from deployment scaling. This allows for a simpler, yet significantly more functional, way to do canary test and rollout.

Intelligent routing in support of canary deployment is just one of the many features of Istio that will make the production deployment of large-scale microservices-based applications much simpler. Check out istio.io for more information and to try it out. The sample code used in this article can be found here.

Using Istio to Improve End-to-End Security

The Istio Team — Thu, 25 May 2017 00:00:00 +0000

Conventional network security approaches fail to address security threats to distributed applications deployed in dynamic production environments. Today, we describe how Istio authentication enables enterprises to transform their security posture from just protecting the edge to consistently securing all inter-service communications deep within their applications. With Istio authentication, developers and operators can protect services with sensitive data against unauthorized insider access and they can achieve this without any changes to the application code!

Istio authentication is the security component of the broader Istio platform. It incorporates the learnings of securing millions of microservice endpoints in Google’s production environment.

Background

Modern application architectures are increasingly based on shared services that are deployed and scaled dynamically on cloud platforms. Traditional network edge security (e.g. firewall) is too coarse-grained and allows access from unintended clients. An example of a security risk is stolen authentication tokens that can be replayed from another client. This is a major risk for companies with sensitive data that are concerned about insider threats. Other network security approaches like IP whitelists have to be statically defined, are hard to manage at scale, and are unsuitable for dynamic production environments.

Thus, security administrators need a tool that enables them to consistently, and by default, secure all communication between services across diverse production environments.

Solution: strong service identity and authentication

Google has, over the years, developed architecture and technology to uniformly secure millions of microservice endpoints in its production environment against external attacks and insider threats. Key security principles include trusting the endpoints and not the network, strong mutual authentication based on service identity and service level authorization. Istio authentication is based on the same principles.

The version 0.1 release of Istio authentication runs on Kubernetes and provides the following features:

Strong identity assertion between services
Access control to limit the identities that can access a service (and its data)
Automatic encryption of data in transit
Management of keys and certificates at scale

Istio authentication is based on industry standards like mutual TLS and X.509. Furthermore, Google is actively contributing to an open, community-driven service security framework called SPIFFE. As the SPIFFE specifications mature, we intend for Istio authentication to become a reference implementation of the same.

The diagram below provides an overview of the Istio’s service authentication architecture on Kubernetes.

Istio Authentication Overview

The above diagram illustrates three key security features:

Strong identity

Istio authentication uses Kubernetes service accounts to identify who the service runs as. The identity is used to establish trust and define service level access policies. The identity is assigned at service deployment time and encoded in the SAN (Subject Alternative Name) field of an X.509 certificate. Using a service account as the identity has the following advantages:

Administrators can configure who has access to a Service Account by using the RBAC feature introduced in Kubernetes 1.6
Flexibility to identify a human user, a service, or a group of services
Stability of the service identity for dynamically placed and auto-scaled workloads

Communication security

Service-to-service communication is tunneled through high performance client side and server side Envoy proxies. The communication between the proxies is secured using mutual TLS. The benefit of using mutual TLS is that the service identity is not expressed as a bearer token that can be stolen or replayed from another source. Istio authentication also introduces the concept of Secure Naming to protect from a server spoofing attacks - the client side proxy verifies that the authenticated server’s service account is allowed to run the named service.

Key management and distribution

Istio authentication provides a per-cluster CA (Certificate Authority) and automated key & certificate management. In this context, Istio authentication:

Generates a key and certificate pair for each service account.
Distributes keys and certificates to the appropriate pods using Kubernetes Secrets.
Rotates keys and certificates periodically.
Revokes a specific key and certificate pair when necessary (future).

The following diagram explains the end to end Istio authentication workflow on Kubernetes:

Istio Authentication Workflow

Istio authentication is part of the broader security story for containers. Red Hat, a partner on the development of Kubernetes, has identified 10 Layers of container security. Istio addresses two of these layers: “Network Isolation” and “API and Service Endpoint Management”. As cluster federation evolves on Kubernetes and other platforms, our intent is for Istio to secure communications across services spanning multiple federated clusters.

Benefits of Istio authentication

Defense in depth: When used in conjunction with Kubernetes (or infrastructure) network policies, users achieve higher levels of confidence, knowing that pod-to-pod or service-to-service communication is secured both at network and application layers.

Secure by default: When used with Istio’s proxy and centralized policy engine, Istio authentication can be configured during deployment with minimal or no application change. Administrators and operators can thus ensure that service communications are secured by default and that they can enforce these policies consistently across diverse protocols and runtimes.

Strong service authentication: Istio authentication secures service communication using mutual TLS to ensure that the service identity is not expressed as a bearer token that can be stolen or replayed from another source. This ensures that services with sensitive data can only be accessed from strongly authenticated and authorized clients.

Join us in this journey

Istio authentication is the first step towards providing a full stack of capabilities to protect services with sensitive data from external attacks and insider threats. While the initial version runs on Kubernetes, our goal is to enable Istio authentication to secure services across diverse production environments. We encourage the community to join us in making robust service security easy and ubiquitous across different application stacks and runtime platforms.