istio.io/content/help/ops/traffic-management/deploy-guidelines/index.md

title	description	weight
Deployment and Configuration Guidelines	Provides specific deployment and configuration guidelines.	5

This section provides specific deployment or configuration guidelines to avoid networking or traffic management issues.

503 errors after setting destination rule

If requests to a service immediately start generating HTTP 503 errors after you applied a DestinationRule and the errors continue until you remove or revert the DestinationRule, then the DesintationRule is probably causing a TLS conflict for the service.

For example, if you configure mutual TLS in the cluster globally, the DestinationRule must include the following trafficPolicy:

{{< text yaml >}} trafficPolicy: tls: mode: ISTIO_MUTUAL {{< /text >}}

Otherwise, the mode defaults to DISABLED causing client proxy sidecars to make plain HTTP requests instead of TLS encrypted requests. Thus, the requests conflict with the server proxy because the server proxy expects encrypted requests.

To confirm there is a conflict, check whether the STATUS field in the output of the istioctl authn tls-check command is set to CONFLICT for your service. For example:

{{< text bash >}} $ istioctl authn tls-check httpbin.default.svc.cluster.local HOST:PORT STATUS SERVER CLIENT AUTHN POLICY DESTINATION RULE httpbin.default.svc.cluster.local:8000 CONFLICT mTLS HTTP default/ httpbin/default {{< /text >}}

Whenever you apply a DestinationRule, ensure the trafficPolicy TLS mode matches the global server configuration.

503 errors while reconfiguring service routes

When setting route rules to direct traffic to specific versions (subsets) of a service, care must be taken to ensure that the subsets are available before they are used in the routes. Otherwise, calls to the service may return 503 errors during a reconfiguration period.

Creating both the VirtualServices and DestinationRules that define the corresponding subsets using a single kubectl call (e.g., kubectl apply -f myVirtualServiceAndDestinationRule.yaml is not sufficient because the resources propagate (from the configuration server, i.e., Kubernetes API server) to the Pilot instances in an eventually consistent manner. If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.

To make sure services will have zero down-time when configuring routes with subsets, follow a "make-before-break" process as described below:

• When adding new subsets:

  1. Update DestinationRules to add a new subset first, before updating any VirtualServices that use it. Apply the rule using kubectl or any platform-specific tooling.

  2. Wait a few seconds for the DestinationRule configuration to propagate to the Envoy sidecars

  3. Update the VirtualService to refer to the newly added subsets.

• When removing subsets:

  1. Update VirtualServices to remove any references to a subset, before removing the subset from a DestinationRule.

  2. Wait a few seconds for the VirtualService configuration to propagate to the Envoy sidecars.

  3. Update the DestinationRule to remove the unused subsets.

Route rules have no effect on ingress gateway requests

Let's assume you are using an ingress Gateway and corresponding VirtualSerive to access an internal service. For example, your VirtualService looks something like this:

{{< text yaml >}} apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts:

• "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello) gateways:
• myapp-gateway http:
• match:
  • uri: prefix: /hello route:
  • destination: host: helloworld.default.svc.cluster.local
• match: ... {{< /text >}}

You also have a VirtualService which routes traffic for the helloworld service to a particular subset:

{{< text yaml >}} apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: helloworld spec: hosts:

• helloworld.default.svc.cluster.local http:
• route:
  • destination: host: helloworld.default.svc.cluster.local subset: v1 {{< /text >}}

In this situation you will notice that requests to the helloworld service via the ingress gateway will not be directed to subset v1 but instead will continue to use default round-robin routing.

The ingress requests are using the gateway host (e.g., myapp.com) which will activate the rules in the myapp VirtualService that routes to any endpoint in the helloworld service. Internal requests with the host helloworld.default.svc.cluster.local will use the helloworld VirtualService which directs traffic exclusively to subset v1.

To control the traffic from the gateway, you need to include the subset rule in the myapp VirtualService:

{{< text yaml >}} apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts:

• "myapp.com" # or maybe "*" if you are testing without DNS using the ingress-gateway IP (e.g., http://1.2.3.4/hello) gateways:
• myapp-gateway http:
• match:
  • uri: prefix: /hello route:
  • destination: host: helloworld.default.svc.cluster.local subset: v1
• match: ... {{< /text >}}

Alternatively, you can combine both VirtualServices into one unit if possible:

{{< text yaml >}} apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: myapp spec: hosts:

• myapp.com # cannot use "*" here since this is being combined with the mesh services
• helloworld.default.svc.cluster.local gateways:
• mesh # applies internally as well as externally
• myapp-gateway http:
• match:
  • uri: prefix: /hello gateways:
    • myapp-gateway #restricts this rule to apply only to ingress gateway route:
  • destination: host: helloworld.default.svc.cluster.local subset: v1
• match:
  • gateways:
    • mesh # applies to all services inside the mesh route:
  • destination: host: helloworld.default.svc.cluster.local subset: v1 {{< /text >}}

Route rules have no effect on my application

If route rules are working perfectly for the Bookinfo sample, but similar version routing rules have no effect on your own application, it may be that your Kubernetes services need to be changed slightly.

Kubernetes services must adhere to certain restrictions in order to take advantage of Istio's L7 routing features. Refer to the Requirements for Pods and Services for details.

Envoy won't connect to my HTTP/1.0 service

Envoy requires HTTP/1.1 or HTTP/2 traffic for upstream services. For example, when using NGINX for serving traffic behind Envoy, you will need to set the proxy_http_version directive in your NGINX configuration to be "1.1", since the NGINX default is 1.0.

Example configuration:

{{< text plain >}} upstream http_backend { server 127.0.0.1:8080;

keepalive 16;

}

server { ...

location /http/ {
    proxy_pass http://http_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    ...
}

} {{< /text >}}

Headless TCP services losing connection

If istio-citadel is deployed, Envoy is restarted every 15 minutes to refresh certificates. This causes the disconnection of TCP streams or long-running connections between services.

You should build resilience into your application for this type of disconnect, but if you still want to prevent the disconnects from happening, you will need to disable mutual TLS and the istio-citadel deployment.

First, edit your istio configuration to disable mutual TLS:

{{< text bash >}} $ kubectl edit configmap -n istio-system istio $ kubectl delete pods -n istio-system -l istio=pilot {{< /text >}}

Next, scale down the istio-citadel deployment to disable Envoy restarts:

{{< text bash >}} $ kubectl scale --replicas=0 deploy/istio-citadel -n istio-system {{< /text >}}

This should stop Istio from restarting Envoy and disconnecting TCP connections.