mirror of https://github.com/istio/istio.io.git
Istio sidecar DNS resolution blog (#8264)
* DNS blog * no period in sub title Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io> * period in desc Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io> * rework Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io> * spellcheck Signed-off-by: Shriram Rajagopalan <rshriram@tetrate.io> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: Frank Budinsky <frankb@ca.ibm.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: Frank Budinsky <frankb@ca.ibm.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: Frank Budinsky <frankb@ca.ibm.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: Frank Budinsky <frankb@ca.ibm.com> * Update index.md * Update index.md * Update index.md * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: craigbox <craigbox@google.com> * Update index.md * Update index.md * Update index.md * Update index.md * Update content/en/blog/2020/dns-proxy/index.md * Update content/en/blog/2020/dns-proxy/index.md Co-authored-by: Frank Budinsky <frankb@ca.ibm.com> Co-authored-by: craigbox <craigbox@google.com>
This commit is contained in:
Shriram Rajagopalan
GitHub
parent
76f87ce4bd
commit
f5ce4ef4bf
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
|
|
@ -0,0 +1,254 @@
|
|||
---
|
||||
title: Expanding into New Frontiers - Smart DNS Proxying in Istio
|
||||
subtitle: Use workload-local DNS resolution to simplify VM integration, multicluster, and more
|
||||
description: Workload Local DNS resolution to simplify VM integration, multicluster, and more.
|
||||
publishdate: 2020-11-12
|
||||
attribution: "Shriram Rajagopalan (Tetrate.io) on behalf of Istio Networking WG"
|
||||
keywords: [dns,sidecar,multicluster,vm,external services]
|
||||
---
|
||||
|
||||
DNS resolution is a vital component of any application infrastructure
|
||||
on Kubernetes. When your application code attempts to access another
|
||||
service in the Kubernetes cluster or even a service on the internet,
|
||||
it has to first lookup the IP address corresponding to the hostname of
|
||||
the service, before initiating a connection to the service. This name
|
||||
lookup process is often referred to as **service discovery**. In
|
||||
Kubernetes, the cluster DNS server, be it `kube-dns` or CoreDNS,
|
||||
resolves the service's hostname to a unique non-routable virtual IP (VIP),
|
||||
if it is a service of type `clusterIP`. The `kube-proxy` on each node
|
||||
maps this VIP to a set of pods of the service, and forwards the traffic
|
||||
to one of them selected at random. When using a service mesh, the
|
||||
sidecar works similarly to the `kube-proxy` as far as traffic forwarding
|
||||
is concerned.
|
||||
|
||||
The following diagram depicts the role of DNS today:
|
||||
|
||||
{{< image width="75%"
|
||||
link="./role-of-dns-today.png"
|
||||
alt="Role of DNS in Istio, today"
|
||||
caption="Role of DNS in Istio, today"
|
||||
>}}
|
||||
|
||||
## Problems posed by DNS
|
||||
|
||||
While the role of DNS within the service mesh may seem insignificant,
|
||||
it has consistently stood in the way of expanding the mesh to VMs and
|
||||
enabling seamless multicluster access.
|
||||
|
||||
### VM access to Kubernetes services
|
||||
|
||||
Consider the case of a VM with a sidecar. As shown in the illustration
|
||||
below, applications on the VM look up the IP addresses of services
|
||||
inside the Kubernetes cluster as they typically have no access to the
|
||||
cluster's DNS server.
|
||||
|
||||
{{< image width="75%"
|
||||
link="./vm-dns-resolution-issues.png"
|
||||
alt="DNS resolution issues on VMs accessing Kubernetes services"
|
||||
caption="DNS resolution issues on VMs accessing Kubernetes services"
|
||||
>}}
|
||||
|
||||
It is technically possible to use `kube-dns` as a name server on the VM if one is
|
||||
willing to engage in some convoluted workarounds involving `dnsmasq` and
|
||||
external exposure of `kube-dns` using `NodePort` services: assuming you
|
||||
manage to convince your cluster administrator to do so. Even so, you are
|
||||
opening the door to a host of [security
|
||||
issues](https://blog.aquasec.com/dns-spoofing-kubernetes-clusters). At
|
||||
the end of the day, these are point solutions that are typically out
|
||||
of scope for those with limited organizational capability and domain
|
||||
expertise.
|
||||
|
||||
### External TCP services without VIPs
|
||||
|
||||
It is not just the VMs in the mesh that suffer from the DNS issue. For
|
||||
the sidecar to accurately distinguish traffic between two different
|
||||
TCP services that are outside the mesh, the services must be on
|
||||
different ports or they need to have a globally unique VIP, much like
|
||||
the `clusterIP` assigned to Kubernetes services. But what if there is
|
||||
no VIP? Cloud hosted services like hosted databases, typically do not
|
||||
have a VIP. Instead, the provider's DNS server returns one of the
|
||||
instance IPs that can then be directly accessed by the
|
||||
application. For example, consider the two service entries below,
|
||||
pointing to two different AWS RDS services:
|
||||
|
||||
{{< text yaml >}}
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: ServiceEntry
|
||||
metadata:
|
||||
name: db1
|
||||
namespace: ns1
|
||||
spec:
|
||||
hosts:
|
||||
- mysql–instance1.us-east-1.rds.amazonaws.com
|
||||
ports:
|
||||
- name: mysql
|
||||
number: 3306
|
||||
protocol: TCP
|
||||
resolution: DNS
|
||||
---
|
||||
apiVersion: networking.istio.io/v1alpha3
|
||||
kind: ServiceEntry
|
||||
metadata:
|
||||
name: db2
|
||||
namespace: ns1
|
||||
spec:
|
||||
hosts:
|
||||
- mysql–instance2.us-east-1.rds.amazonaws.com
|
||||
ports:
|
||||
- name: mysql
|
||||
number: 3306
|
||||
protocol: TCP
|
||||
resolution: DNS
|
||||
{{< /text >}}
|
||||
|
||||
The sidecar has a single listener on `0.0.0.0:3306` that looks up the
|
||||
IP address of `mysql-instance1.us-east1.rds.amazonaws.com` from public
|
||||
DNS servers and forwards traffic to it. It cannot route traffic to
|
||||
`db2` as it has no way of distinguishing whether traffic arriving at
|
||||
`0.0.0.0:3306` is bound for `db1` or `db2`. The only way to accomplish
|
||||
this is to set the resolution to `NONE` causing the sidecar to
|
||||
_blindly forward any traffic_ on port `3306` to the original IP
|
||||
requested by the application. This is akin to punching a hole in the
|
||||
firewall allowing all traffic to port `3306` irrespective of the
|
||||
destination IP. To get traffic flowing, you are now forced to
|
||||
compromise on the security posture of your system.
|
||||
|
||||
### Resolving DNS for services in remote clusters
|
||||
|
||||
The DNS limitations of a multicluster mesh are well known. Services in
|
||||
one cluster cannot lookup the IP addresses of services in other
|
||||
clusters, without clunky workarounds such as creating stub services in
|
||||
the caller namespace.
|
||||
|
||||
## Taking control of DNS
|
||||
|
||||
All in all, DNS has been a thorny issue in Istio for a while. It was
|
||||
time to slay the beast. We (the Istio networking team) decided to
|
||||
tackle the problem once and for all in a way that is completely
|
||||
transparent to you, the end user. Our first attempt involved utilizing
|
||||
Envoy's DNS proxy. It turned out to be very unreliable, and
|
||||
disappointing overall due to the general lack of sophistication in
|
||||
the c-ares DNS library used by Envoy. Determined to solve the
|
||||
problem, we decided to implement the DNS proxy in the Istio sidecar
|
||||
agent, written in Go. We were able to optimize the implementation to
|
||||
handle all the scenarios that we wanted to tackle without compromising
|
||||
on scale and stability. The Go DNS library we use is the same one
|
||||
used by scalable DNS implementations such as CoreDNS, Consul,
|
||||
Mesos, etc. It has been battle tested in production for scale and stability.
|
||||
|
||||
Starting with Istio 1.8, the Istio agent on the sidecar will ship with
|
||||
a caching DNS proxy, programmed dynamically by Istiod. Istiod pushes
|
||||
the hostname-to-IP-address mappings for all the services that the
|
||||
application may access based on the Kubernetes services and service
|
||||
entries in the cluster. DNS lookup queries from the application are
|
||||
transparently intercepted and served by the Istio agent in the pod or
|
||||
VM. If the query is for a service within the mesh, _irrespective of
|
||||
the cluster that the service is in_, the agent responds directly to the
|
||||
application. If not, it forwards the query to the upstream name
|
||||
servers defined in `/etc/resolv.conf`. The following diagram depicts
|
||||
the interactions that occur when an application tries to access a
|
||||
service using its hostname.
|
||||
|
||||
{{< image width="75%"
|
||||
link="./dns-interception-in-istio.png"
|
||||
alt="Smart DNS proxying in Istio sidecar agent"
|
||||
caption="Smart DNS proxying in Istio sidecar agent"
|
||||
>}}
|
||||
|
||||
As you will see in the following sections, _the DNS proxying feature
|
||||
has had an enormous impact across many aspects of Istio._
|
||||
|
||||
### Reduced load on your DNS servers w/ faster resolution
|
||||
|
||||
The load on your cluster’s Kubernetes DNS server drops drastically as
|
||||
almost all DNS queries are resolved within the pod by Istio. The
|
||||
bigger the footprint of mesh on a cluster, the lesser the load on your
|
||||
DNS servers. Implementing our own DNS proxy in the Istio agent has
|
||||
allowed us to implement cool optimizations such as [CoreDNS
|
||||
auto-path](https://coredns.io/plugins/autopath/) without the
|
||||
correctness issues that CoreDNS currently faces.
|
||||
|
||||
To understand the impact of this optimization, lets take a simple DNS
|
||||
lookup scenario, in a standard Kubernetes cluster without any custom
|
||||
DNS setup for pods - i.e., with the default setting of `ndots:5` in `/etc/resolv.conf`.
|
||||
When your application starts a DNS lookup for
|
||||
`productpage.ns1.svc.cluster.local`, it appends the DNS search
|
||||
namespaces in `/etc/resolv.conf` (e.g., `ns1.svc.cluster.local`) as part
|
||||
of the DNS query, before querying the host as-is. As a result, the
|
||||
first DNS query that is actually sent out will look like
|
||||
`productpage.ns1.svc.cluster.local.ns1.svc.cluster.local`, which will
|
||||
inevitably fail DNS resolution when Istio is not involved. If your
|
||||
`/etc/resolv.conf` has 5 search namespaces, the application will send
|
||||
two DNS queries for each search namespace, one for the IPv4 `A` record
|
||||
and another for the IPv6 `AAAA` record, and then a final pair of
|
||||
queries with the exact hostname used in the code. _Before establishing the
|
||||
connection, the application performs 12 DNS lookup queries for each host!_
|
||||
|
||||
With Istio's implementation of the CoreDNS style auto-path technique,
|
||||
the sidecar agent will detect the real hostname being queried within
|
||||
the first query and return a `cname` record to
|
||||
`productpage.ns1.svc.cluster.local` as part of this DNS response, as
|
||||
well as the `A/AAAA` record for
|
||||
`productpage.ns1.svc.cluster.local`. The application receiving this
|
||||
response can now extract the IP address immediately and proceed to
|
||||
establishing a TCP connection to that IP. _The smart DNS proxy in the
|
||||
Istio agent dramatically cuts down the number of DNS queries from 12
|
||||
to just 2!_
|
||||
|
||||
### VMs to Kubernetes integration
|
||||
|
||||
Since the Istio agent performs local DNS resolution for services
|
||||
within the mesh, DNS lookup queries for Kubernetes services from VMs will now
|
||||
succeed without requiring clunky workarounds for exposing `kube-dns`
|
||||
outside the cluster. The ability to seamlessly resolve internal
|
||||
services in a cluster will now simplify your monolith to microservice
|
||||
journey, as the monolith on VMs can now access microservices on
|
||||
Kubernetes without additional levels of indirection via API gateways.
|
||||
|
||||
### Automatic VIP allocation where possible
|
||||
|
||||
You may ask, how does this DNS functionality in the agent solve the
|
||||
problem of distinguishing between multiple external TCP services
|
||||
without VIPs on the same port?
|
||||
|
||||
Taking inspiration from Kubernetes, Istio will now automatically
|
||||
allocate non-routable VIPs (from the Class E subnet) to such services
|
||||
as long as they do not use a wildcard host. The Istio agent on the
|
||||
sidecar will use the VIPs as responses to the DNS lookup queries from
|
||||
the application. Envoy can now clearly distinguish traffic bound for
|
||||
each external TCP service and forward it to the right target. With the
|
||||
introduction of the DNS proxying, you will no longer need to use
|
||||
`resolution: NONE` for non-wildcard TCP services, improving your
|
||||
overall security posture. Istio cannot help much with wildcard
|
||||
external services (e.g., `*.us-east1.rds.amazonaws.com`). You will
|
||||
have to resort to NONE resolution mode to handle such services.
|
||||
|
||||
### Multicluster DNS lookup
|
||||
|
||||
For the adventurous lot, attempting to weave a multicluster mesh where
|
||||
applications directly call internal services of a namespace in a
|
||||
remote cluster, the DNS proxy functionality comes in quite handy. Your
|
||||
applications can _resolve Kubernetes services on any cluster in any
|
||||
namespace_, without the need to create stub Kubernetes services in
|
||||
every cluster.
|
||||
|
||||
The benefits of the DNS proxy extend beyond the multicluster models
|
||||
that are currently described in Istio today. At Tetrate, we use this
|
||||
mechanism extensively in our customers' multicluster deployments to
|
||||
enable sidecars to resolve DNS for hosts exposed at ingress gateways
|
||||
of all the clusters in a mesh, and access them over mutual TLS.
|
||||
|
||||
## Concluding thoughts
|
||||
|
||||
The problems caused by lack of control over DNS have often been
|
||||
overlooked and ignored in its entirety when it comes to weaving a mesh
|
||||
across many clusters, different environments, and integrating external
|
||||
services. The introduction of a caching DNS proxy in the Istio sidecar
|
||||
agent solves these issues. Exercising control over the
|
||||
application’s DNS resolution allows Istio to accurately identify the
|
||||
target service to which traffic is bound, and enhance the overall
|
||||
security, routing, and telemetry posture in Istio within and across
|
||||
clusters.
|
||||
|
||||
Smart DNS proxying is enabled in the `preview`
|
||||
profile in Istio 1.8. Please try it out!
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 29 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
Loading…
Reference in New Issue