--- title: Using Source IP content_template: templates/tutorial --- {{% capture overview %}} Applications running in a Kubernetes cluster find and communicate with each other, and the outside world, through the Service abstraction. This document explains what happens to the source IP of packets sent to different types of Services, and how you can toggle this behavior according to your needs. {{% /capture %}} {{% capture prerequisites %}} {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} ## Terminology This document makes use of the following terms: * [NAT](https://en.wikipedia.org/wiki/Network_address_translation): network address translation * [Source NAT](https://en.wikipedia.org/wiki/Network_address_translation#SNAT): replacing the source IP on a packet, usually with a node's IP * [Destination NAT](https://en.wikipedia.org/wiki/Network_address_translation#DNAT): replacing the destination IP on a packet, usually with a pod IP * [VIP](/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies): a virtual IP, such as the one assigned to every Kubernetes Service * [Kube-proxy](/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies): a network daemon that orchestrates Service VIP management on every node ## Prerequisites You must have a working Kubernetes 1.5 cluster to run the examples in this document. The examples use a small nginx webserver that echoes back the source IP of requests it receives through an HTTP header. You can create it as follows: ```console kubectl run source-ip-app --image=k8s.gcr.io/echoserver:1.4 ``` The output is: ``` deployment.apps/source-ip-app created ``` {{% /capture %}} {{% capture objectives %}} * Expose a simple application through various types of Services * Understand how each Service type handles source IP NAT * Understand the tradeoffs involved in preserving source IP {{% /capture %}} {{% capture lessoncontent %}} ## Source IP for Services with Type=ClusterIP Packets sent to ClusterIP from within the cluster are never source NAT'd if you're running kube-proxy in [iptables mode](/docs/concepts/services-networking/service/#proxy-mode-iptables), which is the default since Kubernetes 1.2. Kube-proxy exposes its mode through a `proxyMode` endpoint: ```console kubectl get nodes ``` The output is similar to this: ``` NAME STATUS ROLES AGE VERSION kubernetes-node-6jst Ready 2h v1.13.0 kubernetes-node-cx31 Ready 2h v1.13.0 kubernetes-node-jj1t Ready 2h v1.13.0 ``` Get the proxy mode on one of the node ```console kubernetes-node-6jst $ curl localhost:10249/proxyMode ``` The output is: ``` iptables ``` You can test source IP preservation by creating a Service over the source IP app: ```console kubectl expose deployment source-ip-app --name=clusterip --port=80 --target-port=8080 ``` The output is: ``` service/clusterip exposed ``` ```console kubectl get svc clusterip ``` The output is similar to: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE clusterip ClusterIP 10.0.170.92 80/TCP 51s ``` And hitting the `ClusterIP` from a pod in the same cluster: ```console kubectl run busybox -it --image=busybox --restart=Never --rm ``` The output is similar to this: ``` Waiting for pod default/busybox to be running, status is Pending, pod ready: false If you don't see a command prompt, try pressing enter. # ip addr 1: lo: mtu 65536 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0: mtu 1460 qdisc noqueue link/ether 0a:58:0a:f4:03:08 brd ff:ff:ff:ff:ff:ff inet 10.244.3.8/24 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::188a:84ff:feb0:26a5/64 scope link valid_lft forever preferred_lft forever # wget -qO - 10.0.170.92 CLIENT VALUES: client_address=10.244.3.8 command=GET ... ``` The client_address is always the client pod's IP address, whether the client pod and server pod are in the same node or in different nodes. ## Source IP for Services with Type=NodePort As of Kubernetes 1.5, packets sent to Services with [Type=NodePort](/docs/concepts/services-networking/service/#nodeport) are source NAT'd by default. You can test this by creating a `NodePort` Service: ```console kubectl expose deployment source-ip-app --name=nodeport --port=80 --target-port=8080 --type=NodePort ``` The output is: ``` service/nodeport exposed ``` ```console NODEPORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services nodeport) NODES=$(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="ExternalIP")].address }') ``` If you're running on a cloudprovider, you may need to open up a firewall-rule for the `nodes:nodeport` reported above. Now you can try reaching the Service from outside the cluster through the node port allocated above. ```console for node in $NODES; do curl -s $node:$NODEPORT | grep -i client_address; done ``` The output is similar to: ``` client_address=10.180.1.1 client_address=10.240.0.5 client_address=10.240.0.3 ``` Note that these are not the correct client IPs, they're cluster internal IPs. This is what happens: * Client sends packet to `node2:nodePort` * `node2` replaces the source IP address (SNAT) in the packet with its own IP address * `node2` replaces the destination IP on the packet with the pod IP * packet is routed to node 1, and then to the endpoint * the pod's reply is routed back to node2 * the pod's reply is sent back to the client Visually: ``` client \ ^ \ \ v \ node 1 <--- node 2 | ^ SNAT | | ---> v | endpoint ``` To avoid this, Kubernetes has a feature to preserve the client source IP [(check here for feature availability)](/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip). Setting `service.spec.externalTrafficPolicy` to the value `Local` will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. If there are no local endpoints, packets sent to the node are dropped, so you can rely on the correct source-ip in any packet processing rules you might apply a packet that make it through to the endpoint. Set the `service.spec.externalTrafficPolicy` field as follows: ```console kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}' ``` The output is: ``` service/nodeport patched ``` Now, re-run the test: ```console for node in $NODES; do curl --connect-timeout 1 -s $node:$NODEPORT | grep -i client_address; done ``` The output is: ``` client_address=104.132.1.79 ``` Note that you only got one reply, with the *right* client IP, from the one node on which the endpoint pod is running. This is what happens: * client sends packet to `node2:nodePort`, which doesn't have any endpoints * packet is dropped * client sends packet to `node1:nodePort`, which *does* have endpoints * node1 routes packet to endpoint with the correct source IP Visually: ``` client ^ / \ / / \ / v X node 1 node 2 ^ | | | | v endpoint ``` ## Source IP for Services with Type=LoadBalancer As of Kubernetes 1.5, packets sent to Services with [Type=LoadBalancer](/docs/concepts/services-networking/service/#loadbalancer) are source NAT'd by default, because all schedulable Kubernetes nodes in the `Ready` state are eligible for loadbalanced traffic. So if packets arrive at a node without an endpoint, the system proxies it to a node *with* an endpoint, replacing the source IP on the packet with the IP of the node (as described in the previous section). You can test this by exposing the source-ip-app through a loadbalancer ```console kubectl expose deployment source-ip-app --name=loadbalancer --port=80 --target-port=8080 --type=LoadBalancer ``` The output is: ``` service/loadbalancer exposed ``` Print IPs of the Service: ```console kubectl get svc loadbalancer ``` The output is similar to this: ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE loadbalancer LoadBalancer 10.0.65.118 104.198.149.140 80/TCP 5m ``` ```console curl 104.198.149.140 ``` The output is similar to this: ``` CLIENT VALUES: client_address=10.240.0.5 ... ``` However, if you're running on Google Kubernetes Engine/GCE, setting the same `service.spec.externalTrafficPolicy` field to `Local` forces nodes *without* Service endpoints to remove themselves from the list of nodes eligible for loadbalanced traffic by deliberately failing health checks. Visually: ``` client | lb VIP / ^ v / health check ---> node 1 node 2 <--- health check 200 <--- ^ | ---> 500 | V endpoint ``` You can test this by setting the annotation: ```console kubectl patch svc loadbalancer -p '{"spec":{"externalTrafficPolicy":"Local"}}' ``` You should immediately see the `service.spec.healthCheckNodePort` field allocated by Kubernetes: ```console kubectl get svc loadbalancer -o yaml | grep -i healthCheckNodePort ``` The output is similar to this: ``` healthCheckNodePort: 32122 ``` The `service.spec.healthCheckNodePort` field points to a port on every node serving the health check at `/healthz`. You can test this: ```console kubectl get pod -o wide -l run=source-ip-app ``` The output is similar to this: ``` NAME READY STATUS RESTARTS AGE IP NODE source-ip-app-826191075-qehz4 1/1 Running 0 20h 10.180.1.136 kubernetes-node-6jst ``` Curl the `/healthz` endpoint on different nodes. ```console kubernetes-node-6jst $ curl localhost:32122/healthz ``` The output is similar to this: ``` 1 Service Endpoints found ``` ```console kubernetes-node-jj1t $ curl localhost:32122/healthz ``` The output is similar to this: ``` No Service Endpoints Found ``` A service controller running on the master is responsible for allocating the cloud loadbalancer, and when it does so, it also allocates HTTP health checks pointing to this port/path on each node. Wait about 10 seconds for the 2 nodes without endpoints to fail health checks, then curl the lb ip: ```console curl 104.198.149.140 ``` The output is similar to this: ``` CLIENT VALUES: client_address=104.132.1.79 ... ``` __Cross platform support__ As of Kubernetes 1.5, support for source IP preservation through Services with Type=LoadBalancer is only implemented in a subset of cloudproviders (GCP and Azure). The cloudprovider you're running on might fulfill the request for a loadbalancer in a few different ways: 1. With a proxy that terminates the client connection and opens a new connection to your nodes/endpoints. In such cases the source IP will always be that of the cloud LB, not that of the client. 2. With a packet forwarder, such that requests from the client sent to the loadbalancer VIP end up at the node with the source IP of the client, not an intermediate proxy. Loadbalancers in the first category must use an agreed upon protocol between the loadbalancer and backend to communicate the true client IP such as the HTTP [X-FORWARDED-FOR](https://en.wikipedia.org/wiki/X-Forwarded-For) header, or the [proxy protocol](http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt). Loadbalancers in the second category can leverage the feature described above by simply creating an HTTP health check pointing at the port stored in the `service.spec.healthCheckNodePort` field on the Service. {{% /capture %}} {{% capture cleanup %}} Delete the Services: ```console kubectl delete svc -l run=source-ip-app ``` Delete the Deployment, ReplicaSet and Pod: ```console kubectl delete deployment source-ip-app ``` {{% /capture %}} {{% capture whatsnext %}} * Learn more about [connecting applications via services](/docs/concepts/services-networking/connect-applications-service/) * Learn more about [loadbalancing](/docs/user-guide/load-balancer) {{% /capture %}}