mirror of https://github.com/kubernetes/kops.git
Merge pull request #9107 from olemarkus/docs-networking-split
Move CNI docs to their own files
This commit is contained in:
Kubernetes Prow Robot
GitHub
commit
f96099c878
|
|
@ -1,117 +0,0 @@
|
|||
# Calico
|
||||
[Calico](https://docs.projectcalico.org/latest/introduction/) is an open source networking and
|
||||
network security solution for containers, virtual machines, and native host-based workloads.
|
||||
|
||||
Calico combines flexible networking capabilities with run-anywhere security enforcement to provide
|
||||
a solution with native Linux kernel performance and true cloud-native scalability. Calico provides
|
||||
developers and cluster operators with a consistent experience and set of capabilities whether
|
||||
running in public cloud or on-prem, on a single node or across a multi-thousand node cluster.
|
||||
|
||||
See [Calico for networking and network policy](networking.md#calico-example-for-cni-and-network-policy) for help configuring kops with Calico.
|
||||
|
||||
For more general information on options available with Calico see the official [Calico docs](https://docs.projectcalico.org/latest/introduction/):
|
||||
* See [Calico Network Policy](https://docs.projectcalico.org/latest/security/calico-network-policy)
|
||||
for details on the additional features not available with Kubernetes Network Policy.
|
||||
* See [Determining best Calico networking option](https://docs.projectcalico.org/latest/networking/determine-best-networking)
|
||||
for help with the network options available with Calico.
|
||||
|
||||
# Calico Version 3
|
||||
In early 2018 Version 3 of Calico was released, it included a reworked data
|
||||
model and with that a switch from the etcd v2 to v3 API. This section covers
|
||||
the requirements, upgrade process, and configuration to install
|
||||
Calico Version 3. By default new Kops installations configured to use Calico
|
||||
will install v3.
|
||||
|
||||
## Requirements
|
||||
|
||||
- The main requirement needed for Calico Version 3 is the etcd v3 API available
|
||||
with etcd server version 3.
|
||||
- Another requirement is for the Kubernetes version to be a minimum of v1.7.0.
|
||||
|
||||
### etcd
|
||||
Due to the etcd v3 API being a requirement of Calico Version 3
|
||||
(when using etcd as the datastore) not all Kops installations will be
|
||||
upgradable to Calico V3. Installations using etcd v2 (or earlier) will need
|
||||
to remain on Calico V2 or update to etcdv3.
|
||||
|
||||
## Configuration of a new cluster
|
||||
To ensure a new cluster will have Calico Version 3 installed the following
|
||||
two configurations options should be set:
|
||||
|
||||
- `spec.etcdClusters.etcdMembers[0].Version` (Main cluster) should be
|
||||
set to a Version of etcd greater than 3.x or the default version
|
||||
needs to be greater than 3.x.
|
||||
- The Networking config must have the Calico MajorVersion set to `v3` like
|
||||
the following:
|
||||
```
|
||||
spec:
|
||||
networking:
|
||||
calico:
|
||||
majorVersion: v3
|
||||
```
|
||||
|
||||
Both of the above two settings can be set by doing a `kops edit cluster ...`
|
||||
before bringing the cluster up for the first time.
|
||||
|
||||
With the above two settings your Kops deployed cluster will be running with
|
||||
Calico Version 3.
|
||||
|
||||
### Create cluster networking flag
|
||||
|
||||
When enabling Calico with the `--networking calico` flag, etcd will be set to
|
||||
a v3 version. Feel free to change to a different v3 version of etcd.
|
||||
|
||||
## Upgrading an existing cluster
|
||||
Assuming your cluster meets the requirements it is possible to upgrade
|
||||
your Calico v2 Kops cluster to Calico v3.
|
||||
|
||||
A few notes about the upgrade:
|
||||
|
||||
- During the first portion of the migration, while the calico-kube-controllers
|
||||
pod is running its Init, no new policies will be applied though already
|
||||
applied policy will be active.
|
||||
- During the migration no new pods will be scheduled as adding new workloads
|
||||
to Calico is blocked. Once the calico-complete-upgrade job has completed
|
||||
pods will once again be schedulable.
|
||||
- The upgrade process that has been automated in kops can be found in
|
||||
[the Upgrading Calico docs](https://docs.projectcalico.org/v3.1/getting-started/kubernetes/upgrade/upgrade).
|
||||
|
||||
Perform the upgrade with the following steps:
|
||||
|
||||
1. First you must ensure that you are running Calico V2.6.5+. With the
|
||||
latest Kops (greater than 1.9) ensuring your cluster is updated can be
|
||||
done by doing a `kops update` on the cluster.
|
||||
1. Verify your Calico data will migrate successfully by installing and
|
||||
configuring the
|
||||
[calico-upgrade command](https://docs.projectcalico.org/v3.1/getting-started/kubernetes/upgrade/setup)
|
||||
and then run `calico-upgrade dry-run` and verify it reports that the
|
||||
migration can be completed successfully.
|
||||
1. Set `majorVersion` field as below by editing
|
||||
your cluster configuration with `kops edit cluster`.
|
||||
```
|
||||
spec:
|
||||
networking:
|
||||
calico:
|
||||
majorVersion: v3
|
||||
```
|
||||
1. Update your cluster with `kops update` like you would normally update.
|
||||
1. Monitor the progress of the migration by using
|
||||
`kubectl get pods -n kube-system` and checking the status of the following pods:
|
||||
- calico-node pods should restart one at a time and all becoming Running
|
||||
- calico-kube-controllers pod will restart and after the first calico-node
|
||||
pod starts running it will start running
|
||||
- calico-complete-upgrade pod will be Completed after all the calico-node
|
||||
pods start running
|
||||
If any of the above fail by entering a crash loop you should investigate
|
||||
by checking the logs with `kubectl -n kube-system logs <pod name>`.
|
||||
1. Once the calico-node and calico-kube-controllers are running and the
|
||||
calico-complete-upgrade pod has completed the migration has finished
|
||||
successfully.
|
||||
|
||||
### Recovering from a partial migration
|
||||
|
||||
The InitContainer of the first calico-node pod that starts will perform the
|
||||
datastore migration necessary for upgrading from Calico v2 to Calico v3, if
|
||||
this InitContainer is killed or restarted when the new datastore is being
|
||||
populated it will be necessary to manually remove the Calico data in the
|
||||
etcd v3 API before the migration will be successful.
|
||||
|
|
@ -1,4 +1,6 @@
|
|||
## Kubernetes Networking Setup
|
||||
## Kubernetes Networking Options
|
||||
|
||||
The networking options determines how the pod and service networking is implemented and managed.
|
||||
|
||||
Kubernetes Operations (kops) currently supports 4 networking modes:
|
||||
|
||||
|
|
@ -7,7 +9,11 @@ Kubernetes Operations (kops) currently supports 4 networking modes:
|
|||
* `classic` Kubernetes native networking, done in-process.
|
||||
* `external` networking is done via a Daemonset. This is used in some custom implementations.
|
||||
|
||||
### kops Default Networking
|
||||
### Specifying network option for cluster creation
|
||||
|
||||
You can specify the network provider via the `--networking` command line switch. However, this will only give a default configuration of the provider. Typically you would often modify the `spec.networking` section of the cluster spec to configure the provider further.
|
||||
|
||||
### Kubenet (default)
|
||||
|
||||
Kubernetes Operations (kops) uses `kubenet` networking by default. This sets up networking on AWS using VPC
|
||||
networking, where the master allocates a /24 CIDR to each Node, drawing from the Node network.
|
||||
|
|
@ -26,559 +32,42 @@ Users running `--topology private` will not be able to choose `kubenet` networki
|
|||
requires a single routing table. These advanced users are usually running in multiple availability zones
|
||||
and NAT gateways are single AZ, multiple route tables are needed to use each NAT gateway.
|
||||
|
||||
### Supported CNI Networking
|
||||
### CNI
|
||||
|
||||
[Container Network Interface](https://github.com/containernetworking/cni) provides a specification
|
||||
and libraries for writing plugins to configure network interfaces in Linux containers. Kubernetes
|
||||
has built in support for CNI networking components.
|
||||
|
||||
Several different CNI providers are currently built into kops:
|
||||
Several CNI providers are currently built into kops:
|
||||
|
||||
* [Calico](https://docs.projectcalico.org/latest/introduction) - use `--networking calico` (See [below](#calico-example-for-cni-and-network-policy) for additional configuration options.)
|
||||
* [Canal (Flannel + Calico)](https://github.com/projectcalico/canal)
|
||||
* [flannel](https://github.com/coreos/flannel) - use `--networking flannel-vxlan` (recommended) or `--networking flannel-udp` (legacy). `--networking flannel` now selects `flannel-vxlan`.
|
||||
* [kopeio-vxlan](https://github.com/kopeio/networking)
|
||||
* [kube-router](./networking.md#kube-router-example-for-cni-ipvs-based-service-proxy-and-network-policy-enforcer)
|
||||
* [romana](https://github.com/romana/romana)
|
||||
* [weave](https://github.com/weaveworks/weave)
|
||||
* [amazon-vpc-routed-eni](./networking.md#amazon-vpc-backend)
|
||||
* [Cilium](http://docs.cilium.io)
|
||||
* [Lyft cni-ipvlan-vpc-k8s](https://github.com/lyft/cni-ipvlan-vpc-k8s)
|
||||
* [AWS VPC](networking/aws-vpc.md)
|
||||
* [Calico](networking/calico.md)
|
||||
* [Canal](networking/canal.md)
|
||||
* [Cilium](networking/cilium.md)
|
||||
* [Flannel](networking/flannel.md)
|
||||
* [Kube-router](networking/kube-router.md)
|
||||
* [Lyft VPC](networking/lyft-vpc.md)
|
||||
* [Romana](networking/romana.md)
|
||||
* [Weave](networking/weave.md)
|
||||
|
||||
The manifests for the providers are included with kops, and you simply use `--networking provider-name`.
|
||||
The manifests for the providers are included with kops, and you simply use `--networking <provider-name>`.
|
||||
Replace the provider name with the names listed above with you `kops cluster create`. For instance
|
||||
to install `kopeio-vxlan` execute the following:
|
||||
to install `calico` execute the following:
|
||||
|
||||
```console
|
||||
$ kops create cluster --networking kopeio-vxlan
|
||||
kops create cluster --networking calico
|
||||
```
|
||||
|
||||
This project has no bias over the CNI provider that you run, we care that we provide the correct setup to run CNI providers.
|
||||
### External CNI
|
||||
|
||||
Both `kubenet` and `classic` networking options are completely baked into kops, while since
|
||||
CNI networking providers are not part of the Kubernetes project, we do not maintain
|
||||
their installation processes. With that in mind, we do not support problems with
|
||||
different CNI providers but support configuring Kubernetes to run CNI providers.
|
||||
When using the flag `--networking cni` on `kops create cluster` or `spec.networking: cni {}` Kops will not install any CNI at all, but expect that you install it.
|
||||
|
||||
## Specifying network option for cluster creation
|
||||
When launching a cluster in this mode, the master nodes will come up in `not ready` state. You will then be able to deploy any CNI daemonset by following vanilla kubernetes install instructions. Once the CNI daemonset has been deployed, the master nodes should enter `ready` state and the remaining nodes should join the cluster shortly after.
|
||||
|
||||
You are able to specify your networking type via command line switch or in your yaml file.
|
||||
The `--networking` option accepts the four different values defined above: `kubenet`, `cni`,
|
||||
`classic`, and `external`. If `--networking` is left undefined `kubenet` is installed.
|
||||
|
||||
### Weave Example for CNI
|
||||
## Validating CNI Installation
|
||||
|
||||
#### Installation Weave on a new Cluster
|
||||
|
||||
The following command sets up a cluster, in HA mode, that is ready for a CNI installation.
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking cni \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
Once the cluster is stable, which you can check with a `kubectl cluster-info` command, the next
|
||||
step is to install CNI networking. Most of the CNI network providers are
|
||||
moving to installing their components plugins via a Daemonset. For instance weave will
|
||||
install with the following command:
|
||||
|
||||
Daemonset installation for K8s 1.6.x or above.
|
||||
```console
|
||||
$ kubectl create -f https://git.io/weave-kube-1.6
|
||||
```
|
||||
|
||||
Daemonset installation for K8s 1.4.x or 1.5.x.
|
||||
```console
|
||||
$ kubectl create -f https://git.io/weave-kube
|
||||
```
|
||||
|
||||
### Configuring Weave MTU
|
||||
|
||||
The Weave MTU is configurable by editing the cluster and setting `mtu` option in the weave configuration.
|
||||
AWS VPCs support jumbo frames, so on cluster creation kops sets the weave MTU to 8912 bytes (9001 minus overhead).
|
||||
|
||||
```
|
||||
spec:
|
||||
networking:
|
||||
weave:
|
||||
mtu: 8912
|
||||
```
|
||||
|
||||
### Configuring Weave Net EXTRA_ARGS
|
||||
|
||||
Weave allows you to pass command line arguments to weave by adding those arguments to the EXTRA_ARGS environmental variable.
|
||||
This can be used for debugging or for customizing the logging level of weave net.
|
||||
|
||||
```
|
||||
spec:
|
||||
networking:
|
||||
weave:
|
||||
netExtraArgs: "--log-level=info"
|
||||
```
|
||||
|
||||
Note that it is possible to break the cluster networking if flags are improperly used and as such this option should be used with caution.
|
||||
|
||||
### Configuring Weave NPC EXTRA_ARGS
|
||||
|
||||
Weave-npc (the Weave network policy controller) allows you to customize arguments of the running binary by setting the EXTRA_ARGS environmental variable.
|
||||
This can be used for debugging or for customizing the logging level of weave npc.
|
||||
|
||||
```
|
||||
spec:
|
||||
networking:
|
||||
weave:
|
||||
npcExtraArgs: "--log-level=info"
|
||||
```
|
||||
|
||||
Note that it is possible to break the cluster networking if flags are improperly used and as such this option should be used with caution.
|
||||
|
||||
### Configuring Weave network encryption
|
||||
|
||||
The Weave network encryption is configurable by creating a weave network secret password.
|
||||
Weaveworks recommends choosing a secret with [at least 50 bits of entropy](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/).
|
||||
If no password is supplied, kops will generate one at random.
|
||||
|
||||
```console
|
||||
$ cat /dev/urandom | tr -dc A-Za-z0-9 | head -c9 > password
|
||||
$ kops create secret weavepassword -f password
|
||||
$ kops update cluster
|
||||
```
|
||||
|
||||
Since unencrypted nodes will not be able to connect to nodes configured with encryption enabled, this configuration cannot be changed easily without downtime.
|
||||
|
||||
### Calico Example for CNI and Network Policy
|
||||
|
||||
#### Installing Calico on a new Cluster
|
||||
|
||||
The following command sets up a cluster, in HA mode, with Calico as the CNI and Network Policy provider.
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking calico \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
The above will deploy a daemonset installation which requires K8s 1.4.x or above.
|
||||
|
||||
##### Enable Cross-Subnet mode in Calico (AWS only)
|
||||
|
||||
Calico [since 2.1](https://www.projectcalico.org/project-calico-2-1-released/) supports a new option for IP-in-IP mode where traffic is only encapsulated
|
||||
when it’s destined to subnets with intermediate infrastructure lacking Calico route awareness
|
||||
– for example, across heterogeneous public clouds or on AWS where traffic is crossing availability zones/ regions.
|
||||
|
||||
With this mode, IP-in-IP encapsulation is only [performed selectively](https://docs.projectcalico.org/v3.10/networking/vxlan-ipip#configure-ip-in-ip-encapsulation-for-only-cross-subnet-traffic).
|
||||
This provides better performance in AWS multi-AZ deployments, and in general when deploying on networks where
|
||||
pools of nodes with L2 connectivity are connected via a router.
|
||||
|
||||
Note that Calico by default, routes between nodes within a subnet are distributed using a full node-to-node BGP mesh.
|
||||
Each node automatically sets up a BGP peering with every other node within the same L2 network.
|
||||
This full node-to-node mesh per L2 network has its scaling challenges for larger scale deployments.
|
||||
BGP route reflectors can be used as a replacement to a full mesh, and is useful for scaling up a cluster. [BGP route reflectors are recommended once the number of nodes goes above ~50-100.](https://docs.projectcalico.org/networking/bgp#topologies-for-public-cloud)
|
||||
The setup of BGP route reflectors is currently out of the scope of kops.
|
||||
|
||||
Read more here: [BGP route reflectors](https://docs.projectcalico.org/reference/architecture/overview#bgp-route-reflector-bird)
|
||||
|
||||
To enable this mode in a cluster, with Calico as the CNI and Network Policy provider, you must edit the cluster after the previous `kops create ...` command.
|
||||
|
||||
`kops edit cluster` will show you a block like this:
|
||||
|
||||
```
|
||||
networking:
|
||||
calico:
|
||||
majorVersion: v3
|
||||
```
|
||||
|
||||
You will need to change that block, and add an additional field, to look like this:
|
||||
|
||||
```
|
||||
networking:
|
||||
calico:
|
||||
majorVersion: v3
|
||||
crossSubnet: true
|
||||
```
|
||||
|
||||
This `crossSubnet` field can also be defined within a cluster specification file, and the entire cluster can be create by running:
|
||||
`kops create -f k8s-cluster.example.com.yaml`
|
||||
|
||||
In the case of AWS, EC2 instances have source/destination checks enabled by default.
|
||||
When you enable cross-subnet mode in kops, an addon controller ([k8s-ec2-srcdst](https://github.com/ottoyiu/k8s-ec2-srcdst))
|
||||
will be deployed as a Pod (which will be scheduled on one of the masters) to facilitate the disabling of said source/destination address checks.
|
||||
Only the masters have the IAM policy (`ec2:*`) to allow k8s-ec2-srcdst to execute `ec2:ModifyInstanceAttribute`.
|
||||
|
||||
### Configuring Calico MTU
|
||||
|
||||
The Calico MTU is configurable by editing the cluster and setting `mtu` option in the calico configuration.
|
||||
AWS VPCs support jumbo frames, so on cluster creation kops sets the calico MTU to 8912 bytes (9001 minus overhead).
|
||||
|
||||
For more details on Calico MTU please see the [Calico Docs](https://docs.projectcalico.org/networking/mtu#determine-mtu-size).
|
||||
|
||||
```
|
||||
spec:
|
||||
networking:
|
||||
calico:
|
||||
mtu: 8912
|
||||
```
|
||||
|
||||
#### More information about Calico
|
||||
|
||||
For Calico specific documentation please visit the [Calico Docs](http://docs.projectcalico.org/latest/getting-started/kubernetes/).
|
||||
|
||||
For details on upgrading a Calico v2 deployment see [Calico Version 3](calico-v3.md#upgrading-an-existing-cluster).
|
||||
|
||||
#### Getting help with Calico
|
||||
|
||||
For help with Calico or to report any issues:
|
||||
|
||||
- [Calico Github](https://github.com/projectcalico/calico)
|
||||
- [Calico Users Slack](https://calicousers.slack.com)
|
||||
|
||||
#### Calico Backend
|
||||
|
||||
In kops 1.12.0 and later Calico uses the k8s APIServer as its datastore.
|
||||
|
||||
In versions <1.12.0 of kops Calico uses etcd as a backend for storing information about workloads and policies. Calico does not interfere with normal etcd operations and does not require special handling when upgrading etcd. For more information please visit the [etcd Docs](https://coreos.com/etcd/docs/latest/)
|
||||
|
||||
#### Configuring Calico to use Typha
|
||||
|
||||
As of Kops 1.12 Calico uses the kube-apiserver as its datastore. The default setup does not make use of [Typha](https://github.com/projectcalico/typha) - a component intended to lower the impact of Calico on the k8s APIServer which is recommended in [clusters over 50 nodes](https://docs.projectcalico.org/latest/getting-started/kubernetes/installation/calico#installing-with-the-kubernetes-api-datastoremore-than-50-nodes) and is strongly recommended in clusters of 100+ nodes.
|
||||
It is possible to configure Calico to use Typha by editing a cluster and adding a
|
||||
`typhaReplicas` option to the Calico spec:
|
||||
|
||||
```
|
||||
networking:
|
||||
calico:
|
||||
typhaReplicas: 3
|
||||
```
|
||||
|
||||
#### Calico troubleshooting
|
||||
|
||||
##### New nodes are taking minutes for syncing ip routes and new pods on them can't reach kubedns
|
||||
|
||||
This is caused by nodes in the Calico etcd nodestore no longer existing. Due to the ephemeral nature of AWS EC2 instances, new nodes are brought up with different hostnames, and nodes that are taken offline remain in the Calico nodestore. This is unlike most datacentre deployments where the hostnames are mostly static in a cluster. Read more about this issue at https://github.com/kubernetes/kops/issues/3224
|
||||
This has been solved in kops 1.9.0, when creating a new cluster no action is needed, but if the cluster was created with a prior kops version the following actions should be taken:
|
||||
|
||||
* Use kops to update the cluster ```kops update cluster <name> --yes``` and wait for calico-kube-controllers deployment and calico-node daemonset pods to be updated
|
||||
* Decommission all invalid nodes, [see here](https://docs.projectcalico.org/v2.6/usage/decommissioning-a-node)
|
||||
* All nodes that are deleted from the cluster after this actions should be cleaned from calico's etcd storage and the delay programming routes should be solved.
|
||||
|
||||
### Canal Example for CNI and Network Policy
|
||||
|
||||
Canal is a project that combines [Flannel](https://github.com/coreos/flannel) and [Calico](http://docs.projectcalico.org/latest/getting-started/kubernetes/installation/) for CNI Networking. It uses Flannel for networking pod traffic between hosts via VXLAN and Calico for network policy enforcement and pod to pod traffic.
|
||||
|
||||
#### Installing Canal on a new Cluster
|
||||
|
||||
The following command sets up a cluster, in HA mode, with Canal as the CNI and networking policy provider
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking canal \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
The above will deploy a daemonset installation which requires K8s 1.4.x or above.
|
||||
|
||||
#### Getting help with Canal
|
||||
|
||||
For problems with deploying Canal please post an issue to Github:
|
||||
|
||||
- [Canal Issues](https://github.com/projectcalico/canal/issues)
|
||||
|
||||
For support with Calico Policies you can reach out on Slack or Github:
|
||||
|
||||
- [Calico Github](https://github.com/projectcalico/calico)
|
||||
- [Calico Users Slack](https://calicousers.slack.com)
|
||||
|
||||
For support with Flannel you can submit an issue on Github:
|
||||
|
||||
- [Flannel](https://github.com/coreos/flannel/issues)
|
||||
|
||||
### Kube-router example for CNI, IPVS based service proxy and Network Policy enforcer
|
||||
|
||||
[Kube-router](https://github.com/cloudnativelabs/kube-router) is project that provides one cohesive solution that provides CNI networking for pods, an IPVS based network service proxy and iptables based network policy enforcement.
|
||||
|
||||
#### Installing kube-router on a new Cluster
|
||||
|
||||
The following command sets up a cluster with Kube-router as the CNI, service proxy and networking policy provider
|
||||
|
||||
```
|
||||
$ kops create cluster \
|
||||
--node-count 2 \
|
||||
--zones us-west-2a \
|
||||
--master-zones us-west-2a \
|
||||
--dns-zone aws.cloudnativelabs.net \
|
||||
--node-size t2.medium \
|
||||
--master-size t2.medium \
|
||||
--networking kube-router \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
Currently kube-router supports 1.6 and above. Please note that kube-router will also provide service proxy, so kube-proxy will not be deployed in to the cluster.
|
||||
|
||||
No additional configurations are required to be done by user. Kube-router automatically disables source-destination check on all AWS EC2 instances. For the traffic within a subnet there is no overlay or tunneling used. For cross-subnet pod traffic ip-ip tunneling is used implicitly and no configuration is required.
|
||||
|
||||
### Romana Example for CNI
|
||||
|
||||
#### Installing Romana on a new Cluster
|
||||
|
||||
The following command sets up a cluster with Romana as the CNI.
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking romana \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
Currently Romana supports Kubernetes 1.6 and above.
|
||||
|
||||
#### Getting help with Romana
|
||||
|
||||
For problems with deploying Romana please post an issue to Github:
|
||||
|
||||
- [Romana Issues](https://github.com/romana/romana/issues)
|
||||
|
||||
You can also contact the Romana team on Slack
|
||||
|
||||
- [Romana Slack](https://romana.slack.com) (invite required - email [info@romana.io](mailto:info@romana.io))
|
||||
|
||||
#### Romana Backend
|
||||
|
||||
Romana uses the cluster's etcd as a backend for storing information about routes, hosts, host-groups and IP allocations.
|
||||
This does not affect normal etcd operations or require special treatment when upgrading etcd.
|
||||
The etcd port (4001) is opened between masters and nodes when using this networking option.
|
||||
|
||||
#### Amazon VPC Backend
|
||||
|
||||
The [Amazon VPC CNI](https://github.com/aws/amazon-vpc-cni-k8s) plugin
|
||||
requires no additional configurations to be done by user.
|
||||
|
||||
To use the Amazon VPC CNI plugin you specify
|
||||
|
||||
```
|
||||
networking:
|
||||
amazonvpc: {}
|
||||
```
|
||||
|
||||
in the cluster spec file or pass the `--networking amazon-vpc-routed-eni` option on the command line to kops:
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking amazon-vpc-routed-eni \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
**Important:** pods use the VPC CIDR, i.e. there is no isolation between the master, node/s and the internal k8s network.
|
||||
|
||||
**Note:** The following permissions are added to all nodes by kops to run the provider:
|
||||
|
||||
```json
|
||||
{
|
||||
"Sid": "kopsK8sEC2NodeAmazonVPCPerms",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ec2:CreateNetworkInterface",
|
||||
"ec2:AttachNetworkInterface",
|
||||
"ec2:DeleteNetworkInterface",
|
||||
"ec2:DetachNetworkInterface",
|
||||
"ec2:DescribeNetworkInterfaces",
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:ModifyNetworkInterfaceAttribute",
|
||||
"ec2:AssignPrivateIpAddresses",
|
||||
"ec2:UnassignPrivateIpAddresses",
|
||||
"tag:TagResources"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:CreateTags",
|
||||
"Resource": "arn:aws:ec2:*:*:network-interface/*"
|
||||
}
|
||||
```
|
||||
|
||||
In case of any issues the directory `/var/log/aws-routed-eni` contains the log files of the CNI plugin. This directory is located in all the nodes in the cluster.
|
||||
|
||||
[Configuration options for the Amazon VPC CNI plugin](https://github.com/aws/amazon-vpc-cni-k8s/tree/master#cni-configuration-variables) can be set through env vars defined in the cluster spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
amazonvpc:
|
||||
env:
|
||||
- name: WARM_IP_TARGET
|
||||
value: "10"
|
||||
- name: AWS_VPC_K8S_CNI_LOGLEVEL
|
||||
value: debug
|
||||
```
|
||||
|
||||
### Cilium Example for CNI and Network Policy
|
||||
|
||||
The Cilium CNI uses a Linux kernel technology called BPF, which enables the dynamic insertion of powerful security visibility and control logic within the Linux kernel.
|
||||
|
||||
#### Installing Cilium on a new Cluster
|
||||
|
||||
The following command sets up a cluster, in HA mode, with Cilium as the CNI and networking policy provider
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--networking cilium\
|
||||
--yes \
|
||||
--name cilium.example.com
|
||||
```
|
||||
|
||||
The above will deploy a Cilium daemonset installation which requires K8s 1.10.x or above.
|
||||
|
||||
#### Configuring Cilium
|
||||
|
||||
The following command registers a cluster, but doesn't create it yet
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--networking cilium\
|
||||
--name cilium.example.com
|
||||
```
|
||||
|
||||
`kops edit cluster` will show you a block like this:
|
||||
|
||||
```
|
||||
networking:
|
||||
cilium: {}
|
||||
```
|
||||
|
||||
You can adjust Cilium agent configuration with most options that are available in [cilium-agent command reference](http://cilium.readthedocs.io/en/stable/cmdref/cilium-agent/).
|
||||
|
||||
The following command will launch your cluster with desired Cilium configuration
|
||||
|
||||
```console
|
||||
$ kops update cluster myclustername.mydns.io --yes
|
||||
```
|
||||
|
||||
##### Using etcd for agent state sync
|
||||
|
||||
By default, Cilium will use CRDs for synchronizing agent state. This can cause performance problems on larger clusters. As of kops 1.18, kops can manage an etcd cluster using etcd-manager dedicated for cilium agent state sync. The [Cilium docs](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-external-etcd/) contains recommendations for this must be enabled.
|
||||
|
||||
Add the following to `spec.etcdClusters`:
|
||||
Make sure `instanceGroup` match the other etcd clusters.
|
||||
|
||||
```
|
||||
- etcdMembers:
|
||||
- instanceGroup: master-az-1a
|
||||
name: a
|
||||
- instanceGroup: master-az-1b
|
||||
name: b
|
||||
- instanceGroup: master-az-1c
|
||||
name: c
|
||||
name: cilium
|
||||
```
|
||||
|
||||
Then enable etcd as kvstore:
|
||||
|
||||
```
|
||||
networking:
|
||||
cilium:
|
||||
etcdManaged: true
|
||||
```
|
||||
|
||||
##### Enabling BPF NodePort
|
||||
|
||||
As of Kops 1.18 you can safely enable Cilium NodePort.
|
||||
|
||||
In this mode, the cluster is fully functional without kube-proxy, with Cilium replacing kube-proxy's NodePort implementation using BPF.
|
||||
Read more about this in the [Cilium docs](https://docs.cilium.io/en/stable/gettingstarted/nodeport/)
|
||||
|
||||
Be aware that you need to use an AMI with at least Linux 4.19.57 for this feature to work.
|
||||
|
||||
```
|
||||
kubeProxy:
|
||||
enabled: false
|
||||
networking:
|
||||
cilium:
|
||||
enableNodePort: true
|
||||
```
|
||||
|
||||
##### Enabling Cilium ENI IPAM
|
||||
|
||||
As of Kops 1.18, you can have Cilium provision AWS managed adresses and attach them directly to Pods much like Lyft VPC and AWS VPC. See [the Cilium docs for more information](https://docs.cilium.io/en/v1.6/concepts/ipam/eni/)
|
||||
|
||||
When using ENI IPAM you need to disable masquerading in Cilium as well.
|
||||
|
||||
```
|
||||
networking:
|
||||
cilium:
|
||||
disableMasquerade: true
|
||||
ipam: eni
|
||||
```
|
||||
|
||||
Note that since Cilium Operator is the entity that interacts with the EC2 API to provision and attaching ENIs, we force it to run on the master nodes when this IPAM is used.
|
||||
|
||||
Also note that this feature has only been tested on the default kops AMIs.
|
||||
|
||||
#### Getting help with Cilium
|
||||
|
||||
For problems with deploying Cilium please post an issue to Github:
|
||||
|
||||
- [Cilium Issues](https://github.com/cilium/cilium/issues)
|
||||
|
||||
For support with Cilium Network Policies you can reach out on Slack or Github:
|
||||
|
||||
- [Cilium Github](https://github.com/cilium/cilium)
|
||||
- [Cilium Slack](https://cilium.io/slack)
|
||||
|
||||
### Flannel Example for CNI
|
||||
|
||||
#### Configuring Flannel iptables resync period
|
||||
|
||||
As of Kops 1.12.0, Flannel iptables resync option is configurable via editing a cluster and adding
|
||||
`iptablesResyncSeconds` option to spec:
|
||||
|
||||
```
|
||||
networking:
|
||||
flannel:
|
||||
iptablesResyncSeconds: 360
|
||||
```
|
||||
|
||||
### Validating CNI Installation
|
||||
|
||||
You will notice that `kube-dns` fails to start properly until you deploy your CNI provider.
|
||||
Pod networking and IP addresses are provided by the CNI provider.
|
||||
You will notice that `kube-dns` and similar pods that depend on pod networks fails to start properly until you deploy your CNI provider.
|
||||
|
||||
Here are some steps items that will confirm a good CNI install:
|
||||
|
||||
|
|
@ -590,105 +79,14 @@ Here are some steps items that will confirm a good CNI install:
|
|||
The sig-networking and sig-cluster-lifecycle channels on K8s slack are always good starting places
|
||||
for Kubernetes specific CNI challenges.
|
||||
|
||||
#### Lyft CNI
|
||||
## How to pick the correct network provider
|
||||
|
||||
The [lyft cni-ipvlan-vpc-k8s](https://github.com/lyft/cni-ipvlan-vpc-k8s) plugin uses Amazon Elastic Network Interfaces (ENI) to assign AWS-managed IPs to Pods using the Linux kernel's IPvlan driver in L2 mode.
|
||||
Kops maintainers have no bias over the CNI provider that you run, we only aim to be flexible and provide a working setup of the CNIs.
|
||||
|
||||
Read the [prerequisites](https://github.com/lyft/cni-ipvlan-vpc-k8s#prerequisites) before starting. In addition to that, you need to specify the VPC ID as `spec.networkID` in the cluster spec file.
|
||||
|
||||
To use the Lyft CNI plugin you specify
|
||||
|
||||
```
|
||||
networking:
|
||||
lyftvpc: {}
|
||||
```
|
||||
|
||||
in the cluster spec file or pass the `--networking lyftvpc` option on the command line to kops:
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking lyftvpc \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
You can specify which subnets to use for allocating Pod IPs by specifying
|
||||
|
||||
```
|
||||
networking:
|
||||
lyftvpc:
|
||||
subnetTags:
|
||||
KubernetesCluster: myclustername.mydns.io
|
||||
```
|
||||
|
||||
In this example, new interfaces will be attached to subnets tagged with `kubernetes_kubelet = true`.
|
||||
|
||||
**Note:** The following permissions are added to all nodes by kops to run the provider:
|
||||
|
||||
```json
|
||||
{
|
||||
"Sid": "kopsK8sEC2NodeAmazonVPCPerms",
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"ec2:CreateNetworkInterface",
|
||||
"ec2:AttachNetworkInterface",
|
||||
"ec2:DeleteNetworkInterface",
|
||||
"ec2:DetachNetworkInterface",
|
||||
"ec2:DescribeNetworkInterfaces",
|
||||
"ec2:DescribeInstances",
|
||||
"ec2:DescribeInstanceTypes",
|
||||
"ec2:ModifyNetworkInterfaceAttribute",
|
||||
"ec2:AssignPrivateIpAddresses",
|
||||
"ec2:UnassignPrivateIpAddresses",
|
||||
"tag:TagResources"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": "ec2:CreateTags",
|
||||
"Resource": "arn:aws:ec2:*:*:network-interface/*"
|
||||
}
|
||||
```
|
||||
|
||||
In case of any issues the directory `/var/log/aws-routed-eni` contains the log files of the CNI plugin. This directory is located in all the nodes in the cluster.
|
||||
We do recommended something other than `kubenet` for production clusters due to `kubenet`'s limitations.
|
||||
|
||||
## Switching between networking providers
|
||||
|
||||
`kops edit cluster` and you will see a block like:
|
||||
Switching between from `classic` and `kubenet` providers to a CNI provider is considered safe. Just update the config and roll the cluster.
|
||||
|
||||
```
|
||||
networking:
|
||||
classic: {}
|
||||
```
|
||||
|
||||
That means you are running with `classic` networking. The `{}` means there are
|
||||
no configuration options, beyond the setting `classic`.
|
||||
|
||||
To switch to kubenet, change the word classic to kubenet.
|
||||
|
||||
```
|
||||
networking:
|
||||
kubenet: {}
|
||||
```
|
||||
|
||||
Now follow the normal update / rolling-update procedure:
|
||||
|
||||
```console
|
||||
$ kops update cluster # to preview
|
||||
$ kops update cluster --yes # to apply
|
||||
$ kops rolling-update cluster # to preview the rolling-update
|
||||
$ kops rolling-update cluster --yes # to roll all your instances
|
||||
```
|
||||
Your cluster should be ready in a few minutes. It is not trivial to see that this
|
||||
has worked; the easiest way seems to be to SSH to the master and verify
|
||||
that kubelet has been run with `--network-plugin=kubenet`.
|
||||
|
||||
Switching from `kubenet` to a CNI network provider has not been tested at this time.
|
||||
It is also possible to switch between CNI providers, but this usually is a distruptive change. Kops will also not clean up any resources left behind by the previous CNI, _including_ then CNI daemonset.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,44 @@
|
|||
# Amazon VPC
|
||||
|
||||
The Amazon VPC CNI uses the native AWS networking for Pods. Every pod gets an Elastic Network Interface (ENI) on the node it is running and an IP address beloning to the subnets assigned to the node.
|
||||
|
||||
## Installing
|
||||
|
||||
To use Amazon VPC, specify the following in the cluster spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
amazonvpc: {}
|
||||
```
|
||||
|
||||
in the cluster spec file or pass the `--networking amazon-vpc-routed-eni` option on the command line to kops:
|
||||
|
||||
```sh
|
||||
export ZONES=<mylistofzones>
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking amazon-vpc-routed-eni \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
**Important:** pods use the VPC CIDR, i.e. there is no isolation between the master, node/s and the internal k8s network. In addition, this CNI does not enforce network policies.
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
[Configuration options for the Amazon VPC CNI plugin](https://github.com/aws/amazon-vpc-cni-k8s/tree/master#cni-configuration-variables) can be set through env vars defined in the cluster spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
amazonvpc:
|
||||
env:
|
||||
- name: WARM_IP_TARGET
|
||||
value: "10"
|
||||
- name: AWS_VPC_K8S_CNI_LOGLEVEL
|
||||
value: debug
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
In case of any issues the directory `/var/log/aws-routed-eni` contains the log files of the CNI plugin. This directory is located in all the nodes in the cluster.
|
||||
|
|
@ -0,0 +1,112 @@
|
|||
# Calico
|
||||
|
||||
[Calico](https://docs.projectcalico.org/latest/introduction/) is an open source networking and
|
||||
network security solution for containers, virtual machines, and native host-based workloads.
|
||||
|
||||
Calico combines flexible networking capabilities with run-anywhere security enforcement to provide
|
||||
a solution with native Linux kernel performance and true cloud-native scalability. Calico provides
|
||||
developers and cluster operators with a consistent experience and set of capabilities whether
|
||||
running in public cloud or on-prem, on a single node or across a multi-thousand node cluster.
|
||||
|
||||
## Installing
|
||||
|
||||
To use the Calico, specify the following in the cluster spec.
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
calico: {}
|
||||
```
|
||||
|
||||
The following command sets up a cluster using Calico.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzones
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking calico \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
## Configuring
|
||||
|
||||
### Enable Cross-Subnet mode in Calico (AWS only)
|
||||
|
||||
Calico supports a new option for IP-in-IP mode where traffic is only encapsulated
|
||||
when it’s destined to subnets with intermediate infrastructure lacking Calico route awareness
|
||||
– for example, across heterogeneous public clouds or on AWS where traffic is crossing availability zones/ regions.
|
||||
|
||||
With this mode, IP-in-IP encapsulation is only [performed selectively](https://docs.projectcalico.org/v3.10/networking/vxlan-ipip#configure-ip-in-ip-encapsulation-for-only-cross-subnet-traffic).
|
||||
This provides better performance in AWS multi-AZ deployments, and in general when deploying on networks where
|
||||
pools of nodes with L2 connectivity are connected via a router.
|
||||
|
||||
Note that Calico by default, routes between nodes within a subnet are distributed using a full node-to-node BGP mesh.
|
||||
Each node automatically sets up a BGP peering with every other node within the same L2 network.
|
||||
This full node-to-node mesh per L2 network has its scaling challenges for larger scale deployments.
|
||||
BGP route reflectors can be used as a replacement to a full mesh, and is useful for scaling up a cluster. [BGP route reflectors are recommended once the number of nodes goes above ~50-100.](https://docs.projectcalico.org/networking/bgp#topologies-for-public-cloud)
|
||||
The setup of BGP route reflectors is currently out of the scope of kops.
|
||||
|
||||
Read more here: [BGP route reflectors](https://docs.projectcalico.org/reference/architecture/overview#bgp-route-reflector-bird)
|
||||
|
||||
To enable this mode in a cluster, add the following to the cluster spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
calico:
|
||||
crossSubnet: true
|
||||
```
|
||||
|
||||
In the case of AWS, EC2 instances have source/destination checks enabled by default.
|
||||
When you enable cross-subnet mode in kops, an addon controller ([k8s-ec2-srcdst](https://github.com/ottoyiu/k8s-ec2-srcdst))
|
||||
will be deployed as a Pod (which will be scheduled on one of the masters) to facilitate the disabling of said source/destination address checks.
|
||||
Only the masters have the IAM policy (`ec2:*`) to allow k8s-ec2-srcdst to execute `ec2:ModifyInstanceAttribute`.
|
||||
|
||||
### Configuring Calico MTU
|
||||
|
||||
The Calico MTU is configurable by editing the cluster and setting `mtu` option in the calico configuration.
|
||||
AWS VPCs support jumbo frames, so on cluster creation kops sets the calico MTU to 8912 bytes (9001 minus overhead).
|
||||
|
||||
For more details on Calico MTU please see the [Calico Docs](https://docs.projectcalico.org/networking/mtu#determine-mtu-size).
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
networking:
|
||||
calico:
|
||||
mtu: 8912
|
||||
```
|
||||
|
||||
### Configuring Calico to use Typha
|
||||
|
||||
As of Kops 1.12 Calico uses the kube-apiserver as its datastore. The default setup does not make use of [Typha](https://github.com/projectcalico/typha) - a component intended to lower the impact of Calico on the k8s APIServer which is recommended in [clusters over 50 nodes](https://docs.projectcalico.org/latest/getting-started/kubernetes/installation/calico#installing-with-the-kubernetes-api-datastoremore-than-50-nodes) and is strongly recommended in clusters of 100+ nodes.
|
||||
It is possible to configure Calico to use Typha by editing a cluster and adding a `typhaReplicas` option to the Calico spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
calico:
|
||||
typhaReplicas: 3
|
||||
```
|
||||
|
||||
## Getting help
|
||||
|
||||
For help with Calico or to report any issues:
|
||||
* [Calico Github](https://github.com/projectcalico/calico)
|
||||
* [Calico Users Slack](https://calicousers.slack.com)
|
||||
|
||||
For more general information on options available with Calico see the official [Calico docs](https://docs.projectcalico.org/latest/introduction/):
|
||||
* See [Calico Network Policy](https://docs.projectcalico.org/latest/security/calico-network-policy)
|
||||
for details on the additional features not available with Kubernetes Network Policy.
|
||||
* See [Determining best Calico networking option](https://docs.projectcalico.org/latest/networking/determine-best-networking)
|
||||
for help with the network options available with Calico.
|
||||
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### New nodes are taking minutes for syncing ip routes and new pods on them can't reach kubedns
|
||||
|
||||
This is caused by nodes in the Calico etcd nodestore no longer existing. Due to the ephemeral nature of AWS EC2 instances, new nodes are brought up with different hostnames, and nodes that are taken offline remain in the Calico nodestore. This is unlike most datacentre deployments where the hostnames are mostly static in a cluster. Read more about this issue at https://github.com/kubernetes/kops/issues/3224
|
||||
This has been solved in kops 1.9.0, when creating a new cluster no action is needed, but if the cluster was created with a prior kops version the following actions should be taken:
|
||||
|
||||
* Use kops to update the cluster ```kops update cluster <name> --yes``` and wait for calico-kube-controllers deployment and calico-node daemonset pods to be updated
|
||||
* Decommission all invalid nodes, [see here](https://docs.projectcalico.org/v2.6/usage/decommissioning-a-node)
|
||||
* All nodes that are deleted from the cluster after this actions should be cleaned from calico's etcd storage and the delay programming routes should be solved.
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# Canal
|
||||
|
||||
[Canal](https://github.com/projectcalico/canal) is a project that combines [Flannel](https://github.com/coreos/flannel) and [Calico](http://docs.projectcalico.org/latest/getting-started/kubernetes/installation/) for CNI Networking. It uses Flannel for networking pod traffic between hosts via VXLAN and Calico for network policy enforcement and pod to pod traffic.
|
||||
|
||||
## Installing
|
||||
|
||||
The following command sets up a cluster using Canal.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzones
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking canal \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
## Getting help
|
||||
|
||||
For problems with deploying Canal please post an issue to Github:
|
||||
|
||||
- [Canal Issues](https://github.com/projectcalico/canal/issues)
|
||||
|
||||
For support with Calico Policies you can reach out on Slack or Github:
|
||||
|
||||
- [Calico Github](https://github.com/projectcalico/calico)
|
||||
- [Calico Users Slack](https://calicousers.slack.com)
|
||||
|
||||
For support with Flannel you can submit an issue on Github:
|
||||
|
||||
- [Flannel](https://github.com/coreos/flannel/issues)
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
# Cilium
|
||||
* [Cilium](http://docs.cilium.io)
|
||||
|
||||
The Cilium CNI uses a Linux kernel technology called BPF, which enables the dynamic insertion of powerful security visibility and control logic within the Linux kernel.
|
||||
|
||||
## Installing Cilium on a new Cluster
|
||||
|
||||
To use the Cilium, specify the following in the cluster spec.
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
cilium: {}
|
||||
```
|
||||
|
||||
The following command sets up a cluster using Cilium.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzones
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking cilium\
|
||||
--yes \
|
||||
--name cilium.example.com
|
||||
```
|
||||
|
||||
## Configuring Cilium
|
||||
|
||||
### Using etcd for agent state sync
|
||||
|
||||
By default, Cilium will use CRDs for synchronizing agent state. This can cause performance problems on larger clusters. As of kops 1.18, kops can manage an etcd cluster using etcd-manager dedicated for cilium agent state sync. The [Cilium docs](https://docs.cilium.io/en/stable/gettingstarted/k8s-install-external-etcd/) contains recommendations for this must be enabled.
|
||||
|
||||
Add the following to `spec.etcdClusters`:
|
||||
Make sure `instanceGroup` match the other etcd clusters.
|
||||
|
||||
```yaml
|
||||
- etcdMembers:
|
||||
- instanceGroup: master-az-1a
|
||||
name: a
|
||||
- instanceGroup: master-az-1b
|
||||
name: b
|
||||
- instanceGroup: master-az-1c
|
||||
name: c
|
||||
name: cilium
|
||||
```
|
||||
|
||||
Then enable etcd as kvstore:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
cilium:
|
||||
etcdManaged: true
|
||||
```
|
||||
|
||||
### Enabling BPF NodePort
|
||||
|
||||
As of Kops 1.18 you can safely enable Cilium NodePort.
|
||||
|
||||
In this mode, the cluster is fully functional without kube-proxy, with Cilium replacing kube-proxy's NodePort implementation using BPF.
|
||||
Read more about this in the [Cilium docs](https://docs.cilium.io/en/stable/gettingstarted/nodeport/)
|
||||
|
||||
Be aware that you need to use an AMI with at least Linux 4.19.57 for this feature to work.
|
||||
|
||||
```yaml
|
||||
kubeProxy:
|
||||
enabled: false
|
||||
networking:
|
||||
cilium:
|
||||
enableNodePort: true
|
||||
```
|
||||
|
||||
### Enabling Cilium ENI IPAM
|
||||
|
||||
As of Kops 1.18, you can have Cilium provision AWS managed adresses and attach them directly to Pods much like Lyft VPC and AWS VPC. See [the Cilium docs for more information](https://docs.cilium.io/en/v1.6/concepts/ipam/eni/)
|
||||
|
||||
When using ENI IPAM you need to disable masquerading in Cilium as well.
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
cilium:
|
||||
disableMasquerade: true
|
||||
ipam: eni
|
||||
```
|
||||
|
||||
Note that since Cilium Operator is the entity that interacts with the EC2 API to provision and attaching ENIs, we force it to run on the master nodes when this IPAM is used.
|
||||
|
||||
Also note that this feature has only been tested on the default kops AMIs.
|
||||
|
||||
## Getting help
|
||||
|
||||
For problems with deploying Cilium please post an issue to Github:
|
||||
|
||||
- [Cilium Issues](https://github.com/cilium/cilium/issues)
|
||||
|
||||
For support with Cilium Network Policies you can reach out on Slack or Github:
|
||||
|
||||
- [Cilium Github](https://github.com/cilium/cilium)
|
||||
- [Cilium Slack](https://cilium.io/slack)
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
# Flannel
|
||||
|
||||
## Installing
|
||||
|
||||
To install [flannel](https://github.com/coreos/flannel) - use `--networking flannel-vxlan` (recommended) or `--networking flannel-udp` (legacy). `--networking flannel` now selects `flannel-vxlan`.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzone
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking flannel \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
## Configuring
|
||||
|
||||
### Configuring Flannel iptables resync period
|
||||
|
||||
As of Kops 1.12.0, Flannel iptables resync option is configurable via editing a cluster and adding
|
||||
`iptablesResyncSeconds` option to spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
flannel:
|
||||
iptablesResyncSeconds: 360
|
||||
```
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
# Kube-router
|
||||
|
||||
[Kube-router](https://github.com/cloudnativelabs/kube-router) is project that provides one cohesive solution that provides CNI networking for pods, an IPVS based network service proxy and iptables based network policy enforcement.
|
||||
|
||||
Kube-router also provides a service proxy, so kube-proxy will not be deployed in to the cluster.
|
||||
|
||||
## Installing kube-router on a new Cluster
|
||||
|
||||
The following command sets up a cluster with Kube-router.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzones
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking kube-router \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
No additional configurations are required to be done by user. Kube-router automatically disables source-destination check on all AWS EC2 instances. For the traffic within a subnet there is no overlay or tunneling used. For cross-subnet pod traffic ip-ip tunneling is used implicitly and no configuration is required.
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
# Lyft CNI
|
||||
|
||||
The [lyft cni-ipvlan-vpc-k8s](https://github.com/lyft/cni-ipvlan-vpc-k8s) plugin uses Amazon Elastic Network Interfaces (ENI) to assign AWS-managed IPs to Pods using the Linux kernel's IPvlan driver in L2 mode.
|
||||
|
||||
## Installing
|
||||
|
||||
Read the [prerequisites](https://github.com/lyft/cni-ipvlan-vpc-k8s#prerequisites) before starting. In addition to that, you need to specify the VPC ID as `spec.networkID` in the cluster spec file.
|
||||
|
||||
To use the Lyft CNI, specify the following in the cluster spec.
|
||||
|
||||
```
|
||||
networking:
|
||||
lyftvpc: {}
|
||||
```
|
||||
|
||||
in the cluster spec file or pass the `--networking lyftvpc` option on the command line to kops:
|
||||
|
||||
```console
|
||||
$ export ZONES=mylistofzones
|
||||
$ kops create cluster \
|
||||
--zones $ZONES \
|
||||
--master-zones $ZONES \
|
||||
--master-size m4.large \
|
||||
--node-size m4.large \
|
||||
--networking lyftvpc \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
## Configuring
|
||||
|
||||
### Specify subnet selector
|
||||
|
||||
You can specify which subnets to use for allocating Pod IPs by specifying
|
||||
|
||||
```
|
||||
networking:
|
||||
lyftvpc:
|
||||
subnetTags:
|
||||
KubernetesCluster: myclustername.mydns.io
|
||||
```
|
||||
|
||||
In this example, new interfaces will be attached to subnets tagged with `KubernetesCluster = myclustername.mydns.io`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
In case of any issues the directory `/var/log/aws-routed-eni` contains the log files of the CNI plugin. This directory is located in all the nodes in the cluster.
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
# Romana
|
||||
|
||||
## Installing
|
||||
|
||||
To use Romana, specify the following in the cluster spec:
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
romana: {}
|
||||
```
|
||||
|
||||
The following command sets up a cluster with Romana as the CNI.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzones
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking romana \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
Romana uses the cluster's etcd as a backend for storing information about routes, hosts, host-groups and IP allocations.
|
||||
This does not affect normal etcd operations or require special treatment when upgrading etcd.
|
||||
The etcd port (4001) is opened between masters and nodes when using this networking option.
|
||||
|
||||
## Getting help
|
||||
|
||||
For problems with deploying Romana please post an issue to Github:
|
||||
|
||||
- [Romana Issues](https://github.com/romana/romana/issues)
|
||||
|
||||
You can also contact the Romana team on Slack
|
||||
|
||||
- [Romana Slack](https://romana.slack.com) (invite required - email [info@romana.io](mailto:info@romana.io))
|
||||
|
|
@ -0,0 +1,76 @@
|
|||
### Weave
|
||||
|
||||
#### Installation
|
||||
|
||||
To use the Weave, specify the following in the cluster spec.
|
||||
|
||||
```yaml
|
||||
networking:
|
||||
weave: {}
|
||||
```
|
||||
|
||||
The following command sets up a cluster using Weave.
|
||||
|
||||
```sh
|
||||
export ZONES=mylistofzone
|
||||
kops create cluster \
|
||||
--zones $ZONES \
|
||||
--networking weave \
|
||||
--yes \
|
||||
--name myclustername.mydns.io
|
||||
```
|
||||
|
||||
### Configuring Weave MTU
|
||||
|
||||
The Weave MTU is configurable by editing the cluster and setting `mtu` option in the weave configuration.
|
||||
AWS VPCs support jumbo frames, so on cluster creation kops sets the weave MTU to 8912 bytes (9001 minus overhead).
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
networking:
|
||||
weave:
|
||||
mtu: 8912
|
||||
```
|
||||
|
||||
### Configuring Weave Net EXTRA_ARGS
|
||||
|
||||
Weave allows you to pass command line arguments to weave by adding those arguments to the EXTRA_ARGS environmental variable.
|
||||
This can be used for debugging or for customizing the logging level of weave net.
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
networking:
|
||||
weave:
|
||||
netExtraArgs: "--log-level=info"
|
||||
```
|
||||
|
||||
Note that it is possible to break the cluster networking if flags are improperly used and as such this option should be used with caution.
|
||||
|
||||
### Configuring Weave NPC EXTRA_ARGS
|
||||
|
||||
Weave-npc (the Weave network policy controller) allows you to customize arguments of the running binary by setting the EXTRA_ARGS environmental variable.
|
||||
This can be used for debugging or for customizing the logging level of weave npc.
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
networking:
|
||||
weave:
|
||||
npcExtraArgs: "--log-level=info"
|
||||
```
|
||||
|
||||
Note that it is possible to break the cluster networking if flags are improperly used and as such this option should be used with caution.
|
||||
|
||||
### Configuring Weave network encryption
|
||||
|
||||
The Weave network encryption is configurable by creating a weave network secret password.
|
||||
Weaveworks recommends choosing a secret with [at least 50 bits of entropy](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/).
|
||||
If no password is supplied, kops will generate one at random.
|
||||
|
||||
```sh
|
||||
cat /dev/urandom | tr -dc A-Za-z0-9 | head -c9 > password
|
||||
kops create secret weavepassword -f password
|
||||
kops update cluster
|
||||
```
|
||||
|
||||
Since unencrypted nodes will not be able to connect to nodes configured with encryption enabled, this configuration cannot be changed easily without downtime.
|
||||
|
||||
|
|
@ -1,75 +0,0 @@
|
|||
# Calico Pod CIDR Migration Procedure
|
||||
Prior to kops 1.7, calico, and other CNI providers was misconfigured to use the
|
||||
`.NonMasqueradeCIDR` field as the CIDR range for Pod IPs. As a result, IP
|
||||
conflict may occur when a Service is allocated an IP that has already been
|
||||
assigned to a Pod, or vice versa. To prevent this from occurring, manual steps
|
||||
are necessary before upgrading your cluster using kops 1.7 onwards.
|
||||
|
||||
|
||||
## Background
|
||||
The field in the clusterSpec, `.NonMasqueradeCIDR`, captures the IP
|
||||
range of the cluster.
|
||||
|
||||
Within this IP range, smaller IP ranges are then carved out for:
|
||||
|
||||
* Service IPs - as defined as `.serviceClusterIPRange`
|
||||
* Pod IPs - as defined as `.kubeControllerManager.clusterCIDR`
|
||||
|
||||
It was found out in Issue [#1171](https://github.com/kubernetes/kops/issues/1171),
|
||||
that weave and calico were misconfigured to use the wider IP range rather than
|
||||
the range dedicated to Pods only. This was fixed in PR [#2717](https://github.com/kubernetes/kops/pull/2717)
|
||||
and [#2768](https://github.com/kubernetes/kops/pull/2768) for the two CNIs, by
|
||||
switching over to using the `.kubeControllerManager.clusterCIDR` field instead.
|
||||
|
||||
With the `--ip-alloc-range` flag changes for weave, it effectively creates a
|
||||
new network. Pods in the existing network will not necessarily be able to talk
|
||||
to those in the new network. Restarting of all nodes will need to happen
|
||||
to guarantee that all Pods spin up with IPs in the new network. See [here](
|
||||
https://github.com/weaveworks/weave/issues/2874) for more details.
|
||||
|
||||
Just like weave, the above change alone is not enough to mitigate the problem
|
||||
on existing clusters running calico. Effectively, a new network will need to be
|
||||
created first (in the form of an IP Pool in Calico terms), and a restart of all
|
||||
nodes will need to happen to have Pods be allocated the proper IP addresses.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
* `kops` >= 1.7
|
||||
* `jq` for retrieving the field values from the clusterSpec
|
||||
* Kubernetes cluster with calico as the CNI, created prior to kops 1.7
|
||||
* Scheduled maintenance window, this procedure *WILL* result in cluster degregation.
|
||||
|
||||
## Procedure
|
||||
**WARNING** - This procedure will cause disruption to Pods running on the cluster.
|
||||
New Pods may not be able to resolve DNS through kube-dns or other services
|
||||
through its service IP during the rolling restart.
|
||||
Attempt this migration procedure on a staging cluster prior to doing this in production.
|
||||
|
||||
---
|
||||
Calico only uses the `CALICO_IPV4POOL_CIDR` to create a default IPv4 pool if a
|
||||
pool doesn't exist already:
|
||||
https://github.com/projectcalico/calicoctl/blob/v1.3.0/calico_node/startup/startup.go#L463
|
||||
|
||||
Therefore, we need to run two jobs. We have provided a manifest and a bash script.
|
||||
job creates a new IPv4 pool that we want, and deletes the existing IP
|
||||
pool that we no longer want. This is to be executed after a
|
||||
`kops update cluster --yes` using kops 1.7 and beyond,
|
||||
and before a `kops rolling-upgrade cluster`.
|
||||
|
||||
1. Using kops >= 1.7, update your cluster using `kops update cluster [--yes]`.
|
||||
2. Specify your cluster name in a `NAME` variable, download the template and bash script, and then run the bash script:
|
||||
```bash
|
||||
export NAME="YOUR_CLUSTER_NAME"
|
||||
wget https://raw.githubusercontent.com/kubernetes/kops/master/docs/calico_cidr_migration/create_migration_manifest.sh -O create_migration_manifest.sh
|
||||
wget https://raw.githubusercontent.com/kubernetes/kops/master/docs/calico_cidr_migration/jobs.yaml.template -O jobs.yaml.template
|
||||
chmod +x create_migration_manifest.sh
|
||||
./create_migration_manifest.sh
|
||||
```
|
||||
This will create a `jobs.yaml` manifest file that is used by the next step.
|
||||
|
||||
3. Make sure the current-context in the kubeconfig is the cluster you want to perform this migration.
|
||||
Run the job: `kubectl apply -f jobs.yaml`
|
||||
4. Run `kops rolling-update cluster --force --yes` to initiate a rolling restart on the cluster.
|
||||
This forces a restart of all nodes in the cluster.
|
||||
|
||||
That's it, you should see new Pods be allocated IPs in the new IP range!
|
||||
13
mkdocs.yml
13
mkdocs.yml
|
|
@ -89,7 +89,17 @@ nav:
|
|||
- etcd3 Migration: "etcd3-migration.md"
|
||||
|
||||
- Networking:
|
||||
- Networking Overview including CNI: "networking.md"
|
||||
- Networking Overview: "networking.md"
|
||||
- CNI:
|
||||
- AWS VPC: "networking/aws-vpc.md"
|
||||
- Calico: "networking/calico.md"
|
||||
- Canal: "networking/canal.md"
|
||||
- Cilium: "networking/cilium.md"
|
||||
- Flannel: "networking/flannel.md"
|
||||
- Kube-Router: "networking/kube-router.md"
|
||||
- Lyft VPC: "networking/lyft-vpc.md"
|
||||
- Romana: "networking/romana.md"
|
||||
- Weave: "networking/weave.md"
|
||||
- Run kops in an existing VPC: "run_in_existing_vpc.md"
|
||||
- Supported network topologies: "topology.md"
|
||||
- Subdomain setup: "creating_subdomain.md"
|
||||
|
|
@ -108,7 +118,6 @@ nav:
|
|||
- Philosophy: "philosophy.md"
|
||||
- State store: "state.md"
|
||||
- AWS China: "aws-china.md"
|
||||
- Calico v3: "calico-v3.md"
|
||||
- Custom CA: "custom_ca.md"
|
||||
- Horizontal Pod Autoscaling: "horizontal_pod_autoscaling.md"
|
||||
- Egress Proxy: "http_proxy.md"
|
||||
|
|
|
|||
Loading…
Reference in New Issue