8.0 KiB

Raw Blame History

Calico

Calico is an open source networking and network security solution for containers, virtual machines, and native host-based workloads.

Calico combines flexible networking capabilities with run-anywhere security enforcement to provide a solution with native Linux kernel performance and true cloud-native scalability. Calico provides developers and cluster operators with a consistent experience and set of capabilities whether running in public cloud or on-prem, on a single node or across a multi-thousand node cluster.

Installing

To use the Calico, specify the following in the cluster spec.

  networking:
    calico: {}

The following command sets up a cluster using Calico.

export ZONES=mylistofzones
kops create cluster \
  --zones $ZONES \
  --networking calico \
  --yes \
  --name myclustername.mydns.io

Configuring

Enable Cross-Subnet mode in Calico (AWS only)

Calico supports a new option for IP-in-IP mode where traffic is only encapsulated when it’s destined to subnets with intermediate infrastructure lacking Calico route awareness – for example, across heterogeneous public clouds or on AWS where traffic is crossing availability zones/ regions.

With this mode, IP-in-IP encapsulation is only performed selectively. This provides better performance in AWS multi-AZ deployments, and in general when deploying on networks where pools of nodes with L2 connectivity are connected via a router.

Note that Calico by default, routes between nodes within a subnet are distributed using a full node-to-node BGP mesh. Each node automatically sets up a BGP peering with every other node within the same L2 network. This full node-to-node mesh per L2 network has its scaling challenges for larger scale deployments. BGP route reflectors can be used as a replacement to a full mesh, and is useful for scaling up a cluster. BGP route reflectors are recommended once the number of nodes goes above ~50-100. The setup of BGP route reflectors is currently out of the scope of kops.

Configuring Calico MTU

The Calico MTU is configurable by editing the cluster and setting mtu option in the calico configuration. AWS VPCs support jumbo frames, so on cluster creation kOps sets the calico MTU to 8912 bytes (9001 minus overhead).

For more details on Calico MTU please see the Calico Docs.

spec:
  networking:
    calico:
      mtu: 8912

Configuring Calico to use Typha

As of kOps 1.12 Calico uses the kube-apiserver as its datastore. The default setup does not make use of Typha - a component intended to lower the impact of Calico on the k8s APIServer which is recommended in clusters over 50 nodes and is strongly recommended in clusters of 100+ nodes. It is possible to configure Calico to use Typha by editing a cluster and adding a typhaReplicas option to the Calico spec:

  networking:
    calico:
      typhaReplicas: 3

Configuring the eBPF dataplane

{{ kops_feature_table(kops_added_default='1.19', k8s_min='1.16') }}

Calico supports using an eBPF dataplane as an alternative to the standard Linux dataplane (which is iptables based). While the standard dataplane focuses on compatibility by inter-working with kube-proxy, and your own iptables rules, the eBPF dataplane focuses on performance, latency and improving user experience with features that aren’t possible in the standard dataplane. As part of that, the eBPF dataplane replaces kube-proxy with an eBPF implementation. The main “user experience” feature is to preserve the source IP of traffic from outside the cluster when traffic hits a NodePort; this makes the server-side logs and network policy much more useful on that path.

For more details on enabling the eBPF dataplane please refer the Calico Docs.

  kubeProxy:
    enabled: false
  networking:
    calico:
      bpfEnabled: true
      bpfExternalServiceMode: Tunnel
      bpfLogLevel: Info

Configuring WireGuard

{{ kops_feature_table(kops_added_default='1.19', k8s_min='1.16') }}

Calico supports WireGuard to encrypt pod-to-pod traffic. If you enable this options, WireGuard encryption is automatically enabled for all nodes. At the moment, kOps installs WireGuard automatically only when the host OS is Ubuntu. For other OSes, WireGuard has to be part of the base image or installed via a hook.

For more details of Calico WireGuard please refer the Calico Docs.

  networking:
    calico:
      wireguardEnabled: true

Getting help

For help with Calico or to report any issues:

For more general information on options available with Calico see the official Calico docs:

See Calico Network Policy for details on the additional features not available with Kubernetes Network Policy.
See Determining best Calico networking option for help with the network options available with Calico.

Troubleshooting

New nodes are taking minutes for syncing ip routes and new pods on them can't reach kubedns

This is caused by nodes in the Calico etcd nodestore no longer existing. Due to the ephemeral nature of AWS EC2 instances, new nodes are brought up with different hostnames, and nodes that are taken offline remain in the Calico nodestore. This is unlike most datacentre deployments where the hostnames are mostly static in a cluster. Read more about this issue at https://github.com/kubernetes/kops/issues/3224 This has been solved in kOps 1.9.0, when creating a new cluster no action is needed, but if the cluster was created with a prior kops version the following actions should be taken:

Use kOps to update the cluster kops update cluster <name> --yes and wait for calico-kube-controllers deployment and calico-node daemonset pods to be updated
Decommission all invalid nodes, see here
All nodes that are deleted from the cluster after this actions should be cleaned from calico's etcd storage and the delay programming routes should be solved.

8.0 KiB Raw Blame History Unescape Escape