docs/ee/ucp/admin/configure/_site/deploy-route-reflectors.html

<p>UCP uses Calico as the default Kubernetes networking solution. Calico is
configured to create a BGP mesh between all nodes in the cluster.</p>

<p>As you add more nodes to the cluster, networking performance starts decreasing.
If your cluster has more than 100 nodes, you should reconfigure Calico to use
Route Reflectors instead of a node-to-node mesh.</p>

<p>This article guides you in deploying Calico Route Reflectors in a UCP cluster.
UCP running on Microsoft Azure uses Azure SDN instead of Calico for
multi-host networking.
If your UCP deployment is running on Azure, you don’t need to configure it this
way.</p>

<h2 id="before-you-begin">Before you begin</h2>

<p>For production-grade systems, you should deploy at least two Route Reflectors,
each running on a dedicated node. These nodes should not be running any other
workloads.</p>

<p>If Route Reflectors are running on a same node as other workloads, swarm ingress
and NodePorts might not work in these workloads.</p>

<h2 id="choose-dedicated-notes">Choose dedicated notes</h2>

<p>Start by tainting the nodes, so that no other workload runs there. Configure
your CLI with a UCP client bundle, and for each dedicated node, run:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl taint node &lt;node-name&gt; \
  com.docker.ucp.kubernetes.calico/route-reflector=true:NoSchedule
</code></pre></div></div>

<p>Then add labels to those nodes, so that you can target them when deploying the
Route Reflectors. For each dedicated node, run:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl label nodes &lt;node-name&gt; \
  com.docker.ucp.kubernetes.calico/route-reflector=true
</code></pre></div></div>

<h2 id="deploy-the-route-reflectors">Deploy the Route Reflectors</h2>

<p>Create a <code class="highlighter-rouge">calico-rr.yaml</code> file with the following content:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: calico-rr
  namespace: kube-system
  labels:
    app: calico-rr
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      k8s-app: calico-rr
  template:
    metadata:
      labels:
        k8s-app: calico-rr
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      tolerations:
        - key: com.docker.ucp.kubernetes.calico/route-reflector
          value: "true"
          effect: NoSchedule
      hostNetwork: true
      containers:
        - name: calico-rr
          image: calico/routereflector:v0.6.1
          env:
            - name: ETCD_ENDPOINTS
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_endpoints
            - name: ETCD_CA_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_ca
            # Location of the client key for etcd.
            - name: ETCD_KEY_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_key # Location of the client certificate for etcd.
            - name: ETCD_CERT_FILE
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: etcd_cert
            - name: IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /calico-secrets
              name: etcd-certs
          securityContext:
            privileged: true
      nodeSelector:
        com.docker.ucp.kubernetes.calico/route-reflector: "true"
      volumes:
      # Mount in the etcd TLS secrets.
        - name: etcd-certs
          secret:
            secretName: calico-etcd-secrets
</code></pre></div></div>

<p>Then, deploy the DaemonSet using:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create -f calico-rr.yaml
</code></pre></div></div>

<h2 id="configure-calicoctl">Configure calicoctl</h2>

<p>To reconfigure Calico to use Route Reflectors instead of a node-to-node mesh,
you’ll need to SSH into a UCP node and download the <code class="highlighter-rouge">calicoctl</code> tool.</p>

<p>Log in to a UCP node using SSH, and run:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo curl --location https://github.com/projectcalico/calicoctl/releases/download/v3.1.1/calicoctl \
  --output /usr/bin/calicoctl
sudo chmod +x /usr/bin/calicoctl
</code></pre></div></div>

<p>Now you need to configure <code class="highlighter-rouge">calicoctl</code> to communicate with the etcd key-value
store managed by UCP. Create a file named <code class="highlighter-rouge">/etc/calico/calicoctl.cfg</code> with
the following content:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "etcdv3"
  etcdEndpoints: "127.0.0.1:12378"
  etcdKeyFile: "/var/lib/docker/volumes/ucp-node-certs/_data/key.pem"
  etcdCertFile: "/var/lib/docker/volumes/ucp-node-certs/_data/cert.pem"
  etcdCACertFile: "/var/lib/docker/volumes/ucp-node-certs/_data/ca.pem"
</code></pre></div></div>

<h2 id="disable-node-to-node-bgp-mesh">Disable node-to-node BGP mesh</h2>

<p>Not that you’ve configured <code class="highlighter-rouge">calicoctl</code>, you can check the current Calico BGP
configuration:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo calicoctl get bgpconfig
</code></pre></div></div>

<p>If you don’t see any configuration listed, create one by running:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cat &lt;&lt; EOF | sudo calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false
  asNumber: 63400
EOF
</code></pre></div></div>

<p>This creates a new configuration with node-to-node mesh BGP disabled.
If you have a configuration, and <code class="highlighter-rouge">meshenabled</code> is set to <code class="highlighter-rouge">true</code>, update your
configuration:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo calicoctl get bgpconfig --output yaml &gt; bgp.yaml
</code></pre></div></div>

<p>Edit the <code class="highlighter-rouge">bgp.yaml</code> file, updating <code class="highlighter-rouge">nodeToNodeMeshEnabled</code> to <code class="highlighter-rouge">false</code>. Then
update Calico configuration by running:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo calicoctl replace -f bgp.yaml
</code></pre></div></div>

<h2 id="configure-calico-to-use-route-reflectors">Configure Calico to use Route Reflectors</h2>

<p>To configure Calico to use the Route Reflectors you need to know the AS number
for your network first. For that, run:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo calicoctl get nodes --output=wide
</code></pre></div></div>

<p>Now that you have the AS number, you can create the Calico configuration.
For each Route Reflector, customize and run the following snippet:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo calicoctl create -f - &lt;&lt; EOF
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: bgppeer-global
spec:
  peerIP: &lt;IP_RR&gt;
  asNumber: &lt;AS_NUMBER&gt;
EOF
</code></pre></div></div>

<p>Where:</p>
<ul>
  <li><code class="highlighter-rouge">IP_RR</code> is the IP of the node where the Route Reflector pod is deployed.</li>
  <li><code class="highlighter-rouge">AS_NUMBER</code> is the same <code class="highlighter-rouge">AS number</code> for your nodes.</li>
</ul>

<p>You can learn more about this configuration in the
<a href="https://docs.projectcalico.org/v3.1/usage/routereflector/calico-routereflector">Calico documentation</a>.</p>

<h2 id="stop-calico-node-pods">Stop calico-node pods</h2>

<p>If you have <code class="highlighter-rouge">calico-node</code> pods running on the nodes dedicated for running the
Route Reflector, manually delete them. This ensures that you don’t have them
both running on the same node.</p>

<p>Using your UCP client bundle, run:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Find the Pod name
kubectl get pods -n kube-system -o wide | grep &lt;node-name&gt;

# Delete the Pod
kubectl delete pod -n kube-system &lt;pod-name&gt;
</code></pre></div></div>

<h2 id="validate-peers">Validate peers</h2>

<p>Now you can check that other <code class="highlighter-rouge">calico-node</code> pods running on other nodes are
peering with the Route Reflector:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo calicoctl node status
</code></pre></div></div>

<p>You should see something like:</p>

<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>IPv4 BGP status
+--------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE |  SINCE   |    INFO     |
+--------------+-----------+-------+----------+-------------+
| 172.31.24.86 | global    | up    | 23:10:04 | Established |
+--------------+-----------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.
</code></pre></div></div>