dragonfly/docs/en/runtime-integration/containerd.md

Use dfget daemon as HTTP proxy for containerd

Currently, ctr command of containerd doesn't support private registries with registry-mirrors, in order to do so, we need to use HTTP proxy for containerd.

Quick Start

Step 1: Generate CA certificate for HTTP proxy

Generate a CA certificate private key.

openssl genrsa -out ca.key 2048

Open openssl config file openssl.conf. Note set basicConstraints to true, that you can modify the values.

[ req ]
#default_bits		= 2048
#default_md		= sha256
#default_keyfile 	= privkey.pem
distinguished_name	= req_distinguished_name
attributes		= req_attributes
extensions               = v3_ca
req_extensions           = v3_ca

[ req_distinguished_name ]
countryName			= Country Name (2 letter code)
countryName_min			= 2
countryName_max			= 2
stateOrProvinceName		= State or Province Name (full name)
localityName			= Locality Name (eg, city)
0.organizationName		= Organization Name (eg, company)
organizationalUnitName		= Organizational Unit Name (eg, section)
commonName			= Common Name (eg, fully qualified host name)
commonName_max			= 64
emailAddress			= Email Address
emailAddress_max		= 64

[ req_attributes ]
challengePassword		= A challenge password
challengePassword_min		= 4
challengePassword_max		= 20

[ v3_ca ]
basicConstraints         = CA:TRUE

Generate the CA certificate.

openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf
openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt

Step 2: Configure dfget daemon

To use dfget daemon as HTTP proxy, first you need to append a proxy rule in /etc/dragonfly/dfget.yaml, This will proxy your.private.registry's requests for image layers:

proxy:
  security:
    insecure: true
  tcpListen:
    listen: 0.0.0.0
    port: 65001
  proxies:
    - regx: blobs/sha256.*
  hijackHTTPS:
    # CA certificate's path used to hijack https requests
    cert: ca.crt
    key: ca.key
    hosts:
      - regx: your.private.registry

Step 3: Configure containerd

Set dfget damone as HTTP_PROXY and HTTPS_PROXY for containerd in /etc/systemd/system/containerd.service.d/http-proxy.conf:

[Service]
Environment="HTTP_PROXY=http://127.0.0.1:65001"
Environment="HTTPS_PROXY=http://127.0.0.1:65001"

Step 4: Pull images with proxy

Through the above steps, we can start to validate if Dragonfly works as expected.

And you can pull the image as usual, for example:

ctr image pull your.private.registry/namespace/image:latest

Custom assets

Registry uses a self-signed certificate

If your registry uses a self-signed certificate, you can either choose to ignore the certificate error with:

proxy:
  security:
    insecure: true
  tcpListen:
    listen: 0.0.0.0
    port: 65001
  proxies:
    - regx: blobs/sha256.*
  hijackHTTPS:
    # CA certificate's path used to hijack https requests
    cert: ca.crt
    key: ca.key
    hosts:
      - regx: your.private.registry
        insecure: true

Or provide a certificate with:

proxy:
  security:
    insecure: true
  tcpListen:
    listen: 0.0.0.0
    port: 65001
  proxies:
    - regx: blobs/sha256.*
  hijackHTTPS:
    # CA certificate's path used to hijack https requests
    cert: ca.crt
    key: ca.key
    hosts:
      - regx: your.private.registry
        certs: ["server.crt"]

You can get the certificate of your server with:

openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null)

Use dfget daemon as registry mirror for Containerd with CRI support

From v1.1.0, Containerd supports registry mirrors, we can configure Containerd via this feature for HA.

Quick Start

Step 1: Configure dfget daemon

To use dfget daemon as registry mirror, first you need to ensure configuration in /etc/dragonfly/dfget.yaml:

proxy:
  security:
    insecure: true
  tcpListen:
    listen: 0.0.0.0
    port: 65001
  registryMirror:
    # multiple registries support, if only mirror single registry, disable this
    dynamic: true
    url: https://index.docker.io
  proxies:
    - regx: blobs/sha256.*

Run dfget daemon

dfget daemon

Step 2: Configure Containerd

Option 1: Single Registry

Enable mirrors in Containerd registries configuration in /etc/containerd/config.toml:

# explicitly use v2 config format, if already v2, skip the "version = 2"
version = 2

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
  endpoint = ["http://127.0.0.1:65001","https://registry-1.docker.io"]

In this config, there is two mirror endpoints for "docker.io", Containerd will pull images with http://127.0.0.1:65001 first. If http://127.0.0.1:65001 is not available, the default https://registry-1.docker.io will be used for HA.

More details about Containerd configuration: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/registry.md#configure-registry-endpoint

Containerd has deprecated the above config from v1.4.0, new format for reference: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/config.md#registry-configuration

Option 2: Multiple Registries

This option only supports Containerd 1.5.0+.

1. Enable Containerd Registries Config Path

Enable mirrors in Containerd registries config path in /etc/containerd/config.toml:

# explicitly use v2 config format, if already v2, skip the "version = 2"
version = 2

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

2. Generate Per Registry hosts.toml

Option 1: Generate hosts.toml manually

Path: /etc/containerd/certs.d/example.com/hosts.toml

Replace example.com according the different registry domains.

Content:

server = "https://example.com"

[host."http://127.0.0.1:65001"]
  capabilities = ["pull", "resolve"]
  [host."http://127.0.0.1:65001".header]
    X-Dragonfly-Registry = ["https://example.com"]

Option 2: Generate hosts.toml automatically

You can also generate hosts.toml with https://github.com/dragonflyoss/Dragonfly2/blob/main/hack/gen-containerd-hosts.sh

bash gen-containerd-hosts.sh example.com

More details about registry configuration: https://github.com/containerd/containerd/blob/main/docs/hosts.md#registry-configuration---examples

Step 3: Restart Containerd Daemon

systemctl restart containerd

Step 4: Pull Image

You can pull image like this:

crictl pull docker.io/library/busybox

Step 5: Validate Dragonfly

You can execute the following command to check if the busybox image is distributed via Dragonfly.

grep 'register peer task result' /var/log/dragonfly/daemon/*.log

If the output of command above has content like

{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null}