277 lines
7.0 KiB
Markdown
277 lines
7.0 KiB
Markdown
# Use dfget daemon as HTTP proxy for containerd
|
|
|
|
Currently, `ctr` command of containerd doesn't support private registries with `registry-mirrors`,
|
|
in order to do so, we need to use HTTP proxy for containerd.
|
|
|
|
## Quick Start
|
|
|
|
### Step 1: Generate CA certificate for HTTP proxy
|
|
|
|
Generate a CA certificate private key.
|
|
|
|
```bash
|
|
openssl genrsa -out ca.key 2048
|
|
```
|
|
|
|
Open openssl config file `openssl.conf`. Note set `basicConstraints` to true, that you can modify the values.
|
|
|
|
```text
|
|
[ req ]
|
|
#default_bits = 2048
|
|
#default_md = sha256
|
|
#default_keyfile = privkey.pem
|
|
distinguished_name = req_distinguished_name
|
|
attributes = req_attributes
|
|
extensions = v3_ca
|
|
req_extensions = v3_ca
|
|
|
|
[ req_distinguished_name ]
|
|
countryName = Country Name (2 letter code)
|
|
countryName_min = 2
|
|
countryName_max = 2
|
|
stateOrProvinceName = State or Province Name (full name)
|
|
localityName = Locality Name (eg, city)
|
|
0.organizationName = Organization Name (eg, company)
|
|
organizationalUnitName = Organizational Unit Name (eg, section)
|
|
commonName = Common Name (eg, fully qualified host name)
|
|
commonName_max = 64
|
|
emailAddress = Email Address
|
|
emailAddress_max = 64
|
|
|
|
[ req_attributes ]
|
|
challengePassword = A challenge password
|
|
challengePassword_min = 4
|
|
challengePassword_max = 20
|
|
|
|
[ v3_ca ]
|
|
basicConstraints = CA:TRUE
|
|
```
|
|
|
|
Generate the CA certificate.
|
|
|
|
```bash
|
|
openssl req -new -key ca.key -nodes -out ca.csr -config openssl.conf
|
|
openssl x509 -req -days 36500 -extfile openssl.conf -extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt
|
|
```
|
|
|
|
### Step 2: Configure dfget daemon
|
|
|
|
To use dfget daemon as HTTP proxy, first you need to append a proxy rule in
|
|
`/etc/dragonfly/dfget.yaml`, This will proxy `your.private.registry`'s requests for image layers:
|
|
|
|
```yaml
|
|
proxy:
|
|
security:
|
|
insecure: true
|
|
tcpListen:
|
|
listen: 0.0.0.0
|
|
port: 65001
|
|
proxies:
|
|
- regx: blobs/sha256.*
|
|
hijackHTTPS:
|
|
# CA certificate's path used to hijack https requests
|
|
cert: ca.crt
|
|
key: ca.key
|
|
hosts:
|
|
- regx: your.private.registry
|
|
```
|
|
|
|
### Step 3: Configure containerd
|
|
|
|
Set dfget damone as `HTTP_PROXY` and `HTTPS_PROXY` for containerd in
|
|
`/etc/systemd/system/containerd.service.d/http-proxy.conf`:
|
|
|
|
```
|
|
[Service]
|
|
Environment="HTTP_PROXY=http://127.0.0.1:65001"
|
|
Environment="HTTPS_PROXY=http://127.0.0.1:65001"
|
|
```
|
|
|
|
### Step 4: Pull images with proxy
|
|
|
|
Through the above steps, we can start to validate if Dragonfly works as expected.
|
|
|
|
And you can pull the image as usual, for example:
|
|
|
|
```bash
|
|
ctr image pull your.private.registry/namespace/image:latest
|
|
```
|
|
|
|
## Custom assets
|
|
|
|
### Registry uses a self-signed certificate
|
|
|
|
If your registry uses a self-signed certificate, you can either choose to
|
|
ignore the certificate error with:
|
|
|
|
```yaml
|
|
proxy:
|
|
security:
|
|
insecure: true
|
|
tcpListen:
|
|
listen: 0.0.0.0
|
|
port: 65001
|
|
proxies:
|
|
- regx: blobs/sha256.*
|
|
hijackHTTPS:
|
|
# CA certificate's path used to hijack https requests
|
|
cert: ca.crt
|
|
key: ca.key
|
|
hosts:
|
|
- regx: your.private.registry
|
|
insecure: true
|
|
```
|
|
|
|
Or provide a certificate with:
|
|
|
|
```yaml
|
|
proxy:
|
|
security:
|
|
insecure: true
|
|
tcpListen:
|
|
listen: 0.0.0.0
|
|
port: 65001
|
|
proxies:
|
|
- regx: blobs/sha256.*
|
|
hijackHTTPS:
|
|
# CA certificate's path used to hijack https requests
|
|
cert: ca.crt
|
|
key: ca.key
|
|
hosts:
|
|
- regx: your.private.registry
|
|
certs: ["server.crt"]
|
|
```
|
|
|
|
You can get the certificate of your server with:
|
|
|
|
```
|
|
openssl x509 -in <(openssl s_client -showcerts -servername xxx -connect xxx:443 -prexit 2>/dev/null)
|
|
```
|
|
|
|
# Use dfget daemon as registry mirror for Containerd with CRI support
|
|
|
|
From v1.1.0, Containerd supports registry mirrors, we can configure Containerd via this feature for HA.
|
|
|
|
## Quick Start
|
|
|
|
### Step 1: Configure dfget daemon
|
|
|
|
To use dfget daemon as registry mirror, first you need to ensure configuration in `/etc/dragonfly/dfget.yaml`:
|
|
|
|
```yaml
|
|
proxy:
|
|
security:
|
|
insecure: true
|
|
tcpListen:
|
|
listen: 0.0.0.0
|
|
port: 65001
|
|
registryMirror:
|
|
# multiple registries support, if only mirror single registry, disable this
|
|
dynamic: true
|
|
url: https://index.docker.io
|
|
proxies:
|
|
- regx: blobs/sha256.*
|
|
```
|
|
|
|
Run dfget daemon
|
|
|
|
```shell
|
|
dfget daemon
|
|
```
|
|
|
|
## Step 2: Configure Containerd
|
|
|
|
### Option 1: Single Registry
|
|
|
|
Enable mirrors in Containerd registries configuration in
|
|
`/etc/containerd/config.toml`:
|
|
|
|
```toml
|
|
# explicitly use v2 config format, if already v2, skip the "version = 2"
|
|
version = 2
|
|
|
|
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
|
|
endpoint = ["http://127.0.0.1:65001","https://registry-1.docker.io"]
|
|
```
|
|
|
|
In this config, there is two mirror endpoints for "docker.io", Containerd will pull images with `http://127.0.0.1:65001` first.
|
|
If `http://127.0.0.1:65001` is not available, the default `https://registry-1.docker.io` will be used for HA.
|
|
|
|
> More details about Containerd configuration: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/registry.md#configure-registry-endpoint
|
|
|
|
> Containerd has deprecated the above config from v1.4.0, new format for reference: https://github.com/containerd/containerd/blob/v1.5.2/docs/cri/config.md#registry-configuration
|
|
|
|
### Option 2: Multiple Registries
|
|
|
|
This option only supports Containerd 1.5.0+.
|
|
|
|
#### 1. Enable Containerd Registries Config Path
|
|
|
|
Enable mirrors in Containerd registries config path in
|
|
`/etc/containerd/config.toml`:
|
|
|
|
```toml
|
|
# explicitly use v2 config format, if already v2, skip the "version = 2"
|
|
version = 2
|
|
|
|
[plugins."io.containerd.grpc.v1.cri".registry]
|
|
config_path = "/etc/containerd/certs.d"
|
|
```
|
|
|
|
#### 2. Generate Per Registry hosts.toml
|
|
|
|
##### Option 1: Generate hosts.toml manually
|
|
|
|
Path: `/etc/containerd/certs.d/example.com/hosts.toml`
|
|
|
|
Replace `example.com` according the different registry domains.
|
|
|
|
Content:
|
|
|
|
```toml
|
|
server = "https://example.com"
|
|
|
|
[host."http://127.0.0.1:65001"]
|
|
capabilities = ["pull", "resolve"]
|
|
[host."http://127.0.0.1:65001".header]
|
|
X-Dragonfly-Registry = ["https://example.com"]
|
|
```
|
|
|
|
##### Option 2: Generate hosts.toml automatically
|
|
|
|
You can also generate hosts.toml with https://github.com/dragonflyoss/Dragonfly2/blob/main/hack/gen-containerd-hosts.sh
|
|
|
|
```shell
|
|
bash gen-containerd-hosts.sh example.com
|
|
```
|
|
|
|
> More details about registry configuration: https://github.com/containerd/containerd/blob/main/docs/hosts.md#registry-configuration---examples
|
|
|
|
## Step 3: Restart Containerd Daemon
|
|
|
|
```
|
|
systemctl restart containerd
|
|
```
|
|
|
|
## Step 4: Pull Image
|
|
|
|
You can pull image like this:
|
|
|
|
```
|
|
crictl pull docker.io/library/busybox
|
|
```
|
|
|
|
## Step 5: Validate Dragonfly
|
|
|
|
You can execute the following command to check if the busybox image is distributed via Dragonfly.
|
|
|
|
```bash
|
|
grep 'register peer task result' /var/log/dragonfly/daemon/*.log
|
|
```
|
|
|
|
If the output of command above has content like
|
|
|
|
```
|
|
{"level":"info","ts":"2021-02-23 20:03:20.306","caller":"client/client.go:83","msg":"register peer task result:true[200] for taskId:adf62a86f001e17037eedeaaba3393f3519b80ce,peerIp:10.15.233.91,securityDomain:,idc:,scheduler:127.0.0.1:8002","peerId":"10.15.233.91-65000-43096-1614081800301788000","errMsg":null}
|
|
```
|