docs: changes to singularity documentation and document routing (#90)

Signed-off-by: zhaoxinxin <1186037180@qq.com>
This commit is contained in:
Zhaoxinxin 2024-04-09 11:29:36 +08:00 committed by GitHub
parent 26463b577e
commit d841bdc91a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
34 changed files with 88 additions and 76 deletions

View File

@ -1,6 +1,7 @@
---
id: binaries
title: Binaries
slug: /getting-started/installation/binaries/
---
This guide shows how to install the Dragonfly. Dragonfly can be installed either from source, or from pre-built binary releases.
@ -246,7 +247,7 @@ telnet 127.0.0.1 65002
Configure Dfdaemon yaml file, The default path in Linux is `/etc/dragonfly/dfget.yaml` in linux,
The default path in Darwin is `$HOME/.dragonfly/config/dfget.yaml`,
refer to [Dfdaemon](../../reference/configuration/dfdaemon.md)
refer to [Dfdaemon](../../reference/configuration/dfdaemon.md).
Set the `scheduler.manager.netAddrs.addr` address in the configuration file to your actual address.
Configuration content is as follows:

View File

@ -1,6 +1,7 @@
---
id: helm-charts
title: Helm Charts
slug: /getting-started/installation/helm-charts/
---
Documentation for deploying Dragonfly on kubernetes using helm.

View File

@ -1,7 +1,7 @@
---
id: quick-start
title: Quick Start
slug: /getting-started/quick-start/
slug: /getting-started/quick-start
---
Table of contents:

View File

@ -1,7 +1,7 @@
---
id: containerd
title: containerd
slug: /setup/runtime/containerd
slug: /operations/integrations/container-runtime/containerd/
---
Documentation for setting Dragonfly's container runtime to containerd.

View File

@ -1,7 +1,7 @@
---
id: cri-o
title: CRI-O
slug: /setup/runtime/cri-o
slug: /operations/integrations/container-runtime/cri-o/
---
Use dfget daemon as registry mirror for CRI-O

View File

@ -1,7 +1,7 @@
---
id: docker
title: Docker
slug: /setup/runtime/docker
slug: /operations/integrations/container-runtime/docker/
---
Use dfget daemon as HTTP proxy for docker daemon

View File

@ -1,6 +1,7 @@
---
id: nydus
title: Nydus
slug: /operations/integrations/container-runtime/nydus/
---
This document will help you experience how to use dragonfly with nydus.

View File

@ -1,7 +1,6 @@
---
id: singularity
title: Singularity/Apptainer
slug: /setup/runtime/singularity
---
This documentation will help you to integrate Dragonfly into
@ -9,5 +8,5 @@ Singularity/Apptainer container runtime.
Table of contents:
- [Oras Resource Client Mode](./singularity/oras_resource_client.md)
- [Oras Resource Client Mode](./singularity/oras-resource-client.md)
- [Proxy Mode](./singularity/proxy.md)

View File

@ -1,22 +1,20 @@
---
id: singularity-oras-resource-client
title: Oras Resource Client Mode
slug: /setup/runtime/singularity/oras-resource-client
title: ORAS Resource Client
slug: /operations/integrations/container-runtime/singularity/oras-resource-client/
---
## Oras Resource Client {#oras-resource-client}
We can use oras resource client to pull image using Dragonfly, the key is the scheme used for the resource client.
Documentation for setting Dragonfly's container runtime to ORAS Resource Client.
This method of image pull through Dragonfly is more efficient when compared to proxy method as
it avoids TLS termination, reduces CPU time and download time as it creates hardlink(insead of copy)
for subsequent file download after downloading the image from source for first time.
## Quick Start {#quick-start}
## Configure dfget daemon {#configure-dfget-daemon}
### Step 1: Configure dfget daemon {#step-1-configure-dfget-daemon}
To use oras resource client to pull image ensure below configuraion in `/etc/dragonfly/dfget.yaml:`.
Configure Dfdaemon yaml file, The default path in Linux is `/etc/dragonfly/dfget.yaml` in linux,
The default path in Darwin is `$HOME/.dragonfly/config/dfget.yaml`,
refer to [Dfdaemon](../../../../reference/configuration/dfdaemon.md).
```yaml
# Peer task storage option.
@ -33,7 +31,7 @@ storage:
# Set to ture for reusing underlying storage for same task id.
multiplex: true
# The singularity oras resources, most of it is same with https scheme
# The singularity ORAS resources, most of it is same with https scheme.
oras:
proxy:
dialTimeout: 30s
@ -46,26 +44,23 @@ oras:
insecureSkipVerify: true
```
### Step 2: Pull images through oras resource client {#step-2-pull-images-through-oras-resource-client}
## ORAS Resource Client downloads images through Dragonfly {#oras-resource-client-downloads-images-through-Dragonfly}
Through the above steps, we can start to validate if Dragonfly works as expected.
And you can pull the image through oras resource client as below:
Pull the image through ORAS Resource Client:
```shell
dfget -u "oras://hostname/path/image:tag" -O /path/to/output
```
### Step 3: Validate Dragonfly {#step-3-validate-dragonfly}
### Verify {#verify}
You can execute the following command to
check if the image is distributed via Dragonfly.
You can execute the following command to check if the image is distributed via Dragonfly.
```shell
grep "peer task done" /var/log/dragonfly/daemon/core.log
```
If the output of command above has content like
The expected output is as follows:
```shell
{

View File

@ -1,12 +1,12 @@
---
id: singularity-proxy
title: Proxy Mode
slug: /setup/runtime/singularity/proxy
title: HTTP Proxy
slug: /operations/integrations/container-runtime/singularity/proxy/
---
Uses dfget daemon as proxy for Singularity/Apptainer.
Documentation for setting up Dragonfly's container runtime as an HTTP Proxy for Singularity/Apptainer.
## Step 1: Generate CA certificate {#step-1-generate-ca-certificate}
## Generate CA certificate {#generate-ca-certificate}
Generate an RSA private key.
@ -57,18 +57,21 @@ openssl x509 -req -days 36500 -extfile openssl.conf \
-extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt
```
## Step 2: Configure dfget daemon {#step-2-configure-dfget-daemon}
## Configure dfget daemon {#configure-dfget-daemon}
To use dfget daemon as proxy, first you need to append a proxy rule in
`/etc/dragonfly/dfget.yaml`, This will proxy
`your.private.registry's` requests for image layers:
Configure Dfdaemon yaml file, The default path in Linux is `/etc/dragonfly/dfget.yaml` in linux,
The default path in Darwin is `$HOME/.dragonfly/config/dfget.yaml`,
refer to [Dfdaemon](../../../../reference/configuration/dfdaemon.md).
Set the `registryMirror.url` and `hijackHTTPS.hosts.regx` addresses under the configuration
file to your actual addresses. Configuration content is as follows:
```yaml
registryMirror:
# When enable, using header "X-Dragonfly-Registry" for remote instead of url.
dynamic: true
# URL for the registry mirror.
url: <your.private.registry>
url: your_registry_mirror_url
# Whether to ignore https certificate errors.
insecure: false
# Optional certificates if the remote server uses self-signed certificates.
@ -87,29 +90,26 @@ hijackHTTPS:
cert: ca.crt
key: ca.key
hosts:
- regx: <your.private.registry>
- regx: your_hijack_https_hosts
```
## Step 3: Pull images with proxy {#step-4-pull-images-with-proxy}
## HTTP Proxy downloads images through Dragonfly {#http-proxy-downloads-images-through-Dragonfly}
Through the above steps, we can start to validate if Dragonfly works as expected.
And you can pull the image through proxy as below:
Pull images through HTTP Proxy:
```bash
no_proxy='' NO_PROXY='' HTTPS_PROXY=127.0.0.1:65001 singularity pull oras://hostname/path/image:tag
```
## Step 4: Validate Dragonfly {#step-4-validate-dragonfly}
### Verify {#verify}
You can execute the following command to
check if the image is distributed via Dragonfly.
You can execute the following command to check if the image is distributed via Dragonfly.
```shell
grep "peer task done" /var/log/dragonfly/daemon/core.log
```
If the output of command above has content like
The expected output is as follows:
```shell
{

View File

@ -1,6 +1,7 @@
---
id: stargz
title: eStargz
slug: /operations/integrations/container-runtime/stargz/
---
This document will help you experience how to use dragonfly with eStargz.

View File

@ -1,6 +1,7 @@
---
id: git-lfs
title: Git LFS
slug: /operations/integrations/git-lfs/
---
## What is Git LFS?

View File

@ -1,6 +1,7 @@
---
id: harbor
title: Harbor
slug: /operations/integrations/harbor/
---
This document will help you experience how to use dragonfly with harbor.

View File

@ -1,6 +1,7 @@
---
id: hugging-face
title: Hugging Face
slug: /operations/integrations/hugging-face/
---
This document will help you experience how to use dragonfly with hugging face.

View File

@ -1,6 +1,7 @@
---
id: torchserve
title: TorchServe
slug: /operations/integrations/torchserve/
---
This document will help you experience how to use dragonfly with [TorchServe](https://github.com/pytorch/serve).

View File

@ -1,6 +1,7 @@
---
id: triton-server
title: Triton Server
slug: /operations/integrations/triton-server/
---
This document will help you experience how to use dragonfly with [TritonServe](https://github.com/pytorch/serve).

View File

@ -1,7 +1,7 @@
---
id: upgrade
title: Upgrade
slug: /setup/upgrade
slug: /operations/integrations/upgrade/
---
## Upgrade the cluster deployed by Helm {#upgrade-the-cluster-deployed-by-helm}

View File

@ -1,6 +1,7 @@
---
id: binaries
title: 使用二进制文件安装
slug: /getting-started/installation/binaries/
---
文档的目标是帮助您快速开始使用源码或构建的二进制版本部署 Dragonfly。

View File

@ -1,6 +1,7 @@
---
id: helm-charts
title: Helm Charts
slug: /getting-started/installation/helm-charts/
---
文档的目标是帮助您快速开始使用 Helm Charts 部署 Dragonfly。

View File

@ -1,7 +1,7 @@
---
id: containerd
title: containerd
slug: /setup/runtime/containerd
slug: /operations/integrations/container-runtime/containerd/
---
文档的目标是帮助您将 Dragonfly 的容器运行时设置为 containerd。

View File

@ -1,7 +1,7 @@
---
id: cri-o
title: CRI-O
slug: /setup/runtime/cri-o
slug: /operations/integrations/container-runtime/cri-o/
---
使用 dfdaemon 支持 CRI-O

View File

@ -1,7 +1,7 @@
---
id: docker
title: Docker
slug: /setup/runtime/docker
slug: /operations/integrations/container-runtime/docker/
---
使用 dfget daemon 作为 docker daemon 的 http 代理

View File

@ -1,6 +1,7 @@
---
id: nydus
title: Nydus
slug: /operations/integrations/container-runtime/nydus/
---
本文档将帮助您将 Dragonfly 与 Nydus 一起使用。

View File

@ -1,12 +1,11 @@
---
id: singularity
title: Singularity/Apptainer
slug: /setup/runtime/singularity
---
该文档将帮助你集成 Dragonfly 到 Singularity/Apptainer。
集成方式:
- [Oras 资源客户端模式](./singularity/oras_resource_client.md)
- [Oras 资源客户端模式](./singularity/oras-resource-client.md)
- [HTTP Proxy 模式](./singularity/proxy.md)

View File

@ -1,26 +1,23 @@
---
id: singularity-oras-resource-client
title: Oras 资源客户端模式
slug: /setup/runtime/singularity/oras-resource-client
title: ORAS 资源客户端
slug: /operations/integrations/container-runtime/singularity/oras-resource-client/
---
## Oras 资源客户端
使用 oras 的资源客户端使用 Dragonfly 拉取镜像。
文档的目标是帮助您将 Dragonfly 的容器运行时设置为 ORAS 的资源客户端。
通过 Dragonfly 资源客户端拉取镜像的这种方法比代理方法相比更加高效,因为它避免了 TLS减少了 CPU 资源使用量和下载时间,
因为它在首次从源下载镜像后为后续文件下载创建了硬链接,而不是复制一份完全一样的资源。
## 快速开始
## 配置 Dfdaemon
### 步骤 1: 配置 dfdaemon
下面为镜像仓库的 dfdaemon 配置,在路径 `/etc/dragonfly/dfget.yaml`:
编辑配置文件 Linux 环境下默认 Dfdaemon 配置路径为 `/etc/dragonfly/dfget.yaml`
Darwin 环境下默认 Dfdaemon 配置路径为 `$HOME/.dragonfly/config/dfget.yaml`,参考文档 [Dfdaemon](../../../../reference/configuration/dfdaemon.md)。
```yaml
# Peer task storage option.
storage:
# Task data expire time,
# Task data expire time.
# when there is no access to a task data, this task will be gc.
taskExpireTime: 6h
strategy: io.d7y.storage.v2.advance.
@ -32,7 +29,7 @@ storage:
# Set to ture for reusing underlying storage for same task id.
multiplex: true
# The singularity oras resources, most of it is same with https scheme
# The singularity oras resources, most of it is same with https scheme.
oras:
proxy:
dialTimeout: 30s
@ -45,7 +42,7 @@ oras:
insecureSkipVerify: true
```
### 步骤 2: 使用 oras 资源客户端下载镜像
## ORAS 资源客户端通过 Dragonfly 下载镜像
使用以下命令拉取镜像:
@ -53,7 +50,7 @@ oras:
dfget -u "oras://hostname/path/image:tag" -O /path/to/output
```
### 步骤 3: 验证 Dragonfly 拉取成功
### 验证镜像下载成功
可以查看日志,判断镜像正常拉取。

View File

@ -1,12 +1,12 @@
---
id: singularity-proxy
title: HTTP Proxy 模式
slug: /setup/runtime/singularity/proxy
title: HTTP Proxy
slug: /operations/integrations/container-runtime/singularity/proxy/
---
使用 dfget daemon 作为 Singularity/Apptainer 的 http 代理。
文档的目标是帮助您将 Dragonfly 的容器运行时设置为 Singularity/Apptainer 的 HTTP 代理。
## 步骤 1为 http 代理生成 CA 证书
## 为 HTTP 代理生成 CA 证书
生成一个 CA 证书私钥。
@ -56,17 +56,19 @@ openssl x509 -req -days 36500 -extfile openssl.conf \
-extensions v3_ca -in ca.csr -signkey ca.key -out ca.crt
```
## 步骤 2配置 dfget daemon
## 配置 Dfdaemon
为了将 dfget daemon 作为 http 代理使用,首先你需要在 `/etc/dragonfly/dfget.yaml` 中增加一条代理规则,
它将会代理 `your.private.registry` 对镜像层的请求:
编辑配置文件 Linux 环境下默认 Dfdaemon 配置路径为 `/etc/dragonfly/dfget.yaml`
Darwin 环境下默认 Dfdaemon 配置路径为 `$HOME/.dragonfly/config/dfget.yaml`,参考文档 [Dfdaemon](../../../../reference/configuration/dfdaemon.md)。
配置文件下设置 `registryMirror.url``hijackHTTPS.hosts.regx` 地址为你的实际地址,配置内容如下:
```yaml
registryMirror:
# When enable, using header "X-Dragonfly-Registry" for remote instead of url.
dynamic: true
# URL for the registry mirror.
url: <your.private.registry>
url: your_registry_mirror_url
# Whether to ignore https certificate errors.
insecure: false
# Optional certificates if the remote server uses self-signed certificates.
@ -85,16 +87,18 @@ hijackHTTPS:
cert: ca.crt
key: ca.key
hosts:
- regx: <your.private.registry>
- regx: your_hijack_https_hosts
```
## 步骤 3使用代理拉取镜像
## HTTP 代理通过 Dragonfly 下载镜像
使用以下命令拉取镜像:
```bash
no_proxy='' NO_PROXY='' HTTPS_PROXY=127.0.0.1:65001 singularity pull oras://hostname/path/image:tag
```
## 步骤 4: 验证 Dragonfly 拉取成功
### 验证镜像下载成功
可以查看日志,判断镜像正常拉取。

View File

@ -1,6 +1,7 @@
---
id: stargz
title: eStargz
slug: /operations/integrations/container-runtime/stargz/
---
本文档将帮助您将 Dragonfly 与 eStargz 一起使用。

View File

@ -1,6 +1,7 @@
---
id: git-lfs
title: Git LFS
slug: /operations/integrations/git-lfs/
---
## Git LFS 是什么?

View File

@ -1,6 +1,7 @@
---
id: harbor
title: Harbor
slug: /operations/integrations/harbor/
---
本文档将帮助您将 Dragonfly 与 Harbor 一起使用。

View File

@ -1,6 +1,7 @@
---
id: hugging-face
title: Hugging Face
slug: /operations/integrations/hugging-face/
---
本文档将帮助您将 Dragonfly 与 Hugging Face 一起使用。

View File

@ -1,6 +1,7 @@
---
id: torchserve
title: TorchServe
slug: /operations/integrations/torchserve/
---
本文主要解决在 [TorchServe](https://github.com/pytorch/serve) 模型拉取时,存在的中心化的模型仓库带宽瓶颈问题。

View File

@ -1,6 +1,7 @@
---
id: triton-server
title: Triton Server
slug: /operations/integrations/triton-server/
---
本文主要解决在 [Triton Server](https://github.com/triton-inference-server/server) 模型拉取时,存在的中心化的模型仓库带宽瓶颈问题。

View File

@ -1,7 +1,7 @@
---
id: upgrade
title: 升级
slug: /setup/upgrade
slug: /operations/integrations/upgrade/
---
## 升级 Helm 部署的集群

View File

@ -18,9 +18,9 @@
"version": "./scripts/version.sh"
},
"dependencies": {
"@docusaurus/core": "^2.0.0",
"@docusaurus/preset-classic": "^2.0.0",
"@docusaurus/core": "^2.4.0",
"@docusaurus/plugin-content-docs": "2.4.3",
"@docusaurus/preset-classic": "^2.4.0",
"@mdx-js/react": "^1.6.21",
"acorn": "^8.7.1",
"clsx": "^2.0.0",