Rewrite README.md (#166)
* docs: add badge Signed-off-by: Gaius <gaius.qi@gmail.com> * chore: add badges Signed-off-by: Gaius <gaius.qi@gmail.com> * feat: change github build status badge to CI badge Signed-off-by: Gaius <gaius.qi@gmail.com> * chore: add todo badge Signed-off-by: Gaius <gaius.qi@gmail.com> * feat: build the document structure Signed-off-by: Gaius <gaius.qi@gmail.com> * docs: init readme Signed-off-by: Gaius <gaius.qi@gmail.com> * docs: rename Dragonfly2 to Dragonfly Signed-off-by: Gaius <gaius.qi@gmail.com> * docs: rename Dragonfly2 to Dragonfly Signed-off-by: Gaius <gaius.qi@gmail.com> * chore: rename Dragonfly2 to Dragonfly Signed-off-by: Gaius <gaius.qi@gmail.com> * docs: add community describe Signed-off-by: Gaius <gaius.qi@gmail.com> * docs: create architecture Signed-off-by: Gaius <gaius.qi@gmail.com>
|
|
@ -1,6 +1,6 @@
|
|||
# Contributing to Dragonfly2
|
||||
# Contributing to Dragonfly
|
||||
|
||||
It is warmly welcomed if you have interest to hack on Dragonfly2. First, we encourage this kind of willing very much. And here is a list of contributing guide for you.
|
||||
It is warmly welcomed if you have interest to hack on Dragonfly. First, we encourage this kind of willing very much. And here is a list of contributing guide for you.
|
||||
|
||||
## Topics
|
||||
|
||||
|
|
@ -8,17 +8,17 @@ It is warmly welcomed if you have interest to hack on Dragonfly2. First, we enco
|
|||
* [Reporting general issues](#reporting-general-issues)
|
||||
* [Code and doc contribution](#code-and-doc-contribution)
|
||||
* [Engage to help anything](#engage-to-help-anything)
|
||||
* [Join Dragonfly2 as a member](#join-dragonfly2-as-a-member)
|
||||
* [Join Dragonfly as a member](#join-dragonfly-as-a-member)
|
||||
|
||||
## Reporting security issues
|
||||
|
||||
Security issues are always treated seriously. As our usual principle, we discourage anyone to spread security issues. If you find a security issue of Dragonfly2, please do not discuss it in public and even do not open a public issue. Instead we encourage you to send us a private email to [Dragonfly2@noreply.github.com](mailto:Dragonfly2@noreply.github.com) to report this.
|
||||
Security issues are always treated seriously. As our usual principle, we discourage anyone to spread security issues. If you find a security issue of Dragonfly, please do not discuss it in public and even do not open a public issue. Instead we encourage you to send us a private email to [Dragonfly@noreply.github.com](mailto:Dragonfly@noreply.github.com) to report this.
|
||||
|
||||
## Reporting general issues
|
||||
|
||||
To be honest, we regard every user of Dragonfly2 as a very kind contributor. After experiencing Dragonfly2, you may have some feedback for the project. Then feel free to open an issue via [NEW ISSUE](https://github.com/dragonflyoss/Dragonfly2/issues/new).
|
||||
To be honest, we regard every user of Dragonfly as a very kind contributor. After experiencing Dragonfly, you may have some feedback for the project. Then feel free to open an issue via [NEW ISSUE](https://github.com/dragonflyoss/Dragonfly2/issues/new).
|
||||
|
||||
Since we collaborate project Dragonfly2 in a distributed way, we appreciate **WELL-WRITTEN**, **DETAILED**, **EXPLICIT** issue reports. To make the communication more efficient, we wish everyone could search if your issue is an existing one in the searching list. If you find it existing, please add your details in comments under the existing issue instead of opening a brand new one.
|
||||
Since we collaborate project Dragonfly in a distributed way, we appreciate **WELL-WRITTEN**, **DETAILED**, **EXPLICIT** issue reports. To make the communication more efficient, we wish everyone could search if your issue is an existing one in the searching list. If you find it existing, please add your details in comments under the existing issue instead of opening a brand new one.
|
||||
|
||||
To make the issue details as standard as possible, we setup an [ISSUE TEMPLATE](./.github/ISSUE_TEMPLATE) for issue reporters. You can find three kinds of issue templates there: question, bug report and feature request. Please **BE SURE** to follow the instructions to fill fields in template.
|
||||
|
||||
|
|
@ -39,7 +39,7 @@ Also we must remind that when filing a new issue, please remember to remove the
|
|||
|
||||
## Code and doc contribution
|
||||
|
||||
Every action to make project Dragonfly2 better is encouraged. On GitHub, every improvement for Dragonfly2 could be via a PR (short for pull request).
|
||||
Every action to make project Dragonfly better is encouraged. On GitHub, every improvement for Dragonfly could be via a PR (short for pull request).
|
||||
|
||||
* If you find a typo, try to fix it!
|
||||
* If you find a bug, try to fix it!
|
||||
|
|
@ -56,7 +56,7 @@ Actually it is impossible to list them completely. Just remember one princinple:
|
|||
|
||||
> WE ARE LOOKING FORWARD TO ANY PR FROM YOU.
|
||||
|
||||
Since you are ready to improve Dragonfly2 with a PR, we suggest you could take a look at the PR rules here.
|
||||
Since you are ready to improve Dragonfly with a PR, we suggest you could take a look at the PR rules here.
|
||||
|
||||
* [Workspace Preparation](#workspace-preparation)
|
||||
* [Branch Definition](#branch-definition)
|
||||
|
|
@ -69,7 +69,7 @@ Since you are ready to improve Dragonfly2 with a PR, we suggest you could take a
|
|||
|
||||
To put forward a PR, we assume you have registered a GitHub ID. Then you could finish the preparation in the following steps:
|
||||
|
||||
1. **FORK** Dragonfly2 to your repository. To make this work, you just need to click the button Fork in right-left of [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) main page. Then you will end up with your repository in `https://github.com/<your-username>/Dragonfly`, in which `your-username` is your GitHub username.
|
||||
1. **FORK** Dragonfly to your repository. To make this work, you just need to click the button Fork in right-left of [dragonflyoss/Dragonfly2](https://github.com/dragonflyoss/Dragonfly2) main page. Then you will end up with your repository in `https://github.com/<your-username>/Dragonfly`, in which `your-username` is your GitHub username.
|
||||
1. **CLONE** your own repository to develop locally. Use `git clone https://github.com/<your-username>/Dragonfly2.git` to clone repository to your local machine. Then you can create new branches to finish the change you wish to make.
|
||||
|
||||
1. **Set Remote** upstream to be `https://github.com/dragonflyoss/Dragonfly2.git` using the following two commands:
|
||||
|
|
@ -112,9 +112,9 @@ To put forward a PR, we assume you have registered a GitHub ID. Then you could f
|
|||
|
||||
### Branch Definition
|
||||
|
||||
Right now we assume every contribution via pull request is for [branch main](https://github.com/dragonflyoss/Dragonfly2/tree/main) in Dragonfly2. Before contributing, be aware of branch definition would help a lot.
|
||||
Right now we assume every contribution via pull request is for [branch main](https://github.com/dragonflyoss/Dragonfly2/tree/main) in Dragonfly. Before contributing, be aware of branch definition would help a lot.
|
||||
|
||||
As a contributor, keep in mind again that every contribution via pull request is for branch main. While in project Dragonfly2, there are several other branches, we generally call them rc branches, release branches and backport branches.
|
||||
As a contributor, keep in mind again that every contribution via pull request is for branch main. While in project Dragonfly, there are several other branches, we generally call them rc branches, release branches and backport branches.
|
||||
|
||||
Before officially releasing a version, we will checkout a rc(release candidate) branch. In this branch, we will test more than branch main.
|
||||
|
||||
|
|
@ -124,7 +124,7 @@ When backporting some fixes to existing released version, we will checkout backp
|
|||
|
||||
### Commit Rules
|
||||
|
||||
Actually in Dragonfly2, we take two rules serious when committing:
|
||||
Actually in Dragonfly, we take two rules serious when committing:
|
||||
|
||||
* [Commit Message](#commit-message)
|
||||
* [Commit Content](#commit-content)
|
||||
|
|
@ -159,11 +159,11 @@ No matter what the commit message, or commit content is, we do take more emphasi
|
|||
|
||||
### PR Description
|
||||
|
||||
PR is the only way to make change to Dragonfly2 project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request.
|
||||
PR is the only way to make change to Dragonfly project files. To help reviewers better get your purpose, PR description could not be too detailed. We encourage contributors to follow the [PR template](./.github/PULL_REQUEST_TEMPLATE.md) to finish the pull request.
|
||||
|
||||
### Developing Environment
|
||||
|
||||
As a contributor, if you want to make any contribution to Dragonfly2 project, we should reach an agreement on the version of tools used in the development environment.
|
||||
As a contributor, if you want to make any contribution to Dragonfly project, we should reach an agreement on the version of tools used in the development environment.
|
||||
Here are some dependents with specific version:
|
||||
|
||||
* golang : v1.15
|
||||
|
|
@ -171,11 +171,11 @@ Here are some dependents with specific version:
|
|||
* shellCheck : latest
|
||||
* docker: latest
|
||||
|
||||
When you develop the Dragonfly2 project at the local environment, you should use subcommands of Makefile to help yourself to check and build the latest version of Dragonfly2. For the convenience of developers, we use the docker to build Dragonfly2. It can reduce problems of the developing environment.
|
||||
When you develop the Dragonfly project at the local environment, you should use subcommands of Makefile to help yourself to check and build the latest version of Dragonfly. For the convenience of developers, we use the docker to build Dragonfly. It can reduce problems of the developing environment.
|
||||
|
||||
### Golang Dependency Management
|
||||
|
||||
The Dragonfly2 project uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages. This requires a working Go environment with version 1.15 or greater installed.
|
||||
The Dragonfly project uses [Go modules](https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more) to manage dependencies on external packages. This requires a working Go environment with version 1.15 or greater installed.
|
||||
|
||||
To add or update a new dependency, use the `go get` command:
|
||||
|
||||
|
|
@ -202,25 +202,25 @@ git commit
|
|||
|
||||
## Engage to help anything
|
||||
|
||||
We choose GitHub as the primary place for Dragonfly2 to collaborate. So the latest updates of Dragonfly2 are always here. Although contributions via PR is an explicit way to help, we still call for any other ways.
|
||||
We choose GitHub as the primary place for Dragonfly to collaborate. So the latest updates of Dragonfly are always here. Although contributions via PR is an explicit way to help, we still call for any other ways.
|
||||
|
||||
* reply to other's issues if you could;
|
||||
* help solve other user's problems;
|
||||
* help review other's PR design;
|
||||
* help review other's codes in PR;
|
||||
* discuss about Dragonfly2 to make things clearer;
|
||||
* advocate Dragonfly2 technology beyond GitHub;
|
||||
* write blogs on Dragonfly2 and so on.
|
||||
* discuss about Dragonfly to make things clearer;
|
||||
* advocate Dragonfly technology beyond GitHub;
|
||||
* write blogs on Dragonfly and so on.
|
||||
|
||||
In a word, **ANY HELP IS CONTRIBUTION.**
|
||||
|
||||
## Join Dragonfly2 as a member
|
||||
## Join Dragonfly as a member
|
||||
|
||||
It is also welcomed to join Dragonfly2 team if you are willing to participate in Dragonfly2 community continuously and keep active.
|
||||
It is also welcomed to join Dragonfly team if you are willing to participate in Dragonfly community continuously and keep active.
|
||||
|
||||
### Requirements
|
||||
|
||||
* Have read the [Contributing to Dragonfly2](CONTRIBUTING.md) carefully
|
||||
* Have read the [Contributing to Dragonfly](CONTRIBUTING.md) carefully
|
||||
* Have read the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md)
|
||||
* Have submitted multi PRs to the community
|
||||
* Be active in the community, may including but not limited
|
||||
|
|
|
|||
136
README.md
|
|
@ -1,89 +1,73 @@
|
|||
# Dragonfly2
|
||||
## 产品定位
|
||||
提供企业级(高效、稳定、安全、低成本、产品化)的文件分发与管理服务,并成为在该领域的云原生最佳实践与标准解决方案
|
||||
## 解决的问题
|
||||
1. 架构设计缺陷:现有架构很难满足日益增长的文件分发业务需要,在稳定性、效率以及安全性等方面逐渐暴露出不足,并受到越来越多的挑战
|
||||
2. 价值渗透不足:当前只能支持HTTP回源协议,针对其他类型的存储(比如HDFS、各云厂商的存储服务、Maven、YUM等等)缺乏适配,从而导致覆盖的场景受到极大制约,进一步阻碍了其在公司和社区的推广以及价值传播
|
||||
3. 分发模式单一:目前只支持主动拉取模式,缺少主动推送以及主动同步能力,而这些能力是满足我们产品定位的基本条件
|
||||
4. 产品化能力缺失:没有提供完善的控制台功能,比如分发任务管理与管控、数据大盘、多租户以及权限控制等等
|
||||
5. 内外版本不一致,维护成本高,新特性及问题修复难以短时间内同步
|
||||
# Dragonfly
|
||||
|
||||
## 核心能力
|
||||
+ 通过统一的回源适配层及实现支持各种不同类型存储(HDFS、各云厂商的存储服务、Maven、YUM等等)的P2P文件分发能力
|
||||
+ 支持更多的分发模式:主动拉取、主动推送、实时同步、远程复制、自动预热、跨云传输等等
|
||||
+ 系统之间分离解耦,调度与CDN插件化,且支持按需部署,可轻可重,可内可外,灵活满足不同场景的实际需要
|
||||
+ 基于grpc全新设计的P2P协议框架,效率及稳定性更佳
|
||||
+ 支持加密传输、基于账号的传输鉴权与限流、多租户隔离机制
|
||||
+ 支持更高效的IO方式:多线程IO、内存映射、DMA等
|
||||
+ 支持动态压缩、内存文件系统以及更高效的调度算法,提升分发效率
|
||||
+ 客户端通过C/S模式支持第三方软件原生集成蜻蜓的P2P能力
|
||||
+ 产品化能力:支持文件上传、各种分发模式的任务管理、数据视图以及全局管控等功能
|
||||
+ 内外一套,核心特性相互共享,非通用特性各自扩展
|
||||
+ 与生态进一步集成:Harbor、Nydus(镜像按需下载解决方案)、各云厂商仓库服务等
|
||||
## 产品架构
|
||||

|
||||
## 技术架构
|
||||

|
||||
## 各子系统主要逻辑单元
|
||||
Manager
|
||||
<img src="docs/en/images/logo/dragonfly-linear.svg" alt="logo" width="400"/>
|
||||
|
||||
+ 动态配置管理
|
||||
+ 数据大盘&带宽计量
|
||||
+ 文件上传&分发管理与管控
|
||||
+ 文件同步管理
|
||||
+ 账号&权限控制
|
||||
+ 订阅&通知
|
||||
+ 多租户管理
|
||||
+ 命令通道服务(通道合一)
|
||||
[](https://github.com/dragonflyoss/Dragonfly2/releases)
|
||||
[](https://github.com/dragonflyoss/Dragonfly2/actions/workflows/ci.yaml)
|
||||
[](https://codecov.io/gh/dragonflyoss/Dragonfly2)
|
||||
[](https://goreportcard.com/report/github.com/dragonflyoss/Dragonfly2)
|
||||
[](https://www.codetriage.com/dragonflyoss/dragonfly2)
|
||||
[](https://www.tickgit.com/browse?repo=github.com/dragonflyoss/Dragonfly2&branch=main)
|
||||
[](https://github.com/dragonflyoss/Dragonfly2/discussions)
|
||||
[](https://twitter.com/dragonfly_oss)
|
||||
[](https://godoc.org/github.com/dragonflyoss/Dragonfly2)
|
||||
[](https://github.com/dragonflyoss/Dragonfly2/blob/main/LICENSE)
|
||||
|
||||
Scheduler
|
||||
Provide efficient, stable, secure, low-cost file and image distribution services to be the best practice and standard solution in the related Cloud-Native area.
|
||||
|
||||
+ 基于多维度的P2P调度策略
|
||||
+ 智能化调度算法(高级)
|
||||
+ 调度结果可量化对比以及支持A/B Testing
|
||||
+ 调度策略插件化以及CDN子系统插件化
|
||||
+ 元信息下发管道
|
||||
+ 客户端下载结果和回源结果统计以及与监控服务的对接
|
||||
## Features
|
||||
|
||||
CdnSystem
|
||||
- Implement P2P files distribution with various storage types (HDFS, storage services from various cloud vendors, Maven, Yum, etc.) through a unified back-to-source adapter layer.
|
||||
- Support more distribution modes: active pull, active push, real-time synchronization, remote replication, automatic warm-up, cross-cloud transmission, etc.
|
||||
- Provide separation and decoupling between systems, scheduling and plug-in CDN. Support on-demand deployment with flexible types: light or heavy, inside or outside, to meet the actual needs of different scenarios.
|
||||
- Newly designed P2P protocol framework based on GRPC with improved efficiency and stability.
|
||||
- Perform encrypted transmission, account-based transmission authentication and rate limit, and multi-tenant isolation mechanism.
|
||||
- Bear more efficient IO methods: multithreaded IO, memory mapping, DMA, etc.
|
||||
- Advocate dynamic compression, in-memory file systems, and more efficient scheduling algorithms to improve distribution efficiency.
|
||||
- Client allows third-party software to natively integrate Dragonfly’s P2P capabilities through C/S mode.
|
||||
- Productivity: Support file uploading, task management of various distribution modes, data visualization, global control, etc.
|
||||
- Consistent internal and external versions, shared core features, and individual extensions of non-generic features.
|
||||
- Enhanced integration with ecology: Harbor, Nydus (on-demand image download), warehouse services for various cloud vendors, etc.
|
||||
|
||||
+ 多源适配能力,包括目录下载和Range下载功能
|
||||
+ DMA读写
|
||||
+ 回源中转
|
||||
+ 分发预热
|
||||
+ 动态压缩
|
||||
+ 存储管理(种子存放、磁盘清理、块级存储)
|
||||
+ 内存文件系统
|
||||
+ 安全传输(对称加密、完整性校验等)
|
||||
+ 存储介质插件化
|
||||
## Architecture
|
||||
|
||||
Client (dfdaemon、dfget)
|
||||
<img src="docs/en/images/architecture.png" alt="logo" width="800"/>
|
||||
|
||||
+ 多源适配能力,包括目录下载和Range下载功能(和CDN一致)
|
||||
+ 高效IO机制
|
||||
+ 多下载任务的IO调度(高IO吞吐量)
|
||||
+ 减少文件读写次数:改进临时文件存放方式、文件完整性校验改进
|
||||
+ DMA读写(和CDN一致)
|
||||
+ 流式传输
|
||||
+ 客户端下载Proxy能力(http、https)
|
||||
+ 单服务进程启动(CS模式)以及常驻和非常驻模式
|
||||
+ 失败回源处理
|
||||
+ 本地缓存能力(客户端seeder模式)
|
||||
+ 客户端优雅升级与部署方案
|
||||
+ 命令执行
|
||||
**Manager:** Used to manage the dynamic configuration that each module depends on, and provide keepalive and metrics functions.
|
||||
|
||||
Framework
|
||||
**Scheduler:** The tracker and scheduler in the P2P network that choose appropriate downloading net-path for each peer.
|
||||
|
||||
+ 高可用、高性能、易集成
|
||||
+ 一致性Hash算法选主
|
||||
+ 客户端连接管理
|
||||
**CDN:** A CDN server that caches downloaded data from source to avoid downloading same files repeatedly.
|
||||
|
||||
## 最近半年任务计划
|
||||
**Daemon:** Used for pulling images only. It establishes a proxy between containerd/CRI-O and registry.
|
||||
|
||||
+ 一月:基本框架开发完成,主流程跑通
|
||||
+ 二月:核心能力初步具备,同时支持客户端Proxy功能以及流式传输功能,Manager和CDN支持可插拔以及多源适配能力;release v2.0.0 alpha 版本
|
||||
+ 三月:进一步改进性能和稳定性并引入相关高级特性支持,生产环境灰度上线;和Nydus、主流镜像仓库服务完成集成;release v2.0.0 beta版本
|
||||
+ 四月:支持数据大盘以及带宽计量等数据化功能;release v2.0.0 正式版本
|
||||
+ 五月:上海外滩大会PR;支持订阅通知、文件主动推送、远程拷贝、跨云传输等产品化能力
|
||||
+ 六月:支持命令通道,完成通道合一;支持伪实时同步及多语言SDK支持;集团全量上线,完成内部版本和社区版本的最终合一
|
||||
**Dfget:** The client of Dragonfly used for downloading files. It's similar to wget.
|
||||
|
||||
## Getting Started
|
||||
|
||||
- [Introduction][introduction]
|
||||
- [Installation][installation]
|
||||
- [Quick start][quickstart]
|
||||
|
||||
## Documentation
|
||||
You can find the Dragonfly documentation [on the website][website].
|
||||
|
||||
## Community
|
||||
Welcome developers to actively participate in community discussions and contribute code to Dragonfly. We will remain concerned about the issues discussed in the community and respond quickly.
|
||||
|
||||
- Discussions: [Github Discussion Forum][discussion]
|
||||
- DingTalk: 23304666
|
||||
|
||||
## Contributing
|
||||
You should check out our [CONTRIBUTING][contributing] and develop the project together.
|
||||
|
||||
## Code of Conduct
|
||||
Please refer to our [Code of Conduct][codeconduct].
|
||||
|
||||
[contributing]: CONTRIBUTING.md
|
||||
[codeconduct]: CODE_OF_CONDUCT.md
|
||||
[introduction]: https://github.com/dragonflyoss/Dragonfly2
|
||||
[installation]: https://github.com/dragonflyoss/Dragonfly2
|
||||
[quickstart]: https://github.com/dragonflyoss/Dragonfly2
|
||||
[website]: https://d7y.io
|
||||
[discussion]: https://github.com/dragonflyoss/Dragonfly2/discussions
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ plugins:
|
|||
- name: memory
|
||||
enable: true
|
||||
config:
|
||||
baseDir: /tmp/memory/dragonfly2
|
||||
baseDir: /tmp/memory/dragonfly
|
||||
gcConfig:
|
||||
youngGCThreshold: 100G
|
||||
fullGCThreshold: 5G
|
||||
|
|
|
|||
|
|
@ -0,0 +1,2 @@
|
|||
# Contributing to Dragonfly Docs
|
||||
|
||||
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 37 KiB |
|
After Width: | Height: | Size: 5.6 KiB |
|
After Width: | Height: | Size: 22 KiB |
|
After Width: | Height: | Size: 5.5 KiB |
|
After Width: | Height: | Size: 425 KiB |
|
Before Width: | Height: | Size: 167 KiB |
|
Before Width: | Height: | Size: 169 KiB |