Commit Graph

72 Commits

Author SHA1 Message Date
Jim Ma 276333fceb
fix: SyncPieceViaHTTPS not work (#2329)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:49 +08:00
Gaius 07976961c6
refactor: package digest (#2085)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:31 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:30 +08:00
cuidajun 304448009f
refactor: piece_dispatcher considering score of parent peer (#1978)
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:26 +08:00
Jim Ma 55d5308b19
fix: open end range in concurrent back source (#1764)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:03 +08:00
Jim Ma 2d46f6424a
feat: update source temporary error logic (#1739)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:01 +08:00
cuidajun c6119d6ce0
fix: backsource temporary error judgement (#1726)
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>

Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
2023-06-28 17:34:00 +08:00
zhaoshang fda6002aac
test: add test for daemon rpcserver (#1704)
Signed-off-by: zhaoshang <zhaoshangsjtu@linux.alibaba.com>
2023-06-28 17:33:58 +08:00
Jim Ma 73de99353d
feat: empty file e2e (#1687)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:56 +08:00
Gaius 198613a357
chore: add disable seed peer action (#1653)
* chore: add disable seed peer action
* fix: concurrent back source

Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:54 +08:00
Jim Ma 98fb1fc427
feat: add grpc mux transport (#1602)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:49 +08:00
Jim Ma 717ac99ef5
feat: daemon support auto issue certificate (#1586)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:48 +08:00
Jim Ma 25e929142f
chore: optimize source error log (#1553)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:44 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:41 +08:00
Jim Ma ddfdb24dbf
fix: auto switch to concurrent back source (#1507)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:40 +08:00
greenhandatsjtu acf4922866
feat: auto switch to concurrent back source based on download speed (#1494)
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
2023-06-28 17:33:40 +08:00
Jim Ma 59060ef277
fix: back source error (#1485)
Signed-off-by: Jim Ma <majinjing3@gmail.com>

Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:38 +08:00
Jim Ma cf7debf9c4
feat: back source error detail (#1437)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:34 +08:00
Jim Ma 5f1ccfe0a8
chore: exit when receive context done (#1432)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:33 +08:00
Jim Ma 776a0fef42
feat: concurrent multiple pieces back source (#1426)
* feat: concurrent multiple pieces back source
* chore: update http source client

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:33 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:31 +08:00
Gaius 30313a4caa
feat: client add create object storage (#1409)
* feat: add create object storage rest api

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:30 +08:00
Gaius 64f5e9357b
refactor: scheduler announce task (#1407)
* feat: scheduler announce normal task

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cid

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:30 +08:00
Gaius e66b567eed
refactor: digest package (#1403)
* refactor: digest package

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:29 +08:00
Jim Ma 1a8b293a66
fix: seed peer did not send done seed result and no content length send (#1316)
fix: seed peer did not send done seed result
fix: unknown length back source seed did not send content length to other peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:21 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292)
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Jim Ma c6bd71cc93
feat: support more digest like sha256 (#1304)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:19 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227)
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:13 +08:00
Jim Ma a413dc4bdd
chore: report client back source error (#1209)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:10 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137)
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:05 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053)
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:56 +08:00
Jim Ma c00f07c2de
Extract storage driver into peer tasks (#998)
* chore: extract storage instead load every time

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: gofmt

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:52 +08:00
Gaius 5479b3a53e
feat: stream send error code (#986)
* feat: change task and peer ttl (#984)

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: send error code

Signed-off-by: Gaius <gaius.qi@gmail.com>

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add retry interval

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfdaemon report successful piece before end of piece

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: update submodule version

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:51 +08:00
Jim Ma c87be98799
chore: parameterize tests in peer task (#994)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
Jim Ma 88278be464
fix: total pieces count not set cause digest invalid (#992)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
Jim Ma 6f76c74838
chore: clarify daemon interface (#991)
1. Rename FilePeerTask to FileTask.
2. Rename StreamPeerTask to StreamTask.
3. Extract pulling piece task logic to pieceTaskPoller.

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:51 +08:00
Jim Ma 17ff9c17be
fix: send piece result error not handled (#987)
* fix: send piece result error not handled
* chore: optimize daemon cancel logic
* chore: remove redundant log and ctx.Done check
* chore: handle piece download with 404 status

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:50 +08:00
Jim Ma cb36b79dd1
feat: when write last piece, generate digest (#982)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:50 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977)
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:49 +08:00
Jim Ma 8d6e4044a4
chore: optimize back source update digest logic (#950)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945)
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:48 +08:00
Jim Ma 029c4eea36
fix: total pieces not set when back source (#908)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:44 +08:00
Jim Ma ff3a542fde
chore: update UnknownSourceFileLen (#888)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:43 +08:00
Jim Ma 5b016d92b4
fix: small size task failed due to digest error (#886)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:43 +08:00
sunwp 456e776bfd
correct metadata spell (#884)
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
sunwp 46105b0986
unify back source interface (#877)
* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* metaData Spelling correction

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs unit test

Signed-off-by: sunwp <244372610@qq.com>

* reset fqdn

Signed-off-by: sunwp <244372610@qq.com>

* new seed task with header

Signed-off-by: sunwp <244372610@qq.com>

* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified

Signed-off-by: sunwp <244372610@qq.com>

* add request test

Signed-off-by: sunwp <244372610@qq.com>

* golang lint

Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:42 +08:00
Gaius 3086147df6
docs: keep alive (#868)
* docs: keep alive

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add transport log

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:41 +08:00
Jim Ma 540cd233a0
feat: support customize transport in daemon (#866)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:41 +08:00