dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,653 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

379 Commits

Author SHA1 Message Date
Gaius 198613a357
chore: add disable seed peer action (#1653) 
* chore: add disable seed peer action
* fix: concurrent back source

Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:54 +08:00
Jim Ma 7bceca51bc
chore: update debug info (#1617) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:51 +08:00
Gaius b6c395a6cb
feat: add cert spec to security configuration (#1621) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:51 +08:00
Jim Ma 56ac9d1801
feat: support mutate all proxy requests (#1623) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:50 +08:00
Gaius 13580f89f2
refactor: dfpath for certify cache dir (#1618) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:50 +08:00
Gaius 43d8478507
feat: add tls policy to scheduler grpc server (#1616) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:50 +08:00
Jim Ma 9050230926
feat: refactor grpc credential (#1612) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:50 +08:00
Gaius 346cc0649e
feat: add tls policy constants (#1610) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:49 +08:00
Jim Ma 98fb1fc427
feat: add grpc mux transport (#1602) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:49 +08:00
Gaius 9ab33635c5
feat: manager init cert for grpc server (#1603) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:49 +08:00
Jim Ma 7224801bae
feat: refactor peertask option (#1600) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma ecfe467585
feat: add client grpc dial timeout (#1599) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma c25cfdd651
feat: support multiple certify cache (#1598) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Jim Ma 717ac99ef5
feat: daemon support auto issue certificate (#1586) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:48 +08:00
Gaius ad553d949b
feat: grpc dial adds context (#1594) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:47 +08:00
Gaius 499d2fb886
feat: remove golang +build tag (#1585) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:47 +08:00
Gaius 2c48148ac8
fix: dfdaemon can not shutdown (#1580) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:46 +08:00
Gaius f631263d25
refactor: dfnet package (#1578) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:46 +08:00
Gaius a6a44269b4
refactor: dfdaemon client and remove rpc connection pool (#1576) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:46 +08:00
Gaius 272f38f567
feat: grpc add ratelimit (#1572) 
feat: grpc ratelimit

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:46 +08:00
Gaius 69a51808d2
feat: refresh dynconfig addresses when grpc requests unavailable (#1571) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:46 +08:00
Gaius 631654f98b
feat: manager client add context (#1562) 
Signed-off-by: Gaius <gaius.qi@gmail.com>

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:45 +08:00
cuidajun 242fb2bc37
add application-level statistics (#1559) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:45 +08:00
Gaius 5bcb53f0a9
feat: grpc consistent hashing (#1554) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:44 +08:00
Jim Ma 25e929142f
chore: optimize source error log (#1553) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:44 +08:00
Gaius 2ce864fbf6
fix: scheduler and manager tracing (#1551) 
fix: scheduler and manager trace

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:44 +08:00
Jim Ma baa64bb0a3
feat: add seed trace (#1549) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:44 +08:00
Gaius 740696a171
feat: when the request has a range header, object storage is no need to to calculate md5 (#1534) 
feat: when the request has a range header, object storage is no need to calculate md5

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:42 +08:00
cuidajun 1258175366
fix daemon socket remove error (#1529) 2023-06-28 17:33:42 +08:00
Jim Ma f6506115a7
feat: support grpc recursive download (#1518) 
* feat: daemon grpc recursive download
* fix: init storage error

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:42 +08:00
Jim Ma 4984cdfbbc
fix: check same peer id for sync pieces (#1525) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:42 +08:00
Gaius e42186acb7
feat: add daemon-socket-path (#1521) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:42 +08:00
Gaius d79f6405ff
feat: replace grpc package with https://github.com/dragonflyoss/api (#1515) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
Co-authored-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:41 +08:00
Jim Ma ddfdb24dbf
fix: auto switch to concurrent back source (#1507) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:40 +08:00
greenhandatsjtu acf4922866
feat: auto switch to concurrent back source based on download speed (#1494) 
Signed-off-by: greenhandatsjtu <sunhengke@sjtu.edu.cn>
 2023-06-28 17:33:40 +08:00
Jim Ma 3b539f9abf
fix: wait first peer packet fail (#1500) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:39 +08:00
cuidajun de35b14db9
fix: one piece task sometimes backsource after succeed (#1499) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:39 +08:00
Gaius 0bd894adad
fix: dfdaemon unix socket (#1489) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:38 +08:00
Jim Ma c921c265c5
fix: init storage error (#1486) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:38 +08:00
Jim Ma 59060ef277
fix: back source error (#1485) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>

Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:38 +08:00
Gaius 7d3c9d539d
refactor: cache key for peer (#1483) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:38 +08:00
Gaius ed4c381383
feat: rename steal peers to candidate peers (#1476) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:37 +08:00
Gaius f9d2736ae8
feat: scheduler merge end of piece and piece from seed peer (#1474) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:37 +08:00
cuidajun abec6d3a84
fix: upload_manager write header in time (#1471) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:37 +08:00
Gaius 4d71217572
feat: generate dag mock 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:36 +08:00
cuidajun c6654cb4ae
fix: upload_manager write header immediately when it is ready (#1466) 
Signed-off-by: bigerous <bigerous@qq.com>
 2023-06-28 17:33:36 +08:00
1037husterljx 31032e05aa
fix [4000] digest not set error (#1465) 
Signed-off-by: ljx373327 <ljx373327@alibaba-inc.com>

Co-authored-by: ljx373327 <ljx373327@alibaba-inc.com>
 2023-06-28 17:33:36 +08:00
Gaius c6a019560d
feat: proxy add defaultTag field (#1462) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:36 +08:00
Gaius 601993aa46
feat: use os.PathSeparator to generate object key 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:35 +08:00
Gaius 812ff5557b
fix: metrics reduces labels (#1457) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:35 +08:00
Jim Ma 9f360c1b03
feat: reload proxy option (#1443) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:35 +08:00
Jim Ma 1cf39f3f9b
chore: check header length before update (#1445) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:35 +08:00
Gaius 8cf344992f
feat: if peer back-to-source failed, return source metadata. (#1444) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:34 +08:00
Jim Ma 3c107f9add
feat: report peer result with source error detail (#1439) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:34 +08:00
Gaius 93a8ebb4bc
feat: add dfstore command (#1441) 
* feat: add dfstore command

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:34 +08:00
Jim Ma cf7debf9c4
feat: back source error detail (#1437) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:34 +08:00
Jim Ma 5f1ccfe0a8
chore: exit when receive context done (#1432) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:33 +08:00
Jim Ma 776a0fef42
feat: concurrent multiple pieces back source (#1426) 
* feat: concurrent multiple pieces back source
* chore: update http source client

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:33 +08:00
Jim Ma baedf398e9
fix: seed task metric panic (#1427) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:33 +08:00
Gaius 567e8e978c
feat: use put object action (#1422) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 3b426d2c54
feat: rename client/clientutil to client/util (#1420) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius a2511cb945
feat: rewrite interface{} to any (#1419) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius 0222649b88
feat: update namely/protoc-all image version to 1.47_0 (#1418) 
* feat: update namely/protoc-all image version to 1.47_0

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:32 +08:00
Gaius ad36eb64a7
feat: remove github/pkg/errors package (#1416) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:31 +08:00
Gaius 1d7cd9dd12
feat: add dfstore client interface (#1415) 
* feat: add dfstore client interface

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:31 +08:00
Gaius ec2a72d872
feat: import object to seed peer with max replicas (#1413) 
* feat: create object storage with max replicas

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: import object storage with max replicas

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add strings.Unique func

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:31 +08:00
Gaius 4ad950a8e3
feat: object storage add filter field (#1412) 
* feat: object storage add filter field

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: object storage add logger

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: GetObjectMetadata return isExist

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:30 +08:00
Gaius 2504f28cd0
feat: dfdaemon add destroyObject rest api (#1410) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:30 +08:00
Gaius 30313a4caa
feat: client add create object storage (#1409) 
* feat: add create object storage rest api

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:30 +08:00
Gaius 64f5e9357b
refactor: scheduler announce task (#1407) 
* feat: scheduler announce normal task

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cid

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:30 +08:00
Gaius 15d3274224
feat: rename digest func and add new digest func (#1405) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius ad827170a4
feat: dfdaemon upload and object storage service add middlewares (#1404) 
* feat: dfdaemon upload and object storage service add middlewares

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius e66b567eed
refactor: digest package (#1403) 
* refactor: digest package

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius 7a30db3fba
refactor: pkg util (#1402) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius f688d341bd
feat: remove cdn (#1401) 
* feat: remove cdn

Signed-off-by: Gaius <gaius.qi@gmail.com>

* docs: generate swagger

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:29 +08:00
Gaius 230966cf8f
feat: dfdaemon add GetObject rest api (#1398) 
* feat: dfdaemon add GetObject rest api

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:28 +08:00
Gaius 171c0d54ee
feat: dfdaemon add object storage rest api (#1390) 
* feat: dfdaemon add object storage rest api

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:28 +08:00
Gaius 4889a37886
feat: replace gin-gonic/gin with gorilla/mux (#1389) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:28 +08:00
Gaius 2362a670bd
fix: preheat tack id (#1375) 
* fix: preheat with task id

Signed-off-by: Gaius <gaius.qi@gmail.com>

* fix: add end time to seed piece

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:27 +08:00
Jim Ma 6070e5a0ef
chore: update content range for partial content (#1357) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:25 +08:00
Jim Ma a3bc931850
fix: register fail panic (#1351) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:25 +08:00
Jim Ma 175a27e278
fix: find partial completed overflow (#1346) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:24 +08:00
Gaius 6fe519a233
fix: seed peer reuse value 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:23 +08:00
Gaius 40e98d6799
fix: dfdaemon seed peer metrics namespace (#1343) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:23 +08:00
Gaius 5763ee86bd
feat: add seed peer metrics (#1342) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:23 +08:00
Jim Ma ebee0adb7d
fix: reuse seed peer id is not exist (#1335) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:22 +08:00
Jim Ma d0ca987798
fix: reuse non-end range request (#1333) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:22 +08:00
Gaius 8ddd197367
test: watchdog 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:22 +08:00
Jim Ma 3c6af10433
feat: add sync piece watchdog (#1272) 
* feat: add sync piece watchdog

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:21 +08:00
Gaius 605865d3a9
feat: client add tls verify config (#1323) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:21 +08:00
Jim Ma 3a9f160e7a
fix: reuse seed panic (#1319) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:21 +08:00
Jim Ma 1a8b293a66
fix: seed peer did not send done seed result and no content length send (#1316) 
fix: seed peer did not send done seed result
fix: unknown length back source seed did not send content length to other peers

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:21 +08:00
Gaius 999f5919c6
feat: reduce dynconfig log (#1312) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:20 +08:00
Gaius 2a0314e45b
refactor: scheduler grpc (#1310) 
* refactor: scheduler grpc

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate protoc

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:20 +08:00
Jim Ma f3750b1ac0
feat: back source when receive seed request (#1309) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:20 +08:00
Jim Ma 823c722c3e
feat: support response header (#1292) 
* feat: support response header via ExtendAttribute
* chore: update http pass through header
* chore: update ExtendAttribute in cdn downloader

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Gaius faa5e4e465
feat: add seed peer logic (#1302) 
* feat: announce seed peer

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cdn logic

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cdn job

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfdaemon change host uuid to host id

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: go generate mocks

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove cdn compatibility

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: change docker compose

Signed-off-by: Gaius <gaius.qi@gmail.com>

* fix: reuse panic

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: compatible with v2.0.3-beta.2

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma c6bd71cc93
feat: support more digest like sha256 (#1304) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 429252a071
feat: implement pattern in client daemon (#1231) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:19 +08:00
Jim Ma 49f9ebde68
feat: implement client seed mode (#1247) 
* feat: implement client seed mode

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:18 +08:00
Jim Ma 474a8d0e5c
fix: keep accept header (#1291) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:18 +08:00
Jim Ma afdac331bb
chore: add target peer id in sync piece trace (#1278) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:16 +08:00
Jim Ma 3682b26647
feat: in tree plugin (#1276) 
* feat: in tree plugin

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:16 +08:00
Jim Ma 16759c0339
chore: optimize create synchronizer logic (#1269) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:16 +08:00
Jim Ma 4d3e4e703f
fix: acquire empty dst pid (#1268) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:15 +08:00
Jim Ma 41f766168b
chore: add sync pieces trace and update sync pieces logic for done task (#1263) 
* chore: add sync pieces trace

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: when task is done, sync piece tasks will still receive remote request

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: go context lint

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* Update client/daemon/rpcserver/rpcserver.go

Co-authored-by: cndoit18 <cndoit18@outlook.com>
Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: record close send error

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: clean PeerPacket when peer task completed

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize sync piece completed check

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix go lint

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: update sync piece test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

Co-authored-by: cndoit18 <cndoit18@outlook.com>
 2023-06-28 17:33:15 +08:00
Jim Ma 7c1e315ec1
fix: client synchronizer report error lock and dial grpc timeout (#1260) 
* fix: client synchronizer report error lock
* chore: remove superfluous retry to sync and get pieces from other peers
* chore: optimize back source and sync piece logic
* chore: print apk output in e2e
* fix: grpc send with io.EOF error

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:15 +08:00
Gaius 076ba20b51
feat: move dfnet to pkg dir (#1265) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:15 +08:00
Jim Ma 902fbb031b
chore: optimize sync pieces (#1253) 
* chore: optimize sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:14 +08:00
Eryu Guan 65fa8a1680
feat: define and implement new dfdaemon APIs to make dragonfly2 work as a distributed cache (#1227) 
Introduce a new dfcache command which uses dfcache SDK interface to
interactive with dfdaemon and operate on P2P cache system.

For example:
- add a file into cache system
  dfcache import --cid sha256:xxxxxx --tag testtag /path/to/file

- check if a file exists in cache system
  dfcache stat --cid testid --local  # only check local cache
  dfcache stat --cid testid          # check other peers as well

- export/download a file from cache system
  dfcache export --cid testid -O /path/to/output

- delete a file from cache system, both local cache and P2P network
  dfcache delete -i testid -t testtag

Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:13 +08:00
Jim Ma 517ed6c51e
fix: client sync piece panic (#1246) 
* fix: client sync piece panic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize sync piece log

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon panic when register fail

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:13 +08:00
Jim Ma 5efbaaed80
fix: client superfluous usage gc (#1243) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:13 +08:00
Jim Ma 41d5ee6beb
fix: client sync send unsafe call (#1240) 
* fix: client sync send unsafe call

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:13 +08:00
Jim Ma 1f79e76a4b
fix: client unexpected timeout (#1239) 
* fix: client unexpected timeout

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:13 +08:00
Gaius 3fdde9b891
fix: make generate (#1228) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:11 +08:00
Jim Ma 73c5e3a575
fix: sync pieces hang (#1221) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:11 +08:00
Jim Ma 76eff082cb
feat: add back source fail metric in client (#1214) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:10 +08:00
Jim Ma 60a1d9e53e
chore: print client stream task error log (#1210) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:10 +08:00
Jim Ma a413dc4bdd
chore: report client back source error (#1209) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:10 +08:00
Gaius 2bcdba6401
feat: add grpc health interface (#1195) 
* feat: add grpc health interface

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add dfdaemon upload server healthy interface

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:09 +08:00
Jim Ma d45bffea40
fix: client miss failed piece (#1194) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:09 +08:00
Gaius 1939ceff8f
refactor: scheduler end and begin of piece (#1189) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:09 +08:00
Jim Ma 9f4b86cb2f
fix: client break error (#1190) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:08 +08:00
Jim Ma 42046c0845
chore: optimize stream peer task (#1186) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:08 +08:00
sunwp bab1ddec60
Fix client initialization trivial things (#1176) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:08 +08:00
Gaius 04bd2ea383
feat: scheduler addresses log (#1183) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:08 +08:00
Jim Ma d3c36f26ce
chore: always fallback to legacy get pieces (#1180) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:08 +08:00
sunwp 104e8359a9
fix daemon cannot exit after alivetime (#1177) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:07 +08:00
Gaius 21d219175f
feat: dfdaemon get available scheduler addresses in the same cluster (#1178) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:07 +08:00
Jim Ma 10da5fa0d6
feat: implement grpc client side sync pieces (#1167) 
* feat: implement grpc client side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: client unit test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: report sync piece error

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize pieceTaskSyncManager cancel

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:07 +08:00
yxxhero 7446cf45fa
feat: replace time.Now().Sub by time.Since (#1173) 
Signed-off-by: yxxhero <aiopsclub@163.com>
 2023-06-28 17:33:07 +08:00
Jim Ma 42e662d8b3
fix: subscriber data race (#1169) 
* fix: subscriber data race

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:06 +08:00
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165) 
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:06 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137) 
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:05 +08:00
Gaius 98e23e4be7
feat: update otel package version and fix otelgrpc goroutine leak (#1141) 
* feat: update otel version

Signed-off-by: Gaius <gaius.qi@gmail.com>

* fix: grpc.WithInsecure is deprecated

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:03 +08:00
Jim Ma 84eaf402c3
chore: fast back source when get pieces task failed (#1123) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:02 +08:00
Jim Ma ebc1aff720
feat: support health probe in daemon (#1120) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:01 +08:00
Jim Ma 977f095c49
chore: optimize reuse logic (#1110) 
* chore: optimize reuse logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:01 +08:00
Jim Ma ac4c4bb912
feat: reuse partial completed task (#1107) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:00 +08:00
Jim Ma fc2b1f8187
chore: init url meta in rpc server (#1098) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078) 
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma 5478414751
feat: add prefetch metric in client (#1068) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:58 +08:00
Gaius 76dbfcfc5c
feat: add client request log (#1069) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:57 +08:00
Jim Ma 6fc2bb4de2
chore: add content length for fast stream peer task (#1061) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053) 
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
Jim Ma 75ce151d3f
chore: optimize https pass through (#1054) 
* chore: optimize https pass through

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: update var name

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
sunwp 25832aa3d2
cdn rpc server test (#1020) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:55 +08:00
Jim Ma 5027f9c998
chore: use buildx to build docker images in e2e (#1018) 
* chore: build images concurrency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:54 +08:00
Jim Ma 23021dea4b
chore: add piece task metrics in daemon (#1030) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:54 +08:00
Jim Ma 621e52e9f1
chore: update outdated log (#1028) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00