dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
672 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

50 Commits

Author SHA1 Message Date
Gaius 7a5857d389
feat: change scheduler and cdn listen (#1205) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:10 +08:00
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165) 
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:06 +08:00
sunwp f3c1080dd9
Optimize CDN GC Debug log (#1163) 
* remove CDN gc Debug log

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:05 +08:00
sunwp 2a33ff6005
lower cdn fullGC threshould (#1144) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:03 +08:00
sunwp a9be3408e7
resolve forget unlock resource lack lock (#1138) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:03 +08:00
Jim Ma bb73b45b46
fix: wrong md5 sign in cdn (#1126) 
* fix: wrong md5 sign in cdn

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* Extract the pieces sorting rule to the GetProgress method

Signed-off-by: sunwp <244372610@qq.com>

* cal sign notice

Signed-off-by: sunwp <244372610@qq.com>

Co-authored-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:02 +08:00
sunwp 4a752a47da
fix concurrent piece map panic in cdn (#1121) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:02 +08:00
sunwp 17e3f933c4
get piece tasks log (#974) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:33:02 +08:00
sunwp 0b0dd27ff8
add CDN piece download cost (#966) 
Signed-off-by: sunwp <244372610@qq.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:01 +08:00
sunwp d0665728e0
fix multi task seeds lead to panic concurrent map writes (#1093) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:59 +08:00
sunwp 84bfe42b2e
cdn returns begin piece (#1045) 
Signed-off-by: sunwp <244372610@qq.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:56 +08:00
sunwp 25832aa3d2
cdn rpc server test (#1020) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:55 +08:00
sunwp 2c31f35ce2
add cdn get piece tasks result logs (#1019) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:54 +08:00
sunwp f63c705b98
Add begin seed piece hint task registered successfully (#997) 
* cdn obtainSeeds add begin seed piece hint register successfully

Signed-off-by: sunwp <244372610@qq.com>

* feat: scheduler handle begin of piece

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: trigger cdn task

Signed-off-by: Gaius <gaius.qi@gmail.com>

Co-authored-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:53 +08:00
sunwp c9b2c0acf8
fix: calculate piece MD5 sign when last piece download (#1006) 
* fix: calculate piece md5 sign when last piece download

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:52 +08:00
Gaius 97c1eb65e7
Refactor scheduler service (#958) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:50 +08:00
Gaius 9f75cabc34
feat: change metrics port to 8000 (#964) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:48 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945) 
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
sunwp a1ab42790f
Optimaze GC log (#923) 
* optimaze gc log

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:47 +08:00
sunwp 7965ed70fe
Set totalPieceCount when the file length is available (#930) 
* set totalPieceCount when the file length is available

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:47 +08:00
Gaius cb4202319e
feat: make idgen package public (#931) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:47 +08:00
sunwp 39787c5bf1
fix Multiple test downloads failed (#928) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:46 +08:00
sunwp 0babd6e519
CDN config (#920) 
* CDN config

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:46 +08:00
Jim Ma 705c5ed342
fix: plugin dir is empty (#922) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:46 +08:00
sunwp e252ef5880
Adjust CDN module package (#893) 
* adjust CDN code package structure

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:46 +08:00
sunwp afdf91976c
feat: update task access time (#909) 
* update task access time when task is downloading

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:45 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906) 
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2023-06-28 17:32:45 +08:00
Gaius c2c721bc78
refactor: dfpath pkg (#879) 
* feat: dfpath

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfpath

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: path

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: plugin

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:43 +08:00
sunwp dc50bd31d5
reslove scheduler obtain seeds cancal result in CDN task download cancel (#890) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:43 +08:00
Jim Ma ff3a542fde
chore: update UnknownSourceFileLen (#888) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:43 +08:00
sunwp 6866ecc3fe
task manager (#885) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:43 +08:00
sunwp 456e776bfd
correct metadata spell (#884) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:42 +08:00
sunwp 46105b0986
unify back source interface (#877) 
* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* metaData Spelling correction

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs compile

Signed-off-by: sunwp <244372610@qq.com>

* hdfs unit test

Signed-off-by: sunwp <244372610@qq.com>

* reset fqdn

Signed-off-by: sunwp <244372610@qq.com>

* new seed task with header

Signed-off-by: sunwp <244372610@qq.com>

* unify back source interface

Signed-off-by: sunwp <244372610@qq.com>

* add unit tests & rename CheckRespCode -> CheckResponseCode & rename GetLastModifiedMillis -> GetLastModified

Signed-off-by: sunwp <244372610@qq.com>

* add request test

Signed-off-by: sunwp <244372610@qq.com>

* golang lint

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:42 +08:00
Gaius f6ce817b72
feat: move dfnet to internal (#862) 
* feat: move dfnet to internal

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: rename dfutils to util

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:41 +08:00
Gaius 848fec57f6
feat: move syncmap pkg(#859) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:40 +08:00
sunwp 44730a2e46
unify piece size (#854) 
* unify piece size

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:40 +08:00
Gaius 1a13ba11a8
feat: log (#852) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:40 +08:00
sunwp c1d84fb56d
validate grpc model (#825) 
* base validate

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:40 +08:00
sunwp d79a2a6877
move RPC code definition to proto file (#829) 
* move rpc code to proto
* replace Cdn to CDN

Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:38 +08:00
Gaius 0cb14903b2
feat: scheduler and cdn report fqdn to manager (#818) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:37 +08:00
Gaius 61f9f83b28
fix: manager typo and cdn peer id (#809) 
* fix: manager typo and cdn peer id 

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:36 +08:00
Gaius 558782af00
feat: add idgen peer id (#800) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:35 +08:00
Gaius e837ae9bdf
chore: add lint errcheck and fix errcheck(#766) 
* feat: add lint errcheck and fix errcheck

Signed-off-by: Gaius <gaius.qi@gmail.com>

* replace assert to require

Signed-off-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>

Co-authored-by: 孙伟鹏 <weipeng.swp@alibaba-inc.com>
 2023-06-28 17:32:35 +08:00
Jim Ma e0142cda8b
fix: cdn AdvertiseIP not used (#782) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:34 +08:00
Jim Ma 3d63ab2ca3
chore: update golang import lint (#780) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:34 +08:00
Gaius 52cf43a31d
feat: clear hashcircler and maputils package (#768) 
* feat: clear hashcircler and maputils package

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:33 +08:00
Gaius 061e25c3e5
feat: preheat cluster (#731) 
* feat: preheat cluster

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:30 +08:00
Gaius 466912aa48
docs: scheduler config (#698) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:25 +08:00
Gaius b4e58e3c08
refactor: scheduler supervisor (#655) 
* refactor: scheduler supervisor

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:31:49 +08:00
sunwp 359d11367e
chore: rename cdnsystem to cdn (#626) 
Signed-off-by: santong <244372610@qq.com>
 2023-06-28 17:31:46 +08:00