Commit Graph

150 Commits

Author SHA1 Message Date
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165)
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:33:06 +08:00
Gaius 603db21e17
feat: scheduler add default biz tag (#1164)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:06 +08:00
Gaius 7adee4c2d9
feat: host remove peer (#1161)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:05 +08:00
Gaius 4e670035ed
feat: change reschdule config (#1158)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:05 +08:00
Gaius 4ab498837c
test: add scheduler constructSuccessPeerPacket case (#1154)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:04 +08:00
Gaius 1ea700f421
feat: scheduler metrics add default value of biz tag (#1151)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:04 +08:00
Gaius b61865e2a9
feat: scheduler trace trigger cdn (#1147)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:04 +08:00
Gaius eb408d136f
test: scheduler service handlePieceFail (#1146)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:04 +08:00
Gaius 7d30dd86ae
feat: add scheduler traffic metrics (#1143)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:03 +08:00
Gaius 5223cc1ce8
feat: add scheduler metrics (#1139)
* feat: add scheduler metrics

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate validate for url meta

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:03 +08:00
Gaius 8f19c11f54
feat: scheduler remove inactive host (#1135)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:03 +08:00
Gaius addcf04df0
feat: task state for register (#1132)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:02 +08:00
Gaius 58aff0c231
feat: scheduler change piece cost from nanosecond to millisecond (#1119)
* feat: change picec cost from nanoseconds to milliseconds

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: scheduler service

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:02 +08:00
Gaius e940af77ca
fix: scheduler piece cost time (#1118)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:01 +08:00
Gaius 8ba4a12fcd
feat: when peer downloads finished, peer deletes parent (#1116)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:01 +08:00
Gaius c5428e0657
feat: optimize scheduler log (#1114)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:01 +08:00
Gaius 0326f781f4
feat: optimize scheduler (#1106)
* feat: optimize scheduler

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:00 +08:00
Gaius e2248b100c
feat: optimize depth limit func (#1102)
* feat: optimize depth limit func (#1099)

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:00 +08:00
Gaius 27fcc904c0
fix: when peer state is PeerStateSucceeded, return size scope is small (#1103)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:00 +08:00
Gaius 83cdf39a9c
feat: limit tree depth (#1099)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:33:00 +08:00
Gaius 557425249c
feat: optimize peer range (#1095)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:59 +08:00
Gaius db93339f9d
test: FilterParentCount (#1094)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:59 +08:00
Gaius 8bdfeab60b
feat: add cdn addresses log (#1091)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:59 +08:00
Gaius e6e9d2102a
feat: scheduler add limit count of filter parent func (#1090)
* feat: scheduler add filter parent count

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: limit filter parent count

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:59 +08:00
Gaius 3c45cdd6ac
feat: add dynamic parallel count (#1088)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:58 +08:00
Eryu Guan f56de57c01
fix: delete peer's parent on PeerEventDownloadSucceeded event (#1085)
When a peer leaves, it calls scheduler's LeaveTask() interface to
notify scheduler that it's leaving. And LeaveTask() will re-schedule
all its children to find a new parent by calling ScheduleParent().

But some of this peer's children might be in PeerStateSucceeded
state, as they have finished downloading tasks from parent. So
ScheduleParent() on such children is pointless.

Let's delete peer's parent on PeerEventDownloadSucceeded event, so
its parent won't call ScheduleParent() on it again, when its parent
is leaving.

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-06-28 17:32:58 +08:00
Gaius 81ea759393
test: scheduler handle failed piece (#1084)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:58 +08:00
Gaius e0a494e8a8
feat: when scheduler blocks cdn, resource does not initialize cdn (#1081)
* feat: when scheduler blocks cdn, resource does not initialize cdn

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: scheduler remove manager enable config

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:58 +08:00
Gaius 060429c675
feat: scheduler blocks cdn (#1079)
* feat: scheduler blocks cdn

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:57 +08:00
Gaius d52782ff38
feat: job trigger cdn by resource (#1076)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:57 +08:00
sunwp f15a04ec73
correct plugin typo (#1060)
Signed-off-by: sunwp <244372610@qq.com>
2023-06-28 17:32:57 +08:00
Gaius 60b4eeb9e6
fix: when cdn download failed, scheduler should set cdn peer state PeerStateFailed (#1067)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:56 +08:00
Gaius e200373abb
fix: evaluate peer's parent (#1064)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:56 +08:00
Jim Ma 54a9ed79d8
Feature: support e2e feature gates (#1056)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:56 +08:00
Jim Ma c14d36ba6a
fix: scheduler download tiny file error (#1052)
* fix: scheduler download tiny file error

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix golang lint

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:55 +08:00
Gaius 6d58de0ef1
refactor: scheduler grpc server (#1046)
* refactor: scheduler grpc server

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: generate mock file

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: scheduler package

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:55 +08:00
Gaius aa5f078a14
feat: remove host and task when peer make tree (#1042)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:55 +08:00
Gaius 99da719ce1
feat: cdn download tiny file (#1040)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:55 +08:00
Gaius cb2f9298ce
fix: cdn trigger peer error (#1035)
* feat: If cdn only updates IP, set cdn peers state to PeerStateLeave

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:54 +08:00
Gaius 65ad608d58
feat: If cdn only updates IP, set cdn peers state to PeerStateLeave (#1038)
* feat: If cdn only updates IP, set cdn peers state to PeerStateLeave

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:54 +08:00
Gaius efcf65c286
fix: retrigger cdn panic (#1034)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:54 +08:00
Gaius 061c3c8752
feat: scheduler download tiny file with range header (#1024)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:53 +08:00
Gaius 61310b55d5
feat: when cdn peer is failed, peer should be back-to-source (#1005)
* feat: schdule peer with cdn failed

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: when cdn peer is failed, peer back-to-source

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: callback

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: change scheduler default config

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:53 +08:00
sunwp f63c705b98
Add begin seed piece hint task registered successfully (#997)
* cdn obtainSeeds add begin seed piece hint register successfully

Signed-off-by: sunwp <244372610@qq.com>

* feat: scheduler handle begin of piece

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: trigger cdn task

Signed-off-by: Gaius <gaius.qi@gmail.com>

Co-authored-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:53 +08:00
Gaius 43d769b0b4
fix: register task with size scope (#1003)
* feat: register task with size scope

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: fix peer manager data race

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:52 +08:00
Gaius 5359ac2e75
feat: set peer state to running when scope size is SizeScope_TINY (#1004)
* chore: sync docker-compose scheduler config (#1001)

Signed-off-by: Jim Ma <majinjing3@gmail.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: set peer state to running when scope size is SizeScope_TINY

Signed-off-by: Gaius <gaius.qi@gmail.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:52 +08:00
Gaius f8ebd597e2
feat: task mutex replace sync kmutex (#1000)
* feat: trace https proxy request (#996)

* feat: trace https proxy request

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon test

Signed-off-by: Jim Ma <majinjing3@gmail.com>
Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: remove sync kmutex and add task mutex

Signed-off-by: Gaius <gaius.qi@gmail.com>

Co-authored-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:32:52 +08:00
Gaius 5479b3a53e
feat: stream send error code (#986)
* feat: change task and peer ttl (#984)

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: send error code

Signed-off-by: Gaius <gaius.qi@gmail.com>

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add retry interval

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfdaemon report successful piece before end of piece

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: update submodule version

Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:51 +08:00
Gaius 02913a5442
feat: add scheduler host gc (#989)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:51 +08:00
Gaius 54d6542aa6
feat: change task and peer ttl (#984)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:32:50 +08:00