dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
625 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

147 Commits

Author SHA1 Message Date
Jim Ma 98e95e693c
feat: implement bidirectional fetch pieces (#1165) 
* feat: implement grpc server side sync pieces

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:06 +08:00
Jim Ma 010887ab66
feat: optimize proxy performance (#1137) 
* feat: optimize back source trace and logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: fix keepalive performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize generate peer id and transport performance

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: io.Copy call superfluous sendfile, then increase network latency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:05 +08:00
Gaius 98e23e4be7
feat: update otel package version and fix otelgrpc goroutine leak (#1141) 
* feat: update otel version

Signed-off-by: Gaius <gaius.qi@gmail.com>

* fix: grpc.WithInsecure is deprecated

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:33:03 +08:00
Jim Ma 84eaf402c3
chore: fast back source when get pieces task failed (#1123) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:02 +08:00
Jim Ma ebc1aff720
feat: support health probe in daemon (#1120) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:01 +08:00
Jim Ma 977f095c49
chore: optimize reuse logic (#1110) 
* chore: optimize reuse logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:01 +08:00
Jim Ma ac4c4bb912
feat: reuse partial completed task (#1107) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:33:00 +08:00
Jim Ma fc2b1f8187
chore: init url meta in rpc server (#1098) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma 4597f74283
feat: merge ranged request storage into parent (#1078) 
* feat: merge ranged request storage into parent

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:59 +08:00
Jim Ma 5478414751
feat: add prefetch metric in client (#1068) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:58 +08:00
Gaius 76dbfcfc5c
feat: add client request log (#1069) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:57 +08:00
Jim Ma 6fc2bb4de2
chore: add content length for fast stream peer task (#1061) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
Jim Ma f80c75efdb
Feature: prefetch ranged requests (#1053) 
1. implement prefetch ranged requests
2. optimize exact http code in transport
3. simplify reuse peer task logic
4. reuse peer task for ranged request size error
5. fix data race for peer task storage

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
Jim Ma 75ce151d3f
chore: optimize https pass through (#1054) 
* chore: optimize https pass through

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: update var name

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:56 +08:00
sunwp 25832aa3d2
cdn rpc server test (#1020) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:55 +08:00
Jim Ma 5027f9c998
chore: use buildx to build docker images in e2e (#1018) 
* chore: build images concurrency

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:54 +08:00
Jim Ma 23021dea4b
chore: add piece task metrics in daemon (#1030) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:54 +08:00
Jim Ma 621e52e9f1
chore: update outdated log (#1028) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma 6e636b1125
chore: optimize metrics and trace in daemon (#1022) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma a0049b17ae
chore: register to scheduler after updated running tasks (#1016) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
Jim Ma e9f824e0a7
chore: optimize defer and test (#1010) 
* chore: optimize defer and test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: random test failed

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:53 +08:00
sunwp 537667cfaa
add cdn totalpieces (#1014) 
Signed-off-by: sunwp <244372610@qq.com>
 2023-06-28 17:32:52 +08:00
Jim Ma c00f07c2de
Extract storage driver into peer tasks (#998) 
* chore: extract storage instead load every time

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: test

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: gofmt

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:52 +08:00
Gaius 43d769b0b4
fix: register task with size scope (#1003) 
* feat: register task with size scope

Signed-off-by: Gaius <gaius.qi@gmail.com>

* test: fix peer manager data race

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:52 +08:00
Gaius 5479b3a53e
feat: stream send error code (#986) 
* feat: change task and peer ttl (#984)

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: send error code

Signed-off-by: Gaius <gaius.qi@gmail.com>

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: add retry interval

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: dfdaemon report successful piece before end of piece

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: update submodule version

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 0c03af1979
fix: when scheduler is not available, replace the scheduler client (#999) 
* fix: when scheduler is not available, just replace the scheduler client in single peer task

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma c3951e0cb3
feat: trace https proxy request (#996) 
* feat: trace https proxy request

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: daemon test

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma c87be98799
chore: parameterize tests in peer task (#994) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 88278be464
fix: total pieces count not set cause digest invalid (#992) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
Jim Ma 6f76c74838
chore: clarify daemon interface (#991) 
1. Rename FilePeerTask to FileTask.
2. Rename StreamPeerTask to StreamTask.
3. Extract pulling piece task logic to pieceTaskPoller.

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:51 +08:00
guoguodan 3b6966c585
feat: update typo in local_storage.go (#955) 2023-06-28 17:32:51 +08:00
Jim Ma 17ff9c17be
fix: send piece result error not handled (#987) 
* fix: send piece result error not handled
* chore: optimize daemon cancel logic
* chore: remove redundant log and ctx.Done check
* chore: handle piece download with 404 status

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:50 +08:00
Jim Ma cb36b79dd1
feat: when write last piece, generate digest (#982) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:50 +08:00
Jim Ma 37b332a366
test: dump goroutine in e2e (#980) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:50 +08:00
Jim Ma c38bba2612
feat: merge same tasks in daemon (#977) 
* feat: merge same tasks in daemon
* test: add tiny file test in daemon
* fix: when validate digest disabled, did not store metadata

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:49 +08:00
Jim Ma a54ae4fae4
feat: add default DiskGCThresholdPercent and ignore it when is 0 (#971) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:49 +08:00
Christian Bourgeois 0706fc959e
feat: improve redirect to allow url rewrite (#969) 
* feat: improve redirect to allow url rewrite

Signed-off-by: momiji <teq1uila-free01@yahoo.fr>
 2023-06-28 17:32:49 +08:00
Christian Bourgeois b4cc13db41
feat: Add useProxies to registryMirror allowing to mirror more anything (#965) 
* feat: add useProxies to registryMirror to allow mirroring anything

Signed-off-by: momiji <teq1uila-free01@yahoo.fr>
 2023-06-28 17:32:49 +08:00
Jim Ma 38178804e8
feat: add daemon metrics support (#960) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
Jim Ma b7f8ad4e20
feat: support disk usage gc in client (#953) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
Jim Ma 8d6e4044a4
chore: optimize back source update digest logic (#950) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
Jim Ma beaf4ce18d
feat: update source.Response and source client interface (#945) 
* feat: update source.Response and source client interface

Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:48 +08:00
Jim Ma ce2503f6c5
fix: not handle base.Code_SchedTaskStatusError in client (#938) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:47 +08:00
Jim Ma 7ba341fc5c
fix: infinitely get pieces when piece num is invalid (#926) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:47 +08:00
Gaius cb4202319e
feat: make idgen package public (#931) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:47 +08:00
Gaius 1cde964259
feat: make dfpath public (#929) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:46 +08:00
Jim Ma 705c5ed342
fix: plugin dir is empty (#922) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:46 +08:00
Eng Zer Jun f579d3c9ce
refactor: move from io/ioutil to io and os packages (#906) 
* chore: run `go fmt ./...`

This commit synchronizes `//go:build` lines with `// +build` lines.

Reference: https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* refactor: move from io/ioutil to io and os packages

The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2023-06-28 17:32:45 +08:00
Jim Ma 029c4eea36
fix: total pieces not set when back source (#908) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:32:44 +08:00
Gaius 75c7be0769
feat: add data storage directory (#907) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:32:44 +08:00