dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,448 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

2448 Commits

Author SHA1 Message Date
Gaius 7306aec508
feat: correct grpc error code and implement StatPeer and LeavePeer (#2115) 
Correct grpc error from codes.Unknow to codes.Internal in gorm
operation. Implement StatPeer and LeavePeer interface in v2 version of
grpc.

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:34 +08:00
Gaius 1b9353ba8d
feat: add SyncNetworkTopology and SyncProbes to scheduler client (#2114) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 31011b3021
refactor: preheat job (#2113) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 81785231e9
feat: add CIDR affinity to searcher (#2111) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 2f4cc5d24f
feat: remove Scopes and SecurityGroup in seed peer cluster (#2110) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius f9759ca926
feat: dynconfig resolves addresses with host (#2109) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
hiigao eebca2416a
feat: enable oss client download object concurrently. (#2105) 
feat: enalbe councrrent oss backsource by set contentLength

pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.

Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.

So let's set response a ContentLength value to enable concurrent
downloads from oss.

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 1d1aec5e0d
refactor: support reload scheduler addresses for local Dynconfig in client (#2107) 
feat: support reload scheduler addresses for local Dynconfig in client

Add OnNotify to client dynconfig, it will listen changes of the
local configuration.

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Eryu Guan c10b6da498
feat: support reload scheduler addresses for local Dynconfig (#2091) 
Dynconfig only supports refresh new configs from manager, but local
dynconfig is unable to reload scheduler addresses when local config file
is updated.

Now we introduce new Dynconfig method 'SetConfig()', which sets the
given DaemonOption to struct dynconfigLocal, and register it as a
watcher of WatchConfig(), which will reload config file periodically. So
that dynconfigLocal.GetResolveSchedulerAddrs() will get new scheduler
addresses.

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:33 +08:00
cuidajun e8822c144b
test: remove random test in pieceDispatcherTest (#2106) 
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
 2023-06-28 17:34:33 +08:00
Gaius 58034e4b1f
refactor: scheduling with v2 grpc (#2104) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:32 +08:00
hiigao 0349f3cf8f
feat: oss client supports STS access (set security token in header) (#2103) 
feat: oss client supports STS access

Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).

Fixes: #2049

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:32 +08:00
Eryu Guan fab5609d3d
feat: don't GC task if expire time is 0 (#2102) 
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:32 +08:00
Jim Ma d7a7e69916
chore: remove unused MarkInvalid in daemon (#2101) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:32 +08:00
Eryu Guan 3352deff39
fix: unregister task from scheduler in storage.deleteTask (#2100) 
On graceful shutdown we will call forceGC() to delete all cached tasks
if 'keepStorage' is false. And we rely on deleteTask() to delete the
given task from both local storage and scheduler.

But commit 86a6030c8f ("feat: unregister failed task storage (#1717)")
changed MarkReclaim() to MarkInvalid() in deleteTask(), and resulted in
task leak in scheduler.

Fix it by calling MarkReclaim() in deleteTask().

Fixes: 86a6030c8f ("feat: unregister failed task storage (#1717)")

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] f8098799c7
chore(deps): bump github.com/bits-and-blooms/bitset from 1.4.0 to 1.5.0 (#2097) 
Bumps [github.com/bits-and-blooms/bitset](https://github.com/bits-and-blooms/bitset) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/bits-and-blooms/bitset/releases)
- [Commits](https://github.com/bits-and-blooms/bitset/compare/v1.4.0...v1.5.0)

---
updated-dependencies:
- dependency-name: github.com/bits-and-blooms/bitset
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] d473fa770c
chore(deps): bump gorm.io/driver/mysql from 1.4.5 to 1.4.7 (#2096) 
Bumps [gorm.io/driver/mysql](https://github.com/go-gorm/mysql) from 1.4.5 to 1.4.7.
- [Release notes](https://github.com/go-gorm/mysql/releases)
- [Commits](https://github.com/go-gorm/mysql/compare/v1.4.5...v1.4.7)

---
updated-dependencies:
- dependency-name: gorm.io/driver/mysql
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] 2f758ab204
chore(deps): bump golang.org/x/oauth2 from 0.4.0 to 0.5.0 (#2094) 
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] 6f4bd07418
chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger from 1.12.0 to 1.13.0 (#2093) 
chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger

Bumps [go.opentelemetry.io/otel/exporters/jaeger](https://github.com/open-telemetry/opentelemetry-go) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/exporters/jaeger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
Changwei Ge 15a505f4f5
feat: avoid checking dir existence before MkdirAll (#2090) 
feat: avoid checking dir existance before MkdirAll

So we can save a syscall stat(2) thus to reduce a litte
overhead.
Besides, Close() can only fail if it is ever called.
We can safely simplifying the code by `defer Close()`

Signed-off-by: Changwei Ge <gechangwei@bytedance.com>
 2023-06-28 17:34:32 +08:00
Gaius da8c45f2a8
feat: add host ttl to scheduler (#2089) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius dacf1f01a6
feat: rename scheduler package to scheduling (#2087) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 010396a0f8
feat: use v2 version of host id and add Addrs func to seed peer (#2086) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 07976961c6
refactor: package digest (#2085) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 59856bda21
refactor: type of digest in task (#2084) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 042bb64ce4
refactor: task.SizeScope with v2 grpc in scheduler (#2082) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Jim Ma cc42334e4d
fix: backsource first piece timeout (#2083) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:31 +08:00
dlut_xz 91c9d83b6a
feat: add networkTopology configuration to scheduler (#2070) 
Signed-off-by: XZ <834756128@qq.com>
 2023-06-28 17:34:31 +08:00
Gaius 286885733a
feat: remove training configuration in scheduler (#2081) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 840c599d04
refactor: task piece with v2 grpc (#2080) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius d8f36a44be
feat: change piece size to length (#2079) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] d30c758411
chore(deps): bump github.com/go-playground/validator/v10 from 10.11.1 to 10.11.2 (#2077) 
chore(deps): bump github.com/go-playground/validator/v10

Bumps [github.com/go-playground/validator/v10](https://github.com/go-playground/validator) from 10.11.1 to 10.11.2.
- [Release notes](https://github.com/go-playground/validator/releases)
- [Commits](https://github.com/go-playground/validator/compare/v10.11.1...v10.11.2)

---
updated-dependencies:
- dependency-name: github.com/go-playground/validator/v10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] 9eebcaa0be
chore(deps): bump github.com/casbin/casbin/v2 from 2.60.0 to 2.61.1 (#2075) 
Bumps [github.com/casbin/casbin/v2](https://github.com/casbin/casbin) from 2.60.0 to 2.61.1.
- [Release notes](https://github.com/casbin/casbin/releases)
- [Changelog](https://github.com/casbin/casbin/blob/master/.releaserc.json)
- [Commits](https://github.com/casbin/casbin/compare/v2.60.0...v2.61.1)

---
updated-dependencies:
- dependency-name: github.com/casbin/casbin/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] de25bab39c
chore(deps): bump go.opentelemetry.io/otel from 1.12.0 to 1.13.0 (#2074) 
Bumps [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] eda9d8e8af
chore(deps): bump github.com/looplab/fsm from 1.0.0 to 1.0.1 (#2073) 
Bumps [github.com/looplab/fsm](https://github.com/looplab/fsm) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/looplab/fsm/releases)
- [Commits](https://github.com/looplab/fsm/compare/v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/looplab/fsm
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
Gaius a5685582ce
refactor: resource task with v2 version of grpc (#2078) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius 1409e37ba3
feat: set gorm log level (#2063) 2023-06-28 17:34:29 +08:00
Gaius a8e05c1b90
chore: change e2e timeout (#2062) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 09713e547c
fix: peer GC clear all peers when peer's count large than PeerCountLimitForTask (#2061) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 951705da98
feat: change PeerCountLimitForTask to 1000 (#2059) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius f7a399b47a
feat: add v2 version of the idgen (#2056) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 2917d91841
chore: add miHoYo to ADOPTERS.md (#2054) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 0f5470b5eb
feat: update task type from v1 to v2 (#2053) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius e1dd1efca1
feat: add AnnouncePeers to task in resource (#2051) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 7dc3c826f2
feat: add v2 version of dfdaemon client (#2050) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 29e57d1129
feat: add DownloadTask to seed peer resource (#2048) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:28 +08:00
dependabot[bot] 4c1287afdf
chore(deps): bump google.golang.org/grpc from 1.52.0 to 1.52.3 (#2046) 
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.52.0 to 1.52.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.52.0...v1.52.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:28 +08:00
dependabot[bot] 3a474ebaa4
chore(deps): bump docker/build-push-action from 3 to 4 (#2047) 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 3 to 4.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:28 +08:00
dependabot[bot] 2d0fc02bdc
chore(deps): bump github.com/shirou/gopsutil/v3 from 3.22.12 to 3.23.1 (#2045) 
Bumps [github.com/shirou/gopsutil/v3](https://github.com/shirou/gopsutil) from 3.22.12 to 3.23.1.
- [Release notes](https://github.com/shirou/gopsutil/releases)
- [Commits](https://github.com/shirou/gopsutil/compare/v3.22.12...v3.23.1)

---
updated-dependencies:
- dependency-name: github.com/shirou/gopsutil/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:28 +08:00