dragonflyoss
/
dragonfly
mirror of https://github.com/dragonflyoss/dragonfly.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,907 Commits 5 Branches 281 Tags 98 MiB
Commit Graph

1907 Commits

Author SHA1 Message Date
Guangwen Feng bb54e7b28a
Fix a minor typo (#2127) 
Signed-off-by: Guangwen Feng <fenggw-fnst@fujitsu.com>
 2023-06-28 17:34:35 +08:00
Gaius 6ae36555be
test: refactor client validate ut (#2126) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:34 +08:00
Gaius 670c892898
feat: implement v2 version of scheduler service (#2125) 
Implement StatTask, AnnounceHost and LeaveHost api in
scheduler service v2.

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:34 +08:00
dependabot[bot] 7f1ca00df8
chore(deps): bump github.com/casbin/casbin/v2 from 2.61.1 to 2.64.0 (#2123) 
Bumps [github.com/casbin/casbin/v2](https://github.com/casbin/casbin) from 2.61.1 to 2.64.0.
- [Release notes](https://github.com/casbin/casbin/releases)
- [Changelog](https://github.com/casbin/casbin/blob/master/.releaserc.json)
- [Commits](https://github.com/casbin/casbin/compare/v2.61.1...v2.64.0)

---
updated-dependencies:
- dependency-name: github.com/casbin/casbin/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:34 +08:00
dependabot[bot] 1993266c6f
chore(deps): bump github.com/aws/aws-sdk-go from 1.44.189 to 1.44.209 (#2122) 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.44.189 to 1.44.209.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.44.189...v1.44.209)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:34 +08:00
dependabot[bot] 1cb90baf42
chore(deps): bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from 0.37.0 to 0.39.0 (#2120) 
chore(deps): bump go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

Bumps [go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc](https://github.com/open-telemetry/opentelemetry-go-contrib) from 0.37.0 to 0.39.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go-contrib/compare/zpages/v0.37.0...zpages/v0.39.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:34 +08:00
Eryu Guan a9df345d5b
test: add unit tests for DaemonConfig.Validate() (#2119) 
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:34 +08:00
Eryu Guan bbeba99da5
fix: validate daemon gcInterval config (#2118) 
If gcInterval is set to 0s or negative value, dfdaemon will panic as

  panic: non-positive interval for NewTicker

  goroutine 132 [running]:
  time.NewTicker(0x0?)

Fix it by validating it as well.

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:34 +08:00
AdamQQQ 7633910441
feat: update golang version to 1.20.1 (#2117) 
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
 2023-06-28 17:34:34 +08:00
Gaius 7306aec508
feat: correct grpc error code and implement StatPeer and LeavePeer (#2115) 
Correct grpc error from codes.Unknow to codes.Internal in gorm
operation. Implement StatPeer and LeavePeer interface in v2 version of
grpc.

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:34 +08:00
Gaius 1b9353ba8d
feat: add SyncNetworkTopology and SyncProbes to scheduler client (#2114) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 31011b3021
refactor: preheat job (#2113) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 81785231e9
feat: add CIDR affinity to searcher (#2111) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 2f4cc5d24f
feat: remove Scopes and SecurityGroup in seed peer cluster (#2110) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Gaius f9759ca926
feat: dynconfig resolves addresses with host (#2109) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
hiigao eebca2416a
feat: enable oss client download object concurrently. (#2105) 
feat: enalbe councrrent oss backsource by set contentLength

pieceManager calls DownloadSource() to download source objects.
DownloadSource() firstly connect to source and get the response from
source. Then, DownloadSource() checks the "ContentLength" value of the
response. If it larger than 0, pieceManager will download the object by
multiple threads.

Unfortunately, oss client doesn't set ContentLength value, even though
ContentLength can be set by the oss.GetObjectResult.Response.Headers.

So let's set response a ContentLength value to enable concurrent
downloads from oss.

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:33 +08:00
Gaius 1d1aec5e0d
refactor: support reload scheduler addresses for local Dynconfig in client (#2107) 
feat: support reload scheduler addresses for local Dynconfig in client

Add OnNotify to client dynconfig, it will listen changes of the
local configuration.

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:33 +08:00
Eryu Guan c10b6da498
feat: support reload scheduler addresses for local Dynconfig (#2091) 
Dynconfig only supports refresh new configs from manager, but local
dynconfig is unable to reload scheduler addresses when local config file
is updated.

Now we introduce new Dynconfig method 'SetConfig()', which sets the
given DaemonOption to struct dynconfigLocal, and register it as a
watcher of WatchConfig(), which will reload config file periodically. So
that dynconfigLocal.GetResolveSchedulerAddrs() will get new scheduler
addresses.

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:33 +08:00
cuidajun e8822c144b
test: remove random test in pieceDispatcherTest (#2106) 
Signed-off-by: bigerous <cuidajun.cdj@alibaba-inc.com>
 2023-06-28 17:34:33 +08:00
Gaius 58034e4b1f
refactor: scheduling with v2 grpc (#2104) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:32 +08:00
hiigao 0349f3cf8f
feat: oss client supports STS access (set security token in header) (#2103) 
feat: oss client supports STS access

Set security token in header and parse securityToken to init oss.Client.
Codes are in ossSourceClient getClient(header source.Header).

Fixes: #2049

Signed-off-by: hiigao <hii_gao@hotmail.com>
 2023-06-28 17:34:32 +08:00
Eryu Guan fab5609d3d
feat: don't GC task if expire time is 0 (#2102) 
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:32 +08:00
Jim Ma d7a7e69916
chore: remove unused MarkInvalid in daemon (#2101) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:32 +08:00
Eryu Guan 3352deff39
fix: unregister task from scheduler in storage.deleteTask (#2100) 
On graceful shutdown we will call forceGC() to delete all cached tasks
if 'keepStorage' is false. And we rely on deleteTask() to delete the
given task from both local storage and scheduler.

But commit 86a6030c8f ("feat: unregister failed task storage (#1717)")
changed MarkReclaim() to MarkInvalid() in deleteTask(), and resulted in
task leak in scheduler.

Fix it by calling MarkReclaim() in deleteTask().

Fixes: 86a6030c8f ("feat: unregister failed task storage (#1717)")

Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] f8098799c7
chore(deps): bump github.com/bits-and-blooms/bitset from 1.4.0 to 1.5.0 (#2097) 
Bumps [github.com/bits-and-blooms/bitset](https://github.com/bits-and-blooms/bitset) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/bits-and-blooms/bitset/releases)
- [Commits](https://github.com/bits-and-blooms/bitset/compare/v1.4.0...v1.5.0)

---
updated-dependencies:
- dependency-name: github.com/bits-and-blooms/bitset
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] d473fa770c
chore(deps): bump gorm.io/driver/mysql from 1.4.5 to 1.4.7 (#2096) 
Bumps [gorm.io/driver/mysql](https://github.com/go-gorm/mysql) from 1.4.5 to 1.4.7.
- [Release notes](https://github.com/go-gorm/mysql/releases)
- [Commits](https://github.com/go-gorm/mysql/compare/v1.4.5...v1.4.7)

---
updated-dependencies:
- dependency-name: gorm.io/driver/mysql
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] 2f758ab204
chore(deps): bump golang.org/x/oauth2 from 0.4.0 to 0.5.0 (#2094) 
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.4.0 to 0.5.0.
- [Release notes](https://github.com/golang/oauth2/releases)
- [Commits](https://github.com/golang/oauth2/compare/v0.4.0...v0.5.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
dependabot[bot] 6f4bd07418
chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger from 1.12.0 to 1.13.0 (#2093) 
chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger

Bumps [go.opentelemetry.io/otel/exporters/jaeger](https://github.com/open-telemetry/opentelemetry-go) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/exporters/jaeger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:32 +08:00
Changwei Ge 15a505f4f5
feat: avoid checking dir existence before MkdirAll (#2090) 
feat: avoid checking dir existance before MkdirAll

So we can save a syscall stat(2) thus to reduce a litte
overhead.
Besides, Close() can only fail if it is ever called.
We can safely simplifying the code by `defer Close()`

Signed-off-by: Changwei Ge <gechangwei@bytedance.com>
 2023-06-28 17:34:32 +08:00
Gaius da8c45f2a8
feat: add host ttl to scheduler (#2089) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius dacf1f01a6
feat: rename scheduler package to scheduling (#2087) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 010396a0f8
feat: use v2 version of host id and add Addrs func to seed peer (#2086) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 07976961c6
refactor: package digest (#2085) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 59856bda21
refactor: type of digest in task (#2084) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 042bb64ce4
refactor: task.SizeScope with v2 grpc in scheduler (#2082) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Jim Ma cc42334e4d
fix: backsource first piece timeout (#2083) 
Signed-off-by: Jim Ma <majinjing3@gmail.com>
 2023-06-28 17:34:31 +08:00
dlut_xz 91c9d83b6a
feat: add networkTopology configuration to scheduler (#2070) 
Signed-off-by: XZ <834756128@qq.com>
 2023-06-28 17:34:31 +08:00
Gaius 286885733a
feat: remove training configuration in scheduler (#2081) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:31 +08:00
Gaius 840c599d04
refactor: task piece with v2 grpc (#2080) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius d8f36a44be
feat: change piece size to length (#2079) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] d30c758411
chore(deps): bump github.com/go-playground/validator/v10 from 10.11.1 to 10.11.2 (#2077) 
chore(deps): bump github.com/go-playground/validator/v10

Bumps [github.com/go-playground/validator/v10](https://github.com/go-playground/validator) from 10.11.1 to 10.11.2.
- [Release notes](https://github.com/go-playground/validator/releases)
- [Commits](https://github.com/go-playground/validator/compare/v10.11.1...v10.11.2)

---
updated-dependencies:
- dependency-name: github.com/go-playground/validator/v10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] 9eebcaa0be
chore(deps): bump github.com/casbin/casbin/v2 from 2.60.0 to 2.61.1 (#2075) 
Bumps [github.com/casbin/casbin/v2](https://github.com/casbin/casbin) from 2.60.0 to 2.61.1.
- [Release notes](https://github.com/casbin/casbin/releases)
- [Changelog](https://github.com/casbin/casbin/blob/master/.releaserc.json)
- [Commits](https://github.com/casbin/casbin/compare/v2.60.0...v2.61.1)

---
updated-dependencies:
- dependency-name: github.com/casbin/casbin/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] de25bab39c
chore(deps): bump go.opentelemetry.io/otel from 1.12.0 to 1.13.0 (#2074) 
Bumps [go.opentelemetry.io/otel](https://github.com/open-telemetry/opentelemetry-go) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
dependabot[bot] eda9d8e8af
chore(deps): bump github.com/looplab/fsm from 1.0.0 to 1.0.1 (#2073) 
Bumps [github.com/looplab/fsm](https://github.com/looplab/fsm) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/looplab/fsm/releases)
- [Commits](https://github.com/looplab/fsm/compare/v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/looplab/fsm
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-06-28 17:34:30 +08:00
Gaius a5685582ce
refactor: resource task with v2 version of grpc (#2078) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius 79024c87ca
refactor: parse http range (#2071) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:30 +08:00
Gaius 1409e37ba3
feat: set gorm log level (#2063) 2023-06-28 17:34:29 +08:00
Gaius a8e05c1b90
chore: change e2e timeout (#2062) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 09713e547c
fix: peer GC clear all peers when peer's count large than PeerCountLimitForTask (#2061) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00
Gaius 951705da98
feat: change PeerCountLimitForTask to 1000 (#2059) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2023-06-28 17:34:29 +08:00