Commit Graph

1241 Commits

Author SHA1 Message Date
dependabot[bot] ba789a20f2
chore(deps): bump k8s.io/component-base from 0.25.3 to 0.25.4 (#1847)
Bumps [k8s.io/component-base](https://github.com/kubernetes/component-base) from 0.25.3 to 0.25.4.
- [Release notes](https://github.com/kubernetes/component-base/releases)
- [Commits](https://github.com/kubernetes/component-base/compare/v0.25.3...v0.25.4)

---
updated-dependencies:
- dependency-name: k8s.io/component-base
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:11 +08:00
Gaius 2421ce88f4
feat: add announcer to dfdameon (#1852)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:11 +08:00
Gaius 43b9c39149
feat: when dfdaemon disable object storage, dynconfig can't fetch manager (#1845)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:11 +08:00
Gaius fc0eccaf74
feat: optimize manager log (#1846)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:11 +08:00
Gaius f194f0961f
feat: scheduler adds announce host handler (#1843)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:11 +08:00
Jim Ma 7e90f43a53
chore: update dst peer log (#1844)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:11 +08:00
Gaius bdf5cd54c5
feat: call all nodes in consistent hashing and reuse grpc connection (#1842)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:11 +08:00
Jim Ma 3fe342dabc
fix: recursive download always return none error (#1841)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:10 +08:00
Jim Ma 1c24c3fc8c
fix: expire header timezone (#1840)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:10 +08:00
dependabot[bot] 3582a10b91
chore(deps): bump github.com/onsi/gomega from 1.23.0 to 1.24.1 (#1832)
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.23.0 to 1.24.1.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.23.0...v1.24.1)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:10 +08:00
Jim Ma ea02bbefde
chore: update e2e test (#1839)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:10 +08:00
Jim Ma f6d8d5c6b2
chore: remove unused code (#1838)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:10 +08:00
dependabot[bot] 900835f277
chore(deps): bump github.com/onsi/ginkgo/v2 from 2.4.0 to 2.5.0 (#1836)
Bumps [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) from 2.4.0 to 2.5.0.
- [Release notes](https://github.com/onsi/ginkgo/releases)
- [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/ginkgo/compare/v2.4.0...v2.5.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/ginkgo/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:10 +08:00
dependabot[bot] f2bcc5c47a
chore(deps): bump github.com/mdlayher/vsock from 1.1.1 to 1.2.0 (#1834)
Bumps [github.com/mdlayher/vsock](https://github.com/mdlayher/vsock) from 1.1.1 to 1.2.0.
- [Release notes](https://github.com/mdlayher/vsock/releases)
- [Changelog](https://github.com/mdlayher/vsock/blob/main/CHANGELOG.md)
- [Commits](https://github.com/mdlayher/vsock/compare/v1.1.1...v1.2.0)

---
updated-dependencies:
- dependency-name: github.com/mdlayher/vsock
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:10 +08:00
Gaius ca9afc5b9d
feat: update concurrent-map version (#1837)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:10 +08:00
Gaius 02543c781d
feat: optimize scope size is error (#1831)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:10 +08:00
Gaius 0f8b75808a
feat: add timeout grpc and job (#1830)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:10 +08:00
Jim Ma fdd189e661
chore: enable cache list metadata e2e (#1829)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Gaius 42223d994f
feat: optimize peer log (#1828)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:09 +08:00
Jim Ma ba58047871
feat: optional save list metadata to p2p (#1822)
* feat: save list metadata to p2p
* chore: optimize rpc download worker logic
* feat: add expire header reclaim logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Jim Ma 5040af8b4f
feat: add s3 resource client and recursive e2e test (#1826)
* feat: add s3 resource client
* chore: add minio server and recursive e2e test
* chore: add download grpc test tool

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Gaius d2f98d1429
feat: optimize preheat log (#1827)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:09 +08:00
Gaius 5c25a325c6
feat: seed peer reuses traffic (#1825)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:09 +08:00
Jim Ma f581d10208
chore: daemon avoid alway open metadata files (#1823)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:09 +08:00
Gaius 3da67311ea
feat: optimize preheat (#1824)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:09 +08:00
Gaius b1c049db46
feat: returns an scheduling error if the peer state is not PeerStateRunning (#1821)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:09 +08:00
Gaius 073c84dd71
feat: optimize peer gc (#1819)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:08 +08:00
Gaius 83ec42cf25
feat: peer.UpdateAt needs to be updated during download process (#1818)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:08 +08:00
Jim Ma 9378ce410a
chore: close out of use client grpc conn (#1817)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:08 +08:00
Gaius 96910887b2
feat: statistical the traffic of reused peer (#1816)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:08 +08:00
Jim Ma 6fd389c27a
fix: otel goroutine leak (#1815)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:08 +08:00
dependabot[bot] b440be333e
chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger from 1.10.0 to 1.11.1 (#1813)
chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger

Bumps [go.opentelemetry.io/otel/exporters/jaeger](https://github.com/open-telemetry/opentelemetry-go) from 1.10.0 to 1.11.1.
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.10.0...v1.11.1)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/exporters/jaeger
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:08 +08:00
dependabot[bot] 7a486afa1b
chore(deps): bump github.com/shirou/gopsutil/v3 from 3.22.9 to 3.22.10 (#1812)
Bumps [github.com/shirou/gopsutil/v3](https://github.com/shirou/gopsutil) from 3.22.9 to 3.22.10.
- [Release notes](https://github.com/shirou/gopsutil/releases)
- [Commits](https://github.com/shirou/gopsutil/compare/v3.22.9...v3.22.10)

---
updated-dependencies:
- dependency-name: github.com/shirou/gopsutil/v3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:08 +08:00
dependabot[bot] 6fd1233dad
chore(deps): bump go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin from 0.36.1 to 0.36.4 (#1811)
chore(deps): bump go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin

Bumps [go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin](https://github.com/gin-gonic/gin) from 0.36.1 to 0.36.4.
- [Release notes](https://github.com/gin-gonic/gin/releases)
- [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md)
- [Commits](https://github.com/gin-gonic/gin/commits)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:08 +08:00
dependabot[bot] 656fdab7c7
chore(deps): bump github.com/gammazero/deque from 0.2.0 to 0.2.1 (#1810)
Bumps [github.com/gammazero/deque](https://github.com/gammazero/deque) from 0.2.0 to 0.2.1.
- [Release notes](https://github.com/gammazero/deque/releases)
- [Commits](https://github.com/gammazero/deque/compare/v0.2.0...v0.2.1)

---
updated-dependencies:
- dependency-name: github.com/gammazero/deque
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:08 +08:00
Gaius 4bd7d413f2
feat: add workHome and pluginDir to configuration (#1807)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:08 +08:00
Jim Ma 8749845aae
chore: make SendMsg in doRecursiveDownload safe (#1806)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:07 +08:00
Jim Ma 0e83c7f58f
feat: add otel trace in log (#1804)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:07 +08:00
Gaius 4ba03bf761
fix: gorm-adaptor pkg version (#1805)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:07 +08:00
Jim Ma 2a7e576778
chore: add list log in rpc download (#1802)
Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:07 +08:00
Gaius d1363f20e7
fix: leave host (#1803)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:07 +08:00
Gaius abf1c43a4e
feat: add leave host logger (#1801)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:07 +08:00
dependabot[bot] cbf63e6400
chore(deps): bump gorm.io/driver/mysql from 1.4.1 to 1.4.3 (#1799)
Bumps [gorm.io/driver/mysql](https://github.com/go-gorm/mysql) from 1.4.1 to 1.4.3.
- [Release notes](https://github.com/go-gorm/mysql/releases)
- [Commits](https://github.com/go-gorm/mysql/compare/v1.4.1...v1.4.3)

---
updated-dependencies:
- dependency-name: gorm.io/driver/mysql
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:07 +08:00
dependabot[bot] c9f1f1eb31
chore(deps): bump google.golang.org/api from 0.97.0 to 0.101.0 (#1800)
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.97.0 to 0.101.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.97.0...v0.101.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:07 +08:00
dependabot[bot] 2a6819c230
chore(deps): bump github.com/onsi/gomega from 1.22.1 to 1.23.0 (#1798)
Bumps [github.com/onsi/gomega](https://github.com/onsi/gomega) from 1.22.1 to 1.23.0.
- [Release notes](https://github.com/onsi/gomega/releases)
- [Changelog](https://github.com/onsi/gomega/blob/master/CHANGELOG.md)
- [Commits](https://github.com/onsi/gomega/compare/v1.22.1...v1.23.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/gomega
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:07 +08:00
dependabot[bot] 9a72705fe9
chore(deps): bump gorm.io/driver/postgres from 1.4.4 to 1.4.5 (#1797)
Bumps [gorm.io/driver/postgres](https://github.com/go-gorm/postgres) from 1.4.4 to 1.4.5.
- [Release notes](https://github.com/go-gorm/postgres/releases)
- [Commits](https://github.com/go-gorm/postgres/compare/v1.4.4...v1.4.5)

---
updated-dependencies:
- dependency-name: gorm.io/driver/postgres
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:07 +08:00
dependabot[bot] 8986243990
chore(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1 (#1796)
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.5.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-06-28 17:34:06 +08:00
Gaius 3587ba9537
feat: scheduler's record adds ParentUploadCount and ParentUploadFailedCount (#1795)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:06 +08:00
Jim Ma 876c2ae17e
feat: support split running tasks (#1794)
* feat: support split running tasks

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: go tests

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: add split runnging tasks e2e config

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: actions

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* fix: actions

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize get running tasks logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

* chore: optimize get running tasks logic

Signed-off-by: Jim Ma <majinjing3@gmail.com>

Signed-off-by: Jim Ma <majinjing3@gmail.com>
2023-06-28 17:34:06 +08:00
Gaius 0c1cf2619d
feat: add download header log (#1793)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2023-06-28 17:34:06 +08:00