Commit Graph

130 Commits

Author SHA1 Message Date
Gaius b7dc472df9
refactor(quic): integrate QUIC and downloader (#1378)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-09-25 19:57:04 +08:00
LXDgit2018 74a6f04b8c
feat: Add QUIC Client and Server Implementation for Piece Download in dragonfly-client-storage (#1376)
Signed-off-by: LXDgit2018 <1289504283@qq.com>
2025-09-25 11:31:20 +08:00
Gaius 7cf2e3d800
feat(export): add overwrite option for output files (#1374)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-09-24 21:33:42 +08:00
Gaius c4fb049aac
feat(metric): add dragonfly-client-metric module (#1362)
* feat(metric): add dragonfly-client-metric module

Signed-off-by: Gaius <gaius.qi@gmail.com>

* chore(deps): update dragonfly-client versions to 1.0.22

Signed-off-by: Gaius <gaius.qi@gmail.com>

---------

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-09-22 10:23:09 +00:00
Gaius 160f51396f
refactor: tcp storage server and client (#1361)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-09-22 17:44:20 +08:00
Fan Tianlan 9ec926f824
feat: Add TCP Client and Server Implementation for Piece Download in dragonfly-client-storage (#1359)
Signed-off-by: ftljk <2510317228@qq.com>
2025-09-18 15:18:03 +08:00
Gaius 344bf3ed4f
refactor(cache): remove WriteCachePieceResponse struct (#1329)
* refactor(cache): remove WriteCachePieceResponse struct

Signed-off-by: Gaius <gaius.qi@gmail.com>

* style(lib): remove unused import from lib.rs

Signed-off-by: Gaius <gaius.qi@gmail.com>

---------

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-09-02 20:18:07 +08:00
this is my name 622ef4ec56
feat: Add support for CacheTask in the storage layer. (#1278)
This pull request modifies the Cache's size field type, adds storage layer support for CacheTask, and implements related processing for CacheTask metadata.

- Reconstruct the Cache structure, changing the size field to Arc<AtomicU64> to achieve internal mutability and state sharing.
- Created a new CacheTask metadata structure and implemented its associated processing functions.
- Added CacheTask support in the storage layer, implementing all necessary functions for its download functionality.

To provide storage layer support for CacheTask, laying the foundation for future CacheTask integration into the client.

Signed-off-by: fu220 <2863318196@qq.com>
2025-09-02 11:38:58 +08:00
Gaius 578fc8d147
feat: add conditional piece content reading based on need_piece_content flag (#1296)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-08-21 11:49:19 +08:00
this is my name 1f8a323665
docs:Modify Cache storage comments (#1277)
Modify Cache storage comments to clarify its usage scenarios.

Signed-off-by: fu220 <2863318196@qq.com>
2025-08-06 17:04:20 +08:00
this is my name 848737e327
feat:removes the load_to_cache field (#1264)
This pull request removes the load_to_cache field and adds trait method definitions to dfdaemon_download.rs and dfdaemon_upload.rs.

- Removed cache-related processing from Task handling.
- Added trait method definitions such as download_cache_task to dfdaemon_download.rs and dfdaemon_upload.rs to comply with the API format of version 2.1.55.

- Aim to allow Task to focus on disk interactions while delegating memory cache operations to CacheTask.

Signed-off-by: fu220 <2863318196@qq.com>
2025-08-06 14:40:39 +08:00
Gaius 7796ee7342
chore(deps): remove unused dependencies (#1274)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-08-06 11:11:15 +08:00
Gaius 4d6ad26d87
feat: move metadata update after content task creation in Storage::download_task_started (#1234)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-07-16 17:52:09 +08:00
Gaius cad36b3a19
feat: remove write buffer and check piece length when write piece finished
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-07-03 02:03:59 +08:00
dependabot[bot] 8bc771e619
chore(deps): Bump num_cpus from 1.16.0 to 1.17.0 (#1187)
Bumps [num_cpus](https://github.com/seanmonstar/num_cpus) from 1.16.0 to 1.17.0.
- [Release notes](https://github.com/seanmonstar/num_cpus/releases)
- [Changelog](https://github.com/seanmonstar/num_cpus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/seanmonstar/num_cpus/compare/v1.16.0...v1.17.0)

---
updated-dependencies:
- dependency-name: num_cpus
  dependency-version: 1.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-16 22:10:57 +08:00
Gaius 8b6758e79e
feat(dragonfly-client-storage): add dist_threshold for disk usage calculation in GC policy (#1191)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-06-16 14:08:16 +00:00
Gaius 7819702b67
feat(tracing): align resource attributes with OpenTelemetry semantic conventions (#1164)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-05-29 00:00:12 +08:00
Gaius f2315f2804
feat(dragonfly-client-storage): add write_piece_timeout to storage config and enhance piece download timeout handling (#1146)
feat(dragonfly-client-storage): add write_piece_timeout to storage config and enhance piece download timeout handling

This change introduces a `write_piece_timeout` configuration to the storage module, allowing customizable timeouts for writing pieces to storage (e.g., disk or cache). The default timeout is set to 90 seconds. The total timeout for piece operations now combines `download.piece was timeout` and `storage.write_piece_timeout`. Key changes include:

- Added `write_piece_timeout` field to `Storage` config with default and serialization support.
- Updated `wait_for_piece_finished` and `wait_for_piece_finished_by_task_id` to use combined timeout (`download.piece_timeout + storage.write_piece_timeout`).
- Introduced `DownloadPieceFinished` error for piece download timeouts and refactored `download_piece_from_source_finished` and `download_piece_from_parent_finished` to enforce timeouts using `tokio::select!`.
- Increased default `download.piece_timeout` from 60s to 120s for robustness.
- Removed `wait_for_piece_count` debug logging to simplify code.
- Updated tests to validate `write_piece_timeout` parsing.

These changes improve timeout granularity and reliability for piece operations.

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-05-12 20:52:21 +08:00
Gaius a2ba39026e
feat: support fallocate for creating task (#1119)
* feat: support fallocate and filling zero for creating task

* feat: support fallocate and filling zero for creating task

---------

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-29 11:43:42 +08:00
Gaius 63ceb47d82
feat(dragonfly-client-storage): Change error log to info for cache task deletion failure (#1108)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-22 13:29:10 +00:00
Gaius 2a0ef8ec19
fix: solve the memory leak caused by rayon (#1102)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-18 09:49:05 +00:00
Gaius 3abe3654ef
chore: update verison to 0.2.23 (#1090)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-11 17:22:41 +08:00
Gaius 8631030c97
test(dragonfly-client-storage/benches): remove 1u64 when create cache (#1089)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-11 16:27:04 +08:00
Gaius f1a9f0af55
feat: remove unnecessary dependencies (#1085)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-10 17:03:04 +08:00
Gaius 2e08f6622a
refactor(dragonfly-client-storage): lru cache support for piece download/upload (#1084)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-04-09 23:19:46 +08:00
SouthWest7 9c3019f88b
feat: add cache support for piece download/upload. (#1074)
Signed-off-by: southwest miao <1403572259@qq.com>
2025-04-09 17:16:19 +08:00
Gaius 83bbf1973a
test(dragonfly-client-storage): add unit tests for content and fix doctest syntax (#1065)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-31 11:52:03 +08:00
Gaius 250f083980
feat: optimize hard link and copy logic (#1064)
* feat: optimize hard link and copy logic

Refer to https://github.com/dragonflyoss/design/blob/main/systems-analysis/file-download-workflow-with-hard-link/README.md

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-28 18:29:20 +08:00
Gaius 4f3e2f6e4f
feat: optimize benches for cache and lru cache (#1063)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-27 22:16:22 +08:00
SouthWest7 47d2222900
test: add benchmarks for dragonfly-client-storage/cache (#1057)
Signed-off-by: southwest miao <1403572259@qq.com>
2025-03-27 18:24:49 +08:00
Gaius 2e46546bb6
feat(dragonfly-client-storage): optimize lru cache
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-24 13:56:49 +08:00
SouthWest7 3111106f4a
test(dragonfly-client-storage): add unit tests for `lru_cache` and `cache` module. (#1050)
Signed-off-by: Gaius <gaius.qi@gmail.com>
Signed-off-by: southwest miao <1403572259@qq.com>
Co-authored-by: Gaius <gaius.qi@gmail.com>
2025-03-24 13:40:06 +08:00
Gaius e5b8dcb383
refactor: optimize storage cache (#1041)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-14 18:49:37 +08:00
Gaius 70eca028cd
refactor(dragonfly-client-storage): optimize lru_cache for storage (#1040)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-14 15:49:56 +08:00
SouthWest7 9ca89909f3
feat: add a LRU cache module for preheat jobs (#945)
Signed-off-by: southwest miao <1403572259@qq.com>
2025-03-14 13:47:53 +08:00
Gaius 6bc3fbbd48
fix(dragonfly-client-storage): improve hard link handling in content storage (#1038)
- Bumped version from 0.2.18 to 0.2.19 across all packages
- Modified hard link logic in content.rs to:
  - Move existence check after initial hard link attempt
  - Handle AlreadyExists error case more gracefully
  - Apply consistent behavior for both regular and persistent cache tasks
- Maintains original functionality while improving error handling flow

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-13 17:51:15 +08:00
Gaius a75988717c
test(dragonfly-client-storage): optimize engine tests (#1036)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-11 22:37:21 +08:00
Chongzhi Deng e34d07f419
test: add unit tests for RocksdbStorageEngine (#1035)
Signed-off-by: BruceAko <chongzhi@hust.edu.cn>
2025-03-11 20:31:43 +08:00
Gaius 68bc6df1b2
feat: change crc to crc32fast (#1023)
* feat: change crc to crc32fast

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: update

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: update

Signed-off-by: Gaius <gaius.qi@gmail.com>

---------

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-06 13:58:03 +08:00
Gaius 00ef5fdfbd
refactor(dragonfly-client-storage): optimize buffer creation sequence for better readability (#1022)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-03-05 22:39:59 +08:00
Gaius 31e4d136d0
feat(dragonfly-client-storage): optimize hard link handling and cache logic (#1005)
- Bump all crate versions from 0.2.15 to 0.2.16
- Add cross-platform device/inode check for hard link operations
- Refactor hard link logic to remove redundant helper methods
- Implement proper file existence checks with same-device/inode validation

Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-27 00:23:03 +08:00
Gaius 300bb7b786
chore(release): set toolchain to 1.82.0 (#1006)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-27 00:22:25 +08:00
Gaius 6d6a802f09
feat: add X-Dragonfly-Output-Path header for proxy (#993)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-24 12:06:28 +08:00
Gaius c3911f0469
test(dragonfly-client-storage): add test case for has_enough_space (#992)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-20 20:21:49 +08:00
Gaius 04c71e6e2b
feat: calculate digest of the persistent cache piece to check the integrity of the metadata (#980)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-14 12:00:53 +08:00
KennyMcCormick 32c99b711e
add checksum in download request to verify its consistency (#979)
Signed-off-by: cormick <cormick1080@gmail.com>
2025-02-14 10:37:57 +08:00
Gaius 3339d5ca2f
feat: bump api from v2.1.23 to v2.1.25 and optimize gc for persistent cache task (#978)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-12 17:23:49 +08:00
Gaius d56dae38ab
feat: add iperf3 for Dockerfile and change is_expired for PersistentCacheTask (#975)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-02-11 20:35:51 +08:00
Gaius 58da38d9f9
feat: support persist cache task when scheduler replicates task (#953)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-01-26 22:25:05 +08:00
Gaius e787afeab3
feat: update ttl, persistent and piece_length when download persistent cache task started (#952)
Signed-off-by: Gaius <gaius.qi@gmail.com>
2025-01-26 10:46:01 +08:00