dragonflyoss
/
client
mirror of https://github.com/dragonflyoss/client.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,235 Commits 11 Branches 201 Tags 8.2 MiB
Commit Graph

130 Commits

Author SHA1 Message Date
Gaius d4096db1e2
feat: support download persistent cache task from parents (#950) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2025-01-21 21:18:22 +08:00
Gaius 80dfd4dc39
feat: upload persistent cache task by piece level (#944) 2025-01-15 19:02:05 +08:00
Gaius 9823eac701
feat: use InspectReader to calculate crc32 (#935) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2025-01-08 23:25:07 +08:00
Gaius f4c991c34d
feat: replace crc32c with crc-rc (#934) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2025-01-08 21:20:07 +08:00
Gaius 6d9cdbf968
feat: remove crc32 iso3309 from write piece func (#933) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2025-01-07 20:29:28 +08:00
Gaius 3d91c09b0a
feat: support transfer piece content from dfdaemon (#926) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2025-01-03 17:15:50 +08:00
Gaius 70b8925b74
test: add unit tests for calculate_piece_range (#907) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-12-20 13:37:40 +08:00
Gaius 6ace03a0e6
feat: calculate piece range and store the actual piece reader (#906) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-12-19 21:56:41 +08:00
Gaius c67f13e231
feat: rename remote peer to parent (#901) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-12-17 16:19:28 +08:00
Gaius 9fd770ffab
feat: add cache store for proxy (#870) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-28 22:27:09 +08:00
Gaius 0d50958dbc
feat: remove unnecessary get_piece (#869) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-26 20:52:42 +08:00
Gaius e7740f46ed
feat: use CRC-32-Castagnoli algorithm with hardware acceleration (#866) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-26 10:32:35 +08:00
Gaius 10bab190a2
fix: rocksdb panic caused by out of bounds (#860) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-22 20:53:33 +08:00
Gaius 28f9f1cbf2
feat: add disk bandwidth for metrics and announcer (#845) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-18 19:57:46 +08:00
Gaius fbca1897c6
feat: remove unnecessary logs (#844) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-13 22:48:41 +08:00
Gaius 99e3080d8c
feat: remove fill_cache from getting in rocksdb (#843) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-13 20:16:15 +08:00
Gaius db7b878b5a
feat: optimize convert headermap to hashmap (#841) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-13 17:21:23 +08:00
Gaius 6d27c92ac6
feat: optimize generater of the piece id (#839) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-12 23:53:54 +08:00
yxxhero 23c651e187
refactor(metadata.rs): simplify is_empty checks for Task and Persiste… (#829) 2024-11-09 21:20:21 +08:00
Gaius ee21989120
feat: when the task is downloading, it is not allowed to delete the task (#828) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-11-08 17:34:13 +08:00
Gaius 1a85ccc7a6
feat: storage needs split by the first 3 characters of task id(sha256) to avoid too many files in one directory (#792) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-10-21 21:08:10 +08:00
Gaius f536bab788
feat: rename cache task to persistent cache task (#753) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-09-27 17:32:43 +08:00
Gaius e2209a8e61
refactor: change comments to rust style (#742) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-09-20 20:15:51 +08:00
Gaius d9f15bdbfe
feat: add log for deleting task content (#741) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-09-20 15:06:32 +08:00
Gaius e0914ff8e9
feat: announce disable shared for scheduler (#735) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-09-14 17:06:45 +08:00
Gaius 91747990d2
feat: ensure the parent directory of the destination exists (#734) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-09-10 19:55:46 +08:00
Zyyeric 9f2f16c71d
fix: Ensure the parent directory of the destination exists (#732) 
Signed-off-by: Zyyeric <eric1976808123@gmail.com>
 2024-09-10 15:46:58 +08:00
Gaius 38de24bee0
test: remove metadata mock const (#697) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-28 10:55:46 +08:00
Chongzhi Deng ee4ce9cc62
feat: add get_pieces() and fix unit tests (#696) 
Signed-off-by: BruceAko <chongzhi@hust.edu.cn>
 2024-08-28 10:34:51 +08:00
Gaius b4c0e817d6
feat: add tracing flame for dfdaemon (#694) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-28 10:34:24 +08:00
Gaius 2fe844ac26
feat: set prefix_extractor for column families (#693) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-26 21:22:22 +08:00
Gaius a19919b0bd
feat: add log for deleting task (#683) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-19 21:09:12 +08:00
Gaius 71efbe9abd
feat: optimize gc with rocksdb (#668) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-14 21:14:42 +08:00
Gaius e7f7c50fa4
feat: calculate piece length by file length (#661) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-12 21:09:10 +08:00
Gaius 28d17f9980
feat: add log for deleting piece (#656) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-06 21:21:47 +08:00
Gaius 5647215c9e
feat: support crc32 for calculating piece content (#649) 
* feat: support crc32 for calculating piece info

Signed-off-by: Gaius <gaius.qi@gmail.com>

* Trigger CI pipeline

Signed-off-by: Gaius <gaius.qi@gmail.com>

---------

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-05 18:54:01 +08:00
Gaius 50fa71e4c9
feat: remove is_uploading condition in gc (#646) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-08-05 16:31:18 +08:00
Gaius c202b14fd2
feat: add garbage collection for cache task (#610) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-07-18 11:54:05 +08:00
Gaius f9c8f2875f
feat: implement export in dfcache (#609) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-07-17 23:17:09 +08:00
Gaius 8d44be49be
feat: optimize log for piece id with tracing instrument (#585) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-07-11 15:09:13 +08:00
Gaius 2366f462be
feat: add import feature in client (#575) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-07-08 15:20:55 +08:00
Gaius 918767934c
feat: add import subcommand (#572) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-07-04 21:41:54 +08:00
Gaius 213a54ecaf
feat: add cache_task implements in storage (#549) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-06-19 21:37:50 +08:00
Gaius 832af14c91
feat: wait for piece finished before update piece metadata (#535) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-06-13 15:51:54 +08:00
Gaius c995295fc1
feat: set fill cache is false (#527) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-06-06 21:47:51 +08:00
Gaius b67fc8d355
feat: remove rocksdb bloom filter (#523) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-06-05 19:47:22 +08:00
Gaius d500a0f6c0
feat: add error log for backend (#522) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-06-05 14:19:17 +08:00
Gaius 8853191cae
feat: add keep to indicate whether keep data when dfdaemon restarts (#511) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-28 23:48:34 +08:00
Gaius 468462e9ae
feat: use public backend request, backend response and backend error (#510) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-28 23:07:24 +08:00
Gaius 672e3f0bc4
feat: define task is_expired by upload_at (#482) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-17 17:01:29 +08:00
Gaius 05d0f931b6
feat: add leave task and remove task finished only download from local (#480) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-16 22:47:46 +08:00
Gaius 5c8e9f5ef2
feat: if task download failed change the metadata (#479) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-16 18:44:13 +08:00
Gaius 5f4476574c
feat: use set_total_order_seek to skip bloom filter (#478) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-16 15:28:18 +08:00
Gaius 033bc13c9a
feat: optimize default params for concurrent (#476) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-16 11:39:22 +08:00
Gaius 1038d79421
feat: only prefetch once (#475) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-16 00:10:50 +08:00
Gaius 014eb7fde9
feat: set set_total_order_seek true (#474) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-15 22:58:10 +08:00
Gaius fd1f368601
feat: change prefix_extractor to 128 (#473) 
* feat: change the size of log max files

Signed-off-by: Gaius <gaius.qi@gmail.com>

* feat: change prefix_extractor to 128

Signed-off-by: Gaius <gaius.qi@gmail.com>

---------

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-15 22:18:58 +08:00
Gaius 079aa2af9f
feat: write sync for rocksdb (#471) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-15 20:02:59 +08:00
Gaius af1fe2e91e
feat: replace TransactionDB with DB (#468) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-15 11:14:43 +08:00
Gaius 1223e4d096
feat: remove transaction in storage (#467) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-14 22:35:20 +08:00
Gaius 17955f3702
fix: txn memory leak in storage (#466) 
feat: txn memory leak in storage

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-14 21:37:47 +08:00
Gaius 9732a92283
fix: transaction memory leak storage engine (#465) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-14 19:57:16 +08:00
Gaius 8030a2338b
fix: add rollback for with_txn (#464) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-14 12:29:50 +08:00
Gaius 781784a883
feat: calculate piece digest by blake3 (#456) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-10 14:36:45 +08:00
Gaius 8c09ed2105
feat: leave host and peer from scheduler (#448) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-08 17:03:56 +08:00
Gaius 2e36483a09
feat: add metrics for dfdaemon (#446) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-05-08 15:43:11 +08:00
Gaius 4849535dc8
feat: add buffer size config for dfdaemon (#434) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-04-29 14:44:46 +08:00
Gaius aa11bf97af
feat: change buffer size to 32KB (#432) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-04-29 12:14:49 +08:00
Gaius d254e13591
fix: hardlink failed when file is not exists (#418) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-04-24 12:06:32 +08:00
Gaius fe7e25c99d
chore: add License for storage_engine (#412) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-04-23 11:55:13 +08:00
imtsuki 76aa34206f
feat: extract storage engine into traits (#401) 
Signed-off-by: imtsuki <me@qjx.app>
 2024-04-23 10:45:16 +08:00
Gaius dc70300f0d
feat: super replace with crate (#395) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-04-17 15:33:20 +08:00
imtsuki c60dd7aa5f
Refactor: wrap rocksdb transaction operations in closures (#393) 
* refactor: wrap rocksdb txn operations in closures

Signed-off-by: imtsuki <me@qjx.app>

* use generics for CF_NAME

Signed-off-by: imtsuki <me@qjx.app>

* address review comments

Change-Id: I86eccaf3cd9cc2b1128ec89c4a2cf041947a87ea
Signed-off-by: imtsuki <jianxin.qiu@outlook.com>

---------

Signed-off-by: imtsuki <me@qjx.app>
Signed-off-by: imtsuki <jianxin.qiu@outlook.com>
 2024-04-17 15:20:11 +08:00
AngrySean 7d18b2d7a0
Refactor: extract error module (#375) 
Co-authored-by: Xu Jihui <jihui.xjh@bytedance.com>
 2024-04-15 11:37:49 +08:00
Gaius 1b85a6346e
feat: change memtable memory limit in rocksdb (#364) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-04-03 16:48:15 +08:00
Gaius ac9d291f5c
feat: add truncate options to content for writing piece (#344) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-03-22 15:29:01 +08:00
Gaius e9010361e1
feat: add read buffer to io::copy (#325) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-03-15 18:37:19 +08:00
Gaius 36f8c62260
feat: add empty task handler for hard_link_or_copy (#310) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-03-12 20:16:01 +08:00
Gaius e7f53a09e1
chore: remove readme in sub crates (#303) 
Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-03-07 10:28:10 +08:00
Gaius a653b401da
refactor: add dragonfly-client-storage crate (#297) 
chore: add dragonfly-client-storage crate

Signed-off-by: Gaius <gaius.qi@gmail.com>
 2024-03-05 16:11:52 +08:00