Compare commits

...

53 Commits

Author SHA1 Message Date
paderlol 0c35e096c2
Merge pull request #384 from nacos-group/develop
Optimize some codes
2024-11-30 15:23:35 +08:00
paderlol 142a36f608
Merge pull request #383 from paderlol/develop
Optimize some codes
2024-11-30 15:22:07 +08:00
paderlol b18afb95c1
Optimize some codes 2024-11-30 15:20:40 +08:00
paderlol aa87b24e1a
Merge pull request #381 from nacos-group/develop
Develop
2024-10-29 10:55:13 +08:00
paderlol fdc51ba605
Merge pull request #380 from paderlol/develop
Write unit tests for all utils classes
2024-10-29 10:53:56 +08:00
paderlol ce6249142f
Write unit tests for all utils classes 2024-10-29 10:52:01 +08:00
paderlol 26c8b307d7
Merge pull request #378 from nacos-group/develop
Develop
2024-09-28 13:29:35 +08:00
paderlol 6585b217bc
Merge pull request #377 from paderlol/develop
1. 对代码中依赖注入方式全部修改成构造函数注入
2024-09-28 13:28:32 +08:00
paderlol b609c729c3
1. 对代码中依赖注入方式全部修改成构造函数注入 2024-09-28 13:26:07 +08:00
paderlol 432ced4e0e
Merge pull request #375 from paderlol/develop
优化部分代码
2024-08-29 16:49:31 +08:00
paderlol 116116c8c4
1. 优化部分集群新增部分冗余代码
2. 升级Java JDK到17
2024-08-29 16:48:58 +08:00
paderlol ca08bd7d6e
1. 优化部分集群新增部分冗余代码
2. 升级Java JDK到17
2024-08-29 16:43:05 +08:00
paderlol 81610e505a
Merge pull request #373 from paderlol/add_validation_cluster
修复有任务运行时可以删除集群信息的问题 #371
2024-07-31 15:57:00 +08:00
paderlol 17439be724
修复有任务运行时可以删除集群信息的问题 #371 2024-07-31 15:45:05 +08:00
paderlol 91999fbdcb
Merge pull request #368 from nacos-group/develop
优化以及修复小错误
2024-07-20 16:44:16 +08:00
paderlol 6645b00f24
Develop (#367)
* 1. 优化部分代码

* 整理代码:
 1. 移除枚举类冗余方法以及SkyWalkerCacheServices中未使用方法
 2. 部分服务类改用构造注入
 3. Entity对象重写equals和hashCode
2024-07-20 16:33:19 +08:00
paderlol acb7b49551
Merge pull request #360 from MajorHe1/patch-1
[#ISSUE 359]Update SkyWalkerCacheServices.java, fix operationId empty-judge problem
2024-07-20 14:26:31 +08:00
MajorHe1 ba416d1496
[#ISSUE 359]Update SkyWalkerCacheServices.java, fix operationId empty-judge problem 2024-06-13 20:05:49 +08:00
paderlol ff5e484997
Merge pull request #357 from nacos-group/develop
Develop
2024-06-10 22:19:41 +08:00
paderlol b900c28347
1. 优化部分代码 (#356) 2024-06-10 22:18:13 +08:00
paderlol 5d17242ffc
Develop (#355)
* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* #346 #350:nacos 2 nacos 同步关闭后心跳没有停止问题 ,nacos-sync删除目标节点问题 (#347)

* update port

* fix #297 (#298)

Co-authored-by: yangchun2 <yangchun2@joyy.com>

* Revert "fix #297 (#298)" (#318)

This reverts commit a9df169b5a.

* 0.4.9-pre (#325)

* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:集群信息保存 cluster_level 为 null

* fix:
1、注册时根据中心化逻辑判断同步,但是删除时逻辑不一致问题
2、如果停止同步nameservice为空问题

* 问题在于对destInstances列表的更新方式。在Java中,方法参数是按值传递的。这意味着当你传递一个对象到方法中时,实际上传递的是对象引用的副本。因此,如果你在方法内部改变了这个引用指向的对象(例如,将其指向一个新的对象),这个改变不会影响到原始的对象引用。

在你的代码中,destInstances = newDestInstance;这一行只是改变了destInstances引用在方法内部的指向,而不会改变方法外部传入的destInstances列表对象。这意味着,尽管你筛选出了需要反注册的实例,但这个改变不会反映到方法调用者那里。

---------

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>

* 0.5.0 (#351)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* Develop (#353)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* 1. 优化部分代码 (#354)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* 1. 优化部分代码

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: 龙竹 <34528665+dragonTalon@users.noreply.github.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
2024-06-10 22:04:01 +08:00
paderlol 39357660ab
Merge branch 'master' into develop 2024-06-10 22:02:39 +08:00
paderlol 49abb7c286
1. 优化部分代码 (#354)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* 1. 优化部分代码
2024-06-10 22:01:12 +08:00
paderlol 8185feca1d
0.5.0 (#352)
* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* #346 #350:nacos 2 nacos 同步关闭后心跳没有停止问题 ,nacos-sync删除目标节点问题 (#347)

* update port

* fix #297 (#298)

Co-authored-by: yangchun2 <yangchun2@joyy.com>

* Revert "fix #297 (#298)" (#318)

This reverts commit a9df169b5a.

* 0.4.9-pre (#325)

* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:集群信息保存 cluster_level 为 null

* fix:
1、注册时根据中心化逻辑判断同步,但是删除时逻辑不一致问题
2、如果停止同步nameservice为空问题

* 问题在于对destInstances列表的更新方式。在Java中,方法参数是按值传递的。这意味着当你传递一个对象到方法中时,实际上传递的是对象引用的副本。因此,如果你在方法内部改变了这个引用指向的对象(例如,将其指向一个新的对象),这个改变不会影响到原始的对象引用。

在你的代码中,destInstances = newDestInstance;这一行只是改变了destInstances引用在方法内部的指向,而不会改变方法外部传入的destInstances列表对象。这意味着,尽管你筛选出了需要反注册的实例,但这个改变不会反映到方法调用者那里。

---------

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>

* 0.5.0 (#351)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* Develop (#353)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: 龙竹 <34528665+dragonTalon@users.noreply.github.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
2024-05-19 00:17:01 +08:00
paderlol 5789d4d774
Develop (#353)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释
2024-05-19 00:15:50 +08:00
paderlol 2fdf44d308
Merge branch 'master' into develop 2024-05-19 00:08:04 +08:00
paderlol 5941bfb9bc
0.5.0 (#351)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释
2024-05-18 23:50:23 +08:00
龙竹 0c24dcb0a2
#346 #350:nacos 2 nacos 同步关闭后心跳没有停止问题 ,nacos-sync删除目标节点问题 (#347)
* update port

* fix #297 (#298)

Co-authored-by: yangchun2 <yangchun2@joyy.com>

* Revert "fix #297 (#298)" (#318)

This reverts commit a9df169b5a.

* 0.4.9-pre (#325)

* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:集群信息保存 cluster_level 为 null

* fix:
1、注册时根据中心化逻辑判断同步,但是删除时逻辑不一致问题
2、如果停止同步nameservice为空问题

* 问题在于对destInstances列表的更新方式。在Java中,方法参数是按值传递的。这意味着当你传递一个对象到方法中时,实际上传递的是对象引用的副本。因此,如果你在方法内部改变了这个引用指向的对象(例如,将其指向一个新的对象),这个改变不会影响到原始的对象引用。

在你的代码中,destInstances = newDestInstance;这一行只是改变了destInstances引用在方法内部的指向,而不会改变方法外部传入的destInstances列表对象。这意味着,尽管你筛选出了需要反注册的实例,但这个改变不会反映到方法调用者那里。

---------

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
2024-05-15 09:53:45 +08:00
paderlol a1d683a7c9
0.4.9-pre (#325)
* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>
2023-05-15 17:33:26 +08:00
paderlol be65db1892
Refactoring the Nacos Sync to Consul Logic (#324)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.
2023-05-15 17:25:22 +08:00
paderlol eb93fdd52a
Fix cyclic dependency code (#323)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.
2023-05-15 17:21:53 +08:00
paderlol e7e52acfa7
Fix #305 (#322)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305
2023-05-15 16:51:11 +08:00
paderlol 02838da959
Develop (#321)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308
2023-05-15 16:13:39 +08:00
paderlol 0e73a0864c
Optimize the code for assigning tasks. (#320) 2023-05-15 15:58:39 +08:00
paderlol 61d347609e
Revert "fix #297 (#298)" (#318)
This reverts commit a9df169b5a.
2023-05-15 09:55:50 +08:00
杨春 a9df169b5a
fix #297 (#298)
Co-authored-by: yangchun2 <yangchun2@joyy.com>
2023-05-15 09:53:25 +08:00
chenhao26 98efe26906
Feat/sync support2.x#mutiple thread sync02 (#304)
* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
2022-08-01 08:53:33 +08:00
paderlol 384a9cc1bb
Merge pull request #303 from nacos-group/develop
0.4.8
2022-07-30 12:47:09 +08:00
paderlol 0ffdca3e0b
Merge pull request #302 from paderlol/develop
Optimize Nacos synchronization Nacos logic.
2022-07-30 12:18:18 +08:00
paderlol 5ae4071508 Optimize Nacos synchronization Nacos logic. 2022-07-30 12:15:56 +08:00
paderlol 4082efa1fc
Merge pull request #301 from paderlol/develop
Optimize Nacos synchronization Eureka logic.
2022-07-30 11:49:28 +08:00
paderlol fc03e6483c Optimize Nacos synchronization Eureka logic. 2022-07-30 11:46:46 +08:00
paderlol 2b8d29a0ab
Merge pull request #296 from jefferson-chern/fix-resync-when-two-way-sync
[fix #295] 双向同步时,重启某个集群下所有实例,可以重新同步
2022-06-14 21:00:22 +08:00
jefferson 881c11d51f refactor code: remove some useless code. 2022-06-14 16:14:07 +08:00
jefferson 855922964e fix #295 2022-06-14 14:59:12 +08:00
paderlol 9e814d2d43
Merge pull request #294 from paderlol/develop
Refactor some code
2022-06-08 21:36:08 +08:00
paderlol cdfbebf407 Refactor some code 2022-06-08 21:34:13 +08:00
paderlol 017cacf12c
Merge pull request #293 from jefferson-chern/fix-log-bug-and-refactor-version-and-upgrade-to-0.4.8
统一版本号;升级到0.4.8;修复日志不删除问题
2022-06-07 13:52:52 +08:00
jefferson ab4a4bbb92 1. dependency的版本号全部挪到父pom的dependencyManagement;
2. plugin的版本号全部挪到父pom的pluginManagement;
3. nacosSync名称修改为nacos-sync;
4. 父pom和子pom的version统一由revision管理,只需一处修改;
5. 修复一些readme.md中的明显错误;
6. 修复日志文件永不删除的问题;
7. upgrade version to 0.4.8;
8. remove some useless newline in redeme.md.
2022-06-05 15:28:04 +08:00
paderlol 853119b615
Merge pull request #289 from CherishCai/feat_TaskAddAll_with_nacos_source
Feat task add all with nacos source
2022-06-04 19:42:15 +08:00
paderlol f99398c649
Merge pull request #290 from Oliverwqcwrw/master-update-port-1
Update README.md port
2022-04-20 09:59:44 +08:00
Oliver a61a79aca9 update port 2022-04-20 09:02:22 +08:00
hongwen.chw 00570523fa feat: 支持从 sourceCluster 获取所有 service,然后生成同步到 destCluster 的任务。// 目前仅支持 Nacos 为源的同步类型,待完善更多类型支持. 2022-03-23 20:14:52 +08:00
95 changed files with 3550 additions and 1800 deletions

1
.gitignore vendored
View File

@ -11,3 +11,4 @@ target
node_modules
test/derby.log
/console-fe
.flattened-pom.xml

View File

@ -5,7 +5,7 @@
## Function
- Console: provide API and console for management
- Worker: provider the service registration synchronization.
- Worker: provide the service registration synchronization.
## Architecture
@ -44,15 +44,9 @@ Info | +------------+ ^
- NacosCluster target will dedup the synchronization information from Nacos.
## Quick Start:
- Swagger API: http://127.0.0.1:8081/swagger-ui.html#/
- Web Console: http://127.0.0.1:8081/
- Swagger API: http://127.0.0.1:8083/swagger-ui.html#/
- Web Console: http://127.0.0.1:8083/
- Others: TBD
# NacosSync Migration User Guide
@ -84,7 +78,7 @@ Before you begin, install the following:
- 64bit OS: Linux/Unix/Mac/Windows supported, Linux/Unix/Mac recommended.
- 64bit JDK 1.8+: downloads, JAVA_HOME settings.
- Maven 3.2.x+: downloads, settings.
- Maven 3.5.2+: [downloads](https://maven.apache.org/download.cgi), [settings](https://maven.apache.org/settings.html).
- MySql 5.6.+
## Download & Build From Release
@ -96,7 +90,7 @@ There are two ways to get NacosSync.
``` xml
cd nacosSync/
cd nacos-sync/
mvn clean package -U
```
@ -105,7 +99,7 @@ The path to the target file:
``` xml
nacos-sync/nacossync-distribution/target/nacosSync.0.3.8.zip
nacos-sync/nacossync-distribution/target/nacos-sync-0.5.0.tar.gz
```
@ -113,7 +107,7 @@ After extracting the installation package, the directory structure:
``` xml
nacosSync
nacos-sync
├── LICENSE
├── NOTICE
├── bin
@ -124,7 +118,7 @@ nacosSync
│   ├── application.properties
│   └── logback-spring.xml
├── logs
└── nacosSync-server.jar
└── nacos-sync-server.jar
```
@ -132,7 +126,7 @@ nacosSync
The default is Mysql database, which can support other relational databases
- Build db schema, the default schema name nacos_Sync.
- Build db schema, the default schema name nacos_sync.
- Tables do not need to be created separately, which is conducive to hibernate's automatic table creation function.
- If the automatic table creation fails, you can build the table nacosSync.sql, the table statement is in the bin folder.
@ -161,15 +155,18 @@ sh startup.sh start
``` xml
http://127.0.0.1:8081/#/serviceSync
http://127.0.0.1:8083/#/serviceSync
```
## Advanced Configuration
### Full Synchronization from Zookeeper to Nacos (Dubbo)
When “*” is entered in the “Service Name” field of this form, it will fully synchronize all services from Zookeeper to Nacos, but only when using Dubbo.
![img_1.png](img_1.png)
### Full Synchronization from Nacos to Nacos
When “All” is entered in the “Service Name” field of this form, it will automatically synchronize all registered services within the **default group** of the current cluster.
This description explains the functionality clearly for English-speaking users.
![img_2.png](img_2.png)

BIN
img_1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

BIN
img_2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

View File

@ -16,7 +16,7 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.7</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -27,5 +27,4 @@
<artifactId>junit</artifactId>
</dependency>
</dependencies>
</project>
</project>

View File

@ -13,7 +13,7 @@ class Layout extends React.Component {
<Header />
<Row className="layout">
<Col fixedSpan="9" className="nav-bar">
<h1 className="title">Nacos-Sync 3.0</h1>
<h1 className="title">Nacos-Sync 0.4.8</h1>
<Menu />
</Col>
<Col className="main-panel">{this.props.children}</Col>

View File

@ -10,6 +10,8 @@ CREATE TABLE `cluster` (
`connect_key_list` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`user_name` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`password` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`namespace` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`cluster_level` int default 0,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
/******************************************/
@ -39,5 +41,6 @@ CREATE TABLE `task` (
`task_status` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`version` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`worker_ip` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`status` int default null ,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;

View File

@ -13,14 +13,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
pid=`ps ax | grep -i 'nacosSync' |grep java | grep -v grep | awk '{print $1}'`
pid=`ps ax | grep -i 'nacos-sync' |grep java | grep -v grep | awk '{print $1}'`
if [ -z "$pid" ] ; then
echo "No nacosSync running."
echo "no nacos-sync running."
exit -1;
fi
echo "The nacosSync(${pid}) is running..."
echo "the nacos-sync(${pid}) is running..."
kill ${pid}
echo "Send shutdown request to nacosSync(${pid}) OK"
echo "Send shutdown request to nacos-sync(${pid}) OK"

View File

@ -1 +1 @@
java -jar ../nacosSync-server.jar -server -Xms2g -Xmx2g -Xmn1g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m --spring.config.location=../conf/application.properties
java -jar ../nacos-sync-server.jar -server -Xms2g -Xmx2g -Xmn1g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m --spring.config.location=../conf/application.properties

View File

@ -47,24 +47,24 @@ export BASE_DIR=`cd $(dirname $0)/..; pwd`
JAVA_OPT="${JAVA_OPT} -server -Xms2g -Xmx2g -Xmn1g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${BASE_DIR}/nacossync_java_heapdump.hprof"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${BASE_DIR}/nacos-sync-java-heapdump.hprof"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages"
JAVA_OPT="${JAVA_OPT} -Dspring.config.location=${BASE_DIR}/conf/application.properties"
JAVA_OPT="${JAVA_OPT} -DnacosSync.home=${BASE_DIR}"
JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
if [[ "$JAVA_MAJOR_VERSION" -ge "9" ]] ; then
JAVA_OPT="${JAVA_OPT} -Xlog:gc*:file=${BASE_DIR}/logs/nacossync_gc.log:time,tags:filecount=10,filesize=102400"
JAVA_OPT="${JAVA_OPT} -Xlog:gc*:file=${BASE_DIR}/logs/nacos-sync-gc.log:time,tags:filecount=10,filesize=102400"
else
JAVA_OPT="${JAVA_OPT} -Xloggc:${BASE_DIR}/logs/nacossync_gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
JAVA_OPT="${JAVA_OPT} -Xloggc:${BASE_DIR}/logs/nacos-sync-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
fi
JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR}/nacosSync-server.jar"
JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR}/nacos-sync-server.jar"
JAVA_OPT="${JAVA_OPT} --logging.config=${BASE_DIR}/conf/logback-spring.xml"
echo "=============JAVA_HOME:"$JAVA_HOME
echo "=============BASE_DIR:"$BASE_DIR
echo "=============JAVA:"$JAVA
echo "JAVA_HOME:"$JAVA_HOME
echo "BASE_DIR:"$BASE_DIR
echo "JAVA:"$JAVA
if [ ! -d "${BASE_DIR}/logs" ]; then
@ -78,9 +78,9 @@ usage(){
start(){
echo "$JAVA ${JAVA_OPT}" > ${BASE_DIR}/logs/nacossync_start.out 2>&1 &
nohup $JAVA ${JAVA_OPT} >> ${BASE_DIR}/logs/nacossync_start.out 2>&1 &
echo "nacossync is startingyou can check the ${BASE_DIR}/logs/nacossync_start.out"
echo "$JAVA ${JAVA_OPT}" > ${BASE_DIR}/logs/nacos-sync-start.out 2>&1 &
nohup $JAVA ${JAVA_OPT} >> ${BASE_DIR}/logs/nacos-sync-start.out 2>&1 &
echo "nacos-sync is startingyou can check the ${BASE_DIR}/logs/nacos-sync-start.out"
}

View File

@ -5,14 +5,14 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.7</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging>
<artifactId>nacossync-distribution</artifactId>
<build>
<finalName>nacosSync.${parent.version}</finalName>
<finalName>nacos-sync-${project.version}</finalName>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
@ -35,4 +35,4 @@
</plugin>
</plugins>
</build>
</project>
</project>

View File

@ -17,7 +17,7 @@
<id>bin</id>
<includeBaseDirectory>true</includeBaseDirectory>
<!-- file name after unzip -->
<baseDirectory>nacosSync</baseDirectory>
<baseDirectory>nacos-sync</baseDirectory>
<formats>
<format>dir</format>
<format>tar.gz</format>
@ -60,9 +60,9 @@
<destName>NOTICE</destName>
</file>
<file>
<source>../nacossync-worker/target/nacosSync-server.${parent.version}.jar</source>
<source>../nacossync-worker/target/nacos-sync-server-${project.version}.jar</source>
<outputDirectory></outputDirectory>
<destName>nacosSync-server.jar</destName>
<destName>nacos-sync-server.jar</destName>
</file>
</files>
</assembly>

View File

@ -17,7 +17,7 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.7</version>
<version>${revision}</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -25,27 +25,19 @@
<artifactId>nacossync-test</artifactId>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba.nacossync</groupId>
<groupId>${project.groupId}</groupId>
<artifactId>nacossync-worker</artifactId>
<version>0.4.7</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
@ -57,10 +49,8 @@
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>1.7.17</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
@ -68,7 +58,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<parallel>methods</parallel>
<useUnlimitedThreads>true</useUnlimitedThreads>
@ -76,6 +65,4 @@
</plugin>
</plugins>
</build>
</project>
</project>

View File

@ -16,29 +16,11 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.7</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>nacossync-worker</artifactId>
<version>0.4.7</version>
<properties>
<zookeeper.version>3.4.9</zookeeper.version>
<curator.version>4.1.0</curator.version>
<cloud.version>2020.0.2</cloud.version>
<mockito.version>1.10.19</mockito.version>
<nacos.client.verison>1.4.2</nacos.client.verison>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
@ -60,12 +42,12 @@
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<!-- 默认使用HikariCP连接池 -->
<dependency>
@ -78,33 +60,26 @@
</dependency>
<!-- mysql -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<!-- swagger2 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>3.0.0</version>
<groupId>com.mysql</groupId>
<artifactId>mysql-connector-j</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>3.0.0</version>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<!--nacos-->
<dependency>
<groupId>com.alibaba.nacos</groupId>
<artifactId>nacos-client</artifactId>
<version>${nacos.client.verison}</version>
</dependency>
<!--zookeeper-->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
@ -112,11 +87,9 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
@ -124,17 +97,13 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>${curator.version}</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-api</artifactId>
@ -142,14 +111,11 @@
</exclusion>
</exclusions>
</dependency>
<!-- eureka -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<!-- consul -->
<dependency>
<groupId>com.ecwid.consul</groupId>
@ -158,16 +124,15 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>29.0-jre</version>
</dependency>
</dependencies>
<build>
<finalName>nacosSync-server.${parent.version}</finalName>
<finalName>nacos-sync-server-${project.version}</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
@ -183,7 +148,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.2.0</version>
<configuration>
<excludes>
<exclude>application.properties</exclude>
@ -197,6 +161,22 @@
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>17</source>
<target>17</target>
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.34</version>
</path>
</annotationProcessorPaths>
</configuration>
</plugin>
</plugins>
</build>
</project>

View File

@ -14,24 +14,32 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration;
import springfox.documentation.swagger2.annotations.EnableSwagger2;
import org.springframework.context.ConfigurableApplicationContext;
/**
* @author NacosSync
* @version $Id: SkyWalkerMain.java, v 0.1 2018-09-24 PM12:42 NacosSync Exp $$
*/
@EnableSwagger2
@SpringBootApplication(exclude = EurekaClientAutoConfiguration.class)
public class NacosSyncMain {
public static void main(String[] args) {
SpringApplication.run(NacosSyncMain.class, args);
ConfigurableApplicationContext context = SpringApplication.run(NacosSyncMain.class, args);
// Register shutdown callback using Spring Boot's context lifecycle
context.registerShutdownHook();
context.addApplicationListener(event -> {
if (event instanceof org.springframework.context.event.ContextClosedEvent) {
BatchTaskExecutor.shutdown();
}
});
}
}

View File

@ -14,6 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.api;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
@ -44,57 +45,54 @@ import org.springframework.web.bind.annotation.RestController;
@Slf4j
@RestController
public class ClusterApi {
private final ClusterAddProcessor clusterAddProcessor;
private final ClusterDeleteProcessor clusterDeleteProcessor;
private final ClusterDetailQueryProcessor clusterDetailQueryProcessor;
private final ClusterListQueryProcessor clusterListQueryProcessor;
public ClusterApi(
ClusterAddProcessor clusterAddProcessor, ClusterDeleteProcessor clusterDeleteProcessor,
ClusterDetailQueryProcessor clusterDetailQueryProcessor, ClusterListQueryProcessor clusterListQueryProcessor) {
public ClusterApi(ClusterAddProcessor clusterAddProcessor, ClusterDeleteProcessor clusterDeleteProcessor,
ClusterDetailQueryProcessor clusterDetailQueryProcessor,
ClusterListQueryProcessor clusterListQueryProcessor) {
this.clusterAddProcessor = clusterAddProcessor;
this.clusterDeleteProcessor = clusterDeleteProcessor;
this.clusterDetailQueryProcessor = clusterDetailQueryProcessor;
this.clusterListQueryProcessor = clusterListQueryProcessor;
}
@RequestMapping(path = "/v1/cluster/list", method = RequestMethod.GET)
public ClusterListQueryResult clusters(ClusterListQueryRequest clusterListQueryRequest) {
return SkyWalkerTemplate.run(clusterListQueryProcessor, clusterListQueryRequest,
new ClusterListQueryResult());
return SkyWalkerTemplate.run(clusterListQueryProcessor, clusterListQueryRequest, new ClusterListQueryResult());
}
@RequestMapping(path = "/v1/cluster/detail", method = RequestMethod.GET)
public ClusterDetailQueryResult getByTaskId(ClusterDetailQueryRequest clusterDetailQueryRequest) {
return SkyWalkerTemplate.run(clusterDetailQueryProcessor, clusterDetailQueryRequest,
new ClusterDetailQueryResult());
}
@RequestMapping(path = "/v1/cluster/delete", method = RequestMethod.DELETE)
public ClusterDeleteResult deleteCluster(ClusterDeleteRequest clusterDeleteRequest) {
return SkyWalkerTemplate.run(clusterDeleteProcessor, clusterDeleteRequest,
new ClusterDeleteResult());
return SkyWalkerTemplate.run(clusterDeleteProcessor, clusterDeleteRequest, new ClusterDeleteResult());
}
@RequestMapping(path = "/v1/cluster/add", method = RequestMethod.POST)
public ClusterAddResult clusterAdd(@RequestBody ClusterAddRequest clusterAddRequest) {
return SkyWalkerTemplate
.run(clusterAddProcessor, clusterAddRequest, new ClusterAddResult());
return SkyWalkerTemplate.run(clusterAddProcessor, clusterAddRequest, new ClusterAddResult());
}
@RequestMapping(path = "/v1/cluster/types", method = RequestMethod.GET)
public ClusterTypeResult getClusterType() {
return new ClusterTypeResult(ClusterTypeEnum.getClusterTypeCodes());
}
}

View File

@ -14,6 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.api;
import lombok.extern.slf4j.Slf4j;
@ -42,34 +43,36 @@ import com.alibaba.nacossync.template.processor.ConfigQueryProcessor;
@Slf4j
@RestController
public class SystemConfigApi {
@Autowired
private ConfigQueryProcessor configQueryProcessor;
@Autowired
private ConfigDeleteProcessor configDeleteProcessor;
@Autowired
private ConfigAddProcessor configAddProcessor;
private final ConfigQueryProcessor configQueryProcessor;
private final ConfigDeleteProcessor configDeleteProcessor;
private final ConfigAddProcessor configAddProcessor;
public SystemConfigApi(ConfigQueryProcessor configQueryProcessor, ConfigDeleteProcessor configDeleteProcessor,
ConfigAddProcessor configAddProcessor) {
this.configQueryProcessor = configQueryProcessor;
this.configDeleteProcessor = configDeleteProcessor;
this.configAddProcessor = configAddProcessor;
}
@RequestMapping(path = "/v1/systemconfig/list", method = RequestMethod.GET)
public ConfigQueryResult tasks(ConfigQueryRequest configQueryRequest) {
return SkyWalkerTemplate.run(configQueryProcessor, configQueryRequest,
new ConfigQueryResult());
return SkyWalkerTemplate.run(configQueryProcessor, configQueryRequest, new ConfigQueryResult());
}
@RequestMapping(path = "/v1/systemconfig/delete", method = RequestMethod.DELETE)
public ConfigDeleteResult deleteTask(@RequestBody ConfigDeleteRequest configDeleteRequest) {
return SkyWalkerTemplate.run(configDeleteProcessor, configDeleteRequest,
new ConfigDeleteResult());
return SkyWalkerTemplate.run(configDeleteProcessor, configDeleteRequest, new ConfigDeleteResult());
}
@RequestMapping(path = "/v1/systemconfig/add", method = RequestMethod.POST)
public ConfigAddResult taskAdd(@RequestBody ConfigAddRequest configAddRequest) {
return SkyWalkerTemplate.run(configAddProcessor, configAddRequest, new ConfigAddResult());
}
}

View File

@ -10,8 +10,10 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.api;
import com.alibaba.nacossync.pojo.request.TaskAddAllRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.alibaba.nacossync.pojo.request.TaskDeleteInBatchRequest;
import com.alibaba.nacossync.pojo.request.TaskDeleteRequest;
@ -23,6 +25,7 @@ import com.alibaba.nacossync.pojo.result.TaskAddResult;
import com.alibaba.nacossync.pojo.result.TaskDetailQueryResult;
import com.alibaba.nacossync.pojo.result.TaskListQueryResult;
import com.alibaba.nacossync.template.SkyWalkerTemplate;
import com.alibaba.nacossync.template.processor.TaskAddAllProcessor;
import com.alibaba.nacossync.template.processor.TaskAddProcessor;
import com.alibaba.nacossync.template.processor.TaskDeleteInBatchProcessor;
import com.alibaba.nacossync.template.processor.TaskDeleteProcessor;
@ -42,67 +45,83 @@ import org.springframework.web.bind.annotation.RestController;
@Slf4j
@RestController
public class TaskApi {
private final TaskUpdateProcessor taskUpdateProcessor;
private final TaskAddProcessor taskAddProcessor;
private final TaskAddAllProcessor taskAddAllProcessor;
private final TaskDeleteProcessor taskDeleteProcessor;
private final TaskDeleteInBatchProcessor taskDeleteInBatchProcessor;
private final TaskListQueryProcessor taskListQueryProcessor;
private final TaskDetailProcessor taskDetailProcessor;
public TaskApi(TaskUpdateProcessor taskUpdateProcessor, TaskAddProcessor taskAddProcessor,
TaskDeleteProcessor taskDeleteProcessor, TaskDeleteInBatchProcessor taskDeleteInBatchProcessor,
TaskListQueryProcessor taskListQueryProcessor, TaskDetailProcessor taskDetailProcessor) {
TaskAddAllProcessor taskAddAllProcessor, TaskDeleteProcessor taskDeleteProcessor,
TaskDeleteInBatchProcessor taskDeleteInBatchProcessor, TaskListQueryProcessor taskListQueryProcessor,
TaskDetailProcessor taskDetailProcessor) {
this.taskUpdateProcessor = taskUpdateProcessor;
this.taskAddProcessor = taskAddProcessor;
this.taskAddAllProcessor = taskAddAllProcessor;
this.taskDeleteProcessor = taskDeleteProcessor;
this.taskDeleteInBatchProcessor = taskDeleteInBatchProcessor;
this.taskListQueryProcessor = taskListQueryProcessor;
this.taskDetailProcessor = taskDetailProcessor;
}
@RequestMapping(path = "/v1/task/list", method = RequestMethod.GET)
public TaskListQueryResult tasks(TaskListQueryRequest taskListQueryRequest) {
return SkyWalkerTemplate.run(taskListQueryProcessor, taskListQueryRequest, new TaskListQueryResult());
}
@RequestMapping(path = "/v1/task/detail", method = RequestMethod.GET)
public TaskDetailQueryResult getByTaskId(TaskDetailQueryRequest taskDetailQueryRequest) {
return SkyWalkerTemplate.run(taskDetailProcessor, taskDetailQueryRequest, new TaskDetailQueryResult());
}
@RequestMapping(path = "/v1/task/delete", method = RequestMethod.DELETE)
public BaseResult deleteTask(TaskDeleteRequest taskDeleteRequest) {
return SkyWalkerTemplate.run(taskDeleteProcessor, taskDeleteRequest, new BaseResult());
}
/**
* @author yongchao9
* @param taskBatchDeleteRequest
* @return
* @author yongchao9
*/
@RequestMapping(path = "/v1/task/deleteInBatch", method = RequestMethod.DELETE)
public BaseResult batchDeleteTask(TaskDeleteInBatchRequest taskBatchDeleteRequest) {
return SkyWalkerTemplate.run(taskDeleteInBatchProcessor, taskBatchDeleteRequest, new BaseResult());
}
@RequestMapping(path = "/v1/task/add", method = RequestMethod.POST)
public BaseResult taskAdd(@RequestBody TaskAddRequest addTaskRequest) {
return SkyWalkerTemplate.run(taskAddProcessor, addTaskRequest, new TaskAddResult());
}
/**
* TODO 目前仅支持 Nacos 为源的同步类型待完善更多类型支持.
* <p>
* 支持从 sourceCluster 获取所有 service然后生成同步到 destCluster 的任务
* </p>
*/
@RequestMapping(path = "/v1/task/addAll", method = RequestMethod.POST)
public BaseResult taskAddAll(@RequestBody TaskAddAllRequest addAllRequest) {
return SkyWalkerTemplate.run(taskAddAllProcessor, addAllRequest, new TaskAddResult());
}
@RequestMapping(path = "/v1/task/update", method = RequestMethod.POST)
public BaseResult updateTask(@RequestBody TaskUpdateRequest taskUpdateRequest) {
return SkyWalkerTemplate.run(taskUpdateProcessor, taskUpdateRequest, new BaseResult());
}
}

View File

@ -25,15 +25,15 @@ import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.SneakyThrows;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import org.springframework.util.StringUtils;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadLocalRandom;
import lombok.SneakyThrows;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import org.springframework.util.StringUtils;
/**
* @author NacosSync
@ -41,18 +41,20 @@ import org.springframework.util.StringUtils;
*/
@Service
public class SkyWalkerCacheServices {
private static final Map<String, FinishedTask> FINISHED_TASK_MAP = new ConcurrentHashMap<>();
private final ClusterAccessService clusterAccessService;
@Autowired
private ClusterAccessService clusterAccessService;
@Autowired
private ObjectMapper objectMapper;
private static Map<String, FinishedTask> finishedTaskMap = new ConcurrentHashMap<>();
private final ObjectMapper objectMapper;
public SkyWalkerCacheServices(ClusterAccessService clusterAccessService, ObjectMapper objectMapper) {
this.clusterAccessService = clusterAccessService;
this.objectMapper = objectMapper;
}
public String getClusterConnectKey(String clusterId) {
List<String> allClusterConnectKey = getAllClusterConnectKey(clusterId);
return allClusterConnectKey.get(ThreadLocalRandom.current().nextInt(allClusterConnectKey.size()));
}
@ -83,23 +85,32 @@ public class SkyWalkerCacheServices {
FinishedTask finishedTask = new FinishedTask();
finishedTask.setOperationId(operationId);
finishedTaskMap.put(operationId, finishedTask);
FINISHED_TASK_MAP.put(operationId, finishedTask);
}
public FinishedTask getFinishedTask(TaskDO taskDO) {
String operationId = SkyWalkerUtil.getOperationId(taskDO);
if (StringUtils.isEmpty(operationId)) {
if (!StringUtils.hasLength(operationId)) {
return null;
}
return finishedTaskMap.get(operationId);
return FINISHED_TASK_MAP.get(operationId);
}
public void removeFinishedTask(String operationId) {
if (!StringUtils.hasLength(operationId)) {
return;
}
FINISHED_TASK_MAP.remove(operationId);
}
public Map<String, FinishedTask> getFinishedTaskMap() {
return finishedTaskMap;
return FINISHED_TASK_MAP;
}
}

View File

@ -12,6 +12,8 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
import java.util.ArrayList;
import java.util.List;
@ -19,6 +21,7 @@ import java.util.List;
* @author NacosSync
* @version $Id: ClusterTypeEnum.java, v 0.1 2018-09-25 下午4:38 NacosSync Exp $$
*/
@Getter
public enum ClusterTypeEnum {
CS("CS", "configserver集群"),
@ -32,18 +35,16 @@ public enum ClusterTypeEnum {
ZK("ZK", "zookeeper集群");
private String code;
private final String code;
private String desc;
ClusterTypeEnum(String code, String desc) {
this.code = code;
this.desc = desc;
}
public static List<String> getClusterTypeCodes() {
List<String> list = new ArrayList<String>();
List<String> list = new ArrayList<>();
for (ClusterTypeEnum clusterTypeEnum : ClusterTypeEnum.values()) {
list.add(clusterTypeEnum.getCode());
@ -51,42 +52,7 @@ public enum ClusterTypeEnum {
return list;
}
/**
* Getter method for property <tt>code</tt>.
*
* @return property value of code
*/
public String getCode() {
return code;
}
/**
* Setter method for property <tt>code </tt>.
*
* @param code value to be assigned to property code
*/
public void setCode(String code) {
this.code = code;
}
/**
* Getter method for property <tt>desc</tt>.
*
* @return property value of desc
*/
public String getDesc() {
return desc;
}
/**
* Setter method for property <tt>desc </tt>.
*
* @param desc value to be assigned to property desc
*/
public void setDesc(String desc) {
this.desc = desc;
}
public static boolean contains(String clusterType) {
for (ClusterTypeEnum clusterTypeEnum : ClusterTypeEnum.values()) {

View File

@ -3,10 +3,13 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
/**
* @author NacosSync
* @version $Id: MetricsStatisticsType.java, v 0.1 2019年02月28日 下午2:17 NacosSync Exp $
*/
@Getter
public enum MetricsStatisticsType {
CACHE_SIZE("nacosSync.finished.taskMap.cacheSize", "任务执行完成缓存列表数"),
@ -28,15 +31,10 @@ public enum MetricsStatisticsType {
/**
* metricsName
*/
private String metricsName;
private String desc;
private final String metricsName;
MetricsStatisticsType(String code, String desc) {
this.metricsName = code;
this.desc = desc;
}
public String getMetricsName() {
return metricsName;
}
}

View File

@ -16,46 +16,23 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
/**
* @author NacosSync
* @version $Id: ResultCodeEnum.java, v 0.1 2018-09-25 PM4:38 NacosSync Exp $$
*/
@Getter
public enum ResultCodeEnum {
SUCCESS("SUCCESS", "请求成功", "请求成功"),
SYSTEM_ERROR("SYSTEM_ERROR", "系统异常", "系统异常");
private String code;
private String errorMessage;
private String detail;
private final String code;
ResultCodeEnum(String code, String errorMessage, String detail) {
this.code = code;
this.errorMessage = errorMessage;
this.detail = detail;
}
public String getCode() {
return code;
}
public void setCode(String code) {
this.code = code;
}
public String getErrorMessage() {
return errorMessage;
}
public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}
public String getDetail() {
return detail;
}
public void setDetail(String detail) {
this.detail = detail;
}
}

View File

@ -24,11 +24,18 @@ public class SkyWalkerConstants {
public static final String UNDERLINE = "_";
public static final String DEST_CLUSTERID_KEY = "destClusterId";
public static final String DEST_CLUSTER_ID_KEY = "destClusterId";
public static final String GROUP_NAME = "groupName";
public static final String SYNC_SOURCE_KEY = "syncSource";
public static final String SOURCE_CLUSTERID_KEY = "sourceClusterId";
public static final String SOURCE_CLUSTER_ID_KEY = "sourceClusterId";
public static final String MANAGEMENT_PORT_KEY="management.port";
public static final String MANAGEMENT_CONTEXT_PATH_KEY="management.context-path";
public static final String SERVICE_NAME_PARAM="serviceNameParam";
public static final String GROUP_NAME_PARAM="groupNameParam";
public static final String PAGE_NO="pageNo";
public static final String PAGE_SIZE="pageSize";
public static final String SYNC_INSTANCE_TAG="sync.instance.tag";
public static final String NACOS_ALL_SERVICE_NAME = "ALL";
}

View File

@ -16,6 +16,8 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
/**
* @author NacosSync
* @version $Id: TaskStatusEnum.java, v 0.1 2018-09-26 上午2:38 NacosSync Exp $$
@ -30,51 +32,18 @@ public enum TaskStatusEnum {
* delete the task
*/
DELETE("DELETE", "任务需要被删除");
private String code;
private String desc;
@Getter
private final String code;
private final String desc;
TaskStatusEnum(String code, String desc) {
this.code = code;
this.desc = desc;
}
/**
* Getter method for property <tt>code</tt>.
*
* @return property value of code
*/
public String getCode() {
return code;
}
/**
* Setter method for property <tt>code </tt>.
*
* @param code value to be assigned to property code
*/
public void setCode(String code) {
this.code = code;
}
/**
* Getter method for property <tt>desc</tt>.
*
* @return property value of desc
*/
public String getDesc() {
return desc;
}
/**
* Setter method for property <tt>desc </tt>.
*
* @param desc value to be assigned to property desc
*/
public void setDesc(String desc) {
this.desc = desc;
}
public static boolean contains(String code) {
for (TaskStatusEnum taskStatusEnum : TaskStatusEnum.values()) {

View File

@ -16,10 +16,10 @@
*/
package com.alibaba.nacossync.dao;
import com.alibaba.nacossync.dao.repository.ClusterRepository;
import com.alibaba.nacossync.pojo.QueryCondition;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
@ -27,9 +27,6 @@ import org.springframework.data.domain.Sort;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.repository.ClusterRepository;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.Predicate;
import javax.persistence.criteria.Root;
@ -44,9 +41,12 @@ import java.util.List;
@Slf4j
public class ClusterAccessService implements PageQueryService<ClusterDO> {
@Autowired
private ClusterRepository clusterRepository;
private final ClusterRepository clusterRepository;
public ClusterAccessService(ClusterRepository clusterRepository) {
this.clusterRepository = clusterRepository;
}
public ClusterDO insert(ClusterDO clusterDO) {
return clusterRepository.save(clusterDO);
@ -103,4 +103,12 @@ public class ClusterAccessService implements PageQueryService<ClusterDO> {
predicates.add(criteriaBuilder.like(root.get("clusterName"), "%" + queryCondition.getServiceName() + "%"));
return predicates;
}
public int findClusterLevel(String sourceClusterId){
ClusterDO clusterDO = clusterRepository.findByClusterId(sourceClusterId);
if (clusterDO != null) {
return clusterDO.getClusterLevel();
}
return -1;
}
}

View File

@ -14,10 +14,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.dao;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.dao.repository.TaskRepository;
import com.alibaba.nacossync.pojo.QueryCondition;
import org.springframework.beans.factory.annotation.Autowired;
import com.alibaba.nacossync.pojo.model.TaskDO;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
@ -25,9 +28,6 @@ import org.springframework.data.domain.Sort;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.repository.TaskRepository;
import com.alibaba.nacossync.pojo.model.TaskDO;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.Predicate;
import javax.persistence.criteria.Root;
@ -40,79 +40,94 @@ import java.util.List;
*/
@Service
public class TaskAccessService implements PageQueryService<TaskDO> {
@Autowired
private TaskRepository taskRepository;
private final TaskRepository taskRepository;
public TaskAccessService(TaskRepository taskRepository) {
this.taskRepository = taskRepository;
}
public TaskDO findByTaskId(String taskId) {
return taskRepository.findByTaskId(taskId);
}
public void deleteTaskById(String taskId) {
taskRepository.deleteByTaskId(taskId);
}
/**
* batch delete tasks by taskIds
* @author yongchao9
*
* @param taskIds
* @author yongchao9
*/
public void deleteTaskInBatch(List<String> taskIds) {
List<TaskDO> tds=taskRepository.findAllByTaskIdIn(taskIds);
taskRepository.deleteInBatch(tds);
List<TaskDO> tds = taskRepository.findAllByTaskIdIn(taskIds);
taskRepository.deleteAllInBatch(tds);
}
public Iterable<TaskDO> findAll() {
return taskRepository.findAll();
}
public void addTask(TaskDO taskDO) {
taskRepository.save(taskDO);
}
public int countByDestClusterIdOrSourceClusterId(String destClusterId, String sourceClusterId) {
return taskRepository.countByDestClusterIdOrSourceClusterId(destClusterId, sourceClusterId);
}
private Predicate getPredicate(CriteriaBuilder criteriaBuilder, List<Predicate> predicates) {
Predicate[] p = new Predicate[predicates.size()];
return criteriaBuilder.and(predicates.toArray(p));
}
private List<Predicate> getPredicates(Root<TaskDO> root, CriteriaBuilder criteriaBuilder, QueryCondition queryCondition) {
private List<Predicate> getPredicates(Root<TaskDO> root, CriteriaBuilder criteriaBuilder,
QueryCondition queryCondition) {
List<Predicate> predicates = new ArrayList<>();
predicates.add(criteriaBuilder.like(root.get("serviceName"), "%" + queryCondition.getServiceName() + "%"));
return predicates;
}
@Override
public Page<TaskDO> findPageNoCriteria(Integer pageNum, Integer size) {
Pageable pageable = PageRequest.of(pageNum, size, Sort.Direction.DESC, "id");
return taskRepository.findAll(pageable);
}
@Override
public Page<TaskDO> findPageCriteria(Integer pageNum, Integer size, QueryCondition queryCondition) {
Pageable pageable = PageRequest.of(pageNum, size, Sort.Direction.DESC, "id");
return getTaskDOS(queryCondition, pageable);
}
private Page<TaskDO> getTaskDOS(QueryCondition queryCondition, Pageable pageable) {
return taskRepository.findAll(
(Specification<TaskDO>) (root, criteriaQuery, criteriaBuilder) -> {
List<Predicate> predicates = getPredicates(root,
criteriaBuilder, queryCondition);
return getPredicate(criteriaBuilder, predicates);
}, pageable);
return taskRepository.findAll((Specification<TaskDO>) (root, criteriaQuery, criteriaBuilder) -> {
List<Predicate> predicates = getPredicates(root, criteriaBuilder, queryCondition);
return getPredicate(criteriaBuilder, predicates);
}, pageable);
}
public List<TaskDO> findAllByServiceNameEqualAll() {
return taskRepository.findAllByServiceNameEqualsIgnoreCase(SkyWalkerConstants.NACOS_ALL_SERVICE_NAME);
}
public List<TaskDO> findAllByServiceNameNotEqualAll() {
return taskRepository.findAllByServiceNameNotIgnoreCase(SkyWalkerConstants.NACOS_ALL_SERVICE_NAME);
}
}

View File

@ -16,13 +16,13 @@
*/
package com.alibaba.nacossync.dao.repository;
import javax.transaction.Transactional;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;
import org.springframework.data.repository.CrudRepository;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import javax.transaction.Transactional;
/**
* @author NacosSync
@ -34,6 +34,6 @@ public interface ClusterRepository extends CrudRepository<ClusterDO, Integer>, J
ClusterDO findByClusterId(String clusterId);
@Transactional
int deleteByClusterId(String clusterId);
void deleteByClusterId(String clusterId);
}

View File

@ -16,15 +16,13 @@
*/
package com.alibaba.nacossync.dao.repository;
import java.util.List;
import javax.transaction.Transactional;
import com.alibaba.nacossync.pojo.model.TaskDO;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;
import org.springframework.data.repository.CrudRepository;
import com.alibaba.nacossync.pojo.model.TaskDO;
import javax.transaction.Transactional;
import java.util.List;
/**
* @author NacosSync
@ -36,10 +34,15 @@ public interface TaskRepository extends CrudRepository<TaskDO, Integer>, JpaRepo
TaskDO findByTaskId(String taskId);
@Transactional
int deleteByTaskId(String taskId);
void deleteByTaskId(String taskId);
List<TaskDO> findAllByTaskIdIn(List<String> taskIds);
/**
* query service is alluse ns leven sync data
*/
List<TaskDO> findAllByServiceNameEqualsIgnoreCase(String serviceName);
List<TaskDO> findAllByServiceNameNotIgnoreCase(String serviceName);
List<TaskDO> getAllByWorkerIp(String workerIp);
int countByDestClusterIdOrSourceClusterId(String destClusterId,String sourceClusterId);
}

View File

@ -0,0 +1,13 @@
package com.alibaba.nacossync.event;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.Data;
@Data
public class DeleteAllSubTaskEvent {
public DeleteAllSubTaskEvent(TaskDO taskDO) {
this.taskDO = taskDO;
}
private final TaskDO taskDO;
}

View File

@ -14,23 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.event.listener;
import javax.annotation.PostConstruct;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.monitor.MetricsManager;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.event.SyncTaskEvent;
import com.alibaba.nacossync.extension.SyncManagerService;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.google.common.base.Stopwatch;
import com.google.common.eventbus.EventBus;
import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
/**
* @author NacosSync
@ -39,56 +38,59 @@ import com.google.common.eventbus.Subscribe;
@Slf4j
@Service
public class EventListener {
@Autowired
private MetricsManager metricsManager;
@Autowired
private SyncManagerService syncManagerService;
@Autowired
private EventBus eventBus;
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
private final MetricsManager metricsManager;
private final SyncManagerService syncManagerService;
private final EventBus eventBus;
private final SkyWalkerCacheServices skyWalkerCacheServices;
public EventListener(MetricsManager metricsManager, SyncManagerService syncManagerService, EventBus eventBus,
SkyWalkerCacheServices skyWalkerCacheServices) {
this.metricsManager = metricsManager;
this.syncManagerService = syncManagerService;
this.eventBus = eventBus;
this.skyWalkerCacheServices = skyWalkerCacheServices;
}
@PostConstruct
public void register() {
eventBus.register(this);
}
@Subscribe
public void listenerSyncTaskEvent(SyncTaskEvent syncTaskEvent) {
public void sync(SyncTaskEvent syncTaskEvent) {
try {
long start = System.currentTimeMillis();
if (syncManagerService.sync(syncTaskEvent.getTaskDO())) {
Stopwatch stopwatch = Stopwatch.createStarted();
if (syncManagerService.sync(syncTaskEvent.getTaskDO(), null)) {
skyWalkerCacheServices.addFinishedTask(syncTaskEvent.getTaskDO());
metricsManager.record(MetricsStatisticsType.SYNC_TASK_RT, System.currentTimeMillis() - start);
metricsManager.record(MetricsStatisticsType.SYNC_TASK_RT, stopwatch.elapsed().toMillis());
} else {
log.warn("listenerSyncTaskEvent sync failure");
}
log.warn("syncTaskEvent process error");
}
} catch (Exception e) {
log.warn("listenerSyncTaskEvent process error", e);
log.warn("syncTaskEvent process error", e);
}
}
@Subscribe
public void listenerDeleteTaskEvent(DeleteTaskEvent deleteTaskEvent) {
public void delete(DeleteTaskEvent deleteTaskEvent) {
try {
long start = System.currentTimeMillis();
Stopwatch stopwatch = Stopwatch.createStarted();
if (syncManagerService.delete(deleteTaskEvent.getTaskDO())) {
skyWalkerCacheServices.addFinishedTask(deleteTaskEvent.getTaskDO());
metricsManager.record(MetricsStatisticsType.DELETE_TASK_RT, System.currentTimeMillis() - start);
skyWalkerCacheServices.removeFinishedTask(deleteTaskEvent.getTaskDO().getOperationId());
metricsManager.record(MetricsStatisticsType.DELETE_TASK_RT, stopwatch.elapsed().toMillis());
} else {
log.warn("listenerDeleteTaskEvent delete failure");
}
log.warn("deleteTaskEvent delete failure");
}
} catch (Exception e) {
log.warn("listenerDeleteTaskEvent process error", e);
log.warn("deleteTaskEvent delete failure.", e);
}
}
}

View File

@ -12,14 +12,12 @@
*/
package com.alibaba.nacossync.extension;
import static com.alibaba.nacossync.util.SkyWalkerUtil.generateSyncKey;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.pojo.model.TaskDO;
import java.util.concurrent.ConcurrentHashMap;
import com.alibaba.nacossync.util.StringUtils;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.InitializingBean;
@ -27,6 +25,10 @@ import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.stereotype.Service;
import java.util.concurrent.ConcurrentHashMap;
import static com.alibaba.nacossync.util.SkyWalkerUtil.generateSyncKey;
/**
* @author NacosSync
* @version $Id: SyncManagerService.java, v 0.1 2018-09-25 PM5:17 NacosSync Exp $$
@ -37,7 +39,7 @@ public class SyncManagerService implements InitializingBean, ApplicationContextA
protected final SkyWalkerCacheServices skyWalkerCacheServices;
private ConcurrentHashMap<String, SyncService> syncServiceMap = new ConcurrentHashMap<String, SyncService>();
private final ConcurrentHashMap<String, SyncService> syncServiceMap = new ConcurrentHashMap<String, SyncService>();
private ApplicationContext applicationContext;
@ -52,19 +54,19 @@ public class SyncManagerService implements InitializingBean, ApplicationContextA
}
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
return getSyncService(taskDO.getSourceClusterId(), taskDO.getDestClusterId()).sync(taskDO);
return getSyncService(taskDO.getSourceClusterId(), taskDO.getDestClusterId()).sync(taskDO, index);
}
@Override
public void afterPropertiesSet() {
this.applicationContext.getBeansOfType(SyncService.class).forEach((key, value) -> {
this.applicationContext.getBeansWithAnnotation(NacosSyncService.class).forEach((key, value) -> {
NacosSyncService nacosSyncService = value.getClass().getAnnotation(NacosSyncService.class);
ClusterTypeEnum sourceCluster = nacosSyncService.sourceCluster();
ClusterTypeEnum destinationCluster = nacosSyncService.destinationCluster();
syncServiceMap.put(generateSyncKey(sourceCluster, destinationCluster), value);
syncServiceMap.put(generateSyncKey(sourceCluster, destinationCluster), (SyncService) value);
});
}
@ -74,7 +76,9 @@ public class SyncManagerService implements InitializingBean, ApplicationContextA
}
public SyncService getSyncService(String sourceClusterId, String destClusterId) {
if (StringUtils.isEmpty(sourceClusterId) || StringUtils.isEmpty(destClusterId)) {
throw new IllegalArgumentException("Source cluster id and destination cluster id must not be null or empty");
}
ClusterTypeEnum sourceClusterType = this.skyWalkerCacheServices.getClusterType(sourceClusterId);
ClusterTypeEnum destClusterType = this.skyWalkerCacheServices.getClusterType(destClusterId);

View File

@ -14,9 +14,10 @@ package com.alibaba.nacossync.extension;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.pojo.model.TaskDO;
import java.util.Map;
import org.apache.commons.lang3.StringUtils;
import java.util.Map;
/**
* @author NacosSync
* @version $Id: SyncManagerService.java, v 0.1 2018-09-25 下午5:17 NacosSync Exp $$
@ -30,20 +31,22 @@ public interface SyncService {
* @return
*/
boolean delete(TaskDO taskDO);
/**
* execute sync
*
* @param taskDO
* @param index
* @return
*/
boolean sync(TaskDO taskDO);
boolean sync(TaskDO taskDO, Integer index);
/**
* Determines that the current instance data is from another source cluster
*/
default boolean needSync(Map<String, String> sourceMetaData) {
return StringUtils.isBlank(sourceMetaData.get(SkyWalkerConstants.SOURCE_CLUSTERID_KEY));
boolean syncTag = StringUtils.isBlank(sourceMetaData.get(SkyWalkerConstants.SYNC_INSTANCE_TAG));
boolean blank = StringUtils.isBlank(sourceMetaData.get(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY));
return syncTag && blank;
}
/**
@ -51,7 +54,7 @@ public interface SyncService {
* cluster ID of the task
*/
default boolean needDelete(Map<String, String> destMetaData, TaskDO taskDO) {
return StringUtils.equals(destMetaData.get(SkyWalkerConstants.SOURCE_CLUSTERID_KEY),
return StringUtils.equals(destMetaData.get(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY),
taskDO.getSourceClusterId());
}

View File

@ -1,15 +0,0 @@
package com.alibaba.nacossync.extension.client;
import lombok.Data;
@Data
public class InstanceQueryModel {
private String sourceClusterId;
private String destClusterId;
private String serviceName;
private String groupName;
private String version;
private int pageNo;
private int pageSize;
}

View File

@ -1,11 +0,0 @@
package com.alibaba.nacossync.extension.client;
import com.alibaba.nacossync.pojo.view.TaskModel;
import java.util.List;
public interface SyncQueryClient {
List<TaskModel> getAllInstance(InstanceQueryModel instanceQueryModel);
}

View File

@ -1,54 +0,0 @@
package com.alibaba.nacossync.extension.client.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.ListView;
import com.alibaba.nacossync.extension.client.InstanceQueryModel;
import com.alibaba.nacossync.extension.client.SyncQueryClient;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.pojo.view.TaskModel;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
@Service
@Slf4j
public class NacosSyncQueryClientImpl implements SyncQueryClient {
private final NacosServerHolder nacosServerHolder;
public NacosSyncQueryClientImpl(
NacosServerHolder nacosServerHolder) {
this.nacosServerHolder = nacosServerHolder;
}
@Override
public List<TaskModel> getAllInstance(InstanceQueryModel instanceQueryModel) {
NamingService namingService = nacosServerHolder
.get(instanceQueryModel.getSourceClusterId());
try {
ListView<String> servicesOfServer = namingService
.getServicesOfServer(instanceQueryModel.getPageNo(),
instanceQueryModel.getPageSize());
return servicesOfServer.getData().stream()
.map(serviceName -> buildTaskModel(instanceQueryModel, serviceName))
.collect(Collectors.toList());
} catch (NacosException e) {
log.error("When using nacos client failure query tasks", e);
return Collections.emptyList();
}
}
private TaskModel buildTaskModel(InstanceQueryModel instanceQueryModel, String serviceName) {
TaskModel taskModel = new TaskModel();
taskModel.setServiceName(serviceName);
taskModel.setSourceClusterId(instanceQueryModel.getSourceClusterId());
taskModel.setDestClusterId(instanceQueryModel.getDestClusterId());
return taskModel;
}
}

View File

@ -29,11 +29,11 @@ import java.util.concurrent.TimeUnit;
*/
@Slf4j
public class EurekaBeatReactor {
private ScheduledExecutorService executorService;
private final ScheduledExecutorService executorService;
private volatile long clientBeatInterval = 5 * 1000;
private static final long CLIENT_BEAT_INTERVAL = 5 * 1000;
private final Map<String, InstanceInfo> eurekaBeat = new ConcurrentHashMap<>();
private EurekaHttpClient eurekaHttpClient;
private final EurekaHttpClient eurekaHttpClient;
public EurekaBeatReactor(EurekaHttpClient eurekaHttpClient) {
this.eurekaHttpClient = eurekaHttpClient;
@ -71,7 +71,7 @@ public class EurekaBeatReactor {
} catch (Exception e) {
log.error("[CLIENT-BEAT] Exception while scheduling beat.", e);
} finally {
executorService.schedule(this, clientBeatInterval, TimeUnit.MILLISECONDS);
executorService.schedule(this, CLIENT_BEAT_INTERVAL, TimeUnit.MILLISECONDS);
}
}
}

View File

@ -26,8 +26,8 @@ import java.util.Objects;
* @date 2019-06-26
*/
public class EurekaNamingService {
private EurekaHttpClient eurekaHttpClient;
private EurekaBeatReactor beatReactor;
private final EurekaHttpClient eurekaHttpClient;
private final EurekaBeatReactor beatReactor;
public EurekaNamingService(EurekaHttpClient eurekaHttpClient) {

View File

@ -25,7 +25,7 @@ import java.util.function.Consumer;
*/
@Service
public class SpecialSyncEventBus {
private ConcurrentHashMap<String, SpecialSyncEvent> specialSyncEventRegistry = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, SpecialSyncEvent> specialSyncEventRegistry = new ConcurrentHashMap<>();
public void subscribe(TaskDO taskDO, Consumer<TaskDO> syncAction) {
SpecialSyncEvent specialSyncEvent = new SpecialSyncEvent();

View File

@ -21,7 +21,6 @@ import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.eventbus.EventBus;
import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
@ -34,9 +33,12 @@ import java.util.function.Consumer;
@Service
@Slf4j
public class SpecialSyncEventListener {
@Autowired
private EventBus eventBus;
private final EventBus eventBus;
public SpecialSyncEventListener(EventBus eventBus) {
this.eventBus = eventBus;
}
@PostConstruct
public void init() {
eventBus.register(this);

View File

@ -13,20 +13,23 @@
package com.alibaba.nacossync.extension.holder;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Supplier;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
/**
* @author paderlol
* @date 2018-12-24 22:08
*/
@Slf4j
public abstract class AbstractServerHolderImpl<T> implements Holder {
public abstract class AbstractServerHolderImpl<T> implements Holder<T> {
private final Map<String, T> serviceMap = new ConcurrentHashMap<>();
@Autowired
protected SkyWalkerCacheServices skyWalkerCacheServices;

View File

@ -10,6 +10,7 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.holder;
import com.alibaba.nacossync.extension.eureka.EurekaNamingService;
@ -30,12 +31,27 @@ import java.util.function.Supplier;
@Service
@Slf4j
public class EurekaServerHolder extends AbstractServerHolderImpl<EurekaNamingService> {
private static final String HTTP_PREFIX = "http://";
private static final String HTTPS_PREFIX = "https://";
@Override
EurekaNamingService createServer(String clusterId, Supplier<String> serverAddressSupplier) throws Exception {
RestTemplateTransportClientFactory restTemplateTransportClientFactory =
new RestTemplateTransportClientFactory();
EurekaEndpoint eurekaEndpoint = new DefaultEndpoint(serverAddressSupplier.get());
RestTemplateTransportClientFactory restTemplateTransportClientFactory = new RestTemplateTransportClientFactory();
EurekaEndpoint eurekaEndpoint = new DefaultEndpoint(addHttpPrefix(serverAddressSupplier.get()));
EurekaHttpClient eurekaHttpClient = restTemplateTransportClientFactory.newClient(eurekaEndpoint);
return new EurekaNamingService(eurekaHttpClient);
}
public String addHttpPrefix(String input) {
if (input == null || input.isEmpty()) {
return input;
}
if (!input.startsWith(HTTP_PREFIX) && !input.startsWith(HTTPS_PREFIX)) {
input = HTTP_PREFIX + input;
}
return input;
}
}

View File

@ -18,13 +18,14 @@ import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.google.common.base.Joiner;
import lombok.extern.slf4j.Slf4j;
import org.apache.logging.log4j.util.Strings;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import java.util.function.Supplier;
import lombok.extern.slf4j.Slf4j;
import org.apache.logging.log4j.util.Strings;
import org.springframework.stereotype.Service;
/**
* @author paderlol
@ -35,7 +36,7 @@ import org.springframework.stereotype.Service;
public class NacosServerHolder extends AbstractServerHolderImpl<NamingService> {
private final ClusterAccessService clusterAccessService;
public NacosServerHolder(ClusterAccessService clusterAccessService) {
this.clusterAccessService = clusterAccessService;
}
@ -60,4 +61,5 @@ public class NacosServerHolder extends AbstractServerHolderImpl<NamingService> {
);
return NamingFactory.createNamingService(properties);
}
}

View File

@ -0,0 +1,265 @@
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacos.common.utils.CollectionUtils;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import com.google.common.base.Stopwatch;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
@Slf4j
public abstract class AbstractNacosSync implements SyncService {
private final Map<String, EventListener> listenerMap = new ConcurrentHashMap<>();
private final Map<String, Set<String>> sourceInstanceSnapshot = new ConcurrentHashMap<>();
private final Map<String, Integer> syncTaskTap = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, TaskDO> allSyncTaskMap = new ConcurrentHashMap<>();
private ScheduledExecutorService executorService;
@Autowired
private MetricsManager metricsManager;
@Getter
@Autowired
private NacosServerHolder nacosServerHolder;
/**
* Due to network issues or other reasons, the Nacos Sync synchronization tasks may fail,
* resulting in the target cluster's registry missing synchronized instances.
* To prevent the target cluster's registry from missing synchronized instances for an extended period,
* a fallback worker thread is started every 5 minutes to execute all synchronization tasks.
*/
@PostConstruct
public void afterPropertiesSet() {
initializeExecutorService();
scheduleSyncTasks();
}
@PreDestroy
public void destroy() {
if (executorService != null && !executorService.isShutdown()) {
executorService.shutdown();
try {
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
}
private void initializeExecutorService() {
executorService = Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r);
t.setDaemon(true);
t.setName("com.alibaba.nacossync.basic.synctask");
return t;
});
}
private void scheduleSyncTasks() {
executorService.scheduleWithFixedDelay(this::executeSyncTasks, 0, 300, TimeUnit.SECONDS);
}
private void executeSyncTasks() {
if (allSyncTaskMap.isEmpty()) {
return;
}
Collection<TaskDO> taskCollections = allSyncTaskMap.values();
List<TaskDO> taskDOList = new ArrayList<>(taskCollections);
if (CollectionUtils.isNotEmpty(taskDOList)) {
BatchTaskExecutor.batchOperation(taskDOList, this::executeTask);
}
}
private void executeTask(TaskDO task) {
Stopwatch stopwatch = Stopwatch.createStarted();
String taskId = task.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(task.getSourceClusterId());
doSync(taskId, task, sourceNamingService);
} catch (NacosException e) {
log.error("sync task from nacos to nacos failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} finally {
stopwatch.stop();
log.debug("Task execution time for taskId {}: {} ms", taskId, stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}
@Override
public boolean delete(TaskDO taskDO) {
String taskId = taskDO.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
//移除订阅
EventListener listener = listenerMap.remove(taskId);
if (listener!= null) {
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()), listener);
}
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.remove(taskId));
sourceInstanceSnapshot.remove(taskId);
allSyncTaskMap.remove(taskId);
// 删除目标集群中同步的实例列表
deregisterInstance(taskDO);
}catch (NacosException e) {
log.error("Delete task from nacos to specify destination was failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO, Integer index) {
String taskId = taskDO.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
allSyncTaskMap.put(taskId, taskDO);
//防止暂停同步任务后,重新同步/或删除任务以后新建任务不会再接收到新的事件导致不能同步,所以每次订阅事件之前,先全量同步一次任务
doSync(taskId, taskDO, sourceNamingService);
this.listenerMap.putIfAbsent(taskId, event -> {
if (event instanceof NamingEvent) {
try {
doSync(taskId, taskDO, sourceNamingService);
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
});
sourceNamingService.subscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.get(taskId));
}catch (NacosException e) {
log.error("Nacos sync task process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
}
return true;
}
private void doSync(String taskId, TaskDO taskDO, NamingService sourceNamingService) throws Exception {
if (syncTaskTap.putIfAbsent(taskId, 1) != null) {
log.info("任务Id:{}上一个同步任务尚未结束", taskId);
return;
}
try {
// 直接从本地保存的serviceInfoMap中取订阅的服务实例
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
// 先删除不存在的
this.removeInvalidInstance(taskDO, sourceInstances);
// 同步实例
this.syncNewInstance(taskDO, sourceInstances);
} finally {
syncTaskTap.remove(taskId);
}
}
private void syncNewInstance(TaskDO taskDO, List<Instance> sourceInstances) throws NacosException {
Set<String> latestSyncInstance = new TreeSet<>();
//再次添加新实例
String taskId = taskDO.getTaskId();
Set<String> instanceKeys = sourceInstanceSnapshot.get(taskId);
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
String instanceKey = composeInstanceKey(instance.getIp(), instance.getPort());
if (CollectionUtils.isEmpty(instanceKeys) || !instanceKeys.contains(instanceKey)) {
register(taskDO, instance);
}
latestSyncInstance.add(instanceKey);
}
}
if (CollectionUtils.isNotEmpty(latestSyncInstance)) {
log.info("任务Id:{},已同步实例个数:{}", taskId, latestSyncInstance.size());
sourceInstanceSnapshot.put(taskId, latestSyncInstance);
} else {
// latestSyncInstance为空表示源集群中需要同步的所有实例即非nacos-sync同步过来的实例已经下线,清除本地持有快照
sourceInstanceSnapshot.remove(taskId);
}
}
private void removeInvalidInstance(TaskDO taskDO, List<Instance> sourceInstances) throws Exception {
String taskId = taskDO.getTaskId();
if (this.sourceInstanceSnapshot.containsKey(taskId)) {
Set<String> oldInstanceKeys = this.sourceInstanceSnapshot.get(taskId);
Set<String> newInstanceKeys = sourceInstances.stream()
.map(instance -> composeInstanceKey(instance.getIp(), instance.getPort()))
.collect(Collectors.toSet());
oldInstanceKeys.removeAll(newInstanceKeys);
if (CollectionUtils.isNotEmpty(oldInstanceKeys)) {
log.info("任务Id:{},移除无效同步实例:{}", taskId, oldInstanceKeys);
removeInvalidInstance(taskDO, oldInstanceKeys);
}
}
}
@Override
public boolean needDelete(Map<String, String> destMetaData, TaskDO taskDO) {
return SyncService.super.needDelete(destMetaData, taskDO);
}
@Override
public boolean needSync(Map<String, String> sourceMetaData) {
return SyncService.super.needSync(sourceMetaData);
}
public abstract String composeInstanceKey(String ip, int port);
public abstract void register(TaskDO taskDO, Instance instance);
public abstract void deregisterInstance(TaskDO taskDO) throws Exception;
public abstract void removeInvalidInstance(TaskDO taskDO, Set<String> invalidInstanceKeys) throws Exception;
}

View File

@ -32,13 +32,13 @@ import com.ecwid.consul.v1.ConsulClient;
import com.ecwid.consul.v1.QueryParams;
import com.ecwid.consul.v1.Response;
import com.ecwid.consul.v1.health.model.HealthService;
import lombok.extern.slf4j.Slf4j;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
/**
* Consul 同步 Nacos
@ -50,8 +50,7 @@ import org.springframework.beans.factory.annotation.Autowired;
@NacosSyncService(sourceCluster = ClusterTypeEnum.CONSUL, destinationCluster = ClusterTypeEnum.NACOS)
public class ConsulSyncToNacosServiceImpl implements SyncService {
@Autowired
private MetricsManager metricsManager;
private final MetricsManager metricsManager;
private final ConsulServerHolder consulServerHolder;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@ -60,14 +59,15 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
private final SpecialSyncEventBus specialSyncEventBus;
@Autowired
public ConsulSyncToNacosServiceImpl(ConsulServerHolder consulServerHolder,
SkyWalkerCacheServices skyWalkerCacheServices, NacosServerHolder nacosServerHolder,
SpecialSyncEventBus specialSyncEventBus) {
SpecialSyncEventBus specialSyncEventBus, MetricsManager metricsManager) {
this.consulServerHolder = consulServerHolder;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
this.specialSyncEventBus = specialSyncEventBus;
this.metricsManager = metricsManager;
}
@Override
@ -96,7 +96,7 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
try {
ConsulClient consulClient = consulServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
@ -106,7 +106,7 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
Set<String> instanceKeys = new HashSet<>();
overrideAllInstance(taskDO, destNamingService, healthServiceList, instanceKeys);
cleanAllOldInstance(taskDO, destNamingService, instanceKeys);
specialSyncEventBus.subscribe(taskDO, this::sync);
specialSyncEventBus.subscribe(taskDO, t->sync(t, index));
} catch (Exception e) {
log.error("Sync task from consul to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -146,10 +146,10 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
temp.setIp(instance.getService().getAddress());
temp.setPort(instance.getService().getPort());
Map<String, String> metaData = new HashMap<>(ConsulUtils.transferMetadata(instance.getService().getTags()));
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
temp.setMetadata(metaData);
return temp;
}

View File

@ -29,13 +29,14 @@ import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.NacosUtils;
import com.netflix.appinfo.InstanceInfo;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.CollectionUtils;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
* eureka
*
@ -87,7 +88,7 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO,Integer index) {
try {
EurekaNamingService eurekaNamingService = eurekaServerHolder.get(taskDO.getSourceClusterId());
@ -107,7 +108,7 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
}
addValidInstance(taskDO, destNamingService, eurekaInstances);
}
specialSyncEventBus.subscribe(taskDO, this::sync);
specialSyncEventBus.subscribe(taskDO, t->sync(t, index));
} catch (Exception e) {
log.error("sync task from eureka to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -182,10 +183,10 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
temp.setHealthy(true);
Map<String, String> metaData = new HashMap<>(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
temp.setMetadata(metaData);
return temp;
}

View File

@ -10,150 +10,97 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.holder.ConsulServerHolder;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.ConsulUtils;
import com.alibaba.nacossync.util.NacosUtils;
import com.ecwid.consul.v1.ConsulClient;
import com.ecwid.consul.v1.QueryParams;
import com.ecwid.consul.v1.Response;
import com.ecwid.consul.v1.agent.model.NewService;
import com.ecwid.consul.v1.health.model.HealthService;
import com.google.common.collect.Lists;
import lombok.extern.slf4j.Slf4j;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
/**
* @author zhanglong
*/
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.CONSUL)
public class NacosSyncToConsulServiceImpl implements SyncService {
private Map<String, EventListener> nacosListenerMap = new ConcurrentHashMap<>();
private final MetricsManager metricsManager;
public class NacosSyncToConsulServiceImpl extends AbstractNacosSync {
private static final String DELIMITER = "=";
private final SkyWalkerCacheServices skyWalkerCacheServices;
private final NacosServerHolder nacosServerHolder;
private final ConsulServerHolder consulServerHolder;
public NacosSyncToConsulServiceImpl(MetricsManager metricsManager, SkyWalkerCacheServices skyWalkerCacheServices,
NacosServerHolder nacosServerHolder, ConsulServerHolder consulServerHolder) {
this.metricsManager = metricsManager;
public NacosSyncToConsulServiceImpl(SkyWalkerCacheServices skyWalkerCacheServices,
ConsulServerHolder consulServerHolder) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
this.consulServerHolder = consulServerHolder;
}
@Override
public boolean delete(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
sourceNamingService.unsubscribe(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), nacosListenerMap.get(taskDO.getTaskId()));
// 删除目标集群中同步的实例列表
Response<List<HealthService>> serviceResponse =
consulClient.getHealthServices(taskDO.getServiceName(), true, QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)) {
consulClient.agentServiceDeregister(URLEncoder
.encode(healthService.getService().getId(), StandardCharsets.UTF_8.name()));
}
public String composeInstanceKey(String ip, int port) {
return String.join(":", ip, String.valueOf(port));
}
@Override
public void register(TaskDO taskDO, Instance instance) {
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
consulClient.agentServiceRegister(buildSyncInstance(instance, taskDO));
}
@Override
public void deregisterInstance(TaskDO taskDO) throws Exception {
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
Response<List<HealthService>> serviceResponse = consulClient.getHealthServices(taskDO.getServiceName(), true,
QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)) {
consulClient.agentServiceDeregister(
URLEncoder.encode(healthService.getService().getId(), StandardCharsets.UTF_8));
}
} catch (Exception e) {
log.error("delete a task from nacos to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
nacosListenerMap.putIfAbsent(taskDO.getTaskId(), event -> {
if (event instanceof NamingEvent) {
try {
Set<String> instanceKeySet = new HashSet<>();
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()));
// 先将新的注册一遍
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
consulClient.agentServiceRegister(buildSyncInstance(instance, taskDO));
instance.getInstanceId();
instanceKeySet.add(composeInstanceKey(instance.getIp(), instance.getPort()));
}
}
// 再将不存在的删掉
Response<List<HealthService>> serviceResponse =
consulClient.getHealthServices(taskDO.getServiceName(), true, QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)
&& !instanceKeySet.contains(composeInstanceKey(healthService.getService().getAddress(),
healthService.getService().getPort()))) {
consulClient.agentServiceDeregister(URLEncoder
.encode(healthService.getService().getId(), StandardCharsets.UTF_8.toString()));
}
}
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
});
sourceNamingService.subscribe(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), nacosListenerMap.get(taskDO.getTaskId()));
} catch (Exception e) {
log.error("sync task from nacos to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
public void removeInvalidInstance(TaskDO taskDO, Set<String> invalidInstanceKeys)
throws UnsupportedEncodingException {
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
Response<List<HealthService>> serviceResponse = consulClient.getHealthServices(taskDO.getServiceName(), true,
QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)
&& !invalidInstanceKeys.contains(composeInstanceKey(healthService.getService().getAddress(),
healthService.getService().getPort()))) {
consulClient.agentServiceDeregister(
URLEncoder.encode(healthService.getService().getId(), StandardCharsets.UTF_8));
}
}
return true;
}
private String composeInstanceKey(String ip, int port) {
return ip + ":" + port;
}
public NewService buildSyncInstance(Instance instance, TaskDO taskDO) {
NewService newService = new NewService();
newService.setAddress(instance.getIp());
@ -162,13 +109,13 @@ public class NacosSyncToConsulServiceImpl implements SyncService {
newService.setId(instance.getInstanceId());
List<String> tags = Lists.newArrayList();
tags.addAll(instance.getMetadata().entrySet().stream()
.map(entry -> String.join("=", entry.getKey(), entry.getValue())).collect(Collectors.toList()));
tags.add(String.join("=", SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId()));
tags.add(String.join("=", SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode()));
tags.add(String.join("=", SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId()));
.map(entry -> String.join(DELIMITER, entry.getKey(), entry.getValue())).collect(Collectors.toList()));
tags.add(String.join(DELIMITER, SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId()));
tags.add(String.join(DELIMITER, SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode()));
tags.add(String.join(DELIMITER, SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId()));
newService.setTags(tags);
return newService;
}
}

View File

@ -10,203 +10,114 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.Event;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacos.common.utils.CollectionUtils;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.eureka.EurekaNamingService;
import com.alibaba.nacossync.extension.holder.EurekaServerHolder;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.NacosUtils;
import com.netflix.appinfo.DataCenterInfo;
import com.netflix.appinfo.InstanceInfo;
import com.netflix.appinfo.LeaseInfo;
import com.netflix.appinfo.MyDataCenterInfo;
import lombok.extern.slf4j.Slf4j;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import lombok.extern.slf4j.Slf4j;
/**
* @author zhanglong
*/
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.EUREKA)
public class NacosSyncToEurekaServiceImpl implements SyncService {
private final Map<String, EventListener> nacosListenerMap = new ConcurrentHashMap<>();
private final MetricsManager metricsManager;
private final SkyWalkerCacheServices skyWalkerCacheServices;
private final NacosServerHolder nacosServerHolder;
public class NacosSyncToEurekaServiceImpl extends AbstractNacosSync {
private final EurekaServerHolder eurekaServerHolder;
public NacosSyncToEurekaServiceImpl(MetricsManager metricsManager, SkyWalkerCacheServices skyWalkerCacheServices,
NacosServerHolder nacosServerHolder, EurekaServerHolder eurekaServerHolder) {
this.metricsManager = metricsManager;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
private final SkyWalkerCacheServices skyWalkerCacheServices;
public NacosSyncToEurekaServiceImpl(EurekaServerHolder eurekaServerHolder,
SkyWalkerCacheServices skyWalkerCacheServices) {
this.eurekaServerHolder = eurekaServerHolder;
this.skyWalkerCacheServices = skyWalkerCacheServices;
}
@Override
public boolean delete(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
EurekaNamingService destNamingService =
eurekaServerHolder.get(taskDO.getDestClusterId());
sourceNamingService.unsubscribe(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), nacosListenerMap.get(taskDO.getTaskId()));
// 删除目标集群中同步的实例列表
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (allInstances != null) {
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance);
}
}
}
} catch (Exception e) {
log.error("delete task from nacos to eureka was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
public String composeInstanceKey(String ip, int port) {
return ip + ":" + port;
}
@Override
public boolean sync(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
EurekaNamingService destNamingService =
eurekaServerHolder.get(taskDO.getDestClusterId());
nacosListenerMap.putIfAbsent(taskDO.getTaskId(), event -> processNamingEvent(taskDO, sourceNamingService, destNamingService, event));
sourceNamingService.subscribe(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), nacosListenerMap.get(taskDO.getTaskId()));
} catch (Exception e) {
log.error("sync task from eureka to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
}
return true;
public void register(TaskDO taskDO, Instance instance) {
EurekaNamingService destNamingService = eurekaServerHolder.get(taskDO.getDestClusterId());
destNamingService.registerInstance(buildSyncInstance(instance, taskDO));
}
private void processNamingEvent(TaskDO taskDO, NamingService sourceNamingService,
EurekaNamingService destNamingService, Event event) {
if (event instanceof NamingEvent) {
try {
Set<String> instanceKeySet = new HashSet<>();
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()));
// 先将新的注册一遍
addAllNewInstance(taskDO, destNamingService, instanceKeySet, sourceInstances);
// 再将不存在的删掉
ifNecessaryDelete(taskDO, destNamingService, instanceKeySet);
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
}
private void ifNecessaryDelete(TaskDO taskDO, EurekaNamingService destNamingService, Set<String> instanceKeySet) {
@Override
public void deregisterInstance(TaskDO taskDO) {
EurekaNamingService destNamingService = eurekaServerHolder.get(taskDO.getDestClusterId());
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (allInstances != null) {
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO) && !instanceKeySet.contains(
composeInstanceKey(instance.getIPAddr(),
instance.getPort()))) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance);
}
}
}
}
private void addAllNewInstance(TaskDO taskDO, EurekaNamingService destNamingService, Set<String> instanceKeySet,
List<Instance> sourceInstances) {
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
destNamingService.registerInstance(buildSyncInstance(instance, taskDO));
instanceKeySet.add(composeInstanceKey(instance.getIp(), instance.getPort()));
@Override
public void removeInvalidInstance(TaskDO taskDO, Set<String> invalidInstanceKeys) {
EurekaNamingService destNamingService = eurekaServerHolder.get(taskDO.getDestClusterId());
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (CollectionUtils.isNotEmpty(allInstances)) {
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO) && invalidInstanceKeys.contains(
composeInstanceKey(instance.getIPAddr(), instance.getPort()))) {
destNamingService.deregisterInstance(instance);
}
}
}
}
private String composeInstanceKey(String ip, int port) {
return ip + ":" + port;
}
private InstanceInfo buildSyncInstance(Instance instance, TaskDO taskDO) {
DataCenterInfo dataCenterInfo = new MyDataCenterInfo(DataCenterInfo.Name.MyOwn);
final Map<String, String> instanceMetadata = instance.getMetadata();
HashMap<String, String> metadata = new HashMap<>(16);
metadata.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metadata.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metadata.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
metadata.putAll(instanceMetadata);
String homePageUrl = obtainHomePageUrl(instance, instanceMetadata);
String serviceName = taskDO.getServiceName();
return new InstanceInfo(
instance.getIp() + ":" + serviceName + ":" + instance.getPort(),
serviceName,
null,
instance.getIp(),
null,
new InstanceInfo.PortWrapper(true, instance.getPort()),
null,
homePageUrl + "/actuator/env",
homePageUrl + "/actuator/info",
homePageUrl + "/actuator/health",
null,
serviceName,
serviceName,
1,
dataCenterInfo,
instance.getIp(),
InstanceInfo.InstanceStatus.UP,
InstanceInfo.InstanceStatus.UNKNOWN,
null,
new LeaseInfo(30, 90,
0L, 0L, 0L, 0L, 0L),
false,
metadata,
System.currentTimeMillis(),
System.currentTimeMillis(),
null,
null
);
return new InstanceInfo(instance.getIp() + ":" + serviceName + ":" + instance.getPort(), serviceName, null,
instance.getIp(), null, new InstanceInfo.PortWrapper(true, instance.getPort()), null,
homePageUrl + "/actuator/env", homePageUrl + "/actuator/info", homePageUrl + "/actuator/health", null,
serviceName, serviceName, 1, dataCenterInfo, instance.getIp(), InstanceInfo.InstanceStatus.UP,
InstanceInfo.InstanceStatus.UNKNOWN, null, new LeaseInfo(30, 90, 0L, 0L, 0L, 0L, 0L), false, metadata,
System.currentTimeMillis(), System.currentTimeMillis(), null, null);
}
private String obtainHomePageUrl(Instance instance, Map<String, String> instanceMetadata) {
final String managementContextPath =
obtainManagementContextPath(instanceMetadata);
final String managementContextPath = obtainManagementContextPath(instanceMetadata);
final String managementPort = instanceMetadata.getOrDefault(SkyWalkerConstants.MANAGEMENT_PORT_KEY,
String.valueOf(instance.getPort()));
String.valueOf(instance.getPort()));
return String.format("http://%s:%s%s", instance.getIp(), managementPort, managementContextPath);
}
private String obtainManagementContextPath(Map<String, String> instanceMetadata) {
final String path = instanceMetadata.getOrDefault(SkyWalkerConstants.MANAGEMENT_CONTEXT_PATH_KEY, "");
if (path.endsWith("/")) {
@ -214,6 +125,6 @@ public class NacosSyncToEurekaServiceImpl implements SyncService {
}
return path;
}
}

View File

@ -10,9 +10,8 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
@ -24,27 +23,32 @@ import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.Collections;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import com.alibaba.nacossync.util.StringUtils;
import com.google.common.base.Stopwatch;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.InitializingBean;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import javax.annotation.PostConstruct;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
/**
* @author yangyshdan
@ -53,109 +57,165 @@ import org.springframework.beans.factory.annotation.Autowired;
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.NACOS)
public class NacosSyncToNacosServiceImpl implements SyncService {
private Map<String, EventListener> listenerMap = new ConcurrentHashMap<>();
private final Map<String, Set<String>> sourceInstanceSnapshot = new ConcurrentHashMap<>();
public class NacosSyncToNacosServiceImpl implements SyncService, InitializingBean, DisposableBean {
private final Map<String, EventListener> listenerMap = new ConcurrentHashMap<>();
private final Map<String, Integer> syncTaskTap = new ConcurrentHashMap<>();
@Autowired
private MetricsManager metricsManager;
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private NacosServerHolder nacosServerHolder;
private ConcurrentHashMap<String, TaskDO> allSyncTaskMap = new ConcurrentHashMap<String, TaskDO>();
private final ConcurrentHashMap<String, TaskDO> allSyncTaskMap = new ConcurrentHashMap<>();
private ScheduledExecutorService executorService;
private final MetricsManager metricsManager;
private final NacosServerHolder nacosServerHolder;
private final ClusterAccessService clusterAccessService;
private final SkyWalkerCacheServices skyWalkerCacheServices;
public NacosSyncToNacosServiceImpl(MetricsManager metricsManager, NacosServerHolder nacosServerHolder,
ClusterAccessService clusterAccessService, SkyWalkerCacheServices skyWalkerCacheServices) {
this.metricsManager = metricsManager;
this.nacosServerHolder = nacosServerHolder;
this.clusterAccessService = clusterAccessService;
this.skyWalkerCacheServices = skyWalkerCacheServices;
}
/**
* 因为网络故障等原因nacos sync的同步任务会失败导致目标集群注册中心缺少同步实例 为避免目标集群注册中心长时间缺少同步实例每隔5分钟启动一个兜底工作线程执行一遍全部的同步任务
* Due to network issues or other reasons, the Nacos Sync synchronization tasks may fail,
* resulting in the target cluster's registry missing synchronized instances.
* To prevent the target cluster's registry from missing synchronized instances for an extended period,
* a fallback worker thread is started every 5 minutes to execute all synchronization tasks.
*/
@PostConstruct
public void startBasicSyncTaskThread() {
ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor(r -> {
@Override
public void afterPropertiesSet() {
initializeExecutorService();
scheduleSyncTasks();
}
@Override
public void destroy() {
if (executorService != null && !executorService.isShutdown()) {
executorService.shutdown();
try {
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
}
private void initializeExecutorService() {
executorService = Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r);
t.setDaemon(true);
t.setName("com.alibaba.nacossync.basic.synctask");
return t;
});
executorService.scheduleWithFixedDelay(() -> {
if (allSyncTaskMap.size() == 0) {
return;
}
try {
for (TaskDO taskDO : allSyncTaskMap.values()) {
String taskId = taskDO.getTaskId();
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService =
nacosServerHolder.get(taskDO.getDestClusterId());
try {
doSync(taskId, taskDO, sourceNamingService, destNamingService);
} catch (Exception e) {
log.error("basic synctask process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
} catch (Throwable e) {
log.warn("basic synctask thread error", e);
}
}, 0, 300, TimeUnit.SECONDS);
}
private void scheduleSyncTasks() {
executorService.scheduleWithFixedDelay(this::executeSyncTasks, 0, 300, TimeUnit.SECONDS);
}
private void executeSyncTasks() {
if (allSyncTaskMap.isEmpty()) {
return;
}
Collection<TaskDO> taskCollections = allSyncTaskMap.values();
List<TaskDO> taskDOList = new ArrayList<>(taskCollections);
if (CollectionUtils.isNotEmpty(taskDOList)) {
BatchTaskExecutor.batchOperation(taskDOList, this::executeTask);
}
}
private void executeTask(TaskDO task) {
Stopwatch stopwatch = Stopwatch.createStarted();
String taskId = task.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(task.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(task.getDestClusterId());
doSync(taskId, task, sourceNamingService, destNamingService);
} catch (NacosException e) {
log.error("sync task from nacos to nacos failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} finally {
stopwatch.stop();
log.debug("Task execution time for taskId {}: {} ms", taskId, stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}
@Override
public boolean delete(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
int level = clusterAccessService.findClusterLevel(taskDO.getSourceClusterId());
String taskId = taskDO.getTaskId();
//Handle individual service
if (StringUtils.isEmpty(taskId)) {
log.warn("taskId is null data synchronization is not currently performed.{}", taskId);
return false;
}
EventListener listener = listenerMap.remove(taskId);
if (listener!= null) {
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()), listener);
}
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), false);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
//移除订阅
sourceNamingService
.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.remove(taskDO.getTaskId()));
sourceInstanceSnapshot.remove(taskDO.getTaskId());
allSyncTaskMap.remove(taskDO.getTaskId());
// 删除目标集群中同步的实例列表
List<Instance> sourceInstances = sourceNamingService
.getAllInstances(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
new ArrayList<>(), false);
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
destNamingService
.deregisterInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
instance.getIp(),
instance.getPort());
if (needSync(instance.getMetadata(), level, taskDO.getDestClusterId())) {
destNamingService.deregisterInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), instance);
}
}
// Remove all tasks that need to be synchronized.
allSyncTaskMap.remove(taskId);
} catch (Exception e) {
log.error("delete task from nacos to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
log.error("delete task from nacos to nacos was failed, operationalId:{}", taskDO.getOperationId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
log.info("Thread {} started synchronization at {}", Thread.currentThread().getId(), System.currentTimeMillis());
String taskId = taskDO.getTaskId();
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
allSyncTaskMap.put(taskId, taskDO);
//防止暂停同步任务后,重新同步/或删除任务以后新建任务不会再接收到新的事件导致不能同步,所以每次订阅事件之前,先全量同步一次任务
// To prevent issues where tasks paused for synchronization, newly created tasks after deletion,
// or resynchronization tasks do not receive new events and hence cannot synchronize,
// perform a full synchronization of tasks before subscribing to events each time.
Stopwatch stopwatch = Stopwatch.createStarted();
doSync(taskId, taskDO, sourceNamingService, destNamingService);
log.debug("Time taken to synchronize a service registration: {} ms",
stopwatch.elapsed(TimeUnit.MILLISECONDS));
this.listenerMap.putIfAbsent(taskId, event -> {
if (event instanceof NamingEvent) {
NamingEvent namingEvent = (NamingEvent) event;
log.info("Detected changes in service {} information, taskId: {}, number of instances: {}, initiating synchronization",
namingEvent.getServiceName(), taskId,
namingEvent.getInstances() == null ? null : namingEvent.getInstances().size());
try {
doSync(taskId, taskDO, sourceNamingService, destNamingService);
log.info("Detected synchronization end for service {}", namingEvent.getServiceName());
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -163,7 +223,7 @@ public class NacosSyncToNacosServiceImpl implements SyncService {
}
});
sourceNamingService.subscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.get(taskId));
listenerMap.get(taskId));
} catch (Exception e) {
log.error("sync task from nacos to nacos was failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -171,85 +231,192 @@ public class NacosSyncToNacosServiceImpl implements SyncService {
}
return true;
}
private void doSync(String taskId, TaskDO taskDO, NamingService sourceNamingService,
NamingService destNamingService) throws NacosException {
NamingService destNamingService) throws NacosException {
if (syncTaskTap.putIfAbsent(taskId, 1) != null) {
log.info("任务Id:{}上一个同步任务尚未结束", taskId);
log.info("Task ID:{} - the previous synchronization task has not finished yet", taskId);
return;
}
try {
// 直接从本地保存的serviceInfoMap中取订阅的服务实例
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
// 先删除不存在的
this.removeInvalidInstance(taskDO, destNamingService, sourceInstances);
// 如果同步实例已经为空代表该服务所有实例已经下线,清除本地持有快照
if (sourceInstances.isEmpty()) {
sourceInstanceSnapshot.remove(taskId);
return;
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
int level = clusterAccessService.findClusterLevel(taskDO.getSourceClusterId());
if (CollectionUtils.isNotEmpty(sourceInstances) && sourceInstances.get(0).isEphemeral()) {
// Handle batch data synchronization of ephemeral instances, need to get all current service instances.
// TODO: When the Client is version 1.x, execute the same synchronization method as persistent instances.
handlerPersistenceInstance(taskDO, destNamingService, sourceInstances, level);
} else if (CollectionUtils.isEmpty(sourceInstances)) {
// If the current source cluster is empty, then directly deregister the instances in the target cluster.
log.debug("service {} needs to sync ephemeral instance num is null: serviceName ", taskDO.getServiceName());
processDeRegisterInstances(taskDO, destNamingService);
} else {
// Handle batch data synchronization of persistent instances.
handlerPersistenceInstance(taskDO, destNamingService, sourceInstances, level);
}
// 同步实例
this.syncNewInstance(taskDO, destNamingService, sourceInstances);
} finally {
syncTaskTap.remove(taskId);
}
}
private void syncNewInstance(TaskDO taskDO, NamingService destNamingService,
List<Instance> sourceInstances) throws NacosException {
Set<String> latestSyncInstance = new TreeSet<>();
//再次添加新实例
String taskId = taskDO.getTaskId();
Set<String> instanceKeys = sourceInstanceSnapshot.get(taskId);
private void handlerPersistenceInstance(TaskDO taskDO, NamingService destNamingService,
List<Instance> sourceInstances, int level) throws NacosException {
List<Instance> needRegisterInstance = new ArrayList<>();
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
String instanceKey = composeInstanceKey(instance);
if (CollectionUtils.isEmpty(instanceKeys) || !instanceKeys.contains(instanceKey)) {
destNamingService.registerInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()),
buildSyncInstance(instance, taskDO));
}
latestSyncInstance.add(instanceKey);
if (needSync(instance.getMetadata(), level, taskDO.getDestClusterId())) {
needRegisterInstance.add(buildSyncInstance(instance, taskDO));
}
}
if (CollectionUtils.isNotEmpty(latestSyncInstance)) {
log.info("任务Id:{},已同步实例个数:{}", taskId, latestSyncInstance.size());
sourceInstanceSnapshot.put(taskId, latestSyncInstance);
List<Instance> destAllInstances = destNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
// Get the instances that the destination cluster has already synchronized
List<Instance> destHasSyncInstances = destAllInstances.stream()
.filter(instance -> hasSync(instance, taskDO.getSourceClusterId())).collect(Collectors.toList());
// The following two conversions are necessary because the Nacos Instance's equals method
// is flawed and cannot be used for direct comparison.
// The reason is that Instance's equals method compares Metadata using the toString method,
// and Metadata is of type HashMap, which does not guarantee order in its toString representation.
// Convert destHasSyncInstances to a Map with the concatenated string as key
Map<String, Instance> destInstanceMap = destHasSyncInstances.stream()
.collect(Collectors.toMap(NacosSyncToNacosServiceImpl::getInstanceKey, instance -> instance));
// Convert newInstances to a Map with the concatenated string as key
Map<String, Instance> needRegisterMap = needRegisterInstance.stream()
.collect(Collectors.toMap(NacosSyncToNacosServiceImpl::getInstanceKey, instance -> instance));
// Remove instances from newInstanceMap that are present in destInstanceMap
List<Instance> newInstances = removeSyncedInstances(destInstanceMap, needRegisterMap);
// Remove instances from destInstanceMap that are present in newInstanceMap
List<Instance> invalidInstances = getInvalidInstances(destInstanceMap, needRegisterMap);
// Register each instance one by one. Take one instance at a time.
for (Instance newInstance : newInstances) {
destNamingService.registerInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
newInstance);
}
if (CollectionUtils.isNotEmpty(invalidInstances)) {
log.info("taskId: {}, service {} deregistered, number of executions: {}", taskDO.getTaskId(), taskDO.getServiceName(),
destHasSyncInstances.size());
}
for (Instance instance : invalidInstances) {
destNamingService.deregisterInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
instance);
}
}
private void removeInvalidInstance(TaskDO taskDO, NamingService destNamingService,
List<Instance> sourceInstances) throws NacosException {
String taskId = taskDO.getTaskId();
if (this.sourceInstanceSnapshot.containsKey(taskId)) {
Set<String> oldInstanceKeys = this.sourceInstanceSnapshot.get(taskId);
List<String> newInstanceKeys = sourceInstances.stream().map(this::composeInstanceKey)
private List<Instance> getInvalidInstances(Map<String, Instance> destInstanceMap, Map<String, Instance> needRegisterMap) {
Map<String, Instance> destClone = new HashMap<>(destInstanceMap);
// Convert newInstances to a Map with the concatenated string as key
Map<String, Instance> needRegisterClone = new HashMap<>(needRegisterMap);
// Remove instances from newInstanceMap that are present in destInstanceMap
destClone.keySet().removeAll(needRegisterClone.keySet());
return new ArrayList<>(destClone.values());
}
public List<Instance> removeSyncedInstances(Map<String, Instance> destInstanceMap, Map<String, Instance> needRegisterMap) {
// Convert destHasSyncInstances to a Map with the concatenated string as key
Map<String, Instance> destClone = new HashMap<>(destInstanceMap);
// Convert newInstances to a Map with the concatenated string as key
Map<String, Instance> needRegisterClone = new HashMap<>(needRegisterMap);
// Remove instances from newInstanceMap that are present in destInstanceMap
needRegisterClone.keySet().removeAll(destClone.keySet());
return new ArrayList<>(needRegisterClone.values());
}
private boolean hasSync(Instance instance, String sourceClusterId) {
if (instance.getMetadata() != null) {
String sourceClusterKey = instance.getMetadata().get(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY);
return sourceClusterKey != null && sourceClusterKey.equals(sourceClusterId);
}
return false;
}
/**
* When the number of instances that the source cluster needs to synchronize is 0,
* if the target cluster still has instances synchronized with the source cluster,
* perform unregistration.
*
* @param destNamingService Destination cluster naming service
* @throws NacosException
*/
private void processDeRegisterInstances(TaskDO taskDO, NamingService destNamingService) throws NacosException {
// If the instances in sourceInstances are empty, it means the instances are offline or do not exist.
List<Instance> destInstances = destNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), false);
// If the instances in the target cluster are also empty, then no operation is needed.
if (CollectionUtils.isEmpty(destInstances)) {
return;
}
destInstances = filterInstancesForRemoval(destInstances, taskDO.getSourceClusterId());
if (CollectionUtils.isNotEmpty(destInstances)) {
// Deregister each instance one by one. Take one instance at a time.
// Need to handle redo, otherwise, it will be registered again.
for (Instance destInstance : destInstances) {
destNamingService.deregisterInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), destInstance);
}
}
}
private List<Instance> filterInstancesForRemoval(List<Instance> destInstances, String sourceClusterId) {
return destInstances.stream().filter(instance -> !instance.getMetadata().isEmpty())
.filter(instance -> needDeregister(instance.getMetadata().get(SOURCE_CLUSTER_ID_KEY), sourceClusterId))
.collect(Collectors.toList());
Collection<String> instanceKeys = Collections.subtract(oldInstanceKeys, newInstanceKeys);
for (String instanceKey : instanceKeys) {
log.info("任务Id:{},移除无效同步实例:{}", taskId, instanceKey);
String[] split = instanceKey.split(":", -1);
destNamingService
.deregisterInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), split[0],
Integer.parseInt(split[1]));
}
}
private boolean needDeregister(String destClusterId, String sourceClusterId) {
if (!StringUtils.isEmpty(destClusterId)) {
return destClusterId.equals(sourceClusterId);
}
return false;
}
private String composeInstanceKey(Instance instance) {
return instance.getIp() + ":" + instance.getPort();
private boolean needSync(Map<String, String> sourceMetaData, int level, String destClusterId) {
// Regular cluster (default)
if (level == 0) {
return SyncService.super.needSync(sourceMetaData);
}
// Central cluster, as long as the instance is not from the target cluster,
// it needs to be synchronized (extended functionality)
return !destClusterId.equals(sourceMetaData.get(SOURCE_CLUSTER_ID_KEY));
}
private Instance buildSyncInstance(Instance instance, TaskDO taskDO) {
Instance temp = getInstance(instance);
temp.addMetadata(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
temp.addMetadata(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
temp.addMetadata(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
//The flag is a synchronous instance
temp.addMetadata(SkyWalkerConstants.SYNC_INSTANCE_TAG,
taskDO.getSourceClusterId() + "@@" + taskDO.getVersion());
return temp;
}
private static Instance getInstance(Instance instance) {
Instance temp = new Instance();
temp.setInstanceId(instance.getInstanceId());
temp.setIp(instance.getIp());
temp.setPort(instance.getPort());
temp.setClusterName(instance.getClusterName());
@ -258,15 +425,19 @@ public class NacosSyncToNacosServiceImpl implements SyncService {
temp.setHealthy(instance.isHealthy());
temp.setWeight(instance.getWeight());
temp.setEphemeral(instance.isEphemeral());
Map<String, String> metaData = new HashMap<>();
metaData.putAll(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
Map<String, String> metaData = new HashMap<>(instance.getMetadata());
temp.setMetadata(metaData);
return temp;
}
private static String getInstanceKey(Instance instance) {
return String.join("|",
instance.getIp(),
String.valueOf(instance.getPort()),
String.valueOf(instance.getWeight()),
String.valueOf(instance.isHealthy()),
String.valueOf(instance.isEphemeral()),
instance.getClusterName(),
instance.getServiceName());
}
}

View File

@ -12,10 +12,6 @@
*/
package com.alibaba.nacossync.extension.impl;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
import static com.alibaba.nacossync.util.StringUtils.convertDubboFullPathForZk;
import static com.alibaba.nacossync.util.StringUtils.convertDubboProvidersPath;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
@ -33,6 +29,13 @@ import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.DubboConstants;
import com.google.common.collect.Sets;
import lombok.extern.slf4j.Slf4j;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.PathChildrenCache;
import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
import org.apache.curator.utils.CloseableUtils;
import org.apache.zookeeper.CreateMode;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashMap;
@ -41,13 +44,10 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import lombok.extern.slf4j.Slf4j;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.PathChildrenCache;
import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
import org.apache.curator.utils.CloseableUtils;
import org.apache.zookeeper.CreateMode;
import org.springframework.beans.factory.annotation.Autowired;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
import static com.alibaba.nacossync.util.StringUtils.convertDubboFullPathForZk;
import static com.alibaba.nacossync.util.StringUtils.convertDubboProvidersPath;
/**
* Nacos 同步 Zk 数据
@ -59,8 +59,7 @@ import org.springframework.beans.factory.annotation.Autowired;
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.ZK)
public class NacosSyncToZookeeperServiceImpl implements SyncService {
@Autowired
private MetricsManager metricsManager;
private final MetricsManager metricsManager;
/**
* @description The Nacos listener map.
@ -92,12 +91,13 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
private final ZookeeperServerHolder zookeeperServerHolder;
@Autowired
public NacosSyncToZookeeperServiceImpl(SkyWalkerCacheServices skyWalkerCacheServices,
NacosServerHolder nacosServerHolder, ZookeeperServerHolder zookeeperServerHolder) {
NacosServerHolder nacosServerHolder, ZookeeperServerHolder zookeeperServerHolder, MetricsManager metricsManager) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
this.zookeeperServerHolder = zookeeperServerHolder;
this.metricsManager = metricsManager;
}
@Override
@ -113,8 +113,10 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
CloseableUtils.closeQuietly(pathChildrenCache);
Set<String> instanceUrlSet = instanceBackupMap.get(taskDO.getTaskId());
CuratorFramework client = zookeeperServerHolder.get(taskDO.getDestClusterId());
for (String instanceUrl : instanceUrlSet) {
client.delete().quietly().forPath(instanceUrl);
if(!instanceUrlSet.isEmpty()){
for (String instanceUrl : instanceUrlSet) {
client.delete().quietly().forPath(instanceUrl);
}
}
} catch (Exception e) {
log.error("delete task from nacos to zk was failed, taskId:{}", taskDO.getTaskId(), e);
@ -125,7 +127,7 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId());
@ -138,11 +140,11 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
Set<String> newInstanceUrlSet = getWaitingToAddInstance(taskDO, client, sourceInstances);
// 获取之前的备份 删除无效实例
// fetch the instance backup
deleteInvalidInstances(taskDO, client, newInstanceUrlSet);
// 替换当前备份为最新备份
// replace the instance backup
instanceBackupMap.put(taskDO.getTaskId(), newInstanceUrlSet);
// 尝试恢复因为zk客户端意外断开导致的实例数据
// try to compensate for the removed instance
tryToCompensate(taskDO, sourceNamingService, sourceInstances);
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskDO.getTaskId(), e);
@ -161,28 +163,45 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
return true;
}
private void tryToCompensate(TaskDO taskDO, NamingService sourceNamingService, List<Instance> sourceInstances) {
if (!CollectionUtils.isEmpty(sourceInstances)) {
final PathChildrenCache pathCache = getPathCache(taskDO);
if (pathCache.getListenable().size() == 0) { // 防止重复注册
pathCache.getListenable().addListener((zkClient, zkEvent) -> {
if (zkEvent.getType() == PathChildrenCacheEvent.Type.CHILD_REMOVED) {
List<Instance> allInstances =
sourceNamingService.getAllInstances(taskDO.getServiceName(),getGroupNameOrDefault(taskDO.getGroupName()));
for (Instance instance : allInstances) {
String instanceUrl = buildSyncInstance(instance, taskDO);
String zkInstancePath = zkEvent.getData().getPath();
if (zkInstancePath.equals(instanceUrl)) {
zkClient.create().creatingParentsIfNeeded().withMode(CreateMode.EPHEMERAL)
.forPath(zkInstancePath);
break;
}
}
}
});
// Avoiding re-registration if there is already a listener registered.
if (pathCache.getListenable().size() == 0) {
registerCompensationListener(pathCache, taskDO, sourceNamingService);
}
}
}
private void registerCompensationListener(PathChildrenCache pathCache, TaskDO taskDO, NamingService sourceNamingService) {
pathCache.getListenable().addListener((zkClient, zkEvent) -> {
try {
if (zkEvent.getType() == PathChildrenCacheEvent.Type.CHILD_REMOVED) {
compensateOnChildRemoval(zkClient, zkEvent, sourceNamingService, taskDO);
}
} catch (Exception e) {
log.error("Error processing ZooKeeper event: {}", zkEvent.getType(), e);
}
});
}
private void compensateOnChildRemoval(CuratorFramework zkClient, PathChildrenCacheEvent zkEvent, NamingService sourceNamingService, TaskDO taskDO) {
String zkInstancePath = null;
try {
List<Instance> allInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()));
zkInstancePath = zkEvent.getData().getPath();
for (Instance instance : allInstances) {
String instanceUrl = buildSyncInstance(instance, taskDO);
if (zkInstancePath.equals(instanceUrl)) {
zkClient.create().creatingParentsIfNeeded().withMode(CreateMode.EPHEMERAL)
.forPath(zkInstancePath);
log.info("Compensated by re-creating the removed node at path: {}", zkInstancePath);
break;
}
}
} catch (Exception e) {
log.error("Failed to compensate for the removed node at path: {}", zkInstancePath, e);
}
}
@ -215,12 +234,11 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
protected String buildSyncInstance(Instance instance, TaskDO taskDO) throws UnsupportedEncodingException {
Map<String, String> metaData = new HashMap<>();
metaData.putAll(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
Map<String, String> metaData = new HashMap<>(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
String servicePath = monitorPath.computeIfAbsent(taskDO.getTaskId(),
key -> convertDubboProvidersPath(metaData.get(DubboConstants.INTERFACE_KEY)));
@ -230,10 +248,10 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
/**
* 获取zk path child 监听缓存类
* fetch zk path cache
*
* @param taskDO 任务对象
* @return zk节点操作缓存对象
* @param taskDO task instance
* @return zk path cache
*/
private PathChildrenCache getPathCache(TaskDO taskDO) {
return pathChildrenCacheMap.computeIfAbsent(taskDO.getTaskId(), (key) -> {

View File

@ -10,8 +10,39 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.extension.holder.ZookeeperServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.TreeCache;
import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
import org.apache.curator.utils.CloseableUtils;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import static com.alibaba.nacossync.util.DubboConstants.ALL_SERVICE_NAME_PATTERN;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_PATH_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_ROOT_PATH;
@ -28,35 +59,6 @@ import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
import static com.alibaba.nacossync.util.StringUtils.parseIpAndPortString;
import static com.alibaba.nacossync.util.StringUtils.parseQueryString;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.extension.holder.ZookeeperServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Predicate;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.TreeCache;
import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
import org.apache.curator.utils.CloseableUtils;
import org.springframework.beans.factory.annotation.Autowired;
/**
* @author paderlol
* @version 1.0
@ -65,59 +67,55 @@ import org.springframework.beans.factory.annotation.Autowired;
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.ZK, destinationCluster = ClusterTypeEnum.NACOS)
public class ZookeeperSyncToNacosServiceImpl implements SyncService {
@Autowired
private MetricsManager metricsManager;
private static final String DEFAULT_WEIGHT = "1.0";
private final MetricsManager metricsManager;
/**
* Listener cache of Zookeeper format taskId -> PathChildrenCache instance
*/
private Map<String, TreeCache> treeCacheMap = new ConcurrentHashMap<>();
private final Map<String, TreeCache> treeCacheMap = new ConcurrentHashMap<>();
/**
* service name cache
*/
private Map<String, String> nacosServiceNameMap = new ConcurrentHashMap<>();
private final Map<String, String> nacosServiceNameMap = new ConcurrentHashMap<>();
private final ZookeeperServerHolder zookeeperServerHolder;
private final NacosServerHolder nacosServerHolder;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
public ZookeeperSyncToNacosServiceImpl(ZookeeperServerHolder zookeeperServerHolder,
NacosServerHolder nacosServerHolder, SkyWalkerCacheServices skyWalkerCacheServices) {
NacosServerHolder nacosServerHolder, SkyWalkerCacheServices skyWalkerCacheServices,
MetricsManager metricsManager) {
this.zookeeperServerHolder = zookeeperServerHolder;
this.nacosServerHolder = nacosServerHolder;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.metricsManager = metricsManager;
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
try {
if (treeCacheMap.containsKey(taskDO.getTaskId())) {
return true;
}
TreeCache treeCache = getTreeCache(taskDO);
if (!initializeTreeCache(taskDO)) {
return false;
}
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
if (destNamingService == null) {
logAndRecordSyncError("Failed to obtain NamingService for destination clusterId: {}", taskDO.getDestClusterId(), null);
return false;
}
// 初次执行任务统一注册所有实例
registerAllInstances(taskDO, destNamingService);
//注册ZK监听
Objects.requireNonNull(treeCache).getListenable().addListener((client, event) -> {
try {
String path = event.getData().getPath();
Map<String, String> queryParam = parseQueryString(path);
if (isMatch(taskDO, queryParam) && needSync(queryParam)) {
processEvent(taskDO, destNamingService, event, path, queryParam);
}
} catch (Exception e) {
log.error("event process from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
});
setupListener(taskDO, destNamingService);
} catch (Exception e) {
log.error("sync task from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -125,37 +123,217 @@ public class ZookeeperSyncToNacosServiceImpl implements SyncService {
}
return true;
}
private boolean initializeTreeCache(TaskDO taskDO) {
TreeCache treeCache = getTreeCache(taskDO);
if (treeCache == null) {
logAndRecordSyncError("Failed to obtain TreeCache for taskId: {}", taskDO.getTaskId(), null);
return false;
}
return true;
}
private void logAndRecordSyncError(String message, String taskId, Exception e) {
if (e != null) {
log.error(message, taskId, e);
} else {
log.error(message, taskId);
}
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
private void setupListener(TaskDO taskDO, NamingService destNamingService) {
TreeCache treeCache = Objects.requireNonNull(getTreeCache(taskDO));
treeCache.getListenable().addListener((client, event) -> {
try {
// INITIALIZED is a special event that is not triggered by the Zookeeper server
if(event.getData()==null){
log.warn("TreeCache event data is null, taskId:{}", taskDO.getTaskId());
return;
}
String path = event.getData().getPath();
Map<String, String> queryParam = parseQueryString(path);
if (isMatch(taskDO, queryParam) && needSync(queryParam)) {
processEvent(taskDO, destNamingService, event, path, queryParam);
}
} catch (Exception e) {
logAndRecordSyncError("Event process from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
}
});
}
public boolean delete(TaskDO taskDO) {
if (taskDO.getServiceName() == null) {
return true;
}
CloseableUtils.closeQuietly(treeCacheMap.get(taskDO.getTaskId()));
try {
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
if (destNamingService == null) {
log.error("Failed to obtain NamingService for destination clusterId: {}", taskDO.getDestClusterId());
return false;
}
Set<String> serviceNames = getServiceNamesToDelete(taskDO);
deleteInstances(serviceNames, destNamingService, taskDO);
} catch (Exception e) {
log.error("Delete task from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
private Set<String> getServiceNamesToDelete(TaskDO taskDO) {
if (!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
return new HashSet<>(Collections.singleton(taskDO.getServiceName()));
} else {
return new HashSet<>(nacosServiceNameMap.keySet());
}
}
private void deleteInstances(Set<String> serviceNames, NamingService destNamingService, TaskDO taskDO) {
for (String serviceName : serviceNames) {
try {
List<Instance> allInstances = destNamingService.getAllInstances(serviceName,
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(), instance.getPort());
}
}
nacosServiceNameMap.remove(serviceName);
} catch (NacosException e) {
log.error("Failed to deregister service instance for serviceName: {}", serviceName, e);
}
}
}
/**
* fetch the Path cache when the task sync
*/
protected TreeCache getTreeCache(TaskDO taskDO) {
return treeCacheMap.computeIfAbsent(taskDO.getTaskId(), (key) -> {
try {
TreeCache treeCache = new TreeCache(zookeeperServerHolder.get(taskDO.getSourceClusterId()),
DUBBO_ROOT_PATH);
treeCache.start();
return treeCache;
} catch (Exception e) {
log.error("zookeeper path children cache start failed, taskId:{}", taskDO.getTaskId(), e);
return null;
}
});
}
/**
* Checks if the instance information needs to be synchronized based on the dubbo version, grouping name,
* and service name.
*/
protected boolean isMatch(TaskDO taskDO, Map<String, String> queryParam) {
return isVersionMatch(taskDO, queryParam) &&
isGroupMatch(taskDO, queryParam) &&
isServiceMatch(taskDO, queryParam) ||
isMatchAllServices(taskDO);
}
private boolean isVersionMatch(TaskDO task, Map<String, String> queryParam) {
return StringUtils.isBlank(task.getVersion()) || StringUtils.equals(task.getVersion(), queryParam.get(VERSION_KEY));
}
private boolean isGroupMatch(TaskDO task, Map<String, String> queryParam) {
return StringUtils.isBlank(task.getGroupName()) || StringUtils.equals(task.getGroupName(), queryParam.get(GROUP_KEY));
}
private boolean isServiceMatch(TaskDO task, Map<String, String> queryParam) {
return StringUtils.isNotBlank(task.getServiceName()) && StringUtils.equals(task.getServiceName(), queryParam.get(INTERFACE_KEY));
}
private boolean isMatchAllServices(TaskDO task) {
return StringUtils.isNotBlank(task.getServiceName()) && StringUtils.equals(task.getServiceName(), ALL_SERVICE_NAME_PATTERN);
}
/**
* Builds a synchronized Nacos instance from Zookeeper data.
*
* @param queryParam Parameters obtained from the query string.
* @param ipAndPortMap IP and port information.
* @param taskDO Task details.
* @return A fully configured Nacos instance.
*/
protected Instance buildSyncInstance(Map<String, String> queryParam, Map<String, String> ipAndPortMap, TaskDO taskDO) {
Instance instance = new Instance();
instance.setIp(ipAndPortMap.get(INSTANCE_IP_KEY));
instance.setPort(Integer.parseInt(ipAndPortMap.get(INSTANCE_PORT_KEY)));
instance.setServiceName(getServiceNameFromCache(taskDO.getTaskId(), queryParam));
instance.setWeight(parseWeight(queryParam));
instance.setHealthy(true);
instance.setMetadata(buildMetadata(queryParam, ipAndPortMap, taskDO));
return instance;
}
private double parseWeight(Map<String, String> queryParam) {
try {
return Double.parseDouble(queryParam.getOrDefault(WEIGHT_KEY, DEFAULT_WEIGHT));
} catch (NumberFormatException e) {
log.error("Error parsing weight: {}", queryParam.get(WEIGHT_KEY), e);
return Double.parseDouble(DEFAULT_WEIGHT); // Default weight in case of error
}
}
private Map<String, String> buildMetadata(Map<String, String> queryParam, Map<String, String> ipAndPortMap, TaskDO taskDO) {
Map<String, String> metaData = new HashMap<>(queryParam);
metaData.put(PROTOCOL_KEY, ipAndPortMap.get(PROTOCOL_KEY));
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY, skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
return metaData;
}
private void processEvent(TaskDO taskDO, NamingService destNamingService, TreeCacheEvent event, String path,
Map<String, String> queryParam) throws NacosException {
Map<String, String> queryParam) throws NacosException {
if (!com.alibaba.nacossync.util.StringUtils.isDubboProviderPath(path)) {
return;
}
Map<String, String> ipAndPortParam = parseIpAndPortString(path);
if (ipAndPortParam.isEmpty()) {
log.error("Invalid IP and Port data extracted from path: {}", path);
return;
}
Instance instance = buildSyncInstance(queryParam, ipAndPortParam, taskDO);
String serviceName = queryParam.get(INTERFACE_KEY);
if (serviceName == null || serviceName.isEmpty()) {
log.error("Service name is missing in the query parameters.");
return;
}
switch (event.getType()) {
case NODE_ADDED:
case NODE_UPDATED:
destNamingService.registerInstance(
getServiceNameFromCache(serviceName, queryParam), getGroupNameOrDefault(taskDO.getGroupName()),
instance);
destNamingService.registerInstance(getServiceNameFromCache(serviceName, queryParam),
getGroupNameOrDefault(taskDO.getGroupName()), instance);
break;
case NODE_REMOVED:
destNamingService.deregisterInstance(
getServiceNameFromCache(serviceName, queryParam), getGroupNameOrDefault(taskDO.getGroupName()),
ipAndPortParam.get(INSTANCE_IP_KEY),
Integer.parseInt(ipAndPortParam.get(INSTANCE_PORT_KEY)));
destNamingService.deregisterInstance(getServiceNameFromCache(serviceName, queryParam),
getGroupNameOrDefault(taskDO.getGroupName()), ipAndPortParam.get(INSTANCE_IP_KEY),
Integer.parseInt(ipAndPortParam.get(INSTANCE_PORT_KEY)));
nacosServiceNameMap.remove(serviceName);
break;
default:
break;
}
}
private void registerAllInstances(TaskDO taskDO, NamingService destNamingService) throws Exception {
CuratorFramework zk = zookeeperServerHolder.get(taskDO.getSourceClusterId());
if (!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
@ -168,9 +346,9 @@ public class ZookeeperSyncToNacosServiceImpl implements SyncService {
}
}
}
private void registerALLInstances0(TaskDO taskDO, NamingService destNamingService, CuratorFramework zk,
String serviceName) throws Exception {
String serviceName) throws Exception {
String path = String.format(DUBBO_PATH_FORMAT, serviceName);
if (zk.getChildren() == null) {
return;
@ -182,122 +360,12 @@ public class ZookeeperSyncToNacosServiceImpl implements SyncService {
Map<String, String> ipAndPortParam = parseIpAndPortString(path + ZOOKEEPER_SEPARATOR + provider);
Instance instance = buildSyncInstance(queryParam, ipAndPortParam, taskDO);
destNamingService.registerInstance(getServiceNameFromCache(serviceName, queryParam),
getGroupNameOrDefault(taskDO.getGroupName()),
instance);
getGroupNameOrDefault(taskDO.getGroupName()), instance);
}
}
}
@Override
public boolean delete(TaskDO taskDO) {
if (taskDO.getServiceName() == null) {
return true;
}
try {
CloseableUtils.closeQuietly(treeCacheMap.get(taskDO.getTaskId()));
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
if (!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
if (nacosServiceNameMap.containsKey(taskDO.getServiceName())) {
List<Instance> allInstances =
destNamingService.getAllInstances(nacosServiceNameMap.get(taskDO.getServiceName()),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(),
instance.getPort());
}
nacosServiceNameMap.remove(taskDO.getServiceName());
}
}
} else {
Set<String> serviceNames = nacosServiceNameMap.keySet();
for (String serviceName : serviceNames) {
if (nacosServiceNameMap.containsKey(serviceName)) {
List<Instance> allInstances =
destNamingService.getAllInstances(serviceName, getGroupNameOrDefault(taskDO.getGroupName()),
new ArrayList<>(), true);
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(),
instance.getPort());
}
nacosServiceNameMap.remove(serviceName);
}
}
}
}
} catch (Exception e) {
log.error("delete task from zookeeper to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
/**
* fetch the Path cache when the task sync
*/
protected TreeCache getTreeCache(TaskDO taskDO) {
return treeCacheMap.computeIfAbsent(taskDO.getTaskId(), (key) -> {
try {
TreeCache treeCache =
new TreeCache(zookeeperServerHolder.get(taskDO.getSourceClusterId()),
DUBBO_ROOT_PATH);
treeCache.start();
return treeCache;
} catch (Exception e) {
log.error("zookeeper path children cache start failed, taskId:{}", taskDO.getTaskId(), e);
return null;
}
});
}
/**
* The instance information that needs to be synchronized is matched based on the dubbo version and the grouping
* name
*/
protected boolean isMatch(TaskDO taskDO, Map<String, String> queryParam) {
Predicate<TaskDO> isVersionEq = (task) -> StringUtils.isBlank(taskDO.getVersion())
|| StringUtils.equals(task.getVersion(), queryParam.get(VERSION_KEY));
Predicate<TaskDO> isGroupEq = (task) -> StringUtils.isBlank(taskDO.getGroupName())
|| StringUtils.equals(task.getGroupName(), queryParam.get(GROUP_KEY));
return isVersionEq.and(isGroupEq).test(taskDO);
}
/**
* create Nacos service instance
*
* @param queryParam dubbo metadata
* @param ipAndPortMap dubbo ip and address
*/
protected Instance buildSyncInstance(Map<String, String> queryParam, Map<String, String> ipAndPortMap,
TaskDO taskDO) {
Instance temp = new Instance();
temp.setIp(ipAndPortMap.get(INSTANCE_IP_KEY));
temp.setPort(Integer.parseInt(ipAndPortMap.get(INSTANCE_PORT_KEY)));
temp.setServiceName(getServiceNameFromCache(taskDO.getTaskId(), queryParam));
temp.setWeight(Double.parseDouble(queryParam.get(WEIGHT_KEY) == null ? "1.0" : queryParam.get(WEIGHT_KEY)));
temp.setHealthy(true);
Map<String, String> metaData = new HashMap<>(queryParam);
metaData.put(PROTOCOL_KEY, ipAndPortMap.get(PROTOCOL_KEY));
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
temp.setMetadata(metaData);
return temp;
}
/**
* cteate Dubbo service name
*
@ -307,5 +375,5 @@ public class ZookeeperSyncToNacosServiceImpl implements SyncService {
protected String getServiceNameFromCache(String serviceName, Map<String, String> queryParam) {
return nacosServiceNameMap.computeIfAbsent(serviceName, (key) -> createServiceName(queryParam));
}
}

View File

@ -23,15 +23,19 @@ import java.util.concurrent.TimeUnit;
@Service
public class MetricsManager implements CommandLineRunner {
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private ClusterAccessService clusterAccessService;
@Autowired
private TaskAccessService taskAccessService;
private final ClusterAccessService clusterAccessService;
private final TaskAccessService taskAccessService;
public MetricsManager(SkyWalkerCacheServices skyWalkerCacheServices, ClusterAccessService clusterAccessService,
TaskAccessService taskAccessService) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.clusterAccessService = clusterAccessService;
this.taskAccessService = taskAccessService;
}
/**
* Callback used to run the bean.
*

View File

@ -17,17 +17,28 @@
package com.alibaba.nacossync.pojo.model;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.proxy.HibernateProxy;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import java.io.Serializable;
import javax.persistence.*;
import lombok.Data;
import java.util.Objects;
/**
* @author NacosSync
* @version $Id: EnvDO.java, v 0.1 2018-09-25 PM 4:17 NacosSync Exp $$
*/
@Data
@Getter
@Setter
@ToString
@RequiredArgsConstructor
@Entity
@Table(name = "cluster")
public class ClusterDO implements Serializable {
@ -67,5 +78,33 @@ public class ClusterDO implements Serializable {
private String password;
private String namespace;
private Integer clusterLevel;
@Override
public final boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null) {
return false;
}
Class<?> oEffectiveClass =
o instanceof HibernateProxy ? ((HibernateProxy) o).getHibernateLazyInitializer().getPersistentClass()
: o.getClass();
Class<?> thisEffectiveClass =
this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass() : this.getClass();
if (thisEffectiveClass != oEffectiveClass) {
return false;
}
ClusterDO clusterDO = (ClusterDO) o;
return getId() != null && Objects.equals(getId(), clusterDO.getId());
}
@Override
public final int hashCode() {
return this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass().hashCode() : getClass().hashCode();
}
}

View File

@ -16,15 +16,28 @@
*/
package com.alibaba.nacossync.pojo.model;
import javax.persistence.*;
import lombok.Data;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.proxy.HibernateProxy;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import java.util.Objects;
/**
* @author NacosSync
* @version $Id: SystemConfig.java, v 0.1 2018-09-26 上午1:48 NacosSync Exp $$
*/
@Data
@Getter
@Setter
@ToString
@RequiredArgsConstructor
@Entity
@Table(name = "system_config")
public class SystemConfigDO {
@ -34,5 +47,31 @@ public class SystemConfigDO {
private String configKey;
private String configValue;
private String configDesc;
@Override
public final boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null) {
return false;
}
Class<?> oEffectiveClass =
o instanceof HibernateProxy ? ((HibernateProxy) o).getHibernateLazyInitializer().getPersistentClass()
: o.getClass();
Class<?> thisEffectiveClass =
this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass() : this.getClass();
if (thisEffectiveClass != oEffectiveClass) {
return false;
}
SystemConfigDO that = (SystemConfigDO) o;
return getId() != null && Objects.equals(getId(), that.getId());
}
@Override
public final int hashCode() {
return this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass().hashCode() : getClass().hashCode();
}
}

View File

@ -12,16 +12,29 @@
*/
package com.alibaba.nacossync.pojo.model;
import lombok.Data;
import javax.persistence.*;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.proxy.HibernateProxy;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import java.io.Serializable;
import java.util.Objects;
/**
* @author NacosSync
* @version $Id: TaskDo.java, v 0.1 2018-09-24 PM11:53 NacosSync Exp $$
*/
@Data
@Getter
@Setter
@ToString
@RequiredArgsConstructor
@Entity
@Table(name = "task")
public class TaskDO implements Serializable {
@ -70,4 +83,31 @@ public class TaskDO implements Serializable {
* operation id,The operation id follow when the task status changes
*/
private String operationId;
@Override
public final boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null) {
return false;
}
Class<?> oEffectiveClass =
o instanceof HibernateProxy ? ((HibernateProxy) o).getHibernateLazyInitializer().getPersistentClass()
: o.getClass();
Class<?> thisEffectiveClass =
this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass() : this.getClass();
if (thisEffectiveClass != oEffectiveClass) {
return false;
}
TaskDO taskDO = (TaskDO) o;
return getId() != null && Objects.equals(getId(), taskDO.getId());
}
@Override
public final int hashCode() {
return this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass().hashCode() : getClass().hashCode();
}
}

View File

@ -17,10 +17,10 @@
package com.alibaba.nacossync.pojo.request;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import java.util.List;
import lombok.Data;
import java.util.List;
/**
* @author NacosSync
* @version $Id: AddClusterRequest.java, v 0.1 2018-09-25 PM 10:27 NacosSync Exp $$

View File

@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.pojo.request;
import lombok.Data;
/**
* @author NacosSync
* @version $Id: TaskAddAllRequest.java, v 0.1 2022-03-23 AM12:13 NacosSync Exp $$
*/
@Data
public class TaskAddAllRequest extends BaseRequest {
/**
* eg: b7bacb110199d5bb83b9757038fadeb0 .
*/
private String sourceClusterId;
/**
* eg: bbdad57833a0e4f0981f6f3349005617 .
*/
private String destClusterId;
/**
* whether to exclude subscriber.
*/
private boolean excludeConsumer = true;
}

View File

@ -14,36 +14,53 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.pojo.view;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import java.io.Serializable;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import lombok.Data;
import java.io.Serializable;
/**
* @author NacosSync
* @version $Id: ClusterModel.java, v 0.1 2018-09-25 下午11:09 NacosSync Exp $$
*/
@Data
public class ClusterModel implements Serializable {
private String clusterId;
/**
* json format["192.168.1:8080","192.168.2?key=1"]
*/
private String connectKeyList;
/**
* cluster name, egcluster of ShangHaiedas-sh
*/
private String clusterName;
/**
* cluster type, eg cluster of CS,cluster of Nacos,
*
* @see ClusterTypeEnum
*/
private String clusterType;
private String namespace;
private String userName;
public static ClusterModel from(ClusterDO clusterDO) {
ClusterModel clusterModel = new ClusterModel();
clusterModel.setClusterId(clusterDO.getClusterId());
clusterModel.setConnectKeyList(clusterDO.getConnectKeyList());
clusterModel.setClusterType(clusterDO.getClusterType());
clusterModel.setClusterName(clusterDO.getClusterName());
clusterModel.setNamespace(clusterDO.getNamespace());
clusterModel.setUserName(clusterDO.getUserName());
return clusterModel;
}
}

View File

@ -14,8 +14,10 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.pojo.view;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.Data;
/**
@ -24,12 +26,27 @@ import lombok.Data;
*/
@Data
public class TaskModel {
private String taskId;
private String sourceClusterId;
private String destClusterId;
private String serviceName;
private String groupName;
private String taskStatus;
public static TaskModel from(TaskDO taskDO) {
TaskModel taskModel = new TaskModel();
taskModel.setDestClusterId(taskDO.getDestClusterId());
taskModel.setGroupName(taskDO.getGroupName());
taskModel.setServiceName(taskDO.getServiceName());
taskModel.setSourceClusterId(taskDO.getSourceClusterId());
taskModel.setTaskStatus(taskDO.getTaskStatus());
taskModel.setTaskId(taskDO.getTaskId());
return taskModel;
}
}

View File

@ -45,8 +45,7 @@ public class SkyWalkerTemplate {
}
private static <T extends BaseResult> void initExceptionResult(T result, Throwable e) {
if (e instanceof SkyWalkerException) {
SkyWalkerException skyWalkerException = (SkyWalkerException) e;
if (e instanceof SkyWalkerException skyWalkerException) {
if (null != skyWalkerException.getResultCode()) {
result.setResultCode(skyWalkerException.getResultCode().getCode());
}

View File

@ -14,12 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterAddRequest;
import com.alibaba.nacossync.pojo.result.ClusterAddResult;
@ -28,7 +28,6 @@ import com.alibaba.nacossync.util.SkyWalkerUtil;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
@ -38,44 +37,45 @@ import org.springframework.stereotype.Service;
@Slf4j
@Service
public class ClusterAddProcessor implements Processor<ClusterAddRequest, ClusterAddResult> {
@Autowired
private MetricsManager metricsManager;
@Autowired
private ClusterAccessService clusterAccessService;
@Autowired
private ObjectMapper objectMapper;
private final ClusterAccessService clusterAccessService;
private final ObjectMapper objectMapper;
public ClusterAddProcessor(ClusterAccessService clusterAccessService, ObjectMapper objectMapper) {
this.clusterAccessService = clusterAccessService;
this.objectMapper = objectMapper;
}
@Override
public void process(ClusterAddRequest clusterAddRequest, ClusterAddResult clusterAddResult,
Object... others) throws Exception {
public void process(ClusterAddRequest clusterAddRequest, ClusterAddResult clusterAddResult, Object... others)
throws Exception {
ClusterDO clusterDO = new ClusterDO();
if (null == clusterAddRequest.getConnectKeyList() || 0 == clusterAddRequest.getConnectKeyList().size()) {
if (null == clusterAddRequest.getConnectKeyList() || clusterAddRequest.getConnectKeyList().isEmpty()) {
throw new SkyWalkerException("集群列表不能为空!");
}
if (StringUtils.isBlank(clusterAddRequest.getClusterName()) || StringUtils
.isBlank(clusterAddRequest.getClusterType())) {
if (StringUtils.isBlank(clusterAddRequest.getClusterName()) || StringUtils.isBlank(
clusterAddRequest.getClusterType())) {
throw new SkyWalkerException("集群名字或者类型不能为空!");
}
if (!ClusterTypeEnum.contains(clusterAddRequest.getClusterType())) {
throw new SkyWalkerException("集群类型不存在:" + clusterAddRequest.getClusterType());
}
String clusterId = SkyWalkerUtil.generateClusterId(clusterAddRequest);
if (null != clusterAccessService.findByClusterId(clusterId)) {
throw new SkyWalkerException("重复插入clusterId已存在" + clusterId);
}
clusterDO.setClusterId(clusterId);
clusterDO.setClusterName(clusterAddRequest.getClusterName());
clusterDO.setClusterType(clusterAddRequest.getClusterType());
@ -83,6 +83,7 @@ public class ClusterAddProcessor implements Processor<ClusterAddRequest, Cluster
clusterDO.setUserName(clusterAddRequest.getUserName());
clusterDO.setPassword(clusterAddRequest.getPassword());
clusterDO.setNamespace(clusterAddRequest.getNamespace());
clusterDO.setClusterLevel(0);
clusterAccessService.insert(clusterDO);
}
}

View File

@ -14,15 +14,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.result.ClusterDeleteResult;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.pojo.request.ClusterDeleteRequest;
import com.alibaba.nacossync.pojo.result.ClusterDeleteResult;
import com.alibaba.nacossync.template.Processor;
import org.springframework.stereotype.Service;
/**
* @author NacosSync
@ -30,15 +31,26 @@ import com.alibaba.nacossync.template.Processor;
*/
@Service
public class ClusterDeleteProcessor implements Processor<ClusterDeleteRequest, ClusterDeleteResult> {
@Autowired
private ClusterAccessService clusterAccessService;
private final ClusterAccessService clusterAccessService;
private final TaskAccessService taskAccessService;
public ClusterDeleteProcessor(ClusterAccessService clusterAccessService, TaskAccessService taskAccessService) {
this.clusterAccessService = clusterAccessService;
this.taskAccessService = taskAccessService;
}
@Override
public void process(ClusterDeleteRequest clusterDeleteRequest,
ClusterDeleteResult clusterDeleteResult, Object... others) throws Exception {
public void process(ClusterDeleteRequest clusterDeleteRequest, ClusterDeleteResult clusterDeleteResult,
Object... others) throws Exception {
int count = taskAccessService.countByDestClusterIdOrSourceClusterId(clusterDeleteRequest.getClusterId(),
clusterDeleteRequest.getClusterId());
if (count > 0) {
throw new SkyWalkerException(String.format("集群下有%d个任务请先删除任务", count));
}
clusterAccessService.deleteByClusterId(clusterDeleteRequest.getClusterId());
}
}

View File

@ -14,45 +14,37 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.result.ClusterDetailQueryResult;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterDetailQueryRequest;
import com.alibaba.nacossync.pojo.result.ClusterDetailQueryResult;
import com.alibaba.nacossync.pojo.view.ClusterModel;
import com.alibaba.nacossync.template.Processor;
import org.springframework.stereotype.Service;
/**
* @author NacosSync
* @version $Id: ClusterDetailQueryProcessor.java, v 0.1 2018-09-30 PM2:39 NacosSync Exp $$
*/
@Service
public class ClusterDetailQueryProcessor
implements
Processor<ClusterDetailQueryRequest, ClusterDetailQueryResult> {
@Autowired
private ClusterAccessService clusterAccessService;
public class ClusterDetailQueryProcessor implements Processor<ClusterDetailQueryRequest, ClusterDetailQueryResult> {
private final ClusterAccessService clusterAccessService;
public ClusterDetailQueryProcessor(ClusterAccessService clusterAccessService) {
this.clusterAccessService = clusterAccessService;
}
@Override
public void process(ClusterDetailQueryRequest clusterDetailQueryRequest,
ClusterDetailQueryResult clusterDetailQueryResult, Object... others)
throws Exception {
ClusterDO clusterDO = clusterAccessService.findByClusterId(clusterDetailQueryRequest
.getClusterId());
ClusterModel clusterModel = new ClusterModel();
clusterModel.setClusterId(clusterDO.getClusterId());
clusterModel.setConnectKeyList(clusterDO.getConnectKeyList());
clusterModel.setClusterType(clusterDO.getClusterType());
clusterModel.setClusterName(clusterDO.getClusterName());
clusterModel.setNamespace(clusterDO.getNamespace());
clusterModel.setUserName(clusterDO.getUserName());
clusterDetailQueryResult.setClusterModel(clusterModel);
ClusterDetailQueryResult clusterDetailQueryResult, Object... others) throws Exception {
ClusterDO clusterDO = clusterAccessService.findByClusterId(clusterDetailQueryRequest.getClusterId());
clusterDetailQueryResult.setClusterModel(ClusterModel.from(clusterDO));
}
}

View File

@ -14,71 +14,60 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import java.util.ArrayList;
import java.util.List;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.QueryCondition;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterListQueryRequest;
import com.alibaba.nacossync.pojo.result.ClusterListQueryResult;
import com.alibaba.nacossync.pojo.view.ClusterModel;
import com.alibaba.nacossync.template.Processor;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.result.ClusterListQueryResult;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterListQueryRequest;
import com.alibaba.nacossync.pojo.view.ClusterModel;
import com.alibaba.nacossync.template.Processor;
import java.util.List;
/**
* @author NacosSync
* @version $Id: ClusterListQueryProcessor.java, v 0.1 2018-09-30 PM2:33 NacosSync Exp $$
*/
@Service
public class ClusterListQueryProcessor implements
Processor<ClusterListQueryRequest, ClusterListQueryResult> {
@Autowired
private ClusterAccessService clusterAccessService;
public class ClusterListQueryProcessor implements Processor<ClusterListQueryRequest, ClusterListQueryResult> {
private final ClusterAccessService clusterAccessService;
public ClusterListQueryProcessor(ClusterAccessService clusterAccessService) {
this.clusterAccessService = clusterAccessService;
}
@Override
public void process(ClusterListQueryRequest clusterListQueryRequest,
ClusterListQueryResult clusterListQueryResult, Object... others) {
public void process(ClusterListQueryRequest clusterListQueryRequest, ClusterListQueryResult clusterListQueryResult,
Object... others) {
Page<ClusterDO> clusterDOS;
if (StringUtils.isNotBlank(clusterListQueryRequest.getClusterName())) {
QueryCondition queryCondition = new QueryCondition();
queryCondition.setServiceName(clusterListQueryRequest.getClusterName());
clusterDOS = clusterAccessService.findPageCriteria(clusterListQueryRequest.getPageNum() - 1,
clusterListQueryRequest.getPageSize(), queryCondition);
} else {
clusterDOS = clusterAccessService.findPageNoCriteria(clusterListQueryRequest.getPageNum() - 1,
clusterListQueryRequest.getPageSize());
}
List<ClusterModel> clusterModels = new ArrayList<>();
clusterDOS.forEach(clusterDO -> {
ClusterModel clusterModel = new ClusterModel();
clusterModel.setClusterId(clusterDO.getClusterId());
clusterModel.setClusterName(clusterDO.getClusterName());
clusterModel.setClusterType(clusterDO.getClusterType());
clusterModel.setConnectKeyList(clusterDO.getConnectKeyList());
clusterModel.setNamespace(clusterDO.getNamespace());
clusterModels.add(clusterModel);
});
List<ClusterModel> clusterModels = clusterDOS.stream().map(ClusterModel::from).toList();
clusterListQueryResult.setClusterModels(clusterModels);
clusterListQueryResult.setTotalPage(clusterDOS.getTotalPages());
clusterListQueryResult.setCurrentSize(clusterModels.size());
clusterListQueryResult.setTotalSize(clusterDOS.getTotalElements());
}
}

View File

@ -0,0 +1,250 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.CommonParams;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.client.naming.NacosNamingService;
import com.alibaba.nacos.client.naming.net.NamingProxy;
import com.alibaba.nacos.client.naming.utils.UtilAndComs;
import com.alibaba.nacos.common.utils.HttpMethod;
import com.alibaba.nacos.common.utils.JacksonUtils;
import com.alibaba.nacos.common.utils.StringUtils;
import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.extension.SyncManagerService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskAddAllRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.alibaba.nacossync.pojo.result.TaskAddResult;
import com.alibaba.nacossync.template.Processor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.util.ReflectionUtils;
import javax.annotation.Nullable;
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.GROUP_NAME_PARAM;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.PAGE_NO;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.PAGE_SIZE;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.SERVICE_NAME_PARAM;
/**
* @author NacosSync
* @version $Id: TaskAddAllProcessor.java, v 0.1 2022-03-23 PM11:40 NacosSync Exp $$
*/
@Slf4j
@Service
public class TaskAddAllProcessor implements Processor<TaskAddAllRequest, TaskAddResult> {
private static final String CONSUMER_PREFIX = "consumers:";
private final NacosServerHolder nacosServerHolder;
private final SyncManagerService syncManagerService;
private final TaskAccessService taskAccessService;
private final ClusterAccessService clusterAccessService;
public TaskAddAllProcessor(NacosServerHolder nacosServerHolder, SyncManagerService syncManagerService,
TaskAccessService taskAccessService, ClusterAccessService clusterAccessService) {
this.nacosServerHolder = nacosServerHolder;
this.syncManagerService = syncManagerService;
this.taskAccessService = taskAccessService;
this.clusterAccessService = clusterAccessService;
}
@Override
public void process(TaskAddAllRequest addAllRequest, TaskAddResult taskAddResult, Object... others)
throws Exception {
ClusterDO destCluster = clusterAccessService.findByClusterId(addAllRequest.getDestClusterId());
ClusterDO sourceCluster = clusterAccessService.findByClusterId(addAllRequest.getSourceClusterId());
if (Objects.isNull(destCluster) || Objects.isNull(sourceCluster)) {
throw new SkyWalkerException("Please check if the source or target cluster exists.");
}
if (Objects.isNull(syncManagerService.getSyncService(sourceCluster.getClusterId(), destCluster.getClusterId()))) {
throw new SkyWalkerException("current sync type not supported.");
}
// TODO 目前仅支持 Nacos 为源的同步类型待完善更多类型支持
final NamingService sourceNamingService = nacosServerHolder.get(sourceCluster.getClusterId());
if (sourceNamingService == null) {
throw new SkyWalkerException("only support sync type that the source of the Nacos.");
}
final EnhanceNamingService enhanceNamingService = new EnhanceNamingService(sourceNamingService);
final CatalogServiceResult catalogServiceResult = enhanceNamingService.catalogServices(null, null);
if (catalogServiceResult == null || catalogServiceResult.getCount() <= 0) {
throw new SkyWalkerException("sourceCluster data empty");
}
for (ServiceView serviceView : catalogServiceResult.getServiceList()) {
// exclude subscriber
if (addAllRequest.isExcludeConsumer() && serviceView.getName().startsWith(CONSUMER_PREFIX)) {
continue;
}
TaskAddRequest taskAddRequest = new TaskAddRequest();
taskAddRequest.setSourceClusterId(sourceCluster.getClusterId());
taskAddRequest.setDestClusterId(destCluster.getClusterId());
taskAddRequest.setServiceName(serviceView.getName());
taskAddRequest.setGroupName(serviceView.getGroupName());
this.dealTask(addAllRequest, taskAddRequest);
}
}
private void dealTask(TaskAddAllRequest addAllRequest, TaskAddRequest taskAddRequest) throws Exception {
String taskId = SkyWalkerUtil.generateTaskId(taskAddRequest);
TaskDO taskDO = taskAccessService.findByTaskId(taskId);
if (null == taskDO) {
taskDO = new TaskDO();
taskDO.setTaskId(taskId);
taskDO.setDestClusterId(addAllRequest.getDestClusterId());
taskDO.setSourceClusterId(addAllRequest.getSourceClusterId());
taskDO.setServiceName(taskAddRequest.getServiceName());
taskDO.setVersion(taskAddRequest.getVersion());
taskDO.setGroupName(taskAddRequest.getGroupName());
taskDO.setNameSpace(taskAddRequest.getNameSpace());
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());
taskDO.setWorkerIp(SkyWalkerUtil.getLocalIp());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
} else {
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
}
taskAccessService.addTask(taskDO);
}
static class EnhanceNamingService {
protected NamingService delegate;
protected NamingProxy serverProxy;
protected EnhanceNamingService(NamingService namingService) {
if (!(namingService instanceof NacosNamingService)) {
throw new IllegalArgumentException(
"namingService only support instance of com.alibaba.nacos.client.naming.NacosNamingService.");
}
this.delegate = namingService;
// serverProxy
final Field serverProxyField = ReflectionUtils.findField(NacosNamingService.class, "serverProxy");
assert serverProxyField != null;
ReflectionUtils.makeAccessible(serverProxyField);
this.serverProxy = (NamingProxy) ReflectionUtils.getField(serverProxyField, delegate);
}
public CatalogServiceResult catalogServices(@Nullable String serviceName, @Nullable String group)
throws NacosException {
int pageNo = 1; // start with 1
int pageSize = 100;
final CatalogServiceResult result = catalogServices(serviceName, group, pageNo, pageSize);
CatalogServiceResult tmpResult = result;
while (Objects.nonNull(tmpResult) && tmpResult.serviceList.size() >= pageSize) {
pageNo++;
tmpResult = catalogServices(serviceName, group, pageNo, pageSize);
if (tmpResult != null) {
result.serviceList.addAll(tmpResult.serviceList);
}
}
return result;
}
/**
* @see com.alibaba.nacos.client.naming.core.HostReactor#getServiceInfoDirectlyFromServer(String, String)
*/
public CatalogServiceResult catalogServices(@Nullable String serviceName, @Nullable String group, int pageNo,
int pageSize) throws NacosException {
// pageNo
// pageSize
// serviceNameParam
// groupNameParam
final Map<String, String> params = new HashMap<>(8);
params.put(CommonParams.NAMESPACE_ID, serverProxy.getNamespaceId());
params.put(SERVICE_NAME_PARAM, serviceName);
params.put(GROUP_NAME_PARAM, group);
params.put(PAGE_NO, String.valueOf(pageNo));
params.put(PAGE_SIZE, String.valueOf(pageSize));
final String result = this.serverProxy.reqApi(UtilAndComs.nacosUrlBase + "/catalog/services", params,
HttpMethod.GET);
if (StringUtils.isNotEmpty(result)) {
return JacksonUtils.toObj(result, CatalogServiceResult.class);
}
return null;
}
}
/**
* Copy from Nacos Server.
*/
@Data
static class ServiceView {
private String name;
private String groupName;
private int clusterCount;
private int ipCount;
private int healthyInstanceCount;
private String triggerFlag;
}
@Data
static class CatalogServiceResult {
/**
* countnot equal serviceList.size .
*/
private int count;
private List<ServiceView> serviceList;
}
}

View File

@ -89,7 +89,7 @@ public class TaskAddProcessor implements Processor<TaskAddRequest, TaskAddResult
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());
taskDO.setWorkerIp(SkyWalkerUtil.getLocalIp());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
} else {
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());

View File

@ -16,19 +16,12 @@
*/
package com.alibaba.nacossync.template.processor;
import java.util.ArrayList;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskDeleteInBatchRequest;
import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.template.Processor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
@ -40,20 +33,15 @@ import lombok.extern.slf4j.Slf4j;
@Service
public class TaskDeleteInBatchProcessor implements Processor<TaskDeleteInBatchRequest, BaseResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskDeleteInBatchProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskDeleteInBatchRequest taskBatchDeleteRequest, BaseResult baseResult,
Object... others) {
//
// String[] taskIds= taskBatchDeleteRequest.getTaskIds();
// List<TaskDO> taskDOs = new ArrayList<TaskDO>();
// for (String taskId : taskIds) {
// TaskDO taskDO = new TaskDO();
// taskDO.setTaskId(taskId);
// taskDOs.add(taskDO);
// }
taskAccessService.deleteTaskInBatch(taskBatchDeleteRequest.getTaskIds());
}
}

View File

@ -14,9 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.event.DeleteAllSubTaskEvent;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskDeleteRequest;
@ -24,7 +27,6 @@ import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.template.Processor;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
@ -34,18 +36,28 @@ import org.springframework.stereotype.Service;
@Slf4j
@Service
public class TaskDeleteProcessor implements Processor<TaskDeleteRequest, BaseResult> {
@Autowired
private TaskAccessService taskAccessService;
@Autowired
private EventBus eventBus;
private final TaskAccessService taskAccessService;
private final EventBus eventBus;
public TaskDeleteProcessor(TaskAccessService taskAccessService, EventBus eventBus) {
this.taskAccessService = taskAccessService;
this.eventBus = eventBus;
}
@Override
public void process(TaskDeleteRequest taskDeleteRequest, BaseResult baseResult,
Object... others) {
public void process(TaskDeleteRequest taskDeleteRequest, BaseResult baseResult, Object... others) {
TaskDO taskDO = taskAccessService.findByTaskId(taskDeleteRequest.getTaskId());
eventBus.post(new DeleteTaskEvent(taskDO));
log.info("删除同步任务数据之前,发出一个同步事件:" + taskDO);
// delete all sub task when ServiceName is all
if (SkyWalkerConstants.NACOS_ALL_SERVICE_NAME.equalsIgnoreCase(taskDO.getServiceName())) {
eventBus.post(new DeleteAllSubTaskEvent(taskDO));
} else {
eventBus.post(new DeleteTaskEvent(taskDO));
}
log.info("删除同步任务数据之前,发出一个同步事件:{}", taskDO);
taskAccessService.deleteTaskById(taskDeleteRequest.getTaskId());
}
}

View File

@ -14,17 +14,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.pojo.result.TaskDetailQueryResult;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskDetailQueryRequest;
import com.alibaba.nacossync.pojo.result.TaskDetailQueryResult;
import com.alibaba.nacossync.pojo.view.TaskModel;
import com.alibaba.nacossync.template.Processor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
@ -34,29 +34,23 @@ import org.springframework.stereotype.Service;
@Slf4j
@Service
public class TaskDetailProcessor implements Processor<TaskDetailQueryRequest, TaskDetailQueryResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskDetailProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskDetailQueryRequest taskDetailQueryRequest, TaskDetailQueryResult taskDetailQueryResult, Object... others)
throws Exception {
public void process(TaskDetailQueryRequest taskDetailQueryRequest, TaskDetailQueryResult taskDetailQueryResult,
Object... others) throws Exception {
TaskDO taskDO = taskAccessService.findByTaskId(taskDetailQueryRequest.getTaskId());
if (null == taskDO) {
throw new SkyWalkerException("taskDo is null,taskId :" + taskDetailQueryRequest.getTaskId());
}
TaskModel taskModel = new TaskModel();
taskModel.setDestClusterId(taskDO.getDestClusterId());
taskModel.setGroupName(taskDO.getGroupName());
taskModel.setServiceName(taskDO.getServiceName());
taskModel.setSourceClusterId(taskDO.getSourceClusterId());
taskModel.setTaskStatus(taskDO.getTaskStatus());
taskModel.setTaskId(taskDO.getTaskId());
taskDetailQueryResult.setTaskModel(taskModel);
taskDetailQueryResult.setTaskModel(TaskModel.from(taskDO));
}
}

View File

@ -14,25 +14,23 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import java.util.ArrayList;
import java.util.List;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.QueryCondition;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskListQueryRequest;
import com.alibaba.nacossync.pojo.result.TaskListQueryResult;
import com.alibaba.nacossync.pojo.view.TaskModel;
import com.alibaba.nacossync.template.Processor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.result.TaskListQueryResult;
import com.alibaba.nacossync.pojo.request.TaskListQueryRequest;
import com.alibaba.nacossync.pojo.view.TaskModel;
import com.alibaba.nacossync.template.Processor;
import java.util.List;
/**
* @author NacosSync
@ -41,42 +39,34 @@ import com.alibaba.nacossync.template.Processor;
@Service
@Slf4j
public class TaskListQueryProcessor implements Processor<TaskListQueryRequest, TaskListQueryResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskListQueryProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskListQueryRequest taskListQueryRequest,
TaskListQueryResult taskListQueryResult, Object... others) {
public void process(TaskListQueryRequest taskListQueryRequest, TaskListQueryResult taskListQueryResult,
Object... others) {
Page<TaskDO> taskDOPage;
if (StringUtils.isNotBlank(taskListQueryRequest.getServiceName())) {
QueryCondition queryCondition = new QueryCondition();
queryCondition.setServiceName(taskListQueryRequest.getServiceName());
taskDOPage = taskAccessService.findPageCriteria(taskListQueryRequest.getPageNum() - 1,
taskListQueryRequest.getPageSize(), queryCondition);
} else {
taskDOPage = taskAccessService.findPageNoCriteria(taskListQueryRequest.getPageNum() - 1,
taskListQueryRequest.getPageSize());
}
List<TaskModel> taskList = new ArrayList<>();
taskDOPage.forEach(taskDO -> {
TaskModel taskModel = new TaskModel();
taskModel.setTaskId(taskDO.getTaskId());
taskModel.setDestClusterId(taskDO.getDestClusterId());
taskModel.setSourceClusterId(taskDO.getSourceClusterId());
taskModel.setServiceName(taskDO.getServiceName());
taskModel.setGroupName(taskDO.getGroupName());
taskModel.setTaskStatus(taskDO.getTaskStatus());
taskList.add(taskModel);
});
List<TaskModel> taskList = taskDOPage.stream().map(TaskModel::from).toList();
taskListQueryResult.setTaskModels(taskList);
taskListQueryResult.setTotalPage(taskDOPage.getTotalPages());
taskListQueryResult.setTotalSize(taskDOPage.getTotalElements());

View File

@ -14,21 +14,19 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskUpdateRequest;
import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.template.Processor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
* @author NacosSync
@ -37,29 +35,33 @@ import com.alibaba.nacossync.template.Processor;
@Slf4j
@Service
public class TaskUpdateProcessor implements Processor<TaskUpdateRequest, BaseResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskUpdateProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskUpdateRequest taskUpdateRequest, BaseResult baseResult,
Object... others) throws Exception {
public void process(TaskUpdateRequest taskUpdateRequest, BaseResult baseResult, Object... others) throws Exception {
TaskDO taskDO = taskAccessService.findByTaskId(taskUpdateRequest.getTaskId());
if (!TaskStatusEnum.contains(taskUpdateRequest.getTaskStatus())) {
throw new SkyWalkerException(
"taskUpdateRequest.getTaskStatus() is not exist , value is :"
+ taskUpdateRequest.getTaskStatus());
"taskUpdateRequest.getTaskStatus() is not exist , value is :" + taskUpdateRequest.getTaskStatus());
}
if (null == taskDO) {
throw new SkyWalkerException("taskDo is null ,taskId is :"
+ taskUpdateRequest.getTaskId());
throw new SkyWalkerException("taskDo is null ,taskId is :" + taskUpdateRequest.getTaskId());
}
taskDO.setTaskStatus(taskUpdateRequest.getTaskStatus());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
taskAccessService.addTask(taskDO);
}
}

View File

@ -0,0 +1,252 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.timer;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.ListView;
import com.alibaba.nacos.client.naming.utils.CollectionUtils;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.event.DeleteAllSubTaskEvent;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.event.SyncTaskEvent;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeanUtils;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
/**
* when the database task service name is empty, check all the services in the cluster and create a synchronization
* task.
*/
@Slf4j
public class CheckRunningStatusAllNacosThread implements Runnable {
private final Map<String, Set<String>> subTaskService = new ConcurrentHashMap<>();
private final MetricsManager metricsManager;
private final TaskAccessService taskAccessService;
private final NacosServerHolder nacosServerHolder;
private final EventBus eventBus;
public CheckRunningStatusAllNacosThread(MetricsManager metricsManager, TaskAccessService taskAccessService,
NacosServerHolder nacosServerHolder, EventBus eventBus) {
this.metricsManager = metricsManager;
this.taskAccessService = taskAccessService;
this.nacosServerHolder = nacosServerHolder;
this.eventBus = eventBus;
}
/**
* Synchronize data based on the ns level.
*/
@Override
public void run() {
try {
List<TaskDO> tasks = taskAccessService.findAllByServiceNameEqualAll();
if (CollectionUtils.isEmpty(tasks)) {
return;
}
// Get the set of all task IDs
Set<String> taskIdSet = tasks.stream().map(TaskDO::getTaskId).collect(Collectors.toSet());
// Filter and handle sub-tasks that need to be deleted. This handles the case where tasks have been deleted
// but sub-tasks still exist.
subTaskService.entrySet().stream()
.filter(entry -> shouldDeleteSubTasks(entry, taskIdSet))
.forEach(entry -> postDeleteAllSubTaskEvent(entry.getKey()));
// Handle regular tasks
tasks.forEach(this::processTask);
} catch (Exception e) {
log.warn("CheckRunningStatusThread Exception", e);
metricsManager.recordError(MetricsStatisticsType.DISPATCHER_TASK);
}
}
/**
* Listens for the event of deleting all sub-tasks and handles the delete operation.
*
* @param deleteAllSubTaskEvent The event object containing the task information to be deleted.
*/
public void listenerDeleteAllTaskEvent(DeleteAllSubTaskEvent deleteAllSubTaskEvent) {
// Retrieve the task object
TaskDO task = deleteAllSubTaskEvent.getTaskDO();
// Retrieve the task ID
String taskId = task.getTaskId();
// Remove the set of service names corresponding to the task ID from subTaskService
Set<String> serviceNameSet = subTaskService.remove(taskId);
// If the set of service names is empty, return immediately
if (CollectionUtils.isEmpty(serviceNameSet)) {
return;
}
// Build the list of sub-tasks pending removal
List<TaskDO> servicesPendingRemoval = serviceNameSet.stream()
.map(serviceName -> buildSubTaskDO(task, serviceName))
.toList();
// Handle the removal of the pending sub-tasks
handleRemoval(servicesPendingRemoval, serviceNameSet);
}
private boolean shouldDeleteSubTasks(Map.Entry<String, Set<String>> entry, Set<String> taskIdSet) {
return !taskIdSet.contains(entry.getKey()) && !entry.getValue().isEmpty();
}
private void postDeleteAllSubTaskEvent(String taskId) {
TaskDO taskDO = new TaskDO();
taskDO.setTaskId(taskId);
eventBus.post(new DeleteAllSubTaskEvent(taskDO));
}
/**
* Processes the given task by determining the services that need to be inserted and removed,
* and performs the corresponding operations.
*
* @param task The task object to be processed.
*/
private void processTask(TaskDO task) {
// Retrieve the set of services for the task, creating a new set if it does not exist
Set<String> serviceSet = subTaskService.computeIfAbsent(task.getTaskId(), k -> ConcurrentHashMap.newKeySet());
// Get the list of all service names associated with the task
List<String> serviceNameList = getAllServiceName(task);
// Determine the services that need to be inserted (those not in the current service set)
List<TaskDO> servicesPendingInsertion = serviceNameList.stream()
.filter(serviceName -> !serviceSet.contains(serviceName))
.map(serviceName -> buildSubTaskDO(task, serviceName))
.toList();
// Determine the services that need to be removed (those in the current service set but not in the service name list)
List<TaskDO> servicesPendingRemoval = serviceSet.stream()
.filter(serviceName -> !serviceNameList.contains(serviceName))
.map(serviceName -> buildSubTaskDO(task, serviceName))
.toList();
// If all lists are empty, there is nothing to process
if (CollectionUtils.isEmpty(serviceNameList) && CollectionUtils.isEmpty(servicesPendingInsertion)
&& CollectionUtils.isEmpty(servicesPendingRemoval)) {
log.debug("No service found for task: {}", task.getTaskId());
return;
}
// If the task status is SYNC, handle the insertion of services
if (TaskStatusEnum.SYNC.getCode().equals(task.getTaskStatus())) {
handleInsertion(servicesPendingInsertion, serviceSet);
}
// Handle the removal of services
handleRemoval(servicesPendingRemoval, serviceSet);
if (TaskStatusEnum.DELETE.getCode().equals(task.getTaskStatus())) {
List<TaskDO> allSubTasks = serviceNameList.stream().map(serviceName -> buildSubTaskDO(task, serviceName))
.collect(Collectors.toList());
handleRemoval(allSubTasks, serviceSet);
}
}
/**
* Handles the insertion of services.
*
* @param servicesPendingInsertion The list of services to be inserted.
* @param serviceSet The set of services.
*/
private void handleInsertion(List<TaskDO> servicesPendingInsertion, Set<String> serviceSet) {
BatchTaskExecutor.batchOperation(servicesPendingInsertion, t -> {
eventBus.post(new SyncTaskEvent(t));
serviceSet.add(t.getServiceName());
});
}
/**
* Handles the removal of services.
*
* @param servicesPendingRemoval The list of services to be removed.
* @param serviceSet The set of services.
*/
private void handleRemoval(List<TaskDO> servicesPendingRemoval, Set<String> serviceSet) {
BatchTaskExecutor.batchOperation(servicesPendingRemoval, t -> {
eventBus.post(new DeleteTaskEvent(t));
serviceSet.remove(t.getServiceName());
});
}
/**
* Builds a sub-task object for the given task and service name.
*
* @param serviceName The service name.
* @return The constructed sub-task object.
*/
private static TaskDO buildSubTaskDO(TaskDO taskDO, String serviceName) {
TaskDO task = new TaskDO();
BeanUtils.copyProperties(taskDO, task);
task.setTaskId(SkyWalkerUtil.generateTaskId(serviceName, taskDO.getGroupName(), taskDO.getSourceClusterId(),
taskDO.getDestClusterId()));
task.setServiceName(serviceName);
return task;
}
/**
* Retrieves all service names associated with the given task.
*
* @return A list of service names.
*/
private List<String> getAllServiceName(TaskDO taskDO) {
NamingService namingService = nacosServerHolder.get(taskDO.getSourceClusterId());
if (namingService == null) {
log.warn("naming service is null or not found, clusterId:{}", taskDO.getSourceClusterId());
return Collections.emptyList();
}
try {
ListView<String> servicesOfServer = namingService.getServicesOfServer(0, Integer.MAX_VALUE,
taskDO.getGroupName());
return servicesOfServer.getData();
} catch (Exception e) {
log.error("query service list failure", e);
}
return Collections.emptyList();
}
}

View File

@ -14,24 +14,23 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.timer;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.FinishedTask;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;
/**
* @author NacosSync
@ -40,57 +39,53 @@ import com.alibaba.nacossync.pojo.model.TaskDO;
@Slf4j
@Service
public class CleanExceedOperationIdTimer implements CommandLineRunner {
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private TaskAccessService taskAccessService;
@Autowired
private ScheduledExecutorService scheduledExecutorService;
private static final long INITIAL_DELAY = 0;
private static final long PERIOD = 12;
private final SkyWalkerCacheServices skyWalkerCacheServices;
private final TaskAccessService taskAccessService;
private final ScheduledExecutorService scheduledExecutorService;
public CleanExceedOperationIdTimer(SkyWalkerCacheServices skyWalkerCacheServices,
TaskAccessService taskAccessService, ScheduledExecutorService scheduledExecutorService) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.taskAccessService = taskAccessService;
this.scheduledExecutorService = scheduledExecutorService;
}
@Override
public void run(String... args) {
/** Clean up the OperationId cache once every 12 hours */
scheduledExecutorService.scheduleWithFixedDelay(new CleanExceedOperationIdThread(), 0, 12,
TimeUnit.HOURS);
scheduledExecutorService.scheduleWithFixedDelay(new CleanExceedOperationIdThread(), INITIAL_DELAY, PERIOD,
TimeUnit.HOURS);
log.info("CleanExceedOperationIdTimer has started successfully");
}
private class CleanExceedOperationIdThread implements Runnable {
@Override
public void run() {
try {
Map<String, FinishedTask> finishedTaskMap = skyWalkerCacheServices
.getFinishedTaskMap();
Iterable<TaskDO> taskDOS = taskAccessService.findAll();
Set<String> operationIds = getDbOperations(taskDOS);
for (String operationId : finishedTaskMap.keySet()) {
if (!operationIds.contains(operationId)) {
finishedTaskMap.remove(operationId);
}
}
Map<String, FinishedTask> finishedTaskMap = skyWalkerCacheServices.getFinishedTaskMap();
Set<String> operationIds = getDbOperations(taskAccessService.findAll());
finishedTaskMap.keySet().removeIf(operationId -> !operationIds.contains(operationId));
} catch (Exception e) {
log.warn("CleanExceedOperationIdThread Exception", e);
}
}
private Set<String> getDbOperations(Iterable<TaskDO> taskDOS) {
Set<String> operationIds = new HashSet<>();
taskDOS.forEach(taskDO -> operationIds.add(taskDO.getOperationId()));
return operationIds;
return StreamSupport.stream(taskDOS.spliterator(), false).map(TaskDO::getOperationId)
.collect(Collectors.toSet());
}
}
}

View File

@ -22,14 +22,15 @@ import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.event.SyncTaskEvent;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
@ -40,27 +41,44 @@ import java.util.concurrent.TimeUnit;
@Slf4j
@Service
public class QuerySyncTaskTimer implements CommandLineRunner {
@Autowired
private MetricsManager metricsManager;
private static final int INITIAL_DELAY = 0;
private static final int DELAY = 3000;
private final MetricsManager metricsManager;
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
@Autowired
private EventBus eventBus;
@Autowired
private ScheduledExecutorService scheduledExecutorService;
private final EventBus eventBus;
private final ScheduledExecutorService scheduledExecutorService;
private final NacosServerHolder nacosServerHolder;
public QuerySyncTaskTimer(MetricsManager metricsManager, SkyWalkerCacheServices skyWalkerCacheServices,
TaskAccessService taskAccessService, EventBus eventBus, ScheduledExecutorService scheduledExecutorService,
NacosServerHolder nacosServerHolder) {
this.metricsManager = metricsManager;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.taskAccessService = taskAccessService;
this.eventBus = eventBus;
this.scheduledExecutorService = scheduledExecutorService;
this.nacosServerHolder = nacosServerHolder;
}
@Override
public void run(String... args) {
/** Fetch the task list from the database every 3 seconds */
scheduledExecutorService.scheduleWithFixedDelay(new CheckRunningStatusThread(), 0, 3000,
scheduledExecutorService.scheduleWithFixedDelay(new CheckRunningStatusThread(), INITIAL_DELAY, DELAY,
TimeUnit.MILLISECONDS);
scheduledExecutorService.scheduleWithFixedDelay(new CheckRunningStatusAllNacosThread(metricsManager,
taskAccessService, nacosServerHolder, eventBus), INITIAL_DELAY, DELAY,
TimeUnit.MILLISECONDS);
log.info("QuerySyncTaskTimer has started successfully");
}
private class CheckRunningStatusThread implements Runnable {
@ -68,10 +86,10 @@ public class QuerySyncTaskTimer implements CommandLineRunner {
@Override
public void run() {
Long start = System.currentTimeMillis();
long start = System.currentTimeMillis();
try {
Iterable<TaskDO> taskDOS = taskAccessService.findAll();
List<TaskDO> taskDOS = taskAccessService.findAllByServiceNameNotEqualAll();
taskDOS.forEach(taskDO -> {
@ -83,13 +101,13 @@ public class QuerySyncTaskTimer implements CommandLineRunner {
if (TaskStatusEnum.SYNC.getCode().equals(taskDO.getTaskStatus())) {
eventBus.post(new SyncTaskEvent(taskDO));
log.info("从数据库中查询到一个同步任务,发出一个同步事件:" + taskDO);
log.info("从数据库中查询到一个同步任务,发出一个同步事件:{}", taskDO);
}
if (TaskStatusEnum.DELETE.getCode().equals(taskDO.getTaskStatus())) {
eventBus.post(new DeleteTaskEvent(taskDO));
log.info("从数据库中查询到一个删除任务,发出一个同步事件:" + taskDO);
log.info("从数据库中查询到一个删除任务,发出一个同步事件:{}", taskDO);
}
});

View File

@ -17,7 +17,6 @@ import com.alibaba.nacossync.extension.event.SpecialSyncEvent;
import com.alibaba.nacossync.extension.event.SpecialSyncEventBus;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
@ -33,20 +32,25 @@ import java.util.concurrent.TimeUnit;
@Service
public class SpecialSyncEventTimer implements CommandLineRunner {
@Autowired
private SpecialSyncEventBus specialSyncEventBus;
private final SpecialSyncEventBus specialSyncEventBus;
@Autowired
private EventBus eventBus;
@Autowired
private ScheduledExecutorService scheduledExecutorService;
private final EventBus eventBus;
private final ScheduledExecutorService scheduledExecutorService;
public SpecialSyncEventTimer(SpecialSyncEventBus specialSyncEventBus, EventBus eventBus,
ScheduledExecutorService scheduledExecutorService) {
this.specialSyncEventBus = specialSyncEventBus;
this.eventBus = eventBus;
this.scheduledExecutorService = scheduledExecutorService;
}
@Override
public void run(String... args) throws Exception {
scheduledExecutorService.scheduleWithFixedDelay(new SpecialSyncEventTimer.SpecialSyncEventThread(), 0, 3000,
TimeUnit.MILLISECONDS);
log.info("SpecialSyncEventTimer has started successfully");
}
private class SpecialSyncEventThread implements Runnable {
@ -58,9 +62,9 @@ public class SpecialSyncEventTimer implements CommandLineRunner {
allSpecialSyncEvent.stream()
.filter(specialSyncEvent -> TaskStatusEnum.SYNC.getCode()
.equals(specialSyncEvent.getTaskDO().getTaskStatus()))
.forEach(specialSyncEvent -> eventBus.post(specialSyncEvent));
.forEach(eventBus::post);
} catch (Exception e) {
log.warn("SpecialSyncEventThread Exception", e);
log.error("Exception occurred while processing special sync events", e);
}
}

View File

@ -0,0 +1,104 @@
package com.alibaba.nacossync.util;
import com.alibaba.nacos.client.naming.utils.CollectionUtils;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.base.Stopwatch;
import lombok.extern.slf4j.Slf4j;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.*;
import java.util.function.Consumer;
@Slf4j
public class BatchTaskExecutor {
private static final int MAX_THREAD_NUM = 200;
private static final ExecutorService executorService = Executors.newFixedThreadPool(MAX_THREAD_NUM);
/**
* Batch operation method
*
* @param items Task list
* @param operation Operation to be executed
*/
public static void batchOperation(List<TaskDO> items, Consumer<TaskDO> operation) {
Stopwatch stopwatch = Stopwatch.createStarted();
List<Tuple<Integer, List<TaskDO>>> taskGroupList = averageAssign(items, MAX_THREAD_NUM);
// Create a CompletableFuture for each task group
CompletableFuture<?>[] futures = taskGroupList.stream().map(tuple -> CompletableFuture.runAsync(() -> {
for (TaskDO taskDO : tuple.getT2()) {
try {
// Add timeout control for each task to avoid long-running tasks
CompletableFuture.runAsync(() -> operation.accept(taskDO), executorService)
.orTimeout(5, TimeUnit.SECONDS) // Task timeout set to 5 seconds
.exceptionally(ex -> {
log.error("Task execution timed out: {}", taskDO.getServiceName(), ex);
return null;
}).join();
} catch (Exception e) {
log.error("Error occurred during task execution: {}", taskDO.getServiceName(), e);
}
}
}, executorService)).toArray(CompletableFuture[]::new);
try {
// Wait for all tasks to complete
CompletableFuture.allOf(futures).join();
} catch (Exception e) {
log.error("Error occurred during sync operation", e);
} finally {
log.info("Total sync tasks: {}, Execution time: {} ms", items.size(), stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}
/**
* Divide a list into n sublists, mainly implemented by offset
* @param source collection to be divided
* @param limit maximum value
* @return list after division
* @param <T> object type
*/
private static <T> List<Tuple<Integer, List<T>>> averageAssign(List<T> source, int limit) {
if (CollectionUtils.isEmpty(source)) {
return Collections.emptyList();
}
int size = source.size();
int listCount = (int) Math.ceil((double) size / limit); // Calculate the number of sublists
int remainder = size % listCount; // Calculate the number of remaining elements after even distribution
List<Tuple<Integer, List<T>>> result = new ArrayList<>(listCount); // Initialize the result list with the expected size
for (int i = 0, assigned = 0; i < listCount; i++) {
int sublistSize = size / listCount + (remainder-- > 0 ? 1 : 0); // Determine the size of each sublist, distribute remaining elements
List<T> sublist = new ArrayList<>(source.subList(assigned, assigned + sublistSize)); // Create the sublist
result.add(Tuple.of(i, sublist)); // Add the sublist to the result
assigned += sublistSize; // Update the assigned index
}
return result;
}
/**
* Shutdown the executor service to avoid resource leakage
*/
public static void shutdown() {
executorService.shutdown();
try {
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
executorService.shutdownNow();
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
log.error("Executor service did not terminate");
}
}
} catch (InterruptedException ie) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
}

View File

@ -1,19 +0,0 @@
package com.alibaba.nacossync.util;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
public abstract class Collections {
public static Collection subtract(final Collection a, final Collection b) {
ArrayList list = new ArrayList( a );
for (Iterator it = b.iterator(); it.hasNext();) {
list.remove(it.next());
}
return list;
}
}

View File

@ -1,15 +1,3 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE
* file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file
* to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
* License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.util;
import com.alibaba.nacos.client.naming.utils.CollectionUtils;
@ -20,16 +8,21 @@ import java.util.Map;
import java.util.stream.Collectors;
/**
* @author paderlol
* @date: 2019-04-25 00:01
* Utility class for handling Consul metadata.
*/
public class ConsulUtils {
public static Map<String, String> transferMetadata(List<String> tags) {
Map<String, String> metadata = new HashMap<>();
if (!CollectionUtils.isEmpty(tags)) {
return tags.stream().filter(tag -> tag.split("=", -1).length == 2).map(tag -> tag.split("=", -1))
.collect(Collectors.toMap(tagSplitArray -> tagSplitArray[0], tagSplitArray -> tagSplitArray[1]));
if (CollectionUtils.isEmpty(tags)) {
return new HashMap<>();
}
return metadata;
return tags.stream()
.map(tag -> tag.split("=", -1))
.filter(tagArray -> tagArray.length == 2)
.collect(Collectors.toMap(
tagArray -> tagArray[0],
tagArray -> tagArray[1],
(existing, replacement) -> existing // In case of duplicate keys, keep the existing value
));
}
}

View File

@ -20,7 +20,6 @@ import org.apache.commons.lang3.StringUtils;
import java.math.BigDecimal;
import java.util.List;
import java.util.Map;
import java.util.function.Predicate;
/**
* @author paderlol
@ -49,45 +48,51 @@ public final class DubboConstants {
public static final String DUBBO_ROOT_PATH = "/dubbo";
public static final String ALL_SERVICE_NAME_PATTERN = "*";
/**
* if Dubbo version greater than 2.7.2, service name is providers:interface:version:
* if Dubbo version less than 2.7.2, service name is providers:interface:version
* @param queryParam
* @return
* Creates a service name based on Dubbo version compatibility.
* if Dubbo version greater than 2.7.2, service name is providers:interface:version:
* if Dubbo version less than 2.7.2, service name is providers:interface:version
*
* @param queryParam the query parameters that include keys such as interface, version, group, etc.
* @return the constructed service name string
*/
public static String createServiceName(Map<String, String> queryParam) {
String group = queryParam.get(GROUP_KEY);
String release = queryParam.get(RELEASE_KEY);
Predicate<String> isBlankGroup = StringUtils::isBlank;
Predicate<String> isNotBlankRelease = StringUtils::isNotBlank;
String serviceName = Joiner.on(SEPARATOR_KEY).skipNulls().join(CATALOG_KEY, queryParam.get(INTERFACE_KEY),
queryParam.get(VERSION_KEY), group);
//TODO The code here is to deal with service metadata format problems caused by dubbo version incompatibility
if (isBlankGroup.test(group) && isNotBlankRelease.test(release)) {
String baseServiceName = Joiner.on(SEPARATOR_KEY).skipNulls().join(CATALOG_KEY, queryParam.get(INTERFACE_KEY),
queryParam.get(VERSION_KEY), group);
if (StringUtils.isBlank(group) && StringUtils.isNotBlank(release)) {
List<String> versions = Splitter.on(RELEASE_SEPARATOR_KEY).splitToList(release);
if (!CollectionUtils.isEmpty(versions) && versions.size() >= DUBBO_VERSION_INDEX) {
String firstVersion = versions.get(0);
String secondVersion = versions.get(1);
if (DUBBO_VERSION_INDEX == Integer.parseInt(firstVersion)) {
if (MIDDLE_DUBBO_VERSION_INDEX <= versions.size()) {
String thirdVersion = versions.get(2);
BigDecimal bigDecimal =
new BigDecimal(Joiner.on(RELEASE_SEPARATOR_KEY).join(secondVersion, thirdVersion));
if (bigDecimal.compareTo(COMPARE_NUMBER) > 0) {
serviceName = serviceName.concat(SEPARATOR_KEY);
}
} else if (versions.size() == DUBBO_VERSION_INDEX && Integer.parseInt(secondVersion) > 7) {
serviceName = serviceName.concat(SEPARATOR_KEY);
}
} else if (MIN_DUBBO_VERSION < Integer.parseInt(firstVersion)) {
serviceName = serviceName.concat(SEPARATOR_KEY);
BigDecimal bigDecimal = new BigDecimal(Joiner.on(RELEASE_SEPARATOR_KEY).join(secondVersion,
versions.size() > 2 ? versions.get(2) : "0"));
if (isVersionRequiresSeparator(firstVersion, secondVersion, bigDecimal)) {
baseServiceName = baseServiceName.concat(SEPARATOR_KEY);
}
}
}
return serviceName;
return baseServiceName;
}
/**
* Checks if the version requires a separator to be appended to the service name.
*
* @param firstVersion the major version
* @param secondVersion the minor version
* @param bigDecimal the version number as BigDecimal
* @return true if separator should be added, otherwise false
*/
private static boolean isVersionRequiresSeparator(String firstVersion, String secondVersion, BigDecimal bigDecimal) {
int majorVersion = Integer.parseInt(firstVersion);
int minorVersion = Integer.parseInt(secondVersion);
return (DUBBO_VERSION_INDEX == majorVersion && (MIDDLE_DUBBO_VERSION_INDEX <= minorVersion ||
bigDecimal.compareTo(COMPARE_NUMBER) > 0)) || (MIN_DUBBO_VERSION < majorVersion);
}
}

View File

@ -1,19 +1,3 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.util;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
@ -22,135 +6,139 @@ import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.ClusterAddRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.google.common.base.Joiner;
import org.apache.commons.lang3.StringUtils;
import java.io.UnsupportedEncodingException;
import java.net.Inet4Address;
import java.net.InetAddress;
import java.net.NetworkInterface;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Enumeration;
import java.util.UUID;
/**
* @author NacosSync
* @version $Id: SkyWalkerUtil.java, v 0.1 2018-09-26 AM12:10 NacosSync Exp $$
*/
* Utility class for various operations in SkyWalker.
*/
public class SkyWalkerUtil {
private static final String SEPARATOR = ":";
private static final String MD5_ALGORITHM = "MD5";
/**
* Generates an MD5 hash for the given string.
*
* Gets the string md5
* @param value
* @return
* @param value The string to be encrypted.
* @return The encrypted string, or an empty string if encryption fails.
*/
public static String StringToMd5(String value) {
{
try {
MessageDigest md5 = MessageDigest.getInstance("MD5");
md5.update(value.getBytes("UTF-8"));
byte[] encryption = md5.digest();
StringBuffer strBuf = new StringBuffer();
for (int i = 0; i < encryption.length; i++) {
if (Integer.toHexString(0xff & encryption[i]).length() == 1) {
strBuf.append("0").append(Integer.toHexString(0xff & encryption[i]));
} else {
strBuf.append(Integer.toHexString(0xff & encryption[i]));
}
}
return strBuf.toString();
} catch (NoSuchAlgorithmException e) {
return "";
} catch (UnsupportedEncodingException e) {
return "";
public static String stringToMd5(String value) {
if (StringUtils.isBlank(value)) {
return "";
}
try {
MessageDigest md5 = MessageDigest.getInstance(MD5_ALGORITHM);
byte[] encryption = md5.digest(value.getBytes(StandardCharsets.UTF_8));
StringBuilder strBuf = new StringBuilder();
for (byte b : encryption) {
strBuf.append(String.format("%02x", b));
}
return strBuf.toString();
} catch (NoSuchAlgorithmException e) {
return "";
}
}
/**
* The rules of generating taskId
* @param addTaskRequest
* @return
* Generates a task ID based on the given TaskAddRequest.
*
* @param addTaskRequest The TaskAddRequest containing task details.
* @return The generated task ID.
*/
public static String generateTaskId(TaskAddRequest addTaskRequest) {
return generateTaskId(addTaskRequest.getServiceName(), addTaskRequest.getGroupName(),
addTaskRequest.getSourceClusterId(), addTaskRequest.getDestClusterId());
}
/**
* The rules of generating taskId
* Generates a task ID based on the given parameters.
*
* @return
* @param serviceName The service name.
* @param groupName The group name.
* @param sourceClusterId The source cluster ID.
* @param destClusterId The destination cluster ID.
* @return The generated task ID.
*/
public static String generateTaskId(String serviceName, String groupName,
String sourceClusterId, String destClusterId) {
StringBuilder sb = new StringBuilder();
sb.append(serviceName);
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(groupName);
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(sourceClusterId);
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(destClusterId);
return SkyWalkerUtil.StringToMd5(sb.toString());
String sourceClusterId, String destClusterId) {
String rawId = String.join(SkyWalkerConstants.UNDERLINE, serviceName, groupName, sourceClusterId, destClusterId);
return stringToMd5(rawId);
}
/**
* 生成集群clusterId的规则
* Generates a cluster ID based on the given ClusterAddRequest.
*
* @param addClusterRequest
* @return
* @param addClusterRequest The ClusterAddRequest containing cluster details.
* @return The generated cluster ID.
*/
public static String generateClusterId(ClusterAddRequest addClusterRequest) {
StringBuilder sb = new StringBuilder();
sb.append(addClusterRequest.getClusterName());
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(addClusterRequest.getClusterType());
return SkyWalkerUtil.StringToMd5(sb.toString());
String rawId = String.join(SkyWalkerConstants.UNDERLINE, addClusterRequest.getClusterName(), addClusterRequest.getClusterType());
return stringToMd5(rawId);
}
/**
* Avoid getting a return address
* @return
* @throws Exception
* Gets the local IP address, avoiding loopback addresses.
*
* @return The local IP address.
* @throws Exception If an error occurs while fetching the IP address.
*/
public static String getLocalIp() throws Exception {
InetAddress addr = InetAddress.getLocalHost();
String localIp = addr.getHostAddress();
if (addr.isLoopbackAddress()) {
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface in = interfaces.nextElement();
Enumeration<InetAddress> addrs = in.getInetAddresses();
while (addrs.hasMoreElements()) {
InetAddress address = addrs.nextElement();
if (!address.isLoopbackAddress() && address instanceof Inet4Address) {
localIp = address.getHostAddress();
}
if (!addr.isLoopbackAddress()) {
return addr.getHostAddress();
}
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface networkInterface = interfaces.nextElement();
Enumeration<InetAddress> addrs = networkInterface.getInetAddresses();
while (addrs.hasMoreElements()) {
InetAddress address = addrs.nextElement();
if (!address.isLoopbackAddress() && address instanceof Inet4Address) {
return address.getHostAddress();
}
}
}
return localIp;
return addr.getHostAddress();
}
/**
* Generates a synchronization key based on source and destination cluster types.
*
* @param sourceClusterType The source cluster type.
* @param destClusterType The destination cluster type.
* @return The generated synchronization key.
*/
public static String generateSyncKey(ClusterTypeEnum sourceClusterType, ClusterTypeEnum destClusterType) {
return Joiner.on(":").join(sourceClusterType.getCode(), destClusterType.getCode());
return Joiner.on(SEPARATOR).join(sourceClusterType.getCode(), destClusterType.getCode());
}
/**
* Gets the operation ID from the given TaskDO.
*
* @param taskDO The TaskDO containing the operation ID.
* @return The operation ID.
*/
public static String getOperationId(TaskDO taskDO) {
return taskDO.getOperationId();
}
/**
* Generates a unique operation ID.
*
* @return The generated operation ID.
*/
public static String generateOperationId() {
return UUID.randomUUID().toString();
}
}

View File

@ -12,6 +12,17 @@
*/
package com.alibaba.nacossync.util;
import com.google.common.base.Joiner;
import lombok.extern.slf4j.Slf4j;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_PATH_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_URL_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_IP_KEY;
@ -20,17 +31,6 @@ import static com.alibaba.nacossync.util.DubboConstants.INTERFACE_KEY;
import static com.alibaba.nacossync.util.DubboConstants.PROTOCOL_KEY;
import static com.alibaba.nacossync.util.DubboConstants.ZOOKEEPER_SEPARATOR;
import com.google.common.base.Joiner;
import com.google.common.collect.Maps;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import lombok.extern.slf4j.Slf4j;
/**
* @author paderlol
* @date: 2018-12-25 21:08
@ -56,9 +56,9 @@ public final class StringUtils {
*/
private static Map<String, String> parseKeyValuePair(String str, String itemSeparator) {
String[] tmp = str.split(itemSeparator);
Map<String, String> map = new HashMap<String, String>(tmp.length);
for (int i = 0; i < tmp.length; i++) {
Matcher matcher = KVP_PATTERN.matcher(tmp[i]);
Map<String, String> map = new HashMap<>(tmp.length);
for (String s : tmp) {
Matcher matcher = KVP_PATTERN.matcher(s);
if (!matcher.matches()) {
continue;
}
@ -74,19 +74,14 @@ public final class StringUtils {
* @return Parameters instance.
*/
public static Map<String, String> parseQueryString(String qs) {
try {
String decodePath = URLDecoder.decode(qs, "UTF-8");
if (isEmpty(decodePath)) {
return new HashMap<>();
}
decodePath = substringAfter(decodePath, "?");
return parseKeyValuePair(decodePath, "&");
} catch (UnsupportedEncodingException e) {
log.warn("parse query string failed", e);
return Maps.newHashMap();
String decodePath = URLDecoder.decode(qs, StandardCharsets.UTF_8);
if (isEmpty(decodePath)) {
return new HashMap<>();
}
decodePath = substringAfter(decodePath, "?");
return parseKeyValuePair(decodePath, "&");
}
/**
@ -100,27 +95,22 @@ public final class StringUtils {
}
public static Map<String, String> parseIpAndPortString(String path) {
try {
String decodePath = URLDecoder.decode(path, "UTF-8");
Matcher matcher = IP_PORT_PATTERN.matcher(decodePath);
// extract the ones that match the rules
Map<String, String> instanceMap = new HashMap<>(3);
while (matcher.find()) {
// protocol
instanceMap.put(PROTOCOL_KEY, matcher.group(1));
// ip address
instanceMap.put(INSTANCE_IP_KEY, matcher.group(2));
// port
instanceMap.put(INSTANCE_PORT_KEY, matcher.group(3));
break;
}
return instanceMap;
} catch (UnsupportedEncodingException e) {
log.warn("parse query string failed", e);
return Maps.newHashMap();
String decodePath = URLDecoder.decode(path, StandardCharsets.UTF_8);
Matcher matcher = IP_PORT_PATTERN.matcher(decodePath);
// extract the ones that match the rules
Map<String, String> instanceMap = new HashMap<>(3);
while (matcher.find()) {
// protocol
instanceMap.put(PROTOCOL_KEY, matcher.group(1));
// ip address
instanceMap.put(INSTANCE_IP_KEY, matcher.group(2));
// port
instanceMap.put(INSTANCE_PORT_KEY, matcher.group(3));
break;
}
return instanceMap;
}
/**
@ -169,17 +159,12 @@ public final class StringUtils {
public static String convertDubboFullPathForZk(Map<String, String> metaData, String providersPath, String ip,
int port) {
try {
String urlParam = Joiner.on("&").withKeyValueSeparator("=").join(metaData);
String instanceUrl = String.format(DUBBO_URL_FORMAT, metaData.get(PROTOCOL_KEY), ip, port,
metaData.get(INTERFACE_KEY), urlParam);
return Joiner.on(ZOOKEEPER_SEPARATOR).join(providersPath, URLEncoder.encode(instanceUrl, "UTF-8"));
} catch (UnsupportedEncodingException e) {
log.warn("convert Dubbo full path", e);
return "";
}
String urlParam = Joiner.on("&").withKeyValueSeparator("=").join(metaData);
String instanceUrl = String.format(DUBBO_URL_FORMAT, metaData.get(PROTOCOL_KEY), ip, port,
metaData.get(INTERFACE_KEY), urlParam);
return Joiner.on(ZOOKEEPER_SEPARATOR).join(providersPath, URLEncoder.encode(instanceUrl, StandardCharsets.UTF_8));
}
public static boolean isDubboProviderPath(String path) {

View File

@ -0,0 +1,207 @@
/*
* Copyright (c) 2016-2021 VMware Inc. or its affiliates, All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.util;
import reactor.util.annotation.NonNull;
import reactor.util.annotation.Nullable;
import reactor.util.function.Tuple2;
import reactor.util.function.Tuples;
import java.io.Serializable;
import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Objects;
import java.util.function.Function;
/**
* A tuple that holds two non-null values.
*
* @param <T1> The type of the first non-null value held by this tuple
* @param <T2> The type of the second non-null value held by this tuple
* @author Jon Brisbin
* @author Stephane Maldini
*/
@SuppressWarnings("rawtypes")
public class Tuple<T1, T2> implements Iterable<Object>, Serializable {
private static final long serialVersionUID = -3518082018884860684L;
@NonNull
final T1 t1;
@NonNull
final T2 t2;
Tuple(T1 t1, T2 t2) {
this.t1 = Objects.requireNonNull(t1, "t1");
this.t2 = Objects.requireNonNull(t2, "t2");
}
/**
* Type-safe way to get the first object of this {@link Tuples}.
*
* @return The first object
*/
public T1 getT1() {
return t1;
}
/**
* Type-safe way to get the second object of this {@link Tuples}.
*
* @return The second object
*/
public T2 getT2() {
return t2;
}
/**
* Map the left-hand part (T1) of this {@link Tuple} into a different value and type, keeping the right-hand part
* (T2).
*
* @param mapper the mapping {@link Function} for the left-hand part
* @param <R> the new type for the left-hand part
* @return a new {@link Tuple2} with a different left (T1) value
*/
public <R> Tuple<R, T2> mapT1(Function<T1, R> mapper) {
return new Tuple<>(mapper.apply(t1), t2);
}
/**
* Map the right-hand part (T2) of this {@link Tuple} into a different value and type, keeping the left-hand part
* (T1).
*
* @param mapper the mapping {@link Function} for the right-hand part
* @param <R> the new type for the right-hand part
* @return a new {@link Tuple2} with a different right (T2) value
*/
public <R> Tuple<T1, R> mapT2(Function<T2, R> mapper) {
return new Tuple<>(t1, mapper.apply(t2));
}
/**
* Get the object at the given index.
*
* @param index The index of the object to retrieve. Starts at 0.
* @return The object or {@literal null} if out of bounds.
*/
@Nullable
public Object get(int index) {
switch (index) {
case 0:
return t1;
case 1:
return t2;
default:
return null;
}
}
/**
* Turn this {@code Tuple} into a {@link List List&lt;Object&gt;}. The list isn't tied to this Tuple but is a
* <strong>copy</strong> with limited mutability ({@code add} and {@code remove} are not supported, but {@code set}
* is).
*
* @return A copy of the tuple as a new {@link List List&lt;Object&gt;}.
*/
public List<Object> toList() {
return Arrays.asList(toArray());
}
/**
* Turn this {@code Tuple} into a plain {@code Object[]}. The array isn't tied to this Tuple but is a
* <strong>copy</strong>.
*
* @return A copy of the tuple as a new {@link Object Object[]}.
*/
public Object[] toArray() {
return new Object[] {t1, t2};
}
/**
* Return an <strong>immutable</strong> {@link Iterator Iterator&lt;Object&gt;} around the content of this
* {@code Tuple}.
*
* @return An unmodifiable {@link Iterator} over the elements in this Tuple.
* @implNote As an {@link Iterator} is always tied to its {@link Iterable} source by definition, the iterator cannot
* be mutable without the iterable also being mutable. Since {@link Tuples} are <strong>immutable</strong>, so is
* the {@link Iterator} returned by this method.
*/
@Override
public Iterator<Object> iterator() {
return Collections.unmodifiableList(toList()).iterator();
}
@Override
public boolean equals(@Nullable Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
Tuple<?, ?> tuple = (Tuple<?, ?>) o;
return t1.equals(tuple.t1) && t2.equals(tuple.t2);
}
@Override
public int hashCode() {
int result = size();
result = 31 * result + t1.hashCode();
result = 31 * result + t2.hashCode();
return result;
}
/**
* Return the number of elements in this {@literal Tuples}.
*
* @return The size of this {@literal Tuples}.
*/
public int size() {
return 2;
}
/**
* A Tuple String representation is the comma separated list of values, enclosed in square brackets.
*
* @return the Tuple String representation
*/
@Override
public final String toString() {
Object[] values = toArray();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < values.length; i++) {
Object t = values[i];
if (i != 0) {
sb.append(',');
}
if (t != null) {
sb.append(t);
}
}
return sb.insert(0, '[').append(']').toString();
}
public static <T1, T2> Tuple<T1, T2> of(T1 t1, T2 t2) {
return new Tuple<>(t1, t2);
}
}

View File

@ -11,6 +11,5 @@ spring.cloud.discovery.enabled=false
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/nacos_sync?characterEncoding=utf8
spring.datasource.username=root
spring.datasource.password=root
management.endpoints.web.exposure.include=*
management.endpoint.health.show-details=always

View File

@ -4,12 +4,13 @@
<springProperty name="logPath" scope="context" source="nacosSync.logs.path" defaultValue="${nacosSync.home}/logs"/>
<property name="LOG_HOME" value="${logPath}" />
<appender name="FILEINFO" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_HOME}/nacosSync.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_HOME}/log-nacosSync-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>200MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<file>${LOG_HOME}/nacos-sync.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_HOME}/log-nacos-sync-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<maxFileSize>200MB</maxFileSize>
<maxHistory>7</maxHistory>
<totalSizeCap>2GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<append>true</append>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
@ -25,7 +26,6 @@
</appender>
<logger name="com.alibaba.nacossync" level="INFO"/>
<logger name="org.hibernate.type.descriptor.sql.BasicBinder" level="INFO"/>
<logger name="org.hibernate.type.descriptor.sql.BasicExtractor" level="INFO"/>
<logger name="org.hibernate.SQL" level="INFO"/>

File diff suppressed because one or more lines are too long

View File

@ -67,7 +67,7 @@ public class ConsulSyncToNacosServiceImplTest {
TaskDO taskDO = mock(TaskDO.class);
mockSync(taskDO);
// TODO Test the core logic in the future
Assert.assertTrue(consulSyncToNacosService.sync(taskDO));
Assert.assertTrue(consulSyncToNacosService.sync(taskDO,null));
}
@Test
@ -80,7 +80,7 @@ public class ConsulSyncToNacosServiceImplTest {
@Test(expected = Exception.class)
public void testConsulSyncToNacosWithException() throws Exception {
Assert.assertFalse(consulSyncToNacosService.sync(null));
Assert.assertFalse(consulSyncToNacosService.sync(null,null));
}
@Test(expected = Exception.class)
@ -91,7 +91,7 @@ public class ConsulSyncToNacosServiceImplTest {
public void mockSync(TaskDO taskDO) throws Exception {
Instance instance = mock(Instance.class);
Map<String, String> metadata = Maps.newHashMap();
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, TEST_SOURCE_CLUSTER_ID);
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, TEST_SOURCE_CLUSTER_ID);
HealthService healthServiceUp = buildHealthService(TEST_INSTANCE_ADDRESS, 8080, Maps.newHashMap());
HealthService healthServiceDown = buildHealthService(TEST_INSTANCE_ADDRESS, 8081, metadata);
List<HealthService> healthServiceList = Lists.newArrayList(healthServiceUp, healthServiceDown);
@ -129,7 +129,7 @@ public class ConsulSyncToNacosServiceImplTest {
when(taskDO.getDestClusterId()).thenReturn(TEST_DEST_CLUSTER_ID);
doReturn(destNamingService).when(nacosServerHolder).get(anyString());
Map<String, String> metadata = Maps.newHashMap();
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, TEST_SOURCE_CLUSTER_ID);
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, TEST_SOURCE_CLUSTER_ID);
List<Instance> allInstances = Lists.newArrayList(instance);
doReturn(allInstances).when(destNamingService).getAllInstances(anyString());
doReturn(metadata).when(instance).getMetadata();

View File

@ -61,7 +61,7 @@ public class EurekaSyncToNacosServiceImplTest {
public void testEurekaSyncToNacos() throws Exception {
TaskDO taskDO = mock(TaskDO.class);
mockSync(taskDO);
Assert.assertTrue(eurekaSyncToNacosService.sync(taskDO));
Assert.assertTrue(eurekaSyncToNacosService.sync(taskDO,null));
}
@ -74,7 +74,7 @@ public class EurekaSyncToNacosServiceImplTest {
}
@Test(expected = Exception.class)
public void testEurekaSyncToNacosWithException() throws Exception {
Assert.assertFalse(eurekaSyncToNacosService.sync(null));
Assert.assertFalse(eurekaSyncToNacosService.sync(null, null));
}
@Test(expected = Exception.class)
public void testEurekaDeleteToNacosWithException() throws Exception {
@ -111,7 +111,7 @@ public class EurekaSyncToNacosServiceImplTest {
List<InstanceInfo> eurekaInstances = Lists.newArrayList();
doReturn(eurekaInstances).when(eurekaNamingService).getApplications(anyString());
Map<String, String> metadata = Maps.newHashMap();
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, TEST_SOURCE_CLUSTER_ID);
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, TEST_SOURCE_CLUSTER_ID);
List<Instance> allInstances = Lists.newArrayList(instance);
doReturn(allInstances).when(destNamingService).getAllInstances(anyString());
doReturn(metadata).when(instance).getMetadata();

View File

@ -1,12 +1,5 @@
package com.alibaba.nacossync.extension.impl;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.anyString;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
@ -14,8 +7,6 @@ import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import java.util.List;
import java.util.Map;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
@ -24,6 +15,16 @@ import org.mockito.Mock;
import org.mockito.Spy;
import org.mockito.runners.MockitoJUnitRunner;
import java.util.List;
import java.util.Map;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.anyString;
import static org.mockito.Mockito.doNothing;
import static org.mockito.Mockito.doReturn;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
/**
* @author paderlol
* @date: 2019-01-12 20:58
@ -51,9 +52,10 @@ public class NacosSyncToNacosServiceImplTest {
TaskDO taskDO = mock(TaskDO.class);
mockSync(taskDO);
// TODO Test the core logic in the future
Assert.assertTrue(nacosSyncToNacosService.sync(taskDO));
Assert.assertTrue(nacosSyncToNacosService.sync(taskDO,null));
}
@Test
public void testDeleteSyncToNacos() throws Exception {
TaskDO taskDO = mock(TaskDO.class);
@ -64,7 +66,7 @@ public class NacosSyncToNacosServiceImplTest {
@Test(expected = Exception.class)
public void testNacosSyncToNacosWithException() throws Exception {
Assert.assertFalse(nacosSyncToNacosService.sync(null));
Assert.assertFalse(nacosSyncToNacosService.sync(null, null));
}
@Test(expected = Exception.class)
public void testNacosDeleteToNacosWithException() throws Exception {
@ -88,7 +90,7 @@ public class NacosSyncToNacosServiceImplTest {
doReturn(sourceNamingService).when(nacosServerHolder).get(anyString());
doNothing().when(sourceNamingService).unsubscribe(any(), any());
Map<String, String> metadata = Maps.newHashMap();
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, TEST_SOURCE_CLUSTER_ID);
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, TEST_SOURCE_CLUSTER_ID);
List<Instance> allInstances = Lists.newArrayList(instance);
doReturn(allInstances).when(sourceNamingService).getAllInstances(anyString());
doReturn(metadata).when(instance).getMetadata();

View File

@ -60,7 +60,7 @@ public class NacosSyncToZookeeperServiceImplTest {
@Test(expected = Exception.class)
public void testNacosSyncToZookeeperWithException() throws Exception {
Assert.assertFalse(nacosSyncToZookeeperService.sync(null));
Assert.assertFalse(nacosSyncToZookeeperService.sync(null,null));
}
@Test(expected = Exception.class)
public void testNacosDeleteToZookeeperWithException() throws Exception {
@ -75,7 +75,7 @@ public class NacosSyncToZookeeperServiceImplTest {
doReturn(sourceNamingService).when(nacosServerHolder).get(any());
//TODO Test the core logic in the future
return nacosSyncToZookeeperService.sync(taskDO);
return nacosSyncToZookeeperService.sync(taskDO, null);
}
public boolean mockDelete(TaskDO taskDO) throws Exception {

View File

@ -71,7 +71,7 @@ public class ZookeeperSyncToNacosServiceImplTest {
public void testZookeeperDeleteToNacos() throws Exception {
TaskDO taskDO = mock(TaskDO.class);
mockSync(taskDO);
zookeeperSyncToNacosService.sync(taskDO);
zookeeperSyncToNacosService.sync(taskDO,null);
Assert.assertTrue(mockDelete(taskDO));
@ -79,7 +79,7 @@ public class ZookeeperSyncToNacosServiceImplTest {
@Test(expected = Exception.class)
public void tesZookeeperSyncToNacosWithException() throws Exception {
Assert.assertFalse(zookeeperSyncToNacosService.sync(null));
Assert.assertFalse(zookeeperSyncToNacosService.sync(null, null));
}
@Test(expected = Exception.class)
@ -100,14 +100,14 @@ public class ZookeeperSyncToNacosServiceImplTest {
when(treeCache.getCurrentData(any())).thenReturn(childData);
doReturn(ClusterTypeEnum.ZK).when(skyWalkerCacheServices).getClusterType(any());
when(treeCache.getListenable()).thenReturn(listeners);
return zookeeperSyncToNacosService.sync(taskDO);
return zookeeperSyncToNacosService.sync(taskDO,null);
}
public boolean mockDelete(TaskDO taskDO) throws Exception {
doNothing().when(treeCache).close();
Instance instance = mock(Instance.class);
Map<String, String> metadata = Maps.newHashMap();
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, TEST_SOURCE_CLUSTER_ID);
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, TEST_SOURCE_CLUSTER_ID);
List<Instance> allInstances = Lists.newArrayList(instance);
doReturn(allInstances).when(destNamingService).getAllInstances(anyString());

View File

@ -0,0 +1,60 @@
package com.alibaba.nacossync.utils;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class BatchTaskExecutorTest {
/**
* Test the batch operation to ensure all tasks are executed.
*/
@Test
public void testBatchOperation() {
// Prepare a list of tasks
List<TaskDO> tasks = new ArrayList<>();
for (int i = 0; i < 10; i++) {
TaskDO task = new TaskDO();
task.setServiceName("Service" + i);
tasks.add(task);
}
// Create a counter to track the number of executed tasks
AtomicInteger counter = new AtomicInteger(0);
// Execute batch operation
BatchTaskExecutor.batchOperation(tasks, task -> counter.incrementAndGet());
// Verify that all tasks have been executed
assertEquals(tasks.size(), counter.get());
}
/**
* Test the averageAssign method to ensure the correct distribution of tasks.
*/
/**
* Test the batch operation with an empty list to ensure no errors occur.
*/
@Test
public void testBatchOperationWithEmptyList() {
// Prepare an empty list of tasks
List<TaskDO> tasks = new ArrayList<>();
// Create a counter to track the number of executed tasks
AtomicInteger counter = new AtomicInteger(0);
// Execute batch operation
BatchTaskExecutor.batchOperation(tasks, task -> counter.incrementAndGet());
// Verify that no tasks have been executed
assertEquals(0, counter.get());
}
}

View File

@ -0,0 +1,81 @@
package com.alibaba.nacossync.utils;
import com.alibaba.nacossync.util.ConsulUtils;
import org.junit.jupiter.api.Test;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class ConsulUtilsTest {
/**
* Test the transferMetadata method with valid tags containing key-value pairs.
* This test ensures that all valid tags are correctly parsed into the resulting metadata map.
*/
@Test
public void testTransferMetadata_withValidTags() {
// Test with valid tags containing key-value pairs
List<String> tags = Arrays.asList("key1=value1", "key2=value2");
Map<String, String> expectedMetadata = new HashMap<>();
expectedMetadata.put("key1", "value1");
expectedMetadata.put("key2", "value2");
Map<String, String> actualMetadata = ConsulUtils.transferMetadata(tags);
// Assert that the actual metadata matches the expected metadata
assertEquals(expectedMetadata, actualMetadata);
}
/**
* Test the transferMetadata method with invalid tags that do not contain key-value pairs.
* This test ensures that only valid key-value pairs are included in the resulting metadata map.
*/
@Test
public void testTransferMetadata_withInvalidTags() {
// Test with tags that do not contain key-value pairs
List<String> tags = Arrays.asList("key1", "key2=value2", "invalidTag");
Map<String, String> expectedMetadata = new HashMap<>();
expectedMetadata.put("key2", "value2");
Map<String, String> actualMetadata = ConsulUtils.transferMetadata(tags);
// Assert that the actual metadata matches the expected metadata
assertEquals(expectedMetadata, actualMetadata);
}
/**
* Test the transferMetadata method with an empty list of tags.
* This test ensures that an empty list results in an empty metadata map.
*/
@Test
public void testTransferMetadata_withEmptyTags() {
// Test with an empty list of tags
List<String> tags = Collections.emptyList();
Map<String, String> expectedMetadata = new HashMap<>();
Map<String, String> actualMetadata = ConsulUtils.transferMetadata(tags);
// Assert that the actual metadata is empty
assertEquals(expectedMetadata, actualMetadata);
}
/**
* Test the transferMetadata method with null tags.
* This test ensures that a null input results in an empty metadata map.
*/
@Test
public void testTransferMetadata_withNullTags() {
// Test with null tags
Map<String, String> expectedMetadata = new HashMap<>();
Map<String, String> actualMetadata = ConsulUtils.transferMetadata(null);
// Assert that the actual metadata is empty
assertEquals(expectedMetadata, actualMetadata);
}
}

View File

@ -0,0 +1,62 @@
package com.alibaba.nacossync.utils;
import com.alibaba.nacossync.util.DubboConstants;
import org.junit.jupiter.api.Test;
import java.util.HashMap;
import java.util.Map;
import static com.alibaba.nacossync.util.DubboConstants.GROUP_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INTERFACE_KEY;
import static com.alibaba.nacossync.util.DubboConstants.RELEASE_KEY;
import static com.alibaba.nacossync.util.DubboConstants.VERSION_KEY;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class DubboConstantsTest {
@Test
public void testCreateServiceName_withValidParameters() {
Map<String, String> queryParams = new HashMap<>();
queryParams.put(INTERFACE_KEY, "com.example.Service");
queryParams.put(VERSION_KEY, "1.0.0");
queryParams.put(GROUP_KEY, "testGroup");
String expectedServiceName = "providers:com.example.Service:1.0.0:testGroup";
String actualServiceName = DubboConstants.createServiceName(queryParams);
assertEquals(expectedServiceName, actualServiceName);
}
@Test
public void testCreateServiceName_withBlankGroup() {
Map<String, String> queryParams = new HashMap<>();
queryParams.put(INTERFACE_KEY, "com.example.Service");
queryParams.put(VERSION_KEY, "1.0.0");
queryParams.put(RELEASE_KEY, "2.7.3");
String expectedServiceName = "providers:com.example.Service:1.0.0:";
String actualServiceName = DubboConstants.createServiceName(queryParams);
assertEquals(expectedServiceName, actualServiceName);
}
@Test
public void testCreateServiceName_withNoGroupOrRelease() {
Map<String, String> queryParams = new HashMap<>();
queryParams.put(INTERFACE_KEY, "com.example.Service");
queryParams.put(VERSION_KEY, "1.0.0");
String expectedServiceName = "providers:com.example.Service:1.0.0";
String actualServiceName = DubboConstants.createServiceName(queryParams);
assertEquals(expectedServiceName, actualServiceName);
}
@Test
public void testCreateServiceName_withComplexRelease() {
Map<String, String> queryParams = new HashMap<>();
queryParams.put(INTERFACE_KEY, "com.example.Service");
queryParams.put(VERSION_KEY, "1.0.0");
queryParams.put(RELEASE_KEY, "2.8.1");
String expectedServiceName = "providers:com.example.Service:1.0.0:";
String actualServiceName = DubboConstants.createServiceName(queryParams);
assertEquals(expectedServiceName, actualServiceName);
}
}

View File

@ -0,0 +1,39 @@
package com.alibaba.nacossync.utils;
import com.alibaba.nacos.api.common.Constants;
import com.alibaba.nacossync.util.NacosUtils;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
public class NacosUtilsTest {
/**
* Test getGroupNameOrDefault with a non-blank group name.
*/
@Test
public void testGetGroupNameOrDefault_withNonBlankGroupName() {
String groupName = "testGroup";
String result = NacosUtils.getGroupNameOrDefault(groupName);
assertEquals("testGroup", result);
}
/**
* Test getGroupNameOrDefault with a blank group name.
*/
@Test
public void testGetGroupNameOrDefault_withBlankGroupName() {
String groupName = " ";
String result = NacosUtils.getGroupNameOrDefault(groupName);
assertEquals(Constants.DEFAULT_GROUP, result);
}
/**
* Test getGroupNameOrDefault with a null group name.
*/
@Test
public void testGetGroupNameOrDefault_withNullGroupName() {
String result = NacosUtils.getGroupNameOrDefault(null);
assertEquals(Constants.DEFAULT_GROUP, result);
}
}

View File

@ -1,22 +1,118 @@
package com.alibaba.nacossync.utils;
import org.junit.Assert;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.pojo.request.ClusterAddRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import org.junit.Test;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import java.net.InetAddress;
import java.net.UnknownHostException;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.fail;
/**
* @author NacosSync
* @version $Id: SkyWalkerUtilTest.java, v 0.1 2018-09-26 下午3:47 NacosSync Exp $$
*/
public class SkyWalkerUtilTest {
/**
* Test the MD5 hashing function with a sample string.
*/
@Test
public void skyWalkerUtilTest() throws Exception {
String ip = SkyWalkerUtil.getLocalIp();
Assert.assertNotEquals("127.0.0.1", ip);
public void testStringToMd5() {
String value = "testValue";
String expectedMd5 = SkyWalkerUtil.stringToMd5(value);
assertFalse(expectedMd5.isEmpty());
assertEquals(32, expectedMd5.length());
}
/**
* Test the task ID generation using a TaskAddRequest object.
*/
@Test
public void testGenerateTaskId_withTaskAddRequest() {
TaskAddRequest request = new TaskAddRequest();
request.setServiceName("testService");
request.setGroupName("testGroup");
request.setSourceClusterId("sourceCluster");
request.setDestClusterId("destCluster");
String taskId = SkyWalkerUtil.generateTaskId(request);
String expectedMd5 = SkyWalkerUtil.stringToMd5("testService_testGroup_sourceCluster_destCluster");
assertEquals(expectedMd5, taskId);
}
/**
* Test the task ID generation using individual parameters.
*/
@Test
public void testGenerateTaskId_withParameters() {
String serviceName = "testService";
String groupName = "testGroup";
String sourceClusterId = "sourceCluster";
String destClusterId = "destCluster";
String taskId = SkyWalkerUtil.generateTaskId(serviceName, groupName, sourceClusterId, destClusterId);
String expectedMd5 = SkyWalkerUtil.stringToMd5("testService_testGroup_sourceCluster_destCluster");
assertEquals(expectedMd5, taskId);
}
/**
* Test the cluster ID generation using a ClusterAddRequest object.
*/
@Test
public void testGenerateClusterId() {
ClusterAddRequest request = new ClusterAddRequest();
request.setClusterName("testCluster");
request.setClusterType("Nacos");
String clusterId = SkyWalkerUtil.generateClusterId(request);
String expectedMd5 = SkyWalkerUtil.stringToMd5("testCluster_Nacos");
assertEquals(expectedMd5, clusterId);
}
/**
* Test the retrieval of the local IP address.
* This ensures that the local IP address is not null and is a valid IP address.
*/
@Test
public void testGetLocalIp() throws Exception {
String localIp = SkyWalkerUtil.getLocalIp();
assertNotNull(localIp);
assertFalse(localIp.isEmpty());
try {
InetAddress ip = InetAddress.getByName(localIp);
assertNotNull(ip);
} catch (UnknownHostException e) {
fail("The IP address is invalid: " + e.getMessage());
}
}
/**
* Test the synchronization key generation using source and destination cluster types.
*/
@Test
public void testGenerateSyncKey() {
ClusterTypeEnum sourceClusterType = ClusterTypeEnum.NACOS;
ClusterTypeEnum destClusterType = ClusterTypeEnum.ZK;
String syncKey = SkyWalkerUtil.generateSyncKey(sourceClusterType, destClusterType);
String expectedSyncKey = sourceClusterType.getCode() + ":" + destClusterType.getCode();
assertEquals(expectedSyncKey, syncKey);
}
/**
* Test the operation ID generation to ensure it is unique each time.
*/
@Test
public void testGenerateOperationId() {
String operationId1 = SkyWalkerUtil.generateOperationId();
String operationId2 = SkyWalkerUtil.generateOperationId();
assertNotNull(operationId1);
assertNotNull(operationId2);
assertNotEquals(operationId1, operationId2);
}
}

203
pom.xml
View File

@ -16,7 +16,7 @@
<groupId>com.alibaba.nacossync</groupId>
<artifactId>nacossync-parent</artifactId>
<version>0.4.7</version>
<version>${revision}</version>
<modules>
<module>nacossync-console</module>
<module>nacossync-worker</module>
@ -25,17 +25,45 @@
</modules>
<packaging>pom</packaging>
<properties>
<revision>0.5.0</revision>
<spring-boot.version>2.7.17</spring-boot.version>
<spring-cloud.version>2021.0.3</spring-cloud.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
<spring.boot.version>2.4.5</spring.boot.version>
<zookeeper.version>3.4.14</zookeeper.version>
<curator.version>4.3.0</curator.version>
<mockito.version>1.10.19</mockito.version>
<nacos-client.verison>1.4.7</nacos-client.verison>
<lombok.verison>1.18.34</lombok.verison>
<consul-api.verison>1.3.1</consul-api.verison>
<commoms-lang3.verison>3.12.0</commoms-lang3.verison>
<guava.verison>33.2.0-jre</guava.verison>
<versions-maven-plugin.version>2.2</versions-maven-plugin.version>
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<maven-jar-plugin.version>3.2.0</maven-jar-plugin.version>
<maven-enforcer-plugin.version>1.4.1</maven-enforcer-plugin.version>
<maven-source-plugin.version>3.0.1</maven-source-plugin.version>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
<maven-resources-plugin.version>3.0.2</maven-resources-plugin.version>
<maven-surefire-plugin.version>2.20</maven-surefire-plugin.version>
<maven-assembly-plugin.version>3.1.1</maven-assembly-plugin.version>
<clirr-maven-plugin.version>2.7</clirr-maven-plugin.version>
<jacoco-maven-plugin.version>0.7.8</jacoco-maven-plugin.version>
<findbugs-maven-plugin.version>3.0.4</findbugs-maven-plugin.version>
<sonar-maven-plugin.version>3.0.2</sonar-maven-plugin.version>
<coveralls-maven-plugin.version>4.3.0</coveralls-maven-plugin.version>
<flatten-maven-plugin.version>1.1.0</flatten-maven-plugin.version>
<dependency-mediator-maven-plugin.version>1.0.2</dependency-mediator-maven-plugin.version>
<extra-enforcer-rules.version>1.0-beta-4</extra-enforcer-rules.version>
<async-http-client.version>1.7.17</async-http-client.version>
<swagger.version>1.8.0</swagger.version>
</properties>
<scm>
<url>git@github.com:nacos-group/nacos-sync.git</url>
<connection>scm:git@github.com:nacos-group/nacos-sync.git</connection>
<developerConnection>scm:git@github.com:nacos-group/nacos-sync.git</developerConnection>
<tag>nacossync-0.4.6</tag>
<tag>nacos-sync-${project.version}</tag>
</scm>
<developers>
<developer>
@ -58,7 +86,14 @@
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring.boot.version}</version>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
@ -66,32 +101,109 @@
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
<version>1.18.2</version>
<version>${lombok.verison}</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
<version>${swagger.version}</version>
</dependency>
<dependency>
<groupId>com.alibaba.nacos</groupId>
<artifactId>nacos-client</artifactId>
<version>${nacos-client.verison}</version>
</dependency>
<!--zookeeper-->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>${curator.version}</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-api</artifactId>
<groupId>org.slf4j</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commoms-lang3.verison}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.verison}</version>
</dependency>
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>${async-http-client.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>nacossync-worker</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>versions-maven-plugin</artifactId>
<version>2.2</version>
<version>${versions-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>com.github.vongosling</groupId>
<artifactId>dependency-mediator-maven-plugin</artifactId>
<version>1.0.2</version>
<version>${dependency-mediator-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>clirr-maven-plugin</artifactId>
<version>2.7</version>
<version>${clirr-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.4.1</version>
<version>${maven-enforcer-plugin.version}</version>
<executions>
<execution>
<id>enforce-ban-circular-dependencies</id>
@ -110,14 +222,14 @@
<dependency>
<groupId>org.codehaus.mojo</groupId>
<artifactId>extra-enforcer-rules</artifactId>
<version>1.0-beta-4</version>
<version>${extra-enforcer-rules.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>${maven.compiler.source}</source>
<target>${maven.compiler.target}</target>
@ -126,11 +238,10 @@
<showWarnings>true</showWarnings>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>3.0.1</version>
<version>${maven-source-plugin.version}</version>
<executions>
<execution>
<id>attach-sources</id>
@ -143,7 +254,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
<version>${maven-resources-plugin.version}</version>
<configuration>
<!-- We are not suppose to setup the customer resources here -->
<encoding>${project.build.sourceEncoding}</encoding>
@ -152,12 +263,12 @@
<plugin>
<groupId>org.eluder.coveralls</groupId>
<artifactId>coveralls-maven-plugin</artifactId>
<version>4.3.0</version>
<version>${coveralls-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.8</version>
<version>${jacoco-maven-plugin.version}</version>
<executions>
<execution>
<id>default-prepare-agent</id>
@ -196,7 +307,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.20</version>
<configuration>
<forkCount>1</forkCount>
<reuseForks>true</reuseForks>
@ -206,12 +316,40 @@
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>3.0.4</version>
<version>${findbugs-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.sonarsource.scanner.maven</groupId>
<artifactId>sonar-maven-plugin</artifactId>
<version>3.0.2</version>
<version>${sonar-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>flatten-maven-plugin</artifactId>
<version>${flatten-maven-plugin.version}</version>
<configuration>
<updatePomFile>true</updatePomFile>
<flattenMode>resolveCiFriendliesOnly</flattenMode>
<pomElements>
<dependencies>expand</dependencies>
</pomElements>
</configuration>
<executions>
<execution>
<id>flatten</id>
<phase>process-resources</phase>
<goals>
<goal>flatten</goal>
</goals>
</execution>
<execution>
<id>flatten.clean</id>
<phase>clean</phase>
<goals>
<goal>clean</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
@ -219,12 +357,22 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.1.1</version>
<version>${maven-assembly-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>${spring.boot.version}</version>
<version>${spring-boot.version}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>${maven-jar-plugin.version}</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${maven-surefire-plugin.version}</version>
</plugin>
</plugins>
</pluginManagement>
@ -234,8 +382,15 @@
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>3.0.4</version>
<version>${findbugs-maven-plugin.version}</version>
</plugin>
</plugins>
</reporting>
<repositories>
<repository>
<id>maven_central</id>
<name>Maven Central</name>
<url>https://repo.maven.apache.org/maven2/</url>
</repository>
</repositories>
</project>