Compare commits

...

88 Commits

Author SHA1 Message Date
paderlol 0c35e096c2
Merge pull request #384 from nacos-group/develop
Optimize some codes
2024-11-30 15:23:35 +08:00
paderlol 142a36f608
Merge pull request #383 from paderlol/develop
Optimize some codes
2024-11-30 15:22:07 +08:00
paderlol b18afb95c1
Optimize some codes 2024-11-30 15:20:40 +08:00
paderlol aa87b24e1a
Merge pull request #381 from nacos-group/develop
Develop
2024-10-29 10:55:13 +08:00
paderlol fdc51ba605
Merge pull request #380 from paderlol/develop
Write unit tests for all utils classes
2024-10-29 10:53:56 +08:00
paderlol ce6249142f
Write unit tests for all utils classes 2024-10-29 10:52:01 +08:00
paderlol 26c8b307d7
Merge pull request #378 from nacos-group/develop
Develop
2024-09-28 13:29:35 +08:00
paderlol 6585b217bc
Merge pull request #377 from paderlol/develop
1. 对代码中依赖注入方式全部修改成构造函数注入
2024-09-28 13:28:32 +08:00
paderlol b609c729c3
1. 对代码中依赖注入方式全部修改成构造函数注入 2024-09-28 13:26:07 +08:00
paderlol 432ced4e0e
Merge pull request #375 from paderlol/develop
优化部分代码
2024-08-29 16:49:31 +08:00
paderlol 116116c8c4
1. 优化部分集群新增部分冗余代码
2. 升级Java JDK到17
2024-08-29 16:48:58 +08:00
paderlol ca08bd7d6e
1. 优化部分集群新增部分冗余代码
2. 升级Java JDK到17
2024-08-29 16:43:05 +08:00
paderlol 81610e505a
Merge pull request #373 from paderlol/add_validation_cluster
修复有任务运行时可以删除集群信息的问题 #371
2024-07-31 15:57:00 +08:00
paderlol 17439be724
修复有任务运行时可以删除集群信息的问题 #371 2024-07-31 15:45:05 +08:00
paderlol 91999fbdcb
Merge pull request #368 from nacos-group/develop
优化以及修复小错误
2024-07-20 16:44:16 +08:00
paderlol 6645b00f24
Develop (#367)
* 1. 优化部分代码

* 整理代码:
 1. 移除枚举类冗余方法以及SkyWalkerCacheServices中未使用方法
 2. 部分服务类改用构造注入
 3. Entity对象重写equals和hashCode
2024-07-20 16:33:19 +08:00
paderlol acb7b49551
Merge pull request #360 from MajorHe1/patch-1
[#ISSUE 359]Update SkyWalkerCacheServices.java, fix operationId empty-judge problem
2024-07-20 14:26:31 +08:00
MajorHe1 ba416d1496
[#ISSUE 359]Update SkyWalkerCacheServices.java, fix operationId empty-judge problem 2024-06-13 20:05:49 +08:00
paderlol ff5e484997
Merge pull request #357 from nacos-group/develop
Develop
2024-06-10 22:19:41 +08:00
paderlol b900c28347
1. 优化部分代码 (#356) 2024-06-10 22:18:13 +08:00
paderlol 5d17242ffc
Develop (#355)
* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* #346 #350:nacos 2 nacos 同步关闭后心跳没有停止问题 ,nacos-sync删除目标节点问题 (#347)

* update port

* fix #297 (#298)

Co-authored-by: yangchun2 <yangchun2@joyy.com>

* Revert "fix #297 (#298)" (#318)

This reverts commit a9df169b5a.

* 0.4.9-pre (#325)

* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:集群信息保存 cluster_level 为 null

* fix:
1、注册时根据中心化逻辑判断同步,但是删除时逻辑不一致问题
2、如果停止同步nameservice为空问题

* 问题在于对destInstances列表的更新方式。在Java中,方法参数是按值传递的。这意味着当你传递一个对象到方法中时,实际上传递的是对象引用的副本。因此,如果你在方法内部改变了这个引用指向的对象(例如,将其指向一个新的对象),这个改变不会影响到原始的对象引用。

在你的代码中,destInstances = newDestInstance;这一行只是改变了destInstances引用在方法内部的指向,而不会改变方法外部传入的destInstances列表对象。这意味着,尽管你筛选出了需要反注册的实例,但这个改变不会反映到方法调用者那里。

---------

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>

* 0.5.0 (#351)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* Develop (#353)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* 1. 优化部分代码 (#354)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* 1. 优化部分代码

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: 龙竹 <34528665+dragonTalon@users.noreply.github.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
2024-06-10 22:04:01 +08:00
paderlol 39357660ab
Merge branch 'master' into develop 2024-06-10 22:02:39 +08:00
paderlol 49abb7c286
1. 优化部分代码 (#354)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* 1. 优化部分代码
2024-06-10 22:01:12 +08:00
paderlol 8185feca1d
0.5.0 (#352)
* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* #346 #350:nacos 2 nacos 同步关闭后心跳没有停止问题 ,nacos-sync删除目标节点问题 (#347)

* update port

* fix #297 (#298)

Co-authored-by: yangchun2 <yangchun2@joyy.com>

* Revert "fix #297 (#298)" (#318)

This reverts commit a9df169b5a.

* 0.4.9-pre (#325)

* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:集群信息保存 cluster_level 为 null

* fix:
1、注册时根据中心化逻辑判断同步,但是删除时逻辑不一致问题
2、如果停止同步nameservice为空问题

* 问题在于对destInstances列表的更新方式。在Java中,方法参数是按值传递的。这意味着当你传递一个对象到方法中时,实际上传递的是对象引用的副本。因此,如果你在方法内部改变了这个引用指向的对象(例如,将其指向一个新的对象),这个改变不会影响到原始的对象引用。

在你的代码中,destInstances = newDestInstance;这一行只是改变了destInstances引用在方法内部的指向,而不会改变方法外部传入的destInstances列表对象。这意味着,尽管你筛选出了需要反注册的实例,但这个改变不会反映到方法调用者那里。

---------

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>

* 0.5.0 (#351)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

* Develop (#353)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: 龙竹 <34528665+dragonTalon@users.noreply.github.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
2024-05-19 00:17:01 +08:00
paderlol 5789d4d774
Develop (#353)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释
2024-05-19 00:15:50 +08:00
paderlol 2fdf44d308
Merge branch 'master' into develop 2024-05-19 00:08:04 +08:00
paderlol 5941bfb9bc
0.5.0 (#351)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

* 1. 重新设计全量 Nacos 同步 Nacos
2. 修复Nacos Instance equals无效导致出现无法注册成功问题
3. 升级Nacos Sync JDK/Spring Boot版本
4. 保底同步从改成并发同步
5. 增加部分注释
2024-05-18 23:50:23 +08:00
龙竹 0c24dcb0a2
#346 #350:nacos 2 nacos 同步关闭后心跳没有停止问题 ,nacos-sync删除目标节点问题 (#347)
* update port

* fix #297 (#298)

Co-authored-by: yangchun2 <yangchun2@joyy.com>

* Revert "fix #297 (#298)" (#318)

This reverts commit a9df169b5a.

* 0.4.9-pre (#325)

* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:nacos 2 nacos 同步关闭后心跳没有停止问题

* fix:集群信息保存 cluster_level 为 null

* fix:
1、注册时根据中心化逻辑判断同步,但是删除时逻辑不一致问题
2、如果停止同步nameservice为空问题

* 问题在于对destInstances列表的更新方式。在Java中,方法参数是按值传递的。这意味着当你传递一个对象到方法中时,实际上传递的是对象引用的副本。因此,如果你在方法内部改变了这个引用指向的对象(例如,将其指向一个新的对象),这个改变不会影响到原始的对象引用。

在你的代码中,destInstances = newDestInstance;这一行只是改变了destInstances引用在方法内部的指向,而不会改变方法外部传入的destInstances列表对象。这意味着,尽管你筛选出了需要反注册的实例,但这个改变不会反映到方法调用者那里。

---------

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
Co-authored-by: 杨春 <chun@kpromise.top>
Co-authored-by: yangchun2 <yangchun2@joyy.com>
Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
2024-05-15 09:53:45 +08:00
paderlol a1d683a7c9
0.4.9-pre (#325)
* Feat/sync support2.x#mutiple thread sync02 (#304)

* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>

* Optimize the code for assigning tasks. (#320)

* Develop (#321)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix #305 (#322)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code (#323)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic (#324)

* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.

---------

Co-authored-by: chenhao26 <35129699+chenhao26-nineteen@users.noreply.github.com>
Co-authored-by: Oliver <wqdyxnbd@163.com>
2023-05-15 17:33:26 +08:00
paderlol be65db1892
Refactoring the Nacos Sync to Consul Logic (#324)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.

* Refactoring the Nacos Sync to Consul Logic.
2023-05-15 17:25:22 +08:00
paderlol eb93fdd52a
Fix cyclic dependency code (#323)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305

* Fix cyclic dependency code.
2023-05-15 17:21:53 +08:00
paderlol e7e52acfa7
Fix #305 (#322)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308

* Fix .#305
2023-05-15 16:51:11 +08:00
paderlol 02838da959
Develop (#321)
* Optimize the code for assigning tasks.

* Adds prefix to the input string if it doesn't already have it.#308
2023-05-15 16:13:39 +08:00
paderlol 0e73a0864c
Optimize the code for assigning tasks. (#320) 2023-05-15 15:58:39 +08:00
paderlol 61d347609e
Revert "fix #297 (#298)" (#318)
This reverts commit a9df169b5a.
2023-05-15 09:55:50 +08:00
杨春 a9df169b5a
fix #297 (#298)
Co-authored-by: yangchun2 <yangchun2@joyy.com>
2023-05-15 09:53:25 +08:00
chenhao26 98efe26906
Feat/sync support2.x#mutiple thread sync02 (#304)
* update port

* Multithreading sync

* solve conflict

* imple SyncService

* adapter deregister

* optimization some code

* fix deregister instance equals logic

Co-authored-by: Oliver <wqdyxnbd@163.com>
Co-authored-by: paderlol <huangmnlove@163.com>
2022-08-01 08:53:33 +08:00
paderlol 384a9cc1bb
Merge pull request #303 from nacos-group/develop
0.4.8
2022-07-30 12:47:09 +08:00
paderlol 0ffdca3e0b
Merge pull request #302 from paderlol/develop
Optimize Nacos synchronization Nacos logic.
2022-07-30 12:18:18 +08:00
paderlol 5ae4071508 Optimize Nacos synchronization Nacos logic. 2022-07-30 12:15:56 +08:00
paderlol 4082efa1fc
Merge pull request #301 from paderlol/develop
Optimize Nacos synchronization Eureka logic.
2022-07-30 11:49:28 +08:00
paderlol fc03e6483c Optimize Nacos synchronization Eureka logic. 2022-07-30 11:46:46 +08:00
paderlol 2b8d29a0ab
Merge pull request #296 from jefferson-chern/fix-resync-when-two-way-sync
[fix #295] 双向同步时,重启某个集群下所有实例,可以重新同步
2022-06-14 21:00:22 +08:00
jefferson 881c11d51f refactor code: remove some useless code. 2022-06-14 16:14:07 +08:00
jefferson 855922964e fix #295 2022-06-14 14:59:12 +08:00
paderlol 9e814d2d43
Merge pull request #294 from paderlol/develop
Refactor some code
2022-06-08 21:36:08 +08:00
paderlol cdfbebf407 Refactor some code 2022-06-08 21:34:13 +08:00
paderlol 017cacf12c
Merge pull request #293 from jefferson-chern/fix-log-bug-and-refactor-version-and-upgrade-to-0.4.8
统一版本号;升级到0.4.8;修复日志不删除问题
2022-06-07 13:52:52 +08:00
jefferson ab4a4bbb92 1. dependency的版本号全部挪到父pom的dependencyManagement;
2. plugin的版本号全部挪到父pom的pluginManagement;
3. nacosSync名称修改为nacos-sync;
4. 父pom和子pom的version统一由revision管理,只需一处修改;
5. 修复一些readme.md中的明显错误;
6. 修复日志文件永不删除的问题;
7. upgrade version to 0.4.8;
8. remove some useless newline in redeme.md.
2022-06-05 15:28:04 +08:00
paderlol 853119b615
Merge pull request #289 from CherishCai/feat_TaskAddAll_with_nacos_source
Feat task add all with nacos source
2022-06-04 19:42:15 +08:00
paderlol f99398c649
Merge pull request #290 from Oliverwqcwrw/master-update-port-1
Update README.md port
2022-04-20 09:59:44 +08:00
Oliver a61a79aca9 update port 2022-04-20 09:02:22 +08:00
hongwen.chw 00570523fa feat: 支持从 sourceCluster 获取所有 service,然后生成同步到 destCluster 的任务。// 目前仅支持 Nacos 为源的同步类型,待完善更多类型支持. 2022-03-23 20:14:52 +08:00
paderlol 508a912c35
Merge pull request #281 from nacos-group/develop
Develop
2021-11-21 17:42:22 +08:00
paderlol 3ee0d33fff
Merge branch 'master' into develop 2021-11-21 17:42:13 +08:00
paderlol 03e4419157
Merge pull request #275 from DavidLei08/fix-257-namespace
Fix 257 namespace
2021-11-01 09:29:15 +08:00
paderlol 3998dfedf1
Merge pull request #276 from paderlol/develop
Add basic auth to Nacos cluster
2021-10-12 18:03:51 +08:00
paderlol 1ffec6a785 Add basic auth to Nacos cluster 2021-10-12 18:02:44 +08:00
DavidLei08 0fad6ef8cc 增加window bat启动脚本 2021-09-29 23:05:31 +08:00
DavidLei08 46b3590fb6 namespace同步无效问题 2021-09-29 22:32:40 +08:00
DavidLei08 a2d870e15e 解决 consul同步到nacos namespace无效的问题 2021-09-29 22:23:16 +08:00
paderlol 758b18cb7d
Merge pull request #274 from paderlol/develop
Fixed Nacos resync invalid issue #272
2021-09-03 12:04:44 +08:00
paderlol d630fb16b4 Fixed Nacos resync invalid issue #272 2021-09-03 12:04:10 +08:00
paderlol 7c6f889e72
Merge pull request #273 from paderlol/develop
Add Nacos Cluster Specify Namespace Features.#272
2021-08-31 20:05:57 +08:00
paderlol 0826dc9c15 Add Nacos Cluster Specify Namespace Features.#272 2021-08-31 20:05:15 +08:00
paderlol 96111de3da
Merge pull request #271 from zyhui98/feature/cluster-sql
feat: cluster表初始化sql脚本缺少字段
2021-08-18 22:43:17 +08:00
zhuyunhui 137fdee7f1 feat: cluster表初始化sql脚本缺少字段 2021-08-18 21:11:36 +08:00
paderlol c4200d7516
Merge pull request #268 from paderlol/develop
Add Eureka sync supports group
2021-08-11 17:14:04 +08:00
paderlol 30b331b4b0 Add Eureka sync Nacos by group 2021-08-11 17:11:07 +08:00
paderlol 8fceffde07
Merge pull request #267 from zrlw/patch-syncNewInstance
add tap control for same sync task
2021-08-05 07:14:44 +08:00
zrlw 8dde539a79 remove useless Date code and correct comment 2021-08-05 00:30:18 +08:00
zrlw 1cd2e3b408 add tap control for same sync task 2021-08-04 19:46:49 +08:00
paderlol ee3040f011
Merge pull request #263 from xyohn/develop
#262 修复Eureka<->Nacos同步问题
2021-07-28 20:43:01 +08:00
xYohn c5d574e357 #262 Fix NacosSyncToEureka Support Nacos NameSpace 2021-07-28 11:13:41 +08:00
xYohn eb3e74a678 #262 Fix EurekaSyncToNacos Support Nacos NameSpace 2021-07-28 11:13:25 +08:00
paderlol 1f16efb043
Merge pull request #260 from paderlol/develop
Fix issue #259
2021-07-24 11:11:32 +08:00
paderlol 467752e355 Fix issue #259 2021-07-24 11:10:32 +08:00
paderlol 91bf0d93f8
Merge pull request #255 from zrlw/patch-addBasicSyncThread
add basic sync task thread
2021-06-30 16:35:23 +08:00
zrlw 9cab680e46 add basic sync task thread 2021-06-30 15:23:04 +08:00
paderlol c62e93a677
Merge pull request #247 from paderlol/develop
Nacos client version temporarily returned to 1.4.2.
2021-06-29 15:58:23 +08:00
paderlol e3354fefb0 Nacos client version temporarily returned to 1.4.2. 2021-06-29 14:31:37 +08:00
paderlol 57e03c8aee
Merge pull request #246 from zrlw/patch-EventListener
do not add FinishedTask if syncManagerService return false,  add group parameter for getAllInstances at sync and only sync healthy instance
2021-06-29 14:02:29 +08:00
paderlol f1c1c00190
Merge pull request #242 from JerryGuTripleUp/develp-Jerry
Update main.cf091959.js
2021-06-29 14:01:21 +08:00
zrlw 4fced18886 only sync healthy instance 2021-06-29 13:02:11 +08:00
zrlw bdfa05670e add group parameter for getAllInstances at sync 2021-06-29 12:17:27 +08:00
zrlw d389587d96 do not add FinishedTask if syncManagerService return false 2021-06-29 12:16:51 +08:00
JerryGuTripleUp e5283d2a23 Update main.cf091959.js 2021-06-23 16:31:47 +08:00
paderlol e65fba55a0 Add link of the example into README. 2021-05-20 17:09:03 +08:00
115 changed files with 4164 additions and 2857 deletions

1
.gitignore vendored
View File

@ -11,3 +11,4 @@ target
node_modules
test/derby.log
/console-fe
.flattened-pom.xml

View File

@ -1,9 +1,11 @@
# Nacos Sync
## [Example](https://github.com/paderlol/nacos-sync-example)
## Function
- Console: provide API and console for management
- Worker: provider the service registration synchronization.
- Worker: provide the service registration synchronization.
## Architecture
@ -42,15 +44,9 @@ Info | +------------+ ^
- NacosCluster target will dedup the synchronization information from Nacos.
## Quick Start:
- Swagger API: http://127.0.0.1:8081/swagger-ui.html#/
- Web Console: http://127.0.0.1:8081/
- Swagger API: http://127.0.0.1:8083/swagger-ui.html#/
- Web Console: http://127.0.0.1:8083/
- Others: TBD
# NacosSync Migration User Guide
@ -82,7 +78,7 @@ Before you begin, install the following:
- 64bit OS: Linux/Unix/Mac/Windows supported, Linux/Unix/Mac recommended.
- 64bit JDK 1.8+: downloads, JAVA_HOME settings.
- Maven 3.2.x+: downloads, settings.
- Maven 3.5.2+: [downloads](https://maven.apache.org/download.cgi), [settings](https://maven.apache.org/settings.html).
- MySql 5.6.+
## Download & Build From Release
@ -94,7 +90,7 @@ There are two ways to get NacosSync.
``` xml
cd nacosSync/
cd nacos-sync/
mvn clean package -U
```
@ -103,7 +99,7 @@ The path to the target file:
``` xml
nacos-sync/nacossync-distribution/target/nacosSync.0.3.8.zip
nacos-sync/nacossync-distribution/target/nacos-sync-0.5.0.tar.gz
```
@ -111,7 +107,7 @@ After extracting the installation package, the directory structure:
``` xml
nacosSync
nacos-sync
├── LICENSE
├── NOTICE
├── bin
@ -122,7 +118,7 @@ nacosSync
│   ├── application.properties
│   └── logback-spring.xml
├── logs
└── nacosSync-server.jar
└── nacos-sync-server.jar
```
@ -130,7 +126,7 @@ nacosSync
The default is Mysql database, which can support other relational databases
- Build db schema, the default schema name nacos_Sync.
- Build db schema, the default schema name nacos_sync.
- Tables do not need to be created separately, which is conducive to hibernate's automatic table creation function.
- If the automatic table creation fails, you can build the table nacosSync.sql, the table statement is in the bin folder.
@ -159,15 +155,18 @@ sh startup.sh start
``` xml
http://127.0.0.1:8081/#/serviceSync
http://127.0.0.1:8083/#/serviceSync
```
## Advanced Configuration
### Full Synchronization from Zookeeper to Nacos (Dubbo)
When “*” is entered in the “Service Name” field of this form, it will fully synchronize all services from Zookeeper to Nacos, but only when using Dubbo.
![img_1.png](img_1.png)
### Full Synchronization from Nacos to Nacos
When “All” is entered in the “Service Name” field of this form, it will automatically synchronize all registered services within the **default group** of the current cluster.
This description explains the functionality clearly for English-speaking users.
![img_2.png](img_2.png)

BIN
img_1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

BIN
img_2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

View File

@ -16,7 +16,7 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.6</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -27,5 +27,4 @@
<artifactId>junit</artifactId>
</dependency>
</dependencies>
</project>
</project>

File diff suppressed because one or more lines are too long

View File

@ -7,8 +7,8 @@
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>Nacos-Sync</title>
<link rel="shortcut icon" href="//www.aliyun.com/favicon.ico" type="image/x-icon">
<link href="./css/main.9093077e.css" rel="stylesheet"></head>
<link href="./css/main.e2917886.css" rel="stylesheet"></head>
<body>
<div id="root"></div>
<script type="text/javascript" src="./js/main.cf091959.js"></script></body>
<script type="text/javascript" src="./js/main.b73436b5.js"></script></body>
</html>

File diff suppressed because one or more lines are too long

View File

@ -18,6 +18,9 @@ class AddConfigDialog extends React.Component {
visible: false,
clusterName: '',
clusterType: '',
namespace: '',
password: '',
userName: '',
connectKeyList: [],
};
}
@ -27,8 +30,8 @@ class AddConfigDialog extends React.Component {
}
save() {
const { clusterName, clusterType, connectKeyList } = this.state;
add({ clusterName, clusterType, connectKeyList })
const { clusterName, namespace, userName, password, clusterType, connectKeyList } = this.state;
add({ clusterName, namespace, userName, password, clusterType, connectKeyList })
.then(() => {
this.props.turnPage(1);
this.close();
@ -37,7 +40,10 @@ class AddConfigDialog extends React.Component {
}
close() {
this.setState({ visible: false });
this.setState({
visible: false,
clusterType: '',
});
}
open = () => this.setState({ visible: true })
@ -71,6 +77,35 @@ class AddConfigDialog extends React.Component {
}
</Select>
</FormItem>
{
this.state.clusterType === 'NACOS' && (
<>
<FormItem
label={`${locale.namespace}:`}
>
<Input
placeholder={locale.namespacePlaceholder}
onChange={ns => this.setState({ namespace: ns })}
/>
</FormItem>
<FormItem
label={`${locale.username}:`}
>
<Input
placeholder={locale.usernamePlaceholder}
onChange={un => this.setState({ userName: un })}
/>
</FormItem>
<FormItem
label={`${locale.password}:`}
>
<Input.Password
placeholder={locale.passwordPlaceholder}
onChange={pw => this.setState({ password: pw })}
/>
</FormItem>
</>)
}
<FormItem label={`${locale.connectKeyList}:`}>
<Input.TextArea
onChange={(connectKeyListStr) => {

View File

@ -98,6 +98,7 @@ class ClusterConfig extends React.Component {
<Table.Column title={locale.clusterName} dataIndex="clusterName" />
<Table.Column title={locale.clusterType} dataIndex="clusterType" />
<Table.Column title={locale.connectKeyList} dataIndex="connectKeyList" />
<Table.Column title={locale.namespace} dataIndex="namespace" />
<Table.Column
title={locale.operation}
cell={(value, index, record) => (

View File

@ -13,7 +13,7 @@ class Layout extends React.Component {
<Header />
<Row className="layout">
<Col fixedSpan="9" className="nav-bar">
<h1 className="title">Nacos-Sync 3.0</h1>
<h1 className="title">Nacos-Sync 0.4.8</h1>
<Menu />
</Col>
<Col className="main-panel">{this.props.children}</Col>

View File

@ -18,7 +18,6 @@ class AddSyncDialog extends React.Component {
this.state = {
visible: false,
destClusterId: '',
nameSpace: '',
groupName: '',
serviceName: '',
sourceClusterId: '',
@ -32,8 +31,8 @@ class AddSyncDialog extends React.Component {
}
save() {
const { destClusterId, nameSpace, groupName, serviceName, sourceClusterId, version } = this.state;
add({ destClusterId, nameSpace, groupName, serviceName, sourceClusterId, version })
const { destClusterId, groupName, serviceName, sourceClusterId, version } = this.state;
add({ destClusterId, groupName, serviceName, sourceClusterId, version })
.then(() => {
this.props.turnPage(1);
this.close();
@ -65,12 +64,6 @@ class AddSyncDialog extends React.Component {
onClose={() => this.close()}
>
<Form>
<FormItem label={`${locale.nameSpace}:`}>
<Input
placeholder={locale.nameSpacePlaceholder}
onChange={nameSpace => this.setState({ nameSpace })}
/>
</FormItem>
<FormItem label={`${locale.serviceName}:`}>
<Input
placeholder={locale.serviceNamePlaceholder}

View File

@ -19,6 +19,9 @@ const I18N_CONF = {
clusterNamePlaceholder: 'Please enter the cluster name',
clusterType: 'Cluster Type',
connectKeyList: 'Connect Key List',
namespace: 'Namespace',
password: 'Password',
username: 'Username',
operation: 'Operation',
deleteBtn: 'Delete',
confirm: 'Prompt',
@ -29,6 +32,8 @@ const I18N_CONF = {
title: 'New Cluster',
clusterName: 'Cluster Name',
clusterNamePlaceholder: 'Please enter the cluster name',
namespace: 'Namespace',
namespacePlaceholder: 'Please enter the namespace',
clusterType: 'Cluster Type',
connectKeyList: 'Connect IP',
connectKeyListPlaceholder: 'Please enter the ip',
@ -38,7 +43,6 @@ const I18N_CONF = {
serviceNamePlaceholder: 'Please enter service name',
search: 'Search',
addSync: 'New Sync',
nameSpace: 'Namespace',
serviceName: 'Service Name',
groupName: 'Group',
sourceCluster: 'Source Cluster',
@ -61,8 +65,6 @@ const I18N_CONF = {
serviceNamePlaceholder: 'Please enter service name',
groupName: 'Group Name',
groupNamePlaceholder: 'Please enter group name',
nameSpace: 'Namespace',
nameSpacePlaceholder: 'Please enter namespace id',
sourceCluster: 'Source Cluster',
destCluster: 'Dest Cluster',
version: 'Version',

View File

@ -19,6 +19,7 @@ const I18N_CONF = {
clusterNamePlaceholder: '请输入集群名',
clusterType: '集群类型',
connectKeyList: '集群IP列表',
namespace: '命名空间',
operation: '操作',
deleteBtn: '删除',
confirm: '提示',
@ -29,6 +30,12 @@ const I18N_CONF = {
title: '新增集群',
clusterName: '集群名',
clusterNamePlaceholder: '请输入集群名',
namespace: '命名空间',
namespacePlaceholder: '请输入命名空间',
password: '密码',
passwordPlaceholder: '请输入密码',
username: '用户名',
usernamePlaceholder: '请输入用户名',
clusterType: '集群类型',
connectKeyList: '集群IP列表',
connectKeyListPlaceholder: '请输入集群IP',
@ -38,7 +45,6 @@ const I18N_CONF = {
serviceNamePlaceholder: '请输入服务名',
search: '搜索',
addSync: '新增同步',
nameSpace: '命名空间',
serviceName: '服务名',
groupName: '分组',
sourceCluster: '源集群',
@ -57,8 +63,6 @@ const I18N_CONF = {
},
AddSyncDialog: {
title: '新增同步',
nameSpace: '命名空间',
nameSpacePlaceholder: '请输入命名空间ID',
serviceName: '服务名',
serviceNamePlaceholder: '请输入服务名',
groupName: '分组名',

File diff suppressed because one or more lines are too long

View File

@ -7,8 +7,8 @@
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>Nacos-Sync</title>
<link rel="shortcut icon" href="//www.aliyun.com/favicon.ico" type="image/x-icon">
<link href="./css/main.9093077e.css" rel="stylesheet"></head>
<link href="./css/main.e2917886.css" rel="stylesheet"></head>
<body>
<div id="root"></div>
<script type="text/javascript" src="./js/main.cf091959.js"></script></body>
<script type="text/javascript" src="./js/main.b73436b5.js"></script></body>
</html>

File diff suppressed because one or more lines are too long

View File

@ -8,6 +8,10 @@ CREATE TABLE `cluster` (
`cluster_name` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`cluster_type` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`connect_key_list` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`user_name` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`password` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`namespace` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`cluster_level` int default 0,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
/******************************************/
@ -37,5 +41,6 @@ CREATE TABLE `task` (
`task_status` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`version` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`worker_ip` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL,
`status` int default null ,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;

View File

@ -13,14 +13,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
pid=`ps ax | grep -i 'nacosSync' |grep java | grep -v grep | awk '{print $1}'`
pid=`ps ax | grep -i 'nacos-sync' |grep java | grep -v grep | awk '{print $1}'`
if [ -z "$pid" ] ; then
echo "No nacosSync running."
echo "no nacos-sync running."
exit -1;
fi
echo "The nacosSync(${pid}) is running..."
echo "the nacos-sync(${pid}) is running..."
kill ${pid}
echo "Send shutdown request to nacosSync(${pid}) OK"
echo "Send shutdown request to nacos-sync(${pid}) OK"

View File

@ -0,0 +1 @@
java -jar ../nacos-sync-server.jar -server -Xms2g -Xmx2g -Xmn1g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m --spring.config.location=../conf/application.properties

View File

@ -47,24 +47,24 @@ export BASE_DIR=`cd $(dirname $0)/..; pwd`
JAVA_OPT="${JAVA_OPT} -server -Xms2g -Xmx2g -Xmn1g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${BASE_DIR}/nacossync_java_heapdump.hprof"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${BASE_DIR}/nacos-sync-java-heapdump.hprof"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages"
JAVA_OPT="${JAVA_OPT} -Dspring.config.location=${BASE_DIR}/conf/application.properties"
JAVA_OPT="${JAVA_OPT} -DnacosSync.home=${BASE_DIR}"
JAVA_MAJOR_VERSION=$($JAVA -version 2>&1 | sed -E -n 's/.* version "([0-9]*).*$/\1/p')
if [[ "$JAVA_MAJOR_VERSION" -ge "9" ]] ; then
JAVA_OPT="${JAVA_OPT} -Xlog:gc*:file=${BASE_DIR}/logs/nacossync_gc.log:time,tags:filecount=10,filesize=102400"
JAVA_OPT="${JAVA_OPT} -Xlog:gc*:file=${BASE_DIR}/logs/nacos-sync-gc.log:time,tags:filecount=10,filesize=102400"
else
JAVA_OPT="${JAVA_OPT} -Xloggc:${BASE_DIR}/logs/nacossync_gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
JAVA_OPT="${JAVA_OPT} -Xloggc:${BASE_DIR}/logs/nacos-sync-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M"
fi
JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR}/nacosSync-server.jar"
JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR}/nacos-sync-server.jar"
JAVA_OPT="${JAVA_OPT} --logging.config=${BASE_DIR}/conf/logback-spring.xml"
echo "=============JAVA_HOME:"$JAVA_HOME
echo "=============BASE_DIR:"$BASE_DIR
echo "=============JAVA:"$JAVA
echo "JAVA_HOME:"$JAVA_HOME
echo "BASE_DIR:"$BASE_DIR
echo "JAVA:"$JAVA
if [ ! -d "${BASE_DIR}/logs" ]; then
@ -78,9 +78,9 @@ usage(){
start(){
echo "$JAVA ${JAVA_OPT}" > ${BASE_DIR}/logs/nacossync_start.out 2>&1 &
nohup $JAVA ${JAVA_OPT} >> ${BASE_DIR}/logs/nacossync_start.out 2>&1 &
echo "nacossync is startingyou can check the ${BASE_DIR}/logs/nacossync_start.out"
echo "$JAVA ${JAVA_OPT}" > ${BASE_DIR}/logs/nacos-sync-start.out 2>&1 &
nohup $JAVA ${JAVA_OPT} >> ${BASE_DIR}/logs/nacos-sync-start.out 2>&1 &
echo "nacos-sync is startingyou can check the ${BASE_DIR}/logs/nacos-sync-start.out"
}

View File

@ -5,14 +5,14 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.6</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<packaging>pom</packaging>
<artifactId>nacossync-distribution</artifactId>
<build>
<finalName>nacosSync.${parent.version}</finalName>
<finalName>nacos-sync-${project.version}</finalName>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
@ -35,4 +35,4 @@
</plugin>
</plugins>
</build>
</project>
</project>

View File

@ -17,7 +17,7 @@
<id>bin</id>
<includeBaseDirectory>true</includeBaseDirectory>
<!-- file name after unzip -->
<baseDirectory>nacosSync</baseDirectory>
<baseDirectory>nacos-sync</baseDirectory>
<formats>
<format>dir</format>
<format>tar.gz</format>
@ -60,9 +60,9 @@
<destName>NOTICE</destName>
</file>
<file>
<source>../nacossync-worker/target/nacosSync-server.${parent.version}.jar</source>
<source>../nacossync-worker/target/nacos-sync-server-${project.version}.jar</source>
<outputDirectory></outputDirectory>
<destName>nacosSync-server.jar</destName>
<destName>nacos-sync-server.jar</destName>
</file>
</files>
</assembly>

View File

@ -17,7 +17,7 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.6</version>
<version>${revision}</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -25,27 +25,19 @@
<artifactId>nacossync-test</artifactId>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
</dependency>
<dependency>
<groupId>com.alibaba.nacossync</groupId>
<groupId>${project.groupId}</groupId>
<artifactId>nacossync-worker</artifactId>
<version>0.4.6</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
@ -57,10 +49,8 @@
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version>1.7.17</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
@ -68,7 +58,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.9</version>
<configuration>
<parallel>methods</parallel>
<useUnlimitedThreads>true</useUnlimitedThreads>
@ -76,6 +65,4 @@
</plugin>
</plugins>
</build>
</project>
</project>

View File

@ -16,29 +16,11 @@
<parent>
<artifactId>nacossync-parent</artifactId>
<groupId>com.alibaba.nacossync</groupId>
<version>0.4.6</version>
<version>${revision}</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>nacossync-worker</artifactId>
<version>0.4.6</version>
<properties>
<zookeeper.version>3.4.9</zookeeper.version>
<curator.version>4.1.0</curator.version>
<cloud.version>2020.0.2</cloud.version>
<mockito.version>1.10.19</mockito.version>
<nacos.client.verison>2.0.1</nacos.client.verison>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
@ -60,12 +42,12 @@
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>${mockito.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<scope>provided</scope>
</dependency>
<!-- 默认使用HikariCP连接池 -->
<dependency>
@ -78,33 +60,26 @@
</dependency>
<!-- mysql -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<!-- swagger2 -->
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger2</artifactId>
<version>2.6.1</version>
<groupId>com.mysql</groupId>
<artifactId>mysql-connector-j</artifactId>
</dependency>
<dependency>
<groupId>io.springfox</groupId>
<artifactId>springfox-swagger-ui</artifactId>
<version>2.6.1</version>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-ui</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
</dependency>
<!--nacos-->
<dependency>
<groupId>com.alibaba.nacos</groupId>
<artifactId>nacos-client</artifactId>
<version>${nacos.client.verison}</version>
</dependency>
<!--zookeeper-->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-log4j12</artifactId>
@ -112,11 +87,9 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
@ -124,17 +97,13 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>${curator.version}</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<artifactId>slf4j-api</artifactId>
@ -142,14 +111,11 @@
</exclusion>
</exclusions>
</dependency>
<!-- eureka -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<!-- consul -->
<dependency>
<groupId>com.ecwid.consul</groupId>
@ -158,16 +124,15 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>29.0-jre</version>
</dependency>
</dependencies>
<build>
<finalName>nacosSync-server.${parent.version}</finalName>
<finalName>nacos-sync-server-${project.version}</finalName>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
@ -183,7 +148,6 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.2.0</version>
<configuration>
<excludes>
<exclude>application.properties</exclude>
@ -197,6 +161,22 @@
</archive>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>17</source>
<target>17</target>
<annotationProcessorPaths>
<path>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.34</version>
</path>
</annotationProcessorPaths>
</configuration>
</plugin>
</plugins>
</build>
</project>

View File

@ -14,24 +14,32 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration;
import springfox.documentation.swagger2.annotations.EnableSwagger2;
import org.springframework.context.ConfigurableApplicationContext;
/**
* @author NacosSync
* @version $Id: SkyWalkerMain.java, v 0.1 2018-09-24 PM12:42 NacosSync Exp $$
*/
@EnableSwagger2
@SpringBootApplication(exclude = EurekaClientAutoConfiguration.class)
public class NacosSyncMain {
public static void main(String[] args) {
SpringApplication.run(NacosSyncMain.class, args);
ConfigurableApplicationContext context = SpringApplication.run(NacosSyncMain.class, args);
// Register shutdown callback using Spring Boot's context lifecycle
context.registerShutdownHook();
context.addApplicationListener(event -> {
if (event instanceof org.springframework.context.event.ContextClosedEvent) {
BatchTaskExecutor.shutdown();
}
});
}
}

View File

@ -14,6 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.api;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
@ -44,57 +45,54 @@ import org.springframework.web.bind.annotation.RestController;
@Slf4j
@RestController
public class ClusterApi {
private final ClusterAddProcessor clusterAddProcessor;
private final ClusterDeleteProcessor clusterDeleteProcessor;
private final ClusterDetailQueryProcessor clusterDetailQueryProcessor;
private final ClusterListQueryProcessor clusterListQueryProcessor;
public ClusterApi(
ClusterAddProcessor clusterAddProcessor, ClusterDeleteProcessor clusterDeleteProcessor,
ClusterDetailQueryProcessor clusterDetailQueryProcessor, ClusterListQueryProcessor clusterListQueryProcessor) {
public ClusterApi(ClusterAddProcessor clusterAddProcessor, ClusterDeleteProcessor clusterDeleteProcessor,
ClusterDetailQueryProcessor clusterDetailQueryProcessor,
ClusterListQueryProcessor clusterListQueryProcessor) {
this.clusterAddProcessor = clusterAddProcessor;
this.clusterDeleteProcessor = clusterDeleteProcessor;
this.clusterDetailQueryProcessor = clusterDetailQueryProcessor;
this.clusterListQueryProcessor = clusterListQueryProcessor;
}
@RequestMapping(path = "/v1/cluster/list", method = RequestMethod.GET)
public ClusterListQueryResult clusters(ClusterListQueryRequest clusterListQueryRequest) {
return SkyWalkerTemplate.run(clusterListQueryProcessor, clusterListQueryRequest,
new ClusterListQueryResult());
return SkyWalkerTemplate.run(clusterListQueryProcessor, clusterListQueryRequest, new ClusterListQueryResult());
}
@RequestMapping(path = "/v1/cluster/detail", method = RequestMethod.GET)
public ClusterDetailQueryResult getByTaskId(ClusterDetailQueryRequest clusterDetailQueryRequest) {
return SkyWalkerTemplate.run(clusterDetailQueryProcessor, clusterDetailQueryRequest,
new ClusterDetailQueryResult());
}
@RequestMapping(path = "/v1/cluster/delete", method = RequestMethod.DELETE)
public ClusterDeleteResult deleteCluster(ClusterDeleteRequest clusterDeleteRequest) {
return SkyWalkerTemplate.run(clusterDeleteProcessor, clusterDeleteRequest,
new ClusterDeleteResult());
return SkyWalkerTemplate.run(clusterDeleteProcessor, clusterDeleteRequest, new ClusterDeleteResult());
}
@RequestMapping(path = "/v1/cluster/add", method = RequestMethod.POST)
public ClusterAddResult clusterAdd(@RequestBody ClusterAddRequest clusterAddRequest) {
return SkyWalkerTemplate
.run(clusterAddProcessor, clusterAddRequest, new ClusterAddResult());
return SkyWalkerTemplate.run(clusterAddProcessor, clusterAddRequest, new ClusterAddResult());
}
@RequestMapping(path = "/v1/cluster/types", method = RequestMethod.GET)
public ClusterTypeResult getClusterType() {
return new ClusterTypeResult(ClusterTypeEnum.getClusterTypeCodes());
}
}

View File

@ -14,6 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.api;
import lombok.extern.slf4j.Slf4j;
@ -42,34 +43,36 @@ import com.alibaba.nacossync.template.processor.ConfigQueryProcessor;
@Slf4j
@RestController
public class SystemConfigApi {
@Autowired
private ConfigQueryProcessor configQueryProcessor;
@Autowired
private ConfigDeleteProcessor configDeleteProcessor;
@Autowired
private ConfigAddProcessor configAddProcessor;
private final ConfigQueryProcessor configQueryProcessor;
private final ConfigDeleteProcessor configDeleteProcessor;
private final ConfigAddProcessor configAddProcessor;
public SystemConfigApi(ConfigQueryProcessor configQueryProcessor, ConfigDeleteProcessor configDeleteProcessor,
ConfigAddProcessor configAddProcessor) {
this.configQueryProcessor = configQueryProcessor;
this.configDeleteProcessor = configDeleteProcessor;
this.configAddProcessor = configAddProcessor;
}
@RequestMapping(path = "/v1/systemconfig/list", method = RequestMethod.GET)
public ConfigQueryResult tasks(ConfigQueryRequest configQueryRequest) {
return SkyWalkerTemplate.run(configQueryProcessor, configQueryRequest,
new ConfigQueryResult());
return SkyWalkerTemplate.run(configQueryProcessor, configQueryRequest, new ConfigQueryResult());
}
@RequestMapping(path = "/v1/systemconfig/delete", method = RequestMethod.DELETE)
public ConfigDeleteResult deleteTask(@RequestBody ConfigDeleteRequest configDeleteRequest) {
return SkyWalkerTemplate.run(configDeleteProcessor, configDeleteRequest,
new ConfigDeleteResult());
return SkyWalkerTemplate.run(configDeleteProcessor, configDeleteRequest, new ConfigDeleteResult());
}
@RequestMapping(path = "/v1/systemconfig/add", method = RequestMethod.POST)
public ConfigAddResult taskAdd(@RequestBody ConfigAddRequest configAddRequest) {
return SkyWalkerTemplate.run(configAddProcessor, configAddRequest, new ConfigAddResult());
}
}

View File

@ -10,8 +10,10 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.api;
import com.alibaba.nacossync.pojo.request.TaskAddAllRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.alibaba.nacossync.pojo.request.TaskDeleteInBatchRequest;
import com.alibaba.nacossync.pojo.request.TaskDeleteRequest;
@ -23,6 +25,7 @@ import com.alibaba.nacossync.pojo.result.TaskAddResult;
import com.alibaba.nacossync.pojo.result.TaskDetailQueryResult;
import com.alibaba.nacossync.pojo.result.TaskListQueryResult;
import com.alibaba.nacossync.template.SkyWalkerTemplate;
import com.alibaba.nacossync.template.processor.TaskAddAllProcessor;
import com.alibaba.nacossync.template.processor.TaskAddProcessor;
import com.alibaba.nacossync.template.processor.TaskDeleteInBatchProcessor;
import com.alibaba.nacossync.template.processor.TaskDeleteProcessor;
@ -42,67 +45,83 @@ import org.springframework.web.bind.annotation.RestController;
@Slf4j
@RestController
public class TaskApi {
private final TaskUpdateProcessor taskUpdateProcessor;
private final TaskAddProcessor taskAddProcessor;
private final TaskAddAllProcessor taskAddAllProcessor;
private final TaskDeleteProcessor taskDeleteProcessor;
private final TaskDeleteInBatchProcessor taskDeleteInBatchProcessor;
private final TaskListQueryProcessor taskListQueryProcessor;
private final TaskDetailProcessor taskDetailProcessor;
public TaskApi(TaskUpdateProcessor taskUpdateProcessor, TaskAddProcessor taskAddProcessor,
TaskDeleteProcessor taskDeleteProcessor, TaskDeleteInBatchProcessor taskDeleteInBatchProcessor,
TaskListQueryProcessor taskListQueryProcessor, TaskDetailProcessor taskDetailProcessor) {
TaskAddAllProcessor taskAddAllProcessor, TaskDeleteProcessor taskDeleteProcessor,
TaskDeleteInBatchProcessor taskDeleteInBatchProcessor, TaskListQueryProcessor taskListQueryProcessor,
TaskDetailProcessor taskDetailProcessor) {
this.taskUpdateProcessor = taskUpdateProcessor;
this.taskAddProcessor = taskAddProcessor;
this.taskAddAllProcessor = taskAddAllProcessor;
this.taskDeleteProcessor = taskDeleteProcessor;
this.taskDeleteInBatchProcessor = taskDeleteInBatchProcessor;
this.taskListQueryProcessor = taskListQueryProcessor;
this.taskDetailProcessor = taskDetailProcessor;
}
@RequestMapping(path = "/v1/task/list", method = RequestMethod.GET)
public TaskListQueryResult tasks(TaskListQueryRequest taskListQueryRequest) {
return SkyWalkerTemplate.run(taskListQueryProcessor, taskListQueryRequest, new TaskListQueryResult());
}
@RequestMapping(path = "/v1/task/detail", method = RequestMethod.GET)
public TaskDetailQueryResult getByTaskId(TaskDetailQueryRequest taskDetailQueryRequest) {
return SkyWalkerTemplate.run(taskDetailProcessor, taskDetailQueryRequest, new TaskDetailQueryResult());
}
@RequestMapping(path = "/v1/task/delete", method = RequestMethod.DELETE)
public BaseResult deleteTask(TaskDeleteRequest taskDeleteRequest) {
return SkyWalkerTemplate.run(taskDeleteProcessor, taskDeleteRequest, new BaseResult());
}
/**
* @author yongchao9
* @param taskBatchDeleteRequest
* @return
* @author yongchao9
*/
@RequestMapping(path = "/v1/task/deleteInBatch", method = RequestMethod.DELETE)
public BaseResult batchDeleteTask(TaskDeleteInBatchRequest taskBatchDeleteRequest) {
return SkyWalkerTemplate.run(taskDeleteInBatchProcessor, taskBatchDeleteRequest, new BaseResult());
}
@RequestMapping(path = "/v1/task/add", method = RequestMethod.POST)
public BaseResult taskAdd(@RequestBody TaskAddRequest addTaskRequest) {
return SkyWalkerTemplate.run(taskAddProcessor, addTaskRequest, new TaskAddResult());
}
/**
* TODO 目前仅支持 Nacos 为源的同步类型待完善更多类型支持.
* <p>
* 支持从 sourceCluster 获取所有 service然后生成同步到 destCluster 的任务
* </p>
*/
@RequestMapping(path = "/v1/task/addAll", method = RequestMethod.POST)
public BaseResult taskAddAll(@RequestBody TaskAddAllRequest addAllRequest) {
return SkyWalkerTemplate.run(taskAddAllProcessor, addAllRequest, new TaskAddResult());
}
@RequestMapping(path = "/v1/task/update", method = RequestMethod.POST)
public BaseResult updateTask(@RequestBody TaskUpdateRequest taskUpdateRequest) {
return SkyWalkerTemplate.run(taskUpdateProcessor, taskUpdateRequest, new BaseResult());
}
}

View File

@ -25,15 +25,15 @@ import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.SneakyThrows;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import org.springframework.util.StringUtils;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadLocalRandom;
import lombok.SneakyThrows;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.util.CollectionUtils;
import org.springframework.util.StringUtils;
/**
* @author NacosSync
@ -41,18 +41,20 @@ import org.springframework.util.StringUtils;
*/
@Service
public class SkyWalkerCacheServices {
private static final Map<String, FinishedTask> FINISHED_TASK_MAP = new ConcurrentHashMap<>();
private final ClusterAccessService clusterAccessService;
@Autowired
private ClusterAccessService clusterAccessService;
@Autowired
private ObjectMapper objectMapper;
private static Map<String, FinishedTask> finishedTaskMap = new ConcurrentHashMap<>();
private final ObjectMapper objectMapper;
public SkyWalkerCacheServices(ClusterAccessService clusterAccessService, ObjectMapper objectMapper) {
this.clusterAccessService = clusterAccessService;
this.objectMapper = objectMapper;
}
public String getClusterConnectKey(String clusterId) {
List<String> allClusterConnectKey = getAllClusterConnectKey(clusterId);
return allClusterConnectKey.get(ThreadLocalRandom.current().nextInt(allClusterConnectKey.size()));
}
@ -83,23 +85,32 @@ public class SkyWalkerCacheServices {
FinishedTask finishedTask = new FinishedTask();
finishedTask.setOperationId(operationId);
finishedTaskMap.put(operationId, finishedTask);
FINISHED_TASK_MAP.put(operationId, finishedTask);
}
public FinishedTask getFinishedTask(TaskDO taskDO) {
String operationId = SkyWalkerUtil.getOperationId(taskDO);
if (StringUtils.isEmpty(operationId)) {
if (!StringUtils.hasLength(operationId)) {
return null;
}
return finishedTaskMap.get(operationId);
return FINISHED_TASK_MAP.get(operationId);
}
public void removeFinishedTask(String operationId) {
if (!StringUtils.hasLength(operationId)) {
return;
}
FINISHED_TASK_MAP.remove(operationId);
}
public Map<String, FinishedTask> getFinishedTaskMap() {
return finishedTaskMap;
return FINISHED_TASK_MAP;
}
}

View File

@ -12,6 +12,8 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
import java.util.ArrayList;
import java.util.List;
@ -19,6 +21,7 @@ import java.util.List;
* @author NacosSync
* @version $Id: ClusterTypeEnum.java, v 0.1 2018-09-25 下午4:38 NacosSync Exp $$
*/
@Getter
public enum ClusterTypeEnum {
CS("CS", "configserver集群"),
@ -32,18 +35,16 @@ public enum ClusterTypeEnum {
ZK("ZK", "zookeeper集群");
private String code;
private final String code;
private String desc;
ClusterTypeEnum(String code, String desc) {
this.code = code;
this.desc = desc;
}
public static List<String> getClusterTypeCodes() {
List<String> list = new ArrayList<String>();
List<String> list = new ArrayList<>();
for (ClusterTypeEnum clusterTypeEnum : ClusterTypeEnum.values()) {
list.add(clusterTypeEnum.getCode());
@ -51,42 +52,7 @@ public enum ClusterTypeEnum {
return list;
}
/**
* Getter method for property <tt>code</tt>.
*
* @return property value of code
*/
public String getCode() {
return code;
}
/**
* Setter method for property <tt>code </tt>.
*
* @param code value to be assigned to property code
*/
public void setCode(String code) {
this.code = code;
}
/**
* Getter method for property <tt>desc</tt>.
*
* @return property value of desc
*/
public String getDesc() {
return desc;
}
/**
* Setter method for property <tt>desc </tt>.
*
* @param desc value to be assigned to property desc
*/
public void setDesc(String desc) {
this.desc = desc;
}
public static boolean contains(String clusterType) {
for (ClusterTypeEnum clusterTypeEnum : ClusterTypeEnum.values()) {

View File

@ -3,10 +3,13 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
/**
* @author NacosSync
* @version $Id: MetricsStatisticsType.java, v 0.1 2019年02月28日 下午2:17 NacosSync Exp $
*/
@Getter
public enum MetricsStatisticsType {
CACHE_SIZE("nacosSync.finished.taskMap.cacheSize", "任务执行完成缓存列表数"),
@ -28,15 +31,10 @@ public enum MetricsStatisticsType {
/**
* metricsName
*/
private String metricsName;
private String desc;
private final String metricsName;
MetricsStatisticsType(String code, String desc) {
this.metricsName = code;
this.desc = desc;
}
public String getMetricsName() {
return metricsName;
}
}

View File

@ -16,46 +16,23 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
/**
* @author NacosSync
* @version $Id: ResultCodeEnum.java, v 0.1 2018-09-25 PM4:38 NacosSync Exp $$
*/
@Getter
public enum ResultCodeEnum {
SUCCESS("SUCCESS", "请求成功", "请求成功"),
SYSTEM_ERROR("SYSTEM_ERROR", "系统异常", "系统异常");
private String code;
private String errorMessage;
private String detail;
private final String code;
ResultCodeEnum(String code, String errorMessage, String detail) {
this.code = code;
this.errorMessage = errorMessage;
this.detail = detail;
}
public String getCode() {
return code;
}
public void setCode(String code) {
this.code = code;
}
public String getErrorMessage() {
return errorMessage;
}
public void setErrorMessage(String errorMessage) {
this.errorMessage = errorMessage;
}
public String getDetail() {
return detail;
}
public void setDetail(String detail) {
this.detail = detail;
}
}

View File

@ -24,11 +24,18 @@ public class SkyWalkerConstants {
public static final String UNDERLINE = "_";
public static final String DEST_CLUSTERID_KEY = "destClusterId";
public static final String DEST_CLUSTER_ID_KEY = "destClusterId";
public static final String GROUP_NAME = "groupName";
public static final String SYNC_SOURCE_KEY = "syncSource";
public static final String SOURCE_CLUSTERID_KEY = "sourceClusterId";
public static final String SOURCE_CLUSTER_ID_KEY = "sourceClusterId";
public static final String MANAGEMENT_PORT_KEY="management.port";
public static final String MANAGEMENT_CONTEXT_PATH_KEY="management.context-path";
public static final String SERVICE_NAME_PARAM="serviceNameParam";
public static final String GROUP_NAME_PARAM="groupNameParam";
public static final String PAGE_NO="pageNo";
public static final String PAGE_SIZE="pageSize";
public static final String SYNC_INSTANCE_TAG="sync.instance.tag";
public static final String NACOS_ALL_SERVICE_NAME = "ALL";
}

View File

@ -16,6 +16,8 @@
*/
package com.alibaba.nacossync.constant;
import lombok.Getter;
/**
* @author NacosSync
* @version $Id: TaskStatusEnum.java, v 0.1 2018-09-26 上午2:38 NacosSync Exp $$
@ -30,51 +32,18 @@ public enum TaskStatusEnum {
* delete the task
*/
DELETE("DELETE", "任务需要被删除");
private String code;
private String desc;
@Getter
private final String code;
private final String desc;
TaskStatusEnum(String code, String desc) {
this.code = code;
this.desc = desc;
}
/**
* Getter method for property <tt>code</tt>.
*
* @return property value of code
*/
public String getCode() {
return code;
}
/**
* Setter method for property <tt>code </tt>.
*
* @param code value to be assigned to property code
*/
public void setCode(String code) {
this.code = code;
}
/**
* Getter method for property <tt>desc</tt>.
*
* @return property value of desc
*/
public String getDesc() {
return desc;
}
/**
* Setter method for property <tt>desc </tt>.
*
* @param desc value to be assigned to property desc
*/
public void setDesc(String desc) {
this.desc = desc;
}
public static boolean contains(String code) {
for (TaskStatusEnum taskStatusEnum : TaskStatusEnum.values()) {

View File

@ -16,10 +16,10 @@
*/
package com.alibaba.nacossync.dao;
import com.alibaba.nacossync.dao.repository.ClusterRepository;
import com.alibaba.nacossync.pojo.QueryCondition;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
@ -27,9 +27,6 @@ import org.springframework.data.domain.Sort;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.repository.ClusterRepository;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.Predicate;
import javax.persistence.criteria.Root;
@ -44,9 +41,12 @@ import java.util.List;
@Slf4j
public class ClusterAccessService implements PageQueryService<ClusterDO> {
@Autowired
private ClusterRepository clusterRepository;
private final ClusterRepository clusterRepository;
public ClusterAccessService(ClusterRepository clusterRepository) {
this.clusterRepository = clusterRepository;
}
public ClusterDO insert(ClusterDO clusterDO) {
return clusterRepository.save(clusterDO);
@ -103,4 +103,12 @@ public class ClusterAccessService implements PageQueryService<ClusterDO> {
predicates.add(criteriaBuilder.like(root.get("clusterName"), "%" + queryCondition.getServiceName() + "%"));
return predicates;
}
public int findClusterLevel(String sourceClusterId){
ClusterDO clusterDO = clusterRepository.findByClusterId(sourceClusterId);
if (clusterDO != null) {
return clusterDO.getClusterLevel();
}
return -1;
}
}

View File

@ -14,10 +14,13 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.dao;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.dao.repository.TaskRepository;
import com.alibaba.nacossync.pojo.QueryCondition;
import org.springframework.beans.factory.annotation.Autowired;
import com.alibaba.nacossync.pojo.model.TaskDO;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.domain.Pageable;
@ -25,9 +28,6 @@ import org.springframework.data.domain.Sort;
import org.springframework.data.jpa.domain.Specification;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.repository.TaskRepository;
import com.alibaba.nacossync.pojo.model.TaskDO;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.Predicate;
import javax.persistence.criteria.Root;
@ -40,79 +40,94 @@ import java.util.List;
*/
@Service
public class TaskAccessService implements PageQueryService<TaskDO> {
@Autowired
private TaskRepository taskRepository;
private final TaskRepository taskRepository;
public TaskAccessService(TaskRepository taskRepository) {
this.taskRepository = taskRepository;
}
public TaskDO findByTaskId(String taskId) {
return taskRepository.findByTaskId(taskId);
}
public void deleteTaskById(String taskId) {
taskRepository.deleteByTaskId(taskId);
}
/**
* batch delete tasks by taskIds
* @author yongchao9
*
* @param taskIds
* @author yongchao9
*/
public void deleteTaskInBatch(List<String> taskIds) {
List<TaskDO> tds=taskRepository.findAllByTaskIdIn(taskIds);
taskRepository.deleteInBatch(tds);
List<TaskDO> tds = taskRepository.findAllByTaskIdIn(taskIds);
taskRepository.deleteAllInBatch(tds);
}
public Iterable<TaskDO> findAll() {
return taskRepository.findAll();
}
public void addTask(TaskDO taskDO) {
taskRepository.save(taskDO);
}
public int countByDestClusterIdOrSourceClusterId(String destClusterId, String sourceClusterId) {
return taskRepository.countByDestClusterIdOrSourceClusterId(destClusterId, sourceClusterId);
}
private Predicate getPredicate(CriteriaBuilder criteriaBuilder, List<Predicate> predicates) {
Predicate[] p = new Predicate[predicates.size()];
return criteriaBuilder.and(predicates.toArray(p));
}
private List<Predicate> getPredicates(Root<TaskDO> root, CriteriaBuilder criteriaBuilder, QueryCondition queryCondition) {
private List<Predicate> getPredicates(Root<TaskDO> root, CriteriaBuilder criteriaBuilder,
QueryCondition queryCondition) {
List<Predicate> predicates = new ArrayList<>();
predicates.add(criteriaBuilder.like(root.get("serviceName"), "%" + queryCondition.getServiceName() + "%"));
return predicates;
}
@Override
public Page<TaskDO> findPageNoCriteria(Integer pageNum, Integer size) {
Pageable pageable = PageRequest.of(pageNum, size, Sort.Direction.DESC, "id");
return taskRepository.findAll(pageable);
}
@Override
public Page<TaskDO> findPageCriteria(Integer pageNum, Integer size, QueryCondition queryCondition) {
Pageable pageable = PageRequest.of(pageNum, size, Sort.Direction.DESC, "id");
return getTaskDOS(queryCondition, pageable);
}
private Page<TaskDO> getTaskDOS(QueryCondition queryCondition, Pageable pageable) {
return taskRepository.findAll(
(Specification<TaskDO>) (root, criteriaQuery, criteriaBuilder) -> {
List<Predicate> predicates = getPredicates(root,
criteriaBuilder, queryCondition);
return getPredicate(criteriaBuilder, predicates);
}, pageable);
return taskRepository.findAll((Specification<TaskDO>) (root, criteriaQuery, criteriaBuilder) -> {
List<Predicate> predicates = getPredicates(root, criteriaBuilder, queryCondition);
return getPredicate(criteriaBuilder, predicates);
}, pageable);
}
public List<TaskDO> findAllByServiceNameEqualAll() {
return taskRepository.findAllByServiceNameEqualsIgnoreCase(SkyWalkerConstants.NACOS_ALL_SERVICE_NAME);
}
public List<TaskDO> findAllByServiceNameNotEqualAll() {
return taskRepository.findAllByServiceNameNotIgnoreCase(SkyWalkerConstants.NACOS_ALL_SERVICE_NAME);
}
}

View File

@ -16,13 +16,13 @@
*/
package com.alibaba.nacossync.dao.repository;
import javax.transaction.Transactional;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;
import org.springframework.data.repository.CrudRepository;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import javax.transaction.Transactional;
/**
* @author NacosSync
@ -34,6 +34,6 @@ public interface ClusterRepository extends CrudRepository<ClusterDO, Integer>, J
ClusterDO findByClusterId(String clusterId);
@Transactional
int deleteByClusterId(String clusterId);
void deleteByClusterId(String clusterId);
}

View File

@ -16,15 +16,13 @@
*/
package com.alibaba.nacossync.dao.repository;
import java.util.List;
import javax.transaction.Transactional;
import com.alibaba.nacossync.pojo.model.TaskDO;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.JpaSpecificationExecutor;
import org.springframework.data.repository.CrudRepository;
import com.alibaba.nacossync.pojo.model.TaskDO;
import javax.transaction.Transactional;
import java.util.List;
/**
* @author NacosSync
@ -36,10 +34,15 @@ public interface TaskRepository extends CrudRepository<TaskDO, Integer>, JpaRepo
TaskDO findByTaskId(String taskId);
@Transactional
int deleteByTaskId(String taskId);
void deleteByTaskId(String taskId);
List<TaskDO> findAllByTaskIdIn(List<String> taskIds);
/**
* query service is alluse ns leven sync data
*/
List<TaskDO> findAllByServiceNameEqualsIgnoreCase(String serviceName);
List<TaskDO> findAllByServiceNameNotIgnoreCase(String serviceName);
List<TaskDO> getAllByWorkerIp(String workerIp);
int countByDestClusterIdOrSourceClusterId(String destClusterId,String sourceClusterId);
}

View File

@ -0,0 +1,13 @@
package com.alibaba.nacossync.event;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.Data;
@Data
public class DeleteAllSubTaskEvent {
public DeleteAllSubTaskEvent(TaskDO taskDO) {
this.taskDO = taskDO;
}
private final TaskDO taskDO;
}

View File

@ -14,23 +14,22 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.event.listener;
import javax.annotation.PostConstruct;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.monitor.MetricsManager;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.event.SyncTaskEvent;
import com.alibaba.nacossync.extension.SyncManagerService;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.google.common.base.Stopwatch;
import com.google.common.eventbus.EventBus;
import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
/**
* @author NacosSync
@ -39,50 +38,59 @@ import com.google.common.eventbus.Subscribe;
@Slf4j
@Service
public class EventListener {
@Autowired
private MetricsManager metricsManager;
@Autowired
private SyncManagerService syncManagerService;
@Autowired
private EventBus eventBus;
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
private final MetricsManager metricsManager;
private final SyncManagerService syncManagerService;
private final EventBus eventBus;
private final SkyWalkerCacheServices skyWalkerCacheServices;
public EventListener(MetricsManager metricsManager, SyncManagerService syncManagerService, EventBus eventBus,
SkyWalkerCacheServices skyWalkerCacheServices) {
this.metricsManager = metricsManager;
this.syncManagerService = syncManagerService;
this.eventBus = eventBus;
this.skyWalkerCacheServices = skyWalkerCacheServices;
}
@PostConstruct
public void register() {
eventBus.register(this);
}
@Subscribe
public void listenerSyncTaskEvent(SyncTaskEvent syncTaskEvent) {
public void sync(SyncTaskEvent syncTaskEvent) {
try {
long start = System.currentTimeMillis();
syncManagerService.sync(syncTaskEvent.getTaskDO());
skyWalkerCacheServices.addFinishedTask(syncTaskEvent.getTaskDO());
metricsManager.record(MetricsStatisticsType.SYNC_TASK_RT, System.currentTimeMillis() - start);
Stopwatch stopwatch = Stopwatch.createStarted();
if (syncManagerService.sync(syncTaskEvent.getTaskDO(), null)) {
skyWalkerCacheServices.addFinishedTask(syncTaskEvent.getTaskDO());
metricsManager.record(MetricsStatisticsType.SYNC_TASK_RT, stopwatch.elapsed().toMillis());
} else {
log.warn("syncTaskEvent process error");
}
} catch (Exception e) {
log.warn("listenerSyncTaskEvent process error", e);
log.warn("syncTaskEvent process error", e);
}
}
@Subscribe
public void listenerDeleteTaskEvent(DeleteTaskEvent deleteTaskEvent) {
public void delete(DeleteTaskEvent deleteTaskEvent) {
try {
long start = System.currentTimeMillis();
syncManagerService.delete(deleteTaskEvent.getTaskDO());
skyWalkerCacheServices.addFinishedTask(deleteTaskEvent.getTaskDO());
metricsManager.record(MetricsStatisticsType.DELETE_TASK_RT, System.currentTimeMillis() - start);
Stopwatch stopwatch = Stopwatch.createStarted();
if (syncManagerService.delete(deleteTaskEvent.getTaskDO())) {
skyWalkerCacheServices.removeFinishedTask(deleteTaskEvent.getTaskDO().getOperationId());
metricsManager.record(MetricsStatisticsType.DELETE_TASK_RT, stopwatch.elapsed().toMillis());
} else {
log.warn("deleteTaskEvent delete failure");
}
} catch (Exception e) {
log.warn("listenerDeleteTaskEvent process error", e);
log.warn("deleteTaskEvent delete failure.", e);
}
}
}

View File

@ -12,14 +12,12 @@
*/
package com.alibaba.nacossync.extension;
import static com.alibaba.nacossync.util.SkyWalkerUtil.generateSyncKey;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.pojo.model.TaskDO;
import java.util.concurrent.ConcurrentHashMap;
import com.alibaba.nacossync.util.StringUtils;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.InitializingBean;
@ -27,6 +25,10 @@ import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.stereotype.Service;
import java.util.concurrent.ConcurrentHashMap;
import static com.alibaba.nacossync.util.SkyWalkerUtil.generateSyncKey;
/**
* @author NacosSync
* @version $Id: SyncManagerService.java, v 0.1 2018-09-25 PM5:17 NacosSync Exp $$
@ -37,7 +39,7 @@ public class SyncManagerService implements InitializingBean, ApplicationContextA
protected final SkyWalkerCacheServices skyWalkerCacheServices;
private ConcurrentHashMap<String, SyncService> syncServiceMap = new ConcurrentHashMap<String, SyncService>();
private final ConcurrentHashMap<String, SyncService> syncServiceMap = new ConcurrentHashMap<String, SyncService>();
private ApplicationContext applicationContext;
@ -52,19 +54,19 @@ public class SyncManagerService implements InitializingBean, ApplicationContextA
}
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
return getSyncService(taskDO.getSourceClusterId(), taskDO.getDestClusterId()).sync(taskDO);
return getSyncService(taskDO.getSourceClusterId(), taskDO.getDestClusterId()).sync(taskDO, index);
}
@Override
public void afterPropertiesSet() {
this.applicationContext.getBeansOfType(SyncService.class).forEach((key, value) -> {
this.applicationContext.getBeansWithAnnotation(NacosSyncService.class).forEach((key, value) -> {
NacosSyncService nacosSyncService = value.getClass().getAnnotation(NacosSyncService.class);
ClusterTypeEnum sourceCluster = nacosSyncService.sourceCluster();
ClusterTypeEnum destinationCluster = nacosSyncService.destinationCluster();
syncServiceMap.put(generateSyncKey(sourceCluster, destinationCluster), value);
syncServiceMap.put(generateSyncKey(sourceCluster, destinationCluster), (SyncService) value);
});
}
@ -74,7 +76,9 @@ public class SyncManagerService implements InitializingBean, ApplicationContextA
}
public SyncService getSyncService(String sourceClusterId, String destClusterId) {
if (StringUtils.isEmpty(sourceClusterId) || StringUtils.isEmpty(destClusterId)) {
throw new IllegalArgumentException("Source cluster id and destination cluster id must not be null or empty");
}
ClusterTypeEnum sourceClusterType = this.skyWalkerCacheServices.getClusterType(sourceClusterId);
ClusterTypeEnum destClusterType = this.skyWalkerCacheServices.getClusterType(destClusterId);

View File

@ -14,9 +14,10 @@ package com.alibaba.nacossync.extension;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.pojo.model.TaskDO;
import java.util.Map;
import org.apache.commons.lang3.StringUtils;
import java.util.Map;
/**
* @author NacosSync
* @version $Id: SyncManagerService.java, v 0.1 2018-09-25 下午5:17 NacosSync Exp $$
@ -30,20 +31,22 @@ public interface SyncService {
* @return
*/
boolean delete(TaskDO taskDO);
/**
* execute sync
*
* @param taskDO
* @param index
* @return
*/
boolean sync(TaskDO taskDO);
boolean sync(TaskDO taskDO, Integer index);
/**
* Determines that the current instance data is from another source cluster
*/
default boolean needSync(Map<String, String> sourceMetaData) {
return StringUtils.isBlank(sourceMetaData.get(SkyWalkerConstants.SOURCE_CLUSTERID_KEY));
boolean syncTag = StringUtils.isBlank(sourceMetaData.get(SkyWalkerConstants.SYNC_INSTANCE_TAG));
boolean blank = StringUtils.isBlank(sourceMetaData.get(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY));
return syncTag && blank;
}
/**
@ -51,7 +54,7 @@ public interface SyncService {
* cluster ID of the task
*/
default boolean needDelete(Map<String, String> destMetaData, TaskDO taskDO) {
return StringUtils.equals(destMetaData.get(SkyWalkerConstants.SOURCE_CLUSTERID_KEY),
return StringUtils.equals(destMetaData.get(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY),
taskDO.getSourceClusterId());
}

View File

@ -1,15 +0,0 @@
package com.alibaba.nacossync.extension.client;
import lombok.Data;
@Data
public class InstanceQueryModel {
private String sourceClusterId;
private String destClusterId;
private String serviceName;
private String groupName;
private String version;
private int pageNo;
private int pageSize;
}

View File

@ -1,11 +0,0 @@
package com.alibaba.nacossync.extension.client;
import com.alibaba.nacossync.pojo.view.TaskModel;
import java.util.List;
public interface SyncQueryClient {
List<TaskModel> getAllInstance(InstanceQueryModel instanceQueryModel);
}

View File

@ -1,54 +0,0 @@
package com.alibaba.nacossync.extension.client.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.ListView;
import com.alibaba.nacossync.extension.client.InstanceQueryModel;
import com.alibaba.nacossync.extension.client.SyncQueryClient;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.pojo.view.TaskModel;
import java.util.Collections;
import java.util.List;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
@Service
@Slf4j
public class NacosSyncQueryClientImpl implements SyncQueryClient {
private final NacosServerHolder nacosServerHolder;
public NacosSyncQueryClientImpl(
NacosServerHolder nacosServerHolder) {
this.nacosServerHolder = nacosServerHolder;
}
@Override
public List<TaskModel> getAllInstance(InstanceQueryModel instanceQueryModel) {
NamingService namingService = nacosServerHolder
.get(instanceQueryModel.getSourceClusterId(), instanceQueryModel.getGroupName());
try {
ListView<String> servicesOfServer = namingService
.getServicesOfServer(instanceQueryModel.getPageNo(),
instanceQueryModel.getPageSize());
return servicesOfServer.getData().stream()
.map(serviceName -> buildTaskModel(instanceQueryModel, serviceName))
.collect(Collectors.toList());
} catch (NacosException e) {
log.error("When using nacos client failure query tasks", e);
return Collections.emptyList();
}
}
private TaskModel buildTaskModel(InstanceQueryModel instanceQueryModel, String serviceName) {
TaskModel taskModel = new TaskModel();
taskModel.setServiceName(serviceName);
taskModel.setSourceClusterId(instanceQueryModel.getSourceClusterId());
taskModel.setDestClusterId(instanceQueryModel.getDestClusterId());
return taskModel;
}
}

View File

@ -29,11 +29,11 @@ import java.util.concurrent.TimeUnit;
*/
@Slf4j
public class EurekaBeatReactor {
private ScheduledExecutorService executorService;
private final ScheduledExecutorService executorService;
private volatile long clientBeatInterval = 5 * 1000;
private static final long CLIENT_BEAT_INTERVAL = 5 * 1000;
private final Map<String, InstanceInfo> eurekaBeat = new ConcurrentHashMap<>();
private EurekaHttpClient eurekaHttpClient;
private final EurekaHttpClient eurekaHttpClient;
public EurekaBeatReactor(EurekaHttpClient eurekaHttpClient) {
this.eurekaHttpClient = eurekaHttpClient;
@ -71,7 +71,7 @@ public class EurekaBeatReactor {
} catch (Exception e) {
log.error("[CLIENT-BEAT] Exception while scheduling beat.", e);
} finally {
executorService.schedule(this, clientBeatInterval, TimeUnit.MILLISECONDS);
executorService.schedule(this, CLIENT_BEAT_INTERVAL, TimeUnit.MILLISECONDS);
}
}
}

View File

@ -26,8 +26,8 @@ import java.util.Objects;
* @date 2019-06-26
*/
public class EurekaNamingService {
private EurekaHttpClient eurekaHttpClient;
private EurekaBeatReactor beatReactor;
private final EurekaHttpClient eurekaHttpClient;
private final EurekaBeatReactor beatReactor;
public EurekaNamingService(EurekaHttpClient eurekaHttpClient) {

View File

@ -25,7 +25,7 @@ import java.util.function.Consumer;
*/
@Service
public class SpecialSyncEventBus {
private ConcurrentHashMap<String, SpecialSyncEvent> specialSyncEventRegistry = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, SpecialSyncEvent> specialSyncEventRegistry = new ConcurrentHashMap<>();
public void subscribe(TaskDO taskDO, Consumer<TaskDO> syncAction) {
SpecialSyncEvent specialSyncEvent = new SpecialSyncEvent();

View File

@ -21,7 +21,6 @@ import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.eventbus.EventBus;
import com.google.common.eventbus.Subscribe;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
@ -34,9 +33,12 @@ import java.util.function.Consumer;
@Service
@Slf4j
public class SpecialSyncEventListener {
@Autowired
private EventBus eventBus;
private final EventBus eventBus;
public SpecialSyncEventListener(EventBus eventBus) {
this.eventBus = eventBus;
}
@PostConstruct
public void init() {
eventBus.register(this);

View File

@ -13,13 +13,10 @@
package com.alibaba.nacossync.extension.holder;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.google.common.base.Joiner;
import lombok.extern.slf4j.Slf4j;
import org.apache.logging.log4j.util.Strings;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Supplier;
@ -28,40 +25,36 @@ import java.util.function.Supplier;
* @date 2018-12-24 22:08
*/
@Slf4j
public abstract class AbstractServerHolderImpl<T> implements Holder {
public abstract class AbstractServerHolderImpl<T> implements Holder<T> {
private final Map<String, T> serviceMap = new ConcurrentHashMap<>();
@Autowired
protected SkyWalkerCacheServices skyWalkerCacheServices;
@Override
public T get(String clusterId, String namespace) {
final String finalNamespace = Optional.ofNullable(namespace).orElse(Strings.EMPTY);
String key = Joiner.on("_").join(clusterId, finalNamespace);
public T get(String clusterId) {
serviceMap.computeIfAbsent(key, clusterKey -> {
return serviceMap.computeIfAbsent(clusterId, clusterKey -> {
try {
log.info("Starting create cluster server, clusterId={}", clusterId);
return createServer(clusterId, () -> skyWalkerCacheServices.getClusterConnectKey(clusterId),
finalNamespace);
return createServer(clusterId, () -> skyWalkerCacheServices.getClusterConnectKey(clusterId));
} catch (Exception e) {
log.error(String.format("clusterId=%s, start server failed", clusterId), e);
return null;
}
});
return serviceMap.get(key);
}
/**
* Create real cluster client instance
*
* @param clusterId cluster id
* @param clusterId cluster id
* @param serverAddressSupplier server address
* @param namespace name space
* @return cluster client instance
*/
abstract T createServer(String clusterId, Supplier<String> serverAddressSupplier, String namespace)
abstract T createServer(String clusterId, Supplier<String> serverAddressSupplier)
throws Exception;
}

View File

@ -30,7 +30,7 @@ public class ConsulServerHolder extends AbstractServerHolderImpl<ConsulClient> {
public static final String HTTP = "http://";
@Override
ConsulClient createServer(String clusterId, Supplier<String> serverAddressSupplier, String namespace) throws Exception {
ConsulClient createServer(String clusterId, Supplier<String> serverAddressSupplier) throws Exception {
String serverAddress = serverAddressSupplier.get();
serverAddress = serverAddress.startsWith(HTTP) ? serverAddress : HTTP + serverAddress;
URL url = new URL(serverAddress);

View File

@ -10,6 +10,7 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.holder;
import com.alibaba.nacossync.extension.eureka.EurekaNamingService;
@ -30,12 +31,27 @@ import java.util.function.Supplier;
@Service
@Slf4j
public class EurekaServerHolder extends AbstractServerHolderImpl<EurekaNamingService> {
private static final String HTTP_PREFIX = "http://";
private static final String HTTPS_PREFIX = "https://";
@Override
EurekaNamingService createServer(String clusterId, Supplier<String> serverAddressSupplier, String namespace) throws Exception {
RestTemplateTransportClientFactory restTemplateTransportClientFactory =
new RestTemplateTransportClientFactory();
EurekaEndpoint eurekaEndpoint = new DefaultEndpoint(serverAddressSupplier.get());
EurekaNamingService createServer(String clusterId, Supplier<String> serverAddressSupplier) throws Exception {
RestTemplateTransportClientFactory restTemplateTransportClientFactory = new RestTemplateTransportClientFactory();
EurekaEndpoint eurekaEndpoint = new DefaultEndpoint(addHttpPrefix(serverAddressSupplier.get()));
EurekaHttpClient eurekaHttpClient = restTemplateTransportClientFactory.newClient(eurekaEndpoint);
return new EurekaNamingService(eurekaHttpClient);
}
public String addHttpPrefix(String input) {
if (input == null || input.isEmpty()) {
return input;
}
if (!input.startsWith(HTTP_PREFIX) && !input.startsWith(HTTPS_PREFIX)) {
input = HTTP_PREFIX + input;
}
return input;
}
}

View File

@ -22,9 +22,8 @@ public interface Holder<T> {
/**
* Through the cluster ID and namespace fetch cluster client service
* @param clusterId cluster id
* @param namespace name space
* @return
* @throws Exception
*/
T get(String clusterId, String namespace) throws Exception;
T get(String clusterId) throws Exception;
}

View File

@ -18,12 +18,14 @@ import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.google.common.base.Joiner;
import lombok.extern.slf4j.Slf4j;
import org.apache.logging.log4j.util.Strings;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.Optional;
import java.util.Properties;
import java.util.function.Supplier;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
* @author paderlol
@ -34,13 +36,13 @@ import org.springframework.stereotype.Service;
public class NacosServerHolder extends AbstractServerHolderImpl<NamingService> {
private final ClusterAccessService clusterAccessService;
public NacosServerHolder(ClusterAccessService clusterAccessService) {
this.clusterAccessService = clusterAccessService;
}
@Override
NamingService createServer(String clusterId, Supplier<String> serverAddressSupplier, String namespace)
NamingService createServer(String clusterId, Supplier<String> serverAddressSupplier)
throws Exception {
List<String> allClusterConnectKey = skyWalkerCacheServices
.getAllClusterConnectKey(clusterId);
@ -48,7 +50,8 @@ public class NacosServerHolder extends AbstractServerHolderImpl<NamingService> {
String serverList = Joiner.on(",").join(allClusterConnectKey);
Properties properties = new Properties();
properties.setProperty(PropertyKeyConst.SERVER_ADDR, serverList);
properties.setProperty(PropertyKeyConst.NAMESPACE, namespace);
properties.setProperty(PropertyKeyConst.NAMESPACE, Optional.ofNullable(clusterDO.getNamespace()).orElse(
Strings.EMPTY));
Optional.ofNullable(clusterDO.getUserName()).ifPresent(value ->
properties.setProperty(PropertyKeyConst.USERNAME, value)
);
@ -58,4 +61,5 @@ public class NacosServerHolder extends AbstractServerHolderImpl<NamingService> {
);
return NamingFactory.createNamingService(properties);
}
}

View File

@ -32,7 +32,7 @@ public class ZookeeperServerHolder extends AbstractServerHolderImpl<CuratorFrame
@Override
CuratorFramework createServer(String clusterId, Supplier<String> serverAddressSupplier, String namespace) {
CuratorFramework createServer(String clusterId, Supplier<String> serverAddressSupplier) {
List<String> allClusterConnectKey = skyWalkerCacheServices
.getAllClusterConnectKey(clusterId);
String serverList = Joiner.on(",").join(allClusterConnectKey);

View File

@ -0,0 +1,265 @@
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacos.common.utils.CollectionUtils;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import com.google.common.base.Stopwatch;
import lombok.Getter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
@Slf4j
public abstract class AbstractNacosSync implements SyncService {
private final Map<String, EventListener> listenerMap = new ConcurrentHashMap<>();
private final Map<String, Set<String>> sourceInstanceSnapshot = new ConcurrentHashMap<>();
private final Map<String, Integer> syncTaskTap = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, TaskDO> allSyncTaskMap = new ConcurrentHashMap<>();
private ScheduledExecutorService executorService;
@Autowired
private MetricsManager metricsManager;
@Getter
@Autowired
private NacosServerHolder nacosServerHolder;
/**
* Due to network issues or other reasons, the Nacos Sync synchronization tasks may fail,
* resulting in the target cluster's registry missing synchronized instances.
* To prevent the target cluster's registry from missing synchronized instances for an extended period,
* a fallback worker thread is started every 5 minutes to execute all synchronization tasks.
*/
@PostConstruct
public void afterPropertiesSet() {
initializeExecutorService();
scheduleSyncTasks();
}
@PreDestroy
public void destroy() {
if (executorService != null && !executorService.isShutdown()) {
executorService.shutdown();
try {
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
}
private void initializeExecutorService() {
executorService = Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r);
t.setDaemon(true);
t.setName("com.alibaba.nacossync.basic.synctask");
return t;
});
}
private void scheduleSyncTasks() {
executorService.scheduleWithFixedDelay(this::executeSyncTasks, 0, 300, TimeUnit.SECONDS);
}
private void executeSyncTasks() {
if (allSyncTaskMap.isEmpty()) {
return;
}
Collection<TaskDO> taskCollections = allSyncTaskMap.values();
List<TaskDO> taskDOList = new ArrayList<>(taskCollections);
if (CollectionUtils.isNotEmpty(taskDOList)) {
BatchTaskExecutor.batchOperation(taskDOList, this::executeTask);
}
}
private void executeTask(TaskDO task) {
Stopwatch stopwatch = Stopwatch.createStarted();
String taskId = task.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(task.getSourceClusterId());
doSync(taskId, task, sourceNamingService);
} catch (NacosException e) {
log.error("sync task from nacos to nacos failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} finally {
stopwatch.stop();
log.debug("Task execution time for taskId {}: {} ms", taskId, stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}
@Override
public boolean delete(TaskDO taskDO) {
String taskId = taskDO.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
//移除订阅
EventListener listener = listenerMap.remove(taskId);
if (listener!= null) {
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()), listener);
}
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.remove(taskId));
sourceInstanceSnapshot.remove(taskId);
allSyncTaskMap.remove(taskId);
// 删除目标集群中同步的实例列表
deregisterInstance(taskDO);
}catch (NacosException e) {
log.error("Delete task from nacos to specify destination was failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO, Integer index) {
String taskId = taskDO.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
allSyncTaskMap.put(taskId, taskDO);
//防止暂停同步任务后,重新同步/或删除任务以后新建任务不会再接收到新的事件导致不能同步,所以每次订阅事件之前,先全量同步一次任务
doSync(taskId, taskDO, sourceNamingService);
this.listenerMap.putIfAbsent(taskId, event -> {
if (event instanceof NamingEvent) {
try {
doSync(taskId, taskDO, sourceNamingService);
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
});
sourceNamingService.subscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.get(taskId));
}catch (NacosException e) {
log.error("Nacos sync task process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
}
return true;
}
private void doSync(String taskId, TaskDO taskDO, NamingService sourceNamingService) throws Exception {
if (syncTaskTap.putIfAbsent(taskId, 1) != null) {
log.info("任务Id:{}上一个同步任务尚未结束", taskId);
return;
}
try {
// 直接从本地保存的serviceInfoMap中取订阅的服务实例
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
// 先删除不存在的
this.removeInvalidInstance(taskDO, sourceInstances);
// 同步实例
this.syncNewInstance(taskDO, sourceInstances);
} finally {
syncTaskTap.remove(taskId);
}
}
private void syncNewInstance(TaskDO taskDO, List<Instance> sourceInstances) throws NacosException {
Set<String> latestSyncInstance = new TreeSet<>();
//再次添加新实例
String taskId = taskDO.getTaskId();
Set<String> instanceKeys = sourceInstanceSnapshot.get(taskId);
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
String instanceKey = composeInstanceKey(instance.getIp(), instance.getPort());
if (CollectionUtils.isEmpty(instanceKeys) || !instanceKeys.contains(instanceKey)) {
register(taskDO, instance);
}
latestSyncInstance.add(instanceKey);
}
}
if (CollectionUtils.isNotEmpty(latestSyncInstance)) {
log.info("任务Id:{},已同步实例个数:{}", taskId, latestSyncInstance.size());
sourceInstanceSnapshot.put(taskId, latestSyncInstance);
} else {
// latestSyncInstance为空表示源集群中需要同步的所有实例即非nacos-sync同步过来的实例已经下线,清除本地持有快照
sourceInstanceSnapshot.remove(taskId);
}
}
private void removeInvalidInstance(TaskDO taskDO, List<Instance> sourceInstances) throws Exception {
String taskId = taskDO.getTaskId();
if (this.sourceInstanceSnapshot.containsKey(taskId)) {
Set<String> oldInstanceKeys = this.sourceInstanceSnapshot.get(taskId);
Set<String> newInstanceKeys = sourceInstances.stream()
.map(instance -> composeInstanceKey(instance.getIp(), instance.getPort()))
.collect(Collectors.toSet());
oldInstanceKeys.removeAll(newInstanceKeys);
if (CollectionUtils.isNotEmpty(oldInstanceKeys)) {
log.info("任务Id:{},移除无效同步实例:{}", taskId, oldInstanceKeys);
removeInvalidInstance(taskDO, oldInstanceKeys);
}
}
}
@Override
public boolean needDelete(Map<String, String> destMetaData, TaskDO taskDO) {
return SyncService.super.needDelete(destMetaData, taskDO);
}
@Override
public boolean needSync(Map<String, String> sourceMetaData) {
return SyncService.super.needSync(sourceMetaData);
}
public abstract String composeInstanceKey(String ip, int port);
public abstract void register(TaskDO taskDO, Instance instance);
public abstract void deregisterInstance(TaskDO taskDO) throws Exception;
public abstract void removeInvalidInstance(TaskDO taskDO, Set<String> invalidInstanceKeys) throws Exception;
}

View File

@ -27,18 +27,22 @@ import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.ConsulUtils;
import com.alibaba.nacossync.util.NacosUtils;
import com.ecwid.consul.v1.ConsulClient;
import com.ecwid.consul.v1.QueryParams;
import com.ecwid.consul.v1.Response;
import com.ecwid.consul.v1.health.model.HealthService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.*;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Consul 同步 Nacos
*
*
* @author paderlol
* @date: 2018-12-31 16:25
*/
@ -46,8 +50,7 @@ import java.util.*;
@NacosSyncService(sourceCluster = ClusterTypeEnum.CONSUL, destinationCluster = ClusterTypeEnum.NACOS)
public class ConsulSyncToNacosServiceImpl implements SyncService {
@Autowired
private MetricsManager metricsManager;
private final MetricsManager metricsManager;
private final ConsulServerHolder consulServerHolder;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@ -56,14 +59,15 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
private final SpecialSyncEventBus specialSyncEventBus;
@Autowired
public ConsulSyncToNacosServiceImpl(ConsulServerHolder consulServerHolder,
SkyWalkerCacheServices skyWalkerCacheServices, NacosServerHolder nacosServerHolder,
SpecialSyncEventBus specialSyncEventBus) {
SpecialSyncEventBus specialSyncEventBus, MetricsManager metricsManager) {
this.consulServerHolder = consulServerHolder;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
this.specialSyncEventBus = specialSyncEventBus;
this.metricsManager = metricsManager;
}
@Override
@ -71,12 +75,15 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
try {
specialSyncEventBus.unsubscribe(taskDO);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), null);
List<Instance> allInstances = destNamingService.getAllInstances(taskDO.getServiceName());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
List<Instance> allInstances = destNamingService.getAllInstances(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()));
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(taskDO.getServiceName(), instance.getIp(), instance.getPort());
destNamingService.deregisterInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(), instance.getPort());
}
}
@ -89,17 +96,17 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
try {
ConsulClient consulClient = consulServerHolder.get(taskDO.getSourceClusterId(), null);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), null);
ConsulClient consulClient = consulServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
Response<List<HealthService>> response =
consulClient.getHealthServices(taskDO.getServiceName(), true, QueryParams.DEFAULT);
List<HealthService> healthServiceList = response.getValue();
Set<String> instanceKeys = new HashSet<>();
overrideAllInstance(taskDO, destNamingService, healthServiceList, instanceKeys);
cleanAllOldInstance(taskDO, destNamingService, instanceKeys);
specialSyncEventBus.subscribe(taskDO, this::sync);
specialSyncEventBus.subscribe(taskDO, t->sync(t, index));
} catch (Exception e) {
log.error("Sync task from consul to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -115,7 +122,8 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
if (needDelete(instance.getMetadata(), taskDO)
&& !instanceKeys.contains(composeInstanceKey(instance.getIp(), instance.getPort()))) {
destNamingService.deregisterInstance(taskDO.getServiceName(), instance.getIp(), instance.getPort());
destNamingService.deregisterInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(), instance.getPort());
}
}
}
@ -125,6 +133,7 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
for (HealthService healthService : healthServiceList) {
if (needSync(ConsulUtils.transferMetadata(healthService.getService().getTags()))) {
destNamingService.registerInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()),
buildSyncInstance(healthService, taskDO));
instanceKeys.add(composeInstanceKey(healthService.getService().getAddress(),
healthService.getService().getPort()));
@ -137,10 +146,10 @@ public class ConsulSyncToNacosServiceImpl implements SyncService {
temp.setIp(instance.getService().getAddress());
temp.setPort(instance.getService().getPort());
Map<String, String> metaData = new HashMap<>(ConsulUtils.transferMetadata(instance.getService().getTags()));
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
temp.setMetadata(metaData);
return temp;
}

View File

@ -27,6 +27,7 @@ import com.alibaba.nacossync.extension.holder.EurekaServerHolder;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.NacosUtils;
import com.netflix.appinfo.InstanceInfo;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
@ -71,8 +72,10 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
try {
specialSyncEventBus.unsubscribe(taskDO);
EurekaNamingService eurekaNamingService = eurekaServerHolder.get(taskDO.getSourceClusterId(), null);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), null);
EurekaNamingService eurekaNamingService = eurekaServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
List<InstanceInfo> eurekaInstances = eurekaNamingService.getApplications(taskDO.getServiceName());
deleteAllInstanceFromEureka(taskDO, destNamingService, eurekaInstances);
@ -85,12 +88,15 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO,Integer index) {
try {
EurekaNamingService eurekaNamingService = eurekaServerHolder.get(taskDO.getSourceClusterId(), null);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), null);
EurekaNamingService eurekaNamingService = eurekaServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
List<InstanceInfo> eurekaInstances = eurekaNamingService.getApplications(taskDO.getServiceName());
List<Instance> nacosInstances = destNamingService.getAllInstances(taskDO.getServiceName());
List<Instance> nacosInstances = destNamingService.getAllInstances(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()));
if (CollectionUtils.isEmpty(eurekaInstances)) {
// Clear all instance from Nacos
@ -102,7 +108,7 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
}
addValidInstance(taskDO, destNamingService, eurekaInstances);
}
specialSyncEventBus.subscribe(taskDO, this::sync);
specialSyncEventBus.subscribe(taskDO, t->sync(t, index));
} catch (Exception e) {
log.error("sync task from eureka to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -117,7 +123,9 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
if (needSync(instance.getMetadata())) {
log.info("Add service instance from Eureka, serviceName={}, Ip={}, port={}",
instance.getAppName(), instance.getIPAddr(), instance.getPort());
destNamingService.registerInstance(taskDO.getServiceName(), buildSyncInstance(instance, taskDO));
destNamingService.registerInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), buildSyncInstance(instance,
taskDO));
}
}
}
@ -132,7 +140,8 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
if (needSync(instance.getMetadata())) {
log.info("Delete service instance from Eureka, serviceName={}, Ip={}, port={}",
instance.getAppName(), instance.getIPAddr(), instance.getPort());
destNamingService.deregisterInstance(taskDO.getServiceName(), buildSyncInstance(instance, taskDO));
destNamingService.deregisterInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), buildSyncInstance(instance, taskDO));
}
}
}
@ -143,7 +152,8 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
if (!isExistInEurekaInstance(eurekaInstances, instance) && needDelete(instance.getMetadata(), taskDO)) {
log.info("Remove invalid service instance from Nacos, serviceName={}, Ip={}, port={}",
instance.getServiceName(), instance.getIp(), instance.getPort());
destNamingService.deregisterInstance(taskDO.getServiceName(), instance.getIp(), instance.getPort());
destNamingService.deregisterInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(), instance.getPort());
}
}
}
@ -158,7 +168,8 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
throws NacosException {
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(taskDO.getServiceName(), instance);
destNamingService.deregisterInstance(taskDO.getServiceName(),
NacosUtils.getGroupNameOrDefault(taskDO.getGroupName()), instance);
}
}
@ -172,10 +183,10 @@ public class EurekaSyncToNacosServiceImpl implements SyncService {
temp.setHealthy(true);
Map<String, String> metaData = new HashMap<>(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
temp.setMetadata(metaData);
return temp;
}

View File

@ -10,21 +10,15 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.holder.ConsulServerHolder;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.ConsulUtils;
import com.ecwid.consul.v1.ConsulClient;
@ -33,15 +27,13 @@ import com.ecwid.consul.v1.Response;
import com.ecwid.consul.v1.agent.model.NewService;
import com.ecwid.consul.v1.health.model.HealthService;
import com.google.common.collect.Lists;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import lombok.extern.slf4j.Slf4j;
import java.util.HashSet;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
/**
@ -49,107 +41,66 @@ import java.util.stream.Collectors;
*/
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.CONSUL)
public class NacosSyncToConsulServiceImpl implements SyncService {
private Map<String, EventListener> nacosListenerMap = new ConcurrentHashMap<>();
private final MetricsManager metricsManager;
public class NacosSyncToConsulServiceImpl extends AbstractNacosSync {
private static final String DELIMITER = "=";
private final SkyWalkerCacheServices skyWalkerCacheServices;
private final NacosServerHolder nacosServerHolder;
private final ConsulServerHolder consulServerHolder;
public NacosSyncToConsulServiceImpl(MetricsManager metricsManager, SkyWalkerCacheServices skyWalkerCacheServices,
NacosServerHolder nacosServerHolder, ConsulServerHolder consulServerHolder) {
this.metricsManager = metricsManager;
public NacosSyncToConsulServiceImpl(SkyWalkerCacheServices skyWalkerCacheServices,
ConsulServerHolder consulServerHolder) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
this.consulServerHolder = consulServerHolder;
}
@Override
public boolean delete(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getGroupName());
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId(), taskDO.getGroupName());
sourceNamingService.unsubscribe(taskDO.getServiceName(), nacosListenerMap.get(taskDO.getTaskId()));
// 删除目标集群中同步的实例列表
Response<List<HealthService>> serviceResponse =
consulClient.getHealthServices(taskDO.getServiceName(), true, QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)) {
consulClient.agentServiceDeregister(URLEncoder
.encode(healthService.getService().getId(), StandardCharsets.UTF_8.name()));
}
public String composeInstanceKey(String ip, int port) {
return String.join(":", ip, String.valueOf(port));
}
@Override
public void register(TaskDO taskDO, Instance instance) {
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
consulClient.agentServiceRegister(buildSyncInstance(instance, taskDO));
}
@Override
public void deregisterInstance(TaskDO taskDO) throws Exception {
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
Response<List<HealthService>> serviceResponse = consulClient.getHealthServices(taskDO.getServiceName(), true,
QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)) {
consulClient.agentServiceDeregister(
URLEncoder.encode(healthService.getService().getId(), StandardCharsets.UTF_8));
}
} catch (Exception e) {
log.error("delete a task from nacos to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getGroupName());
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId(), taskDO.getGroupName());
nacosListenerMap.putIfAbsent(taskDO.getTaskId(), event -> {
if (event instanceof NamingEvent) {
try {
Set<String> instanceKeySet = new HashSet();
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName());
// 先将新的注册一遍
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
consulClient.agentServiceRegister(buildSyncInstance(instance, taskDO));
instance.getInstanceId();
instanceKeySet.add(composeInstanceKey(instance.getIp(), instance.getPort()));
}
}
// 再将不存在的删掉
Response<List<HealthService>> serviceResponse =
consulClient.getHealthServices(taskDO.getServiceName(), true, QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)
&& !instanceKeySet.contains(composeInstanceKey(healthService.getService().getAddress(),
healthService.getService().getPort()))) {
consulClient.agentServiceDeregister(URLEncoder
.encode(healthService.getService().getId(), StandardCharsets.UTF_8.toString()));
}
}
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
});
sourceNamingService.subscribe(taskDO.getServiceName(), nacosListenerMap.get(taskDO.getTaskId()));
} catch (Exception e) {
log.error("sync task from nacos to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
public void removeInvalidInstance(TaskDO taskDO, Set<String> invalidInstanceKeys)
throws UnsupportedEncodingException {
ConsulClient consulClient = consulServerHolder.get(taskDO.getDestClusterId());
Response<List<HealthService>> serviceResponse = consulClient.getHealthServices(taskDO.getServiceName(), true,
QueryParams.DEFAULT);
List<HealthService> healthServices = serviceResponse.getValue();
for (HealthService healthService : healthServices) {
if (needDelete(ConsulUtils.transferMetadata(healthService.getService().getTags()), taskDO)
&& !invalidInstanceKeys.contains(composeInstanceKey(healthService.getService().getAddress(),
healthService.getService().getPort()))) {
consulClient.agentServiceDeregister(
URLEncoder.encode(healthService.getService().getId(), StandardCharsets.UTF_8));
}
}
return true;
}
private String composeInstanceKey(String ip, int port) {
return ip + ":" + port;
}
public NewService buildSyncInstance(Instance instance, TaskDO taskDO) {
NewService newService = new NewService();
newService.setAddress(instance.getIp());
@ -158,13 +109,13 @@ public class NacosSyncToConsulServiceImpl implements SyncService {
newService.setId(instance.getInstanceId());
List<String> tags = Lists.newArrayList();
tags.addAll(instance.getMetadata().entrySet().stream()
.map(entry -> String.join("=", entry.getKey(), entry.getValue())).collect(Collectors.toList()));
tags.add(String.join("=", SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId()));
tags.add(String.join("=", SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode()));
tags.add(String.join("=", SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId()));
.map(entry -> String.join(DELIMITER, entry.getKey(), entry.getValue())).collect(Collectors.toList()));
tags.add(String.join(DELIMITER, SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId()));
tags.add(String.join(DELIMITER, SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode()));
tags.add(String.join(DELIMITER, SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId()));
newService.setTags(tags);
return newService;
}
}

View File

@ -10,23 +10,17 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.Event;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
import com.alibaba.nacos.api.naming.pojo.Instance;
import com.alibaba.nacos.common.utils.CollectionUtils;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.eureka.EurekaNamingService;
import com.alibaba.nacossync.extension.holder.EurekaServerHolder;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.netflix.appinfo.DataCenterInfo;
import com.netflix.appinfo.InstanceInfo;
@ -34,171 +28,96 @@ import com.netflix.appinfo.LeaseInfo;
import com.netflix.appinfo.MyDataCenterInfo;
import lombok.extern.slf4j.Slf4j;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* @author zhanglong
*/
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.EUREKA)
public class NacosSyncToEurekaServiceImpl implements SyncService {
private final Map<String, EventListener> nacosListenerMap = new ConcurrentHashMap<>();
private final MetricsManager metricsManager;
private final SkyWalkerCacheServices skyWalkerCacheServices;
private final NacosServerHolder nacosServerHolder;
public class NacosSyncToEurekaServiceImpl extends AbstractNacosSync {
private final EurekaServerHolder eurekaServerHolder;
public NacosSyncToEurekaServiceImpl(MetricsManager metricsManager, SkyWalkerCacheServices skyWalkerCacheServices,
NacosServerHolder nacosServerHolder, EurekaServerHolder eurekaServerHolder) {
this.metricsManager = metricsManager;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
private final SkyWalkerCacheServices skyWalkerCacheServices;
public NacosSyncToEurekaServiceImpl(EurekaServerHolder eurekaServerHolder,
SkyWalkerCacheServices skyWalkerCacheServices) {
this.eurekaServerHolder = eurekaServerHolder;
this.skyWalkerCacheServices = skyWalkerCacheServices;
}
@Override
public boolean delete(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getGroupName());
EurekaNamingService destNamingService =
eurekaServerHolder.get(taskDO.getDestClusterId(), taskDO.getGroupName());
sourceNamingService.unsubscribe(taskDO.getServiceName(), nacosListenerMap.get(taskDO.getTaskId()));
// 删除目标集群中同步的实例列表
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (allInstances != null) {
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance);
}
}
}
} catch (Exception e) {
log.error("delete task from nacos to eureka was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getGroupName());
EurekaNamingService destNamingService =
eurekaServerHolder.get(taskDO.getDestClusterId(), taskDO.getGroupName());
nacosListenerMap.putIfAbsent(taskDO.getTaskId(), event -> {
processNamingEvent(taskDO, sourceNamingService, destNamingService, event);
});
sourceNamingService.subscribe(taskDO.getServiceName(), nacosListenerMap.get(taskDO.getTaskId()));
} catch (Exception e) {
log.error("sync task from eureka to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
return false;
}
return true;
}
private void processNamingEvent(TaskDO taskDO, NamingService sourceNamingService,
EurekaNamingService destNamingService, Event event) {
if (event instanceof NamingEvent) {
try {
Set<String> instanceKeySet = new HashSet<>();
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName());
// 先将新的注册一遍
addAllNewInstance(taskDO, destNamingService, instanceKeySet, sourceInstances);
// 再将不存在的删掉
ifNecessaryDelete(taskDO, destNamingService, instanceKeySet);
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
}
private void ifNecessaryDelete(TaskDO taskDO, EurekaNamingService destNamingService, Set<String> instanceKeySet) {
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (allInstances != null){
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO) && !instanceKeySet.contains(composeInstanceKey(instance.getIPAddr(),
instance.getPort()))) {
destNamingService.deregisterInstance(instance);
}
}
}
}
private void addAllNewInstance(TaskDO taskDO, EurekaNamingService destNamingService, Set<String> instanceKeySet,
List<Instance> sourceInstances) {
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
destNamingService.registerInstance(buildSyncInstance(instance, taskDO));
instanceKeySet.add(composeInstanceKey(instance.getIp(), instance.getPort()));
}
}
}
private String composeInstanceKey(String ip, int port) {
public String composeInstanceKey(String ip, int port) {
return ip + ":" + port;
}
@Override
public void register(TaskDO taskDO, Instance instance) {
EurekaNamingService destNamingService = eurekaServerHolder.get(taskDO.getDestClusterId());
destNamingService.registerInstance(buildSyncInstance(instance, taskDO));
}
@Override
public void deregisterInstance(TaskDO taskDO) {
EurekaNamingService destNamingService = eurekaServerHolder.get(taskDO.getDestClusterId());
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (allInstances != null) {
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance);
}
}
}
}
@Override
public void removeInvalidInstance(TaskDO taskDO, Set<String> invalidInstanceKeys) {
EurekaNamingService destNamingService = eurekaServerHolder.get(taskDO.getDestClusterId());
List<InstanceInfo> allInstances = destNamingService.getApplications(taskDO.getServiceName());
if (CollectionUtils.isNotEmpty(allInstances)) {
for (InstanceInfo instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO) && invalidInstanceKeys.contains(
composeInstanceKey(instance.getIPAddr(), instance.getPort()))) {
destNamingService.deregisterInstance(instance);
}
}
}
}
private InstanceInfo buildSyncInstance(Instance instance, TaskDO taskDO) {
DataCenterInfo dataCenterInfo = new MyDataCenterInfo(DataCenterInfo.Name.MyOwn);
final Map<String, String> instanceMetadata = instance.getMetadata();
HashMap<String, String> metadata = new HashMap<>(16);
metadata.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metadata.put(SkyWalkerConstants.SYNC_SOURCE_KEY, skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metadata.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metadata.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metadata.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metadata.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
metadata.putAll(instanceMetadata);
String homePageUrl = obtainHomePageUrl(instance, instanceMetadata);
String serviceName = taskDO.getServiceName();
return new InstanceInfo(
instance.getIp() + ":" + serviceName + ":" + instance.getPort(),
serviceName,
null,
instance.getIp(),
null,
new InstanceInfo.PortWrapper(true, instance.getPort()),
null,
homePageUrl+"/actuator/env",
homePageUrl + "/actuator/info",
homePageUrl + "/actuator/health",
null,
serviceName,
serviceName,
1,
dataCenterInfo,
instance.getIp(),
InstanceInfo.InstanceStatus.UP,
InstanceInfo.InstanceStatus.UNKNOWN,
null,
new LeaseInfo(30, 90,
0L, 0L, 0L, 0L, 0L),
false,
metadata,
System.currentTimeMillis(),
System.currentTimeMillis(),
null,
null
);
return new InstanceInfo(instance.getIp() + ":" + serviceName + ":" + instance.getPort(), serviceName, null,
instance.getIp(), null, new InstanceInfo.PortWrapper(true, instance.getPort()), null,
homePageUrl + "/actuator/env", homePageUrl + "/actuator/info", homePageUrl + "/actuator/health", null,
serviceName, serviceName, 1, dataCenterInfo, instance.getIp(), InstanceInfo.InstanceStatus.UP,
InstanceInfo.InstanceStatus.UNKNOWN, null, new LeaseInfo(30, 90, 0L, 0L, 0L, 0L, 0L), false, metadata,
System.currentTimeMillis(), System.currentTimeMillis(), null, null);
}
private String obtainHomePageUrl(Instance instance, Map<String, String> instanceMetadata) {
final String managementContextPath =
obtainManagementContextPath(instanceMetadata);
final String managementContextPath = obtainManagementContextPath(instanceMetadata);
final String managementPort = instanceMetadata.getOrDefault(SkyWalkerConstants.MANAGEMENT_PORT_KEY,
String.valueOf(instance.getPort()));
return String.format("http://%s:%s%s",instance.getIp(),managementPort,managementContextPath);
String.valueOf(instance.getPort()));
return String.format("http://%s:%s%s", instance.getIp(), managementPort, managementContextPath);
}
private String obtainManagementContextPath(Map<String, String> instanceMetadata) {
final String path = instanceMetadata.getOrDefault(SkyWalkerConstants.MANAGEMENT_CONTEXT_PATH_KEY, "");
if (path.endsWith("/")) {
@ -206,7 +125,6 @@ public class NacosSyncToEurekaServiceImpl implements SyncService {
}
return path;
}
}

View File

@ -10,9 +10,9 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.common.Constants;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
@ -23,24 +23,32 @@ import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.extension.SyncService;
import com.alibaba.nacossync.extension.annotation.NacosSyncService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.Collections;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import com.alibaba.nacossync.util.StringUtils;
import com.google.common.base.Stopwatch;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.DisposableBean;
import org.springframework.beans.factory.InitializingBean;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
/**
* @author yangyshdan
@ -49,85 +57,173 @@ import org.springframework.beans.factory.annotation.Autowired;
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.NACOS)
public class NacosSyncToNacosServiceImpl implements SyncService {
private Map<String, EventListener> listenerMap = new ConcurrentHashMap<>();
private final Map<String, Set<String>> sourceInstanceSnapshot = new ConcurrentHashMap<>();
@Autowired
private MetricsManager metricsManager;
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private NacosServerHolder nacosServerHolder;
public class NacosSyncToNacosServiceImpl implements SyncService, InitializingBean, DisposableBean {
private final Map<String, EventListener> listenerMap = new ConcurrentHashMap<>();
private final Map<String, Integer> syncTaskTap = new ConcurrentHashMap<>();
private final ConcurrentHashMap<String, TaskDO> allSyncTaskMap = new ConcurrentHashMap<>();
private ScheduledExecutorService executorService;
private final MetricsManager metricsManager;
private final NacosServerHolder nacosServerHolder;
private final ClusterAccessService clusterAccessService;
private final SkyWalkerCacheServices skyWalkerCacheServices;
public NacosSyncToNacosServiceImpl(MetricsManager metricsManager, NacosServerHolder nacosServerHolder,
ClusterAccessService clusterAccessService, SkyWalkerCacheServices skyWalkerCacheServices) {
this.metricsManager = metricsManager;
this.nacosServerHolder = nacosServerHolder;
this.clusterAccessService = clusterAccessService;
this.skyWalkerCacheServices = skyWalkerCacheServices;
}
/**
* Due to network issues or other reasons, the Nacos Sync synchronization tasks may fail,
* resulting in the target cluster's registry missing synchronized instances.
* To prevent the target cluster's registry from missing synchronized instances for an extended period,
* a fallback worker thread is started every 5 minutes to execute all synchronization tasks.
*/
@Override
public void afterPropertiesSet() {
initializeExecutorService();
scheduleSyncTasks();
}
@Override
public void destroy() {
if (executorService != null && !executorService.isShutdown()) {
executorService.shutdown();
try {
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
executorService.shutdownNow();
}
} catch (InterruptedException e) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
}
private void initializeExecutorService() {
executorService = Executors.newSingleThreadScheduledExecutor(r -> {
Thread t = new Thread(r);
t.setDaemon(true);
t.setName("com.alibaba.nacossync.basic.synctask");
return t;
});
}
private void scheduleSyncTasks() {
executorService.scheduleWithFixedDelay(this::executeSyncTasks, 0, 300, TimeUnit.SECONDS);
}
private void executeSyncTasks() {
if (allSyncTaskMap.isEmpty()) {
return;
}
Collection<TaskDO> taskCollections = allSyncTaskMap.values();
List<TaskDO> taskDOList = new ArrayList<>(taskCollections);
if (CollectionUtils.isNotEmpty(taskDOList)) {
BatchTaskExecutor.batchOperation(taskDOList, this::executeTask);
}
}
private void executeTask(TaskDO task) {
Stopwatch stopwatch = Stopwatch.createStarted();
String taskId = task.getTaskId();
try {
NamingService sourceNamingService = nacosServerHolder.get(task.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(task.getDestClusterId());
doSync(taskId, task, sourceNamingService, destNamingService);
} catch (NacosException e) {
log.error("sync task from nacos to nacos failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} catch (Exception e) {
log.error("Unexpected error during sync task, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
} finally {
stopwatch.stop();
log.debug("Task execution time for taskId {}: {} ms", taskId, stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}
@Override
public boolean delete(TaskDO taskDO) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getNameSpace());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), taskDO.getNameSpace());
//移除订阅
sourceNamingService
.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.remove(taskDO.getTaskId()));
sourceInstanceSnapshot.remove(taskDO.getTaskId());
// 删除目标集群中同步的实例列表
List<Instance> sourceInstances = sourceNamingService
.getAllInstances(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
new ArrayList<>(), false);
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
int level = clusterAccessService.findClusterLevel(taskDO.getSourceClusterId());
String taskId = taskDO.getTaskId();
//Handle individual service
if (StringUtils.isEmpty(taskId)) {
log.warn("taskId is null data synchronization is not currently performed.{}", taskId);
return false;
}
EventListener listener = listenerMap.remove(taskId);
if (listener!= null) {
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()), listener);
}
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), false);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
destNamingService
.deregisterInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
instance.getIp(),
instance.getPort());
if (needSync(instance.getMetadata(), level, taskDO.getDestClusterId())) {
destNamingService.deregisterInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), instance);
}
}
// Remove all tasks that need to be synchronized.
allSyncTaskMap.remove(taskId);
} catch (Exception e) {
log.error("delete task from nacos to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
log.error("delete task from nacos to nacos was failed, operationalId:{}", taskDO.getOperationId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
log.info("Thread {} started synchronization at {}", Thread.currentThread().getId(), System.currentTimeMillis());
String taskId = taskDO.getTaskId();
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getNameSpace());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), taskDO.getNameSpace());
NamingService sourceNamingService = nacosServerHolder.get(taskDO.getSourceClusterId());
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
allSyncTaskMap.put(taskId, taskDO);
// To prevent issues where tasks paused for synchronization, newly created tasks after deletion,
// or resynchronization tasks do not receive new events and hence cannot synchronize,
// perform a full synchronization of tasks before subscribing to events each time.
Stopwatch stopwatch = Stopwatch.createStarted();
doSync(taskId, taskDO, sourceNamingService, destNamingService);
log.debug("Time taken to synchronize a service registration: {} ms",
stopwatch.elapsed(TimeUnit.MILLISECONDS));
this.listenerMap.putIfAbsent(taskId, event -> {
if (event instanceof NamingEvent) {
NamingEvent namingEvent = (NamingEvent) event;
log.info("Detected changes in service {} information, taskId: {}, number of instances: {}, initiating synchronization",
namingEvent.getServiceName(), taskId,
namingEvent.getInstances() == null ? null : namingEvent.getInstances().size());
try {
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
new ArrayList<>(), false);
// 先删除不存在的
this.removeInvalidInstance(taskDO, destNamingService, sourceInstances);
// 如果同步实例已经为空代表该服务所有实例已经下线,清除本地持有快照
if (sourceInstances.isEmpty()) {
sourceInstanceSnapshot.remove(taskId);
return;
}
// 同步实例
this.syncNewInstance(taskDO, destNamingService, sourceInstances);
doSync(taskId, taskDO, sourceNamingService, destNamingService);
log.info("Detected synchronization end for service {}", namingEvent.getServiceName());
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
}
});
sourceNamingService.subscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
listenerMap.get(taskId));
listenerMap.get(taskId));
} catch (Exception e) {
log.error("sync task from nacos to nacos was failed, taskId:{}", taskId, e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -135,63 +231,192 @@ public class NacosSyncToNacosServiceImpl implements SyncService {
}
return true;
}
private void syncNewInstance(TaskDO taskDO, NamingService destNamingService,
List<Instance> sourceInstances) throws NacosException {
Set<String> latestSyncInstance = new TreeSet<>();
//再次添加新实例
String taskId = taskDO.getTaskId();
Set<String> instanceKeys = sourceInstanceSnapshot.get(taskId);
private void doSync(String taskId, TaskDO taskDO, NamingService sourceNamingService,
NamingService destNamingService) throws NacosException {
if (syncTaskTap.putIfAbsent(taskId, 1) != null) {
log.info("Task ID:{} - the previous synchronization task has not finished yet", taskId);
return;
}
try {
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
int level = clusterAccessService.findClusterLevel(taskDO.getSourceClusterId());
if (CollectionUtils.isNotEmpty(sourceInstances) && sourceInstances.get(0).isEphemeral()) {
// Handle batch data synchronization of ephemeral instances, need to get all current service instances.
// TODO: When the Client is version 1.x, execute the same synchronization method as persistent instances.
handlerPersistenceInstance(taskDO, destNamingService, sourceInstances, level);
} else if (CollectionUtils.isEmpty(sourceInstances)) {
// If the current source cluster is empty, then directly deregister the instances in the target cluster.
log.debug("service {} needs to sync ephemeral instance num is null: serviceName ", taskDO.getServiceName());
processDeRegisterInstances(taskDO, destNamingService);
} else {
// Handle batch data synchronization of persistent instances.
handlerPersistenceInstance(taskDO, destNamingService, sourceInstances, level);
}
} finally {
syncTaskTap.remove(taskId);
}
}
private void handlerPersistenceInstance(TaskDO taskDO, NamingService destNamingService,
List<Instance> sourceInstances, int level) throws NacosException {
List<Instance> needRegisterInstance = new ArrayList<>();
for (Instance instance : sourceInstances) {
if (needSync(instance.getMetadata())) {
String instanceKey = composeInstanceKey(instance);
if (CollectionUtils.isEmpty(instanceKeys) || !instanceKeys.contains(instanceKey)) {
destNamingService.registerInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()),
buildSyncInstance(instance, taskDO));
}
latestSyncInstance.add(instanceKey);
if (needSync(instance.getMetadata(), level, taskDO.getDestClusterId())) {
needRegisterInstance.add(buildSyncInstance(instance, taskDO));
}
}
if (CollectionUtils.isNotEmpty(latestSyncInstance)) {
log.info("任务Id:{},已同步实例个数:{}", taskId, latestSyncInstance.size());
sourceInstanceSnapshot.put(taskId, latestSyncInstance);
List<Instance> destAllInstances = destNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
// Get the instances that the destination cluster has already synchronized
List<Instance> destHasSyncInstances = destAllInstances.stream()
.filter(instance -> hasSync(instance, taskDO.getSourceClusterId())).collect(Collectors.toList());
// The following two conversions are necessary because the Nacos Instance's equals method
// is flawed and cannot be used for direct comparison.
// The reason is that Instance's equals method compares Metadata using the toString method,
// and Metadata is of type HashMap, which does not guarantee order in its toString representation.
// Convert destHasSyncInstances to a Map with the concatenated string as key
Map<String, Instance> destInstanceMap = destHasSyncInstances.stream()
.collect(Collectors.toMap(NacosSyncToNacosServiceImpl::getInstanceKey, instance -> instance));
// Convert newInstances to a Map with the concatenated string as key
Map<String, Instance> needRegisterMap = needRegisterInstance.stream()
.collect(Collectors.toMap(NacosSyncToNacosServiceImpl::getInstanceKey, instance -> instance));
// Remove instances from newInstanceMap that are present in destInstanceMap
List<Instance> newInstances = removeSyncedInstances(destInstanceMap, needRegisterMap);
// Remove instances from destInstanceMap that are present in newInstanceMap
List<Instance> invalidInstances = getInvalidInstances(destInstanceMap, needRegisterMap);
// Register each instance one by one. Take one instance at a time.
for (Instance newInstance : newInstances) {
destNamingService.registerInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
newInstance);
}
if (CollectionUtils.isNotEmpty(invalidInstances)) {
log.info("taskId: {}, service {} deregistered, number of executions: {}", taskDO.getTaskId(), taskDO.getServiceName(),
destHasSyncInstances.size());
}
for (Instance instance : invalidInstances) {
destNamingService.deregisterInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
instance);
}
}
private void removeInvalidInstance(TaskDO taskDO, NamingService destNamingService,
List<Instance> sourceInstances) throws NacosException {
String taskId = taskDO.getTaskId();
if (this.sourceInstanceSnapshot.containsKey(taskId)) {
Set<String> oldInstanceKeys = this.sourceInstanceSnapshot.get(taskId);
List<String> newInstanceKeys = sourceInstances.stream().map(this::composeInstanceKey)
private List<Instance> getInvalidInstances(Map<String, Instance> destInstanceMap, Map<String, Instance> needRegisterMap) {
Map<String, Instance> destClone = new HashMap<>(destInstanceMap);
// Convert newInstances to a Map with the concatenated string as key
Map<String, Instance> needRegisterClone = new HashMap<>(needRegisterMap);
// Remove instances from newInstanceMap that are present in destInstanceMap
destClone.keySet().removeAll(needRegisterClone.keySet());
return new ArrayList<>(destClone.values());
}
public List<Instance> removeSyncedInstances(Map<String, Instance> destInstanceMap, Map<String, Instance> needRegisterMap) {
// Convert destHasSyncInstances to a Map with the concatenated string as key
Map<String, Instance> destClone = new HashMap<>(destInstanceMap);
// Convert newInstances to a Map with the concatenated string as key
Map<String, Instance> needRegisterClone = new HashMap<>(needRegisterMap);
// Remove instances from newInstanceMap that are present in destInstanceMap
needRegisterClone.keySet().removeAll(destClone.keySet());
return new ArrayList<>(needRegisterClone.values());
}
private boolean hasSync(Instance instance, String sourceClusterId) {
if (instance.getMetadata() != null) {
String sourceClusterKey = instance.getMetadata().get(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY);
return sourceClusterKey != null && sourceClusterKey.equals(sourceClusterId);
}
return false;
}
/**
* When the number of instances that the source cluster needs to synchronize is 0,
* if the target cluster still has instances synchronized with the source cluster,
* perform unregistration.
*
* @param destNamingService Destination cluster naming service
* @throws NacosException
*/
private void processDeRegisterInstances(TaskDO taskDO, NamingService destNamingService) throws NacosException {
// If the instances in sourceInstances are empty, it means the instances are offline or do not exist.
List<Instance> destInstances = destNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), false);
// If the instances in the target cluster are also empty, then no operation is needed.
if (CollectionUtils.isEmpty(destInstances)) {
return;
}
destInstances = filterInstancesForRemoval(destInstances, taskDO.getSourceClusterId());
if (CollectionUtils.isNotEmpty(destInstances)) {
// Deregister each instance one by one. Take one instance at a time.
// Need to handle redo, otherwise, it will be registered again.
for (Instance destInstance : destInstances) {
destNamingService.deregisterInstance(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), destInstance);
}
}
}
private List<Instance> filterInstancesForRemoval(List<Instance> destInstances, String sourceClusterId) {
return destInstances.stream().filter(instance -> !instance.getMetadata().isEmpty())
.filter(instance -> needDeregister(instance.getMetadata().get(SOURCE_CLUSTER_ID_KEY), sourceClusterId))
.collect(Collectors.toList());
Collection<String> instanceKeys = Collections.subtract(oldInstanceKeys, newInstanceKeys);
for (String instanceKey : instanceKeys) {
log.info("任务Id:{},移除无效同步实例:{}", taskId, instanceKey);
String[] split = instanceKey.split(":", -1);
destNamingService
.deregisterInstance(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()), split[0],
Integer.parseInt(split[1]));
}
}
private boolean needDeregister(String destClusterId, String sourceClusterId) {
if (!StringUtils.isEmpty(destClusterId)) {
return destClusterId.equals(sourceClusterId);
}
return false;
}
private String composeInstanceKey(Instance instance) {
return instance.getIp() + ":" + instance.getPort();
private boolean needSync(Map<String, String> sourceMetaData, int level, String destClusterId) {
// Regular cluster (default)
if (level == 0) {
return SyncService.super.needSync(sourceMetaData);
}
// Central cluster, as long as the instance is not from the target cluster,
// it needs to be synchronized (extended functionality)
return !destClusterId.equals(sourceMetaData.get(SOURCE_CLUSTER_ID_KEY));
}
private String getGroupNameOrDefault(String groupName) {
return StringUtils.defaultIfBlank(groupName, Constants.DEFAULT_GROUP);
}
private Instance buildSyncInstance(Instance instance, TaskDO taskDO) {
Instance temp = getInstance(instance);
temp.addMetadata(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
temp.addMetadata(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
temp.addMetadata(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
//The flag is a synchronous instance
temp.addMetadata(SkyWalkerConstants.SYNC_INSTANCE_TAG,
taskDO.getSourceClusterId() + "@@" + taskDO.getVersion());
return temp;
}
private static Instance getInstance(Instance instance) {
Instance temp = new Instance();
temp.setInstanceId(instance.getInstanceId());
temp.setIp(instance.getIp());
temp.setPort(instance.getPort());
temp.setClusterName(instance.getClusterName());
@ -200,15 +425,19 @@ public class NacosSyncToNacosServiceImpl implements SyncService {
temp.setHealthy(instance.isHealthy());
temp.setWeight(instance.getWeight());
temp.setEphemeral(instance.isEphemeral());
Map<String, String> metaData = new HashMap<>();
metaData.putAll(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
Map<String, String> metaData = new HashMap<>(instance.getMetadata());
temp.setMetadata(metaData);
return temp;
}
private static String getInstanceKey(Instance instance) {
return String.join("|",
instance.getIp(),
String.valueOf(instance.getPort()),
String.valueOf(instance.getWeight()),
String.valueOf(instance.isHealthy()),
String.valueOf(instance.isEphemeral()),
instance.getClusterName(),
instance.getServiceName());
}
}

View File

@ -12,9 +12,6 @@
*/
package com.alibaba.nacossync.extension.impl;
import static com.alibaba.nacossync.util.StringUtils.convertDubboFullPathForZk;
import static com.alibaba.nacossync.util.StringUtils.convertDubboProvidersPath;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.listener.EventListener;
import com.alibaba.nacos.api.naming.listener.NamingEvent;
@ -32,20 +29,25 @@ import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.DubboConstants;
import com.google.common.collect.Sets;
import java.io.UnsupportedEncodingException;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import lombok.extern.slf4j.Slf4j;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.PathChildrenCache;
import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
import org.apache.curator.utils.CloseableUtils;
import org.apache.zookeeper.CreateMode;
import org.springframework.beans.factory.annotation.Autowired;
import java.io.UnsupportedEncodingException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
import static com.alibaba.nacossync.util.StringUtils.convertDubboFullPathForZk;
import static com.alibaba.nacossync.util.StringUtils.convertDubboProvidersPath;
/**
* Nacos 同步 Zk 数据
@ -57,8 +59,7 @@ import org.springframework.beans.factory.annotation.Autowired;
@NacosSyncService(sourceCluster = ClusterTypeEnum.NACOS, destinationCluster = ClusterTypeEnum.ZK)
public class NacosSyncToZookeeperServiceImpl implements SyncService {
@Autowired
private MetricsManager metricsManager;
private final MetricsManager metricsManager;
/**
* @description The Nacos listener map.
@ -90,12 +91,13 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
private final ZookeeperServerHolder zookeeperServerHolder;
@Autowired
public NacosSyncToZookeeperServiceImpl(SkyWalkerCacheServices skyWalkerCacheServices,
NacosServerHolder nacosServerHolder, ZookeeperServerHolder zookeeperServerHolder) {
NacosServerHolder nacosServerHolder, ZookeeperServerHolder zookeeperServerHolder, MetricsManager metricsManager) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.nacosServerHolder = nacosServerHolder;
this.zookeeperServerHolder = zookeeperServerHolder;
this.metricsManager = metricsManager;
}
@Override
@ -103,15 +105,18 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getGroupName());
nacosServerHolder.get(taskDO.getSourceClusterId());
EventListener eventListener = nacosListenerMap.remove(taskDO.getTaskId());
PathChildrenCache pathChildrenCache = pathChildrenCacheMap.get(taskDO.getTaskId());
sourceNamingService.unsubscribe(taskDO.getServiceName(), eventListener);
sourceNamingService.unsubscribe(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()),
eventListener);
CloseableUtils.closeQuietly(pathChildrenCache);
Set<String> instanceUrlSet = instanceBackupMap.get(taskDO.getTaskId());
CuratorFramework client = zookeeperServerHolder.get(taskDO.getDestClusterId(), taskDO.getGroupName());
for (String instanceUrl : instanceUrlSet) {
client.delete().quietly().forPath(instanceUrl);
CuratorFramework client = zookeeperServerHolder.get(taskDO.getDestClusterId());
if(!instanceUrlSet.isEmpty()){
for (String instanceUrl : instanceUrlSet) {
client.delete().quietly().forPath(instanceUrl);
}
}
} catch (Exception e) {
log.error("delete task from nacos to zk was failed, taskId:{}", taskDO.getTaskId(), e);
@ -122,23 +127,24 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
try {
NamingService sourceNamingService =
nacosServerHolder.get(taskDO.getSourceClusterId(), taskDO.getGroupName());
CuratorFramework client = zookeeperServerHolder.get(taskDO.getDestClusterId(), taskDO.getGroupName());
nacosServerHolder.get(taskDO.getSourceClusterId());
CuratorFramework client = zookeeperServerHolder.get(taskDO.getDestClusterId());
nacosListenerMap.putIfAbsent(taskDO.getTaskId(), event -> {
if (event instanceof NamingEvent) {
try {
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName());
List<Instance> sourceInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
Set<String> newInstanceUrlSet = getWaitingToAddInstance(taskDO, client, sourceInstances);
// 获取之前的备份 删除无效实例
// fetch the instance backup
deleteInvalidInstances(taskDO, client, newInstanceUrlSet);
// 替换当前备份为最新备份
// replace the instance backup
instanceBackupMap.put(taskDO.getTaskId(), newInstanceUrlSet);
// 尝试恢复因为zk客户端意外断开导致的实例数据
// try to compensate for the removed instance
tryToCompensate(taskDO, sourceNamingService, sourceInstances);
} catch (Exception e) {
log.error("event process fail, taskId:{}", taskDO.getTaskId(), e);
@ -148,7 +154,8 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
});
sourceNamingService.subscribe(taskDO.getServiceName(), nacosListenerMap.get(taskDO.getTaskId()));
sourceNamingService.subscribe(taskDO.getServiceName(),getGroupNameOrDefault(taskDO.getGroupName()),
nacosListenerMap.get(taskDO.getTaskId()));
} catch (Exception e) {
log.error("sync task from nacos to zk was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -156,28 +163,45 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
return true;
}
private void tryToCompensate(TaskDO taskDO, NamingService sourceNamingService, List<Instance> sourceInstances) {
if (!CollectionUtils.isEmpty(sourceInstances)) {
final PathChildrenCache pathCache = getPathCache(taskDO);
if (pathCache.getListenable().size() == 0) { // 防止重复注册
pathCache.getListenable().addListener((zkClient, zkEvent) -> {
if (zkEvent.getType() == PathChildrenCacheEvent.Type.CHILD_REMOVED) {
List<Instance> allInstances =
sourceNamingService.getAllInstances(taskDO.getServiceName());
for (Instance instance : allInstances) {
String instanceUrl = buildSyncInstance(instance, taskDO);
String zkInstancePath = zkEvent.getData().getPath();
if (zkInstancePath.equals(instanceUrl)) {
zkClient.create().creatingParentsIfNeeded().withMode(CreateMode.EPHEMERAL)
.forPath(zkInstancePath);
break;
}
}
}
});
// Avoiding re-registration if there is already a listener registered.
if (pathCache.getListenable().size() == 0) {
registerCompensationListener(pathCache, taskDO, sourceNamingService);
}
}
}
private void registerCompensationListener(PathChildrenCache pathCache, TaskDO taskDO, NamingService sourceNamingService) {
pathCache.getListenable().addListener((zkClient, zkEvent) -> {
try {
if (zkEvent.getType() == PathChildrenCacheEvent.Type.CHILD_REMOVED) {
compensateOnChildRemoval(zkClient, zkEvent, sourceNamingService, taskDO);
}
} catch (Exception e) {
log.error("Error processing ZooKeeper event: {}", zkEvent.getType(), e);
}
});
}
private void compensateOnChildRemoval(CuratorFramework zkClient, PathChildrenCacheEvent zkEvent, NamingService sourceNamingService, TaskDO taskDO) {
String zkInstancePath = null;
try {
List<Instance> allInstances = sourceNamingService.getAllInstances(taskDO.getServiceName(), getGroupNameOrDefault(taskDO.getGroupName()));
zkInstancePath = zkEvent.getData().getPath();
for (Instance instance : allInstances) {
String instanceUrl = buildSyncInstance(instance, taskDO);
if (zkInstancePath.equals(instanceUrl)) {
zkClient.create().creatingParentsIfNeeded().withMode(CreateMode.EPHEMERAL)
.forPath(zkInstancePath);
log.info("Compensated by re-creating the removed node at path: {}", zkInstancePath);
break;
}
}
} catch (Exception e) {
log.error("Failed to compensate for the removed node at path: {}", zkInstancePath, e);
}
}
@ -210,12 +234,11 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
}
protected String buildSyncInstance(Instance instance, TaskDO taskDO) throws UnsupportedEncodingException {
Map<String, String> metaData = new HashMap<>();
metaData.putAll(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
Map<String, String> metaData = new HashMap<>(instance.getMetadata());
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
String servicePath = monitorPath.computeIfAbsent(taskDO.getTaskId(),
key -> convertDubboProvidersPath(metaData.get(DubboConstants.INTERFACE_KEY)));
@ -225,16 +248,16 @@ public class NacosSyncToZookeeperServiceImpl implements SyncService {
/**
* 获取zk path child 监听缓存类
* fetch zk path cache
*
* @param taskDO 任务对象
* @return zk节点操作缓存对象
* @param taskDO task instance
* @return zk path cache
*/
private PathChildrenCache getPathCache(TaskDO taskDO) {
return pathChildrenCacheMap.computeIfAbsent(taskDO.getTaskId(), (key) -> {
try {
PathChildrenCache pathChildrenCache = new PathChildrenCache(
zookeeperServerHolder.get(taskDO.getDestClusterId(), ""), monitorPath.get(key), false);
zookeeperServerHolder.get(taskDO.getDestClusterId()), monitorPath.get(key), false);
pathChildrenCache.start(PathChildrenCache.StartMode.BUILD_INITIAL_CACHE);
return pathChildrenCache;
} catch (Exception e) {

View File

@ -10,22 +10,8 @@
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.extension.impl;
import static com.alibaba.nacossync.util.DubboConstants.ALL_SERVICE_NAME_PATTERN;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_PATH_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_ROOT_PATH;
import static com.alibaba.nacossync.util.DubboConstants.GROUP_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_IP_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_PORT_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INTERFACE_KEY;
import static com.alibaba.nacossync.util.DubboConstants.PROTOCOL_KEY;
import static com.alibaba.nacossync.util.DubboConstants.VERSION_KEY;
import static com.alibaba.nacossync.util.DubboConstants.WEIGHT_KEY;
import static com.alibaba.nacossync.util.DubboConstants.ZOOKEEPER_SEPARATOR;
import static com.alibaba.nacossync.util.DubboConstants.createServiceName;
import static com.alibaba.nacossync.util.StringUtils.parseIpAndPortString;
import static com.alibaba.nacossync.util.StringUtils.parseQueryString;
package com.alibaba.nacossync.extension.impl;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.NamingService;
@ -40,20 +26,38 @@ import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.extension.holder.ZookeeperServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Predicate;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.TreeCache;
import org.apache.curator.framework.recipes.cache.TreeCacheEvent;
import org.apache.curator.utils.CloseableUtils;
import org.springframework.beans.factory.annotation.Autowired;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import static com.alibaba.nacossync.util.DubboConstants.ALL_SERVICE_NAME_PATTERN;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_PATH_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_ROOT_PATH;
import static com.alibaba.nacossync.util.DubboConstants.GROUP_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_IP_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_PORT_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INTERFACE_KEY;
import static com.alibaba.nacossync.util.DubboConstants.PROTOCOL_KEY;
import static com.alibaba.nacossync.util.DubboConstants.VERSION_KEY;
import static com.alibaba.nacossync.util.DubboConstants.WEIGHT_KEY;
import static com.alibaba.nacossync.util.DubboConstants.ZOOKEEPER_SEPARATOR;
import static com.alibaba.nacossync.util.DubboConstants.createServiceName;
import static com.alibaba.nacossync.util.NacosUtils.getGroupNameOrDefault;
import static com.alibaba.nacossync.util.StringUtils.parseIpAndPortString;
import static com.alibaba.nacossync.util.StringUtils.parseQueryString;
/**
* @author paderlol
@ -63,59 +67,55 @@ import org.springframework.beans.factory.annotation.Autowired;
@Slf4j
@NacosSyncService(sourceCluster = ClusterTypeEnum.ZK, destinationCluster = ClusterTypeEnum.NACOS)
public class ZookeeperSyncToNacosServiceImpl implements SyncService {
@Autowired
private MetricsManager metricsManager;
private static final String DEFAULT_WEIGHT = "1.0";
private final MetricsManager metricsManager;
/**
* Listener cache of Zookeeper format taskId -> PathChildrenCache instance
*/
private Map<String, TreeCache> treeCacheMap = new ConcurrentHashMap<>();
private final Map<String, TreeCache> treeCacheMap = new ConcurrentHashMap<>();
/**
* service name cache
*/
private Map<String, String> nacosServiceNameMap = new ConcurrentHashMap<>();
private final Map<String, String> nacosServiceNameMap = new ConcurrentHashMap<>();
private final ZookeeperServerHolder zookeeperServerHolder;
private final NacosServerHolder nacosServerHolder;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
public ZookeeperSyncToNacosServiceImpl(ZookeeperServerHolder zookeeperServerHolder,
NacosServerHolder nacosServerHolder, SkyWalkerCacheServices skyWalkerCacheServices) {
NacosServerHolder nacosServerHolder, SkyWalkerCacheServices skyWalkerCacheServices,
MetricsManager metricsManager) {
this.zookeeperServerHolder = zookeeperServerHolder;
this.nacosServerHolder = nacosServerHolder;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.metricsManager = metricsManager;
}
@Override
public boolean sync(TaskDO taskDO) {
public boolean sync(TaskDO taskDO, Integer index) {
try {
if (treeCacheMap.containsKey(taskDO.getTaskId())) {
return true;
}
TreeCache treeCache = getTreeCache(taskDO);
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), null);
if (!initializeTreeCache(taskDO)) {
return false;
}
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
if (destNamingService == null) {
logAndRecordSyncError("Failed to obtain NamingService for destination clusterId: {}", taskDO.getDestClusterId(), null);
return false;
}
// 初次执行任务统一注册所有实例
registerAllInstances(taskDO, destNamingService);
//注册ZK监听
Objects.requireNonNull(treeCache).getListenable().addListener((client, event) -> {
try {
String path = event.getData().getPath();
Map<String, String> queryParam = parseQueryString(path);
if (isMatch(taskDO, queryParam) && needSync(queryParam)) {
processEvent(taskDO, destNamingService, event, path, queryParam);
}
} catch (Exception e) {
log.error("event process from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
});
setupListener(taskDO, destNamingService);
} catch (Exception e) {
log.error("sync task from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
@ -123,128 +123,105 @@ public class ZookeeperSyncToNacosServiceImpl implements SyncService {
}
return true;
}
private void processEvent(TaskDO taskDO, NamingService destNamingService, TreeCacheEvent event, String path,
Map<String, String> queryParam) throws NacosException {
if(!com.alibaba.nacossync.util.StringUtils.isDubboProviderPath(path)) {
return;
}
Map<String, String> ipAndPortParam = parseIpAndPortString(path);
Instance instance = buildSyncInstance(queryParam, ipAndPortParam, taskDO);
String serviceName = queryParam.get(INTERFACE_KEY);
switch (event.getType()) {
case NODE_ADDED:
case NODE_UPDATED:
destNamingService.registerInstance(
getServiceNameFromCache(serviceName, queryParam), instance);
break;
case NODE_REMOVED:
destNamingService.deregisterInstance(
getServiceNameFromCache(serviceName, queryParam),
ipAndPortParam.get(INSTANCE_IP_KEY),
Integer.parseInt(ipAndPortParam.get(INSTANCE_PORT_KEY)));
nacosServiceNameMap.remove(serviceName);
break;
default:
break;
}
}
private void registerAllInstances(TaskDO taskDO, NamingService destNamingService) throws Exception {
CuratorFramework zk = zookeeperServerHolder.get(taskDO.getSourceClusterId(), "");
if(!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
registerALLInstances0(taskDO, destNamingService, zk, taskDO.getServiceName());
} else {
// 同步全部
List<String> serviceList = zk.getChildren().forPath(DUBBO_ROOT_PATH);
for(String serviceName : serviceList) {
registerALLInstances0(taskDO, destNamingService, zk, serviceName);
}
}
}
private void registerALLInstances0(TaskDO taskDO, NamingService destNamingService, CuratorFramework zk,
String serviceName) throws Exception {
String path = String.format(DUBBO_PATH_FORMAT, serviceName);
if(zk.getChildren()==null) {
return;
}
List<String> providers = zk.getChildren().forPath(path);
for(String provider : providers) {
Map<String, String> queryParam = parseQueryString(provider);
if (isMatch(taskDO, queryParam) && needSync(queryParam)) {
Map<String, String> ipAndPortParam = parseIpAndPortString(path + ZOOKEEPER_SEPARATOR + provider);
Instance instance = buildSyncInstance(queryParam, ipAndPortParam, taskDO);
destNamingService.registerInstance(getServiceNameFromCache(serviceName, queryParam),
instance);
}
}
}
@Override
public boolean delete(TaskDO taskDO) {
if (taskDO.getServiceName() == null) {
return true;
}
try {
CloseableUtils.closeQuietly(treeCacheMap.get(taskDO.getTaskId()));
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId(), null);
if(!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
if (nacosServiceNameMap.containsKey(taskDO.getServiceName())) {
List<Instance> allInstances =
destNamingService.getAllInstances(nacosServiceNameMap.get(taskDO.getServiceName()));
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance.getServiceName(), instance.getIp(),
instance.getPort());
}
nacosServiceNameMap.remove(taskDO.getServiceName());
}
}
} else {
Set<String> serviceNames = nacosServiceNameMap.keySet();
for(String serviceName : serviceNames) {
if (nacosServiceNameMap.containsKey(serviceName)) {
List<Instance> allInstances =
destNamingService.getAllInstances(serviceName);
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance.getServiceName(), instance.getIp(),
instance.getPort());
}
nacosServiceNameMap.remove(serviceName);
}
}
}
}
} catch (Exception e) {
log.error("delete task from zookeeper to nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
private boolean initializeTreeCache(TaskDO taskDO) {
TreeCache treeCache = getTreeCache(taskDO);
if (treeCache == null) {
logAndRecordSyncError("Failed to obtain TreeCache for taskId: {}", taskDO.getTaskId(), null);
return false;
}
return true;
}
private void logAndRecordSyncError(String message, String taskId, Exception e) {
if (e != null) {
log.error(message, taskId, e);
} else {
log.error(message, taskId);
}
metricsManager.recordError(MetricsStatisticsType.SYNC_ERROR);
}
private void setupListener(TaskDO taskDO, NamingService destNamingService) {
TreeCache treeCache = Objects.requireNonNull(getTreeCache(taskDO));
treeCache.getListenable().addListener((client, event) -> {
try {
// INITIALIZED is a special event that is not triggered by the Zookeeper server
if(event.getData()==null){
log.warn("TreeCache event data is null, taskId:{}", taskDO.getTaskId());
return;
}
String path = event.getData().getPath();
Map<String, String> queryParam = parseQueryString(path);
if (isMatch(taskDO, queryParam) && needSync(queryParam)) {
processEvent(taskDO, destNamingService, event, path, queryParam);
}
} catch (Exception e) {
logAndRecordSyncError("Event process from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
}
});
}
public boolean delete(TaskDO taskDO) {
if (taskDO.getServiceName() == null) {
return true;
}
CloseableUtils.closeQuietly(treeCacheMap.get(taskDO.getTaskId()));
try {
NamingService destNamingService = nacosServerHolder.get(taskDO.getDestClusterId());
if (destNamingService == null) {
log.error("Failed to obtain NamingService for destination clusterId: {}", taskDO.getDestClusterId());
return false;
}
Set<String> serviceNames = getServiceNamesToDelete(taskDO);
deleteInstances(serviceNames, destNamingService, taskDO);
} catch (Exception e) {
log.error("Delete task from Zookeeper to Nacos was failed, taskId:{}", taskDO.getTaskId(), e);
metricsManager.recordError(MetricsStatisticsType.DELETE_ERROR);
return false;
}
return true;
}
private Set<String> getServiceNamesToDelete(TaskDO taskDO) {
if (!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
return new HashSet<>(Collections.singleton(taskDO.getServiceName()));
} else {
return new HashSet<>(nacosServiceNameMap.keySet());
}
}
private void deleteInstances(Set<String> serviceNames, NamingService destNamingService, TaskDO taskDO) {
for (String serviceName : serviceNames) {
try {
List<Instance> allInstances = destNamingService.getAllInstances(serviceName,
getGroupNameOrDefault(taskDO.getGroupName()), new ArrayList<>(), true);
for (Instance instance : allInstances) {
if (needDelete(instance.getMetadata(), taskDO)) {
destNamingService.deregisterInstance(instance.getServiceName(),
getGroupNameOrDefault(taskDO.getGroupName()), instance.getIp(), instance.getPort());
}
}
nacosServiceNameMap.remove(serviceName);
} catch (NacosException e) {
log.error("Failed to deregister service instance for serviceName: {}", serviceName, e);
}
}
}
/**
* fetch the Path cache when the task sync
*/
protected TreeCache getTreeCache(TaskDO taskDO) {
return treeCacheMap.computeIfAbsent(taskDO.getTaskId(), (key) -> {
try {
TreeCache treeCache =
new TreeCache(zookeeperServerHolder.get(taskDO.getSourceClusterId(), ""),
DUBBO_ROOT_PATH);
TreeCache treeCache = new TreeCache(zookeeperServerHolder.get(taskDO.getSourceClusterId()),
DUBBO_ROOT_PATH);
treeCache.start();
return treeCache;
} catch (Exception e) {
@ -252,54 +229,151 @@ public class ZookeeperSyncToNacosServiceImpl implements SyncService {
return null;
}
});
}
/**
* The instance information that needs to be synchronized is matched based on the dubbo version and the grouping
* name
* Checks if the instance information needs to be synchronized based on the dubbo version, grouping name,
* and service name.
*/
protected boolean isMatch(TaskDO taskDO, Map<String, String> queryParam) {
Predicate<TaskDO> isVersionEq = (task) -> StringUtils.isBlank(taskDO.getVersion())
|| StringUtils.equals(task.getVersion(), queryParam.get(VERSION_KEY));
Predicate<TaskDO> isGroupEq = (task) -> StringUtils.isBlank(taskDO.getGroupName())
|| StringUtils.equals(task.getGroupName(), queryParam.get(GROUP_KEY));
return isVersionEq.and(isGroupEq).test(taskDO);
return isVersionMatch(taskDO, queryParam) &&
isGroupMatch(taskDO, queryParam) &&
isServiceMatch(taskDO, queryParam) ||
isMatchAllServices(taskDO);
}
private boolean isVersionMatch(TaskDO task, Map<String, String> queryParam) {
return StringUtils.isBlank(task.getVersion()) || StringUtils.equals(task.getVersion(), queryParam.get(VERSION_KEY));
}
private boolean isGroupMatch(TaskDO task, Map<String, String> queryParam) {
return StringUtils.isBlank(task.getGroupName()) || StringUtils.equals(task.getGroupName(), queryParam.get(GROUP_KEY));
}
private boolean isServiceMatch(TaskDO task, Map<String, String> queryParam) {
return StringUtils.isNotBlank(task.getServiceName()) && StringUtils.equals(task.getServiceName(), queryParam.get(INTERFACE_KEY));
}
private boolean isMatchAllServices(TaskDO task) {
return StringUtils.isNotBlank(task.getServiceName()) && StringUtils.equals(task.getServiceName(), ALL_SERVICE_NAME_PATTERN);
}
/**
* create Nacos service instance
* Builds a synchronized Nacos instance from Zookeeper data.
*
* @param queryParam dubbo metadata
* @param ipAndPortMap dubbo ip and address
* @param queryParam Parameters obtained from the query string.
* @param ipAndPortMap IP and port information.
* @param taskDO Task details.
* @return A fully configured Nacos instance.
*/
protected Instance buildSyncInstance(Map<String, String> queryParam, Map<String, String> ipAndPortMap,
TaskDO taskDO) {
Instance temp = new Instance();
temp.setIp(ipAndPortMap.get(INSTANCE_IP_KEY));
temp.setPort(Integer.parseInt(ipAndPortMap.get(INSTANCE_PORT_KEY)));
temp.setServiceName(getServiceNameFromCache(taskDO.getTaskId(), queryParam));
temp.setWeight(Double.parseDouble(queryParam.get(WEIGHT_KEY) == null ? "1.0" : queryParam.get(WEIGHT_KEY)));
temp.setHealthy(true);
protected Instance buildSyncInstance(Map<String, String> queryParam, Map<String, String> ipAndPortMap, TaskDO taskDO) {
Instance instance = new Instance();
instance.setIp(ipAndPortMap.get(INSTANCE_IP_KEY));
instance.setPort(Integer.parseInt(ipAndPortMap.get(INSTANCE_PORT_KEY)));
instance.setServiceName(getServiceNameFromCache(taskDO.getTaskId(), queryParam));
instance.setWeight(parseWeight(queryParam));
instance.setHealthy(true);
instance.setMetadata(buildMetadata(queryParam, ipAndPortMap, taskDO));
return instance;
}
private double parseWeight(Map<String, String> queryParam) {
try {
return Double.parseDouble(queryParam.getOrDefault(WEIGHT_KEY, DEFAULT_WEIGHT));
} catch (NumberFormatException e) {
log.error("Error parsing weight: {}", queryParam.get(WEIGHT_KEY), e);
return Double.parseDouble(DEFAULT_WEIGHT); // Default weight in case of error
}
}
private Map<String, String> buildMetadata(Map<String, String> queryParam, Map<String, String> ipAndPortMap, TaskDO taskDO) {
Map<String, String> metaData = new HashMap<>(queryParam);
metaData.put(PROTOCOL_KEY, ipAndPortMap.get(PROTOCOL_KEY));
metaData.put(SkyWalkerConstants.DEST_CLUSTERID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY,
skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTERID_KEY, taskDO.getSourceClusterId());
temp.setMetadata(metaData);
return temp;
metaData.put(SkyWalkerConstants.DEST_CLUSTER_ID_KEY, taskDO.getDestClusterId());
metaData.put(SkyWalkerConstants.SYNC_SOURCE_KEY, skyWalkerCacheServices.getClusterType(taskDO.getSourceClusterId()).getCode());
metaData.put(SkyWalkerConstants.SOURCE_CLUSTER_ID_KEY, taskDO.getSourceClusterId());
return metaData;
}
private void processEvent(TaskDO taskDO, NamingService destNamingService, TreeCacheEvent event, String path,
Map<String, String> queryParam) throws NacosException {
if (!com.alibaba.nacossync.util.StringUtils.isDubboProviderPath(path)) {
return;
}
Map<String, String> ipAndPortParam = parseIpAndPortString(path);
if (ipAndPortParam.isEmpty()) {
log.error("Invalid IP and Port data extracted from path: {}", path);
return;
}
Instance instance = buildSyncInstance(queryParam, ipAndPortParam, taskDO);
String serviceName = queryParam.get(INTERFACE_KEY);
if (serviceName == null || serviceName.isEmpty()) {
log.error("Service name is missing in the query parameters.");
return;
}
switch (event.getType()) {
case NODE_ADDED:
case NODE_UPDATED:
destNamingService.registerInstance(getServiceNameFromCache(serviceName, queryParam),
getGroupNameOrDefault(taskDO.getGroupName()), instance);
break;
case NODE_REMOVED:
destNamingService.deregisterInstance(getServiceNameFromCache(serviceName, queryParam),
getGroupNameOrDefault(taskDO.getGroupName()), ipAndPortParam.get(INSTANCE_IP_KEY),
Integer.parseInt(ipAndPortParam.get(INSTANCE_PORT_KEY)));
nacosServiceNameMap.remove(serviceName);
break;
default:
break;
}
}
private void registerAllInstances(TaskDO taskDO, NamingService destNamingService) throws Exception {
CuratorFramework zk = zookeeperServerHolder.get(taskDO.getSourceClusterId());
if (!ALL_SERVICE_NAME_PATTERN.equals(taskDO.getServiceName())) {
registerALLInstances0(taskDO, destNamingService, zk, taskDO.getServiceName());
} else {
// 同步全部
List<String> serviceList = zk.getChildren().forPath(DUBBO_ROOT_PATH);
for (String serviceName : serviceList) {
registerALLInstances0(taskDO, destNamingService, zk, serviceName);
}
}
}
private void registerALLInstances0(TaskDO taskDO, NamingService destNamingService, CuratorFramework zk,
String serviceName) throws Exception {
String path = String.format(DUBBO_PATH_FORMAT, serviceName);
if (zk.getChildren() == null) {
return;
}
List<String> providers = zk.getChildren().forPath(path);
for (String provider : providers) {
Map<String, String> queryParam = parseQueryString(provider);
if (isMatch(taskDO, queryParam) && needSync(queryParam)) {
Map<String, String> ipAndPortParam = parseIpAndPortString(path + ZOOKEEPER_SEPARATOR + provider);
Instance instance = buildSyncInstance(queryParam, ipAndPortParam, taskDO);
destNamingService.registerInstance(getServiceNameFromCache(serviceName, queryParam),
getGroupNameOrDefault(taskDO.getGroupName()), instance);
}
}
}
/**
* cteate Dubbo service name
*
* @param serviceName dubbo service name
* @param queryParam dubbo metadata
* @param queryParam dubbo metadata
*/
protected String getServiceNameFromCache(String serviceName, Map<String, String> queryParam) {
return nacosServiceNameMap.computeIfAbsent(serviceName, (key) -> createServiceName(queryParam));
}
}

View File

@ -23,15 +23,19 @@ import java.util.concurrent.TimeUnit;
@Service
public class MetricsManager implements CommandLineRunner {
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private ClusterAccessService clusterAccessService;
@Autowired
private TaskAccessService taskAccessService;
private final ClusterAccessService clusterAccessService;
private final TaskAccessService taskAccessService;
public MetricsManager(SkyWalkerCacheServices skyWalkerCacheServices, ClusterAccessService clusterAccessService,
TaskAccessService taskAccessService) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.clusterAccessService = clusterAccessService;
this.taskAccessService = taskAccessService;
}
/**
* Callback used to run the bean.
*

View File

@ -17,17 +17,28 @@
package com.alibaba.nacossync.pojo.model;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.proxy.HibernateProxy;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import java.io.Serializable;
import javax.persistence.*;
import lombok.Data;
import java.util.Objects;
/**
* @author NacosSync
* @version $Id: EnvDO.java, v 0.1 2018-09-25 PM 4:17 NacosSync Exp $$
*/
@Data
@Getter
@Setter
@ToString
@RequiredArgsConstructor
@Entity
@Table(name = "cluster")
public class ClusterDO implements Serializable {
@ -66,4 +77,34 @@ public class ClusterDO implements Serializable {
*/
private String password;
private String namespace;
private Integer clusterLevel;
@Override
public final boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null) {
return false;
}
Class<?> oEffectiveClass =
o instanceof HibernateProxy ? ((HibernateProxy) o).getHibernateLazyInitializer().getPersistentClass()
: o.getClass();
Class<?> thisEffectiveClass =
this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass() : this.getClass();
if (thisEffectiveClass != oEffectiveClass) {
return false;
}
ClusterDO clusterDO = (ClusterDO) o;
return getId() != null && Objects.equals(getId(), clusterDO.getId());
}
@Override
public final int hashCode() {
return this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass().hashCode() : getClass().hashCode();
}
}

View File

@ -16,15 +16,28 @@
*/
package com.alibaba.nacossync.pojo.model;
import javax.persistence.*;
import lombok.Data;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.proxy.HibernateProxy;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import java.util.Objects;
/**
* @author NacosSync
* @version $Id: SystemConfig.java, v 0.1 2018-09-26 上午1:48 NacosSync Exp $$
*/
@Data
@Getter
@Setter
@ToString
@RequiredArgsConstructor
@Entity
@Table(name = "system_config")
public class SystemConfigDO {
@ -34,5 +47,31 @@ public class SystemConfigDO {
private String configKey;
private String configValue;
private String configDesc;
@Override
public final boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null) {
return false;
}
Class<?> oEffectiveClass =
o instanceof HibernateProxy ? ((HibernateProxy) o).getHibernateLazyInitializer().getPersistentClass()
: o.getClass();
Class<?> thisEffectiveClass =
this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass() : this.getClass();
if (thisEffectiveClass != oEffectiveClass) {
return false;
}
SystemConfigDO that = (SystemConfigDO) o;
return getId() != null && Objects.equals(getId(), that.getId());
}
@Override
public final int hashCode() {
return this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass().hashCode() : getClass().hashCode();
}
}

View File

@ -12,16 +12,29 @@
*/
package com.alibaba.nacossync.pojo.model;
import lombok.Data;
import javax.persistence.*;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.hibernate.proxy.HibernateProxy;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;
import javax.persistence.Table;
import java.io.Serializable;
import java.util.Objects;
/**
* @author NacosSync
* @version $Id: TaskDo.java, v 0.1 2018-09-24 PM11:53 NacosSync Exp $$
*/
@Data
@Getter
@Setter
@ToString
@RequiredArgsConstructor
@Entity
@Table(name = "task")
public class TaskDO implements Serializable {
@ -70,4 +83,31 @@ public class TaskDO implements Serializable {
* operation id,The operation id follow when the task status changes
*/
private String operationId;
@Override
public final boolean equals(Object o) {
if (this == o) {
return true;
}
if (o == null) {
return false;
}
Class<?> oEffectiveClass =
o instanceof HibernateProxy ? ((HibernateProxy) o).getHibernateLazyInitializer().getPersistentClass()
: o.getClass();
Class<?> thisEffectiveClass =
this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass() : this.getClass();
if (thisEffectiveClass != oEffectiveClass) {
return false;
}
TaskDO taskDO = (TaskDO) o;
return getId() != null && Objects.equals(getId(), taskDO.getId());
}
@Override
public final int hashCode() {
return this instanceof HibernateProxy ? ((HibernateProxy) this).getHibernateLazyInitializer()
.getPersistentClass().hashCode() : getClass().hashCode();
}
}

View File

@ -17,10 +17,10 @@
package com.alibaba.nacossync.pojo.request;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import java.util.List;
import lombok.Data;
import java.util.List;
/**
* @author NacosSync
* @version $Id: AddClusterRequest.java, v 0.1 2018-09-25 PM 10:27 NacosSync Exp $$
@ -52,5 +52,6 @@ public class ClusterAddRequest extends BaseRequest {
* The password of the Nacos.
*/
private String password;
private String namespace;
}

View File

@ -0,0 +1,44 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.pojo.request;
import lombok.Data;
/**
* @author NacosSync
* @version $Id: TaskAddAllRequest.java, v 0.1 2022-03-23 AM12:13 NacosSync Exp $$
*/
@Data
public class TaskAddAllRequest extends BaseRequest {
/**
* eg: b7bacb110199d5bb83b9757038fadeb0 .
*/
private String sourceClusterId;
/**
* eg: bbdad57833a0e4f0981f6f3349005617 .
*/
private String destClusterId;
/**
* whether to exclude subscriber.
*/
private boolean excludeConsumer = true;
}

View File

@ -14,33 +14,53 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.pojo.view;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import java.io.Serializable;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import lombok.Data;
import java.io.Serializable;
/**
* @author NacosSync
* @version $Id: ClusterModel.java, v 0.1 2018-09-25 下午11:09 NacosSync Exp $$
*/
@Data
public class ClusterModel implements Serializable {
private String clusterId;
/**
* json format["192.168.1:8080","192.168.2?key=1"]
*/
private String connectKeyList;
/**
* cluster name, egcluster of ShangHaiedas-sh
*/
private String clusterName;
/**
* cluster type, eg cluster of CS,cluster of Nacos,
*
* @see ClusterTypeEnum
*/
private String clusterType;
private String namespace;
private String userName;
public static ClusterModel from(ClusterDO clusterDO) {
ClusterModel clusterModel = new ClusterModel();
clusterModel.setClusterId(clusterDO.getClusterId());
clusterModel.setConnectKeyList(clusterDO.getConnectKeyList());
clusterModel.setClusterType(clusterDO.getClusterType());
clusterModel.setClusterName(clusterDO.getClusterName());
clusterModel.setNamespace(clusterDO.getNamespace());
clusterModel.setUserName(clusterDO.getUserName());
return clusterModel;
}
}

View File

@ -14,8 +14,10 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.pojo.view;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.Data;
/**
@ -24,12 +26,27 @@ import lombok.Data;
*/
@Data
public class TaskModel {
private String taskId;
private String sourceClusterId;
private String destClusterId;
private String serviceName;
private String groupName;
private String taskStatus;
public static TaskModel from(TaskDO taskDO) {
TaskModel taskModel = new TaskModel();
taskModel.setDestClusterId(taskDO.getDestClusterId());
taskModel.setGroupName(taskDO.getGroupName());
taskModel.setServiceName(taskDO.getServiceName());
taskModel.setSourceClusterId(taskDO.getSourceClusterId());
taskModel.setTaskStatus(taskDO.getTaskStatus());
taskModel.setTaskId(taskDO.getTaskId());
return taskModel;
}
}

View File

@ -45,8 +45,7 @@ public class SkyWalkerTemplate {
}
private static <T extends BaseResult> void initExceptionResult(T result, Throwable e) {
if (e instanceof SkyWalkerException) {
SkyWalkerException skyWalkerException = (SkyWalkerException) e;
if (e instanceof SkyWalkerException skyWalkerException) {
if (null != skyWalkerException.getResultCode()) {
result.setResultCode(skyWalkerException.getResultCode().getCode());
}

View File

@ -14,12 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterAddRequest;
import com.alibaba.nacossync.pojo.result.ClusterAddResult;
@ -28,7 +28,6 @@ import com.alibaba.nacossync.util.SkyWalkerUtil;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
@ -38,50 +37,53 @@ import org.springframework.stereotype.Service;
@Slf4j
@Service
public class ClusterAddProcessor implements Processor<ClusterAddRequest, ClusterAddResult> {
@Autowired
private MetricsManager metricsManager;
@Autowired
private ClusterAccessService clusterAccessService;
@Autowired
private ObjectMapper objectMapper;
private final ClusterAccessService clusterAccessService;
private final ObjectMapper objectMapper;
public ClusterAddProcessor(ClusterAccessService clusterAccessService, ObjectMapper objectMapper) {
this.clusterAccessService = clusterAccessService;
this.objectMapper = objectMapper;
}
@Override
public void process(ClusterAddRequest clusterAddRequest, ClusterAddResult clusterAddResult,
Object... others) throws Exception {
public void process(ClusterAddRequest clusterAddRequest, ClusterAddResult clusterAddResult, Object... others)
throws Exception {
ClusterDO clusterDO = new ClusterDO();
if (null == clusterAddRequest.getConnectKeyList() || 0 == clusterAddRequest.getConnectKeyList().size()) {
if (null == clusterAddRequest.getConnectKeyList() || clusterAddRequest.getConnectKeyList().isEmpty()) {
throw new SkyWalkerException("集群列表不能为空!");
}
if (StringUtils.isBlank(clusterAddRequest.getClusterName()) || StringUtils
.isBlank(clusterAddRequest.getClusterType())) {
if (StringUtils.isBlank(clusterAddRequest.getClusterName()) || StringUtils.isBlank(
clusterAddRequest.getClusterType())) {
throw new SkyWalkerException("集群名字或者类型不能为空!");
}
if (!ClusterTypeEnum.contains(clusterAddRequest.getClusterType())) {
throw new SkyWalkerException("集群类型不存在:" + clusterAddRequest.getClusterType());
}
String clusterId = SkyWalkerUtil.generateClusterId(clusterAddRequest);
if (null != clusterAccessService.findByClusterId(clusterId)) {
throw new SkyWalkerException("重复插入clusterId已存在" + clusterId);
}
clusterDO.setClusterId(clusterId);
clusterDO.setClusterName(clusterAddRequest.getClusterName());
clusterDO.setClusterType(clusterAddRequest.getClusterType());
clusterDO.setConnectKeyList(objectMapper.writeValueAsString(clusterAddRequest.getConnectKeyList()));
clusterDO.setUserName(clusterAddRequest.getUserName());
clusterDO.setPassword(clusterAddRequest.getPassword());
clusterDO.setNamespace(clusterAddRequest.getNamespace());
clusterDO.setClusterLevel(0);
clusterAccessService.insert(clusterDO);
}
}

View File

@ -14,15 +14,16 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.result.ClusterDeleteResult;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.pojo.request.ClusterDeleteRequest;
import com.alibaba.nacossync.pojo.result.ClusterDeleteResult;
import com.alibaba.nacossync.template.Processor;
import org.springframework.stereotype.Service;
/**
* @author NacosSync
@ -30,15 +31,26 @@ import com.alibaba.nacossync.template.Processor;
*/
@Service
public class ClusterDeleteProcessor implements Processor<ClusterDeleteRequest, ClusterDeleteResult> {
@Autowired
private ClusterAccessService clusterAccessService;
private final ClusterAccessService clusterAccessService;
private final TaskAccessService taskAccessService;
public ClusterDeleteProcessor(ClusterAccessService clusterAccessService, TaskAccessService taskAccessService) {
this.clusterAccessService = clusterAccessService;
this.taskAccessService = taskAccessService;
}
@Override
public void process(ClusterDeleteRequest clusterDeleteRequest,
ClusterDeleteResult clusterDeleteResult, Object... others) throws Exception {
public void process(ClusterDeleteRequest clusterDeleteRequest, ClusterDeleteResult clusterDeleteResult,
Object... others) throws Exception {
int count = taskAccessService.countByDestClusterIdOrSourceClusterId(clusterDeleteRequest.getClusterId(),
clusterDeleteRequest.getClusterId());
if (count > 0) {
throw new SkyWalkerException(String.format("集群下有%d个任务请先删除任务", count));
}
clusterAccessService.deleteByClusterId(clusterDeleteRequest.getClusterId());
}
}

View File

@ -14,44 +14,37 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.result.ClusterDetailQueryResult;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterDetailQueryRequest;
import com.alibaba.nacossync.pojo.result.ClusterDetailQueryResult;
import com.alibaba.nacossync.pojo.view.ClusterModel;
import com.alibaba.nacossync.template.Processor;
import org.springframework.stereotype.Service;
/**
* @author NacosSync
* @version $Id: ClusterDetailQueryProcessor.java, v 0.1 2018-09-30 PM2:39 NacosSync Exp $$
*/
@Service
public class ClusterDetailQueryProcessor
implements
Processor<ClusterDetailQueryRequest, ClusterDetailQueryResult> {
@Autowired
private ClusterAccessService clusterAccessService;
public class ClusterDetailQueryProcessor implements Processor<ClusterDetailQueryRequest, ClusterDetailQueryResult> {
private final ClusterAccessService clusterAccessService;
public ClusterDetailQueryProcessor(ClusterAccessService clusterAccessService) {
this.clusterAccessService = clusterAccessService;
}
@Override
public void process(ClusterDetailQueryRequest clusterDetailQueryRequest,
ClusterDetailQueryResult clusterDetailQueryResult, Object... others)
throws Exception {
ClusterDO clusterDO = clusterAccessService.findByClusterId(clusterDetailQueryRequest
.getClusterId());
ClusterModel clusterModel = new ClusterModel();
clusterModel.setClusterId(clusterDO.getClusterId());
clusterModel.setConnectKeyList(clusterDO.getConnectKeyList());
clusterModel.setClusterType(clusterDO.getClusterType());
clusterModel.setClusterName(clusterDO.getClusterName());
clusterDetailQueryResult.setClusterModel(clusterModel);
ClusterDetailQueryResult clusterDetailQueryResult, Object... others) throws Exception {
ClusterDO clusterDO = clusterAccessService.findByClusterId(clusterDetailQueryRequest.getClusterId());
clusterDetailQueryResult.setClusterModel(ClusterModel.from(clusterDO));
}
}

View File

@ -14,70 +14,60 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import java.util.ArrayList;
import java.util.List;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.QueryCondition;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterListQueryRequest;
import com.alibaba.nacossync.pojo.result.ClusterListQueryResult;
import com.alibaba.nacossync.pojo.view.ClusterModel;
import com.alibaba.nacossync.template.Processor;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.pojo.result.ClusterListQueryResult;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.request.ClusterListQueryRequest;
import com.alibaba.nacossync.pojo.view.ClusterModel;
import com.alibaba.nacossync.template.Processor;
import java.util.List;
/**
* @author NacosSync
* @version $Id: ClusterListQueryProcessor.java, v 0.1 2018-09-30 PM2:33 NacosSync Exp $$
*/
@Service
public class ClusterListQueryProcessor implements
Processor<ClusterListQueryRequest, ClusterListQueryResult> {
@Autowired
private ClusterAccessService clusterAccessService;
public class ClusterListQueryProcessor implements Processor<ClusterListQueryRequest, ClusterListQueryResult> {
private final ClusterAccessService clusterAccessService;
public ClusterListQueryProcessor(ClusterAccessService clusterAccessService) {
this.clusterAccessService = clusterAccessService;
}
@Override
public void process(ClusterListQueryRequest clusterListQueryRequest,
ClusterListQueryResult clusterListQueryResult, Object... others) {
public void process(ClusterListQueryRequest clusterListQueryRequest, ClusterListQueryResult clusterListQueryResult,
Object... others) {
Page<ClusterDO> clusterDOS;
if (StringUtils.isNotBlank(clusterListQueryRequest.getClusterName())) {
QueryCondition queryCondition = new QueryCondition();
queryCondition.setServiceName(clusterListQueryRequest.getClusterName());
clusterDOS = clusterAccessService.findPageCriteria(clusterListQueryRequest.getPageNum() - 1,
clusterListQueryRequest.getPageSize(), queryCondition);
} else {
clusterDOS = clusterAccessService.findPageNoCriteria(clusterListQueryRequest.getPageNum() - 1,
clusterListQueryRequest.getPageSize());
}
List<ClusterModel> clusterModels = new ArrayList<>();
clusterDOS.forEach(clusterDO -> {
ClusterModel clusterModel = new ClusterModel();
clusterModel.setClusterId(clusterDO.getClusterId());
clusterModel.setClusterName(clusterDO.getClusterName());
clusterModel.setClusterType(clusterDO.getClusterType());
clusterModel.setConnectKeyList(clusterDO.getConnectKeyList());
clusterModels.add(clusterModel);
});
List<ClusterModel> clusterModels = clusterDOS.stream().map(ClusterModel::from).toList();
clusterListQueryResult.setClusterModels(clusterModels);
clusterListQueryResult.setTotalPage(clusterDOS.getTotalPages());
clusterListQueryResult.setCurrentSize(clusterModels.size());
clusterListQueryResult.setTotalSize(clusterDOS.getTotalElements());
}
}

View File

@ -0,0 +1,250 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.naming.CommonParams;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.client.naming.NacosNamingService;
import com.alibaba.nacos.client.naming.net.NamingProxy;
import com.alibaba.nacos.client.naming.utils.UtilAndComs;
import com.alibaba.nacos.common.utils.HttpMethod;
import com.alibaba.nacos.common.utils.JacksonUtils;
import com.alibaba.nacos.common.utils.StringUtils;
import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.ClusterAccessService;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.extension.SyncManagerService;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.pojo.model.ClusterDO;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskAddAllRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.alibaba.nacossync.pojo.result.TaskAddResult;
import com.alibaba.nacossync.template.Processor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
import org.springframework.util.ReflectionUtils;
import javax.annotation.Nullable;
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.GROUP_NAME_PARAM;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.PAGE_NO;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.PAGE_SIZE;
import static com.alibaba.nacossync.constant.SkyWalkerConstants.SERVICE_NAME_PARAM;
/**
* @author NacosSync
* @version $Id: TaskAddAllProcessor.java, v 0.1 2022-03-23 PM11:40 NacosSync Exp $$
*/
@Slf4j
@Service
public class TaskAddAllProcessor implements Processor<TaskAddAllRequest, TaskAddResult> {
private static final String CONSUMER_PREFIX = "consumers:";
private final NacosServerHolder nacosServerHolder;
private final SyncManagerService syncManagerService;
private final TaskAccessService taskAccessService;
private final ClusterAccessService clusterAccessService;
public TaskAddAllProcessor(NacosServerHolder nacosServerHolder, SyncManagerService syncManagerService,
TaskAccessService taskAccessService, ClusterAccessService clusterAccessService) {
this.nacosServerHolder = nacosServerHolder;
this.syncManagerService = syncManagerService;
this.taskAccessService = taskAccessService;
this.clusterAccessService = clusterAccessService;
}
@Override
public void process(TaskAddAllRequest addAllRequest, TaskAddResult taskAddResult, Object... others)
throws Exception {
ClusterDO destCluster = clusterAccessService.findByClusterId(addAllRequest.getDestClusterId());
ClusterDO sourceCluster = clusterAccessService.findByClusterId(addAllRequest.getSourceClusterId());
if (Objects.isNull(destCluster) || Objects.isNull(sourceCluster)) {
throw new SkyWalkerException("Please check if the source or target cluster exists.");
}
if (Objects.isNull(syncManagerService.getSyncService(sourceCluster.getClusterId(), destCluster.getClusterId()))) {
throw new SkyWalkerException("current sync type not supported.");
}
// TODO 目前仅支持 Nacos 为源的同步类型待完善更多类型支持
final NamingService sourceNamingService = nacosServerHolder.get(sourceCluster.getClusterId());
if (sourceNamingService == null) {
throw new SkyWalkerException("only support sync type that the source of the Nacos.");
}
final EnhanceNamingService enhanceNamingService = new EnhanceNamingService(sourceNamingService);
final CatalogServiceResult catalogServiceResult = enhanceNamingService.catalogServices(null, null);
if (catalogServiceResult == null || catalogServiceResult.getCount() <= 0) {
throw new SkyWalkerException("sourceCluster data empty");
}
for (ServiceView serviceView : catalogServiceResult.getServiceList()) {
// exclude subscriber
if (addAllRequest.isExcludeConsumer() && serviceView.getName().startsWith(CONSUMER_PREFIX)) {
continue;
}
TaskAddRequest taskAddRequest = new TaskAddRequest();
taskAddRequest.setSourceClusterId(sourceCluster.getClusterId());
taskAddRequest.setDestClusterId(destCluster.getClusterId());
taskAddRequest.setServiceName(serviceView.getName());
taskAddRequest.setGroupName(serviceView.getGroupName());
this.dealTask(addAllRequest, taskAddRequest);
}
}
private void dealTask(TaskAddAllRequest addAllRequest, TaskAddRequest taskAddRequest) throws Exception {
String taskId = SkyWalkerUtil.generateTaskId(taskAddRequest);
TaskDO taskDO = taskAccessService.findByTaskId(taskId);
if (null == taskDO) {
taskDO = new TaskDO();
taskDO.setTaskId(taskId);
taskDO.setDestClusterId(addAllRequest.getDestClusterId());
taskDO.setSourceClusterId(addAllRequest.getSourceClusterId());
taskDO.setServiceName(taskAddRequest.getServiceName());
taskDO.setVersion(taskAddRequest.getVersion());
taskDO.setGroupName(taskAddRequest.getGroupName());
taskDO.setNameSpace(taskAddRequest.getNameSpace());
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());
taskDO.setWorkerIp(SkyWalkerUtil.getLocalIp());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
} else {
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
}
taskAccessService.addTask(taskDO);
}
static class EnhanceNamingService {
protected NamingService delegate;
protected NamingProxy serverProxy;
protected EnhanceNamingService(NamingService namingService) {
if (!(namingService instanceof NacosNamingService)) {
throw new IllegalArgumentException(
"namingService only support instance of com.alibaba.nacos.client.naming.NacosNamingService.");
}
this.delegate = namingService;
// serverProxy
final Field serverProxyField = ReflectionUtils.findField(NacosNamingService.class, "serverProxy");
assert serverProxyField != null;
ReflectionUtils.makeAccessible(serverProxyField);
this.serverProxy = (NamingProxy) ReflectionUtils.getField(serverProxyField, delegate);
}
public CatalogServiceResult catalogServices(@Nullable String serviceName, @Nullable String group)
throws NacosException {
int pageNo = 1; // start with 1
int pageSize = 100;
final CatalogServiceResult result = catalogServices(serviceName, group, pageNo, pageSize);
CatalogServiceResult tmpResult = result;
while (Objects.nonNull(tmpResult) && tmpResult.serviceList.size() >= pageSize) {
pageNo++;
tmpResult = catalogServices(serviceName, group, pageNo, pageSize);
if (tmpResult != null) {
result.serviceList.addAll(tmpResult.serviceList);
}
}
return result;
}
/**
* @see com.alibaba.nacos.client.naming.core.HostReactor#getServiceInfoDirectlyFromServer(String, String)
*/
public CatalogServiceResult catalogServices(@Nullable String serviceName, @Nullable String group, int pageNo,
int pageSize) throws NacosException {
// pageNo
// pageSize
// serviceNameParam
// groupNameParam
final Map<String, String> params = new HashMap<>(8);
params.put(CommonParams.NAMESPACE_ID, serverProxy.getNamespaceId());
params.put(SERVICE_NAME_PARAM, serviceName);
params.put(GROUP_NAME_PARAM, group);
params.put(PAGE_NO, String.valueOf(pageNo));
params.put(PAGE_SIZE, String.valueOf(pageSize));
final String result = this.serverProxy.reqApi(UtilAndComs.nacosUrlBase + "/catalog/services", params,
HttpMethod.GET);
if (StringUtils.isNotEmpty(result)) {
return JacksonUtils.toObj(result, CatalogServiceResult.class);
}
return null;
}
}
/**
* Copy from Nacos Server.
*/
@Data
static class ServiceView {
private String name;
private String groupName;
private int clusterCount;
private int ipCount;
private int healthyInstanceCount;
private String triggerFlag;
}
@Data
static class CatalogServiceResult {
/**
* countnot equal serviceList.size .
*/
private int count;
private List<ServiceView> serviceList;
}
}

View File

@ -89,7 +89,7 @@ public class TaskAddProcessor implements Processor<TaskAddRequest, TaskAddResult
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());
taskDO.setWorkerIp(SkyWalkerUtil.getLocalIp());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
} else {
taskDO.setTaskStatus(TaskStatusEnum.SYNC.getCode());

View File

@ -16,19 +16,12 @@
*/
package com.alibaba.nacossync.template.processor;
import java.util.ArrayList;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskDeleteInBatchRequest;
import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.template.Processor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
@ -40,20 +33,15 @@ import lombok.extern.slf4j.Slf4j;
@Service
public class TaskDeleteInBatchProcessor implements Processor<TaskDeleteInBatchRequest, BaseResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskDeleteInBatchProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskDeleteInBatchRequest taskBatchDeleteRequest, BaseResult baseResult,
Object... others) {
//
// String[] taskIds= taskBatchDeleteRequest.getTaskIds();
// List<TaskDO> taskDOs = new ArrayList<TaskDO>();
// for (String taskId : taskIds) {
// TaskDO taskDO = new TaskDO();
// taskDO.setTaskId(taskId);
// taskDOs.add(taskDO);
// }
taskAccessService.deleteTaskInBatch(taskBatchDeleteRequest.getTaskIds());
}
}

View File

@ -14,9 +14,12 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.constant.SkyWalkerConstants;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.event.DeleteAllSubTaskEvent;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskDeleteRequest;
@ -24,7 +27,6 @@ import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.template.Processor;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
@ -34,18 +36,28 @@ import org.springframework.stereotype.Service;
@Slf4j
@Service
public class TaskDeleteProcessor implements Processor<TaskDeleteRequest, BaseResult> {
@Autowired
private TaskAccessService taskAccessService;
@Autowired
private EventBus eventBus;
private final TaskAccessService taskAccessService;
private final EventBus eventBus;
public TaskDeleteProcessor(TaskAccessService taskAccessService, EventBus eventBus) {
this.taskAccessService = taskAccessService;
this.eventBus = eventBus;
}
@Override
public void process(TaskDeleteRequest taskDeleteRequest, BaseResult baseResult,
Object... others) {
public void process(TaskDeleteRequest taskDeleteRequest, BaseResult baseResult, Object... others) {
TaskDO taskDO = taskAccessService.findByTaskId(taskDeleteRequest.getTaskId());
eventBus.post(new DeleteTaskEvent(taskDO));
log.info("删除同步任务数据之前,发出一个同步事件:" + taskDO);
// delete all sub task when ServiceName is all
if (SkyWalkerConstants.NACOS_ALL_SERVICE_NAME.equalsIgnoreCase(taskDO.getServiceName())) {
eventBus.post(new DeleteAllSubTaskEvent(taskDO));
} else {
eventBus.post(new DeleteTaskEvent(taskDO));
}
log.info("删除同步任务数据之前,发出一个同步事件:{}", taskDO);
taskAccessService.deleteTaskById(taskDeleteRequest.getTaskId());
}
}

View File

@ -14,17 +14,17 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.pojo.result.TaskDetailQueryResult;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskDetailQueryRequest;
import com.alibaba.nacossync.pojo.result.TaskDetailQueryResult;
import com.alibaba.nacossync.pojo.view.TaskModel;
import com.alibaba.nacossync.template.Processor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
/**
@ -34,29 +34,23 @@ import org.springframework.stereotype.Service;
@Slf4j
@Service
public class TaskDetailProcessor implements Processor<TaskDetailQueryRequest, TaskDetailQueryResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskDetailProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskDetailQueryRequest taskDetailQueryRequest, TaskDetailQueryResult taskDetailQueryResult, Object... others)
throws Exception {
public void process(TaskDetailQueryRequest taskDetailQueryRequest, TaskDetailQueryResult taskDetailQueryResult,
Object... others) throws Exception {
TaskDO taskDO = taskAccessService.findByTaskId(taskDetailQueryRequest.getTaskId());
if (null == taskDO) {
throw new SkyWalkerException("taskDo is null,taskId :" + taskDetailQueryRequest.getTaskId());
}
TaskModel taskModel = new TaskModel();
taskModel.setDestClusterId(taskDO.getDestClusterId());
taskModel.setGroupName(taskDO.getGroupName());
taskModel.setServiceName(taskDO.getServiceName());
taskModel.setSourceClusterId(taskDO.getSourceClusterId());
taskModel.setTaskStatus(taskDO.getTaskStatus());
taskModel.setTaskId(taskDO.getTaskId());
taskDetailQueryResult.setTaskModel(taskModel);
taskDetailQueryResult.setTaskModel(TaskModel.from(taskDO));
}
}

View File

@ -14,25 +14,23 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import java.util.ArrayList;
import java.util.List;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.QueryCondition;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskListQueryRequest;
import com.alibaba.nacossync.pojo.result.TaskListQueryResult;
import com.alibaba.nacossync.pojo.view.TaskModel;
import com.alibaba.nacossync.template.Processor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.result.TaskListQueryResult;
import com.alibaba.nacossync.pojo.request.TaskListQueryRequest;
import com.alibaba.nacossync.pojo.view.TaskModel;
import com.alibaba.nacossync.template.Processor;
import java.util.List;
/**
* @author NacosSync
@ -41,42 +39,34 @@ import com.alibaba.nacossync.template.Processor;
@Service
@Slf4j
public class TaskListQueryProcessor implements Processor<TaskListQueryRequest, TaskListQueryResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskListQueryProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskListQueryRequest taskListQueryRequest,
TaskListQueryResult taskListQueryResult, Object... others) {
public void process(TaskListQueryRequest taskListQueryRequest, TaskListQueryResult taskListQueryResult,
Object... others) {
Page<TaskDO> taskDOPage;
if (StringUtils.isNotBlank(taskListQueryRequest.getServiceName())) {
QueryCondition queryCondition = new QueryCondition();
queryCondition.setServiceName(taskListQueryRequest.getServiceName());
taskDOPage = taskAccessService.findPageCriteria(taskListQueryRequest.getPageNum() - 1,
taskListQueryRequest.getPageSize(), queryCondition);
} else {
taskDOPage = taskAccessService.findPageNoCriteria(taskListQueryRequest.getPageNum() - 1,
taskListQueryRequest.getPageSize());
}
List<TaskModel> taskList = new ArrayList<>();
taskDOPage.forEach(taskDO -> {
TaskModel taskModel = new TaskModel();
taskModel.setTaskId(taskDO.getTaskId());
taskModel.setDestClusterId(taskDO.getDestClusterId());
taskModel.setSourceClusterId(taskDO.getSourceClusterId());
taskModel.setServiceName(taskDO.getServiceName());
taskModel.setGroupName(taskDO.getGroupName());
taskModel.setTaskStatus(taskDO.getTaskStatus());
taskList.add(taskModel);
});
List<TaskModel> taskList = taskDOPage.stream().map(TaskModel::from).toList();
taskListQueryResult.setTaskModels(taskList);
taskListQueryResult.setTotalPage(taskDOPage.getTotalPages());
taskListQueryResult.setTotalSize(taskDOPage.getTotalElements());

View File

@ -14,21 +14,19 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.template.processor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.exception.SkyWalkerException;
import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.TaskUpdateRequest;
import com.alibaba.nacossync.pojo.result.BaseResult;
import com.alibaba.nacossync.template.Processor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;
/**
* @author NacosSync
@ -37,29 +35,33 @@ import com.alibaba.nacossync.template.Processor;
@Slf4j
@Service
public class TaskUpdateProcessor implements Processor<TaskUpdateRequest, BaseResult> {
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
public TaskUpdateProcessor(TaskAccessService taskAccessService) {
this.taskAccessService = taskAccessService;
}
@Override
public void process(TaskUpdateRequest taskUpdateRequest, BaseResult baseResult,
Object... others) throws Exception {
public void process(TaskUpdateRequest taskUpdateRequest, BaseResult baseResult, Object... others) throws Exception {
TaskDO taskDO = taskAccessService.findByTaskId(taskUpdateRequest.getTaskId());
if (!TaskStatusEnum.contains(taskUpdateRequest.getTaskStatus())) {
throw new SkyWalkerException(
"taskUpdateRequest.getTaskStatus() is not exist , value is :"
+ taskUpdateRequest.getTaskStatus());
"taskUpdateRequest.getTaskStatus() is not exist , value is :" + taskUpdateRequest.getTaskStatus());
}
if (null == taskDO) {
throw new SkyWalkerException("taskDo is null ,taskId is :"
+ taskUpdateRequest.getTaskId());
throw new SkyWalkerException("taskDo is null ,taskId is :" + taskUpdateRequest.getTaskId());
}
taskDO.setTaskStatus(taskUpdateRequest.getTaskStatus());
taskDO.setOperationId(SkyWalkerUtil.generateOperationId());
taskAccessService.addTask(taskDO);
}
}

View File

@ -0,0 +1,252 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.timer;
import com.alibaba.nacos.api.naming.NamingService;
import com.alibaba.nacos.api.naming.pojo.ListView;
import com.alibaba.nacos.client.naming.utils.CollectionUtils;
import com.alibaba.nacossync.constant.MetricsStatisticsType;
import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.event.DeleteAllSubTaskEvent;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.event.SyncTaskEvent;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.util.BatchTaskExecutor;
import com.alibaba.nacossync.util.SkyWalkerUtil;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.BeanUtils;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
/**
* when the database task service name is empty, check all the services in the cluster and create a synchronization
* task.
*/
@Slf4j
public class CheckRunningStatusAllNacosThread implements Runnable {
private final Map<String, Set<String>> subTaskService = new ConcurrentHashMap<>();
private final MetricsManager metricsManager;
private final TaskAccessService taskAccessService;
private final NacosServerHolder nacosServerHolder;
private final EventBus eventBus;
public CheckRunningStatusAllNacosThread(MetricsManager metricsManager, TaskAccessService taskAccessService,
NacosServerHolder nacosServerHolder, EventBus eventBus) {
this.metricsManager = metricsManager;
this.taskAccessService = taskAccessService;
this.nacosServerHolder = nacosServerHolder;
this.eventBus = eventBus;
}
/**
* Synchronize data based on the ns level.
*/
@Override
public void run() {
try {
List<TaskDO> tasks = taskAccessService.findAllByServiceNameEqualAll();
if (CollectionUtils.isEmpty(tasks)) {
return;
}
// Get the set of all task IDs
Set<String> taskIdSet = tasks.stream().map(TaskDO::getTaskId).collect(Collectors.toSet());
// Filter and handle sub-tasks that need to be deleted. This handles the case where tasks have been deleted
// but sub-tasks still exist.
subTaskService.entrySet().stream()
.filter(entry -> shouldDeleteSubTasks(entry, taskIdSet))
.forEach(entry -> postDeleteAllSubTaskEvent(entry.getKey()));
// Handle regular tasks
tasks.forEach(this::processTask);
} catch (Exception e) {
log.warn("CheckRunningStatusThread Exception", e);
metricsManager.recordError(MetricsStatisticsType.DISPATCHER_TASK);
}
}
/**
* Listens for the event of deleting all sub-tasks and handles the delete operation.
*
* @param deleteAllSubTaskEvent The event object containing the task information to be deleted.
*/
public void listenerDeleteAllTaskEvent(DeleteAllSubTaskEvent deleteAllSubTaskEvent) {
// Retrieve the task object
TaskDO task = deleteAllSubTaskEvent.getTaskDO();
// Retrieve the task ID
String taskId = task.getTaskId();
// Remove the set of service names corresponding to the task ID from subTaskService
Set<String> serviceNameSet = subTaskService.remove(taskId);
// If the set of service names is empty, return immediately
if (CollectionUtils.isEmpty(serviceNameSet)) {
return;
}
// Build the list of sub-tasks pending removal
List<TaskDO> servicesPendingRemoval = serviceNameSet.stream()
.map(serviceName -> buildSubTaskDO(task, serviceName))
.toList();
// Handle the removal of the pending sub-tasks
handleRemoval(servicesPendingRemoval, serviceNameSet);
}
private boolean shouldDeleteSubTasks(Map.Entry<String, Set<String>> entry, Set<String> taskIdSet) {
return !taskIdSet.contains(entry.getKey()) && !entry.getValue().isEmpty();
}
private void postDeleteAllSubTaskEvent(String taskId) {
TaskDO taskDO = new TaskDO();
taskDO.setTaskId(taskId);
eventBus.post(new DeleteAllSubTaskEvent(taskDO));
}
/**
* Processes the given task by determining the services that need to be inserted and removed,
* and performs the corresponding operations.
*
* @param task The task object to be processed.
*/
private void processTask(TaskDO task) {
// Retrieve the set of services for the task, creating a new set if it does not exist
Set<String> serviceSet = subTaskService.computeIfAbsent(task.getTaskId(), k -> ConcurrentHashMap.newKeySet());
// Get the list of all service names associated with the task
List<String> serviceNameList = getAllServiceName(task);
// Determine the services that need to be inserted (those not in the current service set)
List<TaskDO> servicesPendingInsertion = serviceNameList.stream()
.filter(serviceName -> !serviceSet.contains(serviceName))
.map(serviceName -> buildSubTaskDO(task, serviceName))
.toList();
// Determine the services that need to be removed (those in the current service set but not in the service name list)
List<TaskDO> servicesPendingRemoval = serviceSet.stream()
.filter(serviceName -> !serviceNameList.contains(serviceName))
.map(serviceName -> buildSubTaskDO(task, serviceName))
.toList();
// If all lists are empty, there is nothing to process
if (CollectionUtils.isEmpty(serviceNameList) && CollectionUtils.isEmpty(servicesPendingInsertion)
&& CollectionUtils.isEmpty(servicesPendingRemoval)) {
log.debug("No service found for task: {}", task.getTaskId());
return;
}
// If the task status is SYNC, handle the insertion of services
if (TaskStatusEnum.SYNC.getCode().equals(task.getTaskStatus())) {
handleInsertion(servicesPendingInsertion, serviceSet);
}
// Handle the removal of services
handleRemoval(servicesPendingRemoval, serviceSet);
if (TaskStatusEnum.DELETE.getCode().equals(task.getTaskStatus())) {
List<TaskDO> allSubTasks = serviceNameList.stream().map(serviceName -> buildSubTaskDO(task, serviceName))
.collect(Collectors.toList());
handleRemoval(allSubTasks, serviceSet);
}
}
/**
* Handles the insertion of services.
*
* @param servicesPendingInsertion The list of services to be inserted.
* @param serviceSet The set of services.
*/
private void handleInsertion(List<TaskDO> servicesPendingInsertion, Set<String> serviceSet) {
BatchTaskExecutor.batchOperation(servicesPendingInsertion, t -> {
eventBus.post(new SyncTaskEvent(t));
serviceSet.add(t.getServiceName());
});
}
/**
* Handles the removal of services.
*
* @param servicesPendingRemoval The list of services to be removed.
* @param serviceSet The set of services.
*/
private void handleRemoval(List<TaskDO> servicesPendingRemoval, Set<String> serviceSet) {
BatchTaskExecutor.batchOperation(servicesPendingRemoval, t -> {
eventBus.post(new DeleteTaskEvent(t));
serviceSet.remove(t.getServiceName());
});
}
/**
* Builds a sub-task object for the given task and service name.
*
* @param serviceName The service name.
* @return The constructed sub-task object.
*/
private static TaskDO buildSubTaskDO(TaskDO taskDO, String serviceName) {
TaskDO task = new TaskDO();
BeanUtils.copyProperties(taskDO, task);
task.setTaskId(SkyWalkerUtil.generateTaskId(serviceName, taskDO.getGroupName(), taskDO.getSourceClusterId(),
taskDO.getDestClusterId()));
task.setServiceName(serviceName);
return task;
}
/**
* Retrieves all service names associated with the given task.
*
* @return A list of service names.
*/
private List<String> getAllServiceName(TaskDO taskDO) {
NamingService namingService = nacosServerHolder.get(taskDO.getSourceClusterId());
if (namingService == null) {
log.warn("naming service is null or not found, clusterId:{}", taskDO.getSourceClusterId());
return Collections.emptyList();
}
try {
ListView<String> servicesOfServer = namingService.getServicesOfServer(0, Integer.MAX_VALUE,
taskDO.getGroupName());
return servicesOfServer.getData();
} catch (Exception e) {
log.error("query service list failure", e);
}
return Collections.emptyList();
}
}

View File

@ -14,24 +14,23 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.timer;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
import com.alibaba.nacossync.cache.SkyWalkerCacheServices;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.pojo.FinishedTask;
import com.alibaba.nacossync.pojo.model.TaskDO;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;
/**
* @author NacosSync
@ -40,57 +39,53 @@ import com.alibaba.nacossync.pojo.model.TaskDO;
@Slf4j
@Service
public class CleanExceedOperationIdTimer implements CommandLineRunner {
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private TaskAccessService taskAccessService;
@Autowired
private ScheduledExecutorService scheduledExecutorService;
private static final long INITIAL_DELAY = 0;
private static final long PERIOD = 12;
private final SkyWalkerCacheServices skyWalkerCacheServices;
private final TaskAccessService taskAccessService;
private final ScheduledExecutorService scheduledExecutorService;
public CleanExceedOperationIdTimer(SkyWalkerCacheServices skyWalkerCacheServices,
TaskAccessService taskAccessService, ScheduledExecutorService scheduledExecutorService) {
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.taskAccessService = taskAccessService;
this.scheduledExecutorService = scheduledExecutorService;
}
@Override
public void run(String... args) {
/** Clean up the OperationId cache once every 12 hours */
scheduledExecutorService.scheduleWithFixedDelay(new CleanExceedOperationIdThread(), 0, 12,
TimeUnit.HOURS);
scheduledExecutorService.scheduleWithFixedDelay(new CleanExceedOperationIdThread(), INITIAL_DELAY, PERIOD,
TimeUnit.HOURS);
log.info("CleanExceedOperationIdTimer has started successfully");
}
private class CleanExceedOperationIdThread implements Runnable {
@Override
public void run() {
try {
Map<String, FinishedTask> finishedTaskMap = skyWalkerCacheServices
.getFinishedTaskMap();
Iterable<TaskDO> taskDOS = taskAccessService.findAll();
Set<String> operationIds = getDbOperations(taskDOS);
for (String operationId : finishedTaskMap.keySet()) {
if (!operationIds.contains(operationId)) {
finishedTaskMap.remove(operationId);
}
}
Map<String, FinishedTask> finishedTaskMap = skyWalkerCacheServices.getFinishedTaskMap();
Set<String> operationIds = getDbOperations(taskAccessService.findAll());
finishedTaskMap.keySet().removeIf(operationId -> !operationIds.contains(operationId));
} catch (Exception e) {
log.warn("CleanExceedOperationIdThread Exception", e);
}
}
private Set<String> getDbOperations(Iterable<TaskDO> taskDOS) {
Set<String> operationIds = new HashSet<>();
taskDOS.forEach(taskDO -> operationIds.add(taskDO.getOperationId()));
return operationIds;
return StreamSupport.stream(taskDOS.spliterator(), false).map(TaskDO::getOperationId)
.collect(Collectors.toSet());
}
}
}

View File

@ -22,14 +22,15 @@ import com.alibaba.nacossync.constant.TaskStatusEnum;
import com.alibaba.nacossync.dao.TaskAccessService;
import com.alibaba.nacossync.event.DeleteTaskEvent;
import com.alibaba.nacossync.event.SyncTaskEvent;
import com.alibaba.nacossync.extension.holder.NacosServerHolder;
import com.alibaba.nacossync.monitor.MetricsManager;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
import java.util.List;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
@ -40,27 +41,44 @@ import java.util.concurrent.TimeUnit;
@Slf4j
@Service
public class QuerySyncTaskTimer implements CommandLineRunner {
@Autowired
private MetricsManager metricsManager;
private static final int INITIAL_DELAY = 0;
private static final int DELAY = 3000;
private final MetricsManager metricsManager;
@Autowired
private SkyWalkerCacheServices skyWalkerCacheServices;
private final SkyWalkerCacheServices skyWalkerCacheServices;
@Autowired
private TaskAccessService taskAccessService;
private final TaskAccessService taskAccessService;
@Autowired
private EventBus eventBus;
@Autowired
private ScheduledExecutorService scheduledExecutorService;
private final EventBus eventBus;
private final ScheduledExecutorService scheduledExecutorService;
private final NacosServerHolder nacosServerHolder;
public QuerySyncTaskTimer(MetricsManager metricsManager, SkyWalkerCacheServices skyWalkerCacheServices,
TaskAccessService taskAccessService, EventBus eventBus, ScheduledExecutorService scheduledExecutorService,
NacosServerHolder nacosServerHolder) {
this.metricsManager = metricsManager;
this.skyWalkerCacheServices = skyWalkerCacheServices;
this.taskAccessService = taskAccessService;
this.eventBus = eventBus;
this.scheduledExecutorService = scheduledExecutorService;
this.nacosServerHolder = nacosServerHolder;
}
@Override
public void run(String... args) {
/** Fetch the task list from the database every 3 seconds */
scheduledExecutorService.scheduleWithFixedDelay(new CheckRunningStatusThread(), 0, 3000,
scheduledExecutorService.scheduleWithFixedDelay(new CheckRunningStatusThread(), INITIAL_DELAY, DELAY,
TimeUnit.MILLISECONDS);
scheduledExecutorService.scheduleWithFixedDelay(new CheckRunningStatusAllNacosThread(metricsManager,
taskAccessService, nacosServerHolder, eventBus), INITIAL_DELAY, DELAY,
TimeUnit.MILLISECONDS);
log.info("QuerySyncTaskTimer has started successfully");
}
private class CheckRunningStatusThread implements Runnable {
@ -68,10 +86,10 @@ public class QuerySyncTaskTimer implements CommandLineRunner {
@Override
public void run() {
Long start = System.currentTimeMillis();
long start = System.currentTimeMillis();
try {
Iterable<TaskDO> taskDOS = taskAccessService.findAll();
List<TaskDO> taskDOS = taskAccessService.findAllByServiceNameNotEqualAll();
taskDOS.forEach(taskDO -> {
@ -83,13 +101,13 @@ public class QuerySyncTaskTimer implements CommandLineRunner {
if (TaskStatusEnum.SYNC.getCode().equals(taskDO.getTaskStatus())) {
eventBus.post(new SyncTaskEvent(taskDO));
log.info("从数据库中查询到一个同步任务,发出一个同步事件:" + taskDO);
log.info("从数据库中查询到一个同步任务,发出一个同步事件:{}", taskDO);
}
if (TaskStatusEnum.DELETE.getCode().equals(taskDO.getTaskStatus())) {
eventBus.post(new DeleteTaskEvent(taskDO));
log.info("从数据库中查询到一个删除任务,发出一个同步事件:" + taskDO);
log.info("从数据库中查询到一个删除任务,发出一个同步事件:{}", taskDO);
}
});

View File

@ -17,7 +17,6 @@ import com.alibaba.nacossync.extension.event.SpecialSyncEvent;
import com.alibaba.nacossync.extension.event.SpecialSyncEventBus;
import com.google.common.eventbus.EventBus;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Service;
@ -33,20 +32,25 @@ import java.util.concurrent.TimeUnit;
@Service
public class SpecialSyncEventTimer implements CommandLineRunner {
@Autowired
private SpecialSyncEventBus specialSyncEventBus;
private final SpecialSyncEventBus specialSyncEventBus;
@Autowired
private EventBus eventBus;
@Autowired
private ScheduledExecutorService scheduledExecutorService;
private final EventBus eventBus;
private final ScheduledExecutorService scheduledExecutorService;
public SpecialSyncEventTimer(SpecialSyncEventBus specialSyncEventBus, EventBus eventBus,
ScheduledExecutorService scheduledExecutorService) {
this.specialSyncEventBus = specialSyncEventBus;
this.eventBus = eventBus;
this.scheduledExecutorService = scheduledExecutorService;
}
@Override
public void run(String... args) throws Exception {
scheduledExecutorService.scheduleWithFixedDelay(new SpecialSyncEventTimer.SpecialSyncEventThread(), 0, 3000,
TimeUnit.MILLISECONDS);
log.info("SpecialSyncEventTimer has started successfully");
}
private class SpecialSyncEventThread implements Runnable {
@ -58,9 +62,9 @@ public class SpecialSyncEventTimer implements CommandLineRunner {
allSpecialSyncEvent.stream()
.filter(specialSyncEvent -> TaskStatusEnum.SYNC.getCode()
.equals(specialSyncEvent.getTaskDO().getTaskStatus()))
.forEach(specialSyncEvent -> eventBus.post(specialSyncEvent));
.forEach(eventBus::post);
} catch (Exception e) {
log.warn("SpecialSyncEventThread Exception", e);
log.error("Exception occurred while processing special sync events", e);
}
}

View File

@ -0,0 +1,104 @@
package com.alibaba.nacossync.util;
import com.alibaba.nacos.client.naming.utils.CollectionUtils;
import com.alibaba.nacossync.pojo.model.TaskDO;
import com.google.common.base.Stopwatch;
import lombok.extern.slf4j.Slf4j;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.concurrent.*;
import java.util.function.Consumer;
@Slf4j
public class BatchTaskExecutor {
private static final int MAX_THREAD_NUM = 200;
private static final ExecutorService executorService = Executors.newFixedThreadPool(MAX_THREAD_NUM);
/**
* Batch operation method
*
* @param items Task list
* @param operation Operation to be executed
*/
public static void batchOperation(List<TaskDO> items, Consumer<TaskDO> operation) {
Stopwatch stopwatch = Stopwatch.createStarted();
List<Tuple<Integer, List<TaskDO>>> taskGroupList = averageAssign(items, MAX_THREAD_NUM);
// Create a CompletableFuture for each task group
CompletableFuture<?>[] futures = taskGroupList.stream().map(tuple -> CompletableFuture.runAsync(() -> {
for (TaskDO taskDO : tuple.getT2()) {
try {
// Add timeout control for each task to avoid long-running tasks
CompletableFuture.runAsync(() -> operation.accept(taskDO), executorService)
.orTimeout(5, TimeUnit.SECONDS) // Task timeout set to 5 seconds
.exceptionally(ex -> {
log.error("Task execution timed out: {}", taskDO.getServiceName(), ex);
return null;
}).join();
} catch (Exception e) {
log.error("Error occurred during task execution: {}", taskDO.getServiceName(), e);
}
}
}, executorService)).toArray(CompletableFuture[]::new);
try {
// Wait for all tasks to complete
CompletableFuture.allOf(futures).join();
} catch (Exception e) {
log.error("Error occurred during sync operation", e);
} finally {
log.info("Total sync tasks: {}, Execution time: {} ms", items.size(), stopwatch.elapsed(TimeUnit.MILLISECONDS));
}
}
/**
* Divide a list into n sublists, mainly implemented by offset
* @param source collection to be divided
* @param limit maximum value
* @return list after division
* @param <T> object type
*/
private static <T> List<Tuple<Integer, List<T>>> averageAssign(List<T> source, int limit) {
if (CollectionUtils.isEmpty(source)) {
return Collections.emptyList();
}
int size = source.size();
int listCount = (int) Math.ceil((double) size / limit); // Calculate the number of sublists
int remainder = size % listCount; // Calculate the number of remaining elements after even distribution
List<Tuple<Integer, List<T>>> result = new ArrayList<>(listCount); // Initialize the result list with the expected size
for (int i = 0, assigned = 0; i < listCount; i++) {
int sublistSize = size / listCount + (remainder-- > 0 ? 1 : 0); // Determine the size of each sublist, distribute remaining elements
List<T> sublist = new ArrayList<>(source.subList(assigned, assigned + sublistSize)); // Create the sublist
result.add(Tuple.of(i, sublist)); // Add the sublist to the result
assigned += sublistSize; // Update the assigned index
}
return result;
}
/**
* Shutdown the executor service to avoid resource leakage
*/
public static void shutdown() {
executorService.shutdown();
try {
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
executorService.shutdownNow();
if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) {
log.error("Executor service did not terminate");
}
}
} catch (InterruptedException ie) {
executorService.shutdownNow();
Thread.currentThread().interrupt();
}
}
}

View File

@ -1,19 +0,0 @@
package com.alibaba.nacossync.util;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
public abstract class Collections {
public static Collection subtract(final Collection a, final Collection b) {
ArrayList list = new ArrayList( a );
for (Iterator it = b.iterator(); it.hasNext();) {
list.remove(it.next());
}
return list;
}
}

View File

@ -1,15 +1,3 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE
* file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file
* to You under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the
* License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
* an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
* specific language governing permissions and limitations under the License.
*/
package com.alibaba.nacossync.util;
import com.alibaba.nacos.client.naming.utils.CollectionUtils;
@ -20,16 +8,21 @@ import java.util.Map;
import java.util.stream.Collectors;
/**
* @author paderlol
* @date: 2019-04-25 00:01
* Utility class for handling Consul metadata.
*/
public class ConsulUtils {
public static Map<String, String> transferMetadata(List<String> tags) {
Map<String, String> metadata = new HashMap<>();
if (!CollectionUtils.isEmpty(tags)) {
return tags.stream().filter(tag -> tag.split("=", -1).length == 2).map(tag -> tag.split("=", -1))
.collect(Collectors.toMap(tagSplitArray -> tagSplitArray[0], tagSplitArray -> tagSplitArray[1]));
if (CollectionUtils.isEmpty(tags)) {
return new HashMap<>();
}
return metadata;
return tags.stream()
.map(tag -> tag.split("=", -1))
.filter(tagArray -> tagArray.length == 2)
.collect(Collectors.toMap(
tagArray -> tagArray[0],
tagArray -> tagArray[1],
(existing, replacement) -> existing // In case of duplicate keys, keep the existing value
));
}
}

View File

@ -20,7 +20,6 @@ import org.apache.commons.lang3.StringUtils;
import java.math.BigDecimal;
import java.util.List;
import java.util.Map;
import java.util.function.Predicate;
/**
* @author paderlol
@ -49,45 +48,51 @@ public final class DubboConstants {
public static final String DUBBO_ROOT_PATH = "/dubbo";
public static final String ALL_SERVICE_NAME_PATTERN = "*";
/**
* if Dubbo version greater than 2.7.2, service name is providers:interface:version:
* if Dubbo version less than 2.7.2, service name is providers:interface:version
* @param queryParam
* @return
* Creates a service name based on Dubbo version compatibility.
* if Dubbo version greater than 2.7.2, service name is providers:interface:version:
* if Dubbo version less than 2.7.2, service name is providers:interface:version
*
* @param queryParam the query parameters that include keys such as interface, version, group, etc.
* @return the constructed service name string
*/
public static String createServiceName(Map<String, String> queryParam) {
String group = queryParam.get(GROUP_KEY);
String release = queryParam.get(RELEASE_KEY);
Predicate<String> isBlankGroup = StringUtils::isBlank;
Predicate<String> isNotBlankRelease = StringUtils::isNotBlank;
String serviceName = Joiner.on(SEPARATOR_KEY).skipNulls().join(CATALOG_KEY, queryParam.get(INTERFACE_KEY),
queryParam.get(VERSION_KEY), group);
//TODO The code here is to deal with service metadata format problems caused by dubbo version incompatibility
if (isBlankGroup.test(group) && isNotBlankRelease.test(release)) {
String baseServiceName = Joiner.on(SEPARATOR_KEY).skipNulls().join(CATALOG_KEY, queryParam.get(INTERFACE_KEY),
queryParam.get(VERSION_KEY), group);
if (StringUtils.isBlank(group) && StringUtils.isNotBlank(release)) {
List<String> versions = Splitter.on(RELEASE_SEPARATOR_KEY).splitToList(release);
if (!CollectionUtils.isEmpty(versions) && versions.size() >= DUBBO_VERSION_INDEX) {
String firstVersion = versions.get(0);
String secondVersion = versions.get(1);
if (DUBBO_VERSION_INDEX == Integer.parseInt(firstVersion)) {
if (MIDDLE_DUBBO_VERSION_INDEX <= versions.size()) {
String thirdVersion = versions.get(2);
BigDecimal bigDecimal =
new BigDecimal(Joiner.on(RELEASE_SEPARATOR_KEY).join(secondVersion, thirdVersion));
if (bigDecimal.compareTo(COMPARE_NUMBER) > 0) {
serviceName = serviceName.concat(SEPARATOR_KEY);
}
} else if (versions.size() == DUBBO_VERSION_INDEX && Integer.parseInt(secondVersion) > 7) {
serviceName = serviceName.concat(SEPARATOR_KEY);
}
} else if (MIN_DUBBO_VERSION < Integer.parseInt(firstVersion)) {
serviceName = serviceName.concat(SEPARATOR_KEY);
BigDecimal bigDecimal = new BigDecimal(Joiner.on(RELEASE_SEPARATOR_KEY).join(secondVersion,
versions.size() > 2 ? versions.get(2) : "0"));
if (isVersionRequiresSeparator(firstVersion, secondVersion, bigDecimal)) {
baseServiceName = baseServiceName.concat(SEPARATOR_KEY);
}
}
}
return serviceName;
return baseServiceName;
}
/**
* Checks if the version requires a separator to be appended to the service name.
*
* @param firstVersion the major version
* @param secondVersion the minor version
* @param bigDecimal the version number as BigDecimal
* @return true if separator should be added, otherwise false
*/
private static boolean isVersionRequiresSeparator(String firstVersion, String secondVersion, BigDecimal bigDecimal) {
int majorVersion = Integer.parseInt(firstVersion);
int minorVersion = Integer.parseInt(secondVersion);
return (DUBBO_VERSION_INDEX == majorVersion && (MIDDLE_DUBBO_VERSION_INDEX <= minorVersion ||
bigDecimal.compareTo(COMPARE_NUMBER) > 0)) || (MIN_DUBBO_VERSION < majorVersion);
}
}

View File

@ -0,0 +1,11 @@
package com.alibaba.nacossync.util;
import com.alibaba.nacos.api.common.Constants;
import org.apache.commons.lang3.StringUtils;
public class NacosUtils {
public static String getGroupNameOrDefault(String groupName) {
return StringUtils.defaultIfBlank(groupName, Constants.DEFAULT_GROUP);
}
}

View File

@ -1,19 +1,3 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.util;
import com.alibaba.nacossync.constant.ClusterTypeEnum;
@ -22,135 +6,139 @@ import com.alibaba.nacossync.pojo.model.TaskDO;
import com.alibaba.nacossync.pojo.request.ClusterAddRequest;
import com.alibaba.nacossync.pojo.request.TaskAddRequest;
import com.google.common.base.Joiner;
import org.apache.commons.lang3.StringUtils;
import java.io.UnsupportedEncodingException;
import java.net.Inet4Address;
import java.net.InetAddress;
import java.net.NetworkInterface;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Enumeration;
import java.util.UUID;
/**
* @author NacosSync
* @version $Id: SkyWalkerUtil.java, v 0.1 2018-09-26 AM12:10 NacosSync Exp $$
*/
* Utility class for various operations in SkyWalker.
*/
public class SkyWalkerUtil {
private static final String SEPARATOR = ":";
private static final String MD5_ALGORITHM = "MD5";
/**
* Generates an MD5 hash for the given string.
*
* Gets the string md5
* @param value
* @return
* @param value The string to be encrypted.
* @return The encrypted string, or an empty string if encryption fails.
*/
public static String StringToMd5(String value) {
{
try {
MessageDigest md5 = MessageDigest.getInstance("MD5");
md5.update(value.getBytes("UTF-8"));
byte[] encryption = md5.digest();
StringBuffer strBuf = new StringBuffer();
for (int i = 0; i < encryption.length; i++) {
if (Integer.toHexString(0xff & encryption[i]).length() == 1) {
strBuf.append("0").append(Integer.toHexString(0xff & encryption[i]));
} else {
strBuf.append(Integer.toHexString(0xff & encryption[i]));
}
}
return strBuf.toString();
} catch (NoSuchAlgorithmException e) {
return "";
} catch (UnsupportedEncodingException e) {
return "";
public static String stringToMd5(String value) {
if (StringUtils.isBlank(value)) {
return "";
}
try {
MessageDigest md5 = MessageDigest.getInstance(MD5_ALGORITHM);
byte[] encryption = md5.digest(value.getBytes(StandardCharsets.UTF_8));
StringBuilder strBuf = new StringBuilder();
for (byte b : encryption) {
strBuf.append(String.format("%02x", b));
}
return strBuf.toString();
} catch (NoSuchAlgorithmException e) {
return "";
}
}
/**
* The rules of generating taskId
* @param addTaskRequest
* @return
* Generates a task ID based on the given TaskAddRequest.
*
* @param addTaskRequest The TaskAddRequest containing task details.
* @return The generated task ID.
*/
public static String generateTaskId(TaskAddRequest addTaskRequest) {
return generateTaskId(addTaskRequest.getServiceName(), addTaskRequest.getGroupName(),
addTaskRequest.getSourceClusterId(), addTaskRequest.getDestClusterId());
}
/**
* The rules of generating taskId
* Generates a task ID based on the given parameters.
*
* @return
* @param serviceName The service name.
* @param groupName The group name.
* @param sourceClusterId The source cluster ID.
* @param destClusterId The destination cluster ID.
* @return The generated task ID.
*/
public static String generateTaskId(String serviceName, String groupName,
String sourceClusterId, String destClusterId) {
StringBuilder sb = new StringBuilder();
sb.append(serviceName);
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(groupName);
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(sourceClusterId);
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(destClusterId);
return SkyWalkerUtil.StringToMd5(sb.toString());
String sourceClusterId, String destClusterId) {
String rawId = String.join(SkyWalkerConstants.UNDERLINE, serviceName, groupName, sourceClusterId, destClusterId);
return stringToMd5(rawId);
}
/**
* 生成集群clusterId的规则
* Generates a cluster ID based on the given ClusterAddRequest.
*
* @param addClusterRequest
* @return
* @param addClusterRequest The ClusterAddRequest containing cluster details.
* @return The generated cluster ID.
*/
public static String generateClusterId(ClusterAddRequest addClusterRequest) {
StringBuilder sb = new StringBuilder();
sb.append(addClusterRequest.getClusterName());
sb.append(SkyWalkerConstants.UNDERLINE);
sb.append(addClusterRequest.getClusterType());
return SkyWalkerUtil.StringToMd5(sb.toString());
String rawId = String.join(SkyWalkerConstants.UNDERLINE, addClusterRequest.getClusterName(), addClusterRequest.getClusterType());
return stringToMd5(rawId);
}
/**
* Avoid getting a return address
* @return
* @throws Exception
* Gets the local IP address, avoiding loopback addresses.
*
* @return The local IP address.
* @throws Exception If an error occurs while fetching the IP address.
*/
public static String getLocalIp() throws Exception {
InetAddress addr = InetAddress.getLocalHost();
String localIp = addr.getHostAddress();
if (addr.isLoopbackAddress()) {
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface in = interfaces.nextElement();
Enumeration<InetAddress> addrs = in.getInetAddresses();
while (addrs.hasMoreElements()) {
InetAddress address = addrs.nextElement();
if (!address.isLoopbackAddress() && address instanceof Inet4Address) {
localIp = address.getHostAddress();
}
if (!addr.isLoopbackAddress()) {
return addr.getHostAddress();
}
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface networkInterface = interfaces.nextElement();
Enumeration<InetAddress> addrs = networkInterface.getInetAddresses();
while (addrs.hasMoreElements()) {
InetAddress address = addrs.nextElement();
if (!address.isLoopbackAddress() && address instanceof Inet4Address) {
return address.getHostAddress();
}
}
}
return localIp;
return addr.getHostAddress();
}
/**
* Generates a synchronization key based on source and destination cluster types.
*
* @param sourceClusterType The source cluster type.
* @param destClusterType The destination cluster type.
* @return The generated synchronization key.
*/
public static String generateSyncKey(ClusterTypeEnum sourceClusterType, ClusterTypeEnum destClusterType) {
return Joiner.on(":").join(sourceClusterType.getCode(), destClusterType.getCode());
return Joiner.on(SEPARATOR).join(sourceClusterType.getCode(), destClusterType.getCode());
}
/**
* Gets the operation ID from the given TaskDO.
*
* @param taskDO The TaskDO containing the operation ID.
* @return The operation ID.
*/
public static String getOperationId(TaskDO taskDO) {
return taskDO.getOperationId();
}
/**
* Generates a unique operation ID.
*
* @return The generated operation ID.
*/
public static String generateOperationId() {
return UUID.randomUUID().toString();
}
}

View File

@ -13,19 +13,23 @@
package com.alibaba.nacossync.util;
import com.google.common.base.Joiner;
import com.google.common.collect.Maps;
import lombok.extern.slf4j.Slf4j;
import java.io.File;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import static com.alibaba.nacossync.util.DubboConstants.*;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_PATH_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.DUBBO_URL_FORMAT;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_IP_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INSTANCE_PORT_KEY;
import static com.alibaba.nacossync.util.DubboConstants.INTERFACE_KEY;
import static com.alibaba.nacossync.util.DubboConstants.PROTOCOL_KEY;
import static com.alibaba.nacossync.util.DubboConstants.ZOOKEEPER_SEPARATOR;
/**
* @author paderlol
@ -35,24 +39,26 @@ import static com.alibaba.nacossync.util.DubboConstants.*;
public final class StringUtils {
private static final Pattern KVP_PATTERN = Pattern
.compile("([_.a-zA-Z0-9][-_.a-zA-Z0-9]*)[=](.*)");
.compile("([_.a-zA-Z0-9][-_.a-zA-Z0-9]*)[=](.*)");
private static final Pattern IP_PORT_PATTERN = Pattern
.compile(".*/(.*)://(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)");
private static final Pattern DUBBO_PROVIDER_PATTERN = Pattern
.compile("/dubbo/(.*)/providers/(.*)");
.compile(".*/(.*)://(\\d+\\.\\d+\\.\\d+\\.\\d+):(\\d+)");
private static final Pattern DUBBO_PROVIDER_PATTERN = Pattern
.compile("/dubbo/(.*)/providers/(.*)");
public static final int INDEX_NOT_FOUND = -1;
public static final String EMPTY = "";
/**
* parse key-value pair.
*
* @param str string.
* @param str string.
* @param itemSeparator item separator.
* @return key-value map;
*/
private static Map<String, String> parseKeyValuePair(String str, String itemSeparator) {
String[] tmp = str.split(itemSeparator);
Map<String, String> map = new HashMap<String, String>(tmp.length);
for (int i = 0; i < tmp.length; i++) {
Matcher matcher = KVP_PATTERN.matcher(tmp[i]);
Map<String, String> map = new HashMap<>(tmp.length);
for (String s : tmp) {
Matcher matcher = KVP_PATTERN.matcher(s);
if (!matcher.matches()) {
continue;
}
@ -68,17 +74,14 @@ public final class StringUtils {
* @return Parameters instance.
*/
public static Map<String, String> parseQueryString(String qs) {
try {
String decodePath = URLDecoder.decode(qs, "UTF-8");
if (isEmpty(qs)) {
return new HashMap<>();
}
return parseKeyValuePair(decodePath, "\\&");
} catch (UnsupportedEncodingException e) {
log.warn("parse query string failed", e);
return Maps.newHashMap();
String decodePath = URLDecoder.decode(qs, StandardCharsets.UTF_8);
if (isEmpty(decodePath)) {
return new HashMap<>();
}
decodePath = substringAfter(decodePath, "?");
return parseKeyValuePair(decodePath, "&");
}
/**
@ -92,49 +95,79 @@ public final class StringUtils {
}
public static Map<String, String> parseIpAndPortString(String path) {
try {
String decodePath = URLDecoder.decode(path, "UTF-8");
Matcher matcher = IP_PORT_PATTERN.matcher(decodePath);
// extract the ones that match the rules
Map<String, String> instanceMap = new HashMap<>(3);
while (matcher.find()) {
// protocol
instanceMap.put(PROTOCOL_KEY, matcher.group(1));
// ip address
instanceMap.put(INSTANCE_IP_KEY, matcher.group(2));
// port
instanceMap.put(INSTANCE_PORT_KEY, matcher.group(3));
break;
}
return instanceMap;
} catch (UnsupportedEncodingException e) {
log.warn("parse query string failed", e);
return Maps.newHashMap();
String decodePath = URLDecoder.decode(path, StandardCharsets.UTF_8);
Matcher matcher = IP_PORT_PATTERN.matcher(decodePath);
// extract the ones that match the rules
Map<String, String> instanceMap = new HashMap<>(3);
while (matcher.find()) {
// protocol
instanceMap.put(PROTOCOL_KEY, matcher.group(1));
// ip address
instanceMap.put(INSTANCE_IP_KEY, matcher.group(2));
// port
instanceMap.put(INSTANCE_PORT_KEY, matcher.group(3));
break;
}
return instanceMap;
}
/**
* <p>Gets the substring after the first occurrence of a separator.
* The separator is not returned.</p>
*
* <p>A {@code null} string input will return {@code null}.
* An empty ("") string input will return the empty string. A {@code null} separator will return the empty string if
* the input string is not {@code null}.</p>
*
* <p>If nothing is found, the empty string is returned.</p>
*
* <pre>
* StringUtils.substringAfter(null, *) = null
* StringUtils.substringAfter("", *) = ""
* StringUtils.substringAfter(*, null) = ""
* StringUtils.substringAfter("abc", "a") = "bc"
* StringUtils.substringAfter("abcba", "b") = "cba"
* StringUtils.substringAfter("abc", "c") = ""
* StringUtils.substringAfter("abc", "d") = ""
* StringUtils.substringAfter("abc", "") = "abc"
* </pre>
*
* @param str the String to get a substring from, may be null
* @param separator the String to search for, may be null
* @return the substring after the first occurrence of the separator, {@code null} if null String input
* @since 2.0
*/
public static String substringAfter(final String str, final String separator) {
if (isEmpty(str)) {
return str;
}
if (separator == null) {
return EMPTY;
}
final int pos = str.indexOf(separator);
if (pos == INDEX_NOT_FOUND) {
return EMPTY;
}
return str.substring(pos + 1);
}
public static String convertDubboProvidersPath(String interfaceName) {
return String.format(DUBBO_PATH_FORMAT, interfaceName);
}
public static String convertDubboFullPathForZk(Map<String, String> metaData, String providersPath, String ip,
int port) {
try {
String urlParam = Joiner.on("&").withKeyValueSeparator("=").join(metaData);
String instanceUrl = String.format(DUBBO_URL_FORMAT, metaData.get(PROTOCOL_KEY), ip, port,
metaData.get(INTERFACE_KEY), urlParam);
public static String convertDubboFullPathForZk(Map<String, String> metaData, String providersPath, String ip,
int port) {
String urlParam = Joiner.on("&").withKeyValueSeparator("=").join(metaData);
String instanceUrl = String.format(DUBBO_URL_FORMAT, metaData.get(PROTOCOL_KEY), ip, port,
metaData.get(INTERFACE_KEY), urlParam);
return Joiner.on(ZOOKEEPER_SEPARATOR).join(providersPath, URLEncoder.encode(instanceUrl, StandardCharsets.UTF_8));
}
return Joiner.on(ZOOKEEPER_SEPARATOR).join(providersPath, URLEncoder.encode(instanceUrl, "UTF-8"));
} catch (UnsupportedEncodingException e) {
log.warn("convert Dubbo full path", e);
return "";
}
}
public static boolean isDubboProviderPath(String path) {
return DUBBO_PROVIDER_PATTERN.matcher(path).matches();
}
public static boolean isDubboProviderPath(String path) {
return DUBBO_PROVIDER_PATTERN.matcher(path).matches();
}
}

View File

@ -0,0 +1,207 @@
/*
* Copyright (c) 2016-2021 VMware Inc. or its affiliates, All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.alibaba.nacossync.util;
import reactor.util.annotation.NonNull;
import reactor.util.annotation.Nullable;
import reactor.util.function.Tuple2;
import reactor.util.function.Tuples;
import java.io.Serializable;
import java.util.Arrays;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;
import java.util.Objects;
import java.util.function.Function;
/**
* A tuple that holds two non-null values.
*
* @param <T1> The type of the first non-null value held by this tuple
* @param <T2> The type of the second non-null value held by this tuple
* @author Jon Brisbin
* @author Stephane Maldini
*/
@SuppressWarnings("rawtypes")
public class Tuple<T1, T2> implements Iterable<Object>, Serializable {
private static final long serialVersionUID = -3518082018884860684L;
@NonNull
final T1 t1;
@NonNull
final T2 t2;
Tuple(T1 t1, T2 t2) {
this.t1 = Objects.requireNonNull(t1, "t1");
this.t2 = Objects.requireNonNull(t2, "t2");
}
/**
* Type-safe way to get the first object of this {@link Tuples}.
*
* @return The first object
*/
public T1 getT1() {
return t1;
}
/**
* Type-safe way to get the second object of this {@link Tuples}.
*
* @return The second object
*/
public T2 getT2() {
return t2;
}
/**
* Map the left-hand part (T1) of this {@link Tuple} into a different value and type, keeping the right-hand part
* (T2).
*
* @param mapper the mapping {@link Function} for the left-hand part
* @param <R> the new type for the left-hand part
* @return a new {@link Tuple2} with a different left (T1) value
*/
public <R> Tuple<R, T2> mapT1(Function<T1, R> mapper) {
return new Tuple<>(mapper.apply(t1), t2);
}
/**
* Map the right-hand part (T2) of this {@link Tuple} into a different value and type, keeping the left-hand part
* (T1).
*
* @param mapper the mapping {@link Function} for the right-hand part
* @param <R> the new type for the right-hand part
* @return a new {@link Tuple2} with a different right (T2) value
*/
public <R> Tuple<T1, R> mapT2(Function<T2, R> mapper) {
return new Tuple<>(t1, mapper.apply(t2));
}
/**
* Get the object at the given index.
*
* @param index The index of the object to retrieve. Starts at 0.
* @return The object or {@literal null} if out of bounds.
*/
@Nullable
public Object get(int index) {
switch (index) {
case 0:
return t1;
case 1:
return t2;
default:
return null;
}
}
/**
* Turn this {@code Tuple} into a {@link List List&lt;Object&gt;}. The list isn't tied to this Tuple but is a
* <strong>copy</strong> with limited mutability ({@code add} and {@code remove} are not supported, but {@code set}
* is).
*
* @return A copy of the tuple as a new {@link List List&lt;Object&gt;}.
*/
public List<Object> toList() {
return Arrays.asList(toArray());
}
/**
* Turn this {@code Tuple} into a plain {@code Object[]}. The array isn't tied to this Tuple but is a
* <strong>copy</strong>.
*
* @return A copy of the tuple as a new {@link Object Object[]}.
*/
public Object[] toArray() {
return new Object[] {t1, t2};
}
/**
* Return an <strong>immutable</strong> {@link Iterator Iterator&lt;Object&gt;} around the content of this
* {@code Tuple}.
*
* @return An unmodifiable {@link Iterator} over the elements in this Tuple.
* @implNote As an {@link Iterator} is always tied to its {@link Iterable} source by definition, the iterator cannot
* be mutable without the iterable also being mutable. Since {@link Tuples} are <strong>immutable</strong>, so is
* the {@link Iterator} returned by this method.
*/
@Override
public Iterator<Object> iterator() {
return Collections.unmodifiableList(toList()).iterator();
}
@Override
public boolean equals(@Nullable Object o) {
if (this == o) {
return true;
}
if (o == null || getClass() != o.getClass()) {
return false;
}
Tuple<?, ?> tuple = (Tuple<?, ?>) o;
return t1.equals(tuple.t1) && t2.equals(tuple.t2);
}
@Override
public int hashCode() {
int result = size();
result = 31 * result + t1.hashCode();
result = 31 * result + t2.hashCode();
return result;
}
/**
* Return the number of elements in this {@literal Tuples}.
*
* @return The size of this {@literal Tuples}.
*/
public int size() {
return 2;
}
/**
* A Tuple String representation is the comma separated list of values, enclosed in square brackets.
*
* @return the Tuple String representation
*/
@Override
public final String toString() {
Object[] values = toArray();
StringBuilder sb = new StringBuilder();
for (int i = 0; i < values.length; i++) {
Object t = values[i];
if (i != 0) {
sb.append(',');
}
if (t != null) {
sb.append(t);
}
}
return sb.insert(0, '[').append(']').toString();
}
public static <T1, T2> Tuple<T1, T2> of(T1 t1, T2 t2) {
return new Tuple<>(t1, t2);
}
}

View File

@ -11,6 +11,5 @@ spring.cloud.discovery.enabled=false
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/nacos_sync?characterEncoding=utf8
spring.datasource.username=root
spring.datasource.password=root
management.endpoints.web.exposure.include=*
management.endpoint.health.show-details=always

View File

@ -4,12 +4,13 @@
<springProperty name="logPath" scope="context" source="nacosSync.logs.path" defaultValue="${nacosSync.home}/logs"/>
<property name="LOG_HOME" value="${logPath}" />
<appender name="FILEINFO" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_HOME}/nacosSync.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${LOG_HOME}/log-nacosSync-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
<maxFileSize>200MB</maxFileSize>
</timeBasedFileNamingAndTriggeringPolicy>
<file>${LOG_HOME}/nacos-sync.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_HOME}/log-nacos-sync-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
<maxFileSize>200MB</maxFileSize>
<maxHistory>7</maxHistory>
<totalSizeCap>2GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<append>true</append>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
@ -25,7 +26,6 @@
</appender>
<logger name="com.alibaba.nacossync" level="INFO"/>
<logger name="org.hibernate.type.descriptor.sql.BasicBinder" level="INFO"/>
<logger name="org.hibernate.type.descriptor.sql.BasicExtractor" level="INFO"/>
<logger name="org.hibernate.SQL" level="INFO"/>

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More