dapr
/
components-contrib
mirror of https://github.com/dapr/components-contrib.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,254 Commits 55 Branches 159 Tags 84 MiB
Commit Graph

40 Commits

Author SHA1 Message Date
Deepanshu Agarwal 2582f154bd
Add back headers metadata pass in kafka subscribe (#2281) 
* Add back headers metadata pass in kafka subscribe

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update internal/component/kafka/consumer.go

Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
 2022-12-20 11:43:52 -08:00
Mukundan Sundararajan cff6d7b6b1 Merge remote-tracking branch 'upstream/master' into optimize-bulkpubres-struct 2022-12-20 23:04:19 +05:30
seachen e59dfd63c0 feature: add context to lock&pubsub API 
Signed-off-by: seachen <seachen@tencent.com>
 2022-12-14 19:37:24 +08:00
Mukundan Sundararajan ace0d1a03c changed to failed entries for bulk publish response 
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
 2022-12-13 16:22:45 +05:30
Mukundan Sundararajan 72695529f6 optimize bulk pub response to contain only failed entries 
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
 2022-12-13 11:49:04 +05:30
Sky Ao 5cc923bcc4 rollback code to set config.net.tls.enable=true when auth is required and tls is not required 
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
 2022-10-13 12:02:05 +00:00
Deepanshu Agarwal 376cfed216 Fix linting 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-29 16:27:04 +05:30
Deepanshu Agarwal f74b4d648d Update EntryID to EntryId 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-29 15:49:45 +05:30
deepanshuagarwal bcd101abad
Keys rename (#2135) 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-28 09:52:59 -07:00
Mukundan Sundararajan 8b60079501 Merge remote-tracking branch 'upstream/master' into fix-kafka-session-done-merge-master 2022-09-24 00:11:19 +05:30
Shubham Sharma 4f406a597f
Add bulk subscribe support to Azure Service Bus (#2100) 
* Initial implementation

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* ✅

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add comment

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor messages.go and add tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update message tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor servicebus.go logic

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor handleAsync

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor ReceiveAndBlock

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update binding

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Review comments addressed

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Lint

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update metadata name

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update activeMessagesChan logic for concurrency control

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update metadata name

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Use request metadata instead of component metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Review comments addressed

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix merge issue with metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Remove unused metadata key

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix merge issue with metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Lint!!!

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
 2022-09-23 10:49:06 -07:00
cong 1a7882fca5
fix(kafka): ConsumeClaim should return when messages channel closed (#2120) 
Signed-off-by: cong <zhangcong1992@gmail.com>

Signed-off-by: cong <zhangcong1992@gmail.com>
 2022-09-23 09:54:59 -07:00
deepanshuagarwal 8500da577c
Conf test for Bulk Subscribe and Kafka fixes (#2113) 
* Conf test for Bulk Sub and Kafka Fixes

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Fix linting

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Incorporating comments

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-23 09:54:21 -07:00
Mukundan Sundararajan e1697ba115 Merge branch 'master' into feature/pubsub-batching 2022-09-23 06:47:47 +05:30
Mukundan Sundararajan 4354f036e4 properly parse response errors from kafka on sending multiple messages in bulk publish 
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
 2022-09-22 16:51:34 +05:30
Bernd Verst ac73313d6f
Merge branch 'master' into master 2022-09-21 23:40:53 -07:00
Mukundan Sundararajan a45f7db8b2 add bulkpublish-conf-test 
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
 2022-09-21 11:46:46 +05:30
Bernd Verst e9e750bce6 fix linter issues 
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
 2022-09-20 13:02:52 -07:00
Deepanshu Agarwal 609f4080fc Use index as entryID 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-19 09:17:40 +05:30
Deepanshu Agarwal 3656a495fc Incorporating comments 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-16 23:03:58 +05:30
Deepanshu Agarwal 217505fd16 Move common costants to common utils 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-16 22:22:54 +05:30
Deepanshu Agarwal d602e0ab0c Move constants to utils 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-16 22:07:38 +05:30
Deepanshu Agarwal 55ad67d510 When UUID could not be generated 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-16 21:44:37 +05:30
Deepanshu Agarwal 81898d5ee5 Config Per Topic and using maxAwaitDuration 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-16 20:04:32 +05:30
Deepanshu Agarwal d31f4d99f5 Merge branch 'feature/pubsub-batching' of github.com:dapr/components-contrib into ps_batch_c_kafka 2022-09-16 11:36:23 +05:30
Mukundan Sundararajan 3f9c42c4b8
Kafka bulk publish (#2059) 
* Contrib updates for Batch Subscribe

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* commit

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Commit pubsub

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update structs

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update Default Batch struct name

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update Default Batch

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update return type for BatchPublish

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Add docs

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Rename batch to bulk

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* snssqs renam batch to bulk

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Renaming Bulker to BulkMessager

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Renaming names as discussed

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Minor updates

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Remove ContentType from BulkPublishRequest

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update error type

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Changes to bulk subscribe and error type

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Segregating Bulk Publisher and Bulk Subscriber and removing them as mandatory

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Event Type name correction

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Docs update as per comments

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* bulk publish implementation for kafka

Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>

* Add ctx to BulkPublish.

Signed-off-by: Artur Souza <artursouza.ms@outlook.com>

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
Signed-off-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
 2022-09-15 13:17:17 -07:00
Deepanshu Agarwal f11660e70d Change to warning normal subscribe part as well 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-15 16:09:47 +05:30
Deepanshu Agarwal f5542c70b9 Incorporate review comments and remove index based entryID 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-15 16:02:30 +05:30
Deepanshu Agarwal 31a38e5aaa Change marking message logic 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-14 13:01:30 +05:30
Deepanshu Agarwal 059461c6f4 Add kafka changes for bulk subscribe 
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
 2022-09-13 01:01:05 +05:30
zcong1993 320ae2d674
fix(kafka): should return when session context is done in ConsumeClaim 
Signed-off-by: zcong1993 <zhangcong1992@gmail.com>
 2022-09-05 16:28:49 +08:00
Mukundan Sundararajan 35590e0857 read record header as metadata if available in kafka 
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
 2022-08-26 12:02:15 +05:30
Bernd Verst e87cd5e4cb
Go 1.19 support and linter fixes (#1975) 
* Go 1.19 support and linter fixes

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update workflows for Go1.19 and new linter version

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Remove unnecessary space in nolint directive

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* disable additional linters which aren't used because of Go generics

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* enable gosec linter again

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update bindings/zeebe/command/publish_message_test.go

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update bindings/output_binding.go

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Use prepared statement for mysql table creation

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Ping is not ping

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* c'mon linter

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Fix MySQL gosec issue

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* revert mysql to be fixed later

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
 2022-08-18 00:45:23 -07:00
cong 38fc439410
fix(kafka): ConsumeClaim should not return err when retry recovery failed (#1965) 
* fix(kafka): ConsumeClaim should not return err when retry recovery failed

Signed-off-by: zcong1993 <zhangcong1992@gmail.com>

* chore: tweak error log message

Signed-off-by: zcong1993 <zhangcong1992@gmail.com>

Signed-off-by: zcong1993 <zhangcong1992@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
 2022-08-12 19:16:02 -07:00
Andrew 2b3712b755
Ft/sasl scram sha 256 512 (#1856) 
* working with hardcoded SHA-512

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* cleanup code

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* Do not hardcode specific testTopicName

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Andrew Duss <andy.duss@storable.com>

* Ensure context propagation in MySQL binding (#1829)

Spin-off from PR adding contexts to input bindings

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Signed-off-by: Andrew Duss <andy.duss@storable.com>

* Add support for AAD auth in Azure Storage Queues binding (#1842)

* Add support for AAD auth in Azure Storage Queues binding

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* 🧹

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Andrew Duss <andy.duss@storable.com>

* Moved authentication to be an internal pkg (#1855)

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Andrew Duss <andy.duss@storable.com>

* Azure AD support in SignalR (#1852)

* WIP: Azure AD support in SignalR

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Correct SignalR AAD details

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Misc fixes

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* azauth package name

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Andrew Duss <andy.duss@storable.com>

* rename SCRAM properly as SASL

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* update gomod/sum

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* gofmt

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* mod tidy

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* goval

Signed-off-by: Andrew Duss <andy.duss@storable.com>

* use xdg-go instead of depreicated xdg library

Signed-off-by: Andrew Duss <andy.duss@storable.com>

Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
 2022-07-08 06:08:40 -07:00
Sky Ao d6dd10e19d
improve log of kafkaf components cert test (#1800) 
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
 2022-06-20 09:49:09 -07:00
Alessandro (Ale) Segala 704f4dd730
Add contexts to pubsub.Subscribe to allow early cancelation (#1756) 
This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.

In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).

This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.

PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
 2022-06-02 15:06:36 -07:00
Alessandro (Ale) Segala 7c35a4e04c
Added test for multiple pubsub handlers (#1743) 
* Added test for multiple pubsub handlers

This patch adds conformance tests for PubSub with multiple handlers, each listening on a different topic.

It's meant to catch issues such as #1709

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Added missing go.mod for storagequeues binding test

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
 2022-06-02 10:22:15 -07:00
Alessandro (Ale) Segala ca9fbf690e
Fixed Kafka PubSub to allow multiple handlers for different topics (#1755) 
* Fixed Kafka PubSub to allow multiple handlers for different topics

With this, tests from #1743 are passing

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Removed read lock (for now)

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Updated as requested by @skyao

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Use interface type everywhere

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed panic

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
 2022-06-01 15:31:11 -07:00
Sky Ao 8b5554dc71
Refactory kafka binding to reuse the kafka common code extracting from kafka pubsub component (#1696) 
* refactory kafka pubsub code to extract common kafka code for reuse

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* fix lint;add unit test for subscribeAdapter

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* move topics filed from internal kafak struct to pubsub kafka struct, since in input binding the topics will confiured in metadata

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* reuse internal  kafka code for bindings

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* add redis standalone_test back which is delete by mistaken

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* small code improvement to trigger test

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* add license headers

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* try to set disbaleTls to true to verify the kafka connection fail

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* don't enable consum retry in kafka binding component;if authenticaion is disabled, need not set TLSDisable at the same time;

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

* fix lint

Signed-off-by: Sky Ao <aoxiaojian@gmail.com>

Co-authored-by: Loong Dai <long.dai@intel.com>
 2022-05-10 20:07:23 -07:00