Commit Graph

89 Commits

Author SHA1 Message Date
Alessandro (Ale) Segala 934e86c4f7
Rename folder "internal" to "common" (#3225)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-11-14 11:36:38 -08:00
Alessandro (Ale) Segala f4e73b0e65
Update dapr/kit (#3205)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-11-02 16:20:56 -07:00
Alessandro (Ale) Segala ec05809ee6
[Metadata] Update validator and some other fixes (#2984)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-07-17 22:17:22 +00:00
Alessandro (Ale) Segala cef854fab6
Some linting in metadata.yaml files (#2977)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-07-11 01:53:58 +00:00
Arturo Trenard ef4780beaa
Pubsub pluggable component bulkpublish (#2931)
Signed-off-by: arturo <jarturotrenard@gmail.com>
Signed-off-by: arturo <ajt@ptytechnologies.com>
Signed-off-by: Arturo Trenard <jarturotrenard@gmail.com>
Co-authored-by: arturo <ajt@ptytechnologies.com>
2023-06-22 15:14:18 +00:00
Bernd Verst 1a751931f7
Add Service Bus Topics Pubsub Metadata (#2776)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-04-12 21:49:49 -07:00
Alessandro (Ale) Segala 5dad8a867c
Metadata parser: add support for built-in authentication profiles and metadata properties (#2760)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-04-12 00:04:34 +00:00
Bernd Verst 2b89d78a2d
Refactor Metadata Parsing of all PubSub Components (#2759)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-04-09 18:21:56 +00:00
Josh van Leeuwen 569c6be070
PubSub: Wait Group and context catching (#2500)
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-03-03 17:44:26 +00:00
Bernd Verst 4068531538
Merge branch 'master' into metadata 2023-02-22 16:46:43 -08:00
ItalyPaleAle f9cf54ca6f Merge branch 'master' of https://github.com/dapr/components-contrib into release-1.10
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-02-16 23:23:23 +00:00
Josh van Leeuwen d098e38d6a
Propagate context from caller to appropriate places in the code (#2474)
* Propagates contexts to callers where appropriate.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates units tests with new func signature

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix linting errors

Signed-off-by: joshvanl <me@joshvanl.dev>

* Add atomic gate to alicloud rocketmq close channel.

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/aws/kinesis use a separate ctx variable name

Signed-off-by: joshvanl <me@joshvanl.dev>

* binding/kafka: use atomic to prevent closing the channel twice

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: use atomic bool to prevent close channel being closed multiple times

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: use Background context for handle operations:w

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/cocroachdb: add context to Ping()

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/postgres: add comment explaining use of context.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds comment header to health/pinger.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/aws/snssqs: add waitgroup to wait for all go routines to finish
and block on Close(). Shuts down the subscription if there are no topic
handlers.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: add atomic bool to prevent multiple channel closes. Add
wait group to block close on all goroutines to finish.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/rabbitmq: fixes race conditions, uses atomic to prevent multiple
closes, add wait group to block close on all goroutines

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/redis: revert ctx passed when it could be cancelled. Add wait
group wait when closing.

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/postges: pass context in init, and wait group on close.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update all `Ping()` to `PingContext()` where possible.

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/in-memory: add atomic bool to prevent closing channel multiple
times. Add wait group to block on close()

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/mysql: don't use same ctx variable name

Signed-off-by: joshvanl <me@joshvanl.dev>

* Pass correct loop context to redis go routines

Signed-off-by: joshvanl <me@joshvanl.dev>

* Rename context when creating timeouts in state

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove state.Features() from requiring a context

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert wasm request handle Close func to be without context to
implement io.Closer interface. Add 5 second timeout. Add io.Closer
assertion in test.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove superfluous go lint vet directive

Signed-off-by: joshvanl <me@joshvanl.dev>

* Change Configuration Init function to take context

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates input binding interface to include a `Close() error` function. `Close`
blocks until all resources have been released and go routines have returned.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Change `Close() error` in input binding struct to `io.Closer` interface.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update go.mod files to point to dapr/dapr PR https://github.com/dapr/dapr/pull/5831

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/redis: watch closeCh to shutdown worker instead of init context.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/aws/snssqs + bindings/kubemq: ensure closeCh is caught so Close
correctly returns

Signed-off-by: joshvanl <me@joshvanl.dev>

* Close kubemq binding client on close. Ensure kafka consumer channel
cannot be closed more than once.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Tweaks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed cert tests

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* binding/mqtt3: add inline Background context instead of passing to
handleMessage

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: remove context from createSubscriberClientOptions

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: Remove `ResetConnection` func

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/kafka: Don't resubscribe if Subscribe is cancelled.

Signed-off-by: joshvanl <me@joshvanl.dev>

* binding/mqtt3: don't use context to control establishing connection

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: Fix linting errors

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2023-02-16 14:18:35 -08:00
Bernd Verst b1c1304a86 Add and update component metadata definition files
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-02-13 17:09:09 -08:00
Alessandro (Ale) Segala 12acf8effe
[release-1.10] Azure Service Bus: fix goroutine "explosion" after recovering from failure and many other bugs (#2523)
* Fixed context leak

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Finalize messages in background

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* More tweaks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Moved a lot more code to the shared implementation package

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed:
- Fixed: if number of messages received in bulk is greater than maxConcurrentHandlers, a deadlock happens
- Fixed: maxRetriableErrorsPerSec was not working

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Some refactorings

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Simplifications

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Avoid checking if topic/queue exists every time we send a message

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Ensuring locks are renewed for all 3 components

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Updated ASB SDK

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Improvements to closing

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Improvements to closing

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Changed context propagation

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Debug logs in cert tests

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Initial delay before renewing locks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* That last change was actually unnecessary

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Tweaks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* More tweaks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* 💄

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

---------

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-02-13 09:23:03 -08:00
ItalyPaleAle 7047ad533d More logging when Service Bus reconnects after publishing error
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-01-31 17:58:30 +00:00
Deepanshu Agarwal b2d8e6013b
Add bulkSubscribe request params to SubscribeRequest (#2405)
* Add bulkSubscribe configuration params

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Replace dapr/dapr package to run cert tests

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Change bulkSubscribe param types

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update latest dapr/dapr fork

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update correct dapr/dapr version

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* change BulkSubscribeRequest to Config

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update dapr/dapr version for cert

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

* Update with latest dapr fork

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>

Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
2023-01-10 09:30:25 -08:00
Joni Collinge 382f84f30c
Add support for Azure Service Bus sessions for topics (#2346)
* WIP: ASB sessions

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* WIP: Support ASB sessions

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Add metadata

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* remove dead code

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* lint

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Add sessionIdleTimeoutInSec md

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Support blank session IDs

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* If require sessions and no session id, set blank

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* move session config to per subscription metadata

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* add some unit tests

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* lint

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* remove auto blank session id

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* go mod tidy

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* add basic cert test

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* update comment

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* remove dead code

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* remove dead code

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Update certification test

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Add comment

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Add cert test for roundrobin

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* go mod tidy

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Update readme

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Use struct for NewSubscription to simplify parameters

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Address feedback

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Address feedback

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Only async receive

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* isTruthy on map without check

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Use if not select for context check

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* Feedback

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

* lint

Signed-off-by: Joni Collinge <jonathancollinge@live.com>

Signed-off-by: Joni Collinge <jonathancollinge@live.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
2023-01-06 09:59:26 -08:00
Mukundan Sundararajan cff6d7b6b1 Merge remote-tracking branch 'upstream/master' into optimize-bulkpubres-struct 2022-12-20 23:04:19 +05:30
ItalyPaleAle fc00023206 Small naming fixes
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-12-19 22:03:53 +00:00
seachen 1ea2cff9fd feature: add context to lock&pubsub API
Signed-off-by: seachen <seachen@tencent.com>
2022-12-15 09:53:32 +08:00
seachen e59dfd63c0 feature: add context to lock&pubsub API
Signed-off-by: seachen <seachen@tencent.com>
2022-12-14 19:37:24 +08:00
Mukundan Sundararajan 72695529f6 optimize bulk pub response to contain only failed entries
Signed-off-by: Mukundan Sundararajan <65565396+mukundansundar@users.noreply.github.com>
2022-12-13 11:49:04 +05:30
Alessandro (Ale) Segala 04242ce38a
New PubSub component: Azure Service Bus Queues (#2211)
* Implementation for Azure Service Bus Queue pubsub component

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed syntax errors in unused files

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Moved components to subfolders and enabled conf tests

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-10-24 14:07:23 -07:00
ItalyPaleAle 254a2141ad Moved sender handling to common implementation too
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-19 21:01:15 +00:00
ItalyPaleAle 8013d96ad6 More tweaks
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-19 20:39:56 +00:00
ItalyPaleAle 065ef86ee5 Moved client creation to common implementation
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-19 20:34:10 +00:00
ItalyPaleAle 7ecfb6edd0 Moved code to add metadata to ASB messages to shared impl too
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-19 20:08:07 +00:00
ItalyPaleAle 8bce0d5200 Moved metadata to impl package
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-19 19:47:26 +00:00
ItalyPaleAle 0057a31b88 Moved errors
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-18 21:51:17 +00:00
Bernd Verst 0f0aaf4e9b Merge remote-tracking branch 'upstream/master' into merge19intomaster 2022-10-14 12:16:57 -07:00
ItalyPaleAle 4de7a2fc9d The people demand defers
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-12 18:46:38 +00:00
ItalyPaleAle 047f3948c5 Fix for handling "context deadline exceeded" on ASB publishing
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-12 18:18:33 +00:00
ItalyPaleAle 3bc20a1de5 Merge branch 'master' into merge-1.9
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-11 20:26:35 +00:00
Alessandro (Ale) Segala ef74e7f8b6
Azure Service Bus: consider receiving 0 messages as an error that requires reconnecting (#2169)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-10-11 12:57:06 -07:00
Deepanshu Agarwal f74b4d648d Update EntryID to EntryId
Signed-off-by: Deepanshu Agarwal <deepanshu.agarwal1984@gmail.com>
2022-09-29 15:49:45 +05:30
Shubham Sharma cd206609ce
Use request metadata instead of component metadata (#2119)
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
2022-09-23 10:49:39 -07:00
Shubham Sharma 4f406a597f
Add bulk subscribe support to Azure Service Bus (#2100)
* Initial implementation

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* 

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Add comment

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor messages.go and add tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update message tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor servicebus.go logic

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor handleAsync

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor ReceiveAndBlock

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update binding

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Review comments addressed

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Lint

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update metadata name

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update activeMessagesChan logic for concurrency control

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Update metadata name

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Use request metadata instead of component metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Review comments addressed

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix merge issue with metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Remove unused metadata key

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Fix merge issue with metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Lint!!!

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
2022-09-23 10:49:06 -07:00
Shubham Sharma b298b65cfe
Add bulk publish support to Azure Service Bus (#2106)
* Initial implementation

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Some improvements

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor and add tests

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Lint

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Refactor

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Use request metadata instead of component metadata

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Remove unused method

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Review comments addressed

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

* Doc comments

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>

Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
2022-09-23 09:57:00 -07:00
Marcos Candeia 8964d8bf41
Define common metadata across components (#1994)
* Add base metadata for all components

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Fix lint issues on consul and mysql tests

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Pin candeia/dapr runtime version

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Add replace directive to use mcandeia/dapr on all go submodules

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Fixed runtime config instantiation

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
2022-08-25 10:10:53 -07:00
Bernd Verst e87cd5e4cb
Go 1.19 support and linter fixes (#1975)
* Go 1.19 support and linter fixes

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update workflows for Go1.19 and new linter version

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Remove unnecessary space in nolint directive

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* disable additional linters which aren't used because of Go generics

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* enable gosec linter again

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update bindings/zeebe/command/publish_message_test.go

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update bindings/output_binding.go

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Use prepared statement for mysql table creation

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Ping is not ping

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* c'mon linter

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Fix MySQL gosec issue

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* revert mysql to be fixed later

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-08-18 00:45:23 -07:00
addjuarez cd740b4a7b
Handle service bus exported errors (#1942)
* Add service bus exported error handling

Signed-off-by: Addison Juarez <adjuarez@microsoft.com>

* Add service bus exported error handling

Signed-off-by: Addison Juarez <adjuarez@microsoft.com>

Signed-off-by: Addison Juarez <adjuarez@microsoft.com>
Co-authored-by: Loong Dai <long.dai@intel.com>
2022-08-12 14:47:14 -05:00
Alessandro (Ale) Segala a2f3a84b96
Moved authentication to be an internal pkg (#1855)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-07-06 11:05:34 -07:00
Alessandro (Ale) Segala a08988c25e
Shared implementation for Azure Service Bus subscriptions for binding and pubsub (#1791)
* Moved ASB subscription code to a shared package

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Moved ASBQ binding to use the shared ASB implementation

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Updating Azure Service Bus SDK version

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-17 15:12:15 -07:00
Bernd Verst f25b26fc0c
Upgrade to Azure Table Storage Track 2 SDK in State Store and add CosmosDB Table API support (#1792)
* Upgrade to Azure Table Storage Track 2 SDK in State Store

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* make modtidy-all

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* don't panic

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* fix error parsing

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Adds support for empty rowKeys again

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Pin aztables v1.0.1

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Add support for Cosmos DB Table API

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Add Azure Table Storage CosmosDB conformance test

Signed-off-by: GitHub <noreply@github.com>

* Address code review

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Create Table Storage State store subtests

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Address linter issue

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* context instead of ctx

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* load state store in conf tests

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* use cosmosDB mode in cosmosDB test

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Add Azure AD support for Table Storage

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* remove shadow declaration

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* make accountKey optional

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Add options to skip table creation

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-06-17 14:04:53 -07:00
Alessandro (Ale) Segala 98aed5c571
Make Service Bus attempt to reconnect forever in case of issues (#1783)
* Make Service Bus attempt to reconnect forever in case of issues

With exponential back-off configurable between min and max time. Fixes #1612

Also includes fixes:

- Binding: make sure it actually retries to connect forever
- Binding: add delay (exponential backoff) before reconnecting
- PubSub: better handling of failures such as topics disabled or other non-connection issues

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* 💄

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Added warning for deprecated metadata options

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* 💄

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-10 15:54:46 -07:00
Alessandro (Ale) Segala 704f4dd730
Add contexts to pubsub.Subscribe to allow early cancelation (#1756)
This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.

In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).

This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.

PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-02 15:06:36 -07:00
Alessandro (Ale) Segala f7834bd232
Fixed typo in metadata JSON key (#1733)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-05-19 12:35:30 -07:00
Alessandro (Ale) Segala e1b6b01223
Tweaks for ASB pubsub (#1729)
* Tweaks for ASB pubsub:
1. Do not attempt to reconnect when context was canceled
2. Message finalization uses a background context and not the running ctx, in case the app is shutting down

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* ASB: abandoned messages shouldn't be retried right away

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Ported fixes to servicebusqueues binding too

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Use a different approach to limit the number of messages processed in case of error

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Added maxRetriableErrorsPerSec to servicebusqueues binding too
Incl other improvements

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Do not reconnect to ASB every time we retrieve a message.

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* ⚙️

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-05-17 22:33:34 -07:00
Alessandro (Ale) Segala e5e9011655
Update Azure Service Bus components to track2 SDK (#1702)
* Updated to Go 1.18

Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* Updated Azure SDKs that are on track2
Includes some minor refactoring of auth code

Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* Updated Service Bus components to track2 SDK

Co-authored-by: halspang <halspang@microsoft.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* Initial Certification test for eventhubs binding [incomplete] (#1670)

* certification test for eventhubs binding

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

* modified go.mod and go.sum

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

* Add connection string testing

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

* iothub testing

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

* address feedback and run test

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

* Install Azure CLI IOT hub extension

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* make modtidy-all

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* covering all eventhubs test cases

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

* dependency changes after go modtidy-all

Signed-off-by: tanvigour <tanvi.gour@gmail.com>

Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* Use revive instead of golint (#1685)

Signed-off-by: pigletfly <wangbing.adam@gmail.com>

Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* Updated to Go 1.18 (#1697)

* Updated to Go 1.18

Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>

* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Removed go.work

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* 💄

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* 💄 & 🧹

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* Add metadata property to configure Batching in Pulsar (#1707)

* Add metadata property to configure BatchingMaxSize&batchingMaxMessages in Pulsar
Signed-off-by: saberwang <saberwang@hotmail.com>

* sort field
Signed-off-by: saberwang <saberwang@hotmail.com>

* [pubsub]fix unit test bug
Signed-off-by: saberwang <saberwang@hotmail.com>

* remove unrelated changes
Signed-off-by: saberwang <saberwang@hotmail.com>

* Delete hard coded Metadata
Signed-off-by: saberwang <saberwang@hotmail.com>

* remove  .history

Signed-off-by: saberwang <saberwang@hotmail.com>

* restore .gitignore

Signed-off-by: saberwang <saberwang@hotmail.com>

* Hard coding default values and adding 'BatchingMaxPublishDelay' metadata

Signed-off-by: saberwang <saberwang@hotmail.com>

* fix code format

Signed-off-by: saberwang <saberwang@hotmail.com>

* formatting code

Signed-off-by: saberwang <saberwang@hotmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* This test can't work with track2 SDKs
The methods to create a message with a body are not exported

Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* 🧹

Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>

* There's such thing as too much logging

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Refactored subscription.go
This greatly simplifies certain parts of the code, reducing the number of goroutines and likely improving performance.
Performance for end-users improves too as there's no need anymore to pause for 2 seconds every time that we reach `maxActiveMessages`.
Additionally, with this change the config options `prefetchCount` and `maxActiveMessagesRecoveryInSec` are removed as unnecessary anymore.

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* 💄

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed pubsub tests

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* These packages should have never been upgraded

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Ensuring we don't fetch 1 message more than max active

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Adding configurable timeout for servicebusqueues operations

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Persistent connection for invoking SB queues

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Reverted Event Hub SDK update

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Revert "Reverted Event Hub SDK update"

This reverts commit 212220aba0.

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fix Azure deploy for users with a first.last email

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Added some sleep that should help reduce flakiness in eventhubs binding cert test

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Changed servicebusqueue cert test
In case of a failure in the handler (ie. users' code), the message should be correctly re-enqueued, which means that messages will be re-delivered later and won't be in order.

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: halspang <halspang@microsoft.com>
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Wang Bing <wangbing.adam@gmail.com>
Co-authored-by: saber-wang <45062099+saber-wang@users.noreply.github.com>
2022-05-13 16:52:22 -07:00
Wang Bing db99b43068
Use revive instead of golint (#1685)
Signed-off-by: pigletfly <wangbing.adam@gmail.com>

Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2022-05-06 12:55:17 -07:00