Commit Graph

71 Commits

Author SHA1 Message Date
Bernd Verst 9095b0e7e7
Switch to Go 1.21, Updates linter, updates workflows, adds sarama 1.42.1 (#3251)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-11-28 18:13:09 -08:00
Alessandro (Ale) Segala f4e73b0e65
Update dapr/kit (#3205)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-11-02 16:20:56 -07:00
Josh van Leeuwen d47d1a7aad
pubsub/rabbitmq: return error if reconnect errors in Init (#3198)
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-10-27 10:38:43 -07:00
Roberto Rojas f074240c0a
[PubSub RabbitMQ] Adds Component Metadata Schema (#3069)
Signed-off-by: robertojrojas <robertojrojas@gmail.com>
Signed-off-by: Roberto Rojas <robertojrojas@gmail.com>
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Tiago Alves Macambira <tmacam@burocrata.org>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
2023-09-12 05:44:51 -07:00
Taction 0c2ce324b2
Rabbitmq pubsub add client name (#2933)
Signed-off-by: zhangchao <zchao9100@gmail.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-09-05 15:14:17 -07:00
Álvaro Aguilar-Tablada Espinosa e10d5b7e86
[RabbitMQ pub/sub] Allow quorum queues (#2816)
Signed-off-by: Bernd Verst <github@bernd.dev>
Signed-off-by: Alvaro Aguilar <alvaro.aguilar@scrm.lidl>
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Álvaro Aguilar <95039001@lidl.de>
Signed-off-by: Álvaro Aguilar <alvaroteleco@hotmail.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Álvaro Aguilar <95039001@lidl.de>
2023-08-07 06:28:58 -07:00
Alessandro (Ale) Segala ec05809ee6
[Metadata] Update validator and some other fixes (#2984)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-07-17 22:17:22 +00:00
Jack Robertson 3bfa20bda1
rabbitmq pub/sub: add queue name to subscription request metadata (#2962)
Signed-off-by: Jack Robertson <jcrobertson97@gmail.com>
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
2023-07-07 21:24:25 +00:00
spike 2576a3150d
Fix rabbitmq exchange declare (#2836)
Signed-off-by: Bernd Verst <github@bernd.dev>
Signed-off-by: spike <hello@spike.wiki>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-05-22 18:44:27 -07:00
Bernd Verst 2b89d78a2d
Refactor Metadata Parsing of all PubSub Components (#2759)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-04-09 18:21:56 +00:00
Yaron Schneider 0e83c4adc5
Add priority queue support to rabbitmq pub/sub (#2680)
Signed-off-by: yaron2 <schneider.yaron@live.com>
2023-03-17 13:00:49 +00:00
Andre Bossard d87f4bb5a1
RabbitMQ: Support for external authentication with SASL (#2515)
Signed-off-by: Andre Bossard <anbossar@microsoft.com>
Signed-off-by: Andre Bossard <abossard@users.noreply.github.com>
Co-authored-by: Andre Bossard <anbossar@microsoft.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2023-03-16 16:30:54 +00:00
Josh van Leeuwen 569c6be070
PubSub: Wait Group and context catching (#2500)
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-03-03 17:44:26 +00:00
Josh van Leeuwen d098e38d6a
Propagate context from caller to appropriate places in the code (#2474)
* Propagates contexts to callers where appropriate.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates units tests with new func signature

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix linting errors

Signed-off-by: joshvanl <me@joshvanl.dev>

* Add atomic gate to alicloud rocketmq close channel.

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/aws/kinesis use a separate ctx variable name

Signed-off-by: joshvanl <me@joshvanl.dev>

* binding/kafka: use atomic to prevent closing the channel twice

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: use atomic bool to prevent close channel being closed multiple times

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: use Background context for handle operations:w

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/cocroachdb: add context to Ping()

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/postgres: add comment explaining use of context.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds comment header to health/pinger.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/aws/snssqs: add waitgroup to wait for all go routines to finish
and block on Close(). Shuts down the subscription if there are no topic
handlers.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: add atomic bool to prevent multiple channel closes. Add
wait group to block close on all goroutines to finish.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/rabbitmq: fixes race conditions, uses atomic to prevent multiple
closes, add wait group to block close on all goroutines

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/redis: revert ctx passed when it could be cancelled. Add wait
group wait when closing.

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/postges: pass context in init, and wait group on close.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update all `Ping()` to `PingContext()` where possible.

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/in-memory: add atomic bool to prevent closing channel multiple
times. Add wait group to block on close()

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/mysql: don't use same ctx variable name

Signed-off-by: joshvanl <me@joshvanl.dev>

* Pass correct loop context to redis go routines

Signed-off-by: joshvanl <me@joshvanl.dev>

* Rename context when creating timeouts in state

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove state.Features() from requiring a context

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert wasm request handle Close func to be without context to
implement io.Closer interface. Add 5 second timeout. Add io.Closer
assertion in test.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove superfluous go lint vet directive

Signed-off-by: joshvanl <me@joshvanl.dev>

* Change Configuration Init function to take context

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates input binding interface to include a `Close() error` function. `Close`
blocks until all resources have been released and go routines have returned.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Change `Close() error` in input binding struct to `io.Closer` interface.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update go.mod files to point to dapr/dapr PR https://github.com/dapr/dapr/pull/5831

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/redis: watch closeCh to shutdown worker instead of init context.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/aws/snssqs + bindings/kubemq: ensure closeCh is caught so Close
correctly returns

Signed-off-by: joshvanl <me@joshvanl.dev>

* Close kubemq binding client on close. Ensure kafka consumer channel
cannot be closed more than once.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Tweaks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed cert tests

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* binding/mqtt3: add inline Background context instead of passing to
handleMessage

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: remove context from createSubscriberClientOptions

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: Remove `ResetConnection` func

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/kafka: Don't resubscribe if Subscribe is cancelled.

Signed-off-by: joshvanl <me@joshvanl.dev>

* binding/mqtt3: don't use context to control establishing connection

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: Fix linting errors

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2023-02-16 14:18:35 -08:00
ItalyPaleAle 8365df0f07 Fix flaky RabbitMQ certification tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-01-26 22:09:35 +00:00
Bernd Verst 8b3672c91e
Merge branch 'master' into feat/rabbitmq-tls 2022-12-19 15:31:47 -08:00
seachen e59dfd63c0 feature: add context to lock&pubsub API
Signed-off-by: seachen <seachen@tencent.com>
2022-12-14 19:37:24 +08:00
spike 6254982cd7 Raise error if protocol does not match connection string
Signed-off-by: spike <hello@spike.wiki>
2022-12-06 22:27:54 +08:00
spike 55a06c40c4 Optimize code according to lint
Signed-off-by: spike <hello@spike.wiki>
2022-11-20 15:33:53 +08:00
spike 6937857bf8 Change error message and comment
Signed-off-by: spike <hello@spike.wiki>
2022-11-13 15:34:01 +08:00
spike 1d9f3358e4 Add rabbitmq tls metadata
Signed-off-by: spike <hello@spike.wiki>
2022-11-13 15:17:50 +08:00
Bernd Verst 462e2faadc Upgrade linter and fix linter issues
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-11-11 13:19:57 -08:00
Taction 9870c5e33b
Add pubsub rabbitmq TTL support (#2093)
* Add pubsub rabbitmq TTL support

Signed-off-by: zhangchao <zchao9100@gmail.com>

* fix review: remove queue level ttl, both ttl is handled when publishing, to make message ttl has priority over queue level ttl

Signed-off-by: zhangchao <zchao9100@gmail.com>

* add rabbitmq ttl certificate test

Signed-off-by: zhangchao <zchao9100@gmail.com>

* fix lint

Signed-off-by: zhangchao <zchao9100@gmail.com>

* fix ttl certificate test precreate topic

Signed-off-by: zhangchao <zchao9100@gmail.com>

* tiny fix tests

Signed-off-by: zhangchao <zchao9100@gmail.com>

Signed-off-by: zhangchao <zchao9100@gmail.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-11-11 11:34:33 -08:00
Bernd Verst f137bc7373 rabbitmq: use ctx for publishing
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-10-10 14:37:21 -07:00
Mario Reggiori b0d267317d
Add individual connection arguments to pubsub.rabbitmq metadata (#2129)
Signed-off-by: Mario Reggiori <git@marioreggiori.com>

Signed-off-by: Mario Reggiori <git@marioreggiori.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2022-09-23 23:37:09 -07:00
Marcos Candeia 8964d8bf41
Define common metadata across components (#1994)
* Add base metadata for all components

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Fix lint issues on consul and mysql tests

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Pin candeia/dapr runtime version

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Add replace directive to use mcandeia/dapr on all go submodules

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Fixed runtime config instantiation

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
2022-08-25 10:10:53 -07:00
ItalyPaleAle b58464ad1c Make RabbitMQ retry more than once
Also updated tests to remove unused options and add more logging

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-27 22:31:31 +00:00
ItalyPaleAle 35034aa877 Disable built-in retries in pubsub components
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-27 20:36:15 +00:00
saber-wang d7eb3b7b27
[pubsub][rabbitmq]add support for binding multiple routingkeys at the same time (#1750)
* Add metadata property to configure BatchingMaxSize&batchingMaxMessages in Pulsar
Signed-off-by: saberwang <saberwang@hotmail.com>

* sort field
Signed-off-by: saberwang <saberwang@hotmail.com>

* [pubsub]fix unit test bug
Signed-off-by: saberwang <saberwang@hotmail.com>

* remove unrelated changes
Signed-off-by: saberwang <saberwang@hotmail.com>

* Delete hard coded Metadata
Signed-off-by: saberwang <saberwang@hotmail.com>

* remove  .history

Signed-off-by: saberwang <saberwang@hotmail.com>

* restore .gitignore

Signed-off-by: saberwang <saberwang@hotmail.com>

* Hard coding default values and adding 'BatchingMaxPublishDelay' metadata

Signed-off-by: saberwang <saberwang@hotmail.com>

* fix code format

Signed-off-by: saberwang <saberwang@hotmail.com>

* formatting code

Signed-off-by: saberwang <saberwang@hotmail.com>

* [pubsub][rabbitmq]Add support for binding multiple routingkeys at the same time

Signed-off-by: saberwang <saberwang@hotmail.com>

* add TestSubscribeBindRoutingKeys

Signed-off-by: saberwang <saberwang@hotmail.com>

* fix: Infof to Debugf

Signed-off-by: saberwang <saberwang@hotmail.com>

* Update pubsub/rabbitmq/rabbitmq_test.go

Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2022-06-10 16:34:15 -07:00
Alessandro (Ale) Segala 6f3234d5ac
Rabbitmq publisher confirm (#1762)
* Added `publishConfirm` option to RabbitMQ

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Changed to publisherConfirm per feedback

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-06-10 12:47:02 -07:00
Alessandro (Ale) Segala 704f4dd730
Add contexts to pubsub.Subscribe to allow early cancelation (#1756)
This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.

In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).

This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.

PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-02 15:06:36 -07:00
Taction 70a86e6d80
supports rabbitmq pubsub set exchangekind (#1520)
Signed-off-by: zhangchao <zchao9100@gmail.com>
2022-02-21 13:08:18 -08:00
Taction 7c7c977d52
Fix rabbitmq pubsub reconnect (#1471)
Signed-off-by: zhangchao <zchao9100@gmail.com>
2022-02-02 11:56:48 -08:00
Dmitry Shmulevich c7adb917f3
update license to Apache v2.0 (#1406) 2022-01-04 19:53:31 -08:00
Dmitry Shmulevich 7a631ef7aa
pub/sub rabbitmq: add publishing retries (#1382)
Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2021-12-17 15:59:38 -08:00
Dmitry Shmulevich 3fcf50c044
pub/sub rabbitmq: fix subscribe error (#1377)
Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com>
2021-12-14 08:22:35 -08:00
Dmitry Shmulevich ad433a27e1
add troubleshooting info (#1278)
add retries for pubsub subscribe
fix race conditions
2021-11-15 16:33:21 -08:00
Phil Kedy 77564a34c0
RabbitMQ certification: Fixing publish/subscribe race condition. (#1271)
* Fixing publish/subscribe race condition in RabbitMQ component.
Added amqpReady function to tests to wait for RabbitMQ readiness.
Tweaks to when subscribed is signaled in the test.

* Update go.mod(s)
2021-11-03 16:37:52 -07:00
Dmitry Shmulevich 3b1249cf87
reconnect rabbitmq on 'unexpected command received' error (#1198) 2021-10-14 21:19:31 -07:00
Taction ca8cf5c1fc
Add dead letter support in RabbitMQ pubsub (#883)
* Add dead letter support

* fix lint

* add maxLen and maxLenBytes support

* Let the DLX and DLQ be free form.

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long0dai@foxmail.com>
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: cvictory <shenglicao2@gmail.com>
2021-10-11 09:49:14 -07:00
Dmitry Shmulevich 4795f75270
properly close channel in rabbitmq (#1190) 2021-10-07 20:25:36 -07:00
Dmitry Shmulevich 494596be0e
fix 'durable' setting in RabbitMQ (#1183) 2021-10-01 17:27:36 -07:00
Long Dai 988fed05ab
ci: standard linter config (#1102)
* ci: standard linter config

Signed-off-by: Long <long.dai@intel.com>

* Update utils.go

* fix pulsar issue

Signed-off-by: Long <long.dai@intel.com>

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-21 15:21:59 -07:00
Phil Kedy 8e6b36ecdf
Importing the retry and config packages from kit instead of internal (#940)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-22 09:38:55 -07:00
Taction 14870cb7bc
Add backoff for rabbitmq PubSub component (#862)
* Add backoff for rabbitmq

* fix lint

* fix lint

* remove backOffEnable config

* Delete backoff enable metadata

* Add a default config setting max retry to 0

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-06-10 00:19:46 -07:00
Taction 7512c77a28
Rabbitmq pubsub component stop has not be well handled (#860)
* fix pubsub rabbitmq stop

* fix lint

* Add read lock while reading stopped

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-09 23:23:06 -07:00
Phil Kedy 688df9bda4
Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807) 2021-04-19 12:06:10 -07:00
Joni Collinge 9fd461b255 Merge branch 'master' into jjcollinge/pubsub-context 2021-03-23 17:45:05 +00:00
Phil Kedy a1a0ad2c25
Adding RabbitMQ to the conformance tests (#739)
* Initial pass at adding RabbitMQ to the conformance tests

* Fixing unit tests
2021-03-04 11:56:35 -08:00
Joni Collinge c054ae6ada Merge branch 'master' into jjcollinge/pubsub-context 2021-02-25 16:15:06 +00:00