Commit Graph

49 Commits

Author SHA1 Message Date
Byron Ruth 36a055ebd8
[pubsub/jetstream] Add missing concurrencyMode support for queue-based handler (#3232)
Signed-off-by: Byron Ruth <byron@nats.io>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
2023-11-28 20:24:34 -08:00
Bernd Verst 9095b0e7e7
Switch to Go 1.21, Updates linter, updates workflows, adds sarama 1.42.1 (#3251)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-11-28 18:13:09 -08:00
Byron Ruth dd8d2ba185
[pubsub/jetstream] Add support for concurrencyMode (#3222)
Signed-off-by: Byron Ruth <byron@nats.io>
2023-11-13 20:33:48 +00:00
Alessandro (Ale) Segala f4e73b0e65
Update dapr/kit (#3205)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-11-02 16:20:56 -07:00
Bernd Verst 247b08a70c
Jetstream pubsub: Use AckWait time for Retry delay (#3102)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-08-24 17:48:30 -07:00
Alessandro (Ale) Segala ec05809ee6
[Metadata] Update validator and some other fixes (#2984)
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-07-17 22:17:22 +00:00
Byron Ruth 54673a0f58
Add JetStream pubsub unit tests (#2845)
Signed-off-by: Byron Ruth <byron@nats.io>
2023-05-17 16:56:08 -07:00
Byron Ruth d2cad27b3f
[jetstream] Fix setting omitted QueueGroupName on ConsumerConfig (#2835)
Signed-off-by: Byron Ruth <byron@nats.io>
2023-05-16 17:56:18 -07:00
Bernd Verst 2b89d78a2d
Refactor Metadata Parsing of all PubSub Components (#2759)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-04-09 18:21:56 +00:00
ItalyPaleAle c20ab76972 Merge branch 'master' of https://github.com/dapr/components-contrib into merge-release-1.10-master
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2023-03-15 19:25:12 +00:00
Bernd Verst b4cdc7a3ca
Restore Kafka CA Cert Auth without Password support, fix Jetstream pubsub issue (#2644)
Signed-off-by: Bernd Verst <github@bernd.dev>
2023-03-07 22:47:35 -08:00
Josh van Leeuwen 569c6be070
PubSub: Wait Group and context catching (#2500)
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Bernd Verst <github@bernd.dev>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <github@bernd.dev>
2023-03-03 17:44:26 +00:00
Josh van Leeuwen d098e38d6a
Propagate context from caller to appropriate places in the code (#2474)
* Propagates contexts to callers where appropriate.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates units tests with new func signature

Signed-off-by: joshvanl <me@joshvanl.dev>

* Fix linting errors

Signed-off-by: joshvanl <me@joshvanl.dev>

* Add atomic gate to alicloud rocketmq close channel.

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/aws/kinesis use a separate ctx variable name

Signed-off-by: joshvanl <me@joshvanl.dev>

* binding/kafka: use atomic to prevent closing the channel twice

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: use atomic bool to prevent close channel being closed multiple times

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: use Background context for handle operations:w

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/cocroachdb: add context to Ping()

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/postgres: add comment explaining use of context.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Adds comment header to health/pinger.go

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/aws/snssqs: add waitgroup to wait for all go routines to finish
and block on Close(). Shuts down the subscription if there are no topic
handlers.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: add atomic bool to prevent multiple channel closes. Add
wait group to block close on all goroutines to finish.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/rabbitmq: fixes race conditions, uses atomic to prevent multiple
closes, add wait group to block close on all goroutines

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/redis: revert ctx passed when it could be cancelled. Add wait
group wait when closing.

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/postges: pass context in init, and wait group on close.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update all `Ping()` to `PingContext()` where possible.

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/in-memory: add atomic bool to prevent closing channel multiple
times. Add wait group to block on close()

Signed-off-by: joshvanl <me@joshvanl.dev>

* state/mysql: don't use same ctx variable name

Signed-off-by: joshvanl <me@joshvanl.dev>

* Pass correct loop context to redis go routines

Signed-off-by: joshvanl <me@joshvanl.dev>

* Rename context when creating timeouts in state

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove state.Features() from requiring a context

Signed-off-by: joshvanl <me@joshvanl.dev>

* Revert wasm request handle Close func to be without context to
implement io.Closer interface. Add 5 second timeout. Add io.Closer
assertion in test.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Remove superfluous go lint vet directive

Signed-off-by: joshvanl <me@joshvanl.dev>

* Change Configuration Init function to take context

Signed-off-by: joshvanl <me@joshvanl.dev>

* Updates input binding interface to include a `Close() error` function. `Close`
blocks until all resources have been released and go routines have returned.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Change `Close() error` in input binding struct to `io.Closer` interface.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Update go.mod files to point to dapr/dapr PR https://github.com/dapr/dapr/pull/5831

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/redis: watch closeCh to shutdown worker instead of init context.

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/aws/snssqs + bindings/kubemq: ensure closeCh is caught so Close
correctly returns

Signed-off-by: joshvanl <me@joshvanl.dev>

* Close kubemq binding client on close. Ensure kafka consumer channel
cannot be closed more than once.

Signed-off-by: joshvanl <me@joshvanl.dev>

* Tweaks

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* Fixed cert tests

Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>

* binding/mqtt3: add inline Background context instead of passing to
handleMessage

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: remove context from createSubscriberClientOptions

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/mqtt3: Remove `ResetConnection` func

Signed-off-by: joshvanl <me@joshvanl.dev>

* pubsub/kafka: Don't resubscribe if Subscribe is cancelled.

Signed-off-by: joshvanl <me@joshvanl.dev>

* binding/mqtt3: don't use context to control establishing connection

Signed-off-by: joshvanl <me@joshvanl.dev>

* bindings/mqtt3: Fix linting errors

Signed-off-by: joshvanl <me@joshvanl.dev>

---------

Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2023-02-16 14:18:35 -08:00
Tomasz Pietrek 766b2df2fe
Fix heartbeat typo (#2430)
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
Co-authored-by: Bernd Verst <github@bernd.dev>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2023-01-12 11:21:05 -08:00
Tomasz Pietrek 97ad52c692 Make domain and prefix setup cleaner
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-01-04 10:48:32 +01:00
Tomasz Pietrek 94894c465b Add domain and apiPrefix for JetStream
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2023-01-04 09:18:35 +01:00
seachen 1ea2cff9fd feature: add context to lock&pubsub API
Signed-off-by: seachen <seachen@tencent.com>
2022-12-15 09:53:32 +08:00
Tomasz Pietrek 72bd117e7e
Rework JetStream delivery policy (#2315)
* Rework JetStream delivery policy

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>

* Add missing deliver policies to error

Co-authored-by: Piotr Piotrowski <piotrpiotrowski94@gmail.com>
Signed-off-by: Tomasz Pietrek <melgaer@gmail.com>
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>

* Rearrange imports

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
Signed-off-by: Tomasz Pietrek <melgaer@gmail.com>
Co-authored-by: Piotr Piotrowski <piotrpiotrowski94@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2022-12-14 15:56:07 -08:00
seachen e59dfd63c0 feature: add context to lock&pubsub API
Signed-off-by: seachen <seachen@tencent.com>
2022-12-14 19:37:24 +08:00
Tomasz Pietrek 9df618959e Fix formatting
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-12-05 17:56:02 +01:00
Tomasz Pietrek d2e3a7e533 Test ack policies metadata
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-12-05 17:47:35 +01:00
Tomasz Pietrek 0e80570949 Fix typo
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-12-05 15:57:14 +01:00
Tomasz Pietrek 3a8a9a0ef6 Fix error handling in Ack
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-12-05 14:33:12 +01:00
Tomasz Pietrek 0463c2cf8b Adopt handler to work properly with each ack policy in NATS
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-12-05 14:19:13 +01:00
Tomasz Pietrek 7ec2b8608f Add ack policy to NATS pub/sub
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-12-05 14:15:00 +01:00
Tomasz Pietrek 86cf34ef8f
Add missing filter subject in Jetstream pubsub (#2312)
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>

Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2022-11-30 10:46:15 -08:00
Armin Schlegel 103624468f
jetstream pubsub now supports token based auth (#2295)
Signed-off-by: Armin Schlegel <armin.schlegel@gmx.de>

Signed-off-by: Armin Schlegel <armin.schlegel@gmx.de>
2022-11-17 23:28:18 -08:00
Bernd Verst 462e2faadc Upgrade linter and fix linter issues
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-11-11 13:19:57 -08:00
Bernd Verst e5a2266fa8 Address lint issues
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-11-10 12:23:04 -08:00
Bernd Verst 9a8bcc5fa5
Merge branch 'master' into explicitly-add-consumer
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-11-10 12:05:51 -08:00
Nick ce87730a73
pubsub jetstream: set Nats-Msg-Id for message deduplication (#2200) (#2202)
Signed-off-by: NickLarsenNZ <nick@aptiv.co.nz>

Signed-off-by: NickLarsenNZ <nick@aptiv.co.nz>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-11-08 19:49:34 -08:00
Tomasz Pietrek 54cb93c786 Cleanup the opts
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-10-31 21:07:36 +01:00
Tomasz Pietrek 83d7d09ab8 Explicitly use AddConsumer
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-10-31 20:11:17 +01:00
Tomasz Pietrek e04a44895f Cleanup debugs 2022-10-31 16:43:38 +01:00
Tomasz Pietrek efe706a41d wip on binding by name 2022-10-31 12:52:38 +01:00
Bernd Verst 47a46673cd PubSub: Fall back to 'consumerID' when group not present
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-10-27 14:08:37 -07:00
Tomasz Pietrek 954164216e Fix another lint error
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-10-12 11:50:49 +02:00
Tomasz Pietrek 7a3746b33a Fix lint errors
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-10-12 11:44:00 +02:00
Tomasz Pietrek 84742aa1a8 Add more JetStream options
Signed-off-by: Tomasz Pietrek <tomasz@nats.io>
2022-10-12 11:23:03 +02:00
Marcos Candeia 8964d8bf41
Define common metadata across components (#1994)
* Add base metadata for all components

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Fix lint issues on consul and mysql tests

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Pin candeia/dapr runtime version

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Add replace directive to use mcandeia/dapr on all go submodules

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

* Fixed runtime config instantiation

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>

Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
2022-08-25 10:10:53 -07:00
Bernd Verst e87cd5e4cb
Go 1.19 support and linter fixes (#1975)
* Go 1.19 support and linter fixes

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update workflows for Go1.19 and new linter version

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Remove unnecessary space in nolint directive

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* disable additional linters which aren't used because of Go generics

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* enable gosec linter again

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update bindings/zeebe/command/publish_message_test.go

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Update bindings/output_binding.go

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Use prepared statement for mysql table creation

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Ping is not ping

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* c'mon linter

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* Fix MySQL gosec issue

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

* revert mysql to be fixed later

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>

Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-08-18 00:45:23 -07:00
Nick 248f7f458d
feat(pubsub/jetstream): allow tls client authentication (#1924)
Signed-off-by: NickLarsenNZ <nick@aptiv.co.nz>

Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2022-08-04 16:29:53 -07:00
ItalyPaleAle 35034aa877 Disable built-in retries in pubsub components
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-27 20:36:15 +00:00
Alessandro (Ale) Segala 704f4dd730
Add contexts to pubsub.Subscribe to allow early cancelation (#1756)
This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.

In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).

This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.

PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
2022-06-02 15:06:36 -07:00
Gehhilfe 6ecc3b6f7b
Add support for nats jetstream wildcard subscriptions (#1465)
* Add support for nats jetstream wildcard subscriptions

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* Add support for nats jetstream wildcard subscriptions

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2022-02-15 09:46:33 -08:00
Gehhilfe b835eb4e1e
pubsub jetstream: add nkey based authentication (#1441)
* pubsub jetstream: add nkey based authentication

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* fix godot: comment should end in a period

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* go mod tidy

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* fix: go mod tidy

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* pubsub jetstream: add e2e test

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* Cleanup not needed files and move jwt

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2022-01-25 21:11:50 -08:00
Dmitry Shmulevich c7adb917f3
update license to Apache v2.0 (#1406) 2022-01-04 19:53:31 -08:00
Long Dai 988fed05ab
ci: standard linter config (#1102)
* ci: standard linter config

Signed-off-by: Long <long.dai@intel.com>

* Update utils.go

* fix pulsar issue

Signed-off-by: Long <long.dai@intel.com>

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-21 15:21:59 -07:00
Jaime Piña f4a5722806
Add JetStream support (#1101)
* Add JetStream support

* Add conformance tests

* Fix lint

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-09-01 11:50:16 -07:00