* Propagates contexts to callers where appropriate.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Updates units tests with new func signature
Signed-off-by: joshvanl <me@joshvanl.dev>
* Fix linting errors
Signed-off-by: joshvanl <me@joshvanl.dev>
* Add atomic gate to alicloud rocketmq close channel.
Signed-off-by: joshvanl <me@joshvanl.dev>
* bindings/aws/kinesis use a separate ctx variable name
Signed-off-by: joshvanl <me@joshvanl.dev>
* binding/kafka: use atomic to prevent closing the channel twice
Signed-off-by: joshvanl <me@joshvanl.dev>
* bindings/mqtt3: use atomic bool to prevent close channel being closed multiple times
Signed-off-by: joshvanl <me@joshvanl.dev>
* bindings/mqtt3: use Background context for handle operations:w
Signed-off-by: joshvanl <me@joshvanl.dev>
* state/cocroachdb: add context to Ping()
Signed-off-by: joshvanl <me@joshvanl.dev>
* bindings/postgres: add comment explaining use of context.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Adds comment header to health/pinger.go
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/aws/snssqs: add waitgroup to wait for all go routines to finish
and block on Close(). Shuts down the subscription if there are no topic
handlers.
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/mqtt3: add atomic bool to prevent multiple channel closes. Add
wait group to block close on all goroutines to finish.
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/rabbitmq: fixes race conditions, uses atomic to prevent multiple
closes, add wait group to block close on all goroutines
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/redis: revert ctx passed when it could be cancelled. Add wait
group wait when closing.
Signed-off-by: joshvanl <me@joshvanl.dev>
* state/postges: pass context in init, and wait group on close.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Update all `Ping()` to `PingContext()` where possible.
Signed-off-by: joshvanl <me@joshvanl.dev>
* state/in-memory: add atomic bool to prevent closing channel multiple
times. Add wait group to block on close()
Signed-off-by: joshvanl <me@joshvanl.dev>
* state/mysql: don't use same ctx variable name
Signed-off-by: joshvanl <me@joshvanl.dev>
* Pass correct loop context to redis go routines
Signed-off-by: joshvanl <me@joshvanl.dev>
* Rename context when creating timeouts in state
Signed-off-by: joshvanl <me@joshvanl.dev>
* Remove state.Features() from requiring a context
Signed-off-by: joshvanl <me@joshvanl.dev>
* Revert wasm request handle Close func to be without context to
implement io.Closer interface. Add 5 second timeout. Add io.Closer
assertion in test.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Remove superfluous go lint vet directive
Signed-off-by: joshvanl <me@joshvanl.dev>
* Change Configuration Init function to take context
Signed-off-by: joshvanl <me@joshvanl.dev>
* Updates input binding interface to include a `Close() error` function. `Close`
blocks until all resources have been released and go routines have returned.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Change `Close() error` in input binding struct to `io.Closer` interface.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Update go.mod files to point to dapr/dapr PR https://github.com/dapr/dapr/pull/5831
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/redis: watch closeCh to shutdown worker instead of init context.
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/aws/snssqs + bindings/kubemq: ensure closeCh is caught so Close
correctly returns
Signed-off-by: joshvanl <me@joshvanl.dev>
* Close kubemq binding client on close. Ensure kafka consumer channel
cannot be closed more than once.
Signed-off-by: joshvanl <me@joshvanl.dev>
* Tweaks
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fixed cert tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* binding/mqtt3: add inline Background context instead of passing to
handleMessage
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/mqtt3: remove context from createSubscriberClientOptions
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/mqtt3: Remove `ResetConnection` func
Signed-off-by: joshvanl <me@joshvanl.dev>
* pubsub/kafka: Don't resubscribe if Subscribe is cancelled.
Signed-off-by: joshvanl <me@joshvanl.dev>
* binding/mqtt3: don't use context to control establishing connection
Signed-off-by: joshvanl <me@joshvanl.dev>
* bindings/mqtt3: Fix linting errors
Signed-off-by: joshvanl <me@joshvanl.dev>
---------
Signed-off-by: joshvanl <me@joshvanl.dev>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
* Fixed context leak
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Finalize messages in background
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* More tweaks
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Moved a lot more code to the shared implementation package
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fixed:
- Fixed: if number of messages received in bulk is greater than maxConcurrentHandlers, a deadlock happens
- Fixed: maxRetriableErrorsPerSec was not working
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Some refactorings
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Simplifications
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Avoid checking if topic/queue exists every time we send a message
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ensuring locks are renewed for all 3 components
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Updated ASB SDK
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Improvements to closing
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Improvements to closing
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Changed context propagation
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Debug logs in cert tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Initial delay before renewing locks
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* That last change was actually unnecessary
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Tweaks
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* More tweaks
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
---------
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Add base metadata for all components
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
* Fix lint issues on consul and mysql tests
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
* Pin candeia/dapr runtime version
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
* Add replace directive to use mcandeia/dapr on all go submodules
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
* Fixed runtime config instantiation
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
Signed-off-by: Marcos Candeia <marrcooos@gmail.com>
* Go 1.19 support and linter fixes
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Update workflows for Go1.19 and new linter version
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Remove unnecessary space in nolint directive
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* disable additional linters which aren't used because of Go generics
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* enable gosec linter again
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Update bindings/zeebe/command/publish_message_test.go
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Update bindings/output_binding.go
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Use prepared statement for mysql table creation
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Ping is not ping
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* c'mon linter
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Fix MySQL gosec issue
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* revert mysql to be fixed later
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Moved ASB subscription code to a shared package
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Moved ASBQ binding to use the shared ASB implementation
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Updating Azure Service Bus SDK version
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Make Service Bus attempt to reconnect forever in case of issues
With exponential back-off configurable between min and max time. Fixes#1612
Also includes fixes:
- Binding: make sure it actually retries to connect forever
- Binding: add delay (exponential backoff) before reconnecting
- PubSub: better handling of failures such as topics disabled or other non-connection issues
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added warning for deprecated metadata options
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.
In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).
This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.
PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Tweaks for ASB pubsub:
1. Do not attempt to reconnect when context was canceled
2. Message finalization uses a background context and not the running ctx, in case the app is shutting down
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* ASB: abandoned messages shouldn't be retried right away
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ported fixes to servicebusqueues binding too
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Use a different approach to limit the number of messages processed in case of error
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added maxRetriableErrorsPerSec to servicebusqueues binding too
Incl other improvements
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Do not reconnect to ASB every time we retrieve a message.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* ⚙️
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Azure SDKs that are on track2
Includes some minor refactoring of auth code
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Service Bus components to track2 SDK
Co-authored-by: halspang <halspang@microsoft.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Initial Certification test for eventhubs binding [incomplete] (#1670)
* certification test for eventhubs binding
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* modified go.mod and go.sum
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Add connection string testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* iothub testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* address feedback and run test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Install Azure CLI IOT hub extension
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* make modtidy-all
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* covering all eventhubs test cases
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* dependency changes after go modtidy-all
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Use revive instead of golint (#1685)
Signed-off-by: pigletfly <wangbing.adam@gmail.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated to Go 1.18 (#1697)
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Removed go.work
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 💄 & 🧹
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Add metadata property to configure Batching in Pulsar (#1707)
* Add metadata property to configure BatchingMaxSize&batchingMaxMessages in Pulsar
Signed-off-by: saberwang <saberwang@hotmail.com>
* sort field
Signed-off-by: saberwang <saberwang@hotmail.com>
* [pubsub]fix unit test bug
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove unrelated changes
Signed-off-by: saberwang <saberwang@hotmail.com>
* Delete hard coded Metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove .history
Signed-off-by: saberwang <saberwang@hotmail.com>
* restore .gitignore
Signed-off-by: saberwang <saberwang@hotmail.com>
* Hard coding default values and adding 'BatchingMaxPublishDelay' metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* fix code format
Signed-off-by: saberwang <saberwang@hotmail.com>
* formatting code
Signed-off-by: saberwang <saberwang@hotmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* This test can't work with track2 SDKs
The methods to create a message with a body are not exported
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 🧹
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* There's such thing as too much logging
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Refactored subscription.go
This greatly simplifies certain parts of the code, reducing the number of goroutines and likely improving performance.
Performance for end-users improves too as there's no need anymore to pause for 2 seconds every time that we reach `maxActiveMessages`.
Additionally, with this change the config options `prefetchCount` and `maxActiveMessagesRecoveryInSec` are removed as unnecessary anymore.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fixed pubsub tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* These packages should have never been upgraded
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ensuring we don't fetch 1 message more than max active
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Adding configurable timeout for servicebusqueues operations
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Persistent connection for invoking SB queues
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Reverted Event Hub SDK update
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Revert "Reverted Event Hub SDK update"
This reverts commit 212220aba0.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fix Azure deploy for users with a first.last email
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added some sleep that should help reduce flakiness in eventhubs binding cert test
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Changed servicebusqueue cert test
In case of a failure in the handler (ie. users' code), the message should be correctly re-enqueued, which means that messages will be re-delivered later and won't be in order.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: halspang <halspang@microsoft.com>
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Wang Bing <wangbing.adam@gmail.com>
Co-authored-by: saber-wang <45062099+saber-wang@users.noreply.github.com>
Alessandro (Ale) Segala
Arturo Trenard
Bernd Verst
Josh van Leeuwen
Deepanshu Agarwal
Joni Collinge
Mukundan Sundararajan
seachen
Bernd Verst
Shubham Sharma
Marcos Candeia
addjuarez
Wang Bing