This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.
In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).
This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.
PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Azure SDKs that are on track2
Includes some minor refactoring of auth code
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Service Bus components to track2 SDK
Co-authored-by: halspang <halspang@microsoft.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Initial Certification test for eventhubs binding [incomplete] (#1670)
* certification test for eventhubs binding
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* modified go.mod and go.sum
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Add connection string testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* iothub testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* address feedback and run test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Install Azure CLI IOT hub extension
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* make modtidy-all
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* covering all eventhubs test cases
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* dependency changes after go modtidy-all
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Use revive instead of golint (#1685)
Signed-off-by: pigletfly <wangbing.adam@gmail.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated to Go 1.18 (#1697)
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Removed go.work
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 💄 & 🧹
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Add metadata property to configure Batching in Pulsar (#1707)
* Add metadata property to configure BatchingMaxSize&batchingMaxMessages in Pulsar
Signed-off-by: saberwang <saberwang@hotmail.com>
* sort field
Signed-off-by: saberwang <saberwang@hotmail.com>
* [pubsub]fix unit test bug
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove unrelated changes
Signed-off-by: saberwang <saberwang@hotmail.com>
* Delete hard coded Metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove .history
Signed-off-by: saberwang <saberwang@hotmail.com>
* restore .gitignore
Signed-off-by: saberwang <saberwang@hotmail.com>
* Hard coding default values and adding 'BatchingMaxPublishDelay' metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* fix code format
Signed-off-by: saberwang <saberwang@hotmail.com>
* formatting code
Signed-off-by: saberwang <saberwang@hotmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* This test can't work with track2 SDKs
The methods to create a message with a body are not exported
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 🧹
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* There's such thing as too much logging
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Refactored subscription.go
This greatly simplifies certain parts of the code, reducing the number of goroutines and likely improving performance.
Performance for end-users improves too as there's no need anymore to pause for 2 seconds every time that we reach `maxActiveMessages`.
Additionally, with this change the config options `prefetchCount` and `maxActiveMessagesRecoveryInSec` are removed as unnecessary anymore.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fixed pubsub tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* These packages should have never been upgraded
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ensuring we don't fetch 1 message more than max active
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Adding configurable timeout for servicebusqueues operations
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Persistent connection for invoking SB queues
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Reverted Event Hub SDK update
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Revert "Reverted Event Hub SDK update"
This reverts commit 212220aba0.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fix Azure deploy for users with a first.last email
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added some sleep that should help reduce flakiness in eventhubs binding cert test
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Changed servicebusqueue cert test
In case of a failure in the handler (ie. users' code), the message should be correctly re-enqueued, which means that messages will be re-delivered later and won't be in order.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: halspang <halspang@microsoft.com>
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Wang Bing <wangbing.adam@gmail.com>
Co-authored-by: saber-wang <45062099+saber-wang@users.noreply.github.com>
* Add conformance test for EventHubs bindings
* Update EventHubs pubsub to pass conformance tests
- Add conformance tests for Azure EventHubs PubSub component
- Add retry & backoff handling on subscriber handling error to EventHubs
PubSub component for conformance tests.
- Add cancellation context to Azure EventHubs and update Close() to
invoke cancel prior to closing the hub, which cleans up both senders
and receivers.
- Update azure-event-hubs-go dependency to currently supported v3 with
IoT Hub SystemProperties.
- Write Event.SystemProperties to ReadResponse.Metadata for EventHubs
input binding.
- Write Event.SystemProperties to NewMessage.Metadata for EventHubs
pubsub.
Shubham Sharma
Shivam Kumar
Alessandro (Ale) Segala
Bernd Verst
Dmitry Shmulevich
Mukundan Sundararajan
Simon Leet
Long Dai
Simon Leet
Phil Kedy
Joni Collinge
Yaron Schneider
Artur Souza
halspang
Young Bu Park
Abhishek Gupta