* improve log of kafkaf components cert test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add close() method to kafka binding component to fix the consumer rebalance bug
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Complete Eventhubs bindings certification test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* check if single test point runs all good
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* check if single test point runs all good
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* increase the sleep time to fix timeout
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* increase the sleep time to fix timeout
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* increase sleep for all other testpoints too
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test behavior of single partition test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* delete eventhub and container afer run
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* delete eventhub and container afer run
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* delete eventhub and container afer run
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test reconnection in one test point only
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test reconnection in one test point only
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test reconnection in one test point only
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test reconnection in one test point only
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test reconnection in one test point only
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* delete the container after every test run for cleanup
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* run service principal test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Go mod tidy-all
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* revert back changes to last clean state
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* add unique consumer id per component yaml
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* remove consumer id from singlepartition test component yaml
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* remove repetitive test case, test service principal case
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* remove repetitive test case, test service principal case
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test iothub
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* test iothub
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* run service principal test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* run service principal test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* run service principal test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* run service principal test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* add connection string to eventhub service principal yaml
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* use a particular partition
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* enable iothub test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* enable iothub test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* enable iothub test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* final cleanup and test run
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* final cleanup and test run
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* final cleanup and test run
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* add test plan readme
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* rollback the bicep change that is not needed
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* make sure through test that a specific partition receives specific data
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* clean the container between two partiton assertions
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* put back partition id in yaml
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* add same level of partition testing for both partitions
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* address feedback
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Moved ASB subscription code to a shared package
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Moved ASBQ binding to use the shared ASB implementation
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Updating Azure Service Bus SDK version
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.
In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).
This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.
PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added test for multiple pubsub handlers
This patch adds conformance tests for PubSub with multiple handlers, each listening on a different topic.
It's meant to catch issues such as #1709
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added missing go.mod for storagequeues binding test
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* add certification test for kafka binding, the first version is copied from kafka pubsub to verify github flow
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* only run kafka pubsub and binding cert test in development
* fix typo
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* run only kafka bindings
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add certification test for kafka binding: with and without retry
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add config.yaml file back for kafka binding certification test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* rollback to run all cert test before PR
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* update dependencies for kafka pubsub certification test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* update dependencies for kafka pubsub certification test(again)
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add network interruption test back
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* Added certification tests for Azure Storage Queue Bindings
Signed-off-by: Amulya Varote <amulyavarote@microsoft.com>
* Changes based on the review comments
Signed-off-by: Amulya Varote <amulyavarote@microsoft.com>
* Added ttl test case to fetch the unexpired messages
Signed-off-by: Amulya Varote <amulyavarote@microsoft.com>
* Tweaks for ASB pubsub:
1. Do not attempt to reconnect when context was canceled
2. Message finalization uses a background context and not the running ctx, in case the app is shutting down
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* ASB: abandoned messages shouldn't be retried right away
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ported fixes to servicebusqueues binding too
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Use a different approach to limit the number of messages processed in case of error
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added maxRetriableErrorsPerSec to servicebusqueues binding too
Incl other improvements
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Do not reconnect to ASB every time we retrieve a message.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* ⚙️
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Azure SDKs that are on track2
Includes some minor refactoring of auth code
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Service Bus components to track2 SDK
Co-authored-by: halspang <halspang@microsoft.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Initial Certification test for eventhubs binding [incomplete] (#1670)
* certification test for eventhubs binding
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* modified go.mod and go.sum
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Add connection string testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* iothub testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* address feedback and run test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Install Azure CLI IOT hub extension
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* make modtidy-all
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* covering all eventhubs test cases
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* dependency changes after go modtidy-all
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Use revive instead of golint (#1685)
Signed-off-by: pigletfly <wangbing.adam@gmail.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated to Go 1.18 (#1697)
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Removed go.work
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 💄 & 🧹
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Add metadata property to configure Batching in Pulsar (#1707)
* Add metadata property to configure BatchingMaxSize&batchingMaxMessages in Pulsar
Signed-off-by: saberwang <saberwang@hotmail.com>
* sort field
Signed-off-by: saberwang <saberwang@hotmail.com>
* [pubsub]fix unit test bug
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove unrelated changes
Signed-off-by: saberwang <saberwang@hotmail.com>
* Delete hard coded Metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove .history
Signed-off-by: saberwang <saberwang@hotmail.com>
* restore .gitignore
Signed-off-by: saberwang <saberwang@hotmail.com>
* Hard coding default values and adding 'BatchingMaxPublishDelay' metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* fix code format
Signed-off-by: saberwang <saberwang@hotmail.com>
* formatting code
Signed-off-by: saberwang <saberwang@hotmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* This test can't work with track2 SDKs
The methods to create a message with a body are not exported
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 🧹
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* There's such thing as too much logging
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Refactored subscription.go
This greatly simplifies certain parts of the code, reducing the number of goroutines and likely improving performance.
Performance for end-users improves too as there's no need anymore to pause for 2 seconds every time that we reach `maxActiveMessages`.
Additionally, with this change the config options `prefetchCount` and `maxActiveMessagesRecoveryInSec` are removed as unnecessary anymore.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fixed pubsub tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* These packages should have never been upgraded
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ensuring we don't fetch 1 message more than max active
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Adding configurable timeout for servicebusqueues operations
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Persistent connection for invoking SB queues
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Reverted Event Hub SDK update
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Revert "Reverted Event Hub SDK update"
This reverts commit 212220aba0.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fix Azure deploy for users with a first.last email
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added some sleep that should help reduce flakiness in eventhubs binding cert test
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Changed servicebusqueue cert test
In case of a failure in the handler (ie. users' code), the message should be correctly re-enqueued, which means that messages will be re-delivered later and won't be in order.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: halspang <halspang@microsoft.com>
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Wang Bing <wangbing.adam@gmail.com>
Co-authored-by: saber-wang <45062099+saber-wang@users.noreply.github.com>
* refactory kafka pubsub code to extract common kafka code for reuse
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint;add unit test for subscribeAdapter
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* move topics filed from internal kafak struct to pubsub kafka struct, since in input binding the topics will confiured in metadata
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* reuse internal kafka code for bindings
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add redis standalone_test back which is delete by mistaken
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* small code improvement to trigger test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add license headers
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* try to set disbaleTls to true to verify the kafka connection fail
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* don't enable consum retry in kafka binding component;if authenticaion is disabled, need not set TLSDisable at the same time;
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
Co-authored-by: Loong Dai <long.dai@intel.com>
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Removed go.work
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* install azure-iot-cli-extension if not found
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* reduce the number of messages from 100 to 10, to make the test faster
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* First iteration for the eventhubs certifications tests
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Fixing typos, adding more description and publishing with multiple partitions
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Moved the secret reference in components definition to secretstore.local.env
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* moving to latest dapr/dapr github.com/dapr/dapr@v1.7.0-rc.2
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updating go.mod
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* changes in infra and workflow for cert eventhubs
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* typo fix in local secret store type
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* correcting the typo in policyName
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* bug fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* publish to a topic with no subscriber fix
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete azure eventhub that got created as part of test run
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updated readme
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* removed dummy sidecar and leveraged connectionstring at namespace level for multi topic operations
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* go mod changes and nit fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* fixing iot device issue
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* NIT : CR fixes
* Enabled iothub test and a bit of code refactoring
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete the checkpoint container for the temp eventhub
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Revert "NIT : CR fixes"
This reverts commit 260f5ed686.
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* First iteration for the eventhubs certifications tests
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Fixing typos, adding more description and publishing with multiple partitions
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Moved the secret reference in components definition to secretstore.local.env
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* moving to latest dapr/dapr github.com/dapr/dapr@v1.7.0-rc.2
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updating go.mod
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* changes in infra and workflow for cert eventhubs
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* typo fix in local secret store type
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* correcting the typo in policyName
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* bug fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* publish to a topic with no subscriber fix
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete azure eventhub that got created as part of test run
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updated readme
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* removed dummy sidecar and leveraged connectionstring at namespace level for multi topic operations
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* go mod changes and nit fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* fixing iot device issue
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* NIT : CR fixes
* Enabled iothub test and a bit of code refactoring
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete the checkpoint container for the temp eventhub
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Revert "NIT : CR fixes"
This reverts commit 260f5ed686.
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Fix eventhub pubsub certification test infra provisioning
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Fixing the scoping issue for role assignment on eh
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
This is fixed by upgrading the upstream SDK
Also added a missing handling of throttling in the Delete method
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Upgrade nacos version
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
* Run go mod tidy all
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
* Add MaxSize to settings
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
* Add support for nats jetstream kv state store
Add tests for jetstream state store
Update go mod
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Fix lint errors
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Mod tidy
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Mod tidy all
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Mod tidy all
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
* pubsub jetstream: add nkey based authentication
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* fix godot: comment should end in a period
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* go mod tidy
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* fix: go mod tidy
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* pubsub jetstream: add e2e test
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Cleanup not needed files and move jwt
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
* conformance tests
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* latest localstack. snssqs to conformance list
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
Co-authored-by: Mukundan Sundararajan <musundar@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600
Support Oauth2 authentication for Kafka
- Utilize the SASL OAUTHBEARER mechanism to support
the Oauth2 client_credentials flow for Kafka
authentication
- Deprecate `authRequired` field and introduce `authType`
to support varied authentication mechanisms
- Add a metadata upgrade mechanism to support backwards
compatability
- Recommend broker specific scopes to guard against a
compromised broker replaying a token
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* kafka test - use volumes instead of root
CI test fails due to losing data written to container
root
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up volumes
- When bringing down the docker-compose context, cleanup
volumes
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up stale comment
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Resume config defaults
- Return some Kafka config to default values to lower config
footprint and stabilize flaky CI runs
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* - Introduce `fmt.Sprintf` where applicable
- Add conformance test for cassandra component.
Signed-off-by: mbimbij <joseph.mbimbi@gmail.com>
* Start cassandra via docker-compose for its conformance test
Signed-off-by: mbimbij <joseph.mbimbi@gmail.com>
* Apply `go fmt` on `tests/conformance/common.go` as part of PR correction
Signed-off-by: mbimbij <joseph.mbimbi@gmail.com>
* Add certification tests for ASB Queue binding
This commit adds certification tests for the Azure Service Bus Queue
Input/Output binding. This utilizes the new certification framework
and performs a series of tests as described in the test plan readme.
The general purpose of these tests is to serve as an integration test
and as such requires an actual Azure Service Bus connection and a
Dapr sidecar.
https://github.com/dapr/components-contrib/issues/957
* Add to github workflow and fix formatting
* Fix dependencies
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* Authentication for new Azure SDK
* Updated keyvault to use new Azure SDK
* 🙈
* Update authentication/azure/auth.go
* Update authentication/azure/auth.go
* Update authentication/azure/auth.go
* Reintroduce changes from PR 1132 without updating ASB
* Lint 💄
* Marking contexts as TODO as they'll need a timeout
As per conversation with @berndverst
* Update certification tests with no auth libraries
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Fixing publish/subscribe race condition in RabbitMQ component.
Added amqpReady function to tests to wait for RabbitMQ readiness.
Tweaks to when subscribed is signaled in the test.
* Update go.mod(s)
* AKV Integration test - WIP
* KeyVault integration test
* Adds network interruption capabilities
* wip
* remove async handling
* Azure Secret Store test
* get with the flow
* Remove network interruption because it is pointless here
* newlines
* Address code review
* Updte Readme to remove section on basic conformance
* Allocate random ports
* Add script to update Azure SQL firewall rules from GitHub meta API
* Update state.azure.sql test to use unique DB names
- Add use of `databaseName` metadata to state.azure.sql test.
- Add dynamic generation of test `databaseName` to conformance.yml to
avoid multiple test instances from racing.
- Add wait before clean-up of Azure SQL DB instance in conformance.yml
to prevent test flakiness and accumulation of undeleted DBs.
- Remove dynamic Azure SQL firewall rule injection from conformance.yml.
- The workflow relies on IPs used by GitHub Actions to be provisioned
in the firewall rules already.
- Update documentation for managing Azure SQL and testing instructions.
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Adds network interruption capabilities
* remove async handling
* update example docstring
* no need for loop
* small optimization
* newline for good measure
* update dapr version
* add state conformance test for postgresql #1117
* make postgresql pass comformance test
* include postgresql to github's workflow
* add newline at end of file for Linux text file
* enhance return
* add newline at end of file for valid Linux text file
* fix lint issue
* handle byte[] in special way so that to stick to json type in DB
* use jsonb instead of json
* fix lint issue
* doesn't allow "", a special case
* enhance assertEquals
* remove unused 'expectedReadResponse' to fix lint issue
* go fmt to fix lint issue
* fix lint issue
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* WIP
* tweaks
* Simplifying runnables
* Increase network timeout
* tweaks
* More tweaks
* Docker compose and structural tweaks
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
* Tweak
* Working Kafka test
* Watcher ordered vs unordered
* Error simulation
* Made port explicit, removed Redis
* package.Run (better naming)
* terraform related changes
* Restructuring certification testing
* Remove copied go-sdk client package
* Fix terraform.go build issues
* Using master of go-sdk
* Using main of go-sdk
* Tweaks per PR
* Adding file headers and some tweaks per PR
* More tweaks
* More comments in Kafka test
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Tanvi Gour <tanvi.gour@gmail.com>
Co-authored-by: tanvigour <>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
* cosmosdb binding - not as flexible as I'd like
conformance tests now pass
* add tests for new UUID injection logic
* Use nested property as partition key value
* avoid repeated string constants
* Disable Identification Hydrator to avoid struct requirement
* Remove struct
* Simplify PR
* Add comments ofor code maintenance
* Add mysql conformance test and fix mysql state to make it pass the test #1122
* remove unused code
* add newline at end of Linux text file
* enhance setValue and relevant unit tests
* fix lint issue
* fix lint issue
* Adding `initialOffset` option to Kafka binding and pubsub components
* Only allow oldest and newest
* Fix tests
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
- Add `OutputData` property to bindings conformance test `TestConfig`.
This is necessary to accommodate bindings like influx, cosmosdbgraph,
alibaba.table and other bindings that require a specific data format.
- Add docker-compose definition and conformance test configuration for
influx binding.
- Add influx bindings to conformance test workflow matrix.
* Fix parameters flag typo in setup-azure-conf-test.sh
* Update Event Hubs conformance tests
- Separate containers and consumer groups used for EventHub pubsub and
bindings conformance tests.
- Update setup-azure-conf-test.sh to reflect new naming targets.
* Add conformance test for EventHubs bindings
* Update EventHubs pubsub to pass conformance tests
- Add conformance tests for Azure EventHubs PubSub component
- Add retry & backoff handling on subscriber handling error to EventHubs
PubSub component for conformance tests.
- Add cancellation context to Azure EventHubs and update Close() to
invoke cancel prior to closing the hub, which cleans up both senders
and receivers.
* Add automation for deploying Azure conformance test resources
- Add .bicep definitions for Azure deployment of conformance test resources.
- Add setup-azure-conf-test.sh script for deploying .bicep definition and
configuring other assets needed for the Azure conformance tests:
- Creates the appropriate service principals and their credentials
for the test.
- Populates the key vault with the test parameters and secrets used by
the GitHub conformance.yml test workflow.
- Generates a .rc file that can be invoked with `source` to configure
the local environment variables and secrets for running conformance
tests described in tests/conformance/README.md.
- Generates a teardown script that can be used to dispose of the
deployed Azure resources when done.
* Move setup-azure-conf-test under azure/ subfolder
* Add documentation for Azure conformance test automation
- Add README.md to .github/infrastructure/conformance/azure describing
use of automation script.
- Update tests/conformance/README.md with reference to Azure
conformance test infrastructure automation.
- Fix spelling and issues in tests/conformance/README.md reported by
markdownlint.
Unlike the other conformance tests, the azure-keyvault.yaml config hardcodes the name of the
target resource which creates an issue for contributors wanting to set up their own test
environments. For example, different users sharing an Azure subscription may not be able
to manage the permissions on the target key vault individually, and will need to edit the
yaml to retarget against their own keyvault instance.
- Add `AzureKeyVaultName` environment variable as the value for `vaultName` in azure-keyvault.yaml
- Update conformance.yml github workflow to require AzureKeyVaultName secret.
* rename ConsumerID to ConsumerGroup
* rename consumerID to consumerGroup
* update the config in the conformance test
* set ConsumerID as alias of ConsumerGroup
* remove extra space
* gofmt
* add ClientID in kafka metadata
* remove ConsumerID field
* gofmt
* Update kafka.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* add `Ping` operation.
* add `Ping` implementation of GA components
* add context to an argument for Ping
* add `Ping` implementation of blobstorage
* fix error string for passing lint
* Update blobstorage.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Add ETag operation to statestore conformance test
ETags are used to enable concurrency guarantees and are part of the
statestore specification. Not all stores need to use them, however
if the store chooses to, it should be part of the certification
process. This commit adds a simple ETag test scenario that verifies
the basic behaviors of ETags.
https://github.com/dapr/components-contrib/issues/711
* Update tests.yml
* Error on mongodb when etag does not match on delete.
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Refactored the conformance tests to use a new config package
Removed dependency on k8s apimachinery sets
* Comment tweak
* Renamed kit package to internal
* Setting pipefail option in github workflow and purposefully making the Redis test fail to verify that the build fails (as expected)
* Disabling in order processing for Redis (re-disabling) and Azure Service Bus
* Improve conformance tests and add pulsar
* Fix linter error
* Fix linter errors
* Tweaks to pulsar pubsub per PR
* Switching uuid import
* Using Stringer interface to print message keys as base64 in logger
* Disabling some checks w/ Redis pubsub until the component can be enhanced
* Adding comment to eventually remove the simulateErrors option
* Add comment asBase64String to explain what it does
* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing
* Fixing linter issues
* using a wait group instead of a sleep to wait for processing to complete
* More comments and refactoring
* comment tweaks
* Tweaks per PR
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.
* Adding backoff/retry logic to the Kafka pub/sub component
* Add logic to return if error from backoff in kafka.
* Addressing PR comments and lint.
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
* Add state transaction conf test.
* Does not return error for key not found in mongodb.
* Rewrite state conf test.
* Handle []byte in cosmosdb state txn (#664)
Fixes: dapr/dotnet-sdk#579
The issue here is that we had missed the case of pre-marshaled bytes in
the state transaction code path, which would result in the encoding/json
package doing it's normal thing and base64 encoding the data as a JSON
string.
The fix is to use json.RawMessage to avoid that. Turns out this code can
be simplified a bit.
* Fix cosmosdb handling of byte[].
* Fix lint.
* Fix UT for cosmosdb.
* Remove config from state store conf test.
Co-authored-by: Ryan Nowak <nowakra@gmail.com>
* no message
* Removing HTTP input binding from conformance test
* Moving // nolint: noctx
* Fixed linter issue
* Tweaks
* Tweak
* Setting request headers from metadata
* More comments and handling of non-200 response codes
* Fixing conformance test for HTTP
* Temporarily bumping up redis timeouts
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Charlie Stanley <charlie.stanley@microsoft.com>
The http server required for the http binding tests was not up
before the tests began to execute. This change adds a channel to
allow the server to block until it's ready.
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Refactor bindings and introduce input bindings
This commit adds Kafka output bindings tests and refactors the
general bindings tests to include input and output bindings in
the same set of tests. Azure Storage Queues are used as the
first input binding.
* Add http input/output bindings test
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Fix running output bindings with errors
Refactor tests to make it simpler for go test filtering.
Add get operation for blobstorage binding.
* remove unused code
* Add azure service bus conformance tests
Bubble up error on required env var not set
* Update common.go
* Fix running output bindings with errors
* update readme
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Tests MongoDB statestore
* Increase test timeout a bit
* Increase durations even more.
* Clarify the mongodb and redis version
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
This commit adds new conformance tests for output binding components.
Redis is being used as a the introductory binding example.
https://github.com/dapr/components-contrib/issues/409
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Bernd Verst
ItalyPaleAle
tanvigour
Sky Ao
Christian Kaps
deepanshuagarwal
dependabot[bot]
Laurence
Yaron Schneider
amulyavarote
pigletfly
Surender Singh Malik
Amit Mor
Ricardo Corrêa
Shubham Sharma
Dmitry Shmulevich
Mukundan Sundararajan
Mukundan Sundararajan
Looong Dai
Dmitry Shmulevich
Artur Souza
Christian Kaps
shivam
ZhangJian He
Gehhilfe
1046102779
Phil Kedy
Lucas Jellema
Zheng Dayu
Yordan Pavlov
Mukundan Sundararajan
Scott Hussey
Taction
mbimbij
RainFD
halspang
Simon Leet
Ian Luo
Simon Leet
Bernd Verst
Manuel Riezebosch
Jaime Piña
Bernd Verst
Long Dai
iamazy
문찬용
Joni Collinge
Yaron Schneider
Nghia Tran