* add certification test for kafka binding, the first version is copied from kafka pubsub to verify github flow
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* only run kafka pubsub and binding cert test in development
* fix typo
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* run only kafka bindings
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add certification test for kafka binding: with and without retry
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add config.yaml file back for kafka binding certification test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* rollback to run all cert test before PR
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* update dependencies for kafka pubsub certification test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* update dependencies for kafka pubsub certification test(again)
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add network interruption test back
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Azure SDKs that are on track2
Includes some minor refactoring of auth code
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated Service Bus components to track2 SDK
Co-authored-by: halspang <halspang@microsoft.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Initial Certification test for eventhubs binding [incomplete] (#1670)
* certification test for eventhubs binding
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* modified go.mod and go.sum
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Add connection string testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* iothub testing
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* address feedback and run test
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* Install Azure CLI IOT hub extension
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* make modtidy-all
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* covering all eventhubs test cases
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
* dependency changes after go modtidy-all
Signed-off-by: tanvigour <tanvi.gour@gmail.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Use revive instead of golint (#1685)
Signed-off-by: pigletfly <wangbing.adam@gmail.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Updated to Go 1.18 (#1697)
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Removed go.work
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 💄 & 🧹
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* Add metadata property to configure Batching in Pulsar (#1707)
* Add metadata property to configure BatchingMaxSize&batchingMaxMessages in Pulsar
Signed-off-by: saberwang <saberwang@hotmail.com>
* sort field
Signed-off-by: saberwang <saberwang@hotmail.com>
* [pubsub]fix unit test bug
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove unrelated changes
Signed-off-by: saberwang <saberwang@hotmail.com>
* Delete hard coded Metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* remove .history
Signed-off-by: saberwang <saberwang@hotmail.com>
* restore .gitignore
Signed-off-by: saberwang <saberwang@hotmail.com>
* Hard coding default values and adding 'BatchingMaxPublishDelay' metadata
Signed-off-by: saberwang <saberwang@hotmail.com>
* fix code format
Signed-off-by: saberwang <saberwang@hotmail.com>
* formatting code
Signed-off-by: saberwang <saberwang@hotmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* This test can't work with track2 SDKs
The methods to create a message with a body are not exported
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* 🧹
Signed-off-by: Alessandro Segala (ItalyPaleAle) <43508+ItalyPaleAle@users.noreply.github.com>
* There's such thing as too much logging
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Refactored subscription.go
This greatly simplifies certain parts of the code, reducing the number of goroutines and likely improving performance.
Performance for end-users improves too as there's no need anymore to pause for 2 seconds every time that we reach `maxActiveMessages`.
Additionally, with this change the config options `prefetchCount` and `maxActiveMessagesRecoveryInSec` are removed as unnecessary anymore.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fixed pubsub tests
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* These packages should have never been upgraded
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Ensuring we don't fetch 1 message more than max active
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Adding configurable timeout for servicebusqueues operations
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Persistent connection for invoking SB queues
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Reverted Event Hub SDK update
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Revert "Reverted Event Hub SDK update"
This reverts commit 212220aba0.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Fix Azure deploy for users with a first.last email
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added some sleep that should help reduce flakiness in eventhubs binding cert test
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Changed servicebusqueue cert test
In case of a failure in the handler (ie. users' code), the message should be correctly re-enqueued, which means that messages will be re-delivered later and won't be in order.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: halspang <halspang@microsoft.com>
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Wang Bing <wangbing.adam@gmail.com>
Co-authored-by: saber-wang <45062099+saber-wang@users.noreply.github.com>
* refactory kafka pubsub code to extract common kafka code for reuse
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint;add unit test for subscribeAdapter
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* move topics filed from internal kafak struct to pubsub kafka struct, since in input binding the topics will confiured in metadata
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* reuse internal kafka code for bindings
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add redis standalone_test back which is delete by mistaken
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* small code improvement to trigger test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add license headers
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* try to set disbaleTls to true to verify the kafka connection fail
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* don't enable consum retry in kafka binding component;if authenticaion is disabled, need not set TLSDisable at the same time;
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
Co-authored-by: Loong Dai <long.dai@intel.com>
* Updated to Go 1.18
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
* Added go.work file
With Go 1.18, this allows gopls (the Go language server used for example in VS Code) to work inside test apps too.
See: https://go.dev/doc/tutorial/workspaces
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Removed go.work
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* 💄
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* install azure-iot-cli-extension if not found
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* reduce the number of messages from 100 to 10, to make the test faster
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* First iteration for the eventhubs certifications tests
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Fixing typos, adding more description and publishing with multiple partitions
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Moved the secret reference in components definition to secretstore.local.env
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* moving to latest dapr/dapr github.com/dapr/dapr@v1.7.0-rc.2
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updating go.mod
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* changes in infra and workflow for cert eventhubs
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* typo fix in local secret store type
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* correcting the typo in policyName
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* bug fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* publish to a topic with no subscriber fix
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete azure eventhub that got created as part of test run
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updated readme
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* removed dummy sidecar and leveraged connectionstring at namespace level for multi topic operations
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* go mod changes and nit fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* fixing iot device issue
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* NIT : CR fixes
* Enabled iothub test and a bit of code refactoring
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete the checkpoint container for the temp eventhub
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Revert "NIT : CR fixes"
This reverts commit 260f5ed686.
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* First iteration for the eventhubs certifications tests
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Fixing typos, adding more description and publishing with multiple partitions
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Moved the secret reference in components definition to secretstore.local.env
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* moving to latest dapr/dapr github.com/dapr/dapr@v1.7.0-rc.2
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updating go.mod
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* changes in infra and workflow for cert eventhubs
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* typo fix in local secret store type
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* correcting the typo in policyName
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* bug fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* publish to a topic with no subscriber fix
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete azure eventhub that got created as part of test run
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* updated readme
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* removed dummy sidecar and leveraged connectionstring at namespace level for multi topic operations
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* go mod changes and nit fixes
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* fixing iot device issue
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* NIT : CR fixes
* Enabled iothub test and a bit of code refactoring
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* delete the checkpoint container for the temp eventhub
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Revert "NIT : CR fixes"
This reverts commit 260f5ed686.
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
* Fix eventhub pubsub certification test infra provisioning
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Fixing the scoping issue for role assignment on eh
Signed-off-by: Surender Singh Malik <surenderssm@users.noreply.github.com>
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Upgrade nacos version
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
* Run go mod tidy all
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
* Add MaxSize to settings
Signed-off-by: Shubham Sharma <shubhash@microsoft.com>
* Add support for nats jetstream kv state store
Add tests for jetstream state store
Update go mod
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Fix lint errors
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Mod tidy
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Mod tidy all
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Mod tidy all
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600
Support Oauth2 authentication for Kafka
- Utilize the SASL OAUTHBEARER mechanism to support
the Oauth2 client_credentials flow for Kafka
authentication
- Deprecate `authRequired` field and introduce `authType`
to support varied authentication mechanisms
- Add a metadata upgrade mechanism to support backwards
compatability
- Recommend broker specific scopes to guard against a
compromised broker replaying a token
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* kafka test - use volumes instead of root
CI test fails due to losing data written to container
root
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up volumes
- When bringing down the docker-compose context, cleanup
volumes
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up stale comment
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Resume config defaults
- Return some Kafka config to default values to lower config
footprint and stabilize flaky CI runs
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Authentication for new Azure SDK
* Updated keyvault to use new Azure SDK
* 🙈
* Update authentication/azure/auth.go
* Update authentication/azure/auth.go
* Update authentication/azure/auth.go
* Reintroduce changes from PR 1132 without updating ASB
* Lint 💄
* Marking contexts as TODO as they'll need a timeout
As per conversation with @berndverst
* Update certification tests with no auth libraries
Co-authored-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
* Fixing publish/subscribe race condition in RabbitMQ component.
Added amqpReady function to tests to wait for RabbitMQ readiness.
Tweaks to when subscribed is signaled in the test.
* Update go.mod(s)
* WIP
* tweaks
* Simplifying runnables
* Increase network timeout
* tweaks
* More tweaks
* Docker compose and structural tweaks
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
* Tweak
* Working Kafka test
* Watcher ordered vs unordered
* Error simulation
* Made port explicit, removed Redis
* package.Run (better naming)
* terraform related changes
* Restructuring certification testing
* Remove copied go-sdk client package
* Fix terraform.go build issues
* Using master of go-sdk
* Using main of go-sdk
* Tweaks per PR
* Adding file headers and some tweaks per PR
* More tweaks
* More comments in Kafka test
Co-authored-by: tanvigour <60332928+tanvigour@users.noreply.github.com>
Co-authored-by: Tanvi Gour <tanvi.gour@gmail.com>
Co-authored-by: tanvigour <>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Sky Ao
Alessandro (Ale) Segala
Bernd Verst
Surender Singh Malik
pigletfly
Shubham Sharma
shivam
Dmitry Shmulevich
ZhangJian He
Artur Souza
Gehhilfe
1046102779
Phil Kedy
Lucas Jellema
Zheng Dayu
Yordan Pavlov
Scott Hussey
Taction
halspang
Simon Leet
Ian Luo