* Added test for multiple pubsub handlers
This patch adds conformance tests for PubSub with multiple handlers, each listening on a different topic.
It's meant to catch issues such as #1709
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* Added missing go.mod for storagequeues binding test
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* refactory kafka pubsub code to extract common kafka code for reuse
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint;add unit test for subscribeAdapter
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* move topics filed from internal kafak struct to pubsub kafka struct, since in input binding the topics will confiured in metadata
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* reuse internal kafka code for bindings
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add redis standalone_test back which is delete by mistaken
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* small code improvement to trigger test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add license headers
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* try to set disbaleTls to true to verify the kafka connection fail
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* don't enable consum retry in kafka binding component;if authenticaion is disabled, need not set TLSDisable at the same time;
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
Co-authored-by: Loong Dai <long.dai@intel.com>
* conformance tests
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* latest localstack. snssqs to conformance list
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
Co-authored-by: Mukundan Sundararajan <musundar@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600
Support Oauth2 authentication for Kafka
- Utilize the SASL OAUTHBEARER mechanism to support
the Oauth2 client_credentials flow for Kafka
authentication
- Deprecate `authRequired` field and introduce `authType`
to support varied authentication mechanisms
- Add a metadata upgrade mechanism to support backwards
compatability
- Recommend broker specific scopes to guard against a
compromised broker replaying a token
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* kafka test - use volumes instead of root
CI test fails due to losing data written to container
root
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up volumes
- When bringing down the docker-compose context, cleanup
volumes
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up stale comment
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Resume config defaults
- Return some Kafka config to default values to lower config
footprint and stabilize flaky CI runs
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* - Introduce `fmt.Sprintf` where applicable
- Add conformance test for cassandra component.
Signed-off-by: mbimbij <joseph.mbimbi@gmail.com>
* Start cassandra via docker-compose for its conformance test
Signed-off-by: mbimbij <joseph.mbimbi@gmail.com>
* Apply `go fmt` on `tests/conformance/common.go` as part of PR correction
Signed-off-by: mbimbij <joseph.mbimbi@gmail.com>
* Add script to update Azure SQL firewall rules from GitHub meta API
* Update state.azure.sql test to use unique DB names
- Add use of `databaseName` metadata to state.azure.sql test.
- Add dynamic generation of test `databaseName` to conformance.yml to
avoid multiple test instances from racing.
- Add wait before clean-up of Azure SQL DB instance in conformance.yml
to prevent test flakiness and accumulation of undeleted DBs.
- Remove dynamic Azure SQL firewall rule injection from conformance.yml.
- The workflow relies on IPs used by GitHub Actions to be provisioned
in the firewall rules already.
- Update documentation for managing Azure SQL and testing instructions.
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* add state conformance test for postgresql #1117
* make postgresql pass comformance test
* include postgresql to github's workflow
* add newline at end of file for Linux text file
* enhance return
* add newline at end of file for valid Linux text file
* fix lint issue
* handle byte[] in special way so that to stick to json type in DB
* use jsonb instead of json
* fix lint issue
* doesn't allow "", a special case
* enhance assertEquals
* remove unused 'expectedReadResponse' to fix lint issue
* go fmt to fix lint issue
* fix lint issue
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* cosmosdb binding - not as flexible as I'd like
conformance tests now pass
* add tests for new UUID injection logic
* Use nested property as partition key value
* avoid repeated string constants
* Disable Identification Hydrator to avoid struct requirement
* Remove struct
* Simplify PR
* Add comments ofor code maintenance
* Add mysql conformance test and fix mysql state to make it pass the test #1122
* remove unused code
* add newline at end of Linux text file
* enhance setValue and relevant unit tests
* fix lint issue
* fix lint issue
* Adding `initialOffset` option to Kafka binding and pubsub components
* Only allow oldest and newest
* Fix tests
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
- Add `OutputData` property to bindings conformance test `TestConfig`.
This is necessary to accommodate bindings like influx, cosmosdbgraph,
alibaba.table and other bindings that require a specific data format.
- Add docker-compose definition and conformance test configuration for
influx binding.
- Add influx bindings to conformance test workflow matrix.
* Fix parameters flag typo in setup-azure-conf-test.sh
* Update Event Hubs conformance tests
- Separate containers and consumer groups used for EventHub pubsub and
bindings conformance tests.
- Update setup-azure-conf-test.sh to reflect new naming targets.
* Add conformance test for EventHubs bindings
* Update EventHubs pubsub to pass conformance tests
- Add conformance tests for Azure EventHubs PubSub component
- Add retry & backoff handling on subscriber handling error to EventHubs
PubSub component for conformance tests.
- Add cancellation context to Azure EventHubs and update Close() to
invoke cancel prior to closing the hub, which cleans up both senders
and receivers.
Unlike the other conformance tests, the azure-keyvault.yaml config hardcodes the name of the
target resource which creates an issue for contributors wanting to set up their own test
environments. For example, different users sharing an Azure subscription may not be able
to manage the permissions on the target key vault individually, and will need to edit the
yaml to retarget against their own keyvault instance.
- Add `AzureKeyVaultName` environment variable as the value for `vaultName` in azure-keyvault.yaml
- Update conformance.yml github workflow to require AzureKeyVaultName secret.
* rename ConsumerID to ConsumerGroup
* rename consumerID to consumerGroup
* update the config in the conformance test
* set ConsumerID as alias of ConsumerGroup
* remove extra space
* gofmt
* add ClientID in kafka metadata
* remove ConsumerID field
* gofmt
* Update kafka.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Refactored the conformance tests to use a new config package
Removed dependency on k8s apimachinery sets
* Comment tweak
* Renamed kit package to internal
* Setting pipefail option in github workflow and purposefully making the Redis test fail to verify that the build fails (as expected)
* Disabling in order processing for Redis (re-disabling) and Azure Service Bus
* Improve conformance tests and add pulsar
* Fix linter error
* Fix linter errors
* Tweaks to pulsar pubsub per PR
* Switching uuid import
* Using Stringer interface to print message keys as base64 in logger
* Disabling some checks w/ Redis pubsub until the component can be enhanced
* Adding comment to eventually remove the simulateErrors option
* Add comment asBase64String to explain what it does
* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing
* Fixing linter issues
* using a wait group instead of a sleep to wait for processing to complete
* More comments and refactoring
* comment tweaks
* Tweaks per PR
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.
* Adding backoff/retry logic to the Kafka pub/sub component
* Add logic to return if error from backoff in kafka.
* Addressing PR comments and lint.
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
* Add state transaction conf test.
* Does not return error for key not found in mongodb.
* Rewrite state conf test.
* Handle []byte in cosmosdb state txn (#664)
Fixes: dapr/dotnet-sdk#579
The issue here is that we had missed the case of pre-marshaled bytes in
the state transaction code path, which would result in the encoding/json
package doing it's normal thing and base64 encoding the data as a JSON
string.
The fix is to use json.RawMessage to avoid that. Turns out this code can
be simplified a bit.
* Fix cosmosdb handling of byte[].
* Fix lint.
* Fix UT for cosmosdb.
* Remove config from state store conf test.
Co-authored-by: Ryan Nowak <nowakra@gmail.com>
* no message
* Removing HTTP input binding from conformance test
* Moving // nolint: noctx
* Fixed linter issue
* Tweaks
* Tweak
* Setting request headers from metadata
* More comments and handling of non-200 response codes
* Fixing conformance test for HTTP
* Temporarily bumping up redis timeouts
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Charlie Stanley <charlie.stanley@microsoft.com>
* Refactor bindings and introduce input bindings
This commit adds Kafka output bindings tests and refactors the
general bindings tests to include input and output bindings in
the same set of tests. Azure Storage Queues are used as the
first input binding.
* Add http input/output bindings test
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Fix running output bindings with errors
Refactor tests to make it simpler for go test filtering.
Add get operation for blobstorage binding.
* remove unused code
* Add azure service bus conformance tests
Bubble up error on required env var not set
* Update common.go
* Fix running output bindings with errors
* update readme
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Tests MongoDB statestore
* Increase test timeout a bit
* Increase durations even more.
* Clarify the mongodb and redis version
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
This commit adds new conformance tests for output binding components.
Redis is being used as a the introductory binding example.
https://github.com/dapr/components-contrib/issues/409
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Bernd Verst
Alessandro (Ale) Segala
Ryan Lettieri
amulyavarote
Sky Ao
Amit Mor
Ricardo Corrêa
Dmitry Shmulevich
Mukundan Sundararajan
Scott Hussey
mbimbij
Simon Leet
Ian Luo
Long Dai
Artur Souza
Bernd Verst
Phil Kedy
Manuel Riezebosch
Jaime Piña
Bernd Verst
Christian Kaps
iamazy
Nghia Tran
halspang