* Add support for nats jetstream wildcard subscriptions
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Add support for nats jetstream wildcard subscriptions
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
* pubsub jetstream: add nkey based authentication
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* fix godot: comment should end in a period
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* go mod tidy
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* fix: go mod tidy
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* pubsub jetstream: add e2e test
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
* Cleanup not needed files and move jwt
Signed-off-by: Tim Burkert <burkert.tim@gmail.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
* squash
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* comment
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* gofumpted
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* breakdown of metadata loading
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* metadata further refactoring
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* Kafka PubSub: Setting metadata as message headers.
Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
* running go mod tidy on certification tests
Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600
Support Oauth2 authentication for Kafka
- Utilize the SASL OAUTHBEARER mechanism to support
the Oauth2 client_credentials flow for Kafka
authentication
- Deprecate `authRequired` field and introduce `authType`
to support varied authentication mechanisms
- Add a metadata upgrade mechanism to support backwards
compatability
- Recommend broker specific scopes to guard against a
compromised broker replaying a token
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* kafka test - use volumes instead of root
CI test fails due to losing data written to container
root
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up volumes
- When bringing down the docker-compose context, cleanup
volumes
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up stale comment
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Resume config defaults
- Return some Kafka config to default values to lower config
footprint and stabilize flaky CI runs
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* not modifying policy if disableEntity is set
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* not modifying policy if disableEntity is set
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
* support tracestate can be propagated in PubSub
* add trace state
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* Move Service Bus Pubsub/Binding to common auth
Both the pubsub and input/output binding for Azure Service Bus were
connecting via a connection string. This is still supported but will
now fallback to using AAD from the common auth library. This is also
the recommended auth pattern going forward.
* Move AMPQ specific auth and fix linter issues
* Make conn string and namespace mutually exclusive
* Move resourceName to a constant
* Update auth_amqp.go
* Update auth.go
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* Fixing publish/subscribe race condition in RabbitMQ component.
Added amqpReady function to tests to wait for RabbitMQ readiness.
Tweaks to when subscribed is signaled in the test.
* Update go.mod(s)
* Add dead letter support
* fix lint
* add maxLen and maxLenBytes support
* Let the DLX and DLQ be free form.
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long0dai@foxmail.com>
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: cvictory <shenglicao2@gmail.com>
* Adding `initialOffset` option to Kafka binding and pubsub components
* Only allow oldest and newest
* Fix tests
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Add configuration for enabling message ordering
* Simplify passing the metadata
* Update field order to satisfy alignment linter
Co-authored-by: Luke Kennedy <luke.kennedy@optiisolutions.com>
* bugfix for sns topic deletion upon termination
* Revert "bugfix for sns topic deletion upon termination"
This reverts commit bcaa9bb562.
* wip on normalizing queue/topic names
* sanitize queue and topic names
* sanitized names. bugfix for close
* # This is a combination of 4 commits.
# This is the 1st commit message:
Improve error message in case of missing property (#1012)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
# This is the commit message #2:
Remove vestigial pubsub/nats code (#1024)
The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
# This is the commit message #3:
bugfix for sns topic deletion upon termination
# This is the commit message #4:
Revert "bugfix for sns topic deletion upon termination"
This reverts commit bcaa9bb562.
* removed debug message
* raw string abort
* merge issues solved
* wip
* gofmt+remove regex and use byte iter
* wip. first impl of dead-letters queue config
* wip. refactor and fallback values
* integration test wip
* wip integration test
* wip integration
* wip on testing
* wip
* still buggy but wip!
* bugfix in dlq creation
* working. still bug in subscription clean up
* Update snssqs_integ_test.go
* golangci-lint fixes
* golangci-lint refactoring
* trying to skip running integrations for snssqs
* testing
* skip integration test if no AWS related envvars are set (skip in CI)
* parallel testing in unittests
* code review fixes
* not using implicit maxReceives
* maxReceives renamed
* unittest refactor
* Update snssqs.go
* integ removed, renaming back of const
Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
* bugfix for sns topic deletion upon termination
* Revert "bugfix for sns topic deletion upon termination"
This reverts commit bcaa9bb562.
* wip on normalizing queue/topic names
* sanitize queue and topic names
* sanitized names. bugfix for close
* # This is a combination of 4 commits.
# This is the 1st commit message:
Improve error message in case of missing property (#1012)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
# This is the commit message #2:
Remove vestigial pubsub/nats code (#1024)
The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
# This is the commit message #3:
bugfix for sns topic deletion upon termination
# This is the commit message #4:
Revert "bugfix for sns topic deletion upon termination"
This reverts commit bcaa9bb562.
* removed debug message
* raw string abort
* gofmt+remove regex and use byte iter
Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
* Fixing the handling of detach errors (#1030)
* Improve error message in case of missing property (#1012)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Remove vestigial pubsub/nats code (#1024)
The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
* Fixing the handling of detach errors
Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
* Revert "Fixing the handling of detach errors (#1030)" (#1031)
This reverts commit 54840c2b83.
* Fixing the handling of detach errors (#1032)
Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
* Add conformance test for EventHubs bindings
* Update EventHubs pubsub to pass conformance tests
- Add conformance tests for Azure EventHubs PubSub component
- Add retry & backoff handling on subscriber handling error to EventHubs
PubSub component for conformance tests.
- Add cancellation context to Azure EventHubs and update Close() to
invoke cancel prior to closing the hub, which cleans up both senders
and receivers.
Kafka pubsub component was not setting `k.authRequired` property based
on parsed `meta.AuthRequired` value, so would not correctly configure
Kafka producer or consumer with appropriate credentials when specified
by the author.
- Fix assignment of `meta.AuthRequired` to `k.authRequired` consumed by
Subscribe().
- Fix initialization of `SyncProducer` to use copy of config with the
authorization credentials added.
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
- Update azure-event-hubs-go dependency to currently supported v3 with
IoT Hub SystemProperties.
- Write Event.SystemProperties to ReadResponse.Metadata for EventHubs
input binding.
- Write Event.SystemProperties to NewMessage.Metadata for EventHubs
pubsub.
* fix: Dapr runtime panic when handle Pub/Sub (dapr#3281)
* use uuid to generate Id when Id is missing
* skip default value for id and content-type
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: beiwei.ly <beiwei.ly@alibaba-inc.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
* rename ConsumerID to ConsumerGroup
* rename consumerID to consumerGroup
* update the config in the conformance test
* set ConsumerID as alias of ConsumerGroup
* remove extra space
* gofmt
* add ClientID in kafka metadata
* remove ConsumerID field
* gofmt
* Update kafka.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Move pubsub redis client to internal
* Add failover for common redis. Refactor state redis.
* refactor binding redis
* fix lint
* Refactor to setting partten
* fix db and lint
* fix lint
* for backward compatibility
* Using a type alias to handle decoding duration values of -1 for Redis.
* Linter fixes
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
* Fix race in pulsar pubsub component
* fix lint
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* no message
* Switching over usages of retry to the new package and making use of the DecodeConfig function
* Added decoding configuration using settings with a specific prefix
* Linter fixes
* Fixing linter error
* time.Sleep is the enemy
* Fix typo in comment
* Moving config to a pointer parameter so that the component can pass in the config with default values that make sense for that component.
* Renamed config struct
* Fix comment
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* master:
add handle timeout to cancel req for it would be redelivered (#832)
Add optional metadata param maxLen for redis stream PubSub component (#835)
Implement bindings for the Zeebe workflow engine (#806)
Automerge in contrib. (#849)
# Conflicts:
# go.mod
# go.sum
# pubsub/redis/redis.go
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
resolve conflict
* master:
Increasing overall test timeout to allow for conformance tests to run (#834)
Improve SMTP binding performance (#782)
add redis cluter mode for pubsub (#823)
downgrade the verbosity of lock renewal errors from warn to debug. This error is expected when messages are processed before beingrenewed (#803)
Improve pulsar publisher performance. (#827)
optimze(state/redis): output detail redis error (#825)
Consul nameresolution component (#750)
Add DNS name resolver. Add unit tests. (#743)
Azure Service Bus: Changed log level of error on close to info (#830)
Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807)
Update state/blobstorage.go to add metadata. (#786)
Azure Service Bus publishing retries when server is "too busy" (#821)
# Conflicts:
# bindings/redis/redis.go
# go.mod
# go.sum
# pubsub/redis/redis.go
* Improve pulsar publisher performance.
* Addressed comments and added some debug logs
* Use peek instead of get for closing producers
* Fixed lint issue
* Added debug log in close
* Change back o 100
* Close producer before client
* Fixed cached number of producer and added todo for making it configurable
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
When component close function called `r.cancel()` and `r.client.Close()` will be excuted.
The pollNewMessagesLoop loop use client to read from redis which will lead to err. And when err occured this loop will continue.
We need to check context on the top of this loop.
* add metadata properties to control max reconn attempts and recovery delay
* add metadata tests
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
* Improve conformance tests and add pulsar
* Fix linter error
* Fix linter errors
* Tweaks to pulsar pubsub per PR
* Switching uuid import
* Using Stringer interface to print message keys as base64 in logger
* Disabling some checks w/ Redis pubsub until the component can be enhanced
* Adding comment to eventually remove the simulateErrors option
* Add comment asBase64String to explain what it does
* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing
* Fixing linter issues
* using a wait group instead of a sleep to wait for processing to complete
* More comments and refactoring
* comment tweaks
* Tweaks per PR
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.
* Adding backoff/retry logic to the Kafka pub/sub component
* Add logic to return if error from backoff in kafka.
* Addressing PR comments and lint.
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
* Added binary data type detection
Added base64 encoding for binary data types
* Update utils.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* update logger form nats-io's to dapr's
* run go mod tidy and go mod delete the nats-io gnatsd
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
* Add pubsub support for mqtt over TLS
* Update mqtt host to include uri scheme.
- This is required to identify mode of connection used while
connecting to broker.
* Add pubsub support for mqtt over TLS.
- add validation while initialisation.
- update test cases.
* Use shared sqs auth package
* Goimports-ed files
* Added support for endpoint param in aws credentials abstraction, since the sqs-sns pubsub module required it
* Added endpoint support for all aws things
* Updated tests
* Add pubsub support for mqtt
- update README.md
- add tests for metadata parsing
- linting
* Add pubsub support for mqtt
- remove topic field from metadata.
* Add pubsub support for mqtt
- use consumerID as clientID for mqtt connection
* Add pubsub support for mqtt
- address review comments
* created pulsar component
* fixed multi topic issue
* moved metadata struct
* finished writing metadata tests
* fixed linter issues
* modifies consumer type to failover
* uncapitalized errors
* changed subscriptionName to consumerID
* acknowledge message only if there's no error returned from handler
Co-authored-by: john verdonck <John@johns-MacBook-Pro.local>
* working but WIP aws pubsub
* working pubsub impl, problematic resource naming
* working implementation with tags and hashed topic/queue names
* add correct permissions for sqs queue to allow sns publishing
* fix logging messages, make awsToken optional, rename test functions, nits
* lowercase comments
* nits, repetitive use of 'the'
Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
* added worker pool for concurrent receive
* fixed missing end quote
* added type alias
* removed shadow decl
* added cleanup logic for subscription receiver
* renamed metadata property to numConcurrentHandlers and added handler specific timeout
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
* NATS streaming pubsub impl
* after go mod tidy
* updates tests, other refactoring
* updated client ID generator
* updated test
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
* Add Azure SignalR binding
* Fixes from go lint
* Go lint
* Use "github.com/dgrijalva/jwt-go" for jwt
Fix bug parsing connection string
Normalize usage of len vs == in strings
* Consistent usage of http header Set
* Add RabbitMQ pub/sub
* Fix lint error (global variable)
* RabbitMQ pub/sub auto ack defaults to false
Fix log message incorrect prefix
* Remove extra line in go.mod
* Revert extra line in go.mod
* [Commit - PubSub changes ]
Changing the names of the packages directory so to be consistent with name of the company .
It is to be noted that package name can not have hyphen "-" in the name.
So changing the directory names and putting under company names.
For example - servicebus package would be
dapr/components-contrib/pubsub/azure/servicebus
* Commit 2 - Changes for Secret Stores
* Commit 3 - for the state store changes
Ricardo Corrêa
Bernd Verst
Taction
1046102779
Gehhilfe
Looong Dai
Jigar
Gehhilfe
Amit Mor
Yordan Pavlov
Shivam Kumar
Phil Kedy
Dmitry Shmulevich
Mukundan Sundararajan
Zach
Scott Hussey
Amit Mor
huazhongming
halspang
Sergey Krutsko
Simon Leet
Joni Collinge
Simon Leet
Manuel Riezebosch
Jaime Piña
Luke Kennedy
weibubli
Ian Luo
Long Dai
iamazy
cinience
Artur Souza
Charlie Stanley
Xueqian Wang
Trond Hindenes
Yaron Schneider
cvictory
AbserAri
Mayank Kumar
Brightside Automation
Leon Mai
Ally Parker
Mark Chmarny
Mark Chmarny
Kyle Housley
John
Young Bu Park
Piotr Jasina
Sivamuthu Kumar
Abhishek Gupta
REDDY PRASAD
thinkerou
Francisco Beltrao
Shalabh Mohan Shrivastava
Francisco Beltrao
Aman Bhardwaj