This commit is related to dapr/dapr#4624. As noted there, we have an issue in the runtime where all components are shut down after the grace period, when the app is likely already stopped. Because of that, certain input components (the subscribe part of pubsub and the input part of bindings - the latter not in scope of this PR), can continue bringing new work when it's known to fail.
In order to fix the issue linked above properly, we need to implement a way for PubSub components to have the "publish" part closed before the "subscribe" one (and in the future that will need to be done for input bindings too).
This commit achieves precisely that by adding a context in the Subscribe method. When that context is canceled (which can be at any time), the subscription is removed.
PS: This API change was implemented so it can one day be used for dapr/dapr#814 too, as it allows canceling individual subscriptions by using a different context. Although that's not possible today because it requires more work on the runtime, it does implement everything that's needed in the pubsub components already.
Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
* refactory kafka pubsub code to extract common kafka code for reuse
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint;add unit test for subscribeAdapter
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* move topics filed from internal kafak struct to pubsub kafka struct, since in input binding the topics will confiured in metadata
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* reuse internal kafka code for bindings
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add redis standalone_test back which is delete by mistaken
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* small code improvement to trigger test
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* add license headers
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* try to set disbaleTls to true to verify the kafka connection fail
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* don't enable consum retry in kafka binding component;if authenticaion is disabled, need not set TLSDisable at the same time;
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
* fix lint
Signed-off-by: Sky Ao <aoxiaojian@gmail.com>
Co-authored-by: Loong Dai <long.dai@intel.com>
* Kafka PubSub: Setting metadata as message headers.
Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
* running go mod tidy on certification tests
Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600
Support Oauth2 authentication for Kafka
- Utilize the SASL OAUTHBEARER mechanism to support
the Oauth2 client_credentials flow for Kafka
authentication
- Deprecate `authRequired` field and introduce `authType`
to support varied authentication mechanisms
- Add a metadata upgrade mechanism to support backwards
compatability
- Recommend broker specific scopes to guard against a
compromised broker replaying a token
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* kafka test - use volumes instead of root
CI test fails due to losing data written to container
root
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up volumes
- When bringing down the docker-compose context, cleanup
volumes
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Clean up stale comment
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Resume config defaults
- Return some Kafka config to default values to lower config
footprint and stabilize flaky CI runs
Signed-off-by: Scott Hussey <sthussey@gmail.com>
* Adding `initialOffset` option to Kafka binding and pubsub components
* Only allow oldest and newest
* Fix tests
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Kafka pubsub component was not setting `k.authRequired` property based
on parsed `meta.AuthRequired` value, so would not correctly configure
Kafka producer or consumer with appropriate credentials when specified
by the author.
- Fix assignment of `meta.AuthRequired` to `k.authRequired` consumed by
Subscribe().
- Fix initialization of `SyncProducer` to use copy of config with the
authorization credentials added.
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* rename ConsumerID to ConsumerGroup
* rename consumerID to consumerGroup
* update the config in the conformance test
* set ConsumerID as alias of ConsumerGroup
* remove extra space
* gofmt
* add ClientID in kafka metadata
* remove ConsumerID field
* gofmt
* Update kafka.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* no message
* Switching over usages of retry to the new package and making use of the DecodeConfig function
* Added decoding configuration using settings with a specific prefix
* Linter fixes
* Fixing linter error
* time.Sleep is the enemy
* Fix typo in comment
* Moving config to a pointer parameter so that the component can pass in the config with default values that make sense for that component.
* Renamed config struct
* Fix comment
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Improve conformance tests and add pulsar
* Fix linter error
* Fix linter errors
* Tweaks to pulsar pubsub per PR
* Switching uuid import
* Using Stringer interface to print message keys as base64 in logger
* Disabling some checks w/ Redis pubsub until the component can be enhanced
* Adding comment to eventually remove the simulateErrors option
* Add comment asBase64String to explain what it does
* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing
* Fixing linter issues
* using a wait group instead of a sleep to wait for processing to complete
* More comments and refactoring
* comment tweaks
* Tweaks per PR
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.
* Adding backoff/retry logic to the Kafka pub/sub component
* Add logic to return if error from backoff in kafka.
* Addressing PR comments and lint.
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Deepanshu Agarwal
Shubham Sharma
Mukundan Sundararajan
Bernd Verst
Alessandro (Ale) Segala
Sky Ao
Phil Kedy
Dmitry Shmulevich
Scott Hussey
Sergey Krutsko
Taction
Long Dai
Simon Leet
iamazy
Joni Collinge
Artur Souza
Yaron Schneider
halspang
Kyle Housley