Commit Graph

28 Commits

Author SHA1 Message Date
Phil Kedy 265fa9a9c2
Kafka PubSub: Use metadata as message headers. (#1409)
* Kafka PubSub: Setting metadata as message headers.

Signed-off-by: Phil Kedy <phil.kedy@gmail.com>

* running go mod tidy on certification tests

Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
2022-01-06 10:10:37 -08:00
Dmitry Shmulevich c7adb917f3
update license to Apache v2.0 (#1406) 2022-01-04 19:53:31 -08:00
Scott Hussey 897c2a4db5
Add OAuth2/OIDC support to Kafka pubsub (#1229)
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600

Support Oauth2 authentication for Kafka

- Utilize the SASL OAUTHBEARER mechanism to support
  the Oauth2 client_credentials flow for Kafka
  authentication

- Deprecate `authRequired` field and introduce `authType`
  to support varied authentication mechanisms

- Add a metadata upgrade mechanism to support backwards
  compatability

- Recommend broker specific scopes to guard against a
  compromised broker replaying a token

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* kafka test - use volumes instead of root

CI test fails due to losing data written to container
root

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Clean up volumes

- When bringing down the docker-compose context, cleanup
  volumes

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Clean up stale comment

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Resume config defaults

- Return some Kafka config to default values to lower config
  footprint and stabilize flaky CI runs

Signed-off-by: Scott Hussey <sthussey@gmail.com>
2021-12-28 10:34:28 -08:00
Sergey Krutsko 418fd96299 add kafkaVersion 2021-11-04 15:54:03 +03:00
Phil Kedy 1d0ce9ce23
Kafka certification: Adding more consumers and verifying messages are consumed (#1262)
* Adding second consumer and verifying messages are consumed

* Removed commented out code

* Adding async steps

* Added third consumer

* Adding consumer rebalance test

* Renaming watcher variables

* Update README.md

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-11-02 13:51:15 -07:00
Phil Kedy 9a3fe3f683
Kafka certification: Adding infra and network interruption tests (#1241)
* Adding infra and network interruption tests

* Tweak

* Using multierr.Combine

* Adding consumeRetryInterval setting to Kafka pubsub component
2021-10-27 17:41:27 -07:00
Taction c63fee5bec
Add tls support for kafka pubsub component (#1196)
* add tls support for kafka pubsub component

* fix review

* fix review

* delete unused json tag

Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-10-25 15:40:53 -07:00
Long Dai 988fed05ab
ci: standard linter config (#1102)
* ci: standard linter config

Signed-off-by: Long <long.dai@intel.com>

* Update utils.go

* fix pulsar issue

Signed-off-by: Long <long.dai@intel.com>

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-21 15:21:59 -07:00
Phil Kedy a0542ec922
Adding `initialOffset` option to Kafka binding and pubsub components (#1120)
* Adding `initialOffset` option to Kafka binding and pubsub components

* Only allow oldest and newest

* Fix tests

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-02 18:02:10 -07:00
Simon Leet f84a8b8914
Fix Kafka pubsub use of AuthRequired metadata (#1015)
Kafka pubsub component was not setting `k.authRequired` property based
on parsed `meta.AuthRequired` value, so would not correctly configure
Kafka producer or consumer with appropriate credentials when specified
by the author.

- Fix assignment of `meta.AuthRequired` to `k.authRequired` consumed by
  Subscribe().
- Fix initialization of `SyncProducer` to use copy of config with the
  authorization credentials added.

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-07-28 10:24:35 -07:00
iamazy 74af5b6528
rename ConsumerID to ConsumerGroup (#965)
* rename ConsumerID to ConsumerGroup

* rename consumerID to consumerGroup

* update the config in the conformance test

* set ConsumerID as alias of ConsumerGroup

* remove extra space

* gofmt

* add ClientID in kafka metadata

* remove ConsumerID field

* gofmt

* Update kafka.go

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-23 10:33:27 -07:00
Phil Kedy 8e6b36ecdf
Importing the retry and config packages from kit instead of internal (#940)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-22 09:38:55 -07:00
iamazy a59dcc44eb
Fix typo (#927)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-06-09 23:58:15 -07:00
Phil Kedy 1b30e75e60
Configurable retry (#854)
* no message

* Switching over usages of retry to the new package and making use of the DecodeConfig function

* Added decoding configuration using settings with a specific prefix

* Linter fixes

* Fixing linter error

* time.Sleep is the enemy

* Fix typo in comment

* Moving config to a pointer parameter so that the component can pass in the config with default values that make sense for that component.

* Renamed config struct

* Fix comment

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-06 09:43:57 -07:00
Phil Kedy 688df9bda4
Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807) 2021-04-19 12:06:10 -07:00
Joni Collinge 36ed5ec84f use existing context for pubsub comps 2021-03-23 20:12:46 +00:00
Joni Collinge c054ae6ada Merge branch 'master' into jjcollinge/pubsub-context 2021-02-25 16:15:06 +00:00
Phil Kedy 4a0011bcbd
Improve conformance tests and add pulsar (#702)
* Improve conformance tests and add pulsar

* Fix linter error

* Fix linter errors

* Tweaks to pulsar pubsub per PR

* Switching uuid import

* Using Stringer interface to print message keys as base64 in logger

* Disabling some checks w/ Redis pubsub until the component can be enhanced

* Adding comment to eventually remove the simulateErrors option

* Add comment asBase64String to explain what it does

* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing

* Fixing linter issues

* using a wait group instead of a sleep to wait for processing to complete

* More comments and refactoring

* comment tweaks

* Tweaks per PR

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-02-23 13:31:52 -08:00
Artur Souza dd3e463b52
Fix pubsub retry for Kafka (#688)
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.

* Adding backoff/retry logic to the Kafka pub/sub component

* Add logic to return if error from backoff in kafka.

* Addressing PR comments and lint.

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-02-12 00:08:46 -08:00
Yaron Schneider 3ef025c604
change headers (#679) 2021-02-09 18:57:55 -08:00
Yaron Schneider 58e92479ac
Add kafka max message bytes and pubsub partition key (#669)
* add kafka max message bytes and pubsub partition key

* linter
2021-02-08 11:14:52 -08:00
Artur Souza e6dadfab6c
TTL in PubSub. (#565)
* TTL in PubSub.

* Handle message TTL only if component cannot handle it.

* Removig dead code for overflow check in pubsub expiration.
2020-12-29 14:28:52 -08:00
Joni Collinge 2e6b84ae40 Merge remote-tracking branch 'origin/master' into jjcollinge/pubsub-context 2020-11-02 09:05:16 +00:00
Yaron Schneider 182d204111
Add close method to pub/sub (#494)
* add close method to pub/sub

* linter
2020-10-12 19:06:33 -07:00
halspang f647d9eedb
Upgrade golang lint to 1.31 (#490)
Upgrade golang lint to 1.31.0

https://github.com/dapr/components-contrib/issues/439
2020-10-12 12:54:15 -07:00
jjcollinge 19442e67c8 update pubsub components to use new context 2020-07-28 10:03:23 +01:00
jjcollinge 410628af3d add context to pubsub handler signature 2020-07-07 12:37:15 +01:00
Kyle Housley 0c4cade71f
Add pubsub support for kafka (#315)
* fixes issue #8: adds kafka as pubsub component

* fixes issue #8: adds kafka as pubsub component

* pubsub kafka: modifies readme

* pubsub kafka: fixes bug

* pubsub kafka: fixes bug

* pubsub kafka: changes are per PR review

* changes as PR review

* removed unused field

* only close channel once

* working kafka pubsub

* add testing for metadata parsing

* linting

* remove unused PublishTopic, remove loglevel

* Use one consumer group for subscriptions

Co-authored-by: Roberto J Rojas <robertojrojas@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
2020-04-27 19:53:04 -07:00