Commit Graph

220 Commits

Author SHA1 Message Date
Yordan Pavlov 575b530bc7 Merge latest changes from upstream and resolve conflict
Signed-off-by: Yordan Pavlov <yordan.pavlov@dunnhumby.com>
2022-03-10 15:47:11 +00:00
Ricardo Corrêa 027204f2d3
nack in error messages (#1562)
Signed-off-by: Ricardo Corrêa <r.c.correa@outlook.com>
2022-03-09 17:21:51 -08:00
Bernd Verst 800d22f4b0 Move errors to pseudo constant
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-02-24 15:44:09 -08:00
Bernd Verst e59a67a1c4 Add retriable error conditions for ASB sending
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-02-24 15:33:49 -08:00
Taction 70a86e6d80
supports rabbitmq pubsub set exchangekind (#1520)
Signed-off-by: zhangchao <zchao9100@gmail.com>
2022-02-21 13:08:18 -08:00
1046102779 2126b96dc6 feature/pulsar: add producer option disableBatching
Signed-off-by: 1046102779 <seachen@tencent.com>
2022-02-16 12:07:02 +08:00
Gehhilfe 6ecc3b6f7b
Add support for nats jetstream wildcard subscriptions (#1465)
* Add support for nats jetstream wildcard subscriptions

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* Add support for nats jetstream wildcard subscriptions

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2022-02-15 09:46:33 -08:00
Looong Dai c4a64ef471
Merge branch 'master' into gcp_pubsub_reconnect_loop 2022-02-07 10:23:22 +08:00
Taction 7c7c977d52
Fix rabbitmq pubsub reconnect (#1471)
Signed-off-by: zhangchao <zchao9100@gmail.com>
2022-02-02 11:56:48 -08:00
Jigar 9078f11985
Added ContentType to pubsub/binding/state request-response (#1376)
Signed-off-by: jigargandhi <jigarr.gandhi@gmail.com>

Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2022-01-28 10:17:04 -08:00
yellow chicks e04fd24a07
feature/pulsar: provide token && http request (#1427)
* feature/pulsar: provide token && http request

Signed-off-by: 1046102779 <seachen@tencent.com>

* feature/pulsar: provide token && http request

Signed-off-by: 1046102779 <seachen@tencent.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2022-01-25 21:23:56 -08:00
Gehhilfe b835eb4e1e
pubsub jetstream: add nkey based authentication (#1441)
* pubsub jetstream: add nkey based authentication

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* fix godot: comment should end in a period

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* go mod tidy

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* fix: go mod tidy

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* pubsub jetstream: add e2e test

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* Cleanup not needed files and move jwt

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2022-01-25 21:11:50 -08:00
Amit Mor db9e071449
bugfixes: visibility timeout, dlq queue attributes (#1446)
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2022-01-18 10:31:53 -08:00
Yordan Pavlov 022c67a57b add reconnect loop for GCP pub/sub subscriptions
Signed-off-by: Yordan Pavlov <yordan.pavlov@dunnhumby.com>
2022-01-17 17:45:00 +00:00
Amit Mor c8844ccaed
Pubsub AWS SNS/SQS - adding context, cancellation, timeouts, retrying \w backoff & disable delete of messages on failure (#1433)
* squash

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* comment

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* gofumpted

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* breakdown of metadata loading

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* metadata further refactoring

Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2022-01-13 07:27:49 -08:00
Shivam Kumar 4631d6e41a
MQTT Pubsub Certification Testing + AutAckOff Fix for MQTT (#1420)
* MQTT Certification Test

Signed-off-by: shivam <shivamkm07@gmail.com>

* Using paho.mqtt.golang fork with AutoAck fix

Signed-off-by: shivam <shivamkm07@gmail.com>

* Adding MQTT component in certification.yml

Signed-off-by: shivam <shivamkm07@gmail.com>

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2022-01-07 10:33:54 -08:00
1046102779 00c99a4027 feature/pulsar: add producer option disableBatching
Signed-off-by: 1046102779 <seachen@tencent.com>
2022-01-07 12:45:14 +08:00
Phil Kedy 265fa9a9c2
Kafka PubSub: Use metadata as message headers. (#1409)
* Kafka PubSub: Setting metadata as message headers.

Signed-off-by: Phil Kedy <phil.kedy@gmail.com>

* running go mod tidy on certification tests

Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
2022-01-06 10:10:37 -08:00
Dmitry Shmulevich c7adb917f3
update license to Apache v2.0 (#1406) 2022-01-04 19:53:31 -08:00
Mukundan Sundararajan fe6c9e37d9
rewrite eventhubs component (#1292)
Signed-off-by: Mukundan Sundararajan <msundar.ms@outlook.com>
2022-01-03 19:36:35 -08:00
Zach 5c9365b314
pub/sub rocketmq:upgrade with v2 (#1383)
* pub/sub rocketmq:upgrade with client v2

Signed-off-by: zach <zachchou016@gmail.com>

* pubsub/rocketmq:remove cache and add start with setup

Signed-off-by: zach <zachchou016@gmail.com>

* pubsub/rocketmq:fix variable golint

Signed-off-by: zach <zachchou016@gmail.com>

* pubsub/rocketmq:change license and fix goimports

Signed-off-by: zach <zachchou016@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2021-12-31 12:18:27 -08:00
Scott Hussey 897c2a4db5
Add OAuth2/OIDC support to Kafka pubsub (#1229)
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600

Support Oauth2 authentication for Kafka

- Utilize the SASL OAUTHBEARER mechanism to support
  the Oauth2 client_credentials flow for Kafka
  authentication

- Deprecate `authRequired` field and introduce `authType`
  to support varied authentication mechanisms

- Add a metadata upgrade mechanism to support backwards
  compatability

- Recommend broker specific scopes to guard against a
  compromised broker replaying a token

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* kafka test - use volumes instead of root

CI test fails due to losing data written to container
root

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Clean up volumes

- When bringing down the docker-compose context, cleanup
  volumes

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Clean up stale comment

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Resume config defaults

- Return some Kafka config to default values to lower config
  footprint and stabilize flaky CI runs

Signed-off-by: Scott Hussey <sthussey@gmail.com>
2021-12-28 10:34:28 -08:00
Taction 6d9d0bca35
fix nits: duplicated err handle (#1341)
* remove duplicated err handle

Signed-off-by: Taction <zchao9100@gmail.com>

* fix test go mod tidy

Signed-off-by: Taction <zchao9100@gmail.com>
2021-12-27 08:47:10 -08:00
Amit Mor b548aa8bcc
1385 disable policy creation (#1386)
* not modifying policy if disableEntity is set

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* not modifying policy if disableEntity is set

Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2021-12-22 12:42:28 -08:00
Dmitry Shmulevich 7a631ef7aa
pub/sub rabbitmq: add publishing retries (#1382)
Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2021-12-17 15:59:38 -08:00
Dmitry Shmulevich 3fcf50c044
pub/sub rabbitmq: fix subscribe error (#1377)
Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com>
2021-12-14 08:22:35 -08:00
Amit Mor 72bae26380
disable entity creation (#1375)
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2021-12-13 09:22:57 -08:00
Amit Mor e97b1bf9f3
1335 bugfix #2 (#1368)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* Update snssqs.go

* dapr bot schedule

* read and append queue attributes

* unnecessary escaping in json tag

* unexporting structs

* bugfix in policy

* bugfix in policy. merged from master

* fifo suffix as const
2021-12-08 14:20:24 -08:00
Amit Mor 4918900c09
AWS PubSub SNS/SQS - FIFO support + refactoring (#1312)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* wrapping errors

* fifo and naming sanitization

* fifo and name santization tests

* component carrying id

* Update snssqs.go

removed unneeded import

* Update snssqs.go

* add missing github actions file

* linting

* fix go.mod

* schedule yml file

* const max lenght, bugfix in ack

* dapr bot schedule

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-30 10:52:00 -08:00
Amit Mor da7e8cfde7
#1335 bugfix - multi SNS topics publishing policy to SQS (#1337)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* Update snssqs.go

* dapr bot schedule

* read and append queue attributes

* unnecessary escaping in json tag

* unexporting structs
2021-11-26 08:39:56 -08:00
huazhongming 63d6329ae5
Support tracestate can be propagated in PubSub (#1264)
* support tracestate can be propagated in PubSub

* add trace state

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-24 12:30:35 -08:00
Taction 4bf0dcbfcf
Fix the Route of subscription API does not work on pubsub.pulsar (#1283)
* feat(pulsar): add tenant and namesapce. fix topic parsing.

* add test

* fix review add topic format unit test

* fix review: add persistent to metadata, fix nits

* fix lint

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Mukundan Sundararajan <musundar@microsoft.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-22 18:57:53 -08:00
halspang d5a68041c9
Move Service Bus Pubsub/Binding to common auth (#1201)
* Move Service Bus Pubsub/Binding to common auth

Both the pubsub and input/output binding for Azure Service Bus were
connecting via a connection string. This is still supported but will
now fallback to using AAD from the common auth library. This is also
the recommended auth pattern going forward.

* Move AMPQ specific auth and fix linter issues

* Make conn string and namespace mutually exclusive

* Move resourceName to a constant

* Update auth_amqp.go

* Update auth.go

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-22 18:20:11 -08:00
Dmitry Shmulevich ad433a27e1
add troubleshooting info (#1278)
add retries for pubsub subscribe
fix race conditions
2021-11-15 16:33:21 -08:00
Sergey Krutsko a269d62138 change assert -> require 2021-11-05 10:22:13 +03:00
Sergey Krutsko 418fd96299 add kafkaVersion 2021-11-04 15:54:03 +03:00
Phil Kedy 77564a34c0
RabbitMQ certification: Fixing publish/subscribe race condition. (#1271)
* Fixing publish/subscribe race condition in RabbitMQ component.
Added amqpReady function to tests to wait for RabbitMQ readiness.
Tweaks to when subscribed is signaled in the test.

* Update go.mod(s)
2021-11-03 16:37:52 -07:00
Phil Kedy 1d0ce9ce23
Kafka certification: Adding more consumers and verifying messages are consumed (#1262)
* Adding second consumer and verifying messages are consumed

* Removed commented out code

* Adding async steps

* Added third consumer

* Adding consumer rebalance test

* Renaming watcher variables

* Update README.md

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-11-02 13:51:15 -07:00
Amit Mor e9deaf3781
Snssqs subscription policy (#1259)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* gitignore

* restrict SQS send message policy

* linting mostly of unwrapped errors

* refactoring

* pr changes

* Update .gitignore

* Update dapr-bot-schedule.yml

* Update dapr-bot-schedule.yml

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-02 10:38:10 -07:00
Phil Kedy 9a3fe3f683
Kafka certification: Adding infra and network interruption tests (#1241)
* Adding infra and network interruption tests

* Tweak

* Using multierr.Combine

* Adding consumeRetryInterval setting to Kafka pubsub component
2021-10-27 17:41:27 -07:00
Long Dai 77bf26a676
Merge branch 'master' into add_amqp_properties 2021-10-26 09:29:57 +08:00
Taction c63fee5bec
Add tls support for kafka pubsub component (#1196)
* add tls support for kafka pubsub component

* fix review

* fix review

* delete unused json tag

Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-10-25 15:40:53 -07:00
Simon Leet 2006bcccb0 Support message-id in Event Hubs components 2021-10-25 18:12:16 +00:00
Dmitry Shmulevich 3b1249cf87
reconnect rabbitmq on 'unexpected command received' error (#1198) 2021-10-14 21:19:31 -07:00
Taction ca8cf5c1fc
Add dead letter support in RabbitMQ pubsub (#883)
* Add dead letter support

* fix lint

* add maxLen and maxLenBytes support

* Let the DLX and DLQ be free form.

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long0dai@foxmail.com>
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: cvictory <shenglicao2@gmail.com>
2021-10-11 09:49:14 -07:00
Dmitry Shmulevich 4795f75270
properly close channel in rabbitmq (#1190) 2021-10-07 20:25:36 -07:00
Joni Collinge 893da2a3ef
Enable ASB Message Metadata (#1071)
* add asb message metadata

* add asb message metadata

* fix lint

* add server metadata to receiver

* add metadata prefix

* fix tests

* alway set delivery count

* lint

* nolint on nested if

* lint fix

* remove trailing period on license header

Co-authored-by: Long Dai <long0dai@foxmail.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-10-07 14:05:05 -07:00
Dmitry Shmulevich 494596be0e
fix 'durable' setting in RabbitMQ (#1183) 2021-10-01 17:27:36 -07:00
Simon Leet a9cf218c04
Support setting up Azure IoT Hub for integration testing with EventHubs components (#1174)
* Add IoT Hub config to setup-azure-conf-test.sh

* Add EventHubs bindings integration test for IoT device event properties

* Add consumer group for IoT Hub events for pubsub EventHubs testing

* Add EventHubs pubsub integration test for IoT device event properties
2021-10-01 16:17:53 -07:00
Bernd Verst ac85ba2a8e
Add dapr user agent for Azure Components (#1152)
* Add dapr useragent for Service Bus and EventHubs

* Add useragent for tablestorage

* send dapr useragent

* more user agents

* formatting
2021-09-27 17:04:27 -07:00