Commit Graph

569 Commits

Author SHA1 Message Date
Ricardo Corrêa 027204f2d3
nack in error messages (#1562)
Signed-off-by: Ricardo Corrêa <r.c.correa@outlook.com>
2022-03-09 17:21:51 -08:00
Bernd Verst 800d22f4b0 Move errors to pseudo constant
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-02-24 15:44:09 -08:00
Bernd Verst e59a67a1c4 Add retriable error conditions for ASB sending
Signed-off-by: Bernd Verst <4535280+berndverst@users.noreply.github.com>
2022-02-24 15:33:49 -08:00
Taction 70a86e6d80
supports rabbitmq pubsub set exchangekind (#1520)
Signed-off-by: zhangchao <zchao9100@gmail.com>
2022-02-21 13:08:18 -08:00
1046102779 2126b96dc6 feature/pulsar: add producer option disableBatching
Signed-off-by: 1046102779 <seachen@tencent.com>
2022-02-16 12:07:02 +08:00
Gehhilfe 6ecc3b6f7b
Add support for nats jetstream wildcard subscriptions (#1465)
* Add support for nats jetstream wildcard subscriptions

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* Add support for nats jetstream wildcard subscriptions

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2022-02-15 09:46:33 -08:00
Looong Dai c4a64ef471
Merge branch 'master' into gcp_pubsub_reconnect_loop 2022-02-07 10:23:22 +08:00
Taction 7c7c977d52
Fix rabbitmq pubsub reconnect (#1471)
Signed-off-by: zhangchao <zchao9100@gmail.com>
2022-02-02 11:56:48 -08:00
Jigar 9078f11985
Added ContentType to pubsub/binding/state request-response (#1376)
Signed-off-by: jigargandhi <jigarr.gandhi@gmail.com>

Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2022-01-28 10:17:04 -08:00
yellow chicks e04fd24a07
feature/pulsar: provide token && http request (#1427)
* feature/pulsar: provide token && http request

Signed-off-by: 1046102779 <seachen@tencent.com>

* feature/pulsar: provide token && http request

Signed-off-by: 1046102779 <seachen@tencent.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
Co-authored-by: Yaron Schneider <schneider.yaron@live.com>
2022-01-25 21:23:56 -08:00
Gehhilfe b835eb4e1e
pubsub jetstream: add nkey based authentication (#1441)
* pubsub jetstream: add nkey based authentication

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* fix godot: comment should end in a period

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* go mod tidy

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* fix: go mod tidy

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* pubsub jetstream: add e2e test

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

* Cleanup not needed files and move jwt

Signed-off-by: Tim Burkert <burkert.tim@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2022-01-25 21:11:50 -08:00
Amit Mor db9e071449
bugfixes: visibility timeout, dlq queue attributes (#1446)
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2022-01-18 10:31:53 -08:00
Yordan Pavlov 022c67a57b add reconnect loop for GCP pub/sub subscriptions
Signed-off-by: Yordan Pavlov <yordan.pavlov@dunnhumby.com>
2022-01-17 17:45:00 +00:00
Amit Mor c8844ccaed
Pubsub AWS SNS/SQS - adding context, cancellation, timeouts, retrying \w backoff & disable delete of messages on failure (#1433)
* squash

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* comment

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* gofumpted

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* breakdown of metadata loading

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* metadata further refactoring

Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2022-01-13 07:27:49 -08:00
Shivam Kumar 4631d6e41a
MQTT Pubsub Certification Testing + AutAckOff Fix for MQTT (#1420)
* MQTT Certification Test

Signed-off-by: shivam <shivamkm07@gmail.com>

* Using paho.mqtt.golang fork with AutoAck fix

Signed-off-by: shivam <shivamkm07@gmail.com>

* Adding MQTT component in certification.yml

Signed-off-by: shivam <shivamkm07@gmail.com>

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2022-01-07 10:33:54 -08:00
1046102779 00c99a4027 feature/pulsar: add producer option disableBatching
Signed-off-by: 1046102779 <seachen@tencent.com>
2022-01-07 12:45:14 +08:00
Phil Kedy 265fa9a9c2
Kafka PubSub: Use metadata as message headers. (#1409)
* Kafka PubSub: Setting metadata as message headers.

Signed-off-by: Phil Kedy <phil.kedy@gmail.com>

* running go mod tidy on certification tests

Signed-off-by: Phil Kedy <phil.kedy@gmail.com>
2022-01-06 10:10:37 -08:00
Dmitry Shmulevich c7adb917f3
update license to Apache v2.0 (#1406) 2022-01-04 19:53:31 -08:00
Mukundan Sundararajan fe6c9e37d9
rewrite eventhubs component (#1292)
Signed-off-by: Mukundan Sundararajan <msundar.ms@outlook.com>
2022-01-03 19:36:35 -08:00
Zach 5c9365b314
pub/sub rocketmq:upgrade with v2 (#1383)
* pub/sub rocketmq:upgrade with client v2

Signed-off-by: zach <zachchou016@gmail.com>

* pubsub/rocketmq:remove cache and add start with setup

Signed-off-by: zach <zachchou016@gmail.com>

* pubsub/rocketmq:fix variable golint

Signed-off-by: zach <zachchou016@gmail.com>

* pubsub/rocketmq:change license and fix goimports

Signed-off-by: zach <zachchou016@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2021-12-31 12:18:27 -08:00
Scott Hussey 897c2a4db5
Add OAuth2/OIDC support to Kafka pubsub (#1229)
* parent dbb18e4d77
author Scott Hussey <sthussey@gmail.com> 1632277042 -0500
committer Scott Hussey <sthussey@gmail.com> 1639111222 -0600

Support Oauth2 authentication for Kafka

- Utilize the SASL OAUTHBEARER mechanism to support
  the Oauth2 client_credentials flow for Kafka
  authentication

- Deprecate `authRequired` field and introduce `authType`
  to support varied authentication mechanisms

- Add a metadata upgrade mechanism to support backwards
  compatability

- Recommend broker specific scopes to guard against a
  compromised broker replaying a token

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* kafka test - use volumes instead of root

CI test fails due to losing data written to container
root

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Clean up volumes

- When bringing down the docker-compose context, cleanup
  volumes

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Clean up stale comment

Signed-off-by: Scott Hussey <sthussey@gmail.com>

* Resume config defaults

- Return some Kafka config to default values to lower config
  footprint and stabilize flaky CI runs

Signed-off-by: Scott Hussey <sthussey@gmail.com>
2021-12-28 10:34:28 -08:00
Taction 6d9d0bca35
fix nits: duplicated err handle (#1341)
* remove duplicated err handle

Signed-off-by: Taction <zchao9100@gmail.com>

* fix test go mod tidy

Signed-off-by: Taction <zchao9100@gmail.com>
2021-12-27 08:47:10 -08:00
Amit Mor b548aa8bcc
1385 disable policy creation (#1386)
* not modifying policy if disableEntity is set

Signed-off-by: Amit Mor <amit.mor@hotmail.com>

* not modifying policy if disableEntity is set

Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2021-12-22 12:42:28 -08:00
Dmitry Shmulevich 7a631ef7aa
pub/sub rabbitmq: add publishing retries (#1382)
Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com>

Co-authored-by: Looong Dai <long.dai@intel.com>
2021-12-17 15:59:38 -08:00
Dmitry Shmulevich 3fcf50c044
pub/sub rabbitmq: fix subscribe error (#1377)
Signed-off-by: Dmitry Shmulevich <dmitry.shmulevich@gmail.com>
2021-12-14 08:22:35 -08:00
Amit Mor 72bae26380
disable entity creation (#1375)
Signed-off-by: Amit Mor <amit.mor@hotmail.com>
2021-12-13 09:22:57 -08:00
Amit Mor e97b1bf9f3
1335 bugfix #2 (#1368)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* Update snssqs.go

* dapr bot schedule

* read and append queue attributes

* unnecessary escaping in json tag

* unexporting structs

* bugfix in policy

* bugfix in policy. merged from master

* fifo suffix as const
2021-12-08 14:20:24 -08:00
Amit Mor 4918900c09
AWS PubSub SNS/SQS - FIFO support + refactoring (#1312)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* wrapping errors

* fifo and naming sanitization

* fifo and name santization tests

* component carrying id

* Update snssqs.go

removed unneeded import

* Update snssqs.go

* add missing github actions file

* linting

* fix go.mod

* schedule yml file

* const max lenght, bugfix in ack

* dapr bot schedule

Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-30 10:52:00 -08:00
Amit Mor da7e8cfde7
#1335 bugfix - multi SNS topics publishing policy to SQS (#1337)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* Update snssqs.go

* dapr bot schedule

* read and append queue attributes

* unnecessary escaping in json tag

* unexporting structs
2021-11-26 08:39:56 -08:00
huazhongming 63d6329ae5
Support tracestate can be propagated in PubSub (#1264)
* support tracestate can be propagated in PubSub

* add trace state

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-24 12:30:35 -08:00
Taction 4bf0dcbfcf
Fix the Route of subscription API does not work on pubsub.pulsar (#1283)
* feat(pulsar): add tenant and namesapce. fix topic parsing.

* add test

* fix review add topic format unit test

* fix review: add persistent to metadata, fix nits

* fix lint

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Mukundan Sundararajan <musundar@microsoft.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-22 18:57:53 -08:00
halspang d5a68041c9
Move Service Bus Pubsub/Binding to common auth (#1201)
* Move Service Bus Pubsub/Binding to common auth

Both the pubsub and input/output binding for Azure Service Bus were
connecting via a connection string. This is still supported but will
now fallback to using AAD from the common auth library. This is also
the recommended auth pattern going forward.

* Move AMPQ specific auth and fix linter issues

* Make conn string and namespace mutually exclusive

* Move resourceName to a constant

* Update auth_amqp.go

* Update auth.go

Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-22 18:20:11 -08:00
Dmitry Shmulevich ad433a27e1
add troubleshooting info (#1278)
add retries for pubsub subscribe
fix race conditions
2021-11-15 16:33:21 -08:00
Sergey Krutsko a269d62138 change assert -> require 2021-11-05 10:22:13 +03:00
Sergey Krutsko 418fd96299 add kafkaVersion 2021-11-04 15:54:03 +03:00
Phil Kedy 77564a34c0
RabbitMQ certification: Fixing publish/subscribe race condition. (#1271)
* Fixing publish/subscribe race condition in RabbitMQ component.
Added amqpReady function to tests to wait for RabbitMQ readiness.
Tweaks to when subscribed is signaled in the test.

* Update go.mod(s)
2021-11-03 16:37:52 -07:00
Phil Kedy 1d0ce9ce23
Kafka certification: Adding more consumers and verifying messages are consumed (#1262)
* Adding second consumer and verifying messages are consumed

* Removed commented out code

* Adding async steps

* Added third consumer

* Adding consumer rebalance test

* Renaming watcher variables

* Update README.md

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-11-02 13:51:15 -07:00
Amit Mor e9deaf3781
Snssqs subscription policy (#1259)
* bugfix for sns topic deletion upon termination

* removed upstream github workflow files

* gitignore

* restrict SQS send message policy

* linting mostly of unwrapped errors

* refactoring

* pr changes

* Update .gitignore

* Update dapr-bot-schedule.yml

* Update dapr-bot-schedule.yml

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-11-02 10:38:10 -07:00
Phil Kedy 9a3fe3f683
Kafka certification: Adding infra and network interruption tests (#1241)
* Adding infra and network interruption tests

* Tweak

* Using multierr.Combine

* Adding consumeRetryInterval setting to Kafka pubsub component
2021-10-27 17:41:27 -07:00
Long Dai 77bf26a676
Merge branch 'master' into add_amqp_properties 2021-10-26 09:29:57 +08:00
Taction c63fee5bec
Add tls support for kafka pubsub component (#1196)
* add tls support for kafka pubsub component

* fix review

* fix review

* delete unused json tag

Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-10-25 15:40:53 -07:00
Simon Leet 2006bcccb0 Support message-id in Event Hubs components 2021-10-25 18:12:16 +00:00
Dmitry Shmulevich 3b1249cf87
reconnect rabbitmq on 'unexpected command received' error (#1198) 2021-10-14 21:19:31 -07:00
Taction ca8cf5c1fc
Add dead letter support in RabbitMQ pubsub (#883)
* Add dead letter support

* fix lint

* add maxLen and maxLenBytes support

* Let the DLX and DLQ be free form.

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Long Dai <long0dai@foxmail.com>
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Long Dai <long.dai@intel.com>
Co-authored-by: Ian Luo <ian.luo@gmail.com>
Co-authored-by: cvictory <shenglicao2@gmail.com>
2021-10-11 09:49:14 -07:00
Dmitry Shmulevich 4795f75270
properly close channel in rabbitmq (#1190) 2021-10-07 20:25:36 -07:00
Joni Collinge 893da2a3ef
Enable ASB Message Metadata (#1071)
* add asb message metadata

* add asb message metadata

* fix lint

* add server metadata to receiver

* add metadata prefix

* fix tests

* alway set delivery count

* lint

* nolint on nested if

* lint fix

* remove trailing period on license header

Co-authored-by: Long Dai <long0dai@foxmail.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-10-07 14:05:05 -07:00
Dmitry Shmulevich 494596be0e
fix 'durable' setting in RabbitMQ (#1183) 2021-10-01 17:27:36 -07:00
Simon Leet a9cf218c04
Support setting up Azure IoT Hub for integration testing with EventHubs components (#1174)
* Add IoT Hub config to setup-azure-conf-test.sh

* Add EventHubs bindings integration test for IoT device event properties

* Add consumer group for IoT Hub events for pubsub EventHubs testing

* Add EventHubs pubsub integration test for IoT device event properties
2021-10-01 16:17:53 -07:00
Bernd Verst ac85ba2a8e
Add dapr user agent for Azure Components (#1152)
* Add dapr useragent for Service Bus and EventHubs

* Add useragent for tablestorage

* send dapr useragent

* more user agents

* formatting
2021-09-27 17:04:27 -07:00
Long Dai 988fed05ab
ci: standard linter config (#1102)
* ci: standard linter config

Signed-off-by: Long <long.dai@intel.com>

* Update utils.go

* fix pulsar issue

Signed-off-by: Long <long.dai@intel.com>

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-21 15:21:59 -07:00
yellow chicks c425f1319a
feature/pubsub: add delay queue params for pulsar (#1112)
* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-17 11:41:47 -07:00
Phil Kedy a0542ec922
Adding `initialOffset` option to Kafka binding and pubsub components (#1120)
* Adding `initialOffset` option to Kafka binding and pubsub components

* Only allow oldest and newest

* Fix tests

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-02 18:02:10 -07:00
Manuel Riezebosch 18cbe25e4e
feat(pubsub): add in-memory pub/sub (#1106) 2021-09-02 17:31:17 -07:00
Jaime Piña f4a5722806
Add JetStream support (#1101)
* Add JetStream support

* Add conformance tests

* Fix lint

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-09-01 11:50:16 -07:00
Luke Kennedy 8fd1ee57a0
GCP Pub/Sub Message Ordering Configuration (#1099)
* Add configuration for enabling message ordering

* Simplify passing the metadata

* Update field order to satisfy alignment linter

Co-authored-by: Luke Kennedy <luke.kennedy@optiisolutions.com>
2021-08-30 08:54:46 -07:00
Amit Mor 284d74488f
pubsub/aws/snssqs to support SQS dead-letter queue (#1066)
* bugfix for sns topic deletion upon termination

* Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* wip on normalizing queue/topic names

* sanitize queue and topic names

* sanitized names. bugfix for close

* # This is a combination of 4 commits.
# This is the 1st commit message:

Improve error message in case of missing property (#1012)

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
# This is the commit message #2:

Remove vestigial pubsub/nats code (#1024)

The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
# This is the commit message #3:

bugfix for sns topic deletion upon termination

# This is the commit message #4:

Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* removed debug message

* raw string abort

* merge issues solved

* wip

* gofmt+remove regex and use byte iter

* wip. first impl of dead-letters queue config

* wip. refactor and fallback values

* integration test wip

* wip integration test

* wip integration

* wip on testing

* wip

* still buggy but wip!

* bugfix in dlq creation

* working. still bug in subscription clean up

* Update snssqs_integ_test.go

* golangci-lint fixes

* golangci-lint refactoring

* trying to skip running integrations for snssqs

* testing

* skip integration test if no AWS related envvars are set (skip in CI)
* parallel testing in unittests

* code review fixes

* not using implicit maxReceives
* maxReceives renamed
* unittest refactor

* Update snssqs.go

* integ removed, renaming back of const

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-08-13 19:02:25 -07:00
weibubli 4d3e4ebee5
rename (#1061)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-08-10 16:24:15 -07:00
Amit Mor 253ef854c5
SNS and SQS topic/queue names + SNS subscription deletion bugfix (#1035)
* bugfix for sns topic deletion upon termination

* Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* wip on normalizing queue/topic names

* sanitize queue and topic names

* sanitized names. bugfix for close

* # This is a combination of 4 commits.
# This is the 1st commit message:

Improve error message in case of missing property (#1012)

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
# This is the commit message #2:

Remove vestigial pubsub/nats code (#1024)

The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
# This is the commit message #3:

bugfix for sns topic deletion upon termination

# This is the commit message #4:

Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* removed debug message

* raw string abort

* gofmt+remove regex and use byte iter

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-08-06 16:31:50 -07:00
Phil Kedy b73d817582
Merge changes from Release 1.3 to master (#1038)
* Fixing the handling of detach errors (#1030)

* Improve error message in case of missing property (#1012)

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>

* Remove vestigial pubsub/nats code (#1024)

The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.

* Fixing the handling of detach errors

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>

* Revert "Fixing the handling of detach errors (#1030)" (#1031)

This reverts commit 54840c2b83.

* Fixing the handling of detach errors (#1032)

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
2021-08-06 10:46:31 -07:00
Simon Leet dc7ee55e2a
Add Event Hubs pubsub and bindings conformance tests (#1040)
* Add conformance test for EventHubs bindings

* Update EventHubs pubsub to pass conformance tests

- Add conformance tests for Azure EventHubs PubSub component
- Add retry & backoff handling on subscriber handling error to EventHubs
  PubSub component for conformance tests.
- Add cancellation context to Azure EventHubs and update Close() to
  invoke cancel prior to closing the hub, which cleans up both senders
  and receivers.
2021-08-02 18:03:24 -07:00
Simon Leet f84a8b8914
Fix Kafka pubsub use of AuthRequired metadata (#1015)
Kafka pubsub component was not setting `k.authRequired` property based
on parsed `meta.AuthRequired` value, so would not correctly configure
Kafka producer or consumer with appropriate credentials when specified
by the author.

- Fix assignment of `meta.AuthRequired` to `k.authRequired` consumed by
  Subscribe().
- Fix initialization of `SyncProducer` to use copy of config with the
  authorization credentials added.

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-07-28 10:24:35 -07:00
Simon Leet f58e0ca26d
Remove vestigial pubsub/nats code (#1024)
The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
2021-07-19 17:46:32 -07:00
Simon Leet 3877f5c793
Return SystemProperties on events from EventHubs bindings and pubsub (#1009)
- Update azure-event-hubs-go dependency to currently supported v3 with
  IoT Hub SystemProperties.
- Write Event.SystemProperties to ReadResponse.Metadata for EventHubs
  input binding.
- Write Event.SystemProperties to NewMessage.Metadata for EventHubs
  pubsub.
2021-07-13 16:04:22 -07:00
Ian Luo 7d2bc9bbdf
fix: Dapr runtime panic when handle Pub/Sub (dapr#3281) (#967)
* fix: Dapr runtime panic when handle Pub/Sub (dapr#3281)

* use uuid to generate Id when Id is missing

* skip default value for id and content-type

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: beiwei.ly <beiwei.ly@alibaba-inc.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-07-08 10:54:36 -07:00
Long Dai fe5e14b78b
Azure Service Bus: ignore amqp🔗detach-forced error (#985)
Signed-off-by: Long Dai <long0dai@foxmail.com>
2021-06-30 17:12:51 -07:00
iamazy 74af5b6528
rename ConsumerID to ConsumerGroup (#965)
* rename ConsumerID to ConsumerGroup

* rename consumerID to consumerGroup

* update the config in the conformance test

* set ConsumerID as alias of ConsumerGroup

* remove extra space

* gofmt

* add ClientID in kafka metadata

* remove ConsumerID field

* gofmt

* Update kafka.go

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-23 10:33:27 -07:00
Phil Kedy 8e6b36ecdf
Importing the retry and config packages from kit instead of internal (#940)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-22 09:38:55 -07:00
Taction 14870cb7bc
Add backoff for rabbitmq PubSub component (#862)
* Add backoff for rabbitmq

* fix lint

* fix lint

* remove backOffEnable config

* Delete backoff enable metadata

* Add a default config setting max retry to 0

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-06-10 00:19:46 -07:00
iamazy a59dcc44eb
Fix typo (#927)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-06-09 23:58:15 -07:00
Taction 7512c77a28
Rabbitmq pubsub component stop has not be well handled (#860)
* fix pubsub rabbitmq stop

* fix lint

* Add read lock while reading stopped

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-09 23:23:06 -07:00
Taction 464d53d94b
Move the common Redis code to internal (#859)
* Move pubsub redis client to internal

* Add failover for common redis. Refactor state redis.

* refactor binding redis

* fix lint

* Refactor to setting partten

* fix db and lint

* fix lint

* for backward compatibility

* Using a type alias to handle decoding duration values of -1 for Redis.

* Linter fixes

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-06-06 23:24:36 -07:00
Long Dai 4ee732e0bd
ci: enable gofumt linter (#887)
Signed-off-by: Long Dai <long0dai@foxmail.com>
2021-05-27 22:21:24 -07:00
Taction 334728a51f
Fix race in pulsar pubsub component (#868)
* Fix race in pulsar pubsub component

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-13 22:50:17 -07:00
cinience 60ad18f505
Add pubsub rocketmq (#816)
* Add pubsub rocketmq

* Fix lint

* Update parseMetadata

* Update logger import

* Tweaks to use the "settings pattern"

* Support backOffConfig

* Fix err log

* Update go.mod

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-05-13 22:31:41 -07:00
Artur Souza 3cc0472a41
Add function to generate CloudEvent for raw subscribed events. (#864)
* Add function to generate CloudEvent for raw subscribed events.

* address code comments.
2021-05-10 10:39:26 -07:00
Phil Kedy 1b30e75e60
Configurable retry (#854)
* no message

* Switching over usages of retry to the new package and making use of the DecodeConfig function

* Added decoding configuration using settings with a specific prefix

* Linter fixes

* Fixing linter error

* time.Sleep is the enemy

* Fix typo in comment

* Moving config to a pointer parameter so that the component can pass in the config with default values that make sense for that component.

* Renamed config struct

* Fix comment

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-06 09:43:57 -07:00
Charlie Stanley f69b9f5c34
Update redis metadata comments to reflect the new maxRetry default from v8 (#856)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-05 10:36:49 -07:00
Taction a173e65ae7
Merge branch 'master' into upgrade_redis_client_to_v8 2021-05-04 23:19:54 +08:00
Charlie Stanley 053ca0b859
Do not block indefinitely on XGroupRead for redis subscriber (#855) 2021-05-03 17:44:20 -07:00
zhangchao 651d754496 Merge branch 'master' into upgrade_redis_client_to_v8
* master:
  add handle timeout to cancel req for it would be redelivered (#832)
  Add optional metadata param maxLen for redis stream PubSub component (#835)
  Implement bindings for the Zeebe workflow engine (#806)
  Automerge in contrib. (#849)

# Conflicts:
#	go.mod
#	go.sum
#	pubsub/redis/redis.go
2021-04-29 11:49:26 +08:00
Taction 61a47e8f56
add handle timeout to cancel req for it would be redelivered (#832)
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-04-28 19:23:36 -07:00
Taction f376450d7c
Add optional metadata param maxLen for redis stream PubSub component (#835)
* Add optional metadata param maxLen for redis stream PubSub component

* fix fmt

* fix error info

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-04-28 18:20:48 -07:00
zhangchao 4c0aef5c4e fix use r.ctx instead of context.Background() in pubsub redis component 2021-04-23 21:57:44 +08:00
zhangchao 5190fe4e7d Merge branch 'master' into upgrade_redis_client_to_v8
resolve conflict

* master:
  Increasing overall test timeout to allow for conformance tests to run (#834)
  Improve SMTP binding performance (#782)
  add redis cluter mode for pubsub (#823)
  downgrade the verbosity of lock renewal errors from warn to debug. This error is expected when messages are processed before beingrenewed (#803)
  Improve pulsar publisher performance. (#827)
  optimze(state/redis): output detail redis error (#825)
  Consul nameresolution component (#750)
  Add DNS name resolver. Add unit tests. (#743)
  Azure Service Bus: Changed log level of error on close to info (#830)
  Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807)
  Update state/blobstorage.go to add metadata. (#786)
  Azure Service Bus publishing retries when server is "too busy" (#821)

# Conflicts:
#	bindings/redis/redis.go
#	go.mod
#	go.sum
#	pubsub/redis/redis.go
2021-04-22 09:13:01 +08:00
Taction 466426894b
add redis cluter mode for pubsub (#823)
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-04-21 11:33:30 -07:00
Joni Collinge 744e5be6fa
downgrade the verbosity of lock renewal errors from warn to debug. This error is expected when messages are processed before beingrenewed (#803)
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-04-21 11:01:19 -07:00
Xueqian Wang f761b51c77
Improve pulsar publisher performance. (#827)
* Improve pulsar publisher performance.

* Addressed comments and added some debug logs

* Use peek instead of get for closing producers

* Fixed lint issue

* Added debug log in close

* Change back o 100

* Close producer before client

* Fixed cached number of producer and added todo for making it configurable

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-04-21 10:53:07 -07:00
Phil Kedy edfa3e5473
Azure Service Bus: Changed log level of error on close to info (#830) 2021-04-19 17:08:27 -07:00
Phil Kedy 688df9bda4
Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807) 2021-04-19 12:06:10 -07:00
Phil Kedy 58b43fbfdb
Azure Service Bus publishing retries when server is "too busy" (#821)
* Adding retry/backoff to publishing to Azure Service Bus when the server is too busy and throttles the connection

* Fixing linter issues
2021-04-14 11:56:55 -07:00
zhangchao 3ec4b336a9 upgrade go-redis/redis to v8, and use context in pubsub/redis to cancel redis request. 2021-04-10 23:41:58 +08:00
Artur Souza 3993d9a559
Merge branch 'master' into jjcollinge/pubsub-context 2021-04-08 14:11:01 -07:00
Phil Kedy 6803b49e56 tls -> ssl for good measure 2021-04-06 17:50:42 -04:00
Phil Kedy bc0c4fd72e Adding small coalesce on scheme to ensure backward compatibility in both the pubsub and binding mqtt components 2021-04-06 17:32:47 -04:00
Phil Kedy 810ec429a7 Adding small coalesce on scheme to ensure backward compatibility in both the pubsub and binding mqtt components 2021-04-06 17:30:33 -04:00
Joni Collinge 617392fcca
Merge branch 'master' into jjcollinge/pubsub-context 2021-03-28 18:50:28 +01:00
zhangchao 9304037b70 rename backoff* to redis* 2021-03-24 20:58:56 +08:00
Joni Collinge 36ed5ec84f use existing context for pubsub comps 2021-03-23 20:12:46 +00:00
Joni Collinge 2d0864d25c use existing context for gcp 2021-03-23 19:45:35 +00:00
Joni Collinge 9fd461b255 Merge branch 'master' into jjcollinge/pubsub-context 2021-03-23 17:45:05 +00:00
Phil Kedy e9095e5081
Merge branch 'master' into add_more_config_for_redis 2021-03-22 18:11:04 -04:00
Trond Hindenes e5d686b56b
refactor GCP pubsub (#760)
* started on refactoring gcp pubsub

* Maybe this works?

* Working implicit auth

* Some fixes

* Updated tests

* Fix lint errors, tests

* trigger ci

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-03-22 14:47:05 -07:00
Phil Kedy a750f3c698
Merge branch 'master' into add_more_config_for_redis 2021-03-19 20:08:32 -04:00
Taction fb09a07e0f
fix pubsub redis close bug (#769)
When component close function called `r.cancel()` and `r.client.Close()` will be excuted.
The pollNewMessagesLoop loop use client to read from redis which will lead to err. And when err occured this loop will continue.
We need to check context on the top of this loop.
2021-03-19 17:01:03 -07:00
zhangchao ddbb039210 rename maxRetries minRetryBackoff maxRetryBackoff to backoffMaxRetries backOffMinInterval backOffMaxInterval. to be consistent with other components 2021-03-19 12:50:09 +08:00
zhangchao 1d376f3f3b fix lint 2021-03-18 22:40:19 +08:00
zhangchao 22f0f43e9b fix: add more connection settings 2021-03-18 22:01:57 +08:00
zhangchao fd1a544877 fix: parse duration-based settings as a time.Duration like redeliverInterval and processingTimeout are and remove the InSec suffix 2021-03-18 21:24:24 +08:00
zhangchao f433307fb1 add some redis client config that may be commonly used 2021-03-17 18:19:48 +08:00
Joni Collinge ca62139f47
Make connection recovery configurable using metadata (#713)
* add metadata properties to control max reconn attempts and recovery delay

* add metadata tests

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-03-12 12:41:38 -08:00
Phil Kedy a1a0ad2c25
Adding RabbitMQ to the conformance tests (#739)
* Initial pass at adding RabbitMQ to the conformance tests

* Fixing unit tests
2021-03-04 11:56:35 -08:00
Phil Kedy 44d64598b8
Added Hazelcast to conformance tests and retry to prevent message loss (#724) 2021-03-03 13:08:58 -08:00
Phil Kedy e8efbca072
Add MQTT pubsub to the conformance tests (#721)
* Add MQTT pubsub to the conformance tests

* Making the MQTT pubsub component conformance test against multiple brokers
2021-02-25 18:16:23 -08:00
Joni Collinge 2ff6ca4302 merge master 2021-02-25 16:15:26 +00:00
Joni Collinge c054ae6ada Merge branch 'master' into jjcollinge/pubsub-context 2021-02-25 16:15:06 +00:00
Phil Kedy 4a0011bcbd
Improve conformance tests and add pulsar (#702)
* Improve conformance tests and add pulsar

* Fix linter error

* Fix linter errors

* Tweaks to pulsar pubsub per PR

* Switching uuid import

* Using Stringer interface to print message keys as base64 in logger

* Disabling some checks w/ Redis pubsub until the component can be enhanced

* Adding comment to eventually remove the simulateErrors option

* Add comment asBase64String to explain what it does

* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing

* Fixing linter issues

* using a wait group instead of a sleep to wait for processing to complete

* More comments and refactoring

* comment tweaks

* Tweaks per PR

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-02-23 13:31:52 -08:00
Artur Souza dd3e463b52
Fix pubsub retry for Kafka (#688)
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.

* Adding backoff/retry logic to the Kafka pub/sub component

* Add logic to return if error from backoff in kafka.

* Addressing PR comments and lint.

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-02-12 00:08:46 -08:00
Yaron Schneider 3ef025c604
change headers (#679) 2021-02-09 18:57:55 -08:00
Artur Souza 859f0ae881
Handles binary data in cloud event. (#674) 2021-02-09 09:45:38 -08:00
Yaron Schneider 58e92479ac
Add kafka max message bytes and pubsub partition key (#669)
* add kafka max message bytes and pubsub partition key

* linter
2021-02-08 11:14:52 -08:00
Yaron Schneider bc0988af42
fix missing subject (#640) 2021-01-28 08:37:36 -08:00
Yaron Schneider 4ca679172f
Fix service bus new topic creation (#636)
* fix service bus new topic creation

* place topic in list
2021-01-26 13:49:51 -08:00
Jigar 27344d4e4c
Added binary data type detection (#631)
* Added binary data type detection
Added base64 encoding for binary data types

* Update utils.go

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-01-25 23:08:47 -08:00
Yaron Schneider 9fae71dada
Fix gcp pub/sub multiple subscriptions (#607)
* fix gcp pub/sub multiple subscriptions

* linter
2021-01-15 13:23:30 -08:00
Artur Souza 40e3d60d72
Remove useless random ID logic for mqtt's consumerID. (#605) 2021-01-15 11:56:54 -08:00
Artur Souza 01147e559d
Refactor and fix contenttype logic for pubsub. (#602)
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-01-14 14:28:52 -08:00
Yaron Schneider d7eeb2413f
Fix RabbitMQ concurrency (#576)
* fix rabbitmq concurrency

* linter

* add concurrency validation

* added tests and validation
2021-01-07 19:29:58 -08:00
Yaron Schneider 7465e4092d
Add concurrency mode to rabbitMQ pub/sub (#575)
* add concurrency mode to rabbitMQ pub/sub

* linter

* simplify
2021-01-07 15:02:58 -08:00
Yaron Schneider 80e5f2ca52
fix data serialization in cloudevent (#571)
* fix data serialization in cloudevent

* fixes
2021-01-04 17:23:07 -08:00
Yaron Schneider a99b605db8
extend custom cloud event (#570) 2021-01-04 12:12:39 -08:00
Yaron Schneider b7584f670b
Add new cloudevents creation method (#568)
* add new cloudevents creation method

* linter
2020-12-30 19:10:47 -08:00
Yaron Schneider a7ff1b0aad
Replace content cloud event check, add settraceid (#567)
* replace content cloud event check, add settraceid

* change cloudeventenvelope to map

* change settraceid method
2020-12-30 17:06:32 -08:00
Yaron Schneider 3e7f473b0c
Add method to check if payload is a cloudevent (#566)
* add method to check if payload is a cloudevent

* linter

* remove old test
2020-12-30 12:26:22 -08:00
Artur Souza e6dadfab6c
TTL in PubSub. (#565)
* TTL in PubSub.

* Handle message TTL only if component cannot handle it.

* Removig dead code for overflow check in pubsub expiration.
2020-12-29 14:28:52 -08:00
Trond Hindenes 660185b08c
Add support for prefetch (#562)
* Add support for prefetch

* Fixed test implementation

* Gofmt-ed files

* Re-arranged struct

* Moar tests
2020-12-21 10:42:16 -08:00
Joni Collinge 31828b3779
Update ASB message finalization (#534)
* remove servicebus context hack

* add more verbose error message

* make error message lowercase

* update go-autorest

* update azure-sdk-for-go

* remove go mod replace go-autorest

* fix lint issues

* go mod tidy

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-12-04 22:13:00 -08:00
Joni Collinge 256f920918 fix lint 2020-11-30 16:47:01 +00:00
Joni Collinge 96a8a4d7ac Merge branch 'master' into jjcollinge/pubsub-context 2020-11-30 16:21:44 +00:00
cvictory 87933ad4bc
The subscribeRequest and PublishRequest both need Metadata. (#543)
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-26 23:15:54 -08:00
Artur Souza 2f4be95b84
Respect content-type for PubSub data. (#541) 2020-11-25 17:20:20 -08:00
Joni Collinge 704cb42383
add pubsub request update (#530)
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-25 17:12:16 -08:00
Trond Hindenes 8d6978372f
Various AWS improvements (#525)
* Support sessiontoken

* Fixed tests and other components

* Fixed tests and things

* fmt

* Fix lint errors

* gofmt

* Fixed lint bugs

* Remove unneeded parameter

* gofmt

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-23 13:21:29 -08:00
Artur Souza 6266934d3f
Auto-reconnect rabbitmq pubsub (#519) 2020-11-09 12:27:53 -08:00
Joni Collinge ff2df86e3c Merge branch 'jjcollinge/pubsub-context' of github.com:jjcollinge/components-contrib into jjcollinge/pubsub-context 2020-11-03 08:29:49 +00:00
Joni Collinge d62e2a9e84 use context.Background() in eventhubs subscriber 2020-11-03 08:29:00 +00:00
Joni Collinge d2016a2e2d
Merge branch 'master' into jjcollinge/pubsub-context 2020-11-02 18:19:48 +00:00
Joni Collinge 7ea3800afd fix lint 2020-11-02 17:20:29 +00:00
AbserAri d09758b502
add ackoptions for nats (#511)
* add ackoptions for nats

* remove redundent variable as golint.

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-02 07:59:58 -08:00
Joni Collinge a1ed1c1662 update pubsub and mqtt 2020-11-02 09:54:54 +00:00
Joni Collinge 2e6b84ae40 Merge remote-tracking branch 'origin/master' into jjcollinge/pubsub-context 2020-11-02 09:05:16 +00:00
Yaron Schneider df5be19205
add close method to event hubs pub/sub (#500) 2020-10-15 08:42:14 -07:00
Yaron Schneider 182d204111
Add close method to pub/sub (#494)
* add close method to pub/sub

* linter
2020-10-12 19:06:33 -07:00
halspang f647d9eedb
Upgrade golang lint to 1.31 (#490)
Upgrade golang lint to 1.31.0

https://github.com/dapr/components-contrib/issues/439
2020-10-12 12:54:15 -07:00
Yaron Schneider 653bdced85
add metadata to pubsub message (#477)
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-09-24 12:04:42 -07:00
Artur Souza fbd04241b0
Adding PubSub response type. (#453) 2020-09-04 12:59:43 -07:00
AbserAri 26b085baba
update logger form nats-io's to dapr's (#449)
* update logger form nats-io's to dapr's

* run go mod tidy and go mod delete the nats-io gnatsd

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-09-03 18:58:34 -07:00
Mayank Kumar 7f9850a04f
Improve mqtt pubsub for multiple topics (#443)
- Listen on multiple topics when those are provided for subscription.
- Improve the current logic to use dedicated function from paho client.
2020-09-03 10:17:27 -07:00
Brightside Automation c344f68718 Merge branch 'master' into jjcollinge/pubsub-context 2020-08-19 14:46:03 +01:00
Yaron Schneider 677e551844
fix consumer exclusivity (#435) 2020-08-17 07:37:25 -07:00
Mayank Kumar f5f807ca73
Add pubsub support for mqtt over TLS (#429)
* Add pubsub support for mqtt over TLS

* Update mqtt host to include uri scheme.

- This is required to identify mode of connection used while
  connecting to broker.

* Add pubsub support for mqtt over TLS.

- add validation while initialisation.
- update test cases.
2020-08-16 11:01:12 -07:00
Leon Mai 08d2538796
Corresponding changes for multi pubsub (#434)
* Corresponding changes for multi pubsub

* Fix quote
2020-08-14 11:05:51 -07:00
jjcollinge 38a8948bd8 Merge branch 'jjcollinge/pubsub-context' of github.com:jjcollinge/components-contrib into jjcollinge/pubsub-context 2020-07-28 10:04:26 +01:00
jjcollinge 19442e67c8 update pubsub components to use new context 2020-07-28 10:03:23 +01:00
Joni Collinge f868dc9497
Merge branch 'master' into jjcollinge/pubsub-context 2020-07-28 09:25:34 +01:00
Yaron Schneider 5892feca63
update Cloud Events to 1.0 (#419) 2020-07-23 17:13:26 -07:00
Ally Parker cf3ee2a46a
Add topic to cloud events envelope (#396)
Co-authored-by: ibrahim.dursun <ibrahim.dursun@gmail.com>
2020-07-22 08:48:32 -07:00
Mark Chmarny 05e71ffdc2
data serialized object (#391)
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-07-13 13:31:32 -07:00
Mark Chmarny dbc1a76dcf lint fix 2020-07-13 09:24:51 -07:00
Mark Chmarny 1589031af6 fiixes json array envelope parsing 2020-07-13 09:20:54 -07:00
Mark Chmarny 68c5a5537c changes cloudevent identification 2020-07-08 15:28:54 -07:00
Mark Chmarny f1f69bf3c8 fixes speeling mistakes in ccmments 2020-07-08 11:23:23 -07:00
Mark Chmarny 43092998ab uses provided CE as is, adds CE test 2020-07-08 06:50:10 -07:00
Mark Chmarny 194ff4f914 fixes double envelop 2020-07-07 16:47:47 -07:00
jjcollinge 07b046f140 rename asb handler type to handle 2020-07-07 13:01:24 +01:00
jjcollinge 7734d05288 Merge branch 'master' into jjcollinge/pubsub-context 2020-07-07 12:41:58 +01:00
jjcollinge 410628af3d add context to pubsub handler signature 2020-07-07 12:37:15 +01:00
Joni Collinge 4b2252b017
ASB lock renewal per topic (#365)
* bump asb sdk version to 0.10.2

* add isolation per topic for lock renewal

* added cancellations on async handlers

* refactored out subscription logic from namespace

* updated error handling

* defer async context cancel

* fixed lint issues

* add connection recovery

* fix lint

* update comments

* update comments

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-07-02 08:02:42 -07:00
Kyle Housley 9ef4fb5fb4
Make enableTLS work correctly (#373)
the enabletls does the opposite of what the config suggests
2020-06-29 14:39:55 -07:00
Mukundan Sundararajan 5c62707aba
Upgrade go version to 1.14.3 and linter version to 1.26.0 (#352)
* Upgrade to go 1.14.3 and linter version to 1.26.0
2020-05-29 15:26:33 -07:00
Trond Hindenes d3e9a7b3a7
Use shared aws auth package (#347)
* Use shared sqs auth package

* Goimports-ed files

* Added support for endpoint param in aws credentials abstraction, since the sqs-sns pubsub module required it

* Added endpoint support for all aws things

* Updated tests
2020-05-26 16:27:16 -07:00
Mayank Kumar 3ed4f0f63c
Add pubsub support for mqtt (#330)
* Add pubsub support for mqtt

- update README.md
- add tests for metadata parsing
- linting

* Add pubsub support for mqtt

- remove topic field from metadata.

* Add pubsub support for mqtt

- use consumerID as clientID for mqtt connection

* Add pubsub support for mqtt

- address review comments
2020-05-22 20:39:09 -07:00
Joni Collinge 1461b0e4a5
Fix issue with ASB pubsub throttling (#329)
* Applied throttling to receiver loop

* added init lock renewal

* added prefect count config option

* renew message locks before processing

* moved lock acquisition per go routine

* fix test

* added max active message throttles

* rename numConcurrentHandlers to maxConcurrentHandlers

* fix typo in abandonMessage func name

* set maxActiveMessages to minimum of prefetchCount

* remove prefetch count constraint

* removed conditional activeMessage tracking

Co-authored-by: Shalabh Mohan Shrivastava <shalabhs@microsoft.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-05-22 20:38:28 -07:00
John b54585c4f0
Apache Pulsar - pubsub component (#346)
* created pulsar component

* fixed multi topic issue

* moved metadata struct

* finished writing metadata tests

* fixed linter issues

* modifies consumer type to failover

* uncapitalized errors

* changed subscriptionName to consumerID

* acknowledge message only if there's no error returned from handler

Co-authored-by: john verdonck <John@johns-MacBook-Pro.local>
2020-05-22 10:04:07 -07:00
Kyle Housley 8eff31b191
Add AWS SNS/SQS support (#342)
* working but WIP aws pubsub

* working pubsub impl, problematic resource naming

* working implementation with tags and hashed topic/queue names

* add correct permissions for sqs queue to allow sns publishing

* fix logging messages, make awsToken optional, rename test functions, nits

* lowercase comments

* nits, repetitive use of 'the'

Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-05-21 14:20:16 -07:00
Yaron Schneider 71eaf77932
Update Readme.md (#320) 2020-04-30 09:06:54 -07:00
Kyle Housley 0c4cade71f
Add pubsub support for kafka (#315)
* fixes issue #8: adds kafka as pubsub component

* fixes issue #8: adds kafka as pubsub component

* pubsub kafka: modifies readme

* pubsub kafka: fixes bug

* pubsub kafka: fixes bug

* pubsub kafka: changes are per PR review

* changes as PR review

* removed unused field

* only close channel once

* working kafka pubsub

* add testing for metadata parsing

* linting

* remove unused PublishTopic, remove loglevel

* Use one consumer group for subscriptions

Co-authored-by: Roberto J Rojas <robertojrojas@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
2020-04-27 19:53:04 -07:00
Young Bu Park 0489fb17f3
Fix linter errors (#293) 2020-04-01 17:22:15 -07:00
Piotr Jasina 1ac30f7be5
Add Hazelcast pubsub support (#254) 2020-03-24 17:00:36 -07:00
Sivamuthu Kumar 16c230aec7
GCP Cloud Pubsub implementation (#266)
* GCP Cloud Pubsub pub sub implementation

* fixing lint

* using consumerID instead of subscription
2020-03-15 11:12:22 -07:00
Abhishek Gupta 807e8893cb
Azure Event Hubs pubsub component (#242)
* event hubs pubsub

* go mod tidy

* go mod

Co-authored-by: yaron2 <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-03-09 22:02:04 -07:00
Yaron Schneider 4602757a45
extend envelop to include a subject field (#249)
* extend envelop to include a subject field

* fix envelop test

* update mod

* try fix mod again
2020-03-04 22:59:26 -08:00
Young Bu Park ee97c3c56a
Use dapr logger (#238)
* Use dapr logger

* skip linter false alarm
2020-02-28 15:49:49 -08:00
Yaron Schneider c5ee4b273f
Add zero block for redis streams (#222)
* add zero block for redis streams

* update go.mod
2020-02-12 08:45:05 -08:00
Yaron Schneider d2a6fa5264
close azure service bus sender connection (#209) 2020-02-06 22:19:55 -08:00
REDDY PRASAD 5f3f850852
Add GCP Secret Manager (#199)
*  Add GCP Secret Manager

* Update tests

* Fix lint errors

* Update as per review

* Run go mod tidy

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-01-30 21:51:52 -08:00
Abhishek Gupta 6de230391d
Aerospike state store implementation (#196)
* aerospike state store impl

* fixed linter issues:

* addressed feedback

* Delete bearer_middleware.go

* Delete cassandra_test.go

* Revert "Delete cassandra_test.go"

This reverts commit 462c066814.

* Revert "Delete bearer_middleware.go"

This reverts commit ae114a1df1.

Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-28 18:55:14 -08:00
Joni Collinge ca2c3821a2 Add concurrency to Azure ServiceBus pubsub (#179)
* added worker pool for concurrent receive

* fixed missing end quote

* added type alias

* removed shadow decl

* added cleanup logic for subscription receiver

* renamed metadata property to numConcurrentHandlers and added handler specific timeout

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-24 10:26:11 -08:00
Abhishek Gupta 3734491718 NATS Streaming pubsub implementation (#173)
* NATS streaming pubsub impl

* after go mod tidy

* updates tests, other refactoring

* updated client ID generator

* updated test

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-17 14:54:01 -08:00
Joni Collinge bcfc66c79b fixed retry logic for servicebus listener (#186) 2020-01-14 12:51:13 -08:00
Abhishek Gupta 9323c44ecf check etag for etcd update and delete ops (#175) 2020-01-09 12:33:23 -08:00
Yaron Schneider 8b89a57b4e
fix redis streams waiting for handler (#172) 2020-01-05 10:54:15 -08:00
thinkerou a2f4b4c510 upgrade logrus to v1.4.2 (#150)
* fix conflict

* upgrade logrus to v1.4.2

* fix lint error

error: `depguard  `github.com/sirupsen/logrus` is in the blacklist`
2019-12-12 09:34:39 -08:00
Francisco Beltrao f89682397b Add Azure SignalR binding (#148)
* Add Azure SignalR binding

* Fixes from go lint

* Go lint

* Use "github.com/dgrijalva/jwt-go" for jwt
Fix bug parsing connection string
Normalize usage of len vs == in strings

* Consistent usage of http header Set
2019-12-03 09:08:53 -08:00
Abhishek Gupta c4bd2a47ea Implementation for Hazelcast based state store (#145)
* hazelcast store impl

* go.mod update
2019-11-27 21:12:47 -08:00
Francisco Beltrao 9e78671fc0 Add RabbitMQ pub/sub (#124)
* Add RabbitMQ pub/sub

* Fix lint error (global variable)

* RabbitMQ pub/sub auto ack defaults to false
Fix log message incorrect prefix

* Remove extra line in go.mod

* Revert extra line in go.mod
2019-11-27 10:37:09 -08:00
Shalabh Mohan Shrivastava bcd23db009
Changes to include Package names pubsub/secret stores/state to be consistent with company name (#137)
* [Commit - PubSub changes ]

Changing the names of the packages directory so to be consistent with name of the company .

It is to be noted that package name can not have hyphen "-" in the name.
So changing the directory names and putting under company names.

For example - servicebus package would be

dapr/components-contrib/pubsub/azure/servicebus

* Commit 2 - Changes for Secret Stores

* Commit 3 - for the state store changes
2019-11-22 12:46:42 -08:00
Francisco Beltrao 8086a01455 Fix azure service bus error message typo (#117) 2019-11-11 12:29:56 -08:00
Joni Collinge c945e70212 fixes wrong metadata config check (#108) 2019-11-05 16:19:26 -08:00
Joni Collinge ed3c829578 Azure ServiceBus PubSub Implementation (#93)
* initial skeleton code

* updated sub message handler

* lowercased errors

* added deadletter

* added context

* remove nil args

* added TODOs

* add max delivery count and timeout config

* refactored ensure methods

* removed confusing comment

* apply go fmt

* removed consumerID default

* updated go mod

* fixed up linter issues

* add package alias

* removed TODOs

* added additional servicebus config

* fix linting
2019-11-05 14:50:48 -08:00
Yaron Schneider 718a51ff40
Use consumerID for queue name (#99)
* use consumerID for queue name

* update tests
2019-11-04 11:42:21 -08:00
Yaron Schneider 34e1757740
Add TLS connection option to Redis pub sub (#96)
* add tls option to redis pub sub

* make default skip verify

* fix tls assignment

* remove gosec check

* retry nosec
2019-11-04 11:00:57 -08:00
Yaron Schneider 298ac1c381
Update Readme.md (#79) 2019-10-30 00:02:25 -07:00
Abhishek Gupta 49b6c9625c NATS pubsub component implementation (#65)
* inital version for  nats pubsub implementation

* added draft impl for nats pubsub

* updated impl, test

* new dependencies

* updated go.mod
2019-10-29 23:48:55 -07:00
Yaron Schneider 99461da958
remove pubsub mock (#33) 2019-10-14 13:06:24 -07:00
Yaron Schneider 4e436865fa
added cloud events content type (#32) 2019-10-14 12:52:39 -07:00
Yaron Schneider 9682bc829c
Exclude mock file from build (#31)
* exclude mock from build

* exclude mock from build
2019-10-12 21:54:34 -07:00
Aman Bhardwaj 04bef3ac3f Adding license header and updating to MIT license. (#26) 2019-10-09 10:58:08 -07:00
Yaron Schneider 2b8dbd2b44
dapr name change (#21) 2019-10-02 13:21:22 -07:00
yaron2 22fea6f43d initial commit 2019-09-26 15:47:41 -07:00