Commit Graph

569 Commits

Author SHA1 Message Date
yellow chicks c425f1319a
feature/pubsub: add delay queue params for pulsar (#1112)
* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

* feature/pubsub: add delay queue params for pulsar

Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-17 11:41:47 -07:00
Phil Kedy a0542ec922
Adding `initialOffset` option to Kafka binding and pubsub components (#1120)
* Adding `initialOffset` option to Kafka binding and pubsub components

* Only allow oldest and newest

* Fix tests

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-09-02 18:02:10 -07:00
Manuel Riezebosch 18cbe25e4e
feat(pubsub): add in-memory pub/sub (#1106) 2021-09-02 17:31:17 -07:00
Jaime Piña f4a5722806
Add JetStream support (#1101)
* Add JetStream support

* Add conformance tests

* Fix lint

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-09-01 11:50:16 -07:00
Luke Kennedy 8fd1ee57a0
GCP Pub/Sub Message Ordering Configuration (#1099)
* Add configuration for enabling message ordering

* Simplify passing the metadata

* Update field order to satisfy alignment linter

Co-authored-by: Luke Kennedy <luke.kennedy@optiisolutions.com>
2021-08-30 08:54:46 -07:00
Amit Mor 284d74488f
pubsub/aws/snssqs to support SQS dead-letter queue (#1066)
* bugfix for sns topic deletion upon termination

* Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* wip on normalizing queue/topic names

* sanitize queue and topic names

* sanitized names. bugfix for close

* # This is a combination of 4 commits.
# This is the 1st commit message:

Improve error message in case of missing property (#1012)

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
# This is the commit message #2:

Remove vestigial pubsub/nats code (#1024)

The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
# This is the commit message #3:

bugfix for sns topic deletion upon termination

# This is the commit message #4:

Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* removed debug message

* raw string abort

* merge issues solved

* wip

* gofmt+remove regex and use byte iter

* wip. first impl of dead-letters queue config

* wip. refactor and fallback values

* integration test wip

* wip integration test

* wip integration

* wip on testing

* wip

* still buggy but wip!

* bugfix in dlq creation

* working. still bug in subscription clean up

* Update snssqs_integ_test.go

* golangci-lint fixes

* golangci-lint refactoring

* trying to skip running integrations for snssqs

* testing

* skip integration test if no AWS related envvars are set (skip in CI)
* parallel testing in unittests

* code review fixes

* not using implicit maxReceives
* maxReceives renamed
* unittest refactor

* Update snssqs.go

* integ removed, renaming back of const

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-08-13 19:02:25 -07:00
weibubli 4d3e4ebee5
rename (#1061)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-08-10 16:24:15 -07:00
Amit Mor 253ef854c5
SNS and SQS topic/queue names + SNS subscription deletion bugfix (#1035)
* bugfix for sns topic deletion upon termination

* Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* wip on normalizing queue/topic names

* sanitize queue and topic names

* sanitized names. bugfix for close

* # This is a combination of 4 commits.
# This is the 1st commit message:

Improve error message in case of missing property (#1012)

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
# This is the commit message #2:

Remove vestigial pubsub/nats code (#1024)

The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
# This is the commit message #3:

bugfix for sns topic deletion upon termination

# This is the commit message #4:

Revert "bugfix for sns topic deletion upon termination"

This reverts commit bcaa9bb562.

* removed debug message

* raw string abort

* gofmt+remove regex and use byte iter

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-08-06 16:31:50 -07:00
Phil Kedy b73d817582
Merge changes from Release 1.3 to master (#1038)
* Fixing the handling of detach errors (#1030)

* Improve error message in case of missing property (#1012)

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>

* Remove vestigial pubsub/nats code (#1024)

The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.

* Fixing the handling of detach errors

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>

* Revert "Fixing the handling of detach errors (#1030)" (#1031)

This reverts commit 54840c2b83.

* Fixing the handling of detach errors (#1032)

Co-authored-by: Maarten Mulders <mthmulders@users.noreply.github.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Simon Leet <31784195+CodeMonkeyLeet@users.noreply.github.com>
2021-08-06 10:46:31 -07:00
Simon Leet dc7ee55e2a
Add Event Hubs pubsub and bindings conformance tests (#1040)
* Add conformance test for EventHubs bindings

* Update EventHubs pubsub to pass conformance tests

- Add conformance tests for Azure EventHubs PubSub component
- Add retry & backoff handling on subscriber handling error to EventHubs
  PubSub component for conformance tests.
- Add cancellation context to Azure EventHubs and update Close() to
  invoke cancel prior to closing the hub, which cleans up both senders
  and receivers.
2021-08-02 18:03:24 -07:00
Simon Leet f84a8b8914
Fix Kafka pubsub use of AuthRequired metadata (#1015)
Kafka pubsub component was not setting `k.authRequired` property based
on parsed `meta.AuthRequired` value, so would not correctly configure
Kafka producer or consumer with appropriate credentials when specified
by the author.

- Fix assignment of `meta.AuthRequired` to `k.authRequired` consumed by
  Subscribe().
- Fix initialization of `SyncProducer` to use copy of config with the
  authorization credentials added.

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-07-28 10:24:35 -07:00
Simon Leet f58e0ca26d
Remove vestigial pubsub/nats code (#1024)
The pubsub/nats component was replaced by pubsub/natsstreaming as part
of https://github.com/dapr/dapr/pull/2003, but the corresponding code
in dapr/components-contrib was not removed, so this change removes it.
2021-07-19 17:46:32 -07:00
Simon Leet 3877f5c793
Return SystemProperties on events from EventHubs bindings and pubsub (#1009)
- Update azure-event-hubs-go dependency to currently supported v3 with
  IoT Hub SystemProperties.
- Write Event.SystemProperties to ReadResponse.Metadata for EventHubs
  input binding.
- Write Event.SystemProperties to NewMessage.Metadata for EventHubs
  pubsub.
2021-07-13 16:04:22 -07:00
Ian Luo 7d2bc9bbdf
fix: Dapr runtime panic when handle Pub/Sub (dapr#3281) (#967)
* fix: Dapr runtime panic when handle Pub/Sub (dapr#3281)

* use uuid to generate Id when Id is missing

* skip default value for id and content-type

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: beiwei.ly <beiwei.ly@alibaba-inc.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-07-08 10:54:36 -07:00
Long Dai fe5e14b78b
Azure Service Bus: ignore amqp🔗detach-forced error (#985)
Signed-off-by: Long Dai <long0dai@foxmail.com>
2021-06-30 17:12:51 -07:00
iamazy 74af5b6528
rename ConsumerID to ConsumerGroup (#965)
* rename ConsumerID to ConsumerGroup

* rename consumerID to consumerGroup

* update the config in the conformance test

* set ConsumerID as alias of ConsumerGroup

* remove extra space

* gofmt

* add ClientID in kafka metadata

* remove ConsumerID field

* gofmt

* Update kafka.go

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-23 10:33:27 -07:00
Phil Kedy 8e6b36ecdf
Importing the retry and config packages from kit instead of internal (#940)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-22 09:38:55 -07:00
Taction 14870cb7bc
Add backoff for rabbitmq PubSub component (#862)
* Add backoff for rabbitmq

* fix lint

* fix lint

* remove backOffEnable config

* Delete backoff enable metadata

* Add a default config setting max retry to 0

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-06-10 00:19:46 -07:00
iamazy a59dcc44eb
Fix typo (#927)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-06-09 23:58:15 -07:00
Taction 7512c77a28
Rabbitmq pubsub component stop has not be well handled (#860)
* fix pubsub rabbitmq stop

* fix lint

* Add read lock while reading stopped

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-06-09 23:23:06 -07:00
Taction 464d53d94b
Move the common Redis code to internal (#859)
* Move pubsub redis client to internal

* Add failover for common redis. Refactor state redis.

* refactor binding redis

* fix lint

* Refactor to setting partten

* fix db and lint

* fix lint

* for backward compatibility

* Using a type alias to handle decoding duration values of -1 for Redis.

* Linter fixes

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-06-06 23:24:36 -07:00
Long Dai 4ee732e0bd
ci: enable gofumt linter (#887)
Signed-off-by: Long Dai <long0dai@foxmail.com>
2021-05-27 22:21:24 -07:00
Taction 334728a51f
Fix race in pulsar pubsub component (#868)
* Fix race in pulsar pubsub component

* fix lint

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-13 22:50:17 -07:00
cinience 60ad18f505
Add pubsub rocketmq (#816)
* Add pubsub rocketmq

* Fix lint

* Update parseMetadata

* Update logger import

* Tweaks to use the "settings pattern"

* Support backOffConfig

* Fix err log

* Update go.mod

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-05-13 22:31:41 -07:00
Artur Souza 3cc0472a41
Add function to generate CloudEvent for raw subscribed events. (#864)
* Add function to generate CloudEvent for raw subscribed events.

* address code comments.
2021-05-10 10:39:26 -07:00
Phil Kedy 1b30e75e60
Configurable retry (#854)
* no message

* Switching over usages of retry to the new package and making use of the DecodeConfig function

* Added decoding configuration using settings with a specific prefix

* Linter fixes

* Fixing linter error

* time.Sleep is the enemy

* Fix typo in comment

* Moving config to a pointer parameter so that the component can pass in the config with default values that make sense for that component.

* Renamed config struct

* Fix comment

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-06 09:43:57 -07:00
Charlie Stanley f69b9f5c34
Update redis metadata comments to reflect the new maxRetry default from v8 (#856)
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-05-05 10:36:49 -07:00
Taction a173e65ae7
Merge branch 'master' into upgrade_redis_client_to_v8 2021-05-04 23:19:54 +08:00
Charlie Stanley 053ca0b859
Do not block indefinitely on XGroupRead for redis subscriber (#855) 2021-05-03 17:44:20 -07:00
zhangchao 651d754496 Merge branch 'master' into upgrade_redis_client_to_v8
* master:
  add handle timeout to cancel req for it would be redelivered (#832)
  Add optional metadata param maxLen for redis stream PubSub component (#835)
  Implement bindings for the Zeebe workflow engine (#806)
  Automerge in contrib. (#849)

# Conflicts:
#	go.mod
#	go.sum
#	pubsub/redis/redis.go
2021-04-29 11:49:26 +08:00
Taction 61a47e8f56
add handle timeout to cancel req for it would be redelivered (#832)
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-04-28 19:23:36 -07:00
Taction f376450d7c
Add optional metadata param maxLen for redis stream PubSub component (#835)
* Add optional metadata param maxLen for redis stream PubSub component

* fix fmt

* fix error info

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Dapr Bot <56698301+dapr-bot@users.noreply.github.com>
2021-04-28 18:20:48 -07:00
zhangchao 4c0aef5c4e fix use r.ctx instead of context.Background() in pubsub redis component 2021-04-23 21:57:44 +08:00
zhangchao 5190fe4e7d Merge branch 'master' into upgrade_redis_client_to_v8
resolve conflict

* master:
  Increasing overall test timeout to allow for conformance tests to run (#834)
  Improve SMTP binding performance (#782)
  add redis cluter mode for pubsub (#823)
  downgrade the verbosity of lock renewal errors from warn to debug. This error is expected when messages are processed before beingrenewed (#803)
  Improve pulsar publisher performance. (#827)
  optimze(state/redis): output detail redis error (#825)
  Consul nameresolution component (#750)
  Add DNS name resolver. Add unit tests. (#743)
  Azure Service Bus: Changed log level of error on close to info (#830)
  Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807)
  Update state/blobstorage.go to add metadata. (#786)
  Azure Service Bus publishing retries when server is "too busy" (#821)

# Conflicts:
#	bindings/redis/redis.go
#	go.mod
#	go.sum
#	pubsub/redis/redis.go
2021-04-22 09:13:01 +08:00
Taction 466426894b
add redis cluter mode for pubsub (#823)
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-04-21 11:33:30 -07:00
Joni Collinge 744e5be6fa
downgrade the verbosity of lock renewal errors from warn to debug. This error is expected when messages are processed before beingrenewed (#803)
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-04-21 11:01:19 -07:00
Xueqian Wang f761b51c77
Improve pulsar publisher performance. (#827)
* Improve pulsar publisher performance.

* Addressed comments and added some debug logs

* Use peek instead of get for closing producers

* Fixed lint issue

* Added debug log in close

* Change back o 100

* Close producer before client

* Fixed cached number of producer and added todo for making it configurable

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-04-21 10:53:07 -07:00
Phil Kedy edfa3e5473
Azure Service Bus: Changed log level of error on close to info (#830) 2021-04-19 17:08:27 -07:00
Phil Kedy 688df9bda4
Removed dependency on dapr/dapr in favor of dapr/kit. Fixed go mod dependencies afterwards. Go 1.16 in go.mod. Removed accidental log dependency. (#807) 2021-04-19 12:06:10 -07:00
Phil Kedy 58b43fbfdb
Azure Service Bus publishing retries when server is "too busy" (#821)
* Adding retry/backoff to publishing to Azure Service Bus when the server is too busy and throttles the connection

* Fixing linter issues
2021-04-14 11:56:55 -07:00
zhangchao 3ec4b336a9 upgrade go-redis/redis to v8, and use context in pubsub/redis to cancel redis request. 2021-04-10 23:41:58 +08:00
Artur Souza 3993d9a559
Merge branch 'master' into jjcollinge/pubsub-context 2021-04-08 14:11:01 -07:00
Phil Kedy 6803b49e56 tls -> ssl for good measure 2021-04-06 17:50:42 -04:00
Phil Kedy bc0c4fd72e Adding small coalesce on scheme to ensure backward compatibility in both the pubsub and binding mqtt components 2021-04-06 17:32:47 -04:00
Phil Kedy 810ec429a7 Adding small coalesce on scheme to ensure backward compatibility in both the pubsub and binding mqtt components 2021-04-06 17:30:33 -04:00
Joni Collinge 617392fcca
Merge branch 'master' into jjcollinge/pubsub-context 2021-03-28 18:50:28 +01:00
zhangchao 9304037b70 rename backoff* to redis* 2021-03-24 20:58:56 +08:00
Joni Collinge 36ed5ec84f use existing context for pubsub comps 2021-03-23 20:12:46 +00:00
Joni Collinge 2d0864d25c use existing context for gcp 2021-03-23 19:45:35 +00:00
Joni Collinge 9fd461b255 Merge branch 'master' into jjcollinge/pubsub-context 2021-03-23 17:45:05 +00:00
Phil Kedy e9095e5081
Merge branch 'master' into add_more_config_for_redis 2021-03-22 18:11:04 -04:00
Trond Hindenes e5d686b56b
refactor GCP pubsub (#760)
* started on refactoring gcp pubsub

* Maybe this works?

* Working implicit auth

* Some fixes

* Updated tests

* Fix lint errors, tests

* trigger ci

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-03-22 14:47:05 -07:00
Phil Kedy a750f3c698
Merge branch 'master' into add_more_config_for_redis 2021-03-19 20:08:32 -04:00
Taction fb09a07e0f
fix pubsub redis close bug (#769)
When component close function called `r.cancel()` and `r.client.Close()` will be excuted.
The pollNewMessagesLoop loop use client to read from redis which will lead to err. And when err occured this loop will continue.
We need to check context on the top of this loop.
2021-03-19 17:01:03 -07:00
zhangchao ddbb039210 rename maxRetries minRetryBackoff maxRetryBackoff to backoffMaxRetries backOffMinInterval backOffMaxInterval. to be consistent with other components 2021-03-19 12:50:09 +08:00
zhangchao 1d376f3f3b fix lint 2021-03-18 22:40:19 +08:00
zhangchao 22f0f43e9b fix: add more connection settings 2021-03-18 22:01:57 +08:00
zhangchao fd1a544877 fix: parse duration-based settings as a time.Duration like redeliverInterval and processingTimeout are and remove the InSec suffix 2021-03-18 21:24:24 +08:00
zhangchao f433307fb1 add some redis client config that may be commonly used 2021-03-17 18:19:48 +08:00
Joni Collinge ca62139f47
Make connection recovery configurable using metadata (#713)
* add metadata properties to control max reconn attempts and recovery delay

* add metadata tests

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-03-12 12:41:38 -08:00
Phil Kedy a1a0ad2c25
Adding RabbitMQ to the conformance tests (#739)
* Initial pass at adding RabbitMQ to the conformance tests

* Fixing unit tests
2021-03-04 11:56:35 -08:00
Phil Kedy 44d64598b8
Added Hazelcast to conformance tests and retry to prevent message loss (#724) 2021-03-03 13:08:58 -08:00
Phil Kedy e8efbca072
Add MQTT pubsub to the conformance tests (#721)
* Add MQTT pubsub to the conformance tests

* Making the MQTT pubsub component conformance test against multiple brokers
2021-02-25 18:16:23 -08:00
Joni Collinge 2ff6ca4302 merge master 2021-02-25 16:15:26 +00:00
Joni Collinge c054ae6ada Merge branch 'master' into jjcollinge/pubsub-context 2021-02-25 16:15:06 +00:00
Phil Kedy 4a0011bcbd
Improve conformance tests and add pulsar (#702)
* Improve conformance tests and add pulsar

* Fix linter error

* Fix linter errors

* Tweaks to pulsar pubsub per PR

* Switching uuid import

* Using Stringer interface to print message keys as base64 in logger

* Disabling some checks w/ Redis pubsub until the component can be enhanced

* Adding comment to eventually remove the simulateErrors option

* Add comment asBase64String to explain what it does

* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing

* Fixing linter issues

* using a wait group instead of a sleep to wait for processing to complete

* More comments and refactoring

* comment tweaks

* Tweaks per PR

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-02-23 13:31:52 -08:00
Artur Souza dd3e463b52
Fix pubsub retry for Kafka (#688)
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.

* Adding backoff/retry logic to the Kafka pub/sub component

* Add logic to return if error from backoff in kafka.

* Addressing PR comments and lint.

Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-02-12 00:08:46 -08:00
Yaron Schneider 3ef025c604
change headers (#679) 2021-02-09 18:57:55 -08:00
Artur Souza 859f0ae881
Handles binary data in cloud event. (#674) 2021-02-09 09:45:38 -08:00
Yaron Schneider 58e92479ac
Add kafka max message bytes and pubsub partition key (#669)
* add kafka max message bytes and pubsub partition key

* linter
2021-02-08 11:14:52 -08:00
Yaron Schneider bc0988af42
fix missing subject (#640) 2021-01-28 08:37:36 -08:00
Yaron Schneider 4ca679172f
Fix service bus new topic creation (#636)
* fix service bus new topic creation

* place topic in list
2021-01-26 13:49:51 -08:00
Jigar 27344d4e4c
Added binary data type detection (#631)
* Added binary data type detection
Added base64 encoding for binary data types

* Update utils.go

Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-01-25 23:08:47 -08:00
Yaron Schneider 9fae71dada
Fix gcp pub/sub multiple subscriptions (#607)
* fix gcp pub/sub multiple subscriptions

* linter
2021-01-15 13:23:30 -08:00
Artur Souza 40e3d60d72
Remove useless random ID logic for mqtt's consumerID. (#605) 2021-01-15 11:56:54 -08:00
Artur Souza 01147e559d
Refactor and fix contenttype logic for pubsub. (#602)
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-01-14 14:28:52 -08:00
Yaron Schneider d7eeb2413f
Fix RabbitMQ concurrency (#576)
* fix rabbitmq concurrency

* linter

* add concurrency validation

* added tests and validation
2021-01-07 19:29:58 -08:00
Yaron Schneider 7465e4092d
Add concurrency mode to rabbitMQ pub/sub (#575)
* add concurrency mode to rabbitMQ pub/sub

* linter

* simplify
2021-01-07 15:02:58 -08:00
Yaron Schneider 80e5f2ca52
fix data serialization in cloudevent (#571)
* fix data serialization in cloudevent

* fixes
2021-01-04 17:23:07 -08:00
Yaron Schneider a99b605db8
extend custom cloud event (#570) 2021-01-04 12:12:39 -08:00
Yaron Schneider b7584f670b
Add new cloudevents creation method (#568)
* add new cloudevents creation method

* linter
2020-12-30 19:10:47 -08:00
Yaron Schneider a7ff1b0aad
Replace content cloud event check, add settraceid (#567)
* replace content cloud event check, add settraceid

* change cloudeventenvelope to map

* change settraceid method
2020-12-30 17:06:32 -08:00
Yaron Schneider 3e7f473b0c
Add method to check if payload is a cloudevent (#566)
* add method to check if payload is a cloudevent

* linter

* remove old test
2020-12-30 12:26:22 -08:00
Artur Souza e6dadfab6c
TTL in PubSub. (#565)
* TTL in PubSub.

* Handle message TTL only if component cannot handle it.

* Removig dead code for overflow check in pubsub expiration.
2020-12-29 14:28:52 -08:00
Trond Hindenes 660185b08c
Add support for prefetch (#562)
* Add support for prefetch

* Fixed test implementation

* Gofmt-ed files

* Re-arranged struct

* Moar tests
2020-12-21 10:42:16 -08:00
Joni Collinge 31828b3779
Update ASB message finalization (#534)
* remove servicebus context hack

* add more verbose error message

* make error message lowercase

* update go-autorest

* update azure-sdk-for-go

* remove go mod replace go-autorest

* fix lint issues

* go mod tidy

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-12-04 22:13:00 -08:00
Joni Collinge 256f920918 fix lint 2020-11-30 16:47:01 +00:00
Joni Collinge 96a8a4d7ac Merge branch 'master' into jjcollinge/pubsub-context 2020-11-30 16:21:44 +00:00
cvictory 87933ad4bc
The subscribeRequest and PublishRequest both need Metadata. (#543)
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-26 23:15:54 -08:00
Artur Souza 2f4be95b84
Respect content-type for PubSub data. (#541) 2020-11-25 17:20:20 -08:00
Joni Collinge 704cb42383
add pubsub request update (#530)
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-25 17:12:16 -08:00
Trond Hindenes 8d6978372f
Various AWS improvements (#525)
* Support sessiontoken

* Fixed tests and other components

* Fixed tests and things

* fmt

* Fix lint errors

* gofmt

* Fixed lint bugs

* Remove unneeded parameter

* gofmt

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-23 13:21:29 -08:00
Artur Souza 6266934d3f
Auto-reconnect rabbitmq pubsub (#519) 2020-11-09 12:27:53 -08:00
Joni Collinge ff2df86e3c Merge branch 'jjcollinge/pubsub-context' of github.com:jjcollinge/components-contrib into jjcollinge/pubsub-context 2020-11-03 08:29:49 +00:00
Joni Collinge d62e2a9e84 use context.Background() in eventhubs subscriber 2020-11-03 08:29:00 +00:00
Joni Collinge d2016a2e2d
Merge branch 'master' into jjcollinge/pubsub-context 2020-11-02 18:19:48 +00:00
Joni Collinge 7ea3800afd fix lint 2020-11-02 17:20:29 +00:00
AbserAri d09758b502
add ackoptions for nats (#511)
* add ackoptions for nats

* remove redundent variable as golint.

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-02 07:59:58 -08:00
Joni Collinge a1ed1c1662 update pubsub and mqtt 2020-11-02 09:54:54 +00:00
Joni Collinge 2e6b84ae40 Merge remote-tracking branch 'origin/master' into jjcollinge/pubsub-context 2020-11-02 09:05:16 +00:00
Yaron Schneider df5be19205
add close method to event hubs pub/sub (#500) 2020-10-15 08:42:14 -07:00
Yaron Schneider 182d204111
Add close method to pub/sub (#494)
* add close method to pub/sub

* linter
2020-10-12 19:06:33 -07:00
halspang f647d9eedb
Upgrade golang lint to 1.31 (#490)
Upgrade golang lint to 1.31.0

https://github.com/dapr/components-contrib/issues/439
2020-10-12 12:54:15 -07:00
Yaron Schneider 653bdced85
add metadata to pubsub message (#477)
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-09-24 12:04:42 -07:00
Artur Souza fbd04241b0
Adding PubSub response type. (#453) 2020-09-04 12:59:43 -07:00
AbserAri 26b085baba
update logger form nats-io's to dapr's (#449)
* update logger form nats-io's to dapr's

* run go mod tidy and go mod delete the nats-io gnatsd

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-09-03 18:58:34 -07:00
Mayank Kumar 7f9850a04f
Improve mqtt pubsub for multiple topics (#443)
- Listen on multiple topics when those are provided for subscription.
- Improve the current logic to use dedicated function from paho client.
2020-09-03 10:17:27 -07:00
Brightside Automation c344f68718 Merge branch 'master' into jjcollinge/pubsub-context 2020-08-19 14:46:03 +01:00
Yaron Schneider 677e551844
fix consumer exclusivity (#435) 2020-08-17 07:37:25 -07:00
Mayank Kumar f5f807ca73
Add pubsub support for mqtt over TLS (#429)
* Add pubsub support for mqtt over TLS

* Update mqtt host to include uri scheme.

- This is required to identify mode of connection used while
  connecting to broker.

* Add pubsub support for mqtt over TLS.

- add validation while initialisation.
- update test cases.
2020-08-16 11:01:12 -07:00
Leon Mai 08d2538796
Corresponding changes for multi pubsub (#434)
* Corresponding changes for multi pubsub

* Fix quote
2020-08-14 11:05:51 -07:00
jjcollinge 38a8948bd8 Merge branch 'jjcollinge/pubsub-context' of github.com:jjcollinge/components-contrib into jjcollinge/pubsub-context 2020-07-28 10:04:26 +01:00
jjcollinge 19442e67c8 update pubsub components to use new context 2020-07-28 10:03:23 +01:00
Joni Collinge f868dc9497
Merge branch 'master' into jjcollinge/pubsub-context 2020-07-28 09:25:34 +01:00
Yaron Schneider 5892feca63
update Cloud Events to 1.0 (#419) 2020-07-23 17:13:26 -07:00
Ally Parker cf3ee2a46a
Add topic to cloud events envelope (#396)
Co-authored-by: ibrahim.dursun <ibrahim.dursun@gmail.com>
2020-07-22 08:48:32 -07:00
Mark Chmarny 05e71ffdc2
data serialized object (#391)
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-07-13 13:31:32 -07:00
Mark Chmarny dbc1a76dcf lint fix 2020-07-13 09:24:51 -07:00
Mark Chmarny 1589031af6 fiixes json array envelope parsing 2020-07-13 09:20:54 -07:00
Mark Chmarny 68c5a5537c changes cloudevent identification 2020-07-08 15:28:54 -07:00
Mark Chmarny f1f69bf3c8 fixes speeling mistakes in ccmments 2020-07-08 11:23:23 -07:00
Mark Chmarny 43092998ab uses provided CE as is, adds CE test 2020-07-08 06:50:10 -07:00
Mark Chmarny 194ff4f914 fixes double envelop 2020-07-07 16:47:47 -07:00
jjcollinge 07b046f140 rename asb handler type to handle 2020-07-07 13:01:24 +01:00
jjcollinge 7734d05288 Merge branch 'master' into jjcollinge/pubsub-context 2020-07-07 12:41:58 +01:00
jjcollinge 410628af3d add context to pubsub handler signature 2020-07-07 12:37:15 +01:00
Joni Collinge 4b2252b017
ASB lock renewal per topic (#365)
* bump asb sdk version to 0.10.2

* add isolation per topic for lock renewal

* added cancellations on async handlers

* refactored out subscription logic from namespace

* updated error handling

* defer async context cancel

* fixed lint issues

* add connection recovery

* fix lint

* update comments

* update comments

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-07-02 08:02:42 -07:00
Kyle Housley 9ef4fb5fb4
Make enableTLS work correctly (#373)
the enabletls does the opposite of what the config suggests
2020-06-29 14:39:55 -07:00
Mukundan Sundararajan 5c62707aba
Upgrade go version to 1.14.3 and linter version to 1.26.0 (#352)
* Upgrade to go 1.14.3 and linter version to 1.26.0
2020-05-29 15:26:33 -07:00
Trond Hindenes d3e9a7b3a7
Use shared aws auth package (#347)
* Use shared sqs auth package

* Goimports-ed files

* Added support for endpoint param in aws credentials abstraction, since the sqs-sns pubsub module required it

* Added endpoint support for all aws things

* Updated tests
2020-05-26 16:27:16 -07:00
Mayank Kumar 3ed4f0f63c
Add pubsub support for mqtt (#330)
* Add pubsub support for mqtt

- update README.md
- add tests for metadata parsing
- linting

* Add pubsub support for mqtt

- remove topic field from metadata.

* Add pubsub support for mqtt

- use consumerID as clientID for mqtt connection

* Add pubsub support for mqtt

- address review comments
2020-05-22 20:39:09 -07:00
Joni Collinge 1461b0e4a5
Fix issue with ASB pubsub throttling (#329)
* Applied throttling to receiver loop

* added init lock renewal

* added prefect count config option

* renew message locks before processing

* moved lock acquisition per go routine

* fix test

* added max active message throttles

* rename numConcurrentHandlers to maxConcurrentHandlers

* fix typo in abandonMessage func name

* set maxActiveMessages to minimum of prefetchCount

* remove prefetch count constraint

* removed conditional activeMessage tracking

Co-authored-by: Shalabh Mohan Shrivastava <shalabhs@microsoft.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-05-22 20:38:28 -07:00
John b54585c4f0
Apache Pulsar - pubsub component (#346)
* created pulsar component

* fixed multi topic issue

* moved metadata struct

* finished writing metadata tests

* fixed linter issues

* modifies consumer type to failover

* uncapitalized errors

* changed subscriptionName to consumerID

* acknowledge message only if there's no error returned from handler

Co-authored-by: john verdonck <John@johns-MacBook-Pro.local>
2020-05-22 10:04:07 -07:00
Kyle Housley 8eff31b191
Add AWS SNS/SQS support (#342)
* working but WIP aws pubsub

* working pubsub impl, problematic resource naming

* working implementation with tags and hashed topic/queue names

* add correct permissions for sqs queue to allow sns publishing

* fix logging messages, make awsToken optional, rename test functions, nits

* lowercase comments

* nits, repetitive use of 'the'

Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-05-21 14:20:16 -07:00
Yaron Schneider 71eaf77932
Update Readme.md (#320) 2020-04-30 09:06:54 -07:00
Kyle Housley 0c4cade71f
Add pubsub support for kafka (#315)
* fixes issue #8: adds kafka as pubsub component

* fixes issue #8: adds kafka as pubsub component

* pubsub kafka: modifies readme

* pubsub kafka: fixes bug

* pubsub kafka: fixes bug

* pubsub kafka: changes are per PR review

* changes as PR review

* removed unused field

* only close channel once

* working kafka pubsub

* add testing for metadata parsing

* linting

* remove unused PublishTopic, remove loglevel

* Use one consumer group for subscriptions

Co-authored-by: Roberto J Rojas <robertojrojas@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
2020-04-27 19:53:04 -07:00
Young Bu Park 0489fb17f3
Fix linter errors (#293) 2020-04-01 17:22:15 -07:00
Piotr Jasina 1ac30f7be5
Add Hazelcast pubsub support (#254) 2020-03-24 17:00:36 -07:00
Sivamuthu Kumar 16c230aec7
GCP Cloud Pubsub implementation (#266)
* GCP Cloud Pubsub pub sub implementation

* fixing lint

* using consumerID instead of subscription
2020-03-15 11:12:22 -07:00
Abhishek Gupta 807e8893cb
Azure Event Hubs pubsub component (#242)
* event hubs pubsub

* go mod tidy

* go mod

Co-authored-by: yaron2 <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-03-09 22:02:04 -07:00
Yaron Schneider 4602757a45
extend envelop to include a subject field (#249)
* extend envelop to include a subject field

* fix envelop test

* update mod

* try fix mod again
2020-03-04 22:59:26 -08:00
Young Bu Park ee97c3c56a
Use dapr logger (#238)
* Use dapr logger

* skip linter false alarm
2020-02-28 15:49:49 -08:00
Yaron Schneider c5ee4b273f
Add zero block for redis streams (#222)
* add zero block for redis streams

* update go.mod
2020-02-12 08:45:05 -08:00
Yaron Schneider d2a6fa5264
close azure service bus sender connection (#209) 2020-02-06 22:19:55 -08:00
REDDY PRASAD 5f3f850852
Add GCP Secret Manager (#199)
*  Add GCP Secret Manager

* Update tests

* Fix lint errors

* Update as per review

* Run go mod tidy

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-01-30 21:51:52 -08:00
Abhishek Gupta 6de230391d
Aerospike state store implementation (#196)
* aerospike state store impl

* fixed linter issues:

* addressed feedback

* Delete bearer_middleware.go

* Delete cassandra_test.go

* Revert "Delete cassandra_test.go"

This reverts commit 462c066814.

* Revert "Delete bearer_middleware.go"

This reverts commit ae114a1df1.

Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-28 18:55:14 -08:00
Joni Collinge ca2c3821a2 Add concurrency to Azure ServiceBus pubsub (#179)
* added worker pool for concurrent receive

* fixed missing end quote

* added type alias

* removed shadow decl

* added cleanup logic for subscription receiver

* renamed metadata property to numConcurrentHandlers and added handler specific timeout

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-24 10:26:11 -08:00
Abhishek Gupta 3734491718 NATS Streaming pubsub implementation (#173)
* NATS streaming pubsub impl

* after go mod tidy

* updates tests, other refactoring

* updated client ID generator

* updated test

Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-17 14:54:01 -08:00
Joni Collinge bcfc66c79b fixed retry logic for servicebus listener (#186) 2020-01-14 12:51:13 -08:00
Abhishek Gupta 9323c44ecf check etag for etcd update and delete ops (#175) 2020-01-09 12:33:23 -08:00
Yaron Schneider 8b89a57b4e
fix redis streams waiting for handler (#172) 2020-01-05 10:54:15 -08:00
thinkerou a2f4b4c510 upgrade logrus to v1.4.2 (#150)
* fix conflict

* upgrade logrus to v1.4.2

* fix lint error

error: `depguard  `github.com/sirupsen/logrus` is in the blacklist`
2019-12-12 09:34:39 -08:00
Francisco Beltrao f89682397b Add Azure SignalR binding (#148)
* Add Azure SignalR binding

* Fixes from go lint

* Go lint

* Use "github.com/dgrijalva/jwt-go" for jwt
Fix bug parsing connection string
Normalize usage of len vs == in strings

* Consistent usage of http header Set
2019-12-03 09:08:53 -08:00
Abhishek Gupta c4bd2a47ea Implementation for Hazelcast based state store (#145)
* hazelcast store impl

* go.mod update
2019-11-27 21:12:47 -08:00
Francisco Beltrao 9e78671fc0 Add RabbitMQ pub/sub (#124)
* Add RabbitMQ pub/sub

* Fix lint error (global variable)

* RabbitMQ pub/sub auto ack defaults to false
Fix log message incorrect prefix

* Remove extra line in go.mod

* Revert extra line in go.mod
2019-11-27 10:37:09 -08:00
Shalabh Mohan Shrivastava bcd23db009
Changes to include Package names pubsub/secret stores/state to be consistent with company name (#137)
* [Commit - PubSub changes ]

Changing the names of the packages directory so to be consistent with name of the company .

It is to be noted that package name can not have hyphen "-" in the name.
So changing the directory names and putting under company names.

For example - servicebus package would be

dapr/components-contrib/pubsub/azure/servicebus

* Commit 2 - Changes for Secret Stores

* Commit 3 - for the state store changes
2019-11-22 12:46:42 -08:00
Francisco Beltrao 8086a01455 Fix azure service bus error message typo (#117) 2019-11-11 12:29:56 -08:00
Joni Collinge c945e70212 fixes wrong metadata config check (#108) 2019-11-05 16:19:26 -08:00
Joni Collinge ed3c829578 Azure ServiceBus PubSub Implementation (#93)
* initial skeleton code

* updated sub message handler

* lowercased errors

* added deadletter

* added context

* remove nil args

* added TODOs

* add max delivery count and timeout config

* refactored ensure methods

* removed confusing comment

* apply go fmt

* removed consumerID default

* updated go mod

* fixed up linter issues

* add package alias

* removed TODOs

* added additional servicebus config

* fix linting
2019-11-05 14:50:48 -08:00
Yaron Schneider 718a51ff40
Use consumerID for queue name (#99)
* use consumerID for queue name

* update tests
2019-11-04 11:42:21 -08:00
Yaron Schneider 34e1757740
Add TLS connection option to Redis pub sub (#96)
* add tls option to redis pub sub

* make default skip verify

* fix tls assignment

* remove gosec check

* retry nosec
2019-11-04 11:00:57 -08:00
Yaron Schneider 298ac1c381
Update Readme.md (#79) 2019-10-30 00:02:25 -07:00
Abhishek Gupta 49b6c9625c NATS pubsub component implementation (#65)
* inital version for  nats pubsub implementation

* added draft impl for nats pubsub

* updated impl, test

* new dependencies

* updated go.mod
2019-10-29 23:48:55 -07:00
Yaron Schneider 99461da958
remove pubsub mock (#33) 2019-10-14 13:06:24 -07:00
Yaron Schneider 4e436865fa
added cloud events content type (#32) 2019-10-14 12:52:39 -07:00
Yaron Schneider 9682bc829c
Exclude mock file from build (#31)
* exclude mock from build

* exclude mock from build
2019-10-12 21:54:34 -07:00
Aman Bhardwaj 04bef3ac3f Adding license header and updating to MIT license. (#26) 2019-10-09 10:58:08 -07:00
Yaron Schneider 2b8dbd2b44
dapr name change (#21) 2019-10-02 13:21:22 -07:00
yaron2 22fea6f43d initial commit 2019-09-26 15:47:41 -07:00