e9095e5081
Merge branch 'master' into add_more_config_for_redis
2021-03-22 18:11:04 -04:00
e5d686b56b
refactor GCP pubsub ( #760 )
...
* started on refactoring gcp pubsub
* Maybe this works?
* Working implicit auth
* Some fixes
* Updated tests
* Fix lint errors, tests
* trigger ci
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-03-22 14:47:05 -07:00
a750f3c698
Merge branch 'master' into add_more_config_for_redis
2021-03-19 20:08:32 -04:00
fb09a07e0f
fix pubsub redis close bug ( #769 )
...
When component close function called `r.cancel()` and `r.client.Close()` will be excuted.
The pollNewMessagesLoop loop use client to read from redis which will lead to err. And when err occured this loop will continue.
We need to check context on the top of this loop.
2021-03-19 17:01:03 -07:00
ddbb039210
rename maxRetries minRetryBackoff maxRetryBackoff to backoffMaxRetries backOffMinInterval backOffMaxInterval. to be consistent with other components
2021-03-19 12:50:09 +08:00
1d376f3f3b
fix lint
2021-03-18 22:40:19 +08:00
22f0f43e9b
fix: add more connection settings
2021-03-18 22:01:57 +08:00
fd1a544877
fix: parse duration-based settings as a time.Duration like redeliverInterval and processingTimeout are and remove the InSec suffix
2021-03-18 21:24:24 +08:00
f433307fb1
add some redis client config that may be commonly used
2021-03-17 18:19:48 +08:00
ca62139f47
Make connection recovery configurable using metadata ( #713 )
...
* add metadata properties to control max reconn attempts and recovery delay
* add metadata tests
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-03-12 12:41:38 -08:00
a1a0ad2c25
Adding RabbitMQ to the conformance tests ( #739 )
...
* Initial pass at adding RabbitMQ to the conformance tests
* Fixing unit tests
2021-03-04 11:56:35 -08:00
44d64598b8
Added Hazelcast to conformance tests and retry to prevent message loss ( #724 )
2021-03-03 13:08:58 -08:00
e8efbca072
Add MQTT pubsub to the conformance tests ( #721 )
...
* Add MQTT pubsub to the conformance tests
* Making the MQTT pubsub component conformance test against multiple brokers
2021-02-25 18:16:23 -08:00
2ff6ca4302
merge master
2021-02-25 16:15:26 +00:00
c054ae6ada
Merge branch 'master' into jjcollinge/pubsub-context
2021-02-25 16:15:06 +00:00
4a0011bcbd
Improve conformance tests and add pulsar ( #702 )
...
* Improve conformance tests and add pulsar
* Fix linter error
* Fix linter errors
* Tweaks to pulsar pubsub per PR
* Switching uuid import
* Using Stringer interface to print message keys as base64 in logger
* Disabling some checks w/ Redis pubsub until the component can be enhanced
* Adding comment to eventually remove the simulateErrors option
* Add comment asBase64String to explain what it does
* Adding redelivery (reclaims) to the Redis pubsub component
Removed the temporary simulateErrors flag now that Redis is passing
* Fixing linter issues
* using a wait group instead of a sleep to wait for processing to complete
* More comments and refactoring
* comment tweaks
* Tweaks per PR
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-02-23 13:31:52 -08:00
dd3e463b52
Fix pubsub retry for Kafka ( #688 )
...
* Add conformance test for kafka pubsub + quickfix for Kafka error handling.
* Adding backoff/retry logic to the Kafka pub/sub component
* Add logic to return if error from backoff in kafka.
* Addressing PR comments and lint.
Co-authored-by: Phil Kedy <phil.kedy@gmail.com>
2021-02-12 00:08:46 -08:00
3ef025c604
change headers ( #679 )
2021-02-09 18:57:55 -08:00
859f0ae881
Handles binary data in cloud event. ( #674 )
2021-02-09 09:45:38 -08:00
58e92479ac
Add kafka max message bytes and pubsub partition key ( #669 )
...
* add kafka max message bytes and pubsub partition key
* linter
2021-02-08 11:14:52 -08:00
bc0988af42
fix missing subject ( #640 )
2021-01-28 08:37:36 -08:00
4ca679172f
Fix service bus new topic creation ( #636 )
...
* fix service bus new topic creation
* place topic in list
2021-01-26 13:49:51 -08:00
27344d4e4c
Added binary data type detection ( #631 )
...
* Added binary data type detection
Added base64 encoding for binary data types
* Update utils.go
Co-authored-by: Artur Souza <artursouza.ms@outlook.com>
2021-01-25 23:08:47 -08:00
9fae71dada
Fix gcp pub/sub multiple subscriptions ( #607 )
...
* fix gcp pub/sub multiple subscriptions
* linter
2021-01-15 13:23:30 -08:00
40e3d60d72
Remove useless random ID logic for mqtt's consumerID. ( #605 )
2021-01-15 11:56:54 -08:00
01147e559d
Refactor and fix contenttype logic for pubsub. ( #602 )
...
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2021-01-14 14:28:52 -08:00
d7eeb2413f
Fix RabbitMQ concurrency ( #576 )
...
* fix rabbitmq concurrency
* linter
* add concurrency validation
* added tests and validation
2021-01-07 19:29:58 -08:00
7465e4092d
Add concurrency mode to rabbitMQ pub/sub ( #575 )
...
* add concurrency mode to rabbitMQ pub/sub
* linter
* simplify
2021-01-07 15:02:58 -08:00
80e5f2ca52
fix data serialization in cloudevent ( #571 )
...
* fix data serialization in cloudevent
* fixes
2021-01-04 17:23:07 -08:00
a99b605db8
extend custom cloud event ( #570 )
2021-01-04 12:12:39 -08:00
b7584f670b
Add new cloudevents creation method ( #568 )
...
* add new cloudevents creation method
* linter
2020-12-30 19:10:47 -08:00
a7ff1b0aad
Replace content cloud event check, add settraceid ( #567 )
...
* replace content cloud event check, add settraceid
* change cloudeventenvelope to map
* change settraceid method
2020-12-30 17:06:32 -08:00
3e7f473b0c
Add method to check if payload is a cloudevent ( #566 )
...
* add method to check if payload is a cloudevent
* linter
* remove old test
2020-12-30 12:26:22 -08:00
e6dadfab6c
TTL in PubSub. ( #565 )
...
* TTL in PubSub.
* Handle message TTL only if component cannot handle it.
* Removig dead code for overflow check in pubsub expiration.
2020-12-29 14:28:52 -08:00
660185b08c
Add support for prefetch ( #562 )
...
* Add support for prefetch
* Fixed test implementation
* Gofmt-ed files
* Re-arranged struct
* Moar tests
2020-12-21 10:42:16 -08:00
31828b3779
Update ASB message finalization ( #534 )
...
* remove servicebus context hack
* add more verbose error message
* make error message lowercase
* update go-autorest
* update azure-sdk-for-go
* remove go mod replace go-autorest
* fix lint issues
* go mod tidy
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-12-04 22:13:00 -08:00
256f920918
fix lint
2020-11-30 16:47:01 +00:00
96a8a4d7ac
Merge branch 'master' into jjcollinge/pubsub-context
2020-11-30 16:21:44 +00:00
87933ad4bc
The subscribeRequest and PublishRequest both need Metadata. ( #543 )
...
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-26 23:15:54 -08:00
2f4be95b84
Respect content-type for PubSub data. ( #541 )
2020-11-25 17:20:20 -08:00
704cb42383
add pubsub request update ( #530 )
...
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-25 17:12:16 -08:00
8d6978372f
Various AWS improvements ( #525 )
...
* Support sessiontoken
* Fixed tests and other components
* Fixed tests and things
* fmt
* Fix lint errors
* gofmt
* Fixed lint bugs
* Remove unneeded parameter
* gofmt
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-23 13:21:29 -08:00
6266934d3f
Auto-reconnect rabbitmq pubsub ( #519 )
2020-11-09 12:27:53 -08:00
ff2df86e3c
Merge branch 'jjcollinge/pubsub-context' of github.com:jjcollinge/components-contrib into jjcollinge/pubsub-context
2020-11-03 08:29:49 +00:00
d62e2a9e84
use context.Background() in eventhubs subscriber
2020-11-03 08:29:00 +00:00
d2016a2e2d
Merge branch 'master' into jjcollinge/pubsub-context
2020-11-02 18:19:48 +00:00
7ea3800afd
fix lint
2020-11-02 17:20:29 +00:00
d09758b502
add ackoptions for nats ( #511 )
...
* add ackoptions for nats
* remove redundent variable as golint.
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-11-02 07:59:58 -08:00
a1ed1c1662
update pubsub and mqtt
2020-11-02 09:54:54 +00:00
2e6b84ae40
Merge remote-tracking branch 'origin/master' into jjcollinge/pubsub-context
2020-11-02 09:05:16 +00:00
df5be19205
add close method to event hubs pub/sub ( #500 )
2020-10-15 08:42:14 -07:00
182d204111
Add close method to pub/sub ( #494 )
...
* add close method to pub/sub
* linter
2020-10-12 19:06:33 -07:00
f647d9eedb
Upgrade golang lint to 1.31 ( #490 )
...
Upgrade golang lint to 1.31.0
https://github.com/dapr/components-contrib/issues/439
2020-10-12 12:54:15 -07:00
653bdced85
add metadata to pubsub message ( #477 )
...
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-09-24 12:04:42 -07:00
fbd04241b0
Adding PubSub response type. ( #453 )
2020-09-04 12:59:43 -07:00
26b085baba
update logger form nats-io's to dapr's ( #449 )
...
* update logger form nats-io's to dapr's
* run go mod tidy and go mod delete the nats-io gnatsd
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-09-03 18:58:34 -07:00
7f9850a04f
Improve mqtt pubsub for multiple topics ( #443 )
...
- Listen on multiple topics when those are provided for subscription.
- Improve the current logic to use dedicated function from paho client.
2020-09-03 10:17:27 -07:00
c344f68718
Merge branch 'master' into jjcollinge/pubsub-context
2020-08-19 14:46:03 +01:00
677e551844
fix consumer exclusivity ( #435 )
2020-08-17 07:37:25 -07:00
f5f807ca73
Add pubsub support for mqtt over TLS ( #429 )
...
* Add pubsub support for mqtt over TLS
* Update mqtt host to include uri scheme.
- This is required to identify mode of connection used while
connecting to broker.
* Add pubsub support for mqtt over TLS.
- add validation while initialisation.
- update test cases.
2020-08-16 11:01:12 -07:00
08d2538796
Corresponding changes for multi pubsub ( #434 )
...
* Corresponding changes for multi pubsub
* Fix quote
2020-08-14 11:05:51 -07:00
38a8948bd8
Merge branch 'jjcollinge/pubsub-context' of github.com:jjcollinge/components-contrib into jjcollinge/pubsub-context
2020-07-28 10:04:26 +01:00
19442e67c8
update pubsub components to use new context
2020-07-28 10:03:23 +01:00
f868dc9497
Merge branch 'master' into jjcollinge/pubsub-context
2020-07-28 09:25:34 +01:00
5892feca63
update Cloud Events to 1.0 ( #419 )
2020-07-23 17:13:26 -07:00
cf3ee2a46a
Add topic to cloud events envelope ( #396 )
...
Co-authored-by: ibrahim.dursun <ibrahim.dursun@gmail.com>
2020-07-22 08:48:32 -07:00
05e71ffdc2
data serialized object ( #391 )
...
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-07-13 13:31:32 -07:00
dbc1a76dcf
lint fix
2020-07-13 09:24:51 -07:00
1589031af6
fiixes json array envelope parsing
2020-07-13 09:20:54 -07:00
68c5a5537c
changes cloudevent identification
2020-07-08 15:28:54 -07:00
f1f69bf3c8
fixes speeling mistakes in ccmments
2020-07-08 11:23:23 -07:00
43092998ab
uses provided CE as is, adds CE test
2020-07-08 06:50:10 -07:00
194ff4f914
fixes double envelop
2020-07-07 16:47:47 -07:00
07b046f140
rename asb handler type to handle
2020-07-07 13:01:24 +01:00
7734d05288
Merge branch 'master' into jjcollinge/pubsub-context
2020-07-07 12:41:58 +01:00
410628af3d
add context to pubsub handler signature
2020-07-07 12:37:15 +01:00
4b2252b017
ASB lock renewal per topic ( #365 )
...
* bump asb sdk version to 0.10.2
* add isolation per topic for lock renewal
* added cancellations on async handlers
* refactored out subscription logic from namespace
* updated error handling
* defer async context cancel
* fixed lint issues
* add connection recovery
* fix lint
* update comments
* update comments
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-07-02 08:02:42 -07:00
9ef4fb5fb4
Make enableTLS work correctly ( #373 )
...
the enabletls does the opposite of what the config suggests
2020-06-29 14:39:55 -07:00
5c62707aba
Upgrade go version to 1.14.3 and linter version to 1.26.0 ( #352 )
...
* Upgrade to go 1.14.3 and linter version to 1.26.0
2020-05-29 15:26:33 -07:00
d3e9a7b3a7
Use shared aws auth package ( #347 )
...
* Use shared sqs auth package
* Goimports-ed files
* Added support for endpoint param in aws credentials abstraction, since the sqs-sns pubsub module required it
* Added endpoint support for all aws things
* Updated tests
2020-05-26 16:27:16 -07:00
3ed4f0f63c
Add pubsub support for mqtt ( #330 )
...
* Add pubsub support for mqtt
- update README.md
- add tests for metadata parsing
- linting
* Add pubsub support for mqtt
- remove topic field from metadata.
* Add pubsub support for mqtt
- use consumerID as clientID for mqtt connection
* Add pubsub support for mqtt
- address review comments
2020-05-22 20:39:09 -07:00
1461b0e4a5
Fix issue with ASB pubsub throttling ( #329 )
...
* Applied throttling to receiver loop
* added init lock renewal
* added prefect count config option
* renew message locks before processing
* moved lock acquisition per go routine
* fix test
* added max active message throttles
* rename numConcurrentHandlers to maxConcurrentHandlers
* fix typo in abandonMessage func name
* set maxActiveMessages to minimum of prefetchCount
* remove prefetch count constraint
* removed conditional activeMessage tracking
Co-authored-by: Shalabh Mohan Shrivastava <shalabhs@microsoft.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-05-22 20:38:28 -07:00
b54585c4f0
Apache Pulsar - pubsub component ( #346 )
...
* created pulsar component
* fixed multi topic issue
* moved metadata struct
* finished writing metadata tests
* fixed linter issues
* modifies consumer type to failover
* uncapitalized errors
* changed subscriptionName to consumerID
* acknowledge message only if there's no error returned from handler
Co-authored-by: john verdonck <John@johns-MacBook-Pro.local>
2020-05-22 10:04:07 -07:00
8eff31b191
Add AWS SNS/SQS support ( #342 )
...
* working but WIP aws pubsub
* working pubsub impl, problematic resource naming
* working implementation with tags and hashed topic/queue names
* add correct permissions for sqs queue to allow sns publishing
* fix logging messages, make awsToken optional, rename test functions, nits
* lowercase comments
* nits, repetitive use of 'the'
Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-05-21 14:20:16 -07:00
71eaf77932
Update Readme.md ( #320 )
2020-04-30 09:06:54 -07:00
0c4cade71f
Add pubsub support for kafka ( #315 )
...
* fixes issue #8 : adds kafka as pubsub component
* fixes issue #8 : adds kafka as pubsub component
* pubsub kafka: modifies readme
* pubsub kafka: fixes bug
* pubsub kafka: fixes bug
* pubsub kafka: changes are per PR review
* changes as PR review
* removed unused field
* only close channel once
* working kafka pubsub
* add testing for metadata parsing
* linting
* remove unused PublishTopic, remove loglevel
* Use one consumer group for subscriptions
Co-authored-by: Roberto J Rojas <robertojrojas@gmail.com>
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
Co-authored-by: Aman Bhardwaj <amanbha@users.noreply.github.com>
2020-04-27 19:53:04 -07:00
0489fb17f3
Fix linter errors ( #293 )
2020-04-01 17:22:15 -07:00
1ac30f7be5
Add Hazelcast pubsub support ( #254 )
2020-03-24 17:00:36 -07:00
16c230aec7
GCP Cloud Pubsub implementation ( #266 )
...
* GCP Cloud Pubsub pub sub implementation
* fixing lint
* using consumerID instead of subscription
2020-03-15 11:12:22 -07:00
807e8893cb
Azure Event Hubs pubsub component ( #242 )
...
* event hubs pubsub
* go mod tidy
* go mod
Co-authored-by: yaron2 <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-03-09 22:02:04 -07:00
4602757a45
extend envelop to include a subject field ( #249 )
...
* extend envelop to include a subject field
* fix envelop test
* update mod
* try fix mod again
2020-03-04 22:59:26 -08:00
ee97c3c56a
Use dapr logger ( #238 )
...
* Use dapr logger
* skip linter false alarm
2020-02-28 15:49:49 -08:00
c5ee4b273f
Add zero block for redis streams ( #222 )
...
* add zero block for redis streams
* update go.mod
2020-02-12 08:45:05 -08:00
d2a6fa5264
close azure service bus sender connection ( #209 )
2020-02-06 22:19:55 -08:00
5f3f850852
Add GCP Secret Manager ( #199 )
...
* ✨ Add GCP Secret Manager
* Update tests
* Fix lint errors
* Update as per review
* Run go mod tidy
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
2020-01-30 21:51:52 -08:00
6de230391d
Aerospike state store implementation ( #196 )
...
* aerospike state store impl
* fixed linter issues:
* addressed feedback
* Delete bearer_middleware.go
* Delete cassandra_test.go
* Revert "Delete cassandra_test.go"
This reverts commit 462c066814ae114a1df1
2020-01-28 18:55:14 -08:00
ca2c3821a2
Add concurrency to Azure ServiceBus pubsub ( #179 )
...
* added worker pool for concurrent receive
* fixed missing end quote
* added type alias
* removed shadow decl
* added cleanup logic for subscription receiver
* renamed metadata property to numConcurrentHandlers and added handler specific timeout
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-24 10:26:11 -08:00
3734491718
NATS Streaming pubsub implementation ( #173 )
...
* NATS streaming pubsub impl
* after go mod tidy
* updates tests, other refactoring
* updated client ID generator
* updated test
Co-authored-by: Yaron Schneider <yaronsc@microsoft.com>
Co-authored-by: Young Bu Park <youngp@microsoft.com>
2020-01-17 14:54:01 -08:00
bcfc66c79b
fixed retry logic for servicebus listener ( #186 )
2020-01-14 12:51:13 -08:00
9323c44ecf
check etag for etcd update and delete ops ( #175 )
2020-01-09 12:33:23 -08:00
8b89a57b4e
fix redis streams waiting for handler ( #172 )
2020-01-05 10:54:15 -08:00
a2f4b4c510
upgrade logrus to v1.4.2 ( #150 )
...
* fix conflict
* upgrade logrus to v1.4.2
* fix lint error
error: `depguard `github.com/sirupsen/logrus` is in the blacklist`
2019-12-12 09:34:39 -08:00
f89682397b
Add Azure SignalR binding ( #148 )
...
* Add Azure SignalR binding
* Fixes from go lint
* Go lint
* Use "github.com/dgrijalva/jwt-go" for jwt
Fix bug parsing connection string
Normalize usage of len vs == in strings
* Consistent usage of http header Set
2019-12-03 09:08:53 -08:00
c4bd2a47ea
Implementation for Hazelcast based state store ( #145 )
...
* hazelcast store impl
* go.mod update
2019-11-27 21:12:47 -08:00
9e78671fc0
Add RabbitMQ pub/sub ( #124 )
...
* Add RabbitMQ pub/sub
* Fix lint error (global variable)
* RabbitMQ pub/sub auto ack defaults to false
Fix log message incorrect prefix
* Remove extra line in go.mod
* Revert extra line in go.mod
2019-11-27 10:37:09 -08:00
bcd23db009
Changes to include Package names pubsub/secret stores/state to be consistent with company name ( #137 )
...
* [Commit - PubSub changes ]
Changing the names of the packages directory so to be consistent with name of the company .
It is to be noted that package name can not have hyphen "-" in the name.
So changing the directory names and putting under company names.
For example - servicebus package would be
dapr/components-contrib/pubsub/azure/servicebus
* Commit 2 - Changes for Secret Stores
* Commit 3 - for the state store changes
2019-11-22 12:46:42 -08:00
8086a01455
Fix azure service bus error message typo ( #117 )
2019-11-11 12:29:56 -08:00
c945e70212
fixes wrong metadata config check ( #108 )
2019-11-05 16:19:26 -08:00
ed3c829578
Azure ServiceBus PubSub Implementation ( #93 )
...
* initial skeleton code
* updated sub message handler
* lowercased errors
* added deadletter
* added context
* remove nil args
* added TODOs
* add max delivery count and timeout config
* refactored ensure methods
* removed confusing comment
* apply go fmt
* removed consumerID default
* updated go mod
* fixed up linter issues
* add package alias
* removed TODOs
* added additional servicebus config
* fix linting
2019-11-05 14:50:48 -08:00
718a51ff40
Use consumerID for queue name ( #99 )
...
* use consumerID for queue name
* update tests
2019-11-04 11:42:21 -08:00
34e1757740
Add TLS connection option to Redis pub sub ( #96 )
...
* add tls option to redis pub sub
* make default skip verify
* fix tls assignment
* remove gosec check
* retry nosec
2019-11-04 11:00:57 -08:00
298ac1c381
Update Readme.md ( #79 )
2019-10-30 00:02:25 -07:00
49b6c9625c
NATS pubsub component implementation ( #65 )
...
* inital version for nats pubsub implementation
* added draft impl for nats pubsub
* updated impl, test
* new dependencies
* updated go.mod
2019-10-29 23:48:55 -07:00
99461da958
remove pubsub mock ( #33 )
2019-10-14 13:06:24 -07:00
4e436865fa
added cloud events content type ( #32 )
2019-10-14 12:52:39 -07:00
9682bc829c
Exclude mock file from build ( #31 )
...
* exclude mock from build
* exclude mock from build
2019-10-12 21:54:34 -07:00
04bef3ac3f
Adding license header and updating to MIT license. ( #26 )
2019-10-09 10:58:08 -07:00
2b8dbd2b44
dapr name change ( #21 )
2019-10-02 13:21:22 -07:00
22fea6f43d
initial commit
2019-09-26 15:47:41 -07:00
Phil Kedy
Trond Hindenes
Taction
Joni Collinge
Artur Souza
Yaron Schneider
Jigar
cvictory
AbserAri
halspang
Mayank Kumar
Brightside Automation
Leon Mai
Ally Parker
Mark Chmarny
Mark Chmarny
Kyle Housley
Mukundan Sundararajan
John
Young Bu Park
Piotr Jasina
Sivamuthu Kumar
Abhishek Gupta
REDDY PRASAD
thinkerou
Francisco Beltrao
Shalabh Mohan Shrivastava
Francisco Beltrao
Aman Bhardwaj