Relay support

Big release - relay support in plumber!

It is now possible to run plumber in "relay" mode. In this mode, plumber will attach to a target messaging system and relay all of the messages/events it sees to Batch via gRPC.

You can run plumber in either foreground mode:

$ plumber relay aws-sqs --queue-name PlumberTestQueue --token 48b30466-e3cb-4a58-9905-123456781234 --auto-delete

Or you can run it via Docker:

docker run --name plumber -p 8080:8080 \
    -e PLUMBER_RELAY_TYPE=aws-sqs \
    -e PLUMBER_RELAY_TOKEN=48b30466-e3cb-4a58-9905-123456781234 \
    -e PLUMBER_RELAY_SQS_QUEUE_NAME=PlumberTestQueue \
    -e PLUMBER_RELAY_SQS_AUTO_DELETE=true \
    -d batchcorp/plumber:latest

Support for AWS SQS

Support for both reading and writing to/from AWS SQS:

# Writing plaintext
$ plumber write message aws-sqs --queue-name PlumberTestQueue --input-data "hello world"

# Writing protobuf
$ plumber write message aws-sqs --queue-name PlumberTestQueue --input-file ./test-assets/messages/sample-outbound.json --input-type jsonpb --output-type protobuf --protobuf-dir ./test-assets/protos/ --protobuf-root-message Outbound

# Reading plaintext
$ plumber read message aws-sqs --queue-name PlumberTestQueue --auto-delete
INFO[0000] Listening for message(s) ...                  pkg=awssqs/read.go
hello world

# Reading protobuf
$ plumber read message aws-sqs --queue-name PlumberTestQueue --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound --output-type protobuf
INFO[0000] Listening for message(s) ...                  pkg=awssqs/read.go
{
  "replayId": "30ddb850-1aca-4ee5-870c-1bb7b339ee5d",
  "blob": "eyJoZWxsbyI6ImRhbiJ9Cg=="
}

For full info: plumber read message aws-sqs --help and plumber write message aws-sqs --help

Support for MQTT

• Support for MQTT

  # Read (forever) example
  $ plumber read message mqtt --topic dantest  -d -f --line-numbers --address tcp://test.mosquitto.org:1883
  
  # Write example
  $ plumber write message mqtt --topic dantest --input-data testing --address tcp://test.mosquitto.org:1883
  
  # Read (forever) with protobuf
  $ plumber read message mqtt --topic dantest --address tcp://test.mosquitto.org:1883 -d -f --line-numbers --output-type protobuf --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound
  
  # Write protobuf
  $ plumber write message mqtt --topic dantest --address tcp://test.mosquitto.org:1883 --input-file ./test-assets/messages/sample-outbound.json --input-type jsonpb --output-type protobuf --protobuf-dir ./test-assets/protos --protobuf-root-message Outbound
  

• Cleaned up debug info from output

CLI arg refactor, cleanup, various bugfixes

Fairly significant cleanup release; significant refactor (moved to a diff lib); cleanup of some flags.

Support for GCP PubSub

Added support for GCP PubSub.

Decompression + Default Exchange

• Added support to gzip decompress output on reads (from kafka and rabbit)
• Removed --exchange-name as required arg (forgot that rabbit has default exchange support)

Initial release

This is the initial release of the port of our internal "plumber" tool that we use for consuming and publishing messages on our event bus.

It is nowhere near feature complete, but it does have the most important part implemented (in our opinion):

It is able to read and write protobuf based messages to either Kafka or RabbitMQ. We think that's pretty damn sweet.

We hope it's useful for you!