skip to Main Content

I am hoping someone has got this to work.
I am the trying to connect RabbitMQ shovel to Azure Event hub or Service Bus.
I found two tutorials on MS, (very short to set up) but both results in the shovel error.
The request to Event hub or Service bus is success in the example.

My guess is that something is missing from the below tutorials or that some components have been changed and breaking changes has been introduced:

Importing Data from RabbitMQ into Azure Data Explorer via Event Hubs
https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/importing-data-from-rabbitmq-into-azure-data-explorer-via-event/ba-p/3777688
Shovel error

AMQP allows for interconnecting connecting brokers directly, for instance using routers like Apache Qpid Dispatch Router or broker-native "shovels" like the one of RabbitMQ.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-overview

How to integrate Service Bus with RabbitMQ
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-integrate-with-rabbitmq

Version:
RabbitMQ 3.10.7, Erlang 25.0

Case and steps:
Adding a new Azure Service Bus Namespace
rabbitmq01
basic
public endpoint
Creating our Azure Service Bus Queue
from-rabbitmq01

Enabling the RabbitMQ Shovel Plugin

rabbitmq-plugins enable rabbitmq_shovel_management

Created queue in Rabbitmq
telemetry01

Connecting RabbitMQ to Azure Service Bus
Add SAS Policy, Manage now, tested with just send also.
rabbitmq-shovel01
Primary Connection String

Connection String to AMQP
https://red-mushroom-0f7446a0f.azurestaticapps.net/
amqps://rabbitmq-shovel01:[email protected]:5671/?sasl=plain

TNC

Test-NetConnection -ComputerName rabbitmq01.servicebus.windows.net -Port 5671 = True

Make the dynamic shovel with above properties.
Shovel was done exactly like the tutorial
The shovel status is just starting, never goes to running.

Log:

2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> supervisor: {<0.742.0>,amqp10_client_connection_sup} 2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> errorContext: start_error 2023-07-08 19:09:56.385000+02:00 [error] <0.742.0> reason: {badmatch,{error,closed}}
[…] 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** Stacktrace = 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> ** [{amqp10_client_frame_reader,init,1, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> [{file,"amqp10_client_frame_reader.erl"}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {line,109}]}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {gen_statem,init_it,6,[{file,"gen_statem.erl"},{line,1001}]}, 2023-07-08 19:09:56.386000+02:00 [error] <0.744.0> {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
[…] 2023-07-08 19:09:56.386000+02:00 [error] <0.688.0> Shovel 'rmq-2-sb' could not connect to destination 2023-07-08 19:09:56.387000+02:00 [error] <0.687.0> supervisor: {<0.687.0>,rabbit_shovel_dyn_worker_sup}

Shovel to Bus requests success

Tested

  1. I have tried to change TLS version in Service bus.
  2. Enabled rabbitmq_amqp1_0 plugin, https://github.com/rabbitmq/rabbitmq-amqp1.0.
  3. Tested a lot of combinations with the URI. After testing https://red-mushroom-0f7446a0f.azurestaticapps.net/ and URL enconding.
  4. Azure ServiceBus can be used with the AMQP 1.0 protocol. Since version 3.7 RabbitMQ supports shovels where either the source or destination (or both) uses AMQP 1.0. https://gist.github.com/kjnilsson/159c643fb34604f8ea20be336109261b

I have sent a mail to RabbitMQ community also.
Thank you in advance

11.07.2023 Update
New RabbitMQ 3.12.1 and Erlang 26.0

Today I did a new test with both tutorials. For the Event hub i followed it exactly. The shovel error is:
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> supervisor: {<0.2006.0>,amqp10_client_connection_sup}
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> errorContext: start_error
2023-07-11 10:01:57.638000+02:00 [error] <0.2006.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[…] 2023-07-11 10:09:27.788000+02:00 [error] <0.2476.0> Shovel 'shovel-07' could not connect to destination

For the Service bus it is the same.

First I tried with shovel Source AMQP 0.9.1, Destination, AMQP 1.0.
:
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> supervisor: {<0.3627.0>,amqp10_client_connection_sup}
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> errorContext: start_error
2023-07-11 10:44:41.042000+02:00 [error] <0.3627.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[…] 2023-07-11 10:44:41.043000+02:00 [error] <0.3610.0> Shovel 'shovel_01' could not connect to destination

Then I tried shovel Source AMQP 1.0, Destination, AMQP 1.0.

And as expected:
2023-07-11 11:08:18.155000+02:00 [error] <0.8999.0> amqp1_0_plugin_not_enabled

Enabled plugin amqp1_0

2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> supervisor: {<0.9613.0>,amqp10_client_connection_sup}
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> errorContext: start_error
2023-07-11 11:14:58.438000+02:00 [error] <0.9613.0> reason: {options,incompatible,[{verify,verify_peer},{cacerts,undefined}]}
[…] 2023-07-11 11:14:58.438000+02:00 [error] <0.9582.0> Shovel 'shovel_03' could not connect to destination

After update to: RabbitMQ 3.12.1 and Erlang 26.0
.So it has do to with TLS, and that was not a step in any of the tutorials.
This is gonna take time 🙂 Any help is appreciated

2

Answers


  1. Chosen as BEST ANSWER

    I got in touch with MS, as per delivered test results, they did not find a issue with the test(s) done. TNC images validated connectivity. Further troubleshooting will be done by support. Looking forward to that. Collaboration is key.


  2. Sample RabbitMQ Shovel to Azure Service Bus:

    • Enable the required

    rabbitmq-plugins enable rabbitmq_shovel_management
    rabbitmq-plugins enable rabbitmq_shovel

    rabbitmq-plugins enable rabbitmq_management

    enter image description here

    enter image description here
    enter image description here

    enter image description here

    enter image description here

    enter image description here

    enter image description here

    Login or Signup to reply.
Please signup or login to give your own answer.
Back To Top
Search