Debian - Azure IoTEdge - Custom IoTEdge module unable to connect to Hub with Communication_Error

user1483086
June 5, 2024
177 views
0 votes
2 Answers

Our custom IoTEdge module was functioning correctly until recently, when it started experiencing loss of connectivity, with our ConnectionStatusChangeHandler reporting status "ConnectionStatus.Disconnected" and reasons "ConnectionStatusChangeReason.Retry_Expired" and
ConnectionStatusChangeReason.Communication_Error".

The handler triggers re-initialization of the client as per recommended practices, but the loss of connectivity is continuous, there are no intermittent successful connections to the hub that would indicate a transient condition.

We checked One thing that stands out is that the EdgeHub module twin of the device does not list the custom module in its reported properties’ "clients" section, which seems to be related to the outage. Only the IoTEdgeMetricsCollector module is listed, which is connected to IoTHub and functioning normally.

  }, "clients": { "$lastUpdated": "2024-05-22T06:41:40.9107753Z", "deviceId/IoTEdgeMetricsCollector": { "$lastUpdated": "2024-05-22T06:41:40.9107753Z", "status": { "$lastUpdated": "2024-05-22T06:41:40.9107753Z" }, "lastConnectedTimeUtc": { "$lastUpdated": "2024-05-22T06:41:40.9107753Z" } } }

It seems that the local EdgeHub module is ignoring our custom module.

Environment

Host: Debian 11, Arm64
aziot-edged: 1.4.27
Edge Agent @ 1.4.35
Edge Hub @ 1.4.35
Docker/Moby: 24.0.9-1

Steps to repro

Unfortunately we are unable to reproduce the issue. We are not sure what started it, but we suspect that an accidental change was saved to the edgeHub’s module twin using Azure IoT Explorer.

What we tried so far :

Removed and re-added the custom module to the device from the Azure portal, and also through a layered deployment
Rebooted the gateway, restarted system modules, ran "iotedge system restart"
Deployed the SimulatedTemperatureSensor module on the device; it correctly registers with EdgeHub’s clients and functions correctly, so new modules other than our own seem unaffected
The "iotedge check" command returns all green with a couple warnings on package versions, no errors.
No errors in the EdgeHub or EdgeAgent logs after restarting

The custom module is successfully deployed, starts up and runs, but fails to send telemetry downstream, since any attempt to connect to the EdgeHub fails with aforementioned ConnectionStatus and ConnectionStatusChangeReason.

We also gathered traces from inside the custom module: we saw a WebSocket exception when the module attempts to connect to the hub, this is to be expected since EdgeHub which acts as a proxy to the custom module is unaware of its presence:

HasStack="True" ThreadID="6,006" ProcessorNumber="0" thisOrContextObject="ErrorDelegatingHandler#34683734" memberName="ExecuteWithErrorHandlingAsync" message="Exception caught: System.Net.WebSockets.WebSocketException (0x80004005): Unable to connect to the remote server —> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. —> System.IO.IOException: Received an unexpected EOF or 0 bytes from the transport stream. at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter) at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) — End of inner exception stack trace — at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options) at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options) at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotTransport.CreateClientWebSocketAsync(Uri websocketUri, CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotTransport.CreateClientWebSocketTransportAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotTransport.InitializeAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpIotConnector.OpenConnectionAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.EnsureConnectionAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder.OpenSessionAsync(IDeviceIdentity deviceIdentity, CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit.EnsureSessionIsOpenAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit.OpenAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken) at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func1 asyncOperation)"

We thought to fully uninstall and reinstall the IoTEdge runtime on the device, but we would like to exhaust all other options before going there.

At this point we are running out of ideas on how to attack this issue, any help would be appreciated.

Answers

Chosen as BEST ANSWER
- user1483086
- June 5, 2024 at 3:15 pm
- 0 votes
0
Solved. The issue was caused due to erroneously deleting the base deployment that puts the system modules on the device, then following up with layered/manual deployments. The edgeAgent and edgeHub modules were working, but would not see any newly deployed custom modules. Strangely enough, edgeHub would happily register other non-custom modules from the masrketplace, like Simulated Temperature Sensor. Restoring the base deployment brought everything back to working order.

(Edit)

The issue where your custom IoT Edge module is unable to connect to the IoT Hub with communication errors can be resolved by following these steps:

Check the modulesContent section to ensure your custom module is correctly listed and the desired properties are correctly set.
Use the iotedge check command to validate the deployment configuration.

I have configured the development environment, creating an IoT Edge module, updating it with custom code, building and pushing the module to a container registry, and finally deploying the module to an IoT Edge device.

Open Visual Studio Code, open the Command Palette (Ctrl+Shift+P) and run Azure IoT Edge: New IoT Edge Solution.
Select a folder to create the solution and enter a name for your solution.
Select a module template (e.g., C#, Node.js, Python) and enter a name for your module.

Provide the module’s image repository. For example, <registry_name>.azurecr.io/<module_name>
Update the Module with the Custom Code:

async Task<MessageResponse> FilterMessages(Message message, object userContext)
      {
          var counterValue = Interlocked.Increment(ref _counter);
          try
          {
              var moduleClient = (ModuleClient)userContext;
              var messageBytes = message.GetBytes();
              var messageString = Encoding.UTF8.GetString(messageBytes);
              Console.WriteLine($"Received message {counterValue}: [{messageString}]");

              var messageBody = JsonConvert.DeserializeObject<MessageBody>(messageString);

              if (messageBody != null && messageBody.machine.temperature > temperatureThreshold)
              {
                  Console.WriteLine($"Machine temperature {messageBody.machine.temperature} exceeds threshold {temperatureThreshold}");
                  using (var filteredMessage = new Message(messageBytes))
                  {
                      foreach (var prop in message.Properties)
                      {
                          filteredMessage.Properties.Add(prop.Key, prop.Value);
                      }

                      filteredMessage.Properties.Add("MessageType", "Alert");
                      await moduleClient.SendEventAsync("output1", filteredMessage);
                  }
              }

              return MessageResponse.Completed;
          }
          catch (Exception ex)
          {
              Console.WriteLine($"Error in sample: {ex.Message}");
              return MessageResponse.Abandoned;
          }
      }

Update the deployment.template.json file:

    {
      "modulesContent": {
        "$edgeAgent": {
          "properties.desired": {
            "modules": {
              "tempSensor": {
                "type": "docker",
                "settings": {
                  "image": "<registry_name>.azurecr.io/tempSensor:1.0",
                  "createOptions": "{}"
                },
                "env": {
                  "SimulatedData": {
                    "value": "true"
                  }
                }
              },
              "filtermodule": {
                "type": "docker",
                "settings": {
                  "image": "<registry_name>.azurecr.io/filtermodule:0.0.1",
                  "createOptions": "{}"
                }
              }
            }
          }
        },
        "$edgeHub": {
          "properties.desired": {
            "routes": {
              "sensorTofiltermodule": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint("/modules/filtermodule/inputs/inputFromSensor")",
              "filtermoduleToIoTHub": "FROM /messages/modules/filtermodule/outputs/output1 INTO $upstream"
            }
          }
        },
        "filtermodule": {
          "properties.desired": {
            "TemperatureThreshold": 25
          }
        }
      }
    }

Right-click on the deployment.template.json file in the Visual Studio Code Explorer and select Build and Push IoT Edge Solution.
This will build the Docker image and push it to your Azure Container Registry.