I am working on designing a solution for sending data from windows based devices to the cloud.
Here is the high level
Device Software: The device software is currently written in C, C++ or C# languages based on the device model and legacy build.
IoT Service: This is a proposed service which might run as a windows service and this might use C# language as well as Azure SDK for IoT Hub connectivity.
Now question is, What is the most efficient way to handle a large number of messages from Device Software to IoT Service and successfully send them to the cloud without any data loss? I mean do I have to use any Queue concept or MQTT libraries which help to handle messages and retry logics?
Should a consumer be implemented to retry failed messages or is an Remote procedure callsufficient for this use case?
This is a real-time use case in this IoTera.
2
Answers
Curious to know what rate you will be sending messages, how many devices and size of message. IoT Hub like most public endpoints implements rate throttling to mitigate attack vectors and you will need to size it properly. The link provides some details. https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-quotas-throttling . e.g. 1 unit of IoT Hub S3 SKU can handle total 6000 send operations/sec and you can add more units to increase it.
If there is not need to analyze and respond immediately to such messages you can batch and upload as file, combination of telemetry and batch on error is effective and supported by IoTHub (file gets uploaded to Blob storage configured in IoTHub)
On the second question IoTHub has internal queue and partitioning mechanism to ingest at high rate and consume in the backend at different rate with default storing up to 7 days. If you are using SDK’s, they implement exponential jitter with back-off to handle transient issues but anything else like authentication failure or network outages you will need to build store-and-forward on the device itself. The IoT Hub service guarantees at least once delivery (MQTT QoS 1), so you will not lose messages unless device fails to retry undelivered messages due to outage or program/device crash.
If the device is capable of running IoT Edge, the runtime has built-in store and forward capabilities but it is not suitable for constrained devices.
If you must avoid data loss in the event of a process crash, OS restart, or power outage, then the only solution I see is to persist data to disk, and then clear it after it was confirmed to be sent.
In the event of a process restart, the program would need to look at the disk where data was stored, ingest it, and attempt to send it. That means there is a chance of duplicate data being sent (data is persisted, telemetry sent, process exits before persisted data could be removed), but if you generate a GUID for each message and store it in the MessageId, you could detect and dedupe on the service side.