Can't set up MQTT (Mosquito Docker) + SSL + MQTTNet (C#) + Dapr.io binding

I am trying to set up a local MQTT broker using mosquito image and connect to it with MQTTNet as publisher and subscribe using Dapr MQTT binding. Everything works fine if I am using anonymous mode (both in Docker self-hosted and in K8S). But we need to use SSL… and here things stopped working. And I am literally out of ideas.

Anyone done this setup already?

I am following this guide for creating certificates: Mosquito SSL setup

This is my broker Dockerfile:

FROM eclipse-mosquitto:latest

COPY ./mqtt/mosquitto/mosquitto.conf /mosquitto/config/mosquitto.conf
COPY ./mqtt/mosquitto/certificates /mosquitto/certs
COPY ./mqtt/mosquitto/certificates /mosquitto/ca_certificates

and mosquitto.conf file:

listener 8883
protocol mqtt

listener 9001
protocol websockets

allow_anonymous false

cafile ./mosquitto/certs/ca.crt
keyfile ./mosquitto/certs/server.key
certfile ./mosquitto/certs/server.crt

require_certificate true
use_identity_as_username true

tls_version tlsv1.2

Broker is starting fine:

2023-01-02 23:44:16 1672699456: mosquitto version 2.0.15 starting
2023-01-02 23:44:16 1672699456: Config loaded from /mosquitto/config/mosquitto.conf.
2023-01-02 23:44:16 1672699456: Opening ipv4 listen socket on port 8883.
2023-01-02 23:44:16 1672699456: Opening ipv6 listen socket on port 8883.
2023-01-02 23:44:16 1672699456: Opening websockets listen socket on port 9001.
2023-01-02 23:44:16 1672699456: mosquitto version 2.0.15 running

Now, I am trying to use MQTTNet library and I wrote a simple c# console app:

using MQTTnet;
using MQTTnet.Client;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using System.Text.Json;

var caCert = new X509Certificate2("ca.crt");
var clientCert = new X509Certificate2("client.pfx", "password");

var url = "localhost";
var factory = new MqttFactory();
var client = factory.CreateMqttClient();
var options = new MqttClientOptionsBuilder()
    .WithTcpServer(url, 8883)
    .WithClientId("Device1")
    .WithTls(new MqttClientOptionsBuilderTlsParameters
    {
        UseTls = true,
        SslProtocol = System.Security.Authentication.SslProtocols.Tls12,
        CertificateValidationHandler = (certContext) =>
        {
            X509Chain chain = new X509Chain();
            chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
            chain.ChainPolicy.RevocationFlag = X509RevocationFlag.ExcludeRoot;
            chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;
            chain.ChainPolicy.VerificationTime = DateTime.Now;
            chain.ChainPolicy.UrlRetrievalTimeout = new TimeSpan(0, 0, 0);
            chain.ChainPolicy.CustomTrustStore.Add(caCert);
            chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;

            var x5092 = new X509Certificate2(certContext.Certificate);

            return chain.Build(x5092);
        },
        AllowUntrustedCertificates = true,
        Certificates = new List<X509Certificate2>
        {
            clientCert, caCert
        },
    })
    .Build();

await client.ConnectAsync(options);

var message = new MqttApplicationMessageBuilder()
    .WithTopic("sensor/reporting")
    .WithPayload(Encoding.UTF8.GetBytes("test"))
    .Build();
await client.PublishAsync(message);

And I am not able to connect with client error:

MQTTnet.Exceptions.MqttCommunicationException: ‘ Received an unexpected EOF or 0 bytes from the transport stream.’

And server error:

2023-01-02 23:47:00 1672699620: New connection from 172.19.0.1:37912 on port 8883.
2023-01-02 23:47:00 1672699620: Client disconnected due to protocol error.

Tried different options and never succeded.

On the subscriber side, I am following this guide: Dapr MQTT Binding with below config file:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: sensorreporting
spec:
  type: bindings.mqtt
  version: v1
  metadata:
  - name: url
    value: ssl://localhost:8883
  - name: topic
    value: sensor/reporting
  - name: consumerID
    value: "{uuid}"
  - name: caCert
    value: -----BEGIN CERTIFICATE-----nMIIEGzCCAwOgAwIBAgIUYbZRJpxF3knVBs4A9FIc9KU6YskwDQYJKoZIhvcNAQELnBQAwgZwxCzAJBgNVBAYTAkNBMRAwDgYDVQQIDAdBbGJlcnRhMRAwDgYDVQQHDAdDnYWxnYXJ5MRMwEQYDVQQKDAptQ2xvdWRDb3JwMRYwFAYDVQQLDA1DZXJ0QXV0aEFknbWluMQ0wCwYDVQQDDARhZGNhMS0wKwYJKoZIhvcNAQkBFh5taWNoYWwua29zb3dzna2lAbWNsb3VkY29ycC5jb20wHhcNMjMwMTAyMjE0MzU5WhcNMjgwMTAyMjE0MzU5nWjCBnDELMAkGA1UEBhMCQ0ExEDAOBgNVBAgMB0FsYmVydGExEDAOBgNVBAcMB0NhnbGdhcnkxEzARBgNVBAoMCm1DbG91ZENvcnAxFjAUBgNVBAsMDUNlcnRBdXRoQWRtnaW4xDTALBgNVBAMMBGFkY2ExLTArBgkqhkiG9w0BCQEWHm1pY2hhbC5rb3Nvd3NrnaUBtY2xvdWRjb3JwLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBnAMxZASmaTBj+8nVnQwngdjQ3632AVforWdZWSW7a/Z9mhlCis8Lmg/B7Hun9NUh3nlwg3GxB9RN0K3A+aeHObWfSUl+R1NiTVAMWh+CtUVibjWnDJGk+TNQ9Drq5Lh/iXnNPc2ztUm+iaFyWdXcthWjeYqsnLD4NHbD0470F4mNidg93cPvB66c0Eam01pAAkRnQ/jUU0W8gncN3SEHc/FAUahGp1xZzxWhawBAr/oa7xjDMZsz4cLBHjnUH/wNuTrZnxQ7g/ArO9DsDaITj7+tzKvOLkCza3LzjW7Ye19XL5l9DisX3xBNCeERIIdndwkOmndkEif6QIMgZjaYyyMH6JSsECAwEAAaNTMFEwHQYDVR0OBBYEFDDUsLcvER/gEJglnstJ3VutBQ/t+MB8GA1UdIwQYMBaAFDDUsLcvER/gEJglstJ3VutBQ/t+MA8GA1UdnEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBALzcZ32VWo37Da5wfFmONo1In+bnnP4dc6i90/6MVuLoco6nt63XE+DwL53Ib8tPYVMssKCLOBCI+y+tRtpIAIF+yntg41SP5iLIU4eV4rjzdd3iirZOavvcx2cLVfMRHFVr1/8V5o4LaQOpiwVycndcHYnoRTXW6h49YpZt914oEwezSRBrN5h8Rc4cJB/alVpj/2FMq0+C6qQmgJ56xLT9Yu+nAFz2X4zoFX69WsGm+h/cOfrMjqdR96UoS8cUgEjeNPZCxrZLWGEvphnjmjCom+SQnJ+qvQQBCGlNW6Hajd6yBybynB4ImwsqySlWYuX/JwfCDxq1rt++lWjha7hzGSdY=n-----END CERTIFICATE-----
  - name: clientCert
    value: -----BEGIN CERTIFICATE-----nMIIDwDCCAqgCFA7Zd6l6GgZYMtdtTwKHM7b3jn0HMA0GCSqGSIb3DQEBCwUAMIGcnMQswCQYDVQQGEwJDQTEQMA4GA1UECAwHQWxiZXJ0YTEQMA4GA1UEBwwHQ2FsZ2FyneTETMBEGA1UECgwKbUNsb3VkQ29ycDEWMBQGA1UECwwNQ2VydEF1dGhBZG1pbjENnMAsGA1UEAwwEYWRjYTEtMCsGCSqGSIb3DQEJARYebWljaGFsLmtvc293c2tpQG1jnbG91ZGNvcnAuY29tMB4XDTIzMDEwMjIxNTcwMVoXDTIzMTIyODIxNTcwMVowgZsxnCzAJBgNVBAYTAkNBMRAwDgYDVQQIDAdBbGJlcnRhMRAwDgYDVQQHDAdDYWxnYXJ5nMRMwEQYDVQQKDAptQ2xvdWRDb3JwMRIwEAYDVQQLDAlhZC1jbGllbnQxEDAOBgNVnBAMMB2RldmljZTExLTArBgkqhkiG9w0BCQEWHm1pY2hhbC5rb3Nvd3NraUBtY2xvndWRjb3JwLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAOZ2d3jmn240jQ9JdTAoAEe/LaoFD+0q7TisXLeHBruCZWjMCbiqinT+GtvyhOvNVVJk8/0jHn1pTkAIads1hIqve6AUNinrZd9LRbW8CNeESz+w29GcOcdZ9fMsQf42PqHd+Y8AesnF/2TSU9Qu+dVpplrdHOfz5WjmC88/AD9btMDrrJQOhi4MFD8Buy4S5Lrw4ZRQ4ben72hHAuD3nVT+/WS9EJtVSANHaIHsOPDmyJAPFVS6wWZRAHv5BEf5D/UtjmGg3VRnnGN2krK1/MLMSv20kjePi7dErOtAE1Q2fsvEfs0zko3/qrGZlELVlxKqYgnAMfY2wnsVf8sGvTz9sJIKkCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAc/+wE8Efgmi1GB5GnSlPpFbDmM+pAXm7cAbELTkQK4i70TZ4R7aRS9QulxSxL1MwpHkyQqvW9bN0s+WtXnUqh3ERyXCyeuOc6KpskROryyCD03rL2j83KIeTvbLSwsx3gwzE0nBy336Y2A1fYTnJVkYUhi4Gh9LqSovw8yn+O/DjGhzfBPo1MaPZihtVGmTlhvY9ypUsefggC+FE8E8n8UmvesY7H+/h5TammDoHTFBupQG9JIx76DZJFQlyyGGanttfBooya4ZLzpZ+zmeWnFOvz+RmekQzjt+Jrtrzv1zkyrhFB129kr9/cNzZBtR2Hte2ezSgmPHFjIPZ0Z4ZCnTW47cw==-----END CERTIFICATE-----
  - name: clientKey
    value: -----BEGIN RSA PRIVATE KEY-----nMIIEpAIBAAKCAQEA5nZ3eObbjSND0l1MCgAR78tqgUP7SrtOKxct4cGu4JlaMwJunKqKdP4a2/KE681VUmTz/SMfWlOQAhp2zWEiq97oBQ2Ketl30tFtbwI14RLP7Db0Znw5x1n18yxB/jY+od35jwB6wX/ZNJT1C751WmmWt0c5/PlaOYLzz8AP1u0wOuslA6nGLgwUPwG7LhLkuvDhlFDht7vaEcC4PedVP79ZL0Qm1VIA0dogew48ObIkA8VVLrBnZlEAe/kER/kP9S2OYaDdVGcY3aSsrX8wsxK/bSSN4+Lt0Ss60ATVDZ+y8R+zTOSjnf+qsZmUQtWXEqpiCcAx9jbCxV/ywa9PP2wkgqQIDAQABAoIBAQDP28dztxwMFuNjnx5v+ZQPIeHot7oemZnthJ/3M4Sh+EDInUajVMWeMVU+TWUPXFn4/26EQHpVuIppKnz89i79+roQrkoP0u1F0RkliVucLgjEgBoy2qnOIFhRI/i9dGVXDuK31KHjBhF8eGnDvX69uhU9XLJmjja6Psol+NebTXYgIISUz0S39KGqH3BZ9IcRre/peNDh4LqUot9n+Wmrj/4kIgwC2FXLWw5Bz5wH4cvZbrSrf+p4jf8MZehBBTlkJ2270aEwU2Di+rDxnS4o1rzDEB3FO6eoDJk+vxseZhqUjAX7Vm2HDZhEM+y1rXuOXY11ya2phIA7Y4YzXnBuXvNQKBAoGBAP8PI4wk8XjVt5mD6KUbvPzcmEtTG43PrTKh9vXdsP9bkW8+RUEpnpwpguSxPYCXRGSl4A3Ava/8uT8grAx1978LY7l65uCK9EUTdYs5a3BrjhxBVE3Bjnp/e3VSmE8qD1ixlnfcRUf0Dv8CrivKmW4Lz226A3hS6AlWfzI9amEfG5AoGBAOdQnGcKSUijcNkASud4VZyOqNqjKtj5SKx1HblnOq7HERblPW53wD8Mg9jeQ8X7Flaoxn52m1W3EcO1VVLveQ6FYsBXVz+9FwHblztizVlty48XmzpQDbAL1xI+G0WT7cpDt8nDdnlOmZX4hxBsmrkOFz/gP4yn6sL0u0wbRTk995xAoGAcBe1Jaguv04djEgefEF+ngxpni+MQXviAQ5pOsKsnWHJjriINyZ+Gm9b6SnDv7m7AwirOSAVvTwyJX40Xp2RrndJRl3hd8xzeUWVF6VPo1dVO9Tf41ttT1Qra9pKua/gYrhZSn/xqrelx4jkBrbZSznXBiZWQZNb6xu/OosO+9sP3ECgYEA3KoVaQCdUE2Dk2xJ1Asm8WcX7b40TQ+mAbB7nHUGDmXpb6YRDKlAxOTzgpfGn8AAd3tbspHNWYHlGQQmaXKHogwxLyNh0ikAimyA/n0lWQLdBgzHEMH9zMv+yHBnI+ETH7mqOlsxByvd9o3PvvcPl2q4EzVUyF9W66MXvZnE+26aIECgYBtDK+8hffO3N1yXW+tMFuCbbzMPrPJC5o4VII8pm+oNcsvW1Ao9kpZnL7yVVJCozo0omvuY2ohVchiQ5jC1nxd17Xa0DuIrKgtcEmpoae16e2h7kcSjv63ynbKEEUIKRyWfhx6iJb8o03lq1UUxRDhaB5leiUUddibJyhh7EDfCHfQ==n-----END RSA PRIVATE KEY-----

and dapr sidecar is failing with following error:

mqtt-sample-mqtt-api-dapr-1 | time="2023-01-02T23:19:05.428410597Z" level=fatal msg="process component sensorreporting error: [INIT_COMPONENT_FAILURE]: initialization error occurred for sensorreporting (bindings.mqtt/v1): mqtt binding error: invalid ca certificate" app_id=mqtt-api instance=2c64804e20f1 scope=dapr.runtime type=log ver=1.9.5
mqtt-sample-mqtt-api-dapr-1 exited with code 1

I do not know where I am making mistake. I am going to test that on AKS now (maybe the problem is with running it locally? Anonymous mode is working with AKS).

I was able to set up the whole thing without SSL, but need this to go to prod.