I'm consuming a Kafka topic published by another team (so I have very limited influence over the message format). The message has a field that holds an ARRAY of STRUCTS (an array of objects), but if the array has only…
Thank you in advance for any help! My company uses multiple AWS accounts to separate region and "global" functionality/storage. We've recently switched to using MQTT IoT Rules and have run tests with regional Kafka vs a global Kafka instance for…
A Kafka topic for my table is not created when using the debezium/connect Docker image. Here's how I'm starting the container: docker run -it --rm --name debezium -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e BOOTSTRAP_SERVERS=192.168.56.1:9092 -e CONNECT_NAME=my-connector -e…
A Kafka topic is not created when I use start the Debezium connector for PostgreSQL. Here's what I have in my properties file: name=testdb connector.class=io.debezium.connector.postgresql.PostgresConnector topic.prefix=test database.hostname=localhost database.port=5432 database.user=postgres database.password=root database.dbname=testdb database.server.name=testdb table.include.list=ipaddrs plugin.name=pgoutput According to this, the topic should…
I am writing a pub/sub implementation which uses Azure EventHub as the underlying event ingestion service. In my application, the publishers will publish events to a particular EventHub partition and the consumers who are subscribed to that particular partition will…
As the title states, I'm using debezium Postgres source connector and I would like MongoDB sink connector to group kafka topics in different collection and databases (also different dbs to isolate unrelated data) according to their names. While inquiring I…
Couple of things to know: Using a raspberry pi 4 Running Unbuntu 20.04 image on ri4 I use ZeroTier and SSH to connect remotely to the ri4 I was able to run 3 containers: nodered, mosquito and portainer. I'm having…
This is how my docker file looks like, version: "3.9" services: zookeeper: image: 'bitnami/zookeeper:latest' ports: - '2181:2181' environment: - ALLOW_ANONYMOUS_LOGIN=yes kafka: image: 'bitnami/kafka:latest' ports: - '9092:9092' - '9093:9093' environment: - KAFKA_BROKER_ID=1 - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,EXTERNAL://:9093 - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092,EXTERNAL://localhost:9093 - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 -…
I'm trying to use Debezium with Kafka connect, I followed this tutorial, and everything connected just fine. However, the problem is that I cannot access Kafka from outside of docker containers anymore. I use these commands to start containers: docker…
I am inserting some data to mongo with springboot application, after that I send this data to kafka. I want to be sure that before I send them to kafka they are commited to mongo. Is there a way I…