to set the context, i have a very basic kafka server setup through a docker-compose.yml file, and then i spin up a ui app for kafka. depending on the config, the ui app will/wont work becuase of port 8080 being…
I'm trying to create schema for avro messages for adding to schema registry in Kafka. Since I get messages from mongodb the structure of messages differs from one message to another. For example, one of the field are present in…
I'm consuming a Kafka topic published by another team (so I have very limited influence over the message format). The message has a field that holds an ARRAY of STRUCTS (an array of objects), but if the array has only…
Thank you in advance for any help! My company uses multiple AWS accounts to separate region and "global" functionality/storage. We've recently switched to using MQTT IoT Rules and have run tests with regional Kafka vs a global Kafka instance for…
A Kafka topic for my table is not created when using the debezium/connect Docker image. Here's how I'm starting the container: docker run -it --rm --name debezium -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e BOOTSTRAP_SERVERS=192.168.56.1:9092 -e CONNECT_NAME=my-connector -e…
A Kafka topic is not created when I use start the Debezium connector for PostgreSQL. Here's what I have in my properties file: name=testdb connector.class=io.debezium.connector.postgresql.PostgresConnector topic.prefix=test database.hostname=localhost database.port=5432 database.user=postgres database.password=root database.dbname=testdb database.server.name=testdb table.include.list=ipaddrs plugin.name=pgoutput According to this, the topic should…
I am writing a pub/sub implementation which uses Azure EventHub as the underlying event ingestion service. In my application, the publishers will publish events to a particular EventHub partition and the consumers who are subscribed to that particular partition will…
As the title states, I'm using debezium Postgres source connector and I would like MongoDB sink connector to group kafka topics in different collection and databases (also different dbs to isolate unrelated data) according to their names. While inquiring I…
Couple of things to know: Using a raspberry pi 4 Running Unbuntu 20.04 image on ri4 I use ZeroTier and SSH to connect remotely to the ri4 I was able to run 3 containers: nodered, mosquito and portainer. I'm having…
This is how my docker file looks like, version: "3.9" services: zookeeper: image: 'bitnami/zookeeper:latest' ports: - '2181:2181' environment: - ALLOW_ANONYMOUS_LOGIN=yes kafka: image: 'bitnami/kafka:latest' ports: - '9092:9092' - '9093:9093' environment: - KAFKA_BROKER_ID=1 - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,EXTERNAL://:9093 - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092,EXTERNAL://localhost:9093 - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181 -…