I want to control the throughtput of a JDBC source kafka connector. I have a lot of data stored in a PostgreSQL table and I wand to ingest it into a Kafka topic. However I would like to avoid a…
to start with - I am a sort of newbie to Kubernetes and I might omit some fundamentals. I have a working containerized app that is orchestrated with docker-compose (and works alright) and I am rewriting it to deploy into…
We are using AWS MSK connect "connectors". The cost of these connectors (runconnect API) by far make up the biggest chuck of the overall costs of using MSK in AWS in our case (75%). However, it looks like it is…
I have a database in MongoDB. That is updating every minute from various user PCs. Now I want to store the data into Amazon S3 bucket (preferable in Parquet, otherwise CSV). But I do not want to store the full…
I'm using docker-compose to setup a kafka broker the following way: kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 29092:29092 - 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 command: sh -c "(sleep 15)…
to set the context, i have a very basic kafka server setup through a docker-compose.yml file, and then i spin up a ui app for kafka. depending on the config, the ui app will/wont work becuase of port 8080 being…
I'm trying to create schema for avro messages for adding to schema registry in Kafka. Since I get messages from mongodb the structure of messages differs from one message to another. For example, one of the field are present in…
I'm consuming a Kafka topic published by another team (so I have very limited influence over the message format). The message has a field that holds an ARRAY of STRUCTS (an array of objects), but if the array has only…
Thank you in advance for any help! My company uses multiple AWS accounts to separate region and "global" functionality/storage. We've recently switched to using MQTT IoT Rules and have run tests with regional Kafka vs a global Kafka instance for…
A Kafka topic for my table is not created when using the debezium/connect Docker image. Here's how I'm starting the container: docker run -it --rm --name debezium -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my-connect-configs -e OFFSET_STORAGE_TOPIC=my-connect-offsets -e BOOTSTRAP_SERVERS=192.168.56.1:9092 -e CONNECT_NAME=my-connector -e…