List topics and their configuration with AWS CLI – Amazon Web Sevices
Is it possible to use the AWS CLI tool to list the Kafka MSK topics and display the configuration of them?
There is error message KafkaAdminClient when try to connect to local kafka broker deployed on local k8s – Nginx
I'm trying to deploy kafka on local k8s, then I need to connect to it by application and using offset explorer so, using kubectl I created zookeeper service and deployment using this yml file apiVersion: v1 kind: Service metadata: labels:…
Does Microsoft Azure Event Hub have support for dead-letter-topic and dead-letter-queues , like vanilla Kafka has?
I was reading through the Microsoft Azure Event Hub documentation , but couldn't find mention of dead-letter-topic and/or dead-letter-queue . I know that Kafka has support for dead-letter-topic and dead-letter-queue My question is , does Microsoft Azure Event Hub have…
Slow down kafka connector JDBC source – control throughput – Postgresql
I want to control the throughtput of a JDBC source kafka connector. I have a lot of data stored in a PostgreSQL table and I wand to ingest it into a Kafka topic. However I would like to avoid a…
Kafka & Python app on Kubernetes in separate pods – NoBrokersAvailable() – Docker
to start with - I am a sort of newbie to Kubernetes and I might omit some fundamentals. I have a working containerized app that is orchestrated with docker-compose (and works alright) and I am rewriting it to deploy into…
is it possible to add tags to kafka connect connectors in AWS MSK – Amazon Web Sevices
We are using AWS MSK connect "connectors". The cost of these connectors (runconnect API) by far make up the biggest chuck of the overall costs of using MSK in AWS in our case (75%). However, it looks like it is…
Incremental Data Storage from MongoDB to Amazon S3 in Parquet Format
I have a database in MongoDB. That is updating every minute from various user PCs. Now I want to store the data into Amazon S3 bucket (preferable in Parquet, otherwise CSV). But I do not want to store the full…
Kafka docker compose configuration only creates one partition
I'm using docker-compose to setup a kafka broker the following way: kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 29092:29092 - 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 command: sh -c "(sleep 15)…
with this docker-compose configuration, why is 8080 considered used?
to set the context, i have a very basic kafka server setup through a docker-compose.yml file, and then i spin up a ui app for kafka. depending on the config, the ui app will/wont work becuase of port 8080 being…
Mongodb – Missing field in avro schema
I'm trying to create schema for avro messages for adding to schema registry in Kafka. Since I get messages from mongodb the structure of messages differs from one message to another. For example, one of the field are present in…