I need to parse data from Kafka in ClickHouse using Kafka Engine, as example, I have created a table, in https://fiddle.clickhouse.com/0e89bec6-4e76-410a-9fc4-cf58ace5f34f, CREATE TABLE json(name String, data Array(Map(String, String)) ) ENGINE = Memory; INSERT INTO JSON FORMAT JSONEachRow {"name": "asd", "data":[{"id":"1",…
From Debezium official page, there's this picture showing that multiple Debezium connector can connect to the same Kafka. So I have 2 databases, 2 Debeziums, 1 Kafka running in docker-compose, but it seems like only 1 debezium sent update to…
Is it possible to use the AWS CLI tool to list the Kafka MSK topics and display the configuration of them?
I'm trying to deploy kafka on local k8s, then I need to connect to it by application and using offset explorer so, using kubectl I created zookeeper service and deployment using this yml file apiVersion: v1 kind: Service metadata: labels:…
I was reading through the Microsoft Azure Event Hub documentation , but couldn't find mention of dead-letter-topic and/or dead-letter-queue . I know that Kafka has support for dead-letter-topic and dead-letter-queue My question is , does Microsoft Azure Event Hub have…
I want to control the throughtput of a JDBC source kafka connector. I have a lot of data stored in a PostgreSQL table and I wand to ingest it into a Kafka topic. However I would like to avoid a…
to start with - I am a sort of newbie to Kubernetes and I might omit some fundamentals. I have a working containerized app that is orchestrated with docker-compose (and works alright) and I am rewriting it to deploy into…
We are using AWS MSK connect "connectors". The cost of these connectors (runconnect API) by far make up the biggest chuck of the overall costs of using MSK in AWS in our case (75%). However, it looks like it is…
I have a database in MongoDB. That is updating every minute from various user PCs. Now I want to store the data into Amazon S3 bucket (preferable in Parquet, otherwise CSV). But I do not want to store the full…
I'm using docker-compose to setup a kafka broker the following way: kafka: image: confluentinc/cp-kafka:latest depends_on: - zookeeper ports: - 29092:29092 - 9092:9092 environment: KAFKA_BROKER_ID: 1 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 command: sh -c "(sleep 15)…