Question 121448 in Redis Question posted in
The official documentation can be found here.

Redis – kafka streams – can I use kafka streams processing in cases where the source is not a kafka topic?

I have an application (call it smscb-router) as shown in the diagram.

  • It reads data from a legacy system (sms).
  • Based on the content (callback type), I have to put into corresponding outgoing topic (such as billing-n-cdr, dr-cdr, …)

I think streams API is better suited in this case, as it has the map functionality to do the content mapping check. What I am unsure is, can I read source data from a non-kafka-topic source.

All the examples that I see on the internet blogs, explain steaming apps with the context of reading from a source topic and put to other destination topics.

So, is this possible to read from a non-topic source, such as say a redis store, or a message queue such as RabbitMQ?

enter image description here

Tags:

2

Answers


  1. 0

    We had a recent implementation, where we had to poll an .xml file from a network attached drive and convert it into the KAFKA Events i.e. publishing each record into an output topic. In such, we wont even call it as something we have developed using a Streams API, but it is just a KAFKA Producer Component.

    Java File Poller Module (Quartz time based) -> XML Schema Management -> KAFKA Producer Component -> Output Topic (KAFKA Broker).

    And you will get all native features of KAKFA Producer API in terms of retries and you can use producer.send (Sync) or producer.send.get(Asyn) with call-back.

    Hope this helps. Streams API is meant for big and something very complex that to be normalized through using Stateful operations.

    Thanks,
    Christopher

    Login or Signup to reply.
  2. 0

    Kafka Streams is only about Topic to Topic Data Streaming

    All external system should be integrated by another method :

    Ideally Kafka Connect : for example with this one :
    https://docs.confluent.io/kafka-connect-rabbitmq-source/current/overview.html

    You may also use a manual consumer for the first step, but it always better to reuse all availability mecanism built in Kafka Connect. (No code, just some Json config).

    In your schema i would recommend to add one topic and one producer or one connector in front of your Pink Component, then it can become a fully standard Kafka Streams microservice.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search