Question 121738 in Redis Question posted in
The official documentation can be found here.

Redis – Why use Thread.sleep in flink decrease performance obviously?

We are comparing storm and flink. What we do is develop storm topology logic to flink app.
There is using Thread.sleep(100) in the storm topology bolt, but when using in flink app, flink app performance is not good.
Here is our test result with Flink parallelism 80:

Consume 161073 data from kafka and compute our business logic in next task, and last sink result to file.
When using Thread.sleep(100) in Compute and Redis task, it spend about 4 minutes. But if we not using Thread.sleep(100), it only spend 6 seconds.

My Flink app

Tags:

2

Answers


  1. 0

    I think this is mainly due to back pressure

    Back pressure will cause upstream processing to be interrupted, waiting for downstream operators to consume

    For more information, please refer to

    https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/monitoring/back_pressure/

    Login or Signup to reply.
  2. 0

    I’m wondering where Thread.sleep(100) is located at? If it happens in the function that processes data, that makes sense. Thread.sleep(100) will be called every time a record is processed. Since there are 161073 records in total, the overall time of sleeping is 161073 / 80 * 100 / 1000 = 201.3 seconds, which is about 4 minutes.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search