Question posted in
The official documentation can be found here.

Redis – Python Multi-Process run multiple times

I am trying to run two python files (A.py and B.py) at the same time. I tried to run them in two different and use two separate commands to run them, it works well.

The problem is, I hope to have a script file to run them in parallel. I tried multi-processing in as the following code:

if __name__ == '__main__':
    jobs=[]
    jobs.append(Process(target=A.start))
    jobs.append(Process(target=B.start))

    for job in jobs:
        job.start()

    for job in jobs:
        job.join()

The result is it runs A and B twice, which I hope them to run only once for each of them.

What is the cause for the problem and how can I solve it? Or is there any other solution that I can run two python files in parallel?

Thanks for your help in advance.

For my import information, I have three files: A.py, B.py and run.py.

In A.py, I have:

from scapy.all import *
from scapy.layers.http import HTTPRequest 
from scapy.layers.http import HTTPResponse
from colorama import init, Fore
import docker
import time
import redis

In B.py, I have:

import json
import docker
import socket
import time
import psutil
import socket
import redis
import prometheus_client
from prometheus_client import Gauge,Counter
from prometheus_client.core import CollectorRegistry
from flask import Response, Flask

And in the run.py, I have:

from multiprocessing import Process
import A
import B

====UPDATE PROBLEM CAUSE====

After hours of playing with the code, I think I found the cause, but I do not understand why.

The cause is in my A.py file, I have a code

#flask starts
app.run(host="0.0.0.0", port=15600,debug=True)

If I removed the debug mode, the code works well. The problem only occurs when the debug mode is on.

Does anyone know why it happens?

Tags:

2

Answers


  1. 0

    First I would make sure, that in your scripts, you have the script you want to run inside a callable function. Then make sure you import the function at the start like from fileA import A.

    from multiprocessing import Process
import time
# from fileA import A
# from fileB import B


def A():
    print("Running A")
    time.sleep(2)
    print("Ending A")


def B():
    print("Running B")
    time.sleep(3)
    print("Ending B")

if __name__ == '__main__':
    jobs=[]
    jobs.append(Process(target=A))
    jobs.append(Process(target=B))

    for job in jobs:
        job.start()

    for job in jobs:
        job.join()

    Output:

    Running A
Ending A
Running B
Ending B
[Finished in 3.2s]
    Login or Signup to reply.
  2. 0

    I think what you are looking for is multiprocessing.Pool, and specifically Pool.apply_async. This will allow you to asynchronously start separate processes that will perform unrelated tasks in parallel. You can use it like this:

    A.py:

    def square(x):
    print('x squared is: ' + str(x**2))

    B.py:

    def cube(x):
    print('x cubed is: ' + str(x**3))

    then:

    import multiprocessing as mp
import A
import B

p = mp.Pool(2)

a = p.apply_async(A.square, args=(5,))
b = p.apply_async(B.cube, args=(7,))

p.close()

    Update

    OK, given the new info, there is a very simple but perhaps non-obvious reason this is happening, and it has nothing to do with your use of multiprocessing (although that is very likely to give you trouble down the road if you’re planning to run this app in a real web server and not the one built in to Flask).

    The Flask debug server starts 2 processes when you run it in debug mode with use_reloader turned on. The parent process monitors your code and when a change is detected it signals the child to reload. This is why the Flask server is able to instantly apply any code changes you make without you needing to perform any action. The problem you are having is that your code is being run in both the parent and the child.

    There are 2 ways to fix it.

    1. Don’t use debug=True.
    2. Use debug=True but also disable the reloader by adding use_reloader=False.

    Option (2) is probably the more attractive solution since you can still access the debugging console, but you will also lose the automatic reload functionality, so you’ll need to restart the server manually before your changes are applied. To do it, change your app.run line to this:

    app.run(host="0.0.0.0", port=15600, debug=True, use_reloader=False)
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search