Question 369192 in Firebase Question posted in
An extensive official documentation can be found here.

Is it possible to call asyncio.run() multiple times with firebase?

I encounter a RuntimeError: Event loop is closed error when trying to run asyncio.run() multiple times with Firebase. Is this expected? Here is a simplified example of the issue:

import streamlit as st
import firebase_admin
from firebase_admin import credentials, firestore_async
import json
import asyncio

key_dict = json.loads(st.secrets["textkey"])
cred = credentials.Certificate(key_dict)
app = firebase_admin.initialize_app(cred)
db_async = firestore_async.client(app)


async def _upload_async():
    dicts_to_upload = {
        "doc_1": {"key_1": "val_1"},
        "doc_2": {"key_2": "val_2"},
    }

    tasks = []

    for key in dicts_to_upload.keys():
        result_dict = db_async.collection("test-collection").document(key)
        task = result_dict.set(dicts_to_upload[key])
        tasks.append(task)

    await asyncio.gather(*tasks)


asyncio.run(_upload_async())
print("nnSuccess1nn")
asyncio.run(_upload_async())
print("nnSuccess2nn")

When running this, the output is:



Success1


Traceback (most recent call last):
  File "/Users/colinrichter/Documents/GitHub/YCN-Calculator/testing.py", line 100, in <module>
    asyncio.run(_upload_async())
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/colinrichter/Documents/GitHub/YCN-Calculator/testing.py", line 95, in _upload_async
    await asyncio.gather(*tasks)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/cloud/firestore_v1/async_document.py", line 131, in set
    write_results = await batch.commit(**kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/cloud/firestore_v1/async_batch.py", line 60, in commit
    commit_response = await self._client._firestore_api.commit(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/cloud/firestore_v1/services/firestore/async_client.py", line 1055, in commit
    response = await rpc(
               ^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary_async.py", line 230, in retry_wrapped_func
    return await retry_target(
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary_async.py", line 160, in retry_target
    _retry_error_helper(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_base.py", line 212, in _retry_error_helper
    raise final_exc from source_exc
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/retry/retry_unary_async.py", line 155, in retry_target
    return await target()
                 ^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/google/api_core/grpc_helpers_async.py", line 166, in error_remapped_callable
    call = callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/grpc/aio/_channel.py", line 150, in __call__
    call = UnaryUnaryCall(
           ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/grpc/aio/_call.py", line 565, in __init__
    self._invocation_task = loop.create_task(self._invoke())
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 434, in create_task
    self._check_closed()
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 519, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'UnaryUnaryCall._invoke' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

I also confirmed in Firebase that it is uploading the test-collection correctly the first time. However, when running it the second time I encounter the error. If I replace the firebase functions with a regular async function that does not use Firebase (like just adding something to doc_1 and doc_2) then it works as expected without this error.

I am doing this because I have a website on Streamlit where users fill out a form which updates some dictionaries, and then I upload those dictionaries to Firebase. I am trying to do this asynchronously to increase performance, but I can only get it to work the first time a user fills out the form. All subsequent times, I run into a RuntimeError: Event loop is closed error.

AmI doing something incorrectly, or is this a bug/feature of Firebase?

Tags:

2

Answers


  1. 0

    If you are happy with this design, just change the control flow lines at the end to use the same loop. Istead of:

    asyncio.run(_upload_async())
print("nnSuccess1nn")
asyncio.run(_upload_async())
print("nnSuccess2nn")

    Do

    loop = asyncio.new_event_loop()  # Maybe change this line to the very
                                # beggining, after the imports.
loop.run_until_complete(_upload_async())
print("nnSuccess1nn")
loop.run_until_complete(_upload_async())
print("nnSuccess2nn")

    What is taking place there is that the firebase async client is binding itself to the running loop the first time it is actually used, and, when you call asyncio.run, a new loop is created. It probably would work if you’d recreate the app and db_async instances between calls.

    The "right thing to do" however, would be to refactor this code and put all of these instances and calls out of "top level" code and into an asynchronous main function, though – the only line of code outside any function should be one calling the loop to run this controlling function.

    Login or Signup to reply.
  2. 0

    It is possible to run asyncio.run twice, but it isn’t recommended. Here is an example that runs it twice:

    import asyncio

async def main():
    await asyncio.sleep(1)
    print("Running main")

asyncio.run(main())
asyncio.run(main())

    This prints out:

    Running main
Running main

    But asyncio.run is meant to run the async part of your program from start to finish. It isn’t necessarily meant to be called multiple times. From the documentation:

    If loop_factory is not None, it is used to create a new event loop; otherwise asyncio.new_event_loop() is used. The loop is closed at the end. This function should be used as a main entry point for asyncio programs, and should ideally only be called once.

    From your error, it indeed seems like Firebase is doing something with the event loop. I haven’t ever used Firebase, so I don’t know. I would look into what the functions:

    app = firebase_admin.initialize_app(cred)
db_async = firestore_async.client(app)

    are doing. Are they startup up an event loop or doing any management of one? I would also investigate the lines:

        for key in dicts_to_upload.keys():
        result_dict = db_async.collection("test-collection").document(key)
        task = result_dict.set(dicts_to_upload[key])
        tasks.append(task)

    and verify that they are creating proper tasks.

    In terms of what to do differently, you should (generally speaking) have a single asyncio event loop that stays running that handles everything you need it to. Calling asyncio.run multiple times should only really be needed when you have multiple asyncio event loops, each running on different threads, but that seems like it would be rare indeed.

    I don’t know what the rest of your program looks like, but a nice way to offload work onto an asyncio event loop is to send it messages via an asyncio.Queue. Here’s some relevant StackOverflow answers covering this:

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search