Question 118927 in Firebase Question posted in
An extensive official documentation can be found here.

Why are we suddenly getting "Function execution took 60000 ms, finished with status: 'error'" on Firebase callable functions / google cloud functions?

Our app is in production since the March 22nd. It makes use of Firebase callable functions. The execution of the functions were flawless until suddenly on March 28th, we started to observe many failed executions get the following debug messages in the cloud functions logs Function execution took 60000 ms, finished with status: 'error'.

While some executions of the same function failed, others worked as usual and completed in on average 250ms.

When checking the cloud platform infrastructure status, there were no indicators, of any outages or any other issues.

For instance we have this very simple function:

 exports.getAccommodations = functions
  .region('asia-east1')
  .https.onCall(async (data, context) => {
    functions.logger.warn(`V3 STARTED`);
    // load some collection from firestore here and return it
  });

Note that the very first line of the function is creating a log warn entry. We were able to observe the following:

  • For failed executions with the above error message, there was no log.warn entry.
  • For succeeded executions, there was always an according log.warn entry.

In other words, it looks like failed executions, do not even bother to execute the functions body. All we see in the failure case is Function Execution started followed by Function execution took 60000 ms, finished with status: 'error'.

These are our logs, corroborating our observations:

  • When filtering for the failure case: (resource.type="cloud_function" ) OR (resource.type="cloud_run_revision" ) textPayload="Function execution took 60000 ms, finished with status: 'error' we get: this chart of failures

  • When listing all executions (resource.type="cloud_function" ) OR (resource.type="cloud_run_revision" ) we get: this chart of executions

Note that we have not deployed anything around the time the issues started to show.

Tags:

2

Answers


  1. 0

    Please check your code thoroughly.

    Often it is a problem with a wrong async/await without any code file specifics.

    If you use any express middleware, do not forget to call the next function, e.g.:

    export const someMiddleware = async (
  req: Request,
  res: Response,
  next: NextFunction,
) {
  // await some of your operations

  // notice that "next" does not need to be awaited, await can cause functions time out
  next(); 
}
    Login or Signup to reply.
  2. 0

    Seeing the same issue, most calls end up with status code 408 and have much higher latency than we usually see.

    Think it’s related to this, which is very broadly phrased:

    Summary: Google Cloud Networking experiencing elevated latency in multiple regions

    https://status.cloud.google.com/incidents/nieR2aLyg1rwFKq1aWZU#2c2sBHWU84yPDJ8y1ar4

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search