Question 102338 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – Too many RDS connections while load testing http server on AWS Lambda with RDS proxy

Introduction

I am hosting a web server implemented by gorilla/mux which expose API endpoints for users to access the database.

I put the server in a AWS Lambda function and use API Gateway to send Lambda proxy requests to the server. The request flow is:

  • API Gateway -> (API Gateway Lambda proxy integration) -> Lambda function -> RDS proxy -> RDS instance

The server code

import (
    "github.com/gorilla/mux"
    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/awslabs/aws-lambda-go-api-proxy/core"
    "github.com/awslabs/aws-lambda-go-api-proxy/gorillamux"
)

var (
    muxLambda *gorillamux.GorillaMuxAdapter
    router = mux.NewRouter().StrictSlash(true)
    // Register handler functions
)

func init() {
    muxLambda = gorillamux.New(router)
}

func LambdaHandler(
    ctx context.Context,
    req events.APIGatewayProxyRequest,
) (*core.SwitchableAPIGatewayResponse, error) {
    response, err := muxLambda.ProxyWithContext(ctx, *core.NewSwitchableAPIGatewayRequestV1(&req))
    if err != nil {
        panic(fmt.Sprintf("Proxy encountered an error: %s", err.Error()))
    }
    return response, nil
}

func main() {
    lambda.Start(LambdaHandler)
}
  • I open the connection to database via:
import "database/sql"
database, err := sql.Open("postgres", connStr)

Problem

While testing the load performance of this server (siege -c 250 -d 0.5 -r 10 'http://<api url>/),
I encountered so many connections to the RDS instance (around 400 connections in the AWS console) and the Lambda function returned timeout (30 sec) for almost all transactions.
If I lower the number of process, the response from the server works as expected.

What are the possible things that cause this?

From my search, I found that maybe it is due to:

  • For each request, a new Lambda instance starts and opens a new database connection to the RDS proxy. With too many connections already there, somehow the Lambda waits and return timeout after 30 seconds.

This might be similar to this question, but I think RDS proxy can handle connection pool for me? Or am I overloading the RDS proxy?

If the cause is overloading, what configuration can I do in the AWS setup?
Scaling up the RDS instances? Or Increase max_connections setting as mentioned in this article?

Or simply change the code structure as mentioned in this SO question or this SO question?

Thanks for reading this far. Any feedback is appreciated.

Updated: The connection metrics of RDS and RDS proxy

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    I found the reason that Lambda timeout might due to the fact of too many RDS connections. So the RDS proxy queues or throttles incoming connections from the Lambda (ref).

    The root cause was because of too many SQL queries from the API server sending to the RDS, which holds up the connections. In my case, the query includes 2 steps:

    1. Getting a list of id
    2. Sends a query for each id such as:
    select row from table where id = 1;

    In this setting, if there are 200 id in the list, then the API server will send 200 + 1 queries in total, which holds up the connection.

    Solution

    • I modified the API server to send a single query like:
    select row from table where id in (1, 2, 3, ....);
    • Close the rows from Db.Query (from database/sql) calls to avoid connection leak:
    rows, err := Db.Query(query, params)
defer rows.Close()

    In summary, this was due to poor understanding of the database/sql go package and SQL. Hopefully this helps those who comes across this.


  2. 0

    Increase value of max_connections is not best practice.
    I think, the better way to use Lambda function init phase to pre-configure RDS connection. This connection will be reused by several lambda functions. This solution decreases number of connections to RDS database.
    There are good example how to do this.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search