Question 256718 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

Amazon web services – API-Gateway integration with AWS Service / DDB with URL Encoding

trying to make PhoneTrack (https://gitlab.com/eneiluj/phonetrack-android/-/wikis/userdoc) work with my API-Gateway to receive location data. I have setup my API to work with a post request, and if I curl it with encoding application/jason, it works fine. the issue is that the app is using encoding of application/x-www-form-urlencoded.

I have setup two mapping templates and I have tried a few combinations to make the mapping work, but the x-www-form-urlencoded template just does not seem to do anything, and I think I have not fully understood how to get this right.

here my terraform resource:

resource "aws_api_gateway_integration" "ddb_integration" {
  depends_on              = [aws_api_gateway_method.generic_post]
  rest_api_id             = aws_api_gateway_rest_api.generic_api.id
  resource_id             = aws_api_gateway_resource.generic_resource.id
  http_method             = aws_api_gateway_method.generic_post.http_method
  passthrough_behavior    = "NEVER"
  content_handling        = "CONVERT_TO_TEXT"
  integration_http_method = "POST"
  uri                     = "arn:aws:apigateway:${local.workspace.aws_region}:dynamodb:action/PutItem"
  credentials             = aws_iam_role.api_gw_to_ddb.arn
  type                    = "AWS"
  request_templates = {
    "application/json" = <<-EOF
    {
      "TableName": "${aws_dynamodb_table.generic_data.name}",
      "Item": {
        "lon": {
          "N": "$input.json('$.lon')"
        },
        "lat": {
          "N": "$input.json('$.lat')"
        },
        "timestamp": {
          "S": "$input.json('$.timestamp')"
        }
      }
    }
    EOF

    "application/x-www-form-urlencoded" = <<-EOF
    {
      $util.log("Starting processing input body...")
      #foreach( $token in $input.body.split('&') )
          $util.log("Current token: $token")
          
          #set( $keyVal = $token.split('=') )
          #set( $keyValSize = $keyVal.size() )
          $util.log("Split token into key and value. KeyValSize: $keyValSize")
          
          #if( $keyValSize >= 1 && !$keyVal[0].isEmpty() )
              #set( $key = $util.urlDecode($keyVal[0]) )
              $util.log("Decoded key: $key")
              
              #set($key = $key.substring(0,1).toLowerCase() + $key.substring(1))
              $util.log("Processed key with first letter lower-cased: $key")
              
              #if( $keyValSize >= 2 )
                  #set( $val = $util.urlDecode($keyVal[1]) )
                  $util.log("Decoded value: $val")
                  
                  #if( $key == "lon" || $key == "lat" || $key == "timestamp" )
                      $util.log("$key: $val")
                  #end
                  
              #else
                  #set( $val = '' )
                  $util.log("No value found for key. Setting value to empty.")
              #end
              
              "$key": "$util.escapeJavaScript($val)"#if($foreach.hasNext),#end
              
          #else
              $util.log("Error processing token: $token. Either key is missing or empty.")
          #end
      #end
      $util.log("Finished processing input body.")
    }
    EOF
  }
}

This works for the application/json version. The logging I have tried to setup in the Apache Velocity Template Language, template code does not seem to be logging out to cloudwatch, so i dont really seem to have a way to look into how this works. I have also tried to use the simplest version of this for url encoding as:

{
    "TableName": "${aws_dynamodb_table.generic_data.name}",
    "Item": {
      "lon": {
        "N": "$util.urlDecode($input.params('lon'))"
      },
      "lat": {
        "N": "$util.urlDecode($input.params('lat'))"
      },
      "timestamp": {
        "S": "$util.urlDecode($input.params('timestamp'))"
      }
    }
  }

but this also does not work. Below my execution logs from APIG, the obvious issue being a serialisation exception.

The problem is obviously that the URL encoded object payload is not being converted into something that DDB can understand. So the issue is that the mapping template is not doing its magic. Will this work, or do I need to use a lambda to get this to function?

The Apache VTL does not seem to do anything for URL encoded payloads, however, it does seem to process fine for json encoded payloads. Is there a way to get the logging feature working with VTL?

(REQUEST_ID_REMOVED) Extended Request Id: MGm12FcyoAMEsgA=
(REQUEST_ID_REMOVED) Verifying Usage Plan for request: REQUEST_ID_REMOVED. API Key:  API Stage: STAGE_ID_REMOVED/prod
(REQUEST_ID_REMOVED) API Key  authorized because method 'POST /data' does not require API Key. Request will not contribute to throttle or quota limits
(REQUEST_ID_REMOVED) Usage Plan check succeeded for API Key  and API Stage STAGE_ID_REMOVED/prod
(REQUEST_ID_REMOVED) Starting execution for request: REQUEST_ID_REMOVED
(REQUEST_ID_REMOVED) HTTP Method: POST, Resource Path: /data
(REQUEST_ID_REMOVED) Method request path: {}
(REQUEST_ID_REMOVED) Method request query string: {}
(REQUEST_ID_REMOVED) Method request headers: {User-Agent=curl/8.3.0, X-Forwarded-Proto=https, X-Forwarded-For=REMOVED_IP, content-type=application/x-www-form-urlencoded, Host=REMOVED, X-Forwarded-Port=443, accept=*/*}
(REQUEST_ID_REMOVED) Method request body before transformations: lon=151.0&lat=-33.0&timestamp=1695994041
(REQUEST_ID_REMOVED) Endpoint request URI: https://dynamodb.us-east-1.amazonaws.com/?Action=PutItem
(REQUEST_ID_REMOVED) Endpoint request headers: {Authorization=REMOVED, X-Amz-Date=20231001T034528Z, x-amzn-apigateway-api-id=STAGE_ID_REMOVED, Accept=application/x-www-form-urlencoded, User-Agent=AmazonAPIGateway_STAGE_ID_REMOVED, X-Amz-Security-Token=REMOVED
(REQUEST_ID_REMOVED) Endpoint request body after transformations: lon=151.0&lat=-33.0&timestamp=1695994041
(REQUEST_ID_REMOVED) Sending request to https://dynamodb.us-east-1.amazonaws.com/?Action=PutItem
(REQUEST_ID_REMOVED) Received response. Status: 400, Integration latency: 6 ms
(REQUEST_ID_REMOVED) Endpoint response headers: {Server=Server, Date=Sun, 01 Oct 2023 03:45:28 GMT, Content-Type=application/x-amz-json-1.0, Content-Length=60, Connection=keep-alive, x-amzn-RequestId=REMOVED, x-amz-crc32=REMOVED}
(REQUEST_ID_REMOVED) Endpoint response body before transformations: {"__type": "com.amazon.coral.service#SerializationException"}
(REQUEST_ID_REMOVED) Execution failed due to configuration error: No match for output mapping and no default output mapping configured. Endpoint Response Status Code: 400
(REQUEST_ID_REMOVED) Gateway response type: DEFAULT_5XX with status code: 500
(REQUEST_ID_REMOVED) Gateway response body: {"message": "An unexpected error occurred."}
(REQUEST_ID_REMOVED) Gateway response headers: {x-amzn-ErrorType=InternalServerErrorException}
(REQUEST_ID_REMOVED) Method completed with status: 500

Ive tried many different combinations of the mapping template, with assistance from GPT. Ive tried to log out data as verbosely as possible in the hope that something will jump out, but I cant get past this.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    To make this work, you need to use the decode function on your data.

    https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html#util-template-reference

    $util.urlDecode() and use it in conjunction with your payload, $util.urlDecode($input.body).

    these commands will get you a list of params with the names in the payload, and then you can reference them with the params.name after.

    #set($decoded = $util.urlDecode($input.body))
#set($params = {})
#foreach($param in $decoded.split("&"))
    #set($keyValue = $param.split("="))
    #set($params[$keyValue[0]] = $keyValue[1])
#end

{
    "TableName": "${aws_dynamodb_table.generic_data.name}",
    "Item": {
        "longitude": {
            "N": "$params.lon"
        },
        "latitude": {
            "N": "$params.lat"
        },
        "timestamp": {
            "S": "$params.timestamp"
        }
    }
},
  "debug": {
        "timestamp": "$params.timestamp",
        "decoded_lon": "$params.lon",
        "decoded_lat": "$params.lat",
        "decoded_body": $util.escapeJavaScript($decoded),
        "request": {
            "parameters": "$util.escapeJavaScript($input.params())"
        }
    }
}

    You cannot grab the parameters out of the query string without decoding it first.

    My working terraform integration resource:

    resource "aws_api_gateway_integration" "ddb_integration" {
  depends_on           = [aws_api_gateway_method.generic_post]
  rest_api_id          = aws_api_gateway_rest_api.generic_api.id
  resource_id          = aws_api_gateway_resource.generic_resource.id
  http_method          = aws_api_gateway_method.generic_post.http_method
  passthrough_behavior = "WHEN_NO_MATCH"
  # content_handling        = "CONVERT_TO_TEXT"
  integration_http_method = "POST"
  uri                     = "arn:aws:apigateway:${local.workspace.aws_region}:dynamodb:action/PutItem"
  credentials             = aws_iam_role.api_gw_to_ddb.arn
  type                    = "AWS"

  request_templates = {
    "application/json" = <<-EOF
#set($decoded = $util.urlDecode($input.body))
#set($params = {})
#foreach($param in $decoded.split("&"))
    #set($keyValue = $param.split("="))
    #set($params[$keyValue[0]] = $keyValue[1])
#end

{
    "TableName": "${aws_dynamodb_table.generic_data.name}",
    "Item": {
        "longitude": {
            "N": "$params.lon"
        },
        "latitude": {
            "N": "$params.lat"
        },
        "timestamp": {
            "S": "$params.timestamp"
        }
    }
},
  "debug": {
        "timestamp": "$params.timestamp",
        "decoded_lon": "$params.lon",
        "decoded_lat": "$params.lat",
        "decoded_body": $util.escapeJavaScript($decoded),
        "request": {
            "parameters": "$util.escapeJavaScript($input.params())"
        }
    }
}
EOF
  }
}

  2. 0

    You are getting a 400 exception, which you have not set up a response mapping for so you’re not sure of the true cause of the error. My hunch is you are getting a ResourceNotFoundException, and I believe your way of parsing the table name in your VTL is not correct. Get the table name and hardcode it into your template and retry. That’ll let you identify the cause of the issue which you can then work on fixing.

    Also set up an exception template so you know what errors you’re getting.

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search