Question 18193 in Amazon Web Sevices Question posted in
The official Amazon Web Services documentation can be found here.

AWS Step Function – Associating inputs and outputs across multiple Map states – Amazon Web Sevices

Say I have a simple input:

{
  "input_array": ["blue", "green", "orange"]
}

Which I am passing into a state machine with a couple of Map states:

{
  "StartAt": "Step One",
  "States": {
    "Step One": {
      "Type": "Map",
      "ItemsPath": "$.input_array",
      "ResultPath": "$.step_one_output",
      "Next": "Step Two",
      "Iterator": {
        "StartAt": "Find fruit matching colour",
        "States": {
          "FindFruitMatchingColour": {
            "Type": "Task",
            "Resource": "[some resource]",
            "End": true
          }
        }
      }
    },
    "Step Two": {
      "Type": "Map",
      "ItemsPath": "$.step_one_output",
      "ResultPath": "$.step_two_output",
      "Next": "End",
      "Iterator": {
        "StartAt": "Do Thing With Colour And Its Fruit",
        "States": {
          "DoThingWithColourAndFruitCombination": {
            "Type": "Task",
            "Resource": "[some resource]",
            "End": true
          }
        }
      }
    },
    "End": {
      "Type": "Succeed"
    }
  }
}

What I’d like to do is associate the input of step one with its corresponding output and pass them together into step two. For example:

[
  {"colour": "blue", "fruit": "blueberry"}, 
  {"colour": "green", "fruit": "apple"}, 
  {"colour": "orange", "fruit": "orange"}
]

It feels like a fairly common use case.

I’ve tried a few ways to solve this, such as using

{
  "ResultSelector": {
    "fruit.$": "$",
    "colour.$": "$$.Map.Item"
  }
}

But I cannot seem to access the individual iterations in ResultSelector, it appears to be in the context of the overall output array. I’ve also explored having the two steps part of the same Iterator, but again struggle to associate the individual inputs to the outputs of step one

Alternatively, I’ve considered having a separate step to zip together the input and the output of step one together before passing to step two. But I’m concerned about not being able to rely on the order of the output of step one and mixing up colours and fruits.

Many thanks for any guidance provided, even if to say it’s not supported. I can always have the resource return both the fruit and the colour, but this is not its current behaviour and I’d like to delegate this to the state machine if possible.

P.S – this is my first SO issue. Don’t hesitate to ask for more detail.

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Based on Justin's help, I think I have come up with the following solution:

    {
  "StartAt": "Step One",
  "States": {
    "Step One": {
      "Type": "Map",
      "ItemsPath": "$.input_array",
      "ResultPath": "$.step_one_output",
      "Next": "Step Two",
      "Iterator": {
        "StartAt": "Find fruit matching colour",
        "States": {
          "FindFruitMatchingColour": {
            "Type": "Task",
            "Resource": "[some resource]",
            "End": true
          }
        }
      }
    },
    "Step Two": {
      "Type": "Map",
      "ItemsPath": "$.step_one_output",
      "ResultPath": "$.step_two_output",
      "Parameters": {
        "colour.$": "States.ArrayGetItem($.input_array, $$.Map.Item.Index)",
        "fruit.$": "$$.Map.Item.Value",
      },
      "Next": "End",
      "Iterator": {
        "StartAt": "Do Thing With Colour And Its Fruit",
        "States": {
          "DoThingWithColourAndFruitCombination": {
            "Type": "Task",
            "Resource": "[some resource]",
            "End": true
          }
        }
      }
    },
    "End": {
      "Type": "Succeed"
    }
  }
}

    Specifically using the output of step on as the input to a Map task step two, and also selecting the appropriate colour from the original input_array based on the index of the current iteration using the Parameters key in 'Step Two'


  2. 0

    I think the best option is to add a Pass state after your task to shape the output you want from each iteration in the Map state. The problem is that you can’t use the ResultSelector to compose an object from both the input to the state and the output of the Task. You can use ResultPath to keep both, but it won’t let you do exactly what you’re looking for.

    Below is an example of how you might do this.

    This State Machine ….

    {
  "StartAt": "Generate Input",
  "States": {
    "Generate Input": {
      "Type": "Pass",
      "Next": "Step One",
      "Result": {
        "input_array": [
          "blue",
          "green",
          "orange"
        ]
      }
    },
    "Step One": {
      "Type": "Map",
      "ItemsPath": "$.input_array",
      "Next": "Step Two",
      "Iterator": {
        "StartAt": "Get Fruit for Colour",
        "States": {
          "Get Fruit for Colour": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "Payload": {
                "colour.$": "$.colour"
              },
              "FunctionName": "arn:aws:lambda:ap-southeast-1:111111111111:function:get-fruit-by-colour:$LATEST"
            },
            "ResultPath": "$.fruit-lookup",
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "Lambda.TooManyRequestsException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 6,
                "BackoffRate": 2
              }
            ],
            "Next": "Shape Result"
          },
          "Shape Result": {
            "Type": "Pass",
            "End": true,
            "Parameters": {
              "colour.$": "$.colour",
              "fruit.$": "$.fruit-lookup.Payload"
            }
          }
        }
      },
      "ItemSelector": {
        "colour.$": "$$.Map.Item.Value"
      }
    },
    "Step Two": {
      "Type": "Map",
      "Next": "End",
      "Iterator": {
        "StartAt": "Do Thing With Colour And Its Fruit",
        "States": {
          "Do Thing With Colour And Its Fruit": {
            "Type": "Pass",
            "End": true
          }
        }
      }
    },
    "End": {
      "Type": "Succeed"
    }
  }
}

    Calls this Lambda Function ….

    import json

def lambda_handler(event, context):
    
    colour_to_fruit_map = {
        "green": "apple",
        "orange": "orange",
        "blue": "blueberry"
    }

    return colour_to_fruit_map[event['colour']]

    And the result will be:

    [
  {
    "colour": "blue",
    "fruit": "blueberry"
  },
  {
    "colour": "green",
    "fruit": "apple"
  },
  {
    "colour": "orange",
    "fruit": "orange"
  }
]

    As for your concerns about ordering, the Map state maintains order. That is, each item you pass into the Map state will initiate an independent branch of execution with that item as input and using the Iterator sub-workflow definition. Then the output of each iteration is returned in the same order as the inputs.

    I hope this helps!

    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search