Question 370342 in Azure Question posted in
The official documentation to get you started can be found here.

How Do I Insert a Page Range Variable in the Relative URL of a REST API Connection Service using an Azure Data Factory?

To start, it seems like pagination works differently when using a Copy data command vs. using a Data Flow to do the same thing, but adding a flatten transformation inbetween the ingest and sink.

When using the Copy data command, this relative url: mytable?type=external&count=400&startPage={startPage}&includeDetails=TRUE with startPage in curly brackets works with this pagination in the Copy data pipeline: absoluteUrl.{startPage} RANGE:1:20:1

However, this same combination does not seem to work in the source of a data flow. The error I receive is that it doesn’t recognize the {} in the relative url. One thing to add as well, is that the RANGE function is not in the drop down list in the pagination rules when an AbsoluteUrl is selected and not sure if this is a bug or on purpose/related.

I am thinking that there is a different way to write the {startPage} variable within the relative url, or potentially a bug with pagination within a data flow. A bug is less likely and hope that there is just a different way to write the relative url. Any help would be greatly appreciated!

I repeated the same configuration for a Copy data command but within a data flow and it caused errors. The data flow does not seem to recognize the relative url the same was a copy command.

Configuration combination:
relative url: mytable?type=external&count=400&startPage={startPage}&includeDetails=TRUE
pagination in the Copy data pipeline: absoluteUrl.{startPage} RANGE:1:20:1

Tags:

2

Answers


  1. Chosen as BEST ANSWER
    0

    Thanks, this is one way to do it and I did get a similar thing working with an Until instead of a For Each. The biggest issue I have with this solution is that it creates a single file per page, which I was trying to avoid. I was hoping to ingest and flatten in one shot, and let the system create the files based on partitions but have sinced learned that it is better to perhaps ingest the raw json and then transform afterwards in more of an ELT approach.

    Thanks again for your help. I also appreciate this, as I figured this was the case but couldn't find any confirmation:

    In Mapping data flow Range pagination is not supported of Rest API.


  2. 0

    I am thinking that there is a different way to write the {startPage} variable within the relative url, or potentially a bug with pagination within a data flow.

    In Mapping data flow Range pagination is not supported of Rest API.

    To implement range like pagination you need to use For-each activity and dataflow under it.

    • You can use range function to iterate over range using @range(1,20) function.
      https://i.imgur.com/JIdRQU7.png
    • Under This For each take dataflow with parameterized dataset as follow:
      In your source dataset create startpage parameter
      enter image description here
      In dataset Add your relative URL like below:
    @concat('mytable?type=external&count=400&startPage=',string(dataset().startPage),'&includeDetails=TRUE')

    enter image description here

    • Then in dataflow source parameters add for each iterator to source parameter.
      enter image description here
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search