Azure Data Factory has native support for copying data from REST APIs. There are some pagination rules supported (listed here), but what to do when the API you are using does not support this?
In my example I do not have any information in the header or body of my requests, but rather use the header to set pages. The call is; https://www.strava.com/api/v3/athlete/activities?page=&per_page=. The easiest way to solve this is with a dataset and pipeline variables.
@concat(‘athlete/activities?per_page=’,dataset().perPage,’&page=’, dataset().page)Relative URL for dataset
When setting this for the dataset, you can have a pipeline parameter or variable with the per page value and simply add this to the copy activity. To set the page number you need to do a bit more. Them simple way to do this would naturally be like this;
So my workaround was to create a new temporary variable that was set to the Activites_page value, and then set the Activites_page based on this. Then place this in an until loop.
But how to stop the until loop when there is no more data? Well, you know how much data there should be, so simply use this as the until expression:
@less(activity(‘Copy Activites JSON’).output.rowsCopied,int(variables(‘Activites_perPage’)))Keep copying until the number of rows in the last copy is less then the number of activites per page