Proc HTTP Cursor Based Pagination

[CR-Horton]

Hello,

I have found this community to be very helpful and have another issue I am working through and looking for some guidance. I am trying to pull in some data from Shopify that uses cursor based pagination. I have 1K+ products I need to pull in and looking for a way to essentially loop through the pages and not just pull in the initial amount which is 50. 

What I have gathered so far is I need to somehow reference the link headers that look something like this: "Link: <https://tXXXXXXXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=50&page_info=eyJsYXN0X2lkI...
YXN0X3ZhbHVlIjoiQmFqYSBEZXNpZ25zIExQOSBTcG9ydCBMRUQgTGlnaHQiLCJkaXJlY3Rpb24iOiJuZXh0In0>; rel="next"" (see: https://shopify.dev/api/usage/pagination-rest ) I was able to generate the link headers in the log using the "headerout=headers" statement shown in my code below. How can I then take that sort of information (specifically the "rel="next") and use it to then pull the next page. Or perhaps there is a much easier way to do this? For other APIs I have been able to loop through the pages, but that doesn't appear to be an option here.

Here is my code that works, but only pulls the first 50:

%let accesskey=XXXXXXXXXX;

%let fullurl=https://XXXXXXX.myshopify.com/admin/api/2021-10/products.json;

filename tb_prods temp;
filename headers temp;

proc http
url= "&fullurl"
method="GET" out=tb_prods headerout=headers;
headers
"X-Shopify-Access-Token"="&accesskey.";
run;

data _null_;
infile headers;
input;
put _infile_;
run;

Any help would be greatly appreciated.

[kelxxx]

Can you try?

%let fullurl=https://XXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=2000;

[CR-Horton]

Unfortunately, Shopify limits the product API calls to 250 or less. 

So the best I can do with one API call is:
%let fullurl=https://XXXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=250;

I am needing to do cursor based pagination. Right now this is printing in the log:
Link: <https://XXXXXX.myshopify.com/admin/api/2021-10/products.json limit=250&page_info=eyJsYXN0X2lkIjo2NjU0NjU1NjYwMDc4LCJsYXN0X3ZhbHVlIjoiQ2FsaSBSYWlzZWQgTG93ZXIgQ29udHJvbCBBcm0gU2tpZCBQbGF0ZSAtIFRveW90YSBUYWNvbWEgKDIwMDUtMjAyMSkiLCJkaXJlY3Rpb24iOiJuZXh0In0>; rel="next"

I am basically needing to reference the "rel="next" in my code to tell SAS to pull the next 250 products.

[CR-Horton]

I wanted to follow-up on my previous message. Can anyone explain the best way to make a paginated request in SAS. Here is the documentation I am just not sure how to translate this into my SAS code. 

https://shopify.dev/api/usage/pagination-rest

It looks like I need to somehow access the rel="next" that prints in the headers file to tell SAS to pull the next set of products.