topic Re: Proc HTTP Cursor Based Pagination in SAS Procedures

Proc HTTP Cursor Based Pagination

CR-Horton — Wed, 22 Dec 2021 21:12:05 GMT

Hello,

I have found this community to be very helpful and have another issue I am working through and looking for some guidance. I am trying to pull in some data from Shopify that uses cursor based pagination. I have 1K+ products I need to pull in and looking for a way to essentially loop through the pages and not just pull in the initial amount which is 50.

What I have gathered so far is I need to somehow reference the link headers that look something like this: "Link: <https://tXXXXXXXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0Njk4MTgzNDAxNTE4LCJs
YXN0X3ZhbHVlIjoiQmFqYSBEZXNpZ25zIExQOSBTcG9ydCBMRUQgTGlnaHQiLCJkaXJlY3Rpb24iOiJuZXh0In0>; rel="next"" (see: https://shopify.dev/api/usage/pagination-rest ) I was able to generate the link headers in the log using the "headerout=headers" statement shown in my code below. How can I then take that sort of information (specifically the "rel="next") and use it to then pull the next page. Or perhaps there is a much easier way to do this? For other APIs I have been able to loop through the pages, but that doesn't appear to be an option here.

Here is my code that works, but only pulls the first 50:

%let accesskey=XXXXXXXXXX;

%let fullurl=https://XXXXXXX.myshopify.com/admin/api/2021-10/products.json;

filename tb_prods temp;
filename headers temp;

proc http
url= "&fullurl"
method="GET" out=tb_prods headerout=headers;
headers
"X-Shopify-Access-Token"="&accesskey.";
run;

data _null_;
infile headers;
input;
put _infile_;
run;

Any help would be greatly appreciated.

Re: Proc HTTP Cursor Based Pagination

kelxxx — Fri, 24 Dec 2021 13:13:12 GMT

Can you try?

%let fullurl=https://XXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=2000;

Re: Proc HTTP Cursor Based Pagination

CR-Horton — Fri, 31 Dec 2021 16:35:21 GMT

Unfortunately, Shopify limits the product API calls to 250 or less.

So the best I can do with one API call is:
%let fullurl=https://XXXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=250;

I am needing to do cursor based pagination. Right now this is printing in the log:
Link: <https://XXXXXX.myshopify.com/admin/api/2021-10/products.json limit=250&page_info=eyJsYXN0X2lkIjo2NjU0NjU1NjYwMDc4LCJsYXN0X3ZhbHVlIjoiQ2FsaSBSYWlzZWQgTG93ZXIgQ29udHJvbCBBcm0gU2tpZCBQbGF0ZSAtIFRveW90YSBUYWNvbWEgKDIwMDUtMjAyMSkiLCJkaXJlY3Rpb24iOiJuZXh0In0>; rel="next"

I am basically needing to reference the "rel="next" in my code to tell SAS to pull the next 250 products.

Re: Proc HTTP Cursor Based Pagination

CR-Horton — Tue, 01 Feb 2022 22:27:01 GMT

I wanted to follow-up on my previous message. Can anyone explain the best way to make a paginated request in SAS. Here is the documentation I am just not sure how to translate this into my SAS code.

https://shopify.dev/api/usage/pagination-rest

It looks like I need to somehow access the rel="next" that prints in the headers file to tell SAS to pull the next set of products.