BookmarkSubscribeRSS Feed
CR-Horton
Fluorite | Level 6

Hello,

I have found this community to be very helpful and have another issue I am working through and looking for some guidance. I am trying to pull in some data from Shopify that uses cursor based pagination. I have 1K+ products I need to pull in and looking for a way to essentially loop through the pages and not just pull in the initial amount which is 50. 

What I have gathered so far is I need to somehow reference the link headers that look something like this: "Link: <https://tXXXXXXXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=50&page_info=eyJsYXN0X2lkI...
YXN0X3ZhbHVlIjoiQmFqYSBEZXNpZ25zIExQOSBTcG9ydCBMRUQgTGlnaHQiLCJkaXJlY3Rpb24iOiJuZXh0In0>; rel="next"" (see: https://shopify.dev/api/usage/pagination-rest ) I was able to generate the link headers in the log using the "headerout=headers" statement shown in my code below. How can I then take that sort of information (specifically the "rel="next") and use it to then pull the next page. Or perhaps there is a much easier way to do this? For other APIs I have been able to loop through the pages, but that doesn't appear to be an option here.

Here is my code that works, but only pulls the first 50:

 

%let accesskey=XXXXXXXXXX;

%let fullurl=https://XXXXXXX.myshopify.com/admin/api/2021-10/products.json;

 

filename tb_prods temp;
filename headers temp;

 

proc http
url= "&fullurl"
method="GET" out=tb_prods headerout=headers;
headers
"X-Shopify-Access-Token"="&accesskey.";
run;

 

data _null_;
infile headers;
input;
put _infile_;
run;

 

Any help would be greatly appreciated.

3 REPLIES 3
kelxxx
Quartz | Level 8

Can you try?

%let fullurl=https://XXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=2000;
CR-Horton
Fluorite | Level 6

Unfortunately, Shopify limits the product API calls to 250 or less. 

 

So the best I can do with one API call is:
%let fullurl=https://XXXXXXXX.myshopify.com/admin/api/2021-10/products.json?limit=250;

 

I am needing to do cursor based pagination. Right now this is printing in the log:
Link: <https://XXXXXX.myshopify.com/admin/api/2021-10/products.json limit=250&page_info=eyJsYXN0X2lkIjo2NjU0NjU1NjYwMDc4LCJsYXN0X3ZhbHVlIjoiQ2FsaSBSYWlzZWQgTG93ZXIgQ29udHJvbCBBcm0gU2tpZCBQbGF0ZSAtIFRveW90YSBUYWNvbWEgKDIwMDUtMjAyMSkiLCJkaXJlY3Rpb24iOiJuZXh0In0>; rel="next"

I am basically needing to reference the "rel="next" in my code to tell SAS to pull the next 250 products.

CR-Horton
Fluorite | Level 6

I wanted to follow-up on my previous message. Can anyone explain the best way to make a paginated request in SAS. Here is the documentation I am just not sure how to translate this into my SAS code. 

https://shopify.dev/api/usage/pagination-rest

It looks like I need to somehow access the rel="next" that prints in the headers file to tell SAS to pull the next set of products. 

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 3 replies
  • 750 views
  • 0 likes
  • 2 in conversation