topic Re: Paging through rest api results? in SAS Programming

Paging through rest api results?

Dwayne-Dibbley — Tue, 26 Mar 2019 16:20:18 GMT

Hi,

I am using the following code to pull a json file from a url :

options NOQUOTELENMAX;

filename usage "/folders/myfolders/sasuser.v94/usage.json";

%let AccessKey = reallylongstring;
	
proc http
 url="https://a.url"
 method="GET" out=usage;
 headers 
   "Authorization"="Bearer &AccessKey.";
run;

libname usage json "/folders/myfolders/sasuser.v94/usage.json";

data usage;
 set usage.data;
run;

proc print data=usage noobs;
run;

however now the results are returning more that 1000 results and i need to check for nextLink property somehow?

in .net you could use:

$usagerest = Invoke-Restmethod -url $usageurl -header $authheaders -method get

while ($null -ne $usageRest.nextLink) {
$usageRest = Invoke-Restmethod -uri $usagerest.nextlink -headers $authheaders -method get
}

is something like this possible with proc http in sas?

Thanks

Re: Paging through rest api results?

ChrisNZ — Wed, 27 Mar 2019 03:01:24 GMT

Not too sure what your .Net code does, but you can post-process the data any way you want in the data step.

Would that help?

Re: Paging through rest api results?

Dwayne-Dibbley — Wed, 27 Mar 2019 07:48:28 GMT

Hi,

the .net code basically fetches the initial json from the url ( which only returns 1000 rows ), then the last row in the json has a link to the next 1000 rows like:

......"}],"nextLink":"https://sameurl/nextpage?sessiontoken.......

it then runs another get request with the url from nextLink, and repeats until no more nextLink in the data

does that help with the explanation?

Thanks

Re: Paging through rest api results?

ChrisNZ — Fri, 29 Mar 2019 01:44:14 GMT

> it then runs another get request with the url from nextLink, and repeats until no more nextLink in the data

In this case I would probably try to read the file as a simple text file using INFILE with the URL engine and FILEVAR= .

You can then update the next file to read using the FILEVAR= option and start over.

The tricky bit may be to identify the *last* nextLink call, depending on how the JSON file is built.