Hello, I am currently using a PROC HTTP request to GET a JSON payload from a server. The response is: 1. Too long to be pulled in one request and must be pulled and concatenated page by page. 2. Contains two data sets (data, data_recorddata) that must be merged together on their corresponding observation. 3. Dynamic in the respect that I cannot anticipate what variables are coming across the API to have a JSON map set up. My issue is that at some point when I merge the data sets or concatenate the pages together I end up truncating some of the data contained in the observations. How do I prevent this? Can I set some parameter so that SAS does not look at length or automatically sets the longest possible length? Unfortunately I cannot anticipate exactly what will come through the API and need to be flexible but also cannot truncate. I have provided my current code below which will pull from the API and perform merging and concatenating of the JSON payload. Any help is greatly appreciated! /*Extract the total number of pages from JSON response and assessbles url for each page */
data urls;
set api.pagination;
call symputx("nobs", sum(totalPages, -1));
do I = 0 to totalPages;
path = catt(&base_url., &study_name., '/', &endpoint., '?size=500&pages=',I, '&filter=formKey==', &crf.);
output;
end;
run;
/*Create macro variables for url paths*/
data _NULL_;
set urls nobs=obs;
call symput("iter", url);
run;
/*Loop through urls based on number of urls and collect API data for each page*/
%do I = 0 %to &nobs.;
%let ping = &url.%str(&)page=&I.;
%put &ping;
proc http
url= "&ping."
method="GET"
out=respAll
proxyhost="http://webproxy-azuse.vsp.sas.com"
proxyport=3128
headerout=hdrs;
headers
"x-imn-security-key"= &security_key.
"x-api-key"= &api_key.;
/*Assign http response to JSON library*/
/*Makes dataset in work library for parsed data set*/
data data_page&I.;
libname page&I. JSON fileref=respAll;
set page&I..data;
run;
proc datasets lib=page&I.; quit;
/*Makes dataset in work library for parsed column recorddata set*/
data recorddata_page&I.;
set page&I..data_recorddata;
run;
/*Merge the two datasets so patient info and CRF data are on same page. Merge by index by page*/
data mergeData&I.;
merge work.data_page&I. work.recorddata_page&I.;
drop ordinal_recordData;
by ordinal_data;
run;
proc transpose data=mergeData&I. out=middle&I.;
var _all_;
run;
proc transpose data=middle&I. out=mergeDataFin&I. (drop=_name_ _label_);
var _all_;
run;
data mergeDataFin&I.;
set mergeDataFin&I.;
if _N_ = 1 then delete;
run;
libname page&I. clear;
%end;
run;
data out.&output.;
set mergeDataFin: truncover;
drop ordinal_root ordinal_data; Seen below, there is truncation at the formId, recordId, and deleted columns when looking at the finalized dataset.
... View more
