- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
I have been working with an API with sends out response in form of 10 ndjson files from 10 requests. Currently these all 10 files are saved separately but the requirement is to write the response in a single ndjson file.
here is the http request and code.
data _null_;
set jbst_rsp.output;
call execute('%download_srcfiles('||ordinal_output||','||type||','||url||')');
run;
%macro download_srcfiles(m_filenum,m_filetype,m_fileurl);
%let date_zz = %sysfunc(today());
%letsource_file%sysfunc(cats(%substr(&m_filetype,1,3),_,%sysfunc(putn(&date_zz,yymmddn6.)),_,%scan(&m_fileurl.,-1,'-')));
filename src_resp "/test/api/&m_filetype./&source_file.";
filename src_head "/%lowcase(&gmv_env.)/%lowcase(&gmv_client.)/source/api_responsefiles/fetch_file_head.txt";
proc http
url="&m_fileurl."
out=src_resp
method="GET"
headerout = src_head;
headers "Accept"="application/json"
"Authorization"="Bearer &access_token. ";
run;
%mend download_srcfiles;
in jbst.output dataset i get list of 10 urls from files will be downloaded.using this list i am calling the download_srcfiles.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
By NDJSON do you just mean JSONL file? Where each LINE in the file is a separate JSON object?
Are you sure the files you are getting back from the Web Service do not include end of line characters in them?
Here is a method to read a text file and write its contents as one line to the end of an existing file.
data _null_;
infile src_rep;
file bigfile mod;
input;
put _infile_ @;
run;
Add that to the end of your macro definition.
Define the filefef BIGFILE before running the data step that generates the macro calls.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
can't we write response of multiple request with the same file?
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
@khandelwalanmol wrote:
yes, ndjson contains separate json object in each line.
can't we write response of multiple request with the same file?
I have no idea what response your PROC HTTP code is generating.
Are you asking whether PROC HTTP will APPEND its output to a file instead of writing a NEW file? If so I have not seen anything to indicate such an option.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
file:https://bcda.cms.gov/assets/data/Patient.ndjson
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
If your API call is generating one JSON file per hit, you can probably get what you need something like this. The data is in separate JSON files in a single folder, but the FILENAME will concatenate them for further processing in SAS. The DATA _NULL_ step could have been written to a consolidated text file instead of the log.
%let path=c:/temp;
%macro getAll(myList);
%local Country Count JSONPath;
%let Country=%scan(%superq(mylist),1);
%let Count=1;
%do %while (%superq(country) ne );
filename resp "&path/json/&country..json";
proc http
url="https://api.worldbank.org/v2/en/country/&country/indicator/DPANUSSPF%nrstr(?mrv=7&gapfill=Y&format=json)"
out=resp;
run;
filename resp;
%let jsonpath=&jsonpath %tslit(&path/json/%superq(country).json);
%put NOTE: %superq(jsonpath);
%let count=%eval(&count+1);
%let Country=%scan(%superq(mylist),&count);
%end;
filename respALL (&jsonpath );
%mend;
%getALL(IND CHN AUS)
data _null_;
infile respALL;
input;
put _infile_;
run;
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
After downloading all 10 files, you can set filerefs to each eg:
filename fref1 "/location/file1";
filename fref2 "/location/file2";
filename fref3 "/location/file3";
...
And then append them all in one go using this macro as follows:
%mp_appendfile(baseref=basefile, appendrefs=fref1 fref2 fref3 ...)