BookmarkSubscribeRSS Feed
khandelwalanmol
Fluorite | Level 6

I have been working with an API with sends out response in form of 10 ndjson files from 10 requests. Currently these all 10 files are saved separately but the requirement is to write the response in a single ndjson file.

here is the http request and code.

data _null_;
	set jbst_rsp.output;
       call execute('%download_srcfiles('||ordinal_output||','||type||','||url||')');
run;

%macro download_srcfiles(m_filenum,m_filetype,m_fileurl);

%let date_zz = %sysfunc(today());
%letsource_file%sysfunc(cats(%substr(&m_filetype,1,3),_,%sysfunc(putn(&date_zz,yymmddn6.)),_,%scan(&m_fileurl.,-1,'-')));
filename src_resp "/test/api/&m_filetype./&source_file.";
	filename src_head "/%lowcase(&gmv_env.)/%lowcase(&gmv_client.)/source/api_responsefiles/fetch_file_head.txt";

	proc http 
		url="&m_fileurl."
		out=src_resp
		method="GET"
		headerout = src_head;
		headers "Accept"="application/json"
			"Authorization"="Bearer &access_token. ";
	run;

%mend download_srcfiles;

in jbst.output dataset i get list of 10 urls from files will be downloaded.using this list i am calling the download_srcfiles. 

 

6 REPLIES 6
Tom
Super User Tom
Super User

By NDJSON do you just mean JSONL file?  Where each LINE in the file is a separate JSON object?

Are you sure the files you are getting back from the Web Service do not include end of line characters in them?

 

Here is a method to read a text file and write its contents as one line to the end of an existing file.

data _null_;
  infile src_rep;
  file bigfile mod;
  input;
  put _infile_ @;
run;

Add that to the end of your macro definition.

Define the filefef BIGFILE before running the data step that generates the macro calls.

 

khandelwalanmol
Fluorite | Level 6
yes, ndjson contains separate json object in each line.
can't we write response of multiple request with the same file?
Tom
Super User Tom
Super User

@khandelwalanmol wrote:
yes, ndjson contains separate json object in each line.
can't we write response of multiple request with the same file?

I have no idea what response your PROC HTTP code is generating.  

Are you asking whether PROC HTTP will APPEND its output to a file instead of writing a NEW file?  If so I have not seen anything to indicate such an option.

 

khandelwalanmol
Fluorite | Level 6
Here is the type of response i receive from the request.
file:https://bcda.cms.gov/assets/data/Patient.ndjson
SASJedi
SAS Super FREQ

If your API call is generating one JSON file per hit, you can probably get what you need something like this. The data is in separate JSON files in a single folder, but the FILENAME will concatenate them for further processing in SAS. The DATA _NULL_ step could have been written to a consolidated text file instead of the log.

%let path=c:/temp;

%macro getAll(myList);
%local Country Count JSONPath;
%let Country=%scan(%superq(mylist),1);
%let Count=1;
    %do %while (%superq(country) ne );
      filename resp "&path/json/&country..json";
      proc http 
         url="https://api.worldbank.org/v2/en/country/&country/indicator/DPANUSSPF%nrstr(?mrv=7&gapfill=Y&format=json)"
         out=resp;
      run;
      filename resp;
      %let jsonpath=&jsonpath %tslit(&path/json/%superq(country).json);
      %put NOTE: %superq(jsonpath);
      %let count=%eval(&count+1);
      %let Country=%scan(%superq(mylist),&count);
   %end;
filename respALL (&jsonpath );
%mend;

%getALL(IND CHN AUS)

data _null_;
   infile respALL;
   input;
   put _infile_;
run;

Check out my Jedi SAS Tricks for SAS Users
AllanBowe
Barite | Level 11

After downloading all 10 files, you can set filerefs to each eg:

 

filename fref1 "/location/file1";
filename fref2 "/location/file2";
filename fref3 "/location/file3";
...

And then append them all in one go using this macro as follows: 

 %mp_appendfile(baseref=basefile, appendrefs=fref1 fref2 fref3 ...)

 

/Allan
SAS Challenges - SASensei
MacroCore library for app developers
SAS networking events (BeLux, Germany, UK&I)

Data Workflows, Data Contracts, Data Lineage, Drag & drop excel EUCs to SAS 9 & Viya - Data Controller
DevOps and AppDev on SAS 9 / Viya / Base SAS - SASjs

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 6 replies
  • 507 views
  • 1 like
  • 4 in conversation