I am trying to loop over all the files in an Azure Data Lake and want to read specific .json files so that I may do something with the data in those files. I have access to the data lake and proved I can see the files stored there from within SAS. I only want to read files with a specific name (in this example, “process.json”). I can access the ADLS location and see the files. I can filter out the files I don’t want. I cannot successfully use the “filename” statement to read the files. I am unsure how to get the actual name of the file into the filename statement within the loop. How do I accomplish this? I can't seem to get the filename statement to incorporate the actual value of "name". Code and log output is below. Note that I have to read json files. I cannot convert to any other file type prior to reading into SAS. options azuretenantid = "###-###-###-";
%let appId="###-###-###-";
%let acctName="###";
%let filesys="###";
filename d adls "/"
applicationId=&appId
accountname=&acctName
filesystem=&filesys;
data _null_;
folder_id = dopen("d");
put folder_id=;
num_files = dnum(folder_id);
put num_files=;
do i = 1 to num_files;
name=dread(folder_id,i);
if prxmatch('/(process)\.(json)/',name) then do;
put name=;
filename myjson adls "&name"
applicationId=&appId
accountname=&acctName
filesystem=&filesys;
libname temp json fileref=myjson;
*do something with data in temp;
end;
end;
closerc = dclose(folder_id);
put closerc=;
run; Log output: data _null_; 80 81 folder_id = dopen("d"); 82 put folder_id=; 83 84 num_files = dnum(folder_id); 85 put num_files=; 86 87 do i = 1 to num_files; 88 name=dread(folder_id,i); 89 90 if prxmatch('/(process)\.(json)/',name) then do; 91 put name=; 92 93 filename myjson adls "&name" WARNING: Apparent symbolic reference NAME not resolved. 94 applicationId=&appId 95 accountname=&acctName 96 filesystem=&filesys; 97 98 libname temp json fileref=myjson; NOTE: JSON data is only read once. To read the JSON again, reassign the JSON LIBNAME. ERROR: The AZURE file &name was not found. ERROR: The endpoint within the Credential Service, azureDeviceCode, could not be found ERROR: The endpoint within the Credential Service, azureDeviceCode, could not be found ERROR: The specified path does not exist. ERROR: Error in the LIBNAME statement. 99 *do something with data in temp; 100 101 end; 102 end; 103 104 closerc = dclose(folder_id); 105 put closerc=; 106 run; folder_id=1 num_files=6 name=process.json closerc=0 NOTE: DATA statement used (Total process time): real time 1.78 seconds cpu time 0.12 seconds 107 108 %studio_hide_wrapper; 118 119
... View more