Read some zipped files from some folders

[eduard1231]

Hi everyone,

I guess I have 3 issues. I need to read files, that have filepath in following format:

x1/x2/x3/x4/file 

So, I want to iterate over SOME x1, ALL x2, SOME x3, ALL x4. Finally when I am in x4 folder, I have 2 types of file's name in following format:

file_name_date

file_name_date.new 

So, from those I want to read only file_name_date files, and I want to skip all file_name_date.new.

file_name part of file is constant the only thing changes in file_name_date is the date part.

Then, this file is zipped. Can sas read zipped file? 

Finally, when I read this file and did some manipulation with it, I want to save it and then start to work with new file. When I iterate over all files in folder, i will go back from x4 level to x3 level, change folder, go to new x4 level and repeat process.

Sorry, I guess it is confusing. You are more than welcome to ask extra questions, I will try to explain as hard as I can

Thanks, 

Ed

[Reeza]

Are the files zipped text files or sas7bdat files? If this is text files everything you want is possible. If it's sas formatted files then I don't think it's as easy.

There's a FILENAME ZIP option that supports reading from zipped files. If you google the term, Chris Hemidinger has written some good blog posts on reading from zipped files, have you explored those posts already?

[eduard1231]

Thanks for this part, yes, it is text files. I see how to read them now, but I am not sure how to iterate over folders I need iterate

[Tom · Accepted solution]

---

@eduard1231 wrote:

Thanks for this part, yes, it is text files. I see how to read them now, but I am not sure how to iterate over folders I need iterate

---

I usually find it easiest to generate the full list of files into a dataset.  Usually by reading output of appropriate operating system command (dir, ls , find etc.).   There are many examples on this site for how to do that.  For example on Unix you might want to use the FIND command:

data filelist;
  infile "find /x1/" pipe truncover ;
  input filename $256. ;
run;

Then I can apply selection/exclusion logic to the file names either in that step or another step.

where scan(filename,3,'/') in ('log','saslog') ;

Once you have the list of file just use it to call the code to process one file.  So if you had a macro that processed one file. Call it %ONEFILE() you can just use a data step to generate one call for each observation in the dataset with the names of the files to process.  

data _null_;
  set filelist ;
  call execute(cats('%nrstr(%onefile)(',filename,')'));
run;