Hello All,
I have a temporary folder from where I used read the different xml files from SAS.
Now I am trying to do something more complicated.
I want to store only 10K files at a single time in my temporary folder and once I am done with reading the first 10K files from SAS, I want to delete them from the temporary folder and store the next 10K files inside the temporary and so on.
Since I am a beginner to SAS, I would also like to know know can I dynamically read the files one by one as all my files are of different names.
Please help
Thanks in advance!
For point 1 - how are the files getting into the folder - that is where you should be limiting things if there are too many in the folder.
For point 2 - get a directory listing - many examples on here for this please search, for example:
You can then use that dir list to either generate code, or call macros etc.
Here's the general idea, just to see if this fits what you are looking for.
1. Put 10,000 files into your temporary folder. I have to assume you can do that already.
2. SAS can issue a system command to get a list of all the files in the folder. (See the X statement, and the %SYSEXEC statement.) Route that list to a file. The format of that system command changes, depending on the operating system.
3. Read the file to get a SAS data set holding the names of all files in your folder.
4. Process each file (possibly using macro language, depends on what "processing" means).
5. Issue another X or %SYSEXEC command to delete all files in the folder.
Repeat as needed.
I have an addition to @Astounding's outline, before permanently deleting files perform some verification that you may have the data correctly read/ processed.
If you are contemplating using Proc Import 10K times expect to spend a lot of time getting variable types and lengths aligned afterwards.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.