07-25-2017 09:41 AM
I have a temporary folder from where I used read the different xml files from SAS.
Now I am trying to do something more complicated.
I want to store only 10K files at a single time in my temporary folder and once I am done with reading the first 10K files from SAS, I want to delete them from the temporary folder and store the next 10K files inside the temporary and so on.
Since I am a beginner to SAS, I would also like to know know can I dynamically read the files one by one as all my files are of different names.
Thanks in advance!
07-25-2017 10:33 AM
For point 1 - how are the files getting into the folder - that is where you should be limiting things if there are too many in the folder.
For point 2 - get a directory listing - many examples on here for this please search, for example:
You can then use that dir list to either generate code, or call macros etc.
07-25-2017 10:36 AM
Here's the general idea, just to see if this fits what you are looking for.
1. Put 10,000 files into your temporary folder. I have to assume you can do that already.
2. SAS can issue a system command to get a list of all the files in the folder. (See the X statement, and the %SYSEXEC statement.) Route that list to a file. The format of that system command changes, depending on the operating system.
3. Read the file to get a SAS data set holding the names of all files in your folder.
4. Process each file (possibly using macro language, depends on what "processing" means).
5. Issue another X or %SYSEXEC command to delete all files in the folder.
Repeat as needed.
07-25-2017 10:43 AM
I have an addition to @Astounding's outline, before permanently deleting files perform some verification that you may have the data correctly read/ processed.
If you are contemplating using Proc Import 10K times expect to spend a lot of time getting variable types and lengths aligned afterwards.