How to Store,Read and then delete multiple files in SAS automatically?

[jjames1]

Hello All,

I have a temporary folder from where I used read the different xml files from SAS.

Now I am trying to do something more complicated.

I want to store only 10K files at a single time in my temporary folder and once I am done with reading the first 10K files from SAS, I want to delete them from the temporary folder and store the next 10K files inside the temporary and so on.

Since I am a beginner to SAS, I would also like to know know can I dynamically read the files one by one as all my files are of different names.

Please help

Thanks in advance!

[RW9]

For point 1 - how are the files getting into the folder - that is where you should be limiting things if there are too many in the folder.

For point 2 - get a directory listing - many examples on here for this please search, for example:

https://communities.sas.com/t5/General-SAS-Programming/read-a-list-of-file-names-with-blanks-by-file...

You can then use that dir list to either generate code, or call macros etc.

[Astounding]

Here's the general idea, just to see if this fits what you are looking for.

1. Put 10,000 files into your temporary folder.  I have to assume you can do that already.

2. SAS can issue a system command to get a list of all the files in the folder.  (See the X statement, and the %SYSEXEC statement.)  Route that list to a file. The format of that system command changes, depending on the operating system.

3. Read the file to get a SAS data set holding the names of all files in your folder.

4. Process each file (possibly using macro language, depends on what "processing" means).

5. Issue another X or %SYSEXEC command to delete all files in the folder.

Repeat as needed.

[ballardw]

I have an addition to @Astounding's outline, before permanently deleting files perform some verification that you may have the data correctly read/ processed.

If you are contemplating using Proc Import 10K times expect to spend a lot of time getting variable types and lengths aligned afterwards.