How to Store,Read and then delete multiple files in SAS automatically?

Reply
Contributor
Posts: 34

How to Store,Read and then delete multiple files in SAS automatically?

Hello All,

 

I have a temporary folder from where I used read the different xml files from SAS.

 

Now I am trying to do something more complicated.

 

I want to store only 10K files at a single time in my temporary folder and once I am done with reading the first 10K files from SAS, I want to delete them from the temporary folder and store the next 10K files inside the temporary and so on.

 

Since I am a beginner to SAS, I would also like to know know can I dynamically read the files one by one as all my files are of different names.

 

Please help

 

Thanks in advance!

Super User
Super User
Posts: 7,433

Re: How to Store,Read and then delete multiple files in SAS automatically?

For point 1 - how are the files getting into the folder - that is where you should be limiting things if there are too many in the folder.

 

For point 2 - get a directory listing - many examples on here for this please search, for example:

https://communities.sas.com/t5/General-SAS-Programming/read-a-list-of-file-names-with-blanks-by-file...

 

You can then use that dir list to either generate code, or call macros etc.

Super User
Posts: 5,099

Re: How to Store,Read and then delete multiple files in SAS automatically?

Here's the general idea, just to see if this fits what you are looking for.

 

1. Put 10,000 files into your temporary folder.  I have to assume you can do that already.

2. SAS can issue a system command to get a list of all the files in the folder.  (See the X statement, and the %SYSEXEC statement.)  Route that list to a file. The format of that system command changes, depending on the operating system.

3. Read the file to get a SAS data set holding the names of all files in your folder.

4. Process each file (possibly using macro language, depends on what "processing" means).

5. Issue another X or %SYSEXEC command to delete all files in the folder.

 

Repeat as needed.

Super User
Posts: 10,552

Re: How to Store,Read and then delete multiple files in SAS automatically?

I have an addition to @Astounding's outline, before permanently deleting files perform some verification that you may have the data correctly read/ processed.

 

If you are contemplating using Proc Import 10K times expect to spend a lot of time getting variable types and lengths aligned afterwards.

Ask a Question
Discussion stats
  • 3 replies
  • 81 views
  • 0 likes
  • 4 in conversation