CODE SAS

[MakiMau]

I have a routine that searches folder by folder and performs some procedures, but do I need a table to be completely loaded so that the routine can continue so how can I make the routine wait until this temporary table is fully loaded?

[Reeza]

Where is this temporary table coming from? How do you know when it's 'fully loaded'? The answer depends on your process and set up which we do not know.

[MakiMau]

This temporary table is created by several csv files that are stacked it is impossible to know when it will be complete by file size and processing will be variable so it needed some code that was looped for a while until this table is fully created, we are talking of csv tables that are stacked with 8,000 rows and 130 columns times 12 files per step.

[MakiMau]

%let DiretorioTodos=&prompt_1;
filename _dir_ "%bquote(&DiretorioTodos.)";

data test;run;
data lil;run;
%Macro Rodar();
data filenames(keep=memname filenames);

handle=dopen( '_dir_' );
if handle > 0 then do;
count=dnum(handle);
do i=1 to count;
memname=dread(handle,i);
output filenames;
call symput("NumArq",compress(i));
end;
end;
rc=dclose(handle);

run;
%put ===========passo 1 ================== &NumArq ==================;
%DO i=1 %to &NumArq;
data _NULL_; set filenames;
if index(memname,".")=0;
if _N_=&i;
call symput("NomePasta",compress(memname));
run;

%let DiretorioIsol=&prompt_1\&NomePasta.;
filename2 _dir2_ "%bquote(&DiretorioIsol.)";

data filenames2(keep=memname filenames2) ;
handle=dopen( '_dir2_' );

if handle > 0 then do;
count=dnum(handle);
do i=1 to count;
memname=dread(handle,i);
output filenames2;
call symput("NumArq2",i);
end;
end;
rc=dclose(handle);
run;
%put ================================= &NumArq ====================;
%put ================================= &NumArq2 ====================;

%do j=1 %to 1; /*&NumArq2;*/
data teste;set filenames2;
if index(memname,".")>0;
if _N_=&j;
call symput("NomeArquivo",memname);
run;

data testemp;
.
.
.
/*here dont have time for export date*/
PROC EXPORT DATA=

outfile="&prompt_1\&NomePasta\IBGE_&NomePasta..csv"
DBMS=csv
REPLACE;
RUN;

[Tom]

Please explain what that code is supposed to be doing. Is it your attempt to solve the problem. Please explain how you think it could solve the problem.

[Tom]

It looks like you are trying to define a macro to use DOPEN() and DREAD() to get the list of files in a directory tree.

Note there is no need to use any macro code for that part of the problem.  See this simple data step for doing that:

https://communities.sas.com/t5/SAS-Programming/listing-all-files-within-a-directory-and-subdirectori...

[Tom]

---

@MakiMau wrote:

This temporary table is created by several csv files that are stacked it is impossible to know when it will be complete by file size and processing will be variable so it needed some code that was looped for a while until this table is fully created, we are talking of csv tables that are stacked with 8,000 rows and 130 columns times 12 files per step.

---

You need to explain in detail what your process is. 

It sounds like you want to READ a number of CSV files but are not sure when they have been fully created.

What is creating the CSV files? 

Is it part of the current SAS program you are running?  If so then you should already know when it is done. Please show the code that is being used to create the CSV files.  

Is it some other SAS program?  Can you call that program from your current program so that you will know when it ends?

Is it some other program/process?  If so how do you know when it is done?  Can you have it update a file or a database when it is done?

[MakiMau]

Given a root folder, the program enters each folder reads the CSV files and stacks creating a temporary table with all files in the folder doing a certain process on the files, and each child folder saves a result, but sometimes that table was not. fully loaded due to file size, network and server processing go to the next step without having yet created this temporary table with the stacked files and the result of the large files saving an empty result. I needed a routine that kept processing without pausing the program so that this table would be fully loaded.

[Tom]

---

@MakiMau wrote:

Given a root folder, the program enters each folder reads the CSV files and stacks creating a temporary table with all files in the folder doing a certain process on the files, and each child folder saves a result, but sometimes that table was not. fully loaded due to file size, network and server processing go to the next step without having yet created this temporary table with the stacked files and the result of the large files saving an empty result. I needed a routine that kept processing without pausing the program so that this table would be fully loaded.

---

Still not clear what is the issue here.  So you are performing some large task and sometimes it means your process runs out of resources? Does it fill the disk you are writing to?  Does it fill up the disk your WORK libref is using?  How do you propose to handle a smooth transition when one of these interruptions occur?  Do you want to restart the whole process?  Do you want to have built some type of table/dataset that contains what files were already successfully processed so you can skip those when you re-run the process?  Can you be sure that dataset will not also experience the same issues that are crashing the main process?

Can you just divide the process into steps?  First get the list of files (folders?) to process.  Then process them one by one and record success or failure.  If you are having intermittent access issues that can fail the processing of a single file then you need some way to trap that and allow the rest of the files to be processed.  The best way is to process that file in a separate SAS session.  So if you have SAS/Connect you could spawn a new process for each file.  That way if it fails your main session is not impacted.  If you can't then you have to deal with the failure in the current session.  So if the failure causes SAS to go into syntaxcheck mode and set OBS=0 then you could try just resetting those options after each file is processed and see if that lets you finish processing the other files.

[MakiMau]

Good afternoon I can't do this because I lose the folder location reference and sub folder saved by a variable

[Tom]

How could you lose the value? Is something deleting the dataset you stored it into?

[Reeza]

Forget file size and SAS, if you're doing this manually, how do you know if it's complete? If there isn't a way to define when the file is completely loaded how can you tell the computer it's ready?

Is it just if the file size stops changing? But that could be an incomplete load still? So how do YOU know when you can run the program. Once you figure that out, we can find a way to turn that into a computer process but until then it's not really possible. 

---

@MakiMau wrote:

This temporary table is created by several csv files that are stacked it is impossible to know when it will be complete by file size and processing will be variable so it needed some code that was looped for a while until this table is fully created, we are talking of csv tables that are stacked with 8,000 rows and 130 columns times 12 files per step.

---

[MakiMau]

I thought about determining a time until it moves on to the next step, but I didn't find any function to do that.

[Reeza]

SLEEP() will tell SAS to sleep for a specific amount of time. But how do you know if you need to wait or if it's ready? It sounds like you don't have a method of determining when the file will be complete. So, can you change the process that generates that file? Have it write a new record at the end, if successful, that you can then check to see if it's finished. Then you can loop your SAS process to wait a specific amount of time or until the file is ready - or a combination of both.