topic Re: file not found in SAS Programming

file not found

p12937 — Fri, 27 Mar 2020 10:41:53 GMT

I have a sas job that reads a csv and writes out to another csv. Then a second sas job reads this new csv. We have these jobs scheduled to run every 3 hours in windows task scheduler (first one at 10 past the hour and the second one at 15 past). Sometimes the second job does not find the new file even though I can see it in the folder. It seems like windows is not refreshed or something like that. Has anyone experienced this? Thanks.

Re: file not found

PaigeMiller — Fri, 27 Mar 2020 10:48:29 GMT

Let's start with basic debugging:

Have you confirmed that the first job has finished writing the CSV before the second job executes? How do you know?

Have you confirmed that the file name is created properly to be read by the second job?

If that isn't it, show us the exact error in the SAS log of the second job.

Can you change the jobs from separated jobs, to one job where the two SAS programs are appended to each other, so the second part can't possibly start running before the first part end. Or better yet, if you can combine the programs, don't write to CSV, save the results of job 1 in a SAS data set and then use that in job 2.

Re: file not found

p12937 — Fri, 27 Mar 2020 11:10:19 GMT

Confirmation of completion: dos window closes and sas log shows that job is complete.

File name is correct. I run the second job again and it finds and processes the file.

There is no error message, the second job executes a windows dir command and looks for that file and 0 records are returned. When I run the job again it finds the file.

Re: file not found

Kurt_Bremser — Fri, 27 Mar 2020 11:11:34 GMT

What do you mean by "I can see it in the folder"? How do you verify that the file is there? Do you work with shared resources on a network?

Anyway, inspect the timestamp of the file; you might find it was created after the second job started. If you use a scheduler, use conditions to prevent a follow-up job from starting too early.

Re: file not found

p12937 — Fri, 27 Mar 2020 11:13:17 GMT

we are creating a second file because the original file has carriage returns and line feed that need to be removed.

Re: file not found

PaigeMiller — Fri, 27 Mar 2020 11:14:40 GMT

@p12937 wrote:

Confirmation of completion: dos window closes and sas log shows that job is complete.

File name is correct. I run the second job again and it finds and processes the file.

There is no error message, the second job executes a windows dir command and looks for that file and 0 records are returned. When I run the job again it finds the file.

Is there some reason why this has to be two jobs? Could these two programs be run sequentially in a single job? If so, could you use a SAS data set instead of writing a CSV?

Re: file not found

p12937 — Fri, 27 Mar 2020 11:35:19 GMT

we are creating a second file because the original file has carriage returns and line feeds that need to be removed.

Re: file not found

PaigeMiller — Fri, 27 Mar 2020 11:42:31 GMT

@p12937 wrote:

we are creating a second file because the original file has carriage returns and line feeds that need to be removed.

I don't see this as a justification for separate programs that launch separately.

Re: file not found

p12937 — Fri, 27 Mar 2020 12:48:10 GMT

maybe I should try a windows community blog.

Re: file not found

Tom — Fri, 27 Mar 2020 13:44:25 GMT

@p12937 wrote:

Confirmation of completion: dos window closes and sas log shows that job is complete.

File name is correct. I run the second job again and it finds and processes the file.

There is no error message, the second job executes a windows dir command and looks for that file and 0 records are returned. When I run the job again it finds the file.

Sounds like disk timing issue. The second job is starting before the operating system has finished really writing the file and updating the directory entries. Try just adding a delay between the two steps. If you can edit the SAS code you could either put a sleep() function call at the end of the first one or the start of the second one.

Re: file not found

PaigeMiller — Fri, 27 Mar 2020 13:53:30 GMT

There is no error message, the second job executes a windows dir command and looks for that file and 0 records are returned.

Have you checked to see that the first program is actually writing the correct contents to the csv file? If this isn't happening correctly, then the timing isn't the problem. Incorrect contents in the csv file could result in 0 records being read, and no error message might appear.

Re: file not found

p12937 — Fri, 27 Mar 2020 14:33:08 GMT

the second job runs 5 minutes after jobs. Is there maybe a windows command that could refresh the directory/cache or something? we are going to try running the windows dir command multiple times

Re: file not found

Kurt_Bremser — Fri, 27 Mar 2020 16:12:13 GMT

Have you looked at the timestamps of the files, as suggested? Also look at the timestamps of the log files, as they tell you exactly when the SAS process closed the file.

I have experienced examples of SAS processes doing literally nothing for minutes because of system congestion.

Re: file not found

p12937 — Fri, 27 Mar 2020 17:18:28 GMT

yes, I have checked file timestamps, log timestamps and log messages. I really don't think this is a SAS issue, but since I am running SAS programs I thought someone might have had a similar issue with files not being available right away on a windows server or sfa.

Re: file not found

PaigeMiller — Fri, 27 Mar 2020 19:19:09 GMT

Did you check to see if the csv file that is created has the proper contents? If it is not created with the proper contents, this could cause the second program to read 0 records.

Re: file not found

Tom — Fri, 27 Mar 2020 19:39:25 GMT

Does the first job take longer than 5 minutes? Why not just change the schedule to run them further apart.