Stop a datastep after certain time

[blooneel]

Hi Guys,

I have a datastep that routinely gets stuck at certain observations for reasons slightly beyond my control or ability. However a workaround for me is to interrupt the data step after sometime and work with whatever records I get. Any IDea how I can achieve that ?

Basically I want the data step to stop after say 5 minutes and give me whatever number of observations it curently holds.

[PaigeMiller]

You can't stop a data step based on minutes.

You can say that you will only work on the first X observations and then stop. Use the OBS= option.

But a far better solution would be to figure out why you are having this problem and prevent the problem, or fix it.

[blooneel]

I have tried that . But I want to get as many OBS as possible. As to your other point im pulling observations from dictionary.tables and the code works absolutely fine on one of the servers and gets randomly stuck on the other one with no errors. There might be a solution but its well beyond my knowledge

[Tom]

FIrst make sure you are taking advantage of the indexes that exist on the dictionary tables (virtual tables?) be avoiding using functions on LIBNAME or MEMNAME fields.  Also use the PROC SQL and the DICTIONARY libref instead of using the views in the SASHELP.

Doing this should allow you to use WHERE conditions to limit the searching to the libraries and tables that you want.

** BAD ** ;
where libname = upcase("&libref");

** GOOD ** ;
where libname = %upcase("&Lifbref") ;

Otherwise switch to use PROC CONTENTS instead.

[blooneel]

I did start with using dictionary.tables and then proc contents and currently im using sashelp views for the reason - my SAS gui atleasts remains accessible and I can manually interrupt the data step from session management. with proc sql on dictionary.tables i had to completely terminate the session with no visible output.

[Shmuel]

Retain a datetime stamp at start running.

Compute end time by adding to 5 minutes or any other amount.

Check at every observation is end time reached and stop when done.

[blooneel]

Need a timer that can run indepently of the Data step (trouble being the processing usually gets stuck at reading some observations and may stay there for upto 3 hours) . so checking at the beginning of each record does not help me. Im open to doing this by somehow embedding OS commands in it. But would like it to be part of overall code.

[Shmuel]

Would knowledge of the last observation executed or read will help to focus on the problematic one ?

You can define a filename.

For each observation save info to that file - rewriting it, one row only.

After canceling the job, you will have info of the last observation saved.

try:

   - read new obs

   - open file rewrite mode

   - write info

   - close file

   - deal with the obs

[Peter_C]

You are at risk of investigating irrelevant libraries.
The list of these librefs might vary vetween servers.
The impact on performance becomes greatest if for example you have one libref pointing into an excel workbook. The collector of metadata for dictionary.tables will investigate how many rows / columns are in each sheet and range name.
If a libref is sufficiently complete to open a connection to a database through SAS/ACCESS it might be trying to count the number of columns in a VIEW.
It might be sufficient to use dictionary.members which populates much more quickly - having fewer columns that risk far less investigation of the platform.

[blooneel]

Hey ,

Dictionary.Members was working quickly. Anyway I can get Encryption information from that ?

[Peter_C]

no

but a join with other metadata can be optimised on libname and memname

##- Please type your reply above this line. Simple formatting, no
attachments. -##