Splitting log and executing a set of subjects together

[Ramya2]

Hi All,

I have enrollment dataset of 10 subjects. Upon matched subjects from enrollment subjects rest of the forms (demog, conmeds,hxsys) subjects are extracted with its data. example , demog extracted  has 10 subjects, conmeds extracted for 23 subjects(same 10 subjects  with different data ) and hxsys has 12 subjects.Now my requirement is, for those 10 subjects of enrollment need to execute 3 at a time and generate log for those 3 and sleep for 1 minute , for next iteration 4 to 6 and sleep for 1 minute and so on.the task here is, for subject '123' we have multiple information in conmeds and hxsys. So considering this , if subject '123' comes in 1st iteration all the irecords for '123'  should be executed. Here I have a major parts of my code.

/***creating enrollment dataset****////

data test.enrlment_1;
set enrlment_1;
run;

data _NULL_;
if 0 then set enrlment_1 nobs=n;
call symputx('nrows',n);
stop;
run;
%put nobs=&nrows;

/**all forms dumpped in final dataset, creating macro ***////
data _null_;
set final_1 end=eof;
call symput('pt'||compress(put(_n_,best.)),trim(left(upcase(subject))));

if eof then call symput('npts',compress(put(_n_,best.)));
run;

****creating macro variable to create or run the report conditionally******;

%let count=0;

data _null_ ;
set final_1 end=eof NOBS=COUNT;;
countc = put(count,best.);
if _n_ = 1 then call symput('count',countc);
run;
%put &count;

/***After sepating each form from enrollment, looper macro created for ods. In Read macro looper macro is called****/////

%macro read;
%do j=1 %to &nrows;
%if %sysfunc(mod(&j,3))=1 %then

%do;

proc printto new log ="/ctshared/cdr/dev/RAMYA/archive/logfile_&j..log";

run;
%let rc=%sysfunc(sleep(60,1));
%end;

%do i=1 %to &npts;

%looper(subject=&&pt&j.);

%end;
%end;
%mend caller();
%caller();

%mend read;
%read;

[Kurt_Bremser]

This won't work:

%macro read; /* here you start defining macro read */
%do j=1 %to &nrows;
%if %sysfunc(mod(&j,3))=1 %then

%do;

proc printto new log ="/ctshared/cdr/dev/RAMYA/archive/logfile_&j..log";

run;
%let rc=%sysfunc(sleep(60,1));
%end;

%do i=1 %to &npts;

%looper(subject=&&pt&j.);

%end;
%end;
%mend caller(); /* but here you end the definition of macro caller ???? */
%caller();

%mend read;
%read;

DO NOT EVEN ATTEMPT to nest macro definitions. All macros are defined in the global table, so nesting macro definitions only causes confusion and serves no other purpose.

[Ramya2]

Then what will be the solution

[Kurt_Bremser]

I pointed out the issues I could see in your (clearly incomplete) code. Fix those, and study the log.

For detailed help, we will need example data in usable form (see my footnotes), the whole macro code and at least a good idea what the macro looper does.

[RW9]

Rethink your process.  Almost all tasks boil down to get data, process data, produce output (there is even an anacronym ETL - Extract, Transform, Load).  So why do you have "get some bits of data", "do something", "sleep", "do it again"?  Doesn't make any sense, and sounds like the whole process is bad.  For instance, from the code you provide it looks like your producing a report, perhaps by three subjects?  If so read all the data in then add a variable called group or something.  Then use proc report with by group; to produce one report per block of three.  

[Quentin]

Agree with @RW9 .  Think about creating the data you want (sounds like it will have a subject ID, and multiple records per subject).  Then think about the reporting you want.  Often the answer is BY-group processing.  If you really have been given a requirement to have a separate log for each by group, then you probably need to use the macro language or some other code generation technique.  

Using SASHELP.SHOES as an example dataset, and making one report per region, if you have a macro like:

%macro MkReport(region=) ;
  proc printto new log ="c:\junk\logfile_&region..log";
  run;
  proc print data=sashelp.shoes ;
    where region="&region" ;
  run ;
  proc printto ;
  run ;
%mend MkReport ;

Which you call like below, to make a report for Canada and a log file for Canada:

options mprint ;
%MkReport(region=Canada)

You could use the dataset to drive the macro via call execute, which for data-driven code is often handier than macro looping:

data _null_ ;
  set sashelp.shoes ;
  by region ;
  if first.region then do ;
    call execute('%nrstr(%MkReport(region='||trim(region)||'))') ;
  end ;
run ;

That will call the macro %MkReport once for each region.  When the macro executes, it redirects the log file and then generates the report.

That said, the requirement to have separate log files is unusual.  Usually this sort of looping over is unnecessary, as  BY group processing is much more efficient.

[Ramya2]

@Quentin , Yes I understand your advice. But as per the requirement I should split the log files. Since data is huge it is consuming more disk spae. So it is adviced to do so.

[Patrick]

---

@Ramya2 wrote:
@Quentin , Yes I understand your advice. But as per the requirement I should split the log files. Since data is huge it is consuming more disk spae. So it is adviced to do so.

---

By group processing will be more efficient than calling the same SAS table many times with a where condition.

The size of the SAS log doesn't depend on the data volumes you're processing but only on the number of steps/lines of code and level of logging.

[Kurt_Bremser]

---

@Ramya2 wrote:
Since data is huge it is consuming more disk spae.

---

That is a patently wrong argument. Splitting files increases disk storage consumption.

[Ramya2]

@Kurt_Bremser
ohhh.. But what you think is a best solution apart from increasing the disk space?Through pro grammatically What can we done?

[Kurt_Bremser]

---

@Ramya2 wrote:
@Kurt_Bremser
ohhh.. But what you think is a best solution apart from increasing the disk space?Through pro grammatically What can we done?

---

When you run out of disk space, start with assessing the "is" situation:

• how much space is available in the first place?
• is there something that limits available space on top of that (user quota)?
• what file exactly causes the overflow? Logs are usually not that big, but datasets or utility files are
• if it is the log, inspect it
• if it is datasets, inspect them for content (columns, observations)
• if it is a certain step that fails, isolate that and see what's happening in detail, especially with SQL

Once you really know what's happening, you can work on a remedy.