topic Re: Splitting log and executing a set of subjects together in SAS Programming

Splitting log and executing a set of subjects together

Ramya2 — Thu, 31 Jan 2019 06:11:51 GMT

Hi All,

I have enrollment dataset of 10 subjects. Upon matched subjects from enrollment subjects rest of the forms (demog, conmeds,hxsys) subjects are extracted with its data. example , demog extracted has 10 subjects, conmeds extracted for 23 subjects(same 10 subjects with different data ) and hxsys has 12 subjects.Now my requirement is, for those 10 subjects of enrollment need to execute 3 at a time and generate log for those 3 and sleep for 1 minute , for next iteration 4 to 6 and sleep for 1 minute and so on.the task here is, for subject '123' we have multiple information in conmeds and hxsys. So considering this , if subject '123' comes in 1st iteration all the irecords for '123' should be executed. Here I have a major parts of my code.

/***creating enrollment dataset****////

data test.enrlment_1;
set enrlment_1;
run;

data _NULL_;
if 0 then set enrlment_1 nobs=n;
call symputx('nrows',n);
stop;
run;
%put nobs=&nrows;

/**all forms dumpped in final dataset, creating macro ***////
data _null_;
set final_1 end=eof;
call symput('pt'||compress(put(_n_,best.)),trim(left(upcase(subject))));

if eof then call symput('npts',compress(put(_n_,best.)));
run;

****creating macro variable to create or run the report conditionally******;

%let count=0;

data _null_ ;
set final_1 end=eof NOBS=COUNT;;
countc = put(count,best.);
if _n_ = 1 then call symput('count',countc);
run;
%put &count;

/***After sepating each form from enrollment, looper macro created for ods. In Read macro looper macro is called****/////

%macro read;
%do j=1 %to &nrows;
%if %sysfunc(mod(&j,3))=1 %then

%do;

proc printto new log ="/ctshared/cdr/dev/RAMYA/archive/logfile_&j..log";

run;
%let rc=%sysfunc(sleep(60,1));
%end;

%do i=1 %to &npts;

%looper(subject=&&pt&j.);

%end;
%end;
%mend caller();
%caller();

%mend read;
%read;

Re: Splitting log and executing a set of subjects together

Kurt_Bremser — Thu, 31 Jan 2019 07:07:30 GMT

This won't work:

%macro read; /* here you start defining macro read */
%do j=1 %to &nrows;
%if %sysfunc(mod(&j,3))=1 %then

%do;

proc printto new log ="/ctshared/cdr/dev/RAMYA/archive/logfile_&j..log";

run;
%let rc=%sysfunc(sleep(60,1));
%end;

%do i=1 %to &npts;

%looper(subject=&&pt&j.);

%end;
%end;
%mend caller(); /* but here you end the definition of macro caller ???? */
%caller();

%mend read;
%read;

DO NOT EVEN ATTEMPT to nest macro definitions. All macros are defined in the global table, so nesting macro definitions only causes confusion and serves no other purpose.

Re: Splitting log and executing a set of subjects together

Ramya2 — Thu, 31 Jan 2019 07:29:34 GMT

Then what will be the solution

Re: Splitting log and executing a set of subjects together

Kurt_Bremser — Thu, 31 Jan 2019 08:16:45 GMT

I pointed out the issues I could see in your (clearly incomplete) code. Fix those, and study the log.

For detailed help, we will need example data in usable form (see my footnotes), the whole macro code and at least a good idea what the macro looper does.

Re: Splitting log and executing a set of subjects together

RW9 — Thu, 31 Jan 2019 08:39:36 GMT

Rethink your process. Almost all tasks boil down to get data, process data, produce output (there is even an anacronym ETL - Extract, Transform, Load). So why do you have "get some bits of data", "do something", "sleep", "do it again"? Doesn't make any sense, and sounds like the whole process is bad. For instance, from the code you provide it looks like your producing a report, perhaps by three subjects? If so read all the data in then add a variable called group or something. Then use proc report with by group; to produce one report per block of three.

Re: Splitting log and executing a set of subjects together

Quentin — Thu, 31 Jan 2019 13:03:12 GMT

Agree with @RW9 . Think about creating the data you want (sounds like it will have a subject ID, and multiple records per subject). Then think about the reporting you want. Often the answer is BY-group processing. If you really have been given a requirement to have a separate log for each by group, then you probably need to use the macro language or some other code generation technique.

Using SASHELP.SHOES as an example dataset, and making one report per region, if you have a macro like:

%macro MkReport(region=) ;
  proc printto new log ="c:\junk\logfile_&region..log";
  run;
  proc print data=sashelp.shoes ;
    where region="&region" ;
  run ;
  proc printto ;
  run ;
%mend MkReport ;

Which you call like below, to make a report for Canada and a log file for Canada:

options mprint ;
%MkReport(region=Canada)

You could use the dataset to drive the macro via call execute, which for data-driven code is often handier than macro looping:

data _null_ ;
  set sashelp.shoes ;
  by region ;
  if first.region then do ;
    call execute('%nrstr(%MkReport(region='||trim(region)||'))') ;
  end ;
run ;

That will call the macro %MkReport once for each region. When the macro executes, it redirects the log file and then generates the report.

That said, the requirement to have separate log files is unusual. Usually this sort of looping over is unnecessary, as BY group processing is much more efficient.

Re: Splitting log and executing a set of subjects together

Ramya2 — Tue, 05 Feb 2019 03:31:43 GMT

@Quentin , Yes I understand your advice. But as per the requirement I should split the log files. Since data is huge it is consuming more disk spae. So it is adviced to do so.

Re: Splitting log and executing a set of subjects together

Patrick — Tue, 05 Feb 2019 04:21:01 GMT

@Ramya2 wrote:
@Quentin , Yes I understand your advice. But as per the requirement I should split the log files. Since data is huge it is consuming more disk spae. So it is adviced to do so.

By group processing will be more efficient than calling the same SAS table many times with a where condition.

The size of the SAS log doesn't depend on the data volumes you're processing but only on the number of steps/lines of code and level of logging.

Re: Splitting log and executing a set of subjects together

Kurt_Bremser — Tue, 05 Feb 2019 09:50:14 GMT

@Ramya2 wrote:
Since data is huge it is consuming more disk spae.

That is a patently wrong argument. Splitting files increases disk storage consumption.

Re: Splitting log and executing a set of subjects together

Ramya2 — Tue, 05 Feb 2019 11:01:02 GMT

@Kurt_Bremser
ohhh.. But what you think is a best solution apart from increasing the disk space?Through pro grammatically What can we done?

Re: Splitting log and executing a set of subjects together

Kurt_Bremser — Tue, 05 Feb 2019 12:06:34 GMT

@Ramya2 wrote:
@Kurt_Bremser
ohhh.. But what you think is a best solution apart from increasing the disk space?Through pro grammatically What can we done?

When you run out of disk space, start with assessing the "is" situation:

how much space is available in the first place?
is there something that limits available space on top of that (user quota)?
what file exactly causes the overflow? Logs are usually not that big, but datasets or utility files are
if it is the log, inspect it
if it is datasets, inspect them for content (columns, observations)
if it is a certain step that fails, isolate that and see what's happening in detail, especially with SQL

Once you really know what's happening, you can work on a remedy.