topic Re: Performing the same Datastep on many similar data files in SAS Programming

Performing the same Datastep on many similar data files

MB_Analyst — Thu, 18 Oct 2018 13:05:34 GMT

I have a number of data files that I want to count observations per group for, in addition to removing id's with only one observation.

To do this with a single data file I can use:

DATA want;
    SET have;
    BY id date;
    IF first.id THEN count=0; 
    count+1;
   IF first.id AND last.id THEN DELETE;
RUN;

Is there anyway I can set this up as a macro to run it on 5+ data files?

Re: Performing the same Datastep on many similar data files

andreas_lds — Thu, 18 Oct 2018 13:12:07 GMT

Untested:

%macro MACRO_NAME(Have=, Want=);
data &want.;
   set &have.;
   by id date;

   if first.id then
      count=0;
   count+1;

   if first.id and last.id then
      delete;
run;
%mend;

%MACRO_NAME(Have= have, Want= want)

Re: Performing the same Datastep on many similar data files

RW9 — Thu, 18 Oct 2018 13:17:53 GMT

You don't need macro, you can simply combine the datasets you want to check, then process them using by groups. It will be faster and use less resources. Post some test data for an exact example but:

data want;
  set dataset1 dataset2 dataset3... indsname=tmp;
  dsname=tmp;
run;
data want;
  set want;
  by dsname id;
  if first.id then count=0;
  count=sum(count,1);
  if first.id and last.id then delete;
run;

Re: Performing the same Datastep on many similar data files

data_null__ — Thu, 18 Oct 2018 13:20:34 GMT

@RW9 wrote:

You don't need macro, you can simply combine the datasets you want to check, then process them using by groups. It will be faster and use less resources. Post some test data for an exact example but:
data want;
  set dataset1 dataset2 dataset3... indsname=tmp;
  dsname=tmp;
run;
data want;
  set want;
  by dsname id;
  if first.id then count=0;
  count=sum(count,1);
  if first.id and last.id then delete;
run;

That gets messy if the only common variable(s) are the ID variables.

Re: Performing the same Datastep on many similar data files

RW9 — Thu, 18 Oct 2018 13:35:13 GMT

True, but then you would assume that if the process is that similar for each of these datasets either - its data split out before, data that can be set together retaining on necessary variables, or data from which the process of counting id could be extracted from the data and then re-merged. Those scenarios are pretty simple, if its more complicated that that, maybe assess the steps before.