BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
ChrisNZ
Tourmaline | Level 20

@Tom 

If the assumptions about the data change, the code needs to adapt to the new conditions.

 

The hierarchy of the run times remains the same.   POINT=   >   READ-AHEAD   >   DOW+BY

 

 

data TEMP3;
  set TEMP1 nobs=NOBS;
  by I T; 
  if ^LASTOBS then set TEMP1(firstobs=4 keep=I rename=(I=I1)) end=LASTOBS;
  by I1 ;
  if I ne I1 then KEEP=1;
  if _N_ > 3 then if first.I1 or lag1(first.I1) or lag2(first.I1) then KEEP=1;
  if _N_ > NOBS-3  then KEEP=1;                        if keep;
run;

Comparison of WORK.TEMP2 with WORK.TEMP3

 

NOTE: No unequal values were found. All values compared equal.

ChrisNZ
Tourmaline | Level 20

Note that an explicit DOW has a speed advantage over the implicit data step loop.

The first step is faster than the second one just because we don't use the implicit loop logic.

 

data TEMP3a;
  do N=1 by 1 until (LASTOBS);
    set TEMP1 nobs=NOBS end=LASTOBS;
    if ^LASTOBS1 then set TEMP1(firstobs=4 keep=I rename=(I=I1)) end=LASTOBS1;
    by I1 ;       
    if I ne I1 then KEEP=1;
    if N > 3 then if first.I1 or lag1(first.I1) or lag2(first.I1) then KEEP=1;
    if N > NOBS-3  then KEEP=1;                        
    if KEEP then output;           
    KEEP=0; 
  end;
run;
 
data TEMP3b;
    set TEMP1 nobs=NOBS ;
    if ^LASTOBS1 then set TEMP1(firstobs=4 keep=I rename=(I=I1)) end=LASTOBS1;
    by I1 ;                         
    if I ne I1 then KEEP=1;
    if _N_ > 3 then if first.I1 or lag1(first.I1) or lag2(first.I1) then KEEP=1;
    if _N_ > NOBS-3  then KEEP=1;                  
    if KEEP ;
run;

Your DOW step is faster than mine though (and arguably more legible), so better for the question at hand.

 

 

 

 

Tom
Super User Tom
Super User

@ChrisNZ  I couldn't figure out the logic of why your step should work.

But I think you are essentially just doing this logic.

data temp3;
  set temp1;
  if not eof then set temp1(in=in1 firstobs=4 keep=I rename=(I=I1)) end=eof;
  else in1=0;
  if (I ne I1) or not in1;
run;

Or put another way.

data temp3;
  set temp1;
  if eof then output;
  else set temp1(firstobs=4 keep=I rename=(I=I1)) end=eof;
  if (I ne I1) then output ;
run;

 

AndrewHowell
Moderator

Are I & T fixed at (say) 500 & 10, respectively?

 

If so, then just filter the top 3 values of T (8, 9, 10)?

 

data temp1;
	set temp;
	where t in (8,9,10);
run;

Or if this is oversimplifying, then try (something like) this:

data temp1;
	set temp;
	by i t;
	OnePrev=lag1(y); TwoPrev=lag2(y);
	if last.i then output;
run;

 

mkeintz
PROC Star

I'd be interested in knowing whether this simpler program is as fast:

 

data want;
  if end_of_temp=0 then set temp (keep=i t) end=end_of_temp;
  by i t ;
  if _n_>=3 then set temp;
  if last.i or lag(last.i) or lag2(last.i);
run;

The "if end_of_temp=0" condition prevents the data step from prematurely stopping, so that the "if _n_>=3 then set ...." statement can read the entire data set.

 

This program will include by groups with less than 3 observations.  If you want to exclude such groups, then modify the subsetting if to

 

  if (last.i or lag(last.i) or lag2(last.i)) and lag2(i)=i;

 

 

--------------------------
The hash OUTPUT method will overwrite a SAS data set, but not append. That can be costly. Consider voting for Add a HASH object method which would append a hash object to an existing SAS data set

Would enabling PROC SORT to simultaneously output multiple datasets be useful? Then vote for
Allow PROC SORT to output multiple datasets

--------------------------
ChrisNZ
Tourmaline | Level 20

>But I think you are essentially just doing this logic.

@Tom Yes, and your code is more legible, again!

@mkeintz The speed is the same

data TEMP1;
  do I=1 to 5e6;
    do T=1 to ceil(ranuni(1)*10);
      Y=rand("normal");
      output;
    end;
  end;
run;

%* POINT=      40 s;
data TEMP2(compress=no);
  set TEMP1;
  by I ;
  if first.I then START = _N_;
  retain START;
  if last.I then do POINT = max(START, _N_ - 2) to _N_;
    set TEMP1 point=POINT;
    output;
  end;
  drop START;
run;

%* DOW + BY      13 s;
data TEMP3(compress=no);
  do _N=1 by 1 until(last.I);
    set TEMP1;
    by I;
  end;
  do _N2=1 to _N;
    set TEMP1;
    if _N2 >= _N-2 then output;
  end;
  drop _N _N2;
run;

%* Read-ahead + BY     18 s ;
data TEMP4(compress=no);
  set TEMP1 nobs=NOBS;
  by I T; 
  if ^LASTOBS then set TEMP1(firstobs=4 keep=I rename=(I=I1)) end=LASTOBS;
  by I1 ;
  if I ne I1 then KEEP=1;
  if _N_ > 3 then if first.I1 or lag1(first.I1) or lag2(first.I1) then KEEP=1;
  if _N_ > NOBS-3  then KEEP=1;                        if keep;
run ;
            
%* Read-ahead    13s ;
data TEMP5(compress=no);
  set TEMP1;
  if LASTOBS then output;
  else set TEMP1(firstobs=4 keep=I rename=(I=I1)) end=LASTOBS;
  if (I ne I1) then output ;
run;
     

 

tatoknight
Calcite | Level 5

Hi all,

 

Just with the code 

POINT = max(START, _N_ - 2) to _N_;

It works completely, but why? What is the purpose of the max function and how does it prevent errors? 

 

Thanks!

PGStats
Opal | Level 21

START contains the position of the first record from the current group. If the group has less than 3 records, the max function prevents the loop from reading records from the previous group.

PG

Ready to join fellow brilliant minds for the SAS Hackathon?

Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.

Register today!
How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 22 replies
  • 10647 views
  • 6 likes
  • 8 in conversation