topic Re: Delete overlapping observation by participant in SAS Programming

Delete overlapping observation by participant

JLang055 — Sat, 19 Nov 2022 15:09:07 GMT

I have a patient dataset with multiple observations and variable. I need to delete overlapping observations for each participant. See example below.

HAVE

FID	MOOD	ANXIETY	EAT	DEVELOP
1		1
1			1
1	1
1	1
2			1
2		1
2			1
3		1
3	1

WANT

FID	MOOD	ANXIETY	EAT	DEVELOP
1		1
1			1
1	1
2			1
2		1
3		1
3	1

Re: Delete overlapping observation by participant

fja — Sat, 19 Nov 2022 15:21:17 GMT

Hello!
Would you like to filter an existing dataset or would you prevent the insertion of superfluous observations?
--fja

Re: Delete overlapping observation by participant

JLang055 — Sat, 19 Nov 2022 15:31:53 GMT

Im trying to prevent double counting a participant in specific categories.

Re: Delete overlapping observation by participant

fja — Sat, 19 Nov 2022 16:01:02 GMT

What about defining an index with option unique?

--fja

Re: Delete overlapping observation by participant

fja — Sat, 19 Nov 2022 16:23:31 GMT

OK, if you just want to have a "clean" intermediary table, than you could use proc sort:

data work.TestData;
	infile datalines dsd;
	input FID MOOD ANXIETY EAT DEVELOP;
datalines;	
1, ,1, , 
1, , ,1, 
1,1, , , 
1,1, , , 
2, , ,1, 
2, ,1, , 
2, , ,1, 
3, ,1, , 
3,1, , , 
;
run;

PROC SORT DATA = work.TestData NODUPKEY out=work.testdata2;
BY MOOD ANXIETY EAT;
RUN;

Re: Delete overlapping observation by participant

Tom — Sat, 19 Nov 2022 19:41:37 GMT

For data like that with values that are either 1 or missing I would probably just collapse to one observation per subject.

data want;
  update have(obs=0) have;
  by fid;
run;

Obs    FID    MOOD    ANXIETY    EAT    DEVELOP

 1      1       1        1        1        .
 2      2       .        1        1        .
 3      3       1        1        .        .

Re: Delete overlapping observation by participant

fja — Sat, 19 Nov 2022 22:14:23 GMT

That resulted in a more sane looking dataset, agreed. It is just that @JLang055 asked for a different kind of output.
--fja

Re: Delete overlapping observation by participant

Patrick — Sat, 19 Nov 2022 22:30:47 GMT

Below should create what you're asking for.

proc sort data=have out=want nodupkey;
  by _all_;
run;

Code not tested because data not provided in a form that doesn't require work to use it directly in code (i.e. a fully working SAS data step creating the data).

Re: Delete overlapping observation by participant

mkeintz — Sun, 20 Nov 2022 02:48:15 GMT

This program will keep the first instance of each unique combination of variables:

data have;
	infile datalines dsd;
	input FID MOOD ANXIETY EAT DEVELOP;
datalines;	
1, ,1, , 
1, , ,1, 
1,1, , , 
1,1, , , 
2, , ,1, 
2, ,1, , 
2, , ,1, 
3, ,1, , 
3,1, , , 
;
run;
data want;
  set have;
  if _n_=1 then do;
    declare hash h (dataset:'have (obs=0)');
      h.definekey(all:'Y');
      h.definedone();
  end;
  if h.add()=0 ;
run;

The hash object is "keyed" on all the variables in dataset have (think of it as using a compound index based on the combination of all variables).

The hash method ADD will be successful (i.e. h.add()=0) only when there is not already a dataitem (i.e. a "row") in it with the same combination of variables. As a result the dataset does not even need to be sorted, even by FID.

Re: Delete overlapping observation by participant

JLang055 — Sun, 20 Nov 2022 22:12:08 GMT

I think this could be a good option. When I run the code you suggest I run into the following error:

NOTE: Writing TAGSETS.SASREPORT13(EGSR) Body file: EGSR
24         
25         GOPTIONS ACCESSIBLE;
26         data DIAG_FULL_F;
27         	update have(obs=0) have;
ERROR: File WORK.HAVE.DATA does not exist.
ERROR: File WORK.HAVE.DATA does not exist.
28         	by FID;
29         run;

NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.DIAG_FULL_F may be incomplete.  When this step was stopped there were 0 observations and 0 variables.
WARNING: Data set WORK.DIAG_FULL_F was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.01 seconds

Do you have any idea whats going on with it?

Re: Delete overlapping observation by participant

Kurt_Bremser — Sun, 20 Nov 2022 23:13:56 GMT

ERROR: File WORK.HAVE.DATA does not exist.

What could be more clear than this? You need to create the HAVE (the name you used in your initial post) dataset before you can use it.

Re: Delete overlapping observation by participant

JLang055 — Mon, 21 Nov 2022 02:28:48 GMT

Hi All,

I took @Tom suggestion to merge all the data into one single row per participant. I ran some code that worked for me.

%macro count (Var); *create new diagnosis count variable;

proc summary data=DIAG_FULL; 
	var &var;

	by FID;

	output out=&var sum=;
run;

data &VAR;
	set &VAR;

	IF &VAR >= 1 then &VAR =1;
	ELSE &VAR =0;

	drop _TYPE_ _FREQ_;
run;

%mend count;

%count(MOOD);
%count(ANXIETY);
%count(DEVELOP);
%count(CONDUCT);
%count(ADHD);

data CLINIC_DIAG;
	merge MOOD ANXIETY DEVELOP CONDUCT ADHD DIAG_MULTI;
	by FID;
run;

Re: Delete overlapping observation by participant

fja — Mon, 21 Nov 2022 07:29:07 GMT

Congratulations to your first solution then ... but could at least spear Toms posting a like. 😉