Could someone please explain me how does the Union set operator work. Does it remove duplicates only from each individual dataset or does it remove overall duplicates from the resultant dataset as well?
I have read that while joining two datasets vertically using Union operator, the duplicates are dropped.
But when I used it to join two datasets, I got the duplicate rows in the final dataset!
Thanks in advance.
It removes duplicates:
data test1;
set sashelp.class (obs=5);
run;
data test2;
set sashelp.class (firstobs=5 obs=10);
run;
proc sql;
create table want as
select * from test1
union
select * from test2
;
quit;
proc print data=want noobs;
run;
See the result:
Name Sex Age Height Weight Alfred M 14 69.0 112.5 Alice F 13 56.5 84.0 Barbara F 13 65.3 98.0 Carol F 14 62.8 102.5 Henry M 14 63.5 102.5 James M 12 57.3 83.0 Jane F 12 59.8 84.5 Janet F 15 62.5 112.5 Jeffrey M 13 62.5 84.0 John M 12 59.0 99.5
Now, if you use union all instead of just union, you get this:
Name Sex Age Height Weight Alfred M 14 69.0 112.5 Alice F 13 56.5 84.0 Barbara F 13 65.3 98.0 Carol F 14 62.8 102.5 Henry M 14 63.5 102.5 Henry M 14 63.5 102.5 James M 12 57.3 83.0 Jane F 12 59.8 84.5 Janet F 15 62.5 112.5 Jeffrey M 13 62.5 84.0 John M 12 59.0 99.5
Henry now appears two times.
It removes duplicates:
data test1;
set sashelp.class (obs=5);
run;
data test2;
set sashelp.class (firstobs=5 obs=10);
run;
proc sql;
create table want as
select * from test1
union
select * from test2
;
quit;
proc print data=want noobs;
run;
See the result:
Name Sex Age Height Weight Alfred M 14 69.0 112.5 Alice F 13 56.5 84.0 Barbara F 13 65.3 98.0 Carol F 14 62.8 102.5 Henry M 14 63.5 102.5 James M 12 57.3 83.0 Jane F 12 59.8 84.5 Janet F 15 62.5 112.5 Jeffrey M 13 62.5 84.0 John M 12 59.0 99.5
Now, if you use union all instead of just union, you get this:
Name Sex Age Height Weight Alfred M 14 69.0 112.5 Alice F 13 56.5 84.0 Barbara F 13 65.3 98.0 Carol F 14 62.8 102.5 Henry M 14 63.5 102.5 Henry M 14 63.5 102.5 James M 12 57.3 83.0 Jane F 12 59.8 84.5 Janet F 15 62.5 112.5 Jeffrey M 13 62.5 84.0 John M 12 59.0 99.5
Henry now appears two times.
Union does not remove duplicates. It merely appends the data in the subsequent step to the data in the current step. Exactly the same as if you use set a b; in a datastep or proc append one set to another.
@RW9 I beg to differ. Please run my example.
Quite true, union removes dups, union all does not.
http://stackoverflow.com/questions/49925/what-is-the-difference-between-union-and-union-all
People usually use union all to avoid the performance penalty caused by the sort distinct (by all! variables) that scans for the duplicates.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.