Hi, I have created a dataset from two different datasets and I want to remove any duplicates in the new datasets. I have tried using the the proc sql distinct approach as well as a Data step with the first. approach and I keep getting a dataset that has duplicates but some of the duplicates are removed.. The dataset has only 3 variable, ID as a character variable and Unit Price and Effective Date both as numeric variables. I double checked and made sure are and were formatted the same before creating the new dataset. Any suggestion on what I may be missing and should check....Thanks in Avance.
I would be interested is seeing the SQL code that did not work. Order of when to appear DISTINCT may have been the issue.
Also you may need to look at the ID in case some of them have one or more leading blank characters as "string" is not equal to " string" and could cause apparent duplicates.
Your join may have issues.
Otherwise you can check unique/duplicate records in SAS with NOUNIQUEKEY in proc sort. Available in SAS 9.3+
proc sort data=have out=duprec nouniquekey uniqueout=want;
by ID;
run;
I would be interested is seeing the SQL code that did not work. Order of when to appear DISTINCT may have been the issue.
Also you may need to look at the ID in case some of them have one or more leading blank characters as "string" is not equal to " string" and could cause apparent duplicates.
Hi Reeza & Ballardw,
Thanks for your help and suggestions. With your suggestions of the possibility of blank characters for the ID variable, it turns out that there must have been a blank or blanks in the Unit Price as I used the Compress Function on both the ID and Unit Price variables having to convert Unit Price to a character variable first. It did the trick. Thanks once againn for your help.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.