Text mining and content categorization

Compare observations in multiple datasets

New User
Posts: 1

Compare observations in multiple datasets



I have several datasets I need to stack, but in some the field is labeled Gender and either has 'M' or 'F' in the observations and in other datasets the field is Gender_DESC and 'Male' or 'Female' in the observations. I need to make sure the field name and values are consistent. Is there an easy way to compare the datasets and see what's inside the observations so I know what needs to be fixed?


Let me know if you have any questions.



Super User
Posts: 6,543

Re: Compare observations in multiple datasets

One possibility would be to take every data set that contains GENDER_DESC and create GENDER.  That's relatively simple:


length gender $ 1;

gender = gender_desc;


Then you could use the same process you use now to see if the GENDER values are equal.


There is a weakness to this approach.  By copying just the first character into GENDER, you may create the impression that all is OK when it is not.  For example:


length gender $ 1;

gender_desc = 'Molly';

gender = gender_desc;


GENDER will be "M" but there is no way to tell that GENDER_DESC is bad without examining in more detail.  Those sorts of comparisons are possible but are more complex to set up so the right solution may depend upon your being comfortable with more complex code.

Ask a Question
Discussion stats
  • 1 reply
  • 2 in conversation