Some advice on the code below please. Is this the right code for a dataset with millions of records where I want to restrict to the most recent record (maximum time_flag) and retain all the columns associated with it for each group ordered and defined by 3 variable? I fear using proc sort would take ages. * Take only the most recent record for each ID / var1 / var2 group; PROC SQL; create table spine as select distinct ID, Var1, Var2, Var3, Var4, time_flag, from mydataset as big group by ID, var1, var2 having time_flag = max(time_flag) order by ID, var1, var2; quit; ... View more

Can people advise on the most efficient/quickest way to check if a dataset with millions of records (millions of individuals where each individual has multiple records), contains a list of unique individual records as specified in a small dataset (30,000 unique IDs). So I have a dataset called ‘small’ with one variable called ID with 30,000 unique records. Then a dataset called ‘big’ with a variables ID and other variables with millions of records (ID not unique). I want to find how many individuals from ‘small’ I can find in ‘Big’. It is taking me days to sort by ID and merge. There must be a more efficient way. Thanks in advance ... View more

Is the code below correct to obtain robust standard errors for a regression where the outcome variable is binary (0,1) and the independent variables are categorical and binaries? The dataset is at individual level: one unique person (identified by ID) per row. Is it right to use ‘cluster id’ to obtain robust S.E.? Is it ok to use individual weights together with the ‘cluster id’ option as done below? proc surveyreg data=matched_dataset; cluster id; class is_treated(ref='0') categorical_covariates; model outcome_variable = is_treated covariate1 covariate2 …/ CLPARM solution; weight _MATCHWGT_; run; ... View more