I am currently trying to merge 2 large data sets using SQL. I haven't had to merge data of this size before, and I need to find an efficient way to merge the data.   The first data set, home._2004, contains 45 variables and 67 million observations. The second data set, home.c2final, contains 73 variables and 24 million observations.   Here are some example observations and variables to show that the data sets have some, but not all variables in common.   Sample Data  home._2004 Year      State          MSA     2004      NE             00561 2004      AL              15646 2004      AL              45646   home.c2final Year      State           Loan_Amount 1999     NE                55000 2004     KS               165000 2005     AK                200000   I have been using: DATA home._2004; SET home._2004data (obs=10); RUN; DATA home.c2final; SET home.c2 (obs=50); RUN; PROC SQL; CREATE TABLE home.merge as SELECT * FROM home._2004 OUTER UNION ALL SELECT * FROM home.c2final; QUIT; This works when I am only using a few observations from each, but they are too large to do all together. When I used 10 observations from home._2004 and all observations from home.c2final it takes 10 minutes. I am trying to find a way to make this more efficient. My thought was to find a way to merge a number of observations from home._2004 with all of home.c2final, and have the program do this with the next group of observations until all of the data is merged.    For example if I used 50 observations from home._2004: Observations 1-50 merge with c2final Followed by Observations 51-100 merge with c2final etc.   I am new to SAS, and I am not sure if this is even a possibility.  ... View more

I am trying to find an efficient way to merge two very large data sets. I am new to SAS and I have only ever merged small data sets, so I am not sure where to start.   The first data set, home._2004, contains 45 variables and 67 million observations. The second data set, home.c2final, contains 73 variables and 24 million observations.   The common variables are Year, MSA, CountyCode, state, and Loan_Amount.    This is the code I would have used if the data wasn't so large: DATA home.testmerge; MERGE home._2004 home.c2final; BY Year MSA CountyCode state Loan_Amount; RUN; As you can see, I have very little experience with SAS. From the research I done, there are more efficient ways to merge data of this size, but I do not know where to start with those methods. I was trying to find some way to merge the data using groups of observations from the home._2004 data, but I don't know how to set it up so that after it's done with one group of observations it movies to the next.     For further clarification here are some sample observations from each set. Like I mentioned, I am very new to SAS, so I did not know if there was an easier way to display the data.   Home._2004 Sample: file:///D:/Research/2004sample.html Home.c2final Sample: file:///D:/Research/c2finalsample.html ... View more