@Reeza wrote: I suspect that appending these is the better solution. Do you have any sort of naming conventions with the data set. @lingcx wrote: Hi all, (each has 400k obs and 7 vars including one unique ID and 6 other vars). Your method will not work because once SAS merges in the data the identical variables will be overwritten so this doesn't make sense to me. Unless it's 6 different variables but I suspect not. I do have 6 vars named differently but systematically in each of the 1000 datasets. For example, dataset 'day1' has ID, day1_v1, day1_v2, day1_v3, day1_v4, day1_v5, and day1_6, and similarly, dataset 'day2' has day2_v1, day2_v2, day2_v3, day2_v4, day2_v5, and day2_6, etc. @lingcx wrote: Hi all, This is my first post asking for help. I have about 1000 medium-sized datasets (each has 400k obs and 7 vars including one unique ID and 6 other vars). I would like to merge all 1000 together. I tried two methods. The first method simply cannot be accomplished because of the huge numbers of obs and vars. The second took a long time to run (more than 12 hours) especially when the "final" dataset gets larger and larger in the end. I'm wondering whether there is a much more efficient way to merge all 1000-ish datasets together. I thought about splitting these into ten batches and merge the 10 semi-final datasets but I'm not sure whether it would help. If you have a better idea how to proceed with this, please please let me know!! Much appreciated. Method 1: Data final; merge day: ; by ID; run; Method 2: %macro merge(data); Data final; merge final c.&data.; by ID; run; %Mend; %merge(day1); %merge(day2); %merge(day4); %merge(day5); %merge(day7); %merge(day8); %merge(day11); %merge(day13); %merge(day20); %merge(day24); .... %merge(day10000); If you really do need it wide, you can still append and then transpose afterwards. How much faster do you think it will be if I append and tranpose? Transpose seems to be a time-consuming procedure to me too...
... View more