Astounding, I'll keep in mind your advices . Thanks ... View more

Hi all Thank you very much for all your help. it's much appreciated for me. Regards ... View more

Morning all. I attached one screenshot of my dataset. The key is only var1 (numeric, 8 length) . It's something like a social Security but with less length. I need almost all field because I need to check if this variables match it with other variables that  there are in the source. Thanks ... View more

Thanks all for your ideas to improve the code. All code related to deal with memory system simply don't work due to limitations of size (only 4 gb) and the impossibility to change it fo a new powerfull one. I'd try splitting in more datasets (perhaps 20 or 25)  but how is the bes option for merge all those datasets?. Could I make an array in the merge statement? Regards ... View more

Thanks all for your comments I have made one text splitting the dataset in 10 smaller datasets, the results given are below options fullstimer; proc sort data = Dataset_A out = Dataset_B (compress=binary); by KEY; run; NOTE: There were 32498768 observations read from the data set Dataset_A. NOTE: The data set WORK.PERSONSHORT has 32498768 observations and 31 variables. NOTE: Compressing data set Dataset_B decreased size by 71.08 percent.       Compressed is 391645 pages; un-compressed would require 1354116 pages. NOTE: PROCEDURE SORT used (Total process time):       real time           44:04.28       user cpu time       1:54.30       system cpu time     1:13.66       Memory                            66929k       OS Memory                         100216k       Timestamp            16/09/2014  17:22:45 Now I splitt the dataset with a macro (thanks to Selvaratnam Sridharma for the tool ) data _null_; 33     if 0 then set ec_dim_per nobs=count; 34     call symput('numobs',put(count,8.)); 35   run; 36   %let n=%sysevalf(&numobs/&num,ceil); 37   data %do J=1 %to &num ; DATASET_B_&J  %end; ; 38   set DATASET_B; 39   %do I=1 %to &num; 40   if %eval(&n*(&i-1)) <_n_ <= %eval(&n*&I) 41   then output DATASET_B_&I; 42   %end; 43   run; 44   %mend split; 45   %split (10) NOTE: DATA STEP stopped due to looping. NOTE: DATA statement used (Total process time):       real time           0.04 seconds       cpu time            0.00 seconds NOTE: There were 32498768 observations read from the data set WORK.DATASET_B. NOTE: The data set WORK.DATASET_B_1 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_2 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_3 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_4 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_5 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_6 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_7 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_8 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_9 has 3249877 observations and 31 variables. NOTE: The data set WORK.DATASET_B_10 has 3249875 observations and 31 variables. NOTE: DATA statement used (Total process time):       real time           15:13.22       cpu time            49.68 seconds NOTE: This SAS session is using a registry in WORK.  All changes will be lost at the end of       this session. Another macro (this time made by me )  to run all the proc sort throught the datasets 46   %macro sort; 47   %do i=1 %to 10; 48   proc sort data = DATASET_B_&i; 49   by KEY; 50   run; 51   %end; 52   run; 53   %mend sort; 54   %short NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_1. NOTE: The data set WORK.DATASET_B_1 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           4:19.91       cpu time            17.59 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_2. NOTE: The data set WORK.DATASET_B_2 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           5:08.46       cpu time            19.56 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_3. NOTE: The data set WORK.DATASET_B_3 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           2:29.01       cpu time            19.51 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_4. NOTE: The data set WORK.DATASET_B_4 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           4:03.43       cpu time            17.03 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_5. NOTE: The data set WORK.DATASET_B_5 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           6:03.90       cpu time            18.95 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_6. NOTE: The data set WORK.DATASET_B_6 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           6:44.73       cpu time            19.23 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_7. NOTE: The data set WORK.DATASET_B_7 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           2:48.26       cpu time            18.72 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_8. NOTE: The data set WORK.DATASET_B_8 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           3:36.37       cpu time            20.17 seconds NOTE: There were 3249877 observations read from the data set WORK.DATASET_B_9. NOTE: The data set WORK.DATASET_B_9 has 3249877 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           4:41.83       cpu time            19.00 seconds NOTE: There were 3249875 observations read from the data set WORK.DATASET_B_10. NOTE: The data set WORK.DATASET_B_10 has 3249875 observations and 31 variables. NOTE: PROCEDURE SORT used (Total process time):       real time           3:17.87       cpu time            19.48 seconds the sum of the all sorts procedures are almost 41 min without counting the 15 min for the macro to work. And we still need a merge to build the dataset again (by the way, what do you think is the best way to merge all the datasets?) As you can see split in 10 dataset isn't faster than the original proc sort. If I able to free space I'd try with perhaps 20 small datasets. AmySwinford, I have SAS/ACCESS ready for use in my PC but I never use it and to be honest I don´t know where to start to make it work in local mode (server/cliente in the same machine). I think defragmenting the hard disk is a very good idea , I´d try if my user have admin privileges,... Regards ... View more

Thanks Jaap for this "free of charge lessons" :smileylaugh:. I`d try to increase the amount of memory in the buffer and total too. As for your questions, I coul say - The most of the datases I need the hole set, in case I only need subset it´s already implemented in the code - Is some dataset there is no order at all, in other the order is not the one I need - In fact it is not what I was thinking, but it works is doesn´t matter. What about multiprocessing in local, I'd try to explain myselft. Is posible to run the two short in parallel?. I read about this but I know sure if it could be posible running a local sesion and "emulate" a server on local machine. Thanks ... View more

Good morning I left here a proc sort, this proc sort is one of the bigger. 72   proc sort data = dataset_1= dataset_2; 73   by key; 74   run; NOTA: Se han leído 32498768 observaciones del conj. datos WORK.EC_DIM_PER. NOTA: El conj. datos WORK.PERSORDENADO tiene 32498768 observaciones y 31 variables. NOTA: PROCEDIMIENTO SORT utilizado (Tiempo de proceso total):       tiempo real          45:25.58       tiempo de cpu del usuario       42.85 segundos       tiempo de cpu del sistema     2:40.40       Memoria                            131842k Jaap, I atttach you a proc sort screenshort, I hope it can resolved your questions. Ballardw, Hash hasn`t enough memory for run in my PC. It couldn't load the entire table in memory. Please see my post above. Thanks all for your help ... View more

client installation? Sorry Linush, I don't know what you mean with it. One of the task of my job is to make sure that the data load from the source is the same that the data that is working with so we take the data set from the source, order it, take the data that is loaded in the system, order it and verify if the data are the same, wich data are in the source and not in the system and viceversa. My code is something like this below data a; set b (where= vardate1<= date1 and vardate2<=date2 and vardate3<=date3 and vardate4<=date4); run; data x; set y (where= vardate1<= date1 and vardate2<=date2 and vardate3<=date3 and vardate4<=date4); run; proc sort data = a; order by key; proc sort data = x (rename=(var_f = key)); order by key; data h i j; merge a (in=w) b(in=q); if w then output h; else if g then output i; else output j; run; The problem is that the proc sort takes tooooooooo much time, so I´m trying to replace this piece of the code. A good alternative was to use a hash object but muy PC hasn't enough memory for perform it with success. Regards ... View more

Thanks guys. I increase the amount of memory without any result. I need more physical memory installed in my pc, only  4 gb is very short Only two images to show you the issue. Before (1.jpg) and after (2.jpg) Thanks for your help P:S : A better way to sort a dataset of 40 millions rows instead of Proc report ? Regards ... View more