Hash
Hi in my code i have created hash objects the data was huge as 15 crores and i need to create the 4 tables ,but i am getting memory issue.where the hash actually stores the files
2.I told that i have created 4 hash tables based on 4 varibale and the variable length
is 100 i have give and the data in this variables will use 12 onlye
ex: attrib x1 length 100$
the data that was used max was up to 12 only Ex:x1=hihru1234545
In this case the hash will creat the table for 100 lenght or for the length the variable that exactly used(12).
If i reduce the length of variable used in the hash tables the memory will increase
150 million rows of data will require a decent about of memory to store the hash object.
1. Make sure your system has enough memory available total to even store the object.
2. Make sure you setup the sessions memory settings appropirately so that SAS will have that memory available to it.
As far as I know the extranious length of a assigned variable does not affect the amount of space required to store it in memory for a hash object. See the following as proof:
16 data foo;
17 length bar $100;
18 set sashelp.class end=done;
19 orig_age=age;
20 do i=1 to 500000;
21 bar=cats(name,put(i,6.));
22 age=orig_age+rand('table',.25,.25,.25,.25);
23 output;
24 end;
25 keep bar age;
26 run;
NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: The data set WORK.FOO has 9500000 observations and 2 variables.
NOTE: DATA statement used (Total process time):
real time 4.83 seconds
user cpu time 4.06 seconds
system cpu time 0.75 seconds
Memory 2493978k
OS Memory 2499944k
27
28 data age10(where=(10<=age<15)) age15(where=(15<=age<20));
29 if 0 then set foo;
30 declare hash ha(dataset:'foo');
31 declare hiter iter('ha');
32 ha.definekey('bar');
33 ha.definedata(all:'y');
34 ha.definedone();
35 iter.first();
36 do while(iter.next()=0);
37 output;
38 end;
39 stop;
40 run;
NOTE: There were 9500000 observations read from the data set WORK.FOO.
NOTE: The data set WORK.AGE10 has 2374832 observations and 2 variables.
NOTE: The data set WORK.AGE15 has 7000595 observations and 2 variables.
NOTE: DATA statement used (Total process time):
real time 13.27 seconds
user cpu time 11.93 seconds
system cpu time 1.32 seconds
Memory 2493978k
OS Memory 2499944k
16 data foo;
17 length bar $13;
18 set sashelp.class;
19 orig_age=age;
20 do i=1 to 500000;
21 bar=cats(name,put(i,6.));
22 age=orig_age+rand('table',.25,.25,.25,.25);
23 output;
24 end;
25 keep bar age;
26 run;
NOTE: There were 19 observations read from the data set SASHELP.CLASS.
NOTE: The data set WORK.FOO has 9500000 observations and 2 variables.
NOTE: DATA statement used (Total process time):
real time 5.71 seconds
user cpu time 4.08 seconds
system cpu time 1.56 seconds
Memory 2493978k
OS Memory 2499944k
27
28 data age10(where=(10<=age<15)) age15(where=(15<=age<20));
29 if 0 then set foo;
30 declare hash ha(dataset:'foo');
31 declare hiter iter('ha');
32 ha.definekey('bar');
33 ha.definedata(all:'y');
34 ha.definedone();
35 iter.first();
36 do while(iter.next()=0);
37 output;
38 end;
39 stop;
40 run;
NOTE: There were 9500000 observations read from the data set WORK.FOO.
NOTE: The data set WORK.AGE10 has 2373910 observations and 2 variables.
NOTE: The data set WORK.AGE15 has 7000782 observations and 2 variables.
NOTE: DATA statement used (Total process time):
real time 13.78 seconds
user cpu time 11.93 seconds
system cpu time 1.82 seconds
Memory 2493978k
OS Memory 2499944k
Available on demand!
Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.