Hash Memory used the varible length kept are the data having in that variables

sas_ — Fri, 27 Jan 2012 06:33:34 GMT

Hash

Hi in my code i have created hash objects the data was huge as 15 crores and i need to create the 4 tables ,but i am getting memory issue.where the hash actually stores the files
2.I told that i have created 4 hash tables based on 4 varibale and the variable length
is 100 i have give and the data in this variables will use 12 onlye
ex: attrib x1 length 100$

the data that was used max was up to 12 only Ex:x1=hihru1234545

In this case the hash will creat the table for 100 lenght or for the length the variable that exactly used(12).

If i reduce the length of variable used in the hash tables the memory will increase

Hash Memory used the varible length kept are the data having in that variables

FriedEgg — Tue, 31 Jan 2012 21:44:03 GMT

150 million rows of data will require a decent about of memory to store the hash object.

1. Make sure your system has enough memory available total to even store the object.

2. Make sure you setup the sessions memory settings appropirately so that SAS will have that memory available to it.

As far as I know the extranious length of a assigned variable does not affect the amount of space required to store it in memory for a hash object. See the following as proof:

16 data foo;

17 length bar $100;

18 set sashelp.class end=done;

19 orig_age=age;

20 do i=1 to 500000;

21 bar=cats(name,put(i,6.));

22 age=orig_age+rand('table',.25,.25,.25,.25);

23 output;

24 end;

25 keep bar age;

26 run;

NOTE: There were 19 observations read from the data set SASHELP.CLASS.

NOTE: The data set WORK.FOO has 9500000 observations and 2 variables.

NOTE: DATA statement used (Total process time):

real time 4.83 seconds

user cpu time 4.06 seconds

system cpu time 0.75 seconds

Memory 2493978k

OS Memory 2499944k

28 data age10(where=(10<=age<15)) age15(where=(15<=age<20));

29 if 0 then set foo;

30 declare hash ha(dataset:'foo');

31 declare hiter iter('ha');

32 ha.definekey('bar');

33 ha.definedata(all:'y');

34 ha.definedone();

35 iter.first();

36 do while(iter.next()=0);

37 output;

38 end;

39 stop;

40 run;

NOTE: There were 9500000 observations read from the data set WORK.FOO.

NOTE: The data set WORK.AGE10 has 2374832 observations and 2 variables.

NOTE: The data set WORK.AGE15 has 7000595 observations and 2 variables.

NOTE: DATA statement used (Total process time):

real time 13.27 seconds

user cpu time 11.93 seconds

system cpu time 1.32 seconds

Memory 2493978k

OS Memory 2499944k

16 data foo;

17 length bar $13;

18 set sashelp.class;

19 orig_age=age;

20 do i=1 to 500000;

21 bar=cats(name,put(i,6.));

22 age=orig_age+rand('table',.25,.25,.25,.25);

23 output;

24 end;

25 keep bar age;

26 run;

NOTE: There were 19 observations read from the data set SASHELP.CLASS.

NOTE: The data set WORK.FOO has 9500000 observations and 2 variables.

NOTE: DATA statement used (Total process time):

real time 5.71 seconds

user cpu time 4.08 seconds

system cpu time 1.56 seconds

Memory 2493978k

OS Memory 2499944k

28 data age10(where=(10<=age<15)) age15(where=(15<=age<20));

29 if 0 then set foo;

30 declare hash ha(dataset:'foo');

31 declare hiter iter('ha');

32 ha.definekey('bar');

33 ha.definedata(all:'y');

34 ha.definedone();

35 iter.first();

36 do while(iter.next()=0);

37 output;

38 end;

39 stop;

40 run;

NOTE: There were 9500000 observations read from the data set WORK.FOO.

NOTE: The data set WORK.AGE10 has 2373910 observations and 2 variables.

NOTE: The data set WORK.AGE15 has 7000782 observations and 2 variables.

NOTE: DATA statement used (Total process time):

real time 13.78 seconds

user cpu time 11.93 seconds

system cpu time 1.82 seconds

Memory 2493978k

OS Memory 2499944k

topic Hash Memory used the varible length kept are the data having in that variables in SAS Programming

Hash Memory used the varible length kept are the data having in that variables

Hash Memory used the varible length kept are the data having in that variables