the following is test code i am running to try and get a feel for how SPDE works:
libname tst SPDE "c:/test/" partsize=1G;
do i=1 to 4000000;
proc datasets library=tst;
index create i;
do i=1 to 2000000;
proc append base=tst.test1 data=tst.test2;
the first data step produces a file with size 24m
the proc datasets produces an hbx file with size of 133m
the second data step produces a file with size of 11m
the proc append increases the size of the first dataset and also creates an idx file with size of 140m
i am using SPDE to deal with large datasets in the first place and now i'm finding that the hbx/idx files combined are going to require 7 or 8 times the size of my original data?
can someone explain where i am going wrong or if there is a way around these large hbx/idx files?