topic Re: SAS Compress and Reuse - Risks? in SAS Data Management

SAS Compress and Reuse - Risks?

tedway — Wed, 01 Nov 2017 20:18:59 GMT

I have a dataset on our network drive that is 37 GB in size. Doing a simple count of one variable using proc sql can take over an hour, which makes the dataset unusable.

I read about compress and gave it a try.

data new (compress=Yes reuse=yes);
set old;
run;

The dataset is now 1.5 GB and the same count query takes two minutes (still slow but way better than before).

The only negatives I've been about to find in regards to using compress are that it can make the file slower to access in some cases and apparently you can't address observations by observation number.

Is there any other reason why I shouldn't use these settings by default with larger datasets going forward?

Re: SAS Compress and Reuse - Risks?

Reeza — Wed, 01 Nov 2017 20:42:26 GMT

Have you added indexes to your data set?

Re: SAS Compress and Reuse - Risks?

tedway — Wed, 01 Nov 2017 20:47:10 GMT

I did but I didn't see a performance improvement. I may need to play around with indexing a different variable or multiple variables.

My main concern is finding out later that there is a downside to compressing that I'm not aware of now.

Re: SAS Compress and Reuse - Risks?

Kurt_Bremser — Wed, 01 Nov 2017 21:13:38 GMT

First of all, stop working on network drives. Especially when your network is so slow. Any modern storage reads a GB in less than 10 seconds.

Other than reading by obs number, compressed datasets are unproblematic. Just keep in mind that sometimes using compress=yes can actually increase the size, so keep an eye on your logs.

Re: SAS Compress and Reuse - Risks?

LinusH — Thu, 02 Nov 2017 07:47:37 GMT

Indexes must be designed with the most common queries in mind. And for Base SAS data sets a general rule is they only perform when they return less than 10% of the observations.

Reuse makes sense only when you delete a lot of observations in place.

The main downside with compress is that it requires more CPU cycles in both read and write operations. But in your case it seems that the slow IO pays off the compression.

Options msglevel=i;
gives feedback in the log about both compression and index usage.

Re: SAS Compress and Reuse - Risks?

Kurt_Bremser — Thu, 02 Nov 2017 12:50:10 GMT

An addendum: almost all our production datasets are stored with compress=yes. The cost in CPU cycles for uncompressing a RLE compressed dataset is negligible compared to the savings in space and therefore the savings in I/O. Only datasets where the compression rate is too small or non-existent (actual increase in physical size) are excluded.