topic Re: Follow up question on the post "Insufficient space on file" in SAS Programming

Follow up question on the post "Insufficient space on file"

Emma_at_SAS — Wed, 26 Jul 2023 13:58:56 GMT

I have a dataset with thousands of rows and when I tried to make several new datasets step-by-step at some point I received a similar error to the following post https://communities.sas.com/t5/SAS-Enterprise-Guide/Insufficient-Space-in-file/m-p/593266#M34787

ERROR: Insufficient space in file MYLIBRARY.MYTABLE

ERROR: File MYLIBRARY.MYTABLE is damaged. I/O processing did not complete.

@Kurt_Bremser recommended using COMPRESS=YES

I used the COMPRESS command like the following, and I did not get that ERROR message anymore BUT I wanted to make sure that I've applied the COMPRESS command correctly and that I will not miss any important data or space in my new WANT data.

DATA WANT;

SET HAVE (COMPRESS=YES);

RUN;

Thanks

Re: Follow up question on the post "Insufficient space on file"

Tom — Wed, 26 Jul 2023 15:12:22 GMT

The COMPRESS= dataset option can be used to override the value of the COMPRESS option set in the system level COMPRESS option or defined in the LIBNAME statement that defined the libref.

When reading from a dataset SAS will just look at the dataset to tell if it was stored with compression or not. So using COMPRESS= makes no sense as a dataset option on the INPUT dataset.

If you want to use it in that code then add is as a dataset option on the OUTPUT dataset.

Re: Follow up question on the post "Insufficient space on file"

Emma_at_SAS — Wed, 26 Jul 2023 15:29:07 GMT

Thank you @Tom for your thoughts. To check if I am understanding your suggestion

You suggest that it does not make sense to use COMPRESS=YES in the SET statement of a DATA step. At most, I may use COMPRESS=YES at the DATA statement of a data step like

DATA WANT (COMPRESS=YES);

SET HAVE;

RUN;

but if I want to save the WANT dataset for my future permanent dataset then I should avoid using COMPRESS at any point in a data step.

I appreciate it if you may check my understanding.

Thanks

Re: Follow up question on the post "Insufficient space on file"

Kurt_Bremser — Wed, 26 Jul 2023 16:51:38 GMT

Compression will never be an issue in the future use of a dataset, compressed datasets are always readable by SAS.

Re: Follow up question on the post "Insufficient space on file"

Emma_at_SAS — Wed, 26 Jul 2023 17:15:25 GMT

Thank you, @Kurt_Bremser for sharing your insight. Is there a risk that I remove a meaningful SPACE in my character variables if I use COMPRESS=YES in the SET statement of a DATA step?

Thanks so much

Re: Follow up question on the post "Insufficient space on file"

Tom — Wed, 26 Jul 2023 17:27:14 GMT

@Emma_at_SAS wrote:

Thank you, @Kurt_Bremser for sharing your insight. Is there a risk that I remove a meaningful SPACE in my character variables if I use COMPRESS=YES in the SET statement of a DATA step?

Thanks so much

It does not change the values. Just how the file is formatted on the disk.

Re: Follow up question on the post "Insufficient space on file"

Kurt_Bremser — Wed, 26 Jul 2023 18:38:30 GMT

Character variables are always padded with blanks to their defined length. Even if those blanks are compressed, they will be recreated upon reading observations into the PDV.

Almost all permanent datasets at the place I used to work at were compressed, and we never lost data.