SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

general question about SAS datasets-expansion/compression during set creation

Reply
Frequent Contributor
Frequent Contributor
Posts: 133

general question about SAS datasets-expansion/compression during set creation

Hello all-

 

This is a technical question regarding to what occurs during a set creation-

If you are already creating a new dataset from an old dataset and you have some added variables or some other changes can the dataset sort of balloon or expand during its creation and then upon it finishing-then applies compression (if option is set) ?  

 

 

And can this happen during a data step and/or sql procedure?

The reason I ask is that last week-someone's data step suddenly hit the space constraints on a unix server.

Although the dataset was only 20 gigs with 30 gigs of space left it was pretty evident it was the activity that led to some further space constraint issues.

 

Lawrence

 

Lawrence

Super User
Posts: 3,115

Re: general question about SAS datasets-expansion/compression during set creation

When SAS is creating a new table the same name as an existing table, there will be two versions of this table kept until SAS successfully finishes creating the new version. The reason for this is so that SAS does not over-write the existing table if there is an error. Once the step is successful the old version table is deleted and the new version table is renamed to be the same as the old version one.

 

To summarise you need enough disk space to store two versions of the same table. 

Frequent Contributor
Frequent Contributor
Posts: 133

Re: general question about SAS datasets-expansion/compression during set creation

SASkiwi-

That is useful to know-

what if you are creating a new dataset from a perm dataset?

 

Lawrence

Super User
Posts: 3,115

Re: general question about SAS datasets-expansion/compression during set creation

Then there wouldn't be two versions of the new table. However as soon as you try to overwrite the new table by running the same program twice, two versions will be kept. This applies for both permanent and temporary tables.

Super User
Posts: 6,971

Re: general question about SAS datasets-expansion/compression during set creation

Some operations may even need more space.

eg sorting a data set will need triple space: the original file, the intermediate (utility) file and the new file (the utility file is either in WORK or where UTILLOC points to)

If the dataset is compressed, this will not affect the utility file, so you may need eben more space than just triple the original filesize.

SQL operations also may create huge utility files, depending on the type of operation.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Ask a Question
Discussion stats
  • 4 replies
  • 326 views
  • 0 likes
  • 3 in conversation