topic Re: resulting data set size becomes much bigger after concatenating two data sets in New SAS User

resulting data set size becomes much bigger after concatenating two data sets

hanfei28 — Tue, 26 Nov 2019 21:29:08 GMT

Hi All,

I used

data lib.A;

set B C;

run;

to concatenate two data sets B and C. Data set B is 80.9GB, and data set C is 78.4GB. But I see resulting data set A is 378 GB. Could any one explain why data set A has much bigger size than sum of data sets B and C?

Thanks.

Re: resulting data set size becomes much bigger after concatenating two data sets

PeterClemmensen — Tue, 26 Nov 2019 21:31:05 GMT

Are one of the two data sets compressed?

Re: resulting data set size becomes much bigger after concatenating two data sets

hanfei28 — Tue, 26 Nov 2019 21:33:54 GMT

Data sets B and C are not compressed.

Re: resulting data set size becomes much bigger after concatenating two data sets

Reeza — Tue, 26 Nov 2019 21:44:55 GMT

Do you get any messages about mismatching lengths or such in your log?

If so, that's likely why. You probably have a character field or multiple ones that are large in one of the data sets. You have mentioned the size but how how many records are in each data?

If you post a proc contents on each data set, the A,B and C, it should be relatively easy to see where the issue is.

Re: resulting data set size becomes much bigger after concatenating two data sets

hanfei28 — Tue, 26 Nov 2019 21:52:22 GMT

I compared the proc contents output for the three sets. They have the same variables names and variable length.

Data B has 198,840,456 rows

Data C has 192,560,707 rows.

Data A has 391,401,163 rows.

Re: resulting data set size becomes much bigger after concatenating two data sets

r_behata — Tue, 26 Nov 2019 21:55:19 GMT

Are Data A and Data B Created on the same Environment or were they copied from a different machine / Server ?

Re: resulting data set size becomes much bigger after concatenating two data sets

hanfei28 — Tue, 26 Nov 2019 21:58:07 GMT

Two contributing data sets, B and C are from the same system(Linux system). The SAS code is run on the same system and data set A is saved to the same directory.

Re: resulting data set size becomes much bigger after concatenating two data sets

r_behata — Tue, 26 Nov 2019 22:10:38 GMT

Check the default compression :

proc options option=compress;
run;

Apply Binary compression , if not the same and try .

options compress=binary;

Re: resulting data set size becomes much bigger after concatenating two data sets

Reeza — Tue, 26 Nov 2019 22:17:31 GMT

@hanfei28 wrote:

I compared the proc contents output for the three sets. They have the same variables names and variable length.

Data B has 198,840,456 rows

Data C has 192,560,707 rows.

Data A has 391,401,163 rows.

If you believe their identical grab the attributes from sashelp.vcolumn and run proc compare to see the difference.

data a_details;
set sashelp.vcolumn;
where libname='WORK' and memname='A';
run;

data b_details;
set sashelp.vcolumn;
where libname='WORK' and memname='B';
run;

proc compare data=a_details compare=b_details;
run;

Re: resulting data set size becomes much bigger after concatenating two data sets

hanfei28 — Tue, 26 Nov 2019 22:19:42 GMT

Here is the output: @r_behata

rsubmit;

NOTE: Remote submit to POE commencing.
proc options option=compress;
run;

SAS (r) Proprietary Software Release 9.4 TS1M4

COMPRESS=NO Specifies the type of compression to use for observations in output SAS data
sets.

Re: resulting data set size becomes much bigger after concatenating two data sets

ballardw — Tue, 26 Nov 2019 22:20:40 GMT

If there are any variables not common to both sets you end up with missing values that will take up space. Each numeric would use 8 byes for each row of the data set not containing the variable.

So when you say you have

Data B has 198,840,456 rows

Data C has 192,560,707 rows.

if you have one variable in B not in C then you potentially have 192560707*8 bytes of storage reserved. Multiply times number of variables in B not in C. If you have a variable in C not in B then 198840456*8 bytes potential disk use.

Long character values can make this potentially many more bytes.

Re: resulting data set size becomes much bigger after concatenating two data sets

Reeza — Tue, 26 Nov 2019 22:20:49 GMT

You can't check just options unfortunately. You can have compression set on data sets, in general via proc options or on a library.

Re: resulting data set size becomes much bigger after concatenating two data sets

Kurt_Bremser — Wed, 27 Nov 2019 09:18:27 GMT

Please post the complete(!) output of proc contents for all three datasets.