topic Re: Break Up Large File into Multiple Small Files in SAS Programming

Break Up Large File into Multiple Small Files

deleted_user — Tue, 28 Jul 2009 20:13:24 GMT

I have a large input file that I need to break up into smaller files of 250 records. The size of the input file changes so the number of observations is never the same. Can I alter my code below to create multiple out files of 250 records?

Thanks

//S02 EXEC SAS
//WORK DD SPACE=(CYL,(150,150))
//IN1 DD DSN=DC.XXXXX.ORS.CONV.WORK1(+1),
// DISP=SHR
//*OUT DD SYSOUT=*
//OUT DD DSN=DC.XXXXX.ORS.CONV.WORK2(+1),
// DISP=(,CATLG,DELETE),
// UNIT=SYSDA,
// LRECL=80,
// BLKSIZE=6160,
// RECFM=FB,
// SPACE=(TRK,(15,15),RLSE)
//SYSIN DD *
DATA RECORD;
INFILE IN1 FIRSTOBS=1 OBS=250;
INPUT @39 VOLS $6.;
FILE OUT;
IDVAR+1;
PUT @1 '//DD'IDVAR HEX2.' DD UNIT=3390,DISP=SHR,VOL=SER='VOLS +(-1) /
@3 'CONVERTV SMS REDET ALLMULTI DDN(DD'IDVAR HEX2. ')';

Re: Break Up Large File into Multiple Small Files

sbb — Tue, 28 Jul 2009 20:41:23 GMT

Aside from a SAS-based solution, if you are licensed for DFSORT, there is a SPLIT function (part of OUTFIL coding, also, it uses the SPLITBY=nnn) to perform this function. No hassles with DCB= coding - very effective and can run on any LPAR (if you have a SAS-licensed LPAR arrangement).

Scott Barry
SBBWorks, Inc.

Re: Break Up Large File into Multiple Small Files

deleted_user — Wed, 29 Jul 2009 14:43:49 GMT

Thanks Scott
I was playing a little with _n_ and thought that might be a solution but will look into yours.

Re: Break Up Large File into Multiple Small Files

sbb — Wed, 29 Jul 2009 16:38:18 GMT

Your post indicates that your input file (a plus 1 generation - maybe it's created earlier in the same job?) -- but you are attempting to generate DD statements from your SAS program *AND* in the same job/step (SAS DATA step) you are showing an INFILE / FILE combination to filter in some increments (one input file to "n" output files).

Suggest you review your SAS program processing in detail, as you have the "cart" before the "horse" -- you will need to investigate using FILENAME to dynamically allocate your output file from within your SAS program -- or if you must generate the JCL, you will need to have two jobs, one to generate the JCL (and that job will get submitted for execution), and the second "generated" jobstream will perform the data split.

However, given this thread, I still recommend you consider DFSORT (or a suitable alternative with your site's SORT package).

Scott Barry
SBBWorks, Inc.

Re: Break Up Large File into Multiple Small Files

sbb — Wed, 29 Jul 2009 16:39:59 GMT

Sorry - I also wanted to mention that you can avoid the DCB attributes altogether with SAS by using the JFCB= parameter on the FILE statement in your DATA step.

Scott Barry
SBBWorks, Inc.

Re: Break Up Large File into Multiple Small Files

Kwok — Fri, 11 Sep 2009 16:51:25 GMT

%LET OBS=1009 ;
%MACRO TEST;
%LET I=1;
%DO %UNTIL(%EVAL(&POBS <= 0));
%LET ENDP= %EVAL(&I * 250) ;
%LET STARTP =%EVAL(&ENDP - 250 + 1);
%LET POBS =%EVAL(&OBS - &ENDP) ;
%IF %EVAL(&POBS <= 0) %THEN %DO;
%LET ENDP=&OBS ;
%LET POBS =0;
%END;
%LET FILE=FILE&I ;
%LET REC=%EVAL(&ENDP - &STARTP +1);
%LET I = %EVAL(&I + 1) ;

%PUT FILE_NO= &FILE RECORD=&REC START= &STARTP END= &ENDP REMAINDER= &POBS ;
%END;
%MEND;
%TEST;

Just change the value of %LET OBS=1009 for different size file.

Hope this help.

Re: Break Up Large File into Multiple Small Files

Kwok — Fri, 11 Sep 2009 16:57:20 GMT

Sorry, for some reason the last post was truncated. Do you know how to increase the size of the page ?

Re: Break Up Large File into Multiple Small Files

sbb — Fri, 11 Sep 2009 17:12:15 GMT

This continuing problem exists with the software used to host the SAS Discussion Forums. You will need to find some alternate technique to post some code, possibly leaving out or masking "special" characters, unfortunately.

Scott Barry
SBBWorks, Inc.