topic Re: How do I split one text file into multiple text file according to file size limit in New SAS User

How do I split one text file into multiple text file according to file size limit

paparock — Wed, 16 Nov 2022 03:46:55 GMT

Hi... need assistance on how to split one text file to multiple text file to follow certain file size limit (eg. not exceeding 30MB/file).

any help rendered is much appreciated. TQ

Re: How do I split one text file into multiple text file according to file size limit

fja — Wed, 16 Nov 2022 08:25:37 GMT

Hello!
Do you need to do that in SAS? This is more conveniently done on OS level.
--FJa

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Wed, 16 Nov 2022 11:07:29 GMT

Do you need to repeat a header line in all files?

How should the filenames for the split files be built?

Re: How do I split one text file into multiple text file according to file size limit

andreas_lds — Wed, 16 Nov 2022 11:09:50 GMT

Do you have to solve the problem using sas?

Re: How do I split one text file into multiple text file according to file size limit

paparock — Thu, 17 Nov 2022 01:34:10 GMT

Yes. In SAS pls... TQ

Re: How do I split one text file into multiple text file according to file size limit

paparock — Thu, 17 Nov 2022 01:34:59 GMT

Yes. If doable, in SAS pls. TQ

Re: How do I split one text file into multiple text file according to file size limit

paparock — Thu, 17 Nov 2022 01:37:32 GMT

- currently the file that i'm working with doesn't have any header. Just need to split the text file so that it doesn't exceed the 30MB/file.
- filenames eg... filenameV1, filenamev2, etc
TQ

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Thu, 17 Nov 2022 08:12:38 GMT

See this code, successfully tested on SAS On Demand:

/* let's make a text file */

%let textfile = ~/test;

data _null_;
file "&textfile.";
do _n_ = 1 to 10000;
  do i = 1 to nobs;
    set sashelp.class point=i nobs=nobs;
    put _all_;
  end;
end;
stop;
run;

/* now, split it */

%let chunk = %eval(1024*1024); * size of individual output files, 1MB here;

%let outfile = ~/testv; * base name of output;

data _null_;
retain count 1 size 0;
length fname $200;
fname = cats("&outfile.",put(count,z3.)); * use a z. format to keep proper order of files;
infile "&textfile.";
file dummy filevar=fname; * filevar= option creates dynamic filenames;
input;
put _infile_;
size + length(_infile_);
if size > &chunk.
then do;
  count + 1;
  size = 0;
end;
run;

Re: How do I split one text file into multiple text file according to file size limit

fja — Thu, 17 Nov 2022 08:33:14 GMT

Again, let me draw your attention to the OS. In Unix it is a one line command:
split --bytes=30M filename

Re: How do I split one text file into multiple text file according to file size limit

fja — Thu, 17 Nov 2022 08:35:37 GMT

Does Kurt's reply solve your Problem?

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Thu, 17 Nov 2022 08:36:51 GMT

@fja wrote:
Again, let me draw your attention to the OS. In Unix it is a one line command:
split --bytes=30M filename

which one can run in a data step:

data _null_;
infile pipe "split --bytes=30M &filename. 2>&1";
input;
put _infile_;
run;

so that all diagnostics are written to the SAS log.

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Thu, 17 Nov 2022 08:40:33 GMT

PS my code is just there to show how to tackle such an issue in SAS. See it as a thought experiment and an opportunity to learn SAS coding options which are not that widely known.

Using the proper tool (Maxim 14), as @fja shows, is the proper way to go. The UNIX command will be much faster in execution, and requires a s..tload less coding.

Re: How do I split one text file into multiple text file according to file size limit

paparock — Thu, 17 Nov 2022 08:55:47 GMT

Tq Sir for your reply.
I'm still new user to SAS, if my file is in drive d:\test\abcd.csv, where to i put the command for this specific file? TQ

Re: How do I split one text file into multiple text file according to file size limit

fja — Thu, 17 Nov 2022 09:01:55 GMT

which one can run in a data step:

data _null_;
infile pipe "split --bytes=30M &filename. 2>&1";
input;
put _infile_;
run;

Awesome! I didn't think of that one ... But you are absolutely right!

Re: How do I split one text file into multiple text file according to file size limit

paparock — Thu, 17 Nov 2022 09:20:56 GMT

Sorry Sir, one more question.
Can the code run on desktop version? TQ for any feedback

Re: How do I split one text file into multiple text file according to file size limit

paparock — Thu, 17 Nov 2022 09:45:20 GMT

Hi jfa,
Still trying but able to obtain the result. TQ

Re: How do I split one text file into multiple text file according to file size limit

Ksharp — Thu, 17 Nov 2022 12:15:47 GMT

%let file= c:\temp\have.txt  ;     /*the original big text file*/
%let output=  c:\temp\   ;    /*the path you want to output*/
%let block=1  ;   /*Here is 1M for a file,if you need 30M for a file,specify 30*/


data _null_;
rc=filename('x',"&file.");
fid=fopen('x');
do i=1 to foptnum(fid);
  option=foptname(fid,i);
  value=finfo(fid,option);
  if i=4 then do;   
    size=input(value,best32.); /*the size of this file*/
    n= ceil(size/1024/1000/&block.);   
    call symputx('n',n);  /*put n into a macro variable*/
  end;
end;
run;

/*get the number of records of the splitted text file*/
data _null_;
 infile "&file." end=last;
 input;
 if last then do;
   length split $ 32000;
   f="%scan(&file.,-2,.\)";
   s="%scan(&file.,-1,.\)";
   size=ceil(_n_/&n.);
   do i=1 to &n.;
     if i=1 then split=catt("if _n_<",size," then fname='&output.\",f,"_",i,".",s,"';");
       else split=catt(split,"else if _n_<",i*size," then fname='&output.\",f,"_",i,".",s,"';");
   end;
   call symputx('split',split);
 end;
run;
%put &=n;
%put %bquote(&split) ;


/*start to split */
data _null_;
length fname $ 200;
&split.
infile "&file." lrecl=32767 ;
file dummy filevar=fname ;
input;
put _infile_;
run;

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Thu, 17 Nov 2022 13:16:24 GMT

@paparock wrote:
Tq Sir for your reply.
I'm still new user to SAS, if my file is in drive d:\test\abcd.csv, where to i put the command for this specific file? TQ

You only need to adapt the TEXTFILE and OUTFILE macro variables to your needs.

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Thu, 17 Nov 2022 13:21:08 GMT

This method was part of my presentation at WUSS 2022.

Re: How do I split one text file into multiple text file according to file size limit

Kurt_Bremser — Thu, 17 Nov 2022 13:23:06 GMT

@paparock wrote:
Sorry Sir, one more question.
Can the code run on desktop version? TQ for any feedback

Of course. The main issue: the file paths must be available to your SAS process.