topic reading data from huge flatfile in SAS Programming https://communities.sas.com/t5/SAS-Programming/reading-data-from-huge-flatfile/m-p/141302#M261538 <HTML><HEAD></HEAD><BODY><P>I have flatfile of size 247 gb .. with help of this flatfile I am creating 8 different datasets which happens to read the sequentially into datasets one after the other .. so overall time thats taking is around 23 hrs to complete my job. As this file exists in unix enviromnent .. intially I thought of using split command and to divide the file&nbsp; but issue here is its going to take lot of space on the server .. I would really appreciate if anyone could suggest better approch to reduce the time .. I am thinking for a solution where I can reduce time by reading data parallelism type ...</P><P>Thanks in advance</P><P>Mvk&nbsp; </P></BODY></HTML> Wed, 12 Mar 2014 17:56:36 GMT MVK 2014-03-12T17:56:36Z reading data from huge flatfile https://communities.sas.com/t5/SAS-Programming/reading-data-from-huge-flatfile/m-p/141302#M261538 <HTML><HEAD></HEAD><BODY><P>I have flatfile of size 247 gb .. with help of this flatfile I am creating 8 different datasets which happens to read the sequentially into datasets one after the other .. so overall time thats taking is around 23 hrs to complete my job. As this file exists in unix enviromnent .. intially I thought of using split command and to divide the file&nbsp; but issue here is its going to take lot of space on the server .. I would really appreciate if anyone could suggest better approch to reduce the time .. I am thinking for a solution where I can reduce time by reading data parallelism type ...</P><P>Thanks in advance</P><P>Mvk&nbsp; </P></BODY></HTML> Wed, 12 Mar 2014 17:56:36 GMT https://communities.sas.com/t5/SAS-Programming/reading-data-from-huge-flatfile/m-p/141302#M261538 MVK 2014-03-12T17:56:36Z Re: reading data from huge flatfile https://communities.sas.com/t5/SAS-Programming/reading-data-from-huge-flatfile/m-p/141303#M261539 <HTML><HEAD></HEAD><BODY><P>Check the admin of your system, they probably will be able to give more practical advice.</P><P></P><P>Reading that large of a file will take a long time, but writing is most likely the bottleneck. Probably the biggest improvement will be in using as many different physical disks as possible. <SPAN style="font-size: 10pt; line-height: 1.5em;">So replicate your program 8 times with each one creating one of the output files and run it in parallel , but make sure to write to different output disks.&nbsp; </SPAN></P></BODY></HTML> Wed, 12 Mar 2014 18:18:44 GMT https://communities.sas.com/t5/SAS-Programming/reading-data-from-huge-flatfile/m-p/141303#M261539 Tom 2014-03-12T18:18:44Z