BookmarkSubscribeRSS Feed
MVK
Calcite | Level 5 MVK
Calcite | Level 5

I have flatfile of size 247 gb .. with help of this flatfile I am creating 8 different datasets which happens to read the sequentially into datasets one after the other .. so overall time thats taking is around 23 hrs to complete my job. As this file exists in unix enviromnent .. intially I thought of using split command and to divide the file  but issue here is its going to take lot of space on the server .. I would really appreciate if anyone could suggest better approch to reduce the time .. I am thinking for a solution where I can reduce time by reading data parallelism type ...

Thanks in advance

Mvk 

1 REPLY 1
Tom
Super User Tom
Super User

Check the admin of your system, they probably will be able to give more practical advice.

Reading that large of a file will take a long time, but writing is most likely the bottleneck. Probably the biggest improvement will be in using as many different physical disks as possible. So replicate your program 8 times with each one creating one of the output files and run it in parallel , but make sure to write to different output disks. 

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 1 reply
  • 351 views
  • 0 likes
  • 2 in conversation