BookmarkSubscribeRSS Feed
DeepSiv
Fluorite | Level 6

Hi

I am trying to read a big file via FTP (from Mainframe to AIX). Any tips on efficiency so as to improve the speed at which the file is being read?

I introduced a single trailing in INPUT statement to filter the data the data I need before being written to SAS dataset. However it doesn't seem to make much difference in the overall run time. I used compress dataset option. That also didn't help much.

Any tips to fasten the process would be highly appreciated.

Thanks

8 REPLIES 8
jakarman
Barite | Level 11

Network get as close to the machine as possible direct AIX/Mainframe.

IO mainframe nothing to configure anymore as it is there.   Use a full binary  download when it contains binary fields (zoned packed are also binary).

Check delay is set to off on AIX (should be part of SAS installation)

Storing data on AIX can be a litte optimized by buffers alingSASiofiles  / compress=binary(long records) using SSS etc. All should already be in place.

---->-- ja karman --<-----
Kurt_Bremser
Super User

Your bottleneck will be the network. If you have 1G network speed, that translates at best to 60-80 MB / sec, which is way below the read/write speed of the disks, which would be the next point to consider performancewise.

So there's not much you can do.

If you have a large varchar field in the data, move it to the end of the record structure and have the MF file saved in VB format, so you can use the s370vb option in SAS. This will significantly reduce the amount of data transferred over the net, if said varchar field is rarely completely filled. In such a case, also use the compress option in SAS.

DeepSiv
Fluorite | Level 6

I used view in the data step (where I use infile/input) and it seemed to help big time. I created a view with the subsetting conditions and used that view for further processing. The overall run time came down by 50%. Any thoughts/comments on this or improvizing this approach?

Kurt_Bremser
Super User

When using the view is faster than creating a dataset, the disk I/O performance seems to be lower than your network speed, which is very unusual, to say the least.

Is the AIX in question a physical server or a LPAR?

jakarman
Barite | Level 11

You are having saved time on writing data your local system. There cannot be anything saved by the ftp process servers side and nework.

When you have this kind of saving on you local system than that performance is not good, (bad). work to do on that.

You platform admin? Your AIX Admin?  Wich SAS release buffering IO speed. Margaret Crevar did update the posting for that level 

---->-- ja karman --<-----
DeepSiv
Fluorite | Level 6

Hi Jaap, I am SAS developer. One of my business partner had this issue. So was trying to help her out.

jakarman
Barite | Level 11

ok DeepSiv, so you could verify things as mentioned by Margaret http://blogs.sas.com/content/sgf/2014/10/08/configuring-sas-what-to-know-before-you-install/

for working ftp mainframe to AIX this one is a nice doc: TS-DOC: TS-642 - Reading EBCDIC Files on ASCII Systems

My be they are helpful but tuning and performance is a technical area. You can do some important things as developer for those.       

---->-- ja karman --<-----
DeepSiv
Fluorite | Level 6

Thanks Jaap!

Ready to join fellow brilliant minds for the SAS Hackathon?

Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.

Register today!
How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 8 replies
  • 1218 views
  • 3 likes
  • 3 in conversation