10-09-2014 12:01 PM
I am trying to read a big file via FTP (from Mainframe to AIX). Any tips on efficiency so as to improve the speed at which the file is being read?
I introduced a single trailing in INPUT statement to filter the data the data I need before being written to SAS dataset. However it doesn't seem to make much difference in the overall run time. I used compress dataset option. That also didn't help much.
Any tips to fasten the process would be highly appreciated.
10-10-2014 05:13 AM
Network get as close to the machine as possible direct AIX/Mainframe.
IO mainframe nothing to configure anymore as it is there. Use a full binary download when it contains binary fields (zoned packed are also binary).
Check delay is set to off on AIX (should be part of SAS installation)
Storing data on AIX can be a litte optimized by buffers alingSASiofiles / compress=binary(long records) using SSS etc. All should already be in place.
10-10-2014 07:53 AM
Your bottleneck will be the network. If you have 1G network speed, that translates at best to 60-80 MB / sec, which is way below the read/write speed of the disks, which would be the next point to consider performancewise.
So there's not much you can do.
If you have a large varchar field in the data, move it to the end of the record structure and have the MF file saved in VB format, so you can use the s370vb option in SAS. This will significantly reduce the amount of data transferred over the net, if said varchar field is rarely completely filled. In such a case, also use the compress option in SAS.
10-10-2014 09:39 AM
I used view in the data step (where I use infile/input) and it seemed to help big time. I created a view with the subsetting conditions and used that view for further processing. The overall run time came down by 50%. Any thoughts/comments on this or improvizing this approach?
10-13-2014 02:04 AM
When using the view is faster than creating a dataset, the disk I/O performance seems to be lower than your network speed, which is very unusual, to say the least.
Is the AIX in question a physical server or a LPAR?
10-10-2014 09:45 AM
You are having saved time on writing data your local system. There cannot be anything saved by the ftp process servers side and nework.
When you have this kind of saving on you local system than that performance is not good, (bad). work to do on that.
You platform admin? Your AIX Admin? Wich SAS release buffering IO speed. Margaret Crevar did update the posting for that level
10-10-2014 02:57 PM
ok DeepSiv, so you could verify things as mentioned by Margaret http://blogs.sas.com/content/sgf/2014/10/08/configuring-sas-what-to-know-before-you-install/
for working ftp mainframe to AIX this one is a nice doc: TS-DOC: TS-642 - Reading EBCDIC Files on ASCII Systems
My be they are helpful but tuning and performance is a technical area. You can do some important things as developer for those.