Reading big file through FILENAME FTP

Reply
Contributor
Posts: 50

Reading big file through FILENAME FTP

Hi

I am trying to read a big file via FTP (from Mainframe to AIX). Any tips on efficiency so as to improve the speed at which the file is being read?

I introduced a single trailing in INPUT statement to filter the data the data I need before being written to SAS dataset. However it doesn't seem to make much difference in the overall run time. I used compress dataset option. That also didn't help much.

Any tips to fasten the process would be highly appreciated.

Thanks

Valued Guide
Posts: 3,208

Re: Reading big file through FILENAME FTP

Network get as close to the machine as possible direct AIX/Mainframe.

IO mainframe nothing to configure anymore as it is there.   Use a full binary  download when it contains binary fields (zoned packed are also binary).

Check delay is set to off on AIX (should be part of SAS installation)

Storing data on AIX can be a litte optimized by buffers alingSASiofiles  / compress=binary(long records) using SSS etc. All should already be in place.

---->-- ja karman --<-----
Super User
Posts: 7,444

Re: Reading big file through FILENAME FTP

Your bottleneck will be the network. If you have 1G network speed, that translates at best to 60-80 MB / sec, which is way below the read/write speed of the disks, which would be the next point to consider performancewise.

So there's not much you can do.

If you have a large varchar field in the data, move it to the end of the record structure and have the MF file saved in VB format, so you can use the s370vb option in SAS. This will significantly reduce the amount of data transferred over the net, if said varchar field is rarely completely filled. In such a case, also use the compress option in SAS.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Contributor
Posts: 50

Re: Reading big file through FILENAME FTP

I used view in the data step (where I use infile/input) and it seemed to help big time. I created a view with the subsetting conditions and used that view for further processing. The overall run time came down by 50%. Any thoughts/comments on this or improvizing this approach?

Super User
Posts: 7,444

Re: Reading big file through FILENAME FTP

When using the view is faster than creating a dataset, the disk I/O performance seems to be lower than your network speed, which is very unusual, to say the least.

Is the AIX in question a physical server or a LPAR?

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: Reading big file through FILENAME FTP

You are having saved time on writing data your local system. There cannot be anything saved by the ftp process servers side and nework.

When you have this kind of saving on you local system than that performance is not good, (bad). work to do on that.

You platform admin? Your AIX Admin?  Wich SAS release buffering IO speed. Margaret Crevar did update the posting for that level 

---->-- ja karman --<-----
Contributor
Posts: 50

Re: Reading big file through FILENAME FTP

Hi Jaap, I am SAS developer. One of my business partner had this issue. So was trying to help her out.

Valued Guide
Posts: 3,208

Re: Reading big file through FILENAME FTP

ok DeepSiv, so you could verify things as mentioned by Margaret http://blogs.sas.com/content/sgf/2014/10/08/configuring-sas-what-to-know-before-you-install/

for working ftp mainframe to AIX this one is a nice doc: TS-DOC: TS-642 - Reading EBCDIC Files on ASCII Systems

My be they are helpful but tuning and performance is a technical area. You can do some important things as developer for those.       

---->-- ja karman --<-----
Contributor
Posts: 50

Re: Reading big file through FILENAME FTP

Thanks Jaap!

Ask a Question
Discussion stats
  • 8 replies
  • 430 views
  • 3 likes
  • 3 in conversation