topic Re: FTP transfer issues -corrupt files in SAS Programming

FTP transfer issues -corrupt files

LB — Fri, 10 Feb 2017 20:51:11 GMT

So I have a FTP script that takes a file from my home folder on one linux server to another.

This file is used as a credential file for SAMBA. This will be used as part of a larger program for other developers to shortcut having to manually ftp this file to different servers.

When the file transfers through FTP it gets corrupted. This is verified by manually downloading the file and reloading it through a FTP client (reflections FTP)

This is the second time this has happened in that the same thing happened to excel files even when the binary option is used.

Anyone know why this might occur-

Here is the script-It works just fine in terms of moving the data.

filename linhome "P:\cs.txt";
filename srvrdest ftp "/home/&sysuserid/cs.txt"
user=&sysuserid host="&serv" cd="/home/&sysuserid" pass="&RITOPASS" binary debug;
data _null_;
infile linhome;
file srvrdest;
input;
put _infile_;
run;
filename linhome clear;
filename srvrdest clear;

Re: FTP transfer issues -corrupt files

LB — Fri, 10 Feb 2017 20:57:47 GMT

If I run this-

filename COPY PIPE "scp /home/bxxx/cs.txt bxxx@xxxx8x.ox.kx.org:/home/bxxx/cs.txt";
data _null_ ;
infile COPY;
input ;
put _infile_;
run;

I get this error-

(gnome-ssh-askpass:72206): Gtk-WARNING **: cannot open display: localhost:10.0
Host key verification failed.
lost connection

Re: FTP transfer issues -corrupt files

Kurt_Bremser — Fri, 10 Feb 2017 21:04:15 GMT

scp will ask for credentials if you do not have public/private key authentication enabled. Trying to prompt for a password probably causes your error.

@LB wrote:

If I run this-

filename COPY PIPE "scp /home/bxxx/cs.txt bxxx@xxxx8x.ox.kx.org:/home/bxxx/cs.txt";
data _null_ ;
infile COPY;
input ;
put _infile_;
run;

I get this error-

(gnome-ssh-askpass:72206): Gtk-WARNING **: cannot open display: localhost:10.0
Host key verification failed.
lost connection

Re: FTP transfer issues -corrupt files

Kurt_Bremser — Fri, 10 Feb 2017 21:07:31 GMT

@LB wrote:

So I have a FTP script that takes a file from my home folder on one linux server to another.

This file is used as a credential file for SAMBA. This will be used as part of a larger program for other developers to shortcut having to manually ftp this file to different servers.

When the file transfers through FTP it gets corrupted. This is verified by manually downloading the file and reloading it through a FTP client (reflections FTP)

This is the second time this has happened in that the same thing happened to excel files even when the binary option is used.

Anyone know why this might occur-

Here is the script-It works just fine in terms of moving the data.

filename linhome "P:\cs.txt";
filename srvrdest ftp "/home/&sysuserid/cs.txt"
user=&sysuserid host="&serv" cd="/home/&sysuserid" pass="&RITOPASS" binary debug;
data _null_;
infile linhome;
file srvrdest;
input;
put _infile_;
run;
filename linhome clear;
filename srvrdest clear;

Could it be that the binary option prevents conversion of CRLF to LF?

Re: FTP transfer issues -corrupt files

LB — Mon, 13 Feb 2017 21:02:01 GMT

That I don't know.

But FTP is still being problematic-

I am trying scp w/o much success.

Lawrence

Re: FTP transfer issues -corrupt files

LB — Mon, 13 Feb 2017 22:00:40 GMT

Is there is a SCP solution-

I have tried different variants here-

filename COPY PIPE "scp file='/home/b335489/cs.txt' todir='xxx9@xxx.xx.x.org:/home/b335489/'" ;
data _null_ ;
infile COPY;
input ;
put _infile_;
run;

AND

filename COPY PIPE "scp file='/home/b335489/cs.txt' todir='xx: password@xxx.xx.x.org:/home/b335489/'" ;

The first fails do to authentication and the second fails as it cannot resolve the host.

Re: FTP transfer issues -corrupt files

Kurt_Bremser — Mon, 13 Feb 2017 22:58:56 GMT

When using SSH (the protocol used by scp) you need to set up public/private key authentication to prevent being prompted for passwords. Once the scp command works from a commandline, you can start using it from SAS.

Re: FTP transfer issues -corrupt files

LB — Mon, 13 Feb 2017 23:13:26 GMT

IS there a way to actually input the password or do I need to set up the SSH keys?

This is being set up in a macro so that the user inserts their password in a macro that is carried through-It is run once to copy a file-all to use SAMBA. I would just FTP the file-but it, like my excel files gets corrupted in transit.

I don't think setting up the public private keys will work in this case.

Thanks

Lawrence

Re: FTP transfer issues -corrupt files

Kurt_Bremser — Mon, 13 Feb 2017 23:20:43 GMT

Software with a respect for security will NOT let you supply passwords on the commandline or other means that can be sniffed (a simple ps -ef would reveal the password). So set up public/private keys for SSH.

And could you please show an example how a file is corrupted by FTP?

Re: FTP transfer issues -corrupt files

LB — Mon, 13 Feb 2017 23:46:43 GMT

Good to know- It makes sense to avoid what could be an inherent error on a users part-

For the excel files-it just gives an error that the file is corrupted and cannot be opened.

can't replicate it per se as it takes too much time to do so.

But the text file is different-

I have a simple script that outputs a file on the local unix home folder with the credentials for SAMBA

data _null_;
file "/home/&sysuserid/cs.txt";

put "username = &SYSUSERID";
put "password = &WINPASS";
put "domain = CS";

run;

The FTP script that does work ports it to the other server-when I download the file it looks the same-however something happens in transit. If I run the program through the management console as a batch job on the first server-the SAMBA credential file works.

However when I attempt to use it on the second server with the same process it fails.

I know the FTP is the issue as if I copy the file down through a FTP client and then reload it into the second, the SAMBA credential file now works. It is an ascii file so I really don't understand the issue.

Thanks for your help. I know SAS well but not Unix/linux.

Lawrence

Re: FTP transfer issues -corrupt files

Kurt_Bremser — Tue, 14 Feb 2017 05:25:03 GMT

More advanced text editors (ie notepad++) will tell you the text file format (DOS or UNIX) in the status line.

Or allow you to view the file in hexadecimal mode, so you can find differences that are not visible otherwise.

Re: FTP transfer issues -corrupt files

LB — Tue, 14 Feb 2017 22:12:33 GMT

Ok-thanks.

Lawrence