SFTP

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 7
Accepted Solution

SFTP

Hello all,

I'm running in circles over here and finally decided to ask for help Smiley Happy. Hopefully someone will know the answer Smiley Happy

I'm trying to pull some files  (it will be a daily process) but without success... I have the line of code below inserted in a user written code in a DIS job :

x "sftp -oIdentityFile=~/bronto_private_key_openssh bronto@ramjet.vcint.com:/*.zip /sas/corporate-dev/Files/Bronto/.";

The code is working perfectly in unix  so i know that the private key was correctly built and that we have all the permissions to access the files but through DIS or EG it doesn't work and there's no error.

log:

122        x "sftp -oIdentityFile=~/bronto_private_key_openssh bronto@ramjet.vcint.com:/*20140425.csv.zip

122      ! /sas/corporate/Files/Bronto/.";

123       

I'm using SAS Data Integration Studio 4.5 .

Any idea of what the problem can be?

Thanks in advance.

Susana


Accepted Solutions
Solution
‎05-16-2014 11:15 AM
Super User
Posts: 6,938

Re: SFTP

You run the commands on different servers. Look at the HOSTNAMEs in the set resp. env commands.

In the ssh it is sasmeta01tst, in the SAS x command it is sasgrid01tst. If your home is not shared across all machines, this may be the reason.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers

View solution in original post


All Replies
Super User
Posts: 10,500

Re: SFTP

Any chance you are running this script on a non-Unix system?

Occasional Contributor
Posts: 7

Re: SFTP

no,no chance Smiley Happy

N/A
Posts: 1

Re: SFTP

Susana, When EG is installed it does not allow for an escape "x" command.  It can be changed though.  Too it depends on whether you're running on the Unix side or have it installed on your PC.

I spoke with John at tech support several years ago, sorry I don't have a last name. You may have to call tech support to see how to change on your PC otherwise you'll have to work with your Unix Admin who is responsible for SAS configuration.

Here is what I got for modifying my registery file:

The SAS command defined in the registry is listed in the Help > About SAS
EG > Configuration Details > 'SAS System Command' row. From my testing the
correct registry key to remove the -noxcmd option on a Win7x64 machine is
below. Back up your registry and make the change.

HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{440196D4-90F0-11D0-9F41-00A024BB830C}\
LocalServer32

Valued Guide
Posts: 3,208

Re: SFTP

similar (x-cmd) as https://communities.sas.com/thread/50072  You SAS-admin can set this option. As of 9.3 there is a way using SMC.
With 9.3 with a local SAS installation the x-cmd is by default setting open again.

Uhhh seeing you sftp command it could be you have access to the system. X-command is working.

It would make sense something is missing in the path definition finding the sftp program.
Pitfall: The unix session behind is not set-up the same way as is done in your ssh approach.

Please try:


filename oscmd pipe "set   2>1" ;  /* the set command should show your settings the same as in your ssh session */
data _null_ ;

  infile oscmd  ; input; put_infile_ ;

run;

This way you you should see your OS command working or getting a clear error.

---->-- ja karman --<-----
Occasional Contributor
Posts: 7

Re: SFTP

Hi Jaap,

First of all,thank you very much for your reply.

I've tried the command and I got this:

NOTE: The infile OSCMD is:

      Pipe command="set   2>1"

/bin/bash: 1: Permission denied

NOTE: 1 record was read from the infile OSCMD.

      The minimum record length was 31.

      The maximum record length was 31.

NOTE: DATA statement used (Total process time):

      real time           0.00 seconds

      cpu time            0.00 seconds

Super User
Posts: 6,938

Re: SFTP

I think that should be

filename oscmd pipe 'set   2>&1' ;

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: SFTP

Yep, that is my typo (not verifying)  == Kurt your are right. Thx for correction.

I often used that to redirect error messages goint into fileid=2 to fileid=1 (normal output) to see everything.
A at least something is coming the X-cmd looks working. I expect seering the value of path that one to SFTP being missing   

---->-- ja karman --<-----
Occasional Contributor
Posts: 7

Re: SFTP

My knowledge about the setting is really limited but, as long as i understand, apparently everything looks ok:

APFMTLIB=SASEnvironment/SASFormats

BASH=/bin/bash

BASH_ARGC=()

BASH_ARGV=()

BASH_EXECUTION_STRING='set   2>&1'

BASH_LINENO=()

BASH_SOURCE=()

BASH_VERSINFO=([0]="3" [1]="2" [2]="25" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")

BASH_VERSION='3.2.25(1)-release'

BINARY_TYPE_HPC=

CLUSTER_DIAG=/var/VRTSvcs/diag//agents/Application

CLUSTER_HOME=/opt/VRTSvcs

CLUSTER_LOGDBG=

CLUSTER_PRODUCT=VCS

CVS_RSH=ssh

DIRSTACK=()

DTJ_CLASSPATH=/sas/sas93/software/SASFoundation/9.3/misc/dmine/dtj.jar:/sas/sas93/software/jre1.6.0_21/lib/rt.jar

EGO_BINDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/bin

EGO_CONFDIR=/sas/sas93/thirdparty/platformcomputing/lsf/conf/ego/sas93cluster/kernel

EGO_ESRVDIR=/sas/sas93/thirdparty/platformcomputing/lsf/conf/ego/sas93cluster/eservice

EGO_LIBDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/lib

EGO_LOCAL_CONFDIR=/sas/sas93/thirdparty/platformcomputing/lsf/conf/ego/sas93cluster/kernel

EGO_SERVERDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/etc

EGO_TOP=/sas/sas93/thirdparty/platformcomputing/lsf

EUID=16777234

GREP_COLOR='1;32'

GREP_OPTIONS=--color=auto

GROUPS=()

G_BROKEN_FILENAMES=1

HISTIGNORE=' cd "`printf "%b": PROMPT_COMMAND='

2                                                          The SAS System                               16:51 Thursday, May 15, 2014

HISTSIZE=1000

HISTTIMEFORMAT='%h/%d - %H:%M:%S '

HOME=/export/home/sasbatch

HOSTNAME=sasgrid01tst

HOSTTYPE=x86_64

IFS=$' \t\n'

INPUTRC=/etc/inputrc

JAVA_HOME=/sas/sas93/software/jre1.6.0_21

LANG=en_US.UTF-8

LD_LIBRARY_PATH=/sas/sas93/software/jre1.6.0_21/lib/i386/server:/sas/sas93/software/jre1.6.0_21/lib/i386:/sas/sas93/software/SASFoun

dation/9.3/sasexe:/sas/sas93/software/jre1.6.0_21/lib/i386/server:/sas/sas93/software/jre1.6.0_21/lib/i386:/sas/sas93/softwa

LD_LIBRARY_PATH_64=/usr/lib/oracle/10.2.0.5/client64/lib

LESSOPEN='|/usr/bin/lesspipe.sh %s'

LOGNAME=sasbatch

LSF_BINDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/bin

LSF_ENVDIR=/sas/sas93/thirdparty/platformcomputing/lsf/conf

LSF_FULL_VERSION=7.06

LSF_LIBDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/lib

LSF_SERVERDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/etc

LS_COLORS='no=00:fi=00:di=00;34:ln=00;36Smiley Tonguei=40;33:so=00;35:bd=40;33;01:cd=40;33;01Smiley Surprisedr=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;3

2:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.l

MACHTYPE=x86_64-redhat-linux-gnu

MAIL=/var/spool/mail/root

MANPATH=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/man:

METAPASS=69Cc6740A03dbE589eCf3a592Fb2F65B

METAUSER='sasbatch@!*(generatedpassworddomain)*!'

NLS_LANG=AMERICAN_AMERICA.UTF8

NOSSLNAMECHECK=1

OPTERR=1

OPTIND=1

ORACLE_HOME=/usr/lib/oracle/10.2.0.5/client64

OSTYPE=linux-gnu

PATH=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/etc:/sas/sas93/thirdparty/platformcomputing/lsf/7.0/li

nux2.6-glibc2.3-x86_64/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:

PPID=26637

PS4='+ '

PWD=/sas/config/sasCompute/Lev1/SASApp

SAMPSIO='(                  '\''!SASROOT/samples/base'\''         '\''!SASROOT/samples/connect'\''         '\''!SASROOT/samples/dmin

e'\''         '\''!SASROOT/samples/ets'\''         '\''!SASROOT/samples/graph'\''         '\''!SASROOT/samples/inttech'\'' 

SAMPSRC='(                  '\''!SASROOT/samples/base'\''         '\''!SASROOT/samples/connect'\''         '\''!SASROOT/samples/ets'

\''         '\''!SASROOT/samples/graph'\''         '\''!SASROOT/samples/inttech'\''         '\''!SASROOT/samples/or'\''    

SASAUTOS='(                  '\''!SASROOT/sasautos'\''         )'

SASCFGPATH='!SASROOT/sasv9.cfg, !SASROOT/nls/u8/sasv9.cfg, !SASROOT/sasv9_local.cfg,  /sas/config/sasCompute/Lev1/SASApp/sasv9.cfg,

/sas/config/sasCompute/Lev1/SASApp/sasv9_usermods.cfg, /sas/config/sasCompute/Lev1/SASApp/WorkspaceServer/sasv9.cfg, /sas/co

SASROOT=/sas/sas93/software/SASFoundation/9.3

SAS_INSTALL_ROOT=/sas/sas93/software/SASFoundation/9.3

SHELL=/bin/bash

SHELLOPTS=braceexpand:hashall:interactive-comments

SHLVL=9

TERM=xterm

TKELS_HANDLE_PREFIX__SAS_SOCKET=0020

TK_PATHLIST=/sas/sas93/software/SASFoundation/9.3/sasexe:/sas/sas93/software/SASFoundation/9.3/utilities/bin

TNS_ADMIN=/sas/sas93/thirdparty/oracle/

UID=16777234

USER=sasbatch

VCSAG_MONITOR_LEVELS=

VCSAG_PIPE_FD=

VCSAG_REG_VERSION=4

3                                                          The SAS System                               16:51 Thursday, May 15, 2014

VCSAG_RES_CONF_LEVEL=

VCSONE_HOSTS=

VCSONE_LOG_RESOURCE_NAME=ObjectSpawner2

VCS_AGFW=1

VCS_CONF=/etc/VRTSvcs

VCS_DIAG=/var/VRTSvcs

VCS_HOME=/opt/VRTSvcs

VCS_LOG=/var/VRTSvcs

VCS_LOG_AGENT_NAME=Application

VCS_LOG_CATEGORY=10031

VCS_LOG_RESOURCE_NAME=ObjectSpawner2

VCS_LOG_SCRIPT_NAME=online

XLSF_UIDDIR=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/lib/uid

_=/bin/bash

Anyway, I'm not sure if i got you,what do you mean with "I expect seeing the value of path that one to SFTP being missing" ?

That i might be passing the wrong path? the only thing that i can think that can be causing this, is the fact that the file names have spaces in between, example : "Daily data dump Bronto20140329.csv.zip" , but in order to get around of this , I've tried to pass the value: *.zip, unfortunately, without success Smiley Sad

Super User
Posts: 6,938

Re: SFTP

I suggest you apply Jaap's encapsulation method to read all the output into the SAS log.

filename oscmd pipe "sftp -oIdentityFile=~/bronto_private_key_openssh bronto@ramjet.vcint.com:/*.zip /sas/corporate-dev/Files/Bronto/. 2>&1";

data _null_ ;

  infile oscmd  ; input; put_infile_ ;

run;

I suspect that the / before *.zip may be the culprit

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: SFTP

This one:

PATH=/sas/sas93/thirdparty/platformcomputing/lsf/7.0/linux2.6-glibc2.3-x86_64/etc:/sas/sas93/thirdparty/platformcomputing/lsf/7.0/li

nux2.6-glibc2.3-x86_64/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:

This setting has been modified by SAS unix scripting. using the key sasbatch?  (EUID/UID=16777234)

Nice Oracle LSF seeing being used probabley also clustered (Ego - Veritas).

== Possible cause, the path setting validation:

Do you have in your ssh shell the following?  you are using a personal key or sasbatch?

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/kerberos/sbin:/usr/kerberos/bin:

An there the sftp is working?  The .profile script at that key could be a cause for some difference.

Where is the sftp program located?

If there is a difference, the full path to the SFTP program could be needed

== Possible cause, something different:

You are not trusting your command. Indeed something to be checked/verified.

x "sftp -oIdentityFile=~/bronto_private_key_openssh bronto@ramjet.vcint.com:/*.zip /sas/corporate-dev/Files/Bronto/.";

The Unix namings are case-sensitive. I assume they are correct. The mailto is a copy issue to this trhead as you tested the sftp manually
The ~ in the command I don not really trust. As you are running this script by sasbatch and I think another key when using the SSH shell.
Another thing is I used single quotes '  instead of the double " hardly visible. It will prevent the SAS macro-processor doing things with that string.       

filename oscmd pipe 'sftp -oIdentityFile=/home/?/bronto_private_key_openssh bronto@ramjet.vcint.com:/*.zip /sas/corporate-dev/Files/Bronto/.      2>&1' ;

data _null_ ;

  infile oscmd  ; input; put_infile_ ;

run;

Anyway when you code the full vaild name (green /home) it should give a clear message on what is happening.

---->-- ja karman --<-----
Occasional Contributor
Posts: 7

Re: SFTP

I'm using a private key located in sasbatch (user) home.

what i have in the ssh shell is:

[sasbatch@sasmeta01tst ~]$ echo $PATH

/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/VRTS/bin:/var/cfengine/bin:/export/home/sasbatch/bin

[sasbatch@sasmeta01tst ~]$ "env"

GREP_COLOR=1;32

HOSTNAME=sasmeta01tst

SHELL=/bin/bash

TERM=xterm

HISTSIZE=1000

USER=sasbatch

LS_COLORS=no=00:fi=00:di=00;34:ln=00;36Smiley Tonguei=40;33:so=00;35:bd=40;33;01:cd=40;33;01Smiley Surprisedr=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35:

MAIL=/var/spool/mail/sasbatch

PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/opt/VRTS/bin:/var/cfengine/bin:/export/home/sasbatch/bin

INPUTRC=/etc/inputrc

PWD=/export/home/sasbatch

LANG=en_US.UTF-8

SHLVL=1

HOME=/export/home/sasbatch

GREP_OPTIONS=--color=auto

LOGNAME=sasbatch

CVS_RSH=ssh

LESSOPEN=|/usr/bin/lesspipe.sh %s

HISTTIMEFORMAT=%h/%d - %H:%M:%S

G_BROKEN_FILENAMES=1

_=/bin/env

OLDPWD=/sas/corporate/Files/Bronto

Now i ran:

filename oscmd pipe '/usr/bin/sftp -oIdentityFile=/export/home/sasbatch/bronto_private_key_openssh bronto@ramjet.vcint.com:/*Bronto20140516.csv.zip /sas/corporate/Files/Bronto/. 2>&1';

data _null_ ;

  infile oscmd  ;

  input;

  put _infile_ ;

run;

and the output was:

     bronto@ramjet.vcint.com:/*Bronto20140516.csv.zip /sas/corporate/Files/Bronto/. 2>&1"

Connecting to ramjet.vcint.com...

Permission denied, please try again.

Permission denied, please try again.

Received disconnect from 172.31.1.5: 2: Too many attempts.

Couldn't read packet: Connection reset by peer

For some reason it doesn't access the private key.....

if i run in unix:

/usr/bin/sftp -oIdentityFile=/export/home/sasbatch/bronto_private_key_openssh bronto@ramjet.vcint.com:/*Bronto20140516.csv.zip /sas/corporate/Files/Bronto/.

Connecting to ramjet.vcint.com...

Fetching /Daily data dump Bronto20140516.csv.zip to /sas/corporate/Files/Bronto/./Daily data dump Bronto20140516.csv.zip

/Daily data dump Bronto20140516.csv.zip                                                                                                                                                                    100%  224KB 223.7KB/s   00:00

it works perfectly....

Not sure if i gave you all the info you asked.

Valued Guide
Posts: 3,208

Re: SFTP

SusanaB, A computer is doing what is told, often not what is intendend. By that I am very stubborn at some moment as they look like not willing to listen.
As soon as response will be "it is Monday I have no energy" we need to dump them or see them a living beings. Not trusting their reliability on doing what is told anymore.:smileylaugh:

As you are able to run that command form a command line we are just in the process of closing it to a situation with no differences any more.
The last run showing sftp being run from /usr/bin and using the sasbatch home coded has eliminated those options.
You are getting nice messages of "permission denied".  Not being happy as not showing what the issue is complaining ..

There are still known differences to investigate.     
1/  PWD=/export/home/sasbatch and  PWD=/sas/config/sasCompute/Lev1/SASApp   (The current dir - ithere is a sas-note on this quirk)
The left one of your SSH session the right one of SAS-DI.  The right one is not accessible for writing any files. When the SFTP is doing that (some intermediate file) it could explain the messages.
To solve/try that change the filename pipe to:  ' cd /export/home/sasbatch ;  /usr/bin/sftp -oIdentityFile=/export/home/sasbatch/bronto_private_key_openssh ......

2/ umask ulimit .. and more settings are not set according the sasbatch user-profile of Unix but are coming from the key running the objectspawner probably the sasinst key.
I do not expect some trouble in this area for your SFTP usage.
uhh using mail. MAIL=/var/spool/mail/sasbatch   MAIL=/var/spool/mail/root .. (DI environment)  There you have another differnce

---->-- ja karman --<-----
Occasional Contributor
Posts: 7

Re: SFTP

ahahah! it's true Smiley Happy.

I have to admit that, this time, I'm starting to think that it might be an option to yell at it , to make sure that he does what I'm telling himSmiley Happy

1/  You mean, changing it like the following?

x 'cd /export/home/sasbatch;/usr/bin/sftp -oIdentityFile=/export/home/sasbatch/bronto_private_key_openssh bronto@ramjet.vcint.com:/*Bronto20140516.csv.zip /sas/corporate/Files/Bronto/.';

I've tried... again without success Smiley Indifferent

2/ yeah..I also expect that one not to be an issue...

Solution
‎05-16-2014 11:15 AM
Super User
Posts: 6,938

Re: SFTP

You run the commands on different servers. Look at the HOSTNAMEs in the set resp. env commands.

In the ssh it is sasmeta01tst, in the SAS x command it is sasgrid01tst. If your home is not shared across all machines, this may be the reason.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 15 replies
  • 1775 views
  • 8 likes
  • 5 in conversation