SFTP, RSYNC or SCP not working via SAS Enterprise Guide 8.3 execution

[Tom007]

HI 

After in place upgrade to RHEL 7.9 to 8.9 (having SAS 9.4 M7 installed), post upgrade any file transfer protocols related SAS program with filename  SFTP or  X or pipe protocol commands  only are not working & encountering below error.
Note: Password less authentication is in place , key exchange done between source & destination.

Error:

/usr/bin/ssh:symbol lookup error: /lib64/libgssapi_krb5.so.2: undefined symbol: k5_internalize_principal, version krb5_3_MIT lost connection

However, on Linux server command line OS commands rsync /sftp /scp are working fine.

Sample codes that are not working:

filename myfile sftp '/users/xxxx/test.dat' host="unixhost1" options="-oPort=4117";
data _null_;
   infile myfile truncover;
   input a $25.;;
run;

Error:

/usr/bin/ssh:symbol lookup error: /lib64/libgssapi_krb5.so.2: undefined symbol: k5_internalize_principal, version krb5_3_MIT lost connection

/*Password less authentication setup and working fine */
%let rsynccmd= rsync -avph /dir/filename user@hostname:/dir/path;

filename oscmd pipe "&rsynccmd";

data _null_;
infile oscmd;
input;
put _infile_;
run;


Error:

/usr/bin/ssh:symbol lookup error: /lib64/libgssapi_krb5.so.2: undefined symbol: k5_internalize_principal, version krb5_3_MIT lost connection

[SASKiwi]

You are using single quotes which won't resolve your embedded macro variable. Try double quotes:

filename oscmd pipe "&rsyncmd";

[Tom007]

Thanks SASKiwi for your attention and response, I did double check the code there is double quotes I updated in my query also.

Issue remains with same error.

[Kurt_Bremser]

Run the set command from the commandline and from within SAS, and look for differences in the output. I suspect a missing library path.

[Tom007]

Thank you @Kurt_Bremser for you recommendation.

After RHEL 8.9 upgrade , As a work around unset $LD_LIBRARY_PATH ; set $/usr/bin; is working for protocols in SAS Code having like X command , pipe command via SAS Enterprise Guide 8.3. Where as sas batch mode and crontab no issues (no need to unset/set).

Our Aim to execute as before without unset & set like before upgrade.

However, this is not behavior during OS RHEL 7.9, SAS code having the scp/ rsync/ sftp commands absolutely works fine in with X command , pipe command Enterprise Guide 8.3 , sas batch mode & crontab.  

[gwootton]

This is a Kerberos error that is the result of mismatched library versions. As @Kurt_Bremser mentioned, this is likely the result of your library path setting (value of LD_LIBRARY_PATH environment variable) including a path that contains a different version of the Kerberos libraries that does not contain the k5_internalize_principal symbol being called by /lib64/libgssapi_krb5.so.2.

[Sajid01]

Hello @Tom007 
One approach to resolving your issue could be using public key authentication.
The solution to this ticket has the sample code that should help

https://communities.sas.com/t5/SAS-Programming/Move-files-between-server-using-SFTP/td-p/350673

[Tom007]

Hi @Sajid01 

Thank you for looking into my issue and providing your thoughts , I appreciate your time and suggestion.

We have made sure the public key authentication b/w source and destination hosts, via OS command line testing, connection work seamless without password prompt.

Even by examples works fine in OS Command line and even embedded in sas code in batch mode or crontab schedule sas job

rsync -v  /source_path/file_name  userid@FQDN:/destination_path/

scp -p   /source_path/file_name  userid@FQDN:/destination_path/

Issue encountering same via SAS Enterprise Guide 8.3 code execution, no error in log file, nor file isn't getting transferred to destination host.

When we user pipe command, following is error: ssh: symbol lookup error: /lib64/libgssapi_krb5.so.2: undefined symbol: k5_internalize_principal, version krb5_3_MIT

%let rsynccmd= rsync -v source_path/baseball.sas7bdat userid@FQDN:/destination_path;

filename cmdpipe pipe "&rsynccmd";
data _null_;
infile cmdpipe;
input;
put _infile_;
run;

[Tom007]

Workaround: When we remove below highlighted path from sasenv.local from Foundation path of SAS. Then we are able to perform execute via SAS Enterprise Guide 8.3 successfully.
LD_LIBRARY_PATH=/opt/Progress/DataDirect/ODBC_80_64bit/lib:/usr/local/sap/IQ-16_1/lib64:/usr/local/nz/lib64:/usr/local/nz/lib:;

But the hitch is , this is impacting crontab scheduled jobs, encountering with below error

ERROR: Could not load /<SASHOME>/SASFoundation/9.4/sasexe/sasnza (40 images loaded)
ERROR: libnzodbc.so: wrong ELF class: ELFCLASS32
ERROR: The SAS/ACCESS Interface to Netezza cannot be loaded. The SASNZA   code appendage could not be loaded.
ERROR: Error in the LIBNAME statement.

For the above SASNZA error we are exporting the LD_LIBRARY_PATH in user's home .bash_profile 

00 09 * * *  source ~/.bash_profile; /sashome/SASFoundation/9.4/sas  sasprogram.sas  -log  sasprogram.log

[Sajid01]

Hello
Try updating the LD_LIBRARY_PATH as follows

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/Progress/DataDirect/ODBC_80_64bit/lib:/usr/local/sap/IQ-16_1/lib64:/usr/local/nz/lib64:/usr/local/nz/lib:;

If that does not work add /<SASHOME>/SASFoundation/9.4/sasexe/sasnza to your path variable

${PATH}=${PATH}: /full_path_here/SASFoundation/9.4/sasexe/sasnzaSeek 

It will be good if you can rope in the assistance of your local LInux Administrators. The local SAS/Linux Admins know your environment well and can be of great help.

[Tom007]

thank you  @Sajid01 for the recommendation we did tweak the LD_LIBRARY_PATH as you mentioned but no luck and we omitted '/usr/local/nz/lib64' from the file /opt/sas/sashome/SASFoundation/9.4/bin/sasenv_local

which is allowing SAS users to perform the transfer protocols.

Appreciate your inputs.

[gwootton]

The error saying wrong ELF class is because /usr/local/nz/lib is still present, so it's finding the 32 bit drivers which are the wrong ELF class.

The main issue here is you have two sets of Kerberos libraries, one in /lib64 and one in /usr/local/nz/lib64, and they are not compatible.

Depending on which function you are using, the appropriate path would need to be first in LD_LIBRARY_PATH so it can find it's associated libraries and not the ones from the incompatible location.

If LD_LIBRARY_PATH contains /lib64 before /usr/local/nz/lib64, the OS functions will work.
If LD_LIBRARY_PATH contains /usr/local/nz/lib64 before /lib64 (or /lib64 is omitted and implicit), then Netezza connectivity will work.

So if your sasenv_local contains an LD_LIBRARY_PATH that includes /usr/local/nz/lib64 (which would be needed to get Netezza working) you would probably need to add an options statement to modify the LD_LIBRARY_PATH environment variable prior to running your OS functions. e.g.

options set=LD_LIBRARY_PATH="/lib64/:%sysget(LD_LIBRARY_PATH)";

[gwootton · Accepted solution]

It sounds like "/local/nz/lib64" contains its own version of the kerberos libraries. I might try adding /lib64/ in front of it so that path is checked first.
As @Sajid01 points out you should also include the existing definition of LD_LIBRARY_PATH (e.g. LD_LIBRARY_PATH=$LD_LIBRARY_PATH:..., otherwise you would overwrite what is already there.