09-27-2017 04:35 AM
I'm at my wits end....
I'm trying to set up Kerberos TGTs in SAS 9.4 M3 Grid (RHEL6) for Hadoop purposes. I have....
- Applied the required SAS TK hotfixes to allow TGT forwarding
- Set SAS_GRID_USE_KERBEROS=1 in ObjectSpawner_usermods.sh and level_env_usermods.sh
- Set LSB_KRB_TGT_FWD=Y in lsf.conf
- Restarted the environment
- Set the required krb5 libs in /lib64 as LSF expects them
SAS on the LSF submission host will generate the TGT ok. TGT is valid, forwardable and renewable. On the LSF execution host I can see that KRB5CCNAME is set to the same TGT name that was generated on LSF submission host. However, the only missing component is that the physical TGT is not being generated on the LSF execution host. KRB5CCNAME points there, but the TGT is physically not there.
What am I missing ?
09-27-2017 08:31 AM
Thanks for your response. That filename is not valid. That is KRB5CCNAME for gridRun, not for LSF. First of all, make sure that you have no scripts that can potentially change KRB5CCNAME in WorkspaceServer_usermods.sh. Second, are you sure that LSF ticket wasn't created in /tmp directory?
ls -lrt /tmp/lsf* | grep <YOUR_NAME>
Also, are you using MIT libraries or third-party software such as Centrify?
09-27-2017 09:04 AM
Correct, so the value of KRB5CCNAME is passed from the LSF submission host (gridrun) and is of course not valid on the LSF execution host. There is nothing that sets this value explicitly, SAS does it on it's own in some way.
I checked, the TGT wasn't created for me on the LSF execution host at all.
$ ls -lrt /tmp/lsf* | grep myuserid
ls: cannot access /tmp/lsf*: No such file or directory
We are using AD as a KDC. I would assume the Kerberos libraries are those from RHEL in lib64
09-27-2017 09:14 AM
Ensure that the krb5 libs (libkrb5.so, libcom_err.so, libk5crypto.so, and libkrb5support.so) are in the default directories. On 64-bit platforms, the directories are /lib64, /usr/lib64, and /usr/local/lib64. If the libs are not in these directories, specify the directories in the LSB_KRB_LIB_PATH= parameter. You can specify multiple paths, separated by blanks, commas, or semicolons.
Have you restarted LSF after changing lsf.conf?
If yes, please add these loggers to /<SASConfig>/Lev<X>/ObjectSpawner/logconfig.xml:
<logger name="Audit.Authentication"> <level value="Trace"/> </logger> <logger name="App.tk.tkegrid"> <level value="Trace"/> </logger>
In the LSF configuration file, add these entries:
LSB_LOG_MASK=LOG_DEBUG LSB_DEBUG_CMD="LC2_KRB" LSB_CMD_LOGDIR=logging_directory LSF_LOG_MASK=LOG_DEBUG LSF_DEBUG_CMD="LC2_KRB" LSF_CMD_LOGDIR=logging_directory
Restart both the object spawner and LSF, repeat a problem and send to me log files for further analysis.