LSF not receiving Kerberos TGT

[nhvdwalt]

Hi team,

I'm at my wits end....

I'm trying to set up Kerberos TGTs in SAS 9.4 M3 Grid (RHEL6) for Hadoop purposes. I have....

- Applied the required SAS TK hotfixes to allow TGT forwarding

- Set SAS_GRID_USE_KERBEROS=1 in ObjectSpawner_usermods.sh and level_env_usermods.sh

- Set LSB_KRB_TGT_FWD=Y in lsf.conf

- Restarted the environment

- Set the required krb5 libs in /lib64 as LSF expects them

SAS on the LSF submission host will generate the TGT ok. TGT is valid, forwardable and renewable. On the LSF execution host I can see that KRB5CCNAME is set to the same TGT name that was generated on  LSF submission host. However, the only missing component is that the physical TGT is not being generated on the LSF execution host. KRB5CCNAME points there, but the TGT is physically not there.

What am I missing ?

Thanks,

[alexal]

@nhvdwalt,

What is the current value of KRB5CCNAME on the execution host?

%put KRB5CCNAME: %sysget(KRB5CCNAME);

[nhvdwalt]

Hi @alexal

It's...

24 %put KRB5CCNAME: %sysget(KRB5CCNAME);
KRB5CCNAME: FILE:/tmp/krb5cc_sasgrid_myuserid_13922_7fa13b507c40
25 run;

[alexal]

@nhvdwalt,

Thanks for your response. That filename is not valid. That is KRB5CCNAME for gridRun, not for LSF. First of all, make sure that you have no scripts that can potentially change KRB5CCNAME in WorkspaceServer_usermods.sh. Second, are you sure that LSF ticket wasn't created in /tmp directory?

ls -lrt /tmp/lsf* | grep <YOUR_NAME>

Also, are you using MIT libraries or third-party software such as Centrify?

[nhvdwalt]

Thanks @alexal

Correct, so the value of KRB5CCNAME is passed from the LSF submission host (gridrun) and is of course not valid on the LSF execution host. There is nothing that sets this value explicitly, SAS does it on it's own in some way.

I checked, the TGT wasn't created for me on the LSF execution host at all.

$ ls -lrt /tmp/lsf* | grep myuserid
ls: cannot access /tmp/lsf*: No such file or directory
$

We are using AD as a KDC. I would assume the Kerberos libraries are those from RHEL in lib64

[alexal]

@nhvdwalt,

Ensure that the krb5 libs (libkrb5.so, libcom_err.so, libk5crypto.so, and libkrb5support.so) are in the default directories. On 64-bit platforms, the directories are /lib64, /usr/lib64, and /usr/local/lib64. If the libs are not in these directories, specify the directories in the LSB_KRB_LIB_PATH= parameter. You can specify multiple paths, separated by blanks, commas, or semicolons.

Have you restarted LSF after changing lsf.conf?

If yes, please add these loggers to /<SASConfig>/Lev<X>/ObjectSpawner/logconfig.xml:

<logger name="Audit.Authentication">
<level value="Trace"/>
</logger>
<logger name="App.tk.tkegrid">
<level value="Trace"/>
</logger>

In the LSF configuration file, add these entries:

LSB_LOG_MASK=LOG_DEBUG
LSB_DEBUG_CMD="LC2_KRB"
LSB_CMD_LOGDIR=logging_directory
LSF_LOG_MASK=LOG_DEBUG
LSF_DEBUG_CMD="LC2_KRB"
LSF_CMD_LOGDIR=logging_directory

Restart both the object spawner and LSF, repeat a problem and send to me log files for further analysis.

[nhvdwalt]

Thanks @alexal, will do.

This is a PROD environment, so I just have to get a slot where I can do the changes and restarts.

Thanks,

[alexal]

@nhvdwalt,

Sure, keep me posted.