BookmarkSubscribeRSS Feed
nhvdwalt
Barite | Level 11

Hi team,

 

I'm at my wits end....

 

I'm trying to set up Kerberos TGTs in SAS 9.4 M3 Grid (RHEL6) for Hadoop purposes. I have....

 

- Applied the required SAS TK hotfixes to allow TGT forwarding

- Set SAS_GRID_USE_KERBEROS=1 in ObjectSpawner_usermods.sh and level_env_usermods.sh

- Set LSB_KRB_TGT_FWD=Y in lsf.conf

- Restarted the environment

- Set the required krb5 libs in /lib64 as LSF expects them

 

SAS on the LSF submission host will generate the TGT ok. TGT is valid, forwardable and renewable. On the LSF execution host I can see that KRB5CCNAME is set to the same TGT name that was generated on  LSF submission host. However, the only missing component is that the physical TGT is not being generated on the LSF execution host. KRB5CCNAME points there, but the TGT is physically not there.

 

What am I missing ?

 

Thanks,

7 REPLIES 7
alexal
SAS Employee

@nhvdwalt,

 

What is the current value of KRB5CCNAME on the execution host?

 

%put KRB5CCNAME: %sysget(KRB5CCNAME);
nhvdwalt
Barite | Level 11

Hi @alexal

 

It's...

 

24 %put KRB5CCNAME: %sysget(KRB5CCNAME);
KRB5CCNAME: FILE:/tmp/krb5cc_sasgrid_myuserid_13922_7fa13b507c40
25 run;

 

 

alexal
SAS Employee

@nhvdwalt,

 

Thanks for your response. That filename is not valid. That is KRB5CCNAME for gridRun, not for LSF. First of all, make sure that you have no scripts that can potentially change KRB5CCNAME in WorkspaceServer_usermods.sh. Second, are you sure that LSF ticket wasn't created in /tmp directory?

 

ls -lrt /tmp/lsf* | grep <YOUR_NAME>

Also, are you using MIT libraries or third-party software such as Centrify?

nhvdwalt
Barite | Level 11

Thanks @alexal

 

Correct, so the value of KRB5CCNAME is passed from the LSF submission host (gridrun) and is of course not valid on the LSF execution host. There is nothing that sets this value explicitly, SAS does it on it's own in some way.

 

I checked, the TGT wasn't created for me on the LSF execution host at all.

 

$ ls -lrt /tmp/lsf* | grep myuserid
ls: cannot access /tmp/lsf*: No such file or directory
$

 

We are using AD as a KDC. I would assume the Kerberos libraries are those from RHEL in lib64

alexal
SAS Employee

@nhvdwalt,

 

Ensure that the krb5 libs (libkrb5.so, libcom_err.so, libk5crypto.so, and libkrb5support.so) are in the default directories. On 64-bit platforms, the directories are /lib64, /usr/lib64, and /usr/local/lib64. If the libs are not in these directories, specify the directories in the LSB_KRB_LIB_PATH= parameter. You can specify multiple paths, separated by blanks, commas, or semicolons.

 

Have you restarted LSF after changing lsf.conf?

 

If yes, please add these loggers to /<SASConfig>/Lev<X>/ObjectSpawner/logconfig.xml:

 

<logger name="Audit.Authentication">
<level value="Trace"/>
</logger>
<logger name="App.tk.tkegrid">
<level value="Trace"/>
</logger>

In the LSF configuration file, add these entries:

 

LSB_LOG_MASK=LOG_DEBUG
LSB_DEBUG_CMD="LC2_KRB"
LSB_CMD_LOGDIR=logging_directory
LSF_LOG_MASK=LOG_DEBUG
LSF_DEBUG_CMD="LC2_KRB"
LSF_CMD_LOGDIR=logging_directory

Restart both the object spawner and LSF, repeat a problem and send to me log files for further analysis.

nhvdwalt
Barite | Level 11

Thanks @alexal, will do.

 

This is a PROD environment, so I just have to get a slot where I can do the changes and restarts.

 

Thanks,

alexal
SAS Employee

@nhvdwalt,

 

Sure, keep me posted.

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

CLI in SAS Viya

Learn how to install the SAS Viya CLI and a few commands you may find useful in this video by SAS’ Darrell Barton.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 7 replies
  • 1952 views
  • 0 likes
  • 2 in conversation