How does a Linux batch job connect to metadata?

[jklaverstijn]

Dear fellow admins,

On one of our grid environments (SAS 9.4M4, Linux, LSF 9.1.3, Kerberos authentication), batch jobs give the message:

ERROR: Unable to establish a SAS Metadata Server connection.

I have re-deployed this one after 7 grid environments that work properly and now I get this. Pre-assigned libraries don't work where they do al the others environments

I must admit to not being able to entirely explain the anatomy of how a batch job connects to the metadata server in absence of user credentials. I would think a trusted peer sort of thing but that doesn't apply to this kind of server.The metaautoresources option is set to SASApp. The other (IOM) servers work properly. File metadataConfig.xml is correct.

So I am a bit lost. I was a bit reluctant to ask this question, afraid of RTFM responses and a facepalm moment for asking about something so obvious. But I have to come clean as time becomes an issue.

So there you have it: can someone explain this and help me diagnose the issue?

Many thanks in advance,

-- Jan.

[JuanS_OCS]

Hello @jklaverstijn,

the Batch server is not exactly an IOM SAS server. That is why, when you deploy the DI jobs, they would contain at the begining the connection to the metadata, because they require it at execution time. Same goes for batch jobs launched with just SAS Foundation.

This might help:

http://support.sas.com/kb/32/507.html

Problem Note 32507: Batch jobs that fail to connect to a SAS Metadata Server might issue a misleading warning message intead of an error

Does it help?

Juan

[jklaverstijn]

Hi Juan,

Thanks for chiming in. I have seen that note and it doesn't help me any further. I test with a job that does

proc options group=meta;
run;

It shows all options (server, port, repos, ...) correctly set and no metauser or metapassword. This agrees with other environments where it does work. One of the reasons why I am so confused.

Cheers Jan.

[JuanS_OCS]

That is quite interesting, Jan.

And, can you connect with this user to metadata, with SAS Management Console in that environment? I would assume yes.

And with EG, as example?

Also, if you check if this user has issued a proper Kerberos TGT ticket? With the kinit, klist commands.

If you cannot get an answer soon enough, I would recommend to extend the log information level of the metadata server, for short time, and test the connections again. The log should be able to tell us more.

[PaulHomes · Accepted solution]

What makes you say Trusted Peer Connections doesn’t apply?

[jklaverstijn]

Hi Paul,

Yes thanks you got me there. I will have to rethink that. Trustedpeer is also for batch jobs and is likely how it works. I will look into how this may fail.

- Jan.

[PaulHomes]

Hi Jan,

I'd start by looking at Lev1/SASMeta/MetadataServer/trustedPeers.xml to see who the SAS Metadata Server has been configured to accept as trusted peers. Has it been modified from the defaults of the initial install?

Then check the series of config files used (with a starting point of sasbatch.sh) on the problem node and especially Lev1/metadataConfig.xml to see if it is pointing to the wrong metadata server. Also check DNS resolution for the metadata server node host name(s) is working as expected on that node.

As Juan suggested, have a look at the metadata server logs. Is there a failed connection on the metadata server from the problem node when the job is run? Is there any connection registered at all? If so, as whom? Does it fail with a specific error?

Verify who the Linux process owner of the batch job is? Do they have a corresponding SAS metadata identity?

Try running a simple test by SSH-ing into the machine as the same Linux user id and running something like this to see if the metadata libraries can be assigned:

/opt/sas/config/Lev1/SASApp/BatchServer/sasbatch.sh -sysin ~/simple-proc-options.sas -log ~/simple-proc-options.log

Hope this helps.

Cheers
Paul

[jklaverstijn]

Hi Paul,

Thanks for the elaborate answer. I have done most of these tests and nothing stands out. There is a mis-configuration, I just haven't found it yet. I do now know the answer to my basic question:

Q: How does a batch job connect to the metadata server in the absence of a password?

A: Trusted peer authentication

And yes, a bit of a "DOH!" moment for me here.

I will have to dig even deeper. I will keep you guys posted.

Thanks,

- Jan.

[PaulHomes]

No worries Jan. Perhaps turn up the level of logging for the metadata server and batch process, trace the process/connection, and see if anything stands out.

[jklaverstijn]

Well, I finally found it and it was less than obvious. It turned out that we had an inconsistent mix of values for the NETENCRYPTALGORITHM option. After systematically eliminating lines from that file it turned out to be related to this option. This revealed that several other components, the metadata server being one of them, had this option set to SASProprietary. The batch server had AES in its config. Our internal requirement is AES for everything. Once we fixed this things started working again.

So, in the end it was a simple solution to a tough problem. Which is not unusual at all.

Thanks to all for contributing. Understanding the physics of trustedpeer authentication allowed me to rethink my misconceptions and zoom in on this configuration setting. Therefore I will accept @PaulHomes contribution as the solution. It steered me in the right direction. 

Regards, 

- Jan

[PaulHomes]

Thanks Jan! ... I'm glad I was able to help a little but I think you should have probably marked your own response as the answer to the post .... since it actually contained the real answer! 😉

[saivenkat]

Check the methods below, if it could be helpful in debugging

proc metaoperate
server="sasmetadataservername"
port=8561
userid="id"
password="pwd"
action=status;
run;

recently we had fixed the access issue for Ad users using the links below, above method was helpful to ensure sas meta config changes were effective

http://support.sas.com/kb/39/891.html

http://support.sas.com/kb/51/911.html

Did u manage to find fix?