Hadoop connectivity issues

[WorkingMan]

Hi SAS Communities. I am having Hadoop connectivity issues whereby connection will fail intermittently. This is affecting my ETL flow that tries to connect to Hadoop.

I referred to this thread: https://communities.sas.com/t5/SAS-Data-Management/Hadoop-connectivity-issues/td-p/114069

And I hope @JBailey can further assist me. I have already raised SAS Support track for more than 3 weeks but there is no solution from them.

My ETL schedule to run at midnight 12am but it never connect to hadoop sucessfully. I tried to simulate the issue by scheduling it to run half-hourly from 12am to 7:30am. Seems like only 7:30am onwards will be able to connect to Hadoop. However, I have discussed with my client and there is no connection issue or settings stopping from any connection to Hadoop at the mentioned time.

They(Client IT Team) suspect this issue could be due to Kerberos ticket. They mentioned that this could be due to Kerberos ticket was generated during day is expired whenever our job run during midnight. Is there a way for us to know if this is the real cause of it? Or does it make any sense at all as the only thing i did was a LIBNAME statement?

Can I know how Kerberos ticket is generated before calling HADOOP resources via SAS job? Secondly, could you provide the klist command as user sas.admin for us to see the output from our linux production server?

Appreciate if anyone can further assist me on this 😞

Your help is very much appreciated.

[JBailey]

Hi @WorkingMan 

I found your Tech Support track. It does sound like there is Kerberos weirdness taking place. @RPoumarede has created a great Hadoop/Kerberos article on the SAS communities. Take a look and see if it helps.

https://communities.sas.com/t5/SAS-Communities-Library/5-tips-for-SAS-ACCESS-TO-KERBERIZED-HADOOP/ta...

Let me know if you need more assistance.

Best wishes,

Jeff

[WorkingMan]

@JBaileyThanks for the guide. However, I am still unable to fix this issue until now.

This is a Production server and I do not wish to do any installation to it for testing with beeline. I understand that it is hard to proceed if we dont do further testing. Is there any other method instead of using beeline?

Based on your experience, what could be the possible cause? Could it be something is prohibiting the connection at the specific time? If yes, could it be other script that is executing at that time etc?

[WorkingMan]

@JBaileyI just found something out that could lead us to a new direction. I created a dummy flow that will connect to Hadoop that runs every 30mins for 24 hours. I found out that none of the scheduled flow has successfully connected to HADOOP. However, during day time, if i trigger manually with LSF Flow Manager, connection to HADOOP is successful. But, the flow will not be able to connect to HADOOP even during daytime.

Could this be something to do with scheduler itself?

[JBailey]

Hi @WorkingMan 

Is the scheduler running the job as the same user the manually triggered job? If it isn't then it may be an authentication issue. Are there error messages in the scheduler logs?

Best wishes,

Jeff