topic Re: AD Account is locked at SAS Server in Administration and Deployment

AD Account is locked at SAS Server

DHS_SASADM — Wed, 30 Sep 2015 20:36:03 GMT

I have a user who has been having his account locked out at the domain level and has happened numerous times since Friday. This user is high profile. This started on Friday and has been continuing for the last few days. I have my server and security teams working the issue with no results. The user gets his account unlocked and within the hour it is getting locked. The security monitor SPLUNK reports that my SAS server is the one locking him out. Over the last three days we have rebooted the server, deleted and re-added the account, and had him try to access production, backup and devel servers. Today we opened a SAS Track and are waiting for a response. Since this user supports higher management here, I used a "SERVICE" account that I had in reserve and logged him in as that from his PC through to the server. He has been stable all afternoon via that method. I, in turn is masquerading as the user to see if I while impersonating him get locked out.

I'm runnning Office Analytics single machine 9.4 M2. The user is running EG 6.1 and I am running EG 7.1.

I'm open to any and all suggestions.

Thanks,

Jerry Coppa SAS Admin at PA DHS

Re: AD Account is locked at SAS Server

SASKiwi — Wed, 30 Sep 2015 20:58:57 GMT

I had a problem very similar to this and it was caused by old remote login sessions to our SAS server that were just disconnected but not signed out. In the meantime I had changed my password but the old sessions kept trying to authenticate and kept locking me out.

Does your user have remote login access? If not another possibility is having his old password stored in SAS metadata or in scheduled jobs. Use SAS Management Console to update the metadata-stored password or to re-schedule the jobs.

Re: AD Account is locked at SAS Server

twocanbazza — Wed, 30 Sep 2015 23:01:11 GMT

Have seen this before also but can't recall the exact problem...

I gather the user can't log onto the server directly

Given you have rebooted the server etc, a rouge session doesn't seem to be the issue...

do you have credentials stored in the metadata? or hardcoded in SAS code? are there any SAS jobs running on the server when the account is locked? Is it after they do something in SAS or is it even when they don't do anything?

Although it is the SAS server that is seen to be be doing this it may not be SAS that is the issue....

Barry

Re: AD Account is locked at SAS Server

jakarman — Thu, 01 Oct 2015 21:54:46 GMT

There are a lot of possible causes.

- The existance of some user ghostprocesses still running and connecting at intervals

- The coding of the old/wrong password in a connection profile. (eguide amo/ .net DI SMC /java)

- the caching (eg conncet) in a SAS metadatadata autentication location

- the usage hard coded of user/password combination.
Finding and seeing these user errors can be hard.

Than you can ahve problems in the SAS system itself.

- The login can be delayed by failed logins. In those cases the metadata login can get delayed in a unusable way.

When the user does a login and wille retry with different password thinking it are typos it can cause a lock.
- After changing the password the sasauthentication can be delayed separtely of your OS setttings.

Getting a new password after an unlock can cause the marvelous sitaution you can loging at the OS level but using SAS for that will fail. After several retries it can get locked.

Re: AD Account is locked at SAS Server

mfab — Fri, 17 Nov 2017 10:57:30 GMT

Hello @jakarman,

I have stumbled upon a problem that might be related to what you were mentioning.

"the caching (eg conncet) in a SAS metadatadata autentication location"

and

"After changing the password the sasauthentication can be delayed separtely of your OS setttings"

Situation is as follows:

Host is unix and the password is changed.

User tries to login from EG/WebReportStudio/Excel Add-In.

User receives promt to enter valid username and password

User enters new information.

SAS says that the information is still invalid.

I then deleted the user account, created it as new account - still no success.

After a while (15-30 minutes) the login suddenly works.

I have been trying to figure out if any setting in the omaconfig.xml (hints) could have had any effect, but no clue was found.

After 3 unsuccessful attempts, the account should have been "locked", but what does this mean? How long is it locked and why did it suddenly work out?

Also, where would the metadata-server or connect spawner (?) cache any information? Are there any hints to how frequently cashing will take place?

Also log files did not provide any useful information aside from the fact, that the user login was denied.

Thank you for any hints and suggestions.

Cheers,

Michael

Re: AD Account is locked at SAS Server

Kurt_Bremser — Fri, 17 Nov 2017 11:42:04 GMT

This is something that I have noticed repeatedly. Once the metadata server has received a "failed" message from the OS for an authentication attempt, it takes some time before it re-checks the authentication.

- user attempts connection, is rejected because password is expired in OS

- user logs in via ssh, changes password as required

- user tries with new password, but is rejected; metadata log reports "Access denied"

- user waits a given amount of time and tries again - voila, works

Re: AD Account is locked at SAS Server

JuanS_OCS — Fri, 17 Nov 2017 12:29:53 GMT

Hello @mfab,

this is a behaviour quite old ( I know it since back 9.1.3 ) and it remains the same, with some improvements, now it happens less often.

General solution is to refresh the SAS Application Server, or even the Object Spawner.

If you have a Grid environment, this might happen on the LSF level as well.

Re: AD Account is locked at SAS Server

mfab — Fri, 17 Nov 2017 12:47:49 GMT

Thanks for the answers so far!

@Kurt_Bremser: do you see any chance, that SAS will document the default behaviour anywhere? It would be pleasant to at least know how the system works, even if there is no chance in changing things. Would you open a support ticket for this or does anyone from SAS have a look into the communities from time to time?

@JuanS_OCS: what do you mean by "refresh" - are you thinking of restart? This would not be applicable for our purposes.

We want to prevent user logins by using the "passwd --lock <username>" command in Cronjobs. So users can't login during given times. We would not want to restart any services because some users did try to login and have their login in SAS ... ahm ... let's say deactivated or not refreshed.

A solution from within SAS to deactivate accounts or prevent login at certain times would also be desirable, but there does not seem to be any option like this (and I wonder why, since the rest of the metadata management in SAS is quite nice).

Cheers,

Michael

Re: AD Account is locked at SAS Server

alexal — Fri, 17 Nov 2017 13:48:03 GMT

@mfab,

I support authentication on UNIX/Linux.

Are you guys talking about PAM or Host authentication?

@Kurt_Bremser

user tries with new password, but is rejected; metadata log reports "Access denied"

Please enable sasauth-debug. Repeat the problem and show me:

The metadata server log
sasauth-debug log
An output from this command (must be started as root): grep sasauth /var/log/secure

Re: AD Account is locked at SAS Server

Kurt_Bremser — Fri, 17 Nov 2017 13:22:05 GMT

Hi @alexal. It's not really a problem, as it can be solved by just giving the metadata server time to "drop the cache", or whatever it is.

I just had a case right now, where a user changed the password after failing to log in to the metadata server (because of password expiration), and could not log in to the metadata server for about half an hour. Then the EG connection profile sudddenly worked (with the already entered new password) without asking for credentials again.

We are using host authentication on AIX, the phenomenon has been there since at least 9.2 on AIX 5.3, and it persists with 9.4 TS1M2 on AIX 7.1

I can find no sasauth-debug in my !SASROOT/utilities/bin

Re: AD Account is locked at SAS Server

alexal — Fri, 17 Nov 2017 13:31:43 GMT

@Kurt_Bremser,

SAS relies on the underlying operating system and APIs to handle authentication request, so most likely the cache was somewhere on the system. Without an additional debug I cannot help. If you wish, open a track and ask to assign it to me, I'm sure we will find the root cause.

Re: AD Account is locked at SAS Server

Kurt_Bremser — Fri, 17 Nov 2017 13:45:22 GMT

Is KB

15231

really the one that you wanted to direct me to? It's the one that deals with the necessary setuid bits, but that is surely not a problem here.

I think you meant http://support.sas.com/kb/39/891.html

I have enabled the logging, will try if I can glean something from the logs if the effect happens again.

Re: AD Account is locked at SAS Server

alexal — Fri, 17 Nov 2017 13:47:26 GMT

@Kurt_Bremser,

You are right, sorry about that, I've copied the wrong link 🙂

Re: AD Account is locked at SAS Server

mfab — Fri, 17 Nov 2017 14:28:21 GMT

@alexal and @Kurt_Bremser: I really appreciate all your effort! I hope, you will track this thing down.

I can add, that we use Host authentication as well (SLES11.4 / SAS9.4M1).

Unfortunately I cannot enable sasauth-debug as of now and as quickly as Kurt.

Our timeline was roughly as follows:

t0 - "passwd --lock ..." on commandline

t1 - first login attempt: not possible (as expected)

t5 - "passwd --unlock ..." on commandline

t6 - login attempt: not possible (not as expected)

t7 - login via SSH: possible (as expected)

t8 - setting user passwort in credentials via Management Console + login attempt: no effect

t11 - removing user login credentials in Management Console + login attempt: no effect

t12 - removed user completely in Management Console, added user again + login attempt: no effect

t30 - randomly tried another login attempt: possible

So I would also think that there might be some sort of caching on the host side. Especially since I removed the login credentials and even removed and added the complete user (I would strongly hope that a new user has his credentials checked against the host, not against some cache).

However, I am wondering that SSH-Login was working properly way earlier than via SAS Metadata.

Hope this helps!

Michael

Re: AD Account is locked at SAS Server

alexal — Fri, 17 Nov 2017 14:41:54 GMT

@mfab,

...

I can add, that we use Host authentication as well (SLES11.4 / SAS9.4M1).

...

I am wondering that SSH-Login was working properly way earlier than via SAS Metadata.

SSHD uses PAM for the authentication. You can also switch SAS to use PAM for the authentication, even for local users:

http://support.sas.com/kb/49/432.html

Re: AD Account is locked at SAS Server

jakarman — Sat, 18 Nov 2017 20:07:26 GMT

Michael, with the omaconfig you have reviewed the internal account definitions
With that you can validate passwords against LDAP/Kerberos/AD but there is no immediate connection to the OS security itself.

It is a kind of working in ignoring security at het data/os-level and combine those to a shared grouped account that is maintained and owned by SAS. I hope SAS institute is not responsible or if they are. Anyway that way of working should be described well as losing auditablity traceablity by normal tools for that. --- high profile user, hurts ----

For unix security technical way of working, there are many options:
http://support.sas.com/documentation/installcenter/en/ikfdtnunxcg/66380/PDF/default/config.pdf review PAM / sasauth.
Getting into LDAP/AD depending on the size of the company there can be many LDAP/AD servers.
Updating the password in a windows environment and expecting it immediately active can be frustrating. At windows you get the immediate response of the update password as you are being connected to that one.
https://blogs.technet.microsoft.com/kenstcyr/2008/07/05/understanding-urgent-replication/   Normally it should be implemented well but you never know.

There is an other weird behavior of the sasauth module. It is that module checking the password while runnig at root-level.
What will happen when validating a user/password combination when there are delays/lock wait policies being activated at the AD origin. Will the user account behave als normal Windows or will there happen something different?
The latter having seen happening. Trying a locked out user several times and the delay time went ever up never being reset by a correct login. You must be very patient and confident to see the login happening after an hour or so.

Re: AD Account is locked at SAS Server

mfab — Mon, 20 Nov 2017 09:10:34 GMT

Hello all,

thanks for the suggestions so far.

I think I might be onto the issue (at least in my case):

Our sasauth.conf is configured as follows:

maxtries=5

maxtriesPeriod=60

maxtriesWait=300

So there should be a maximum wait of 5 Minutes. However I have seen longer wait times. My best guess is that the wait time of 5 minutes (in this example) is prolonged or refreshed everytime the user tries to login within the 5 minutes of the maxtriesWait.

So:

maxtriesWait set to 5 minutes

etc...

But then again this is just my guess. The maxtriesWait could also be accumulated for each unsuccessful login attempt while maxtriesWait is still active.

It would be good to know, how sasauth works when the maxtriesWait has been reached and another login attempt is made. The documentation (Configuration Guide for SAS® 9.4 Foundation for UNIX Environments) might be more helpful with a few explaining lines. Or am I searching in the wrong place?

Also it says that authentication might be

sasauth => Unix OS => /etc/passwd

sasauth => /etc/passwd

Our sasauth.conf says "method = pw". Does this mean we go through the Unix OS and then /etc/passwd or does this mean sasauth does the check versus /etc/passwd? This might be good to know in order to tighten the search, whether the OS might be involved at all in this process or if it a behaviour of sasauth.

Cheers,

Michael

Re: AD Account is locked at SAS Server

jakarman — Mon, 20 Nov 2017 21:18:56 GMT

Michael, yep that is the terrible thing we struggled quite a lot on that.
The updates time-outs on the OS and the SAS were done seperately not aligned to each other.

The pw checking was calling the OS and the OS being redirected to LDAP (AD) for only normal users as service accounts being local.

When this is also you case testing / validating will get more easy. Have at least two accounts availiable.
One being getting locked and at every attempt increasing the time (accumulating). The other as normal user just logging in an measuring the time in responding for normal usage.
Having a test for infra validation of a sas installation available you can try those settings in the sasauth.conf on that scenario changing values and just replay the user-locking and changing passwords. That environment should be very similar to the real one. A stand alone one on a desktop will not do that job.

Re: AD Account is locked at SAS Server

NZ_Hockey_Mum — Mon, 20 Aug 2018 23:41:12 GMT

I had the SAS server locking my AD account repeatedly. It was tracked it back to when I tested the SAS studio installation after which I had then changed my password. I did not logon to SAS studio again. The password was changed in the my profile from within SAS EG but somehow SAS would still pick up my old password stored by SAS studio run that against the SAS server and lock my AD account.

Removing the password from the SAS studio saved information worked for me.

Hope that helps.

Re: AD Account is locked at SAS Server

JuanS_OCS — Wed, 12 Sep 2018 13:14:09 GMT

Thanks a lot @NZ_Hockey_Mum ! Hope it can help many others.