SAS Connect Issue

[Vinnie_MOH]

Hi,

In our production environment our PC SAS users use the following autoexec to connect to the server. It prompts them for their credentials which they enter, and then connects to the server

options metaserver="saspd1mds"

     metaport=8561
    metarepository=Foundation
    metauser="SASConnectUser@saspw"
    metapass="{SAS002}XXXXXXXXXXXXXXXXXXXXXXXX"

       ;
signon server="SASApp - Connect Server";
/*password = _Prompt_;*/

However when I try and use this autoexec in our test environment(I have updated the code and changed metaserver, metaport, metauser etc to reflect test) I get the following error...

ERROR: A communication subsystem partner link setup request failure has occurred.

ERROR: Cannot start remote process.

ERROR: Remote signon to SASAPP - CONNECT SERVER canceled.

I have also tried running the code below as well but get another error.

%LET MYNODE=SASTSTMDS;

OPTIONS REMOTE=MYNODE COMAMID=TCP;

FILENAME RLINK 'C:\Program Files\SASHome\SASFoundation\9.3\connect\saslink\tcpunix.scr';

SIGNON;

ERROR: A communication subsystem partner link setup request failure has occurred.

ERROR: The connection was refused.

ERROR: Remote signon to MYNODE canceled.

We have recently upgraded our environment to 9.3 and was told the Test environment was set-up exactly like Production. In each environment we have a Metadata Server, Application Server and Web Server all installed on AIX, and our client machines are Windows 7. No one ever uses the Test environment and there have been no changes to the environment, so I'm a little confused as to what is going on. I have checked the server logs and SAS Support but I haven't been able to find anything useful.

Has anyone out there experienced something similar or have any ideas on what I could try? I am definitely no SAS expert so any help would be much appreciated.

Thanks,

Vinnie

[Scott_Mitchell]

Hi Vinnie,

I am absolutely no SAS technical guru, however a quick Google search resulted in the following.

http://support.sas.com/kb/6/522.html

Perhaps your IT department can make some progress knowing the above.

Regards,

Scott

[Vinnie_MOH]

Thanks Scott. I did find this SAS Note earlier and is something I have already checked.

[SASKiwi]

The first thing I would ask is: has anyone else successfully connected to your test server? If yes check what they did for it to work. If no then there may well be a problem which is blocking everyone.

A search through SAS notes will bring up many potential reasons for your errors including Scott's one.

At this point you could consider raising a track with SAS Tech Support. They will be able to advise you on further diagnostics to do to identify the cause.

[Vinnie_MOH]

Thanks for replying SASKiwi. I have asked someone else to test and they get the same error. I thought it may have been something with the SAS Test Connect User I have created as well, but I have been able to successfully create a Connection Profile for this user in SAS MC.

I actually have had a track open with SAS Support for the last couple of weeks, but they haven't been able to sort it yet, so that's why I have put something up on this forum.

Thanks,

Vinnie

[Kurt_Bremser]

Have you tried to connect to the server port via telnet? That can at least give you an idea which process is listening for incoming SAS/Connect requests.

[jakarman]

You are using the new approach of SAS/connect with 9.3. It would help when understanding the classic way and this one.

The spawner. (server side)

The classic SAS/connect is having spawner process visible at the OS level. Started by a OS-script containing all options 

The metadata based SAS/connect is having a spawner process visible at the OS level. Starte by a OS-script getting the options from a metdatabase

SAS(R) 9.3 Intelligence Platform: Application Server Administration Guide (Initial Configuration of the SAS/CONNECT Server)

The server (server side) please do not get confused it are just script file that are started

the classic SAS/connect is referring the spawner port and starts the script for the sas-process. This script file starting a SAS process is not being parameterized unless you build you own Connect script. That scripting language is a segregated part within the SAS/connenct documentation. SAS/CONNECT(R) 9.3 User's Guide

The metadata based SAS/connect is having that script with all other parameters being defined as part of metadata SAS/App servers. You can recognize all parameters being there as originally defined in the metadatabase. You can configure at  .../Lev-/SASApp/Connect   the OS usermod configuration-file in the same way as for a WorkspaceServer , Stored process Server etc.

No matter which method, classic/metdatabase, the started SAS session must have a valid OS-user being defined capable (authorized) doing the work you want.

The metadata-server approach of all parameters is still requiring started spawners new is that iit is  for each one defines by a appserver. It still needs to be defined all Connect-options now not with a signon statement but needed for each known appserver a user want to get connected. I am wondering how to solve this when crossing boundaries definitions maintained by different metadatabases.

Remark:

The use of Rlink file is inherited from SAS V6, afterwards is was better an more easy to use the SAS-config setting.  That is: sasconfig SASSCRIPT= SAS/CONNECT(R) 9.3 User's Guide and using CSCRIPT at the signon statement.   

Now your issue question.

- You are connecting to "saspd1mds"

  I hope this is a DNS-name when that is a single name it must be in the same network segment. Crossing network segments is asking fully names

- you are connecting the default app-server SASApp.

  You did not segregate you usergroups in different ones to accomplish the business hierarchy?

- you are using a SAS internal account "SASConnectUser", this one can read SAS metadata.

  How does that one possible start a SAS process on the OS-level? Where is the account for that.

  Are you using a shared account for that being defined in the metadata with its associated password and is that correct working?

  Are you expecting the users are using their personal account (defined at OS-level) in what way?

The error message you have is telling you there is something wrong at starting that sas script at the OS-level.

[Vinnie_MOH]

Thanks for replying Jaap.

To answer some of your questions:

1. saspd1mds is our production metadata server. saststmds is our test metadata server. So basically the autoexec runs successfully in production using the SASConnectUser@saspw account. Therefore I have created a new account in test called SASTestConnectUser@saspw, encoded the password and updated the autoexec with the test server details.

2. I have also tried using the fully qualified name but I get the same error in test

3. Yes we are connecting to the default SAS-App server

4. We have a very basic set-up here, but I have mirrored the access in test off production

5. Yes,  SASTestConnectUser@saspw has read metadata permissions (I have based this off SASConnectUser@saspw in prod)

6. We had an external  SAS consultant create the original autoexec and SASConnectUser for production, when we upgraded our environment last year. However they didn't document the process. Like I said I have tried to base this test account from the production account but obviously something is not right. I'm just wondering if they made a change to a config file on the server as well?

7. I have tried updating the autoexec with my credentials and other local sas accounts but get the same error. I have also compared the OS permissions between environments, but have had no luck.

Thanks,

Vinnie

[jakarman]

Vinnie,

The --@saspw indication is an indication that user has only SAS-metadata access rights it is an SAS internal account.

You can connect to a metadata-server with that but as is not existing on the OS level you can not run any SAS processes with that.

You described a situation it succeeded in doing so, there must be a trick behind the curtains you have missed.

I am trusting the spawner port is working (default 7551).

You could check the mirror of production process is some server-name within the metadata could still point to you production server.

A SAS/connect session on a server is running by an OS-account.  You can verify that on your prod machine. Choose a silent moment on that machine, start the connect session. Check the running processes before and after (ps -ef ...). When verifying that I hope you will find a process started by your personal key. When you find a process started by a generic account that is important information as then you should ask yourself where that one is coming from.

Let us make a checklist:

a- verify the SAS/connect is attempted on your indicated machine not on a other.

  Search the metadata en SAS/connect server loggings for those events.

b- verify the account that SAS/connect is running at OS-level of your operational machine (this cannot be the SAS internal account)

c- verify the authdomain setting that is (possible) associated with your Connect-server definition in the metadata

    This can be the hidden trick to use different users/passwords.   

d1- having that account (b) verify the scripting and settings at the OS-level are correct

  Do a login with that account at the OS-level.

  Go to the (config) .../lev1/SASApp/ConnectServer and start the *.sh script found there (use X11 mode) 

  When there are OS-errors or SAS errors it will give better understandable error messages. Solve those when occurring.

  It could be as easy as a missing home-dir for the used account.

d2- having that account verify the way the SAS-metadata to get that account with password from the metadata. It must be somewhere there as you are not typing that

  It could be a group membership with your SAS internal account, but somewhere there must be a login-field in the SAS-metadata having that necessary information.

[jakarman]

Assuming the port of the connenctspawner is 7551 you could also try to connect in not metadata defined scripting

As the 7551 is connected to the SASAPP connectserver scripts

OPTIONS COMAMID=TCP;                    /* should be the default  */

%LET MYNODE=SASTSTMDS 7551;    /* this is the dns-server name and port number of the connect spawner */

SIGNON MYNODE;                                /* this is the signon with all default settings. perhaps more options to try SAS/CONNECT(R) 9.3 User's Guide */

Using the sigon command you can use serverv to show the metadata definitions. That is the info available with metauser / metapass.

When you use this way to connect I expect a pop-up for the OS-user. You can define that one with the user-= password= options

[Vinnie_MOH]

Thank you Jaap for all your suggestions. I will try these today and let you know how I get on.

Thanks,

Vinnie

[SASKiwi]

Since you can connect OK to the test SAS server via SAS MC that suggests that the server is fundamentally working OK. I suggest you look at the SAS metadata server log for evidence of what is happening when you connect from PC SAS. This log should have error messages relating to your failed connections and may offer some further clues as to what is wrong. Jaap's test is a good one too.

[Vinnie_MOH]

Hi Jaap,

Thanks again for all your help so far.

The Connect Spawner Port is configured for 7552 in our Test Environment.

I have tried checking the processes running on our Production server before and after starting the autoexec and the only entries that are different are the following:
vellis 18481230 25821360 0 15:11:36 - 0:00 sasels 12 9 13 2 12a37e
vellis 19071220 5439520 1 15:11:37 pts/1 0:00 ps –ef
vellis 25821360 5701760 16 15:11:36 - 0:00 /opt/sas/sas93/sw/SASFoundation/9.3/sasexe/sas -dmr -noterminal –nosyn

I have also been checking the metadata logs in both environments, and from comparing them both, it looks like the connection for SASTestConnectUser is being accepted.
Prod:
SASConnectUser@saspw. Encryption level is Credentials using encryption algorithm SASPROPRIETARY. Peer IP address and port are [::ffff:10.64.7.110]:49170 for APPNAME=SAS.
2014-04-09T09:10:40,919 INFO  [07234153] 134718:sas - Client connection 134718 for user SASConnectUser@saspw closed.
2014-04-09T09:10:52,276 INFO  [07234155] :SASConnectUser@saspw - New client connection (142042) accepted from server port 8561 for user SASConnectUser@saspw. Encryption level is Credentials using encryption algorithm SASPROPRIETARY. Peer IP address and port are [::ffff:10.64.7.110]:49173 for APPNAME=SAS.

Test:
2014-04-09T09:59:44,674 INFO  [00002437] :SASTestConnectUser@saspw - New client connection (118) accepted from server port 8562 for user SASTestConnectUser@saspw. Encryption level is Credentials using encryption algorithm SASPROPRIETARY. Peer IP address and port are [::ffff:10.64.4.140]:56108 for APPNAME=SAS.
2014-04-09T09:59:45,023 INFO  [00002448] 118:sas - Client connection 118 for user SASTestConnectUser@saspw closed.

The local sas account is running the Connect Spawner:

Prod:

bash-4.2$ ps -ef | grep Connect

     sas  5636288        1   0   25 May      -  0:00 /bin/sh -p /opt/sas/sas93/config/Lev1/ConnectSpawner/ConnectSpawner.sh start2_tag

     sas  6881508  5636288   0   25 May      -  0:45 /opt/sas/sas93/sw/SASFoundation/9.3/utilities/bin/sastcpd -service 7551 -shell -netencralg SASPROPRIETARY -xmlconfigfile /opt/sas/sas93/config/Lev1/ConnectSpawner/metadataConfig.xml -logconfigloc /opt/sas/sas93/config/Lev1/ConnectSpawner/logconfig.xml -sasspawnercn Connect Spawner - SASApp -logfile /opt/sas/sas93/config/Lev1/Logs/ConnectSpawner_saspd1cts.log

bash-4.2$

Test:

$ ps -ef | grep Connect

     sas 11337962        1   0   09 Apr  pts/1  0:00 /bin/sh -p /opt/sas/sas93/config/Lev2/ConnectSpawner/ConnectSpawner.sh start2_tag

     sas 12517574 11337962   0   09 Apr      -  0:00 /opt/sas/sas93/sw/SASFoundation/9.3/utilities/bin/sastcpd -service 7552 -netencralg SASPROPRIETARY -xmlconfigfile /opt/sas/sas93/config/Lev2/ConnectSpawner/metadataConfig.xml -logconfigloc /opt/sas/sas93/config/Lev2/ConnectSpawner/logconfig.xml -sasspawnercn Connect Spawner - SASApp -logfile /opt/sas/sas93/config/Lev2/Logs/ConnectSpawner_saststcts.log

[2014-04-10 14:50:08 sas@alsasts2] ~

I have then tried logging in as this user and starting the ConnectServer.sh and get the following...

$ cd SASApp
$ ls
appserver_autoexec.sas            SASEnvironment
appserver_autoexec_usermods.sas   sas.sh
appservercontext_env.sh           sasv9.cfg
appservercontext_env.sh.Original  sasv9_usermods.cfg
BatchServer                       sasv9_usermods.cfg.old
ConnectServer                     sasv9_usermods.cfg.older
Data                              StoredProcessServer
PooledWorkspaceServer             WorkspaceServer
Reports
$ cd ConnectServer
$ ls
autoexec.sas               logconfig.apm.xml
autoexec_usermods.sas      logconfig.trace.xml
ConnectServer.sh           logconfig.xml
ConnectServer.sh.older     sasv9.cfg
ConnectServer.sh.today     sasv9_usermods.cfg
ConnectServer_usermods.sh
$ ConnectServer.sh

SAS(R) TCPIP REMOTE LINK PORT=43967    SESSION ESTABLISHED.^C$

I'm not sure if it is related, but I also found the following SAS Note: 18527 - "ERROR: Cannot start remote process."

We also get the following errors in the Connect Spawner Console log:

ERROR: (SASXKINI): PHASE 3 KERNEL INITIALIZATION FAILED.
ERROR: Unable to initialize the SAS kernel.

Then if I look in the sav9_usermods.cfg, on Test (/opt/sas/sas93/config/Lev2/SASApp) there appears to be an entry for SASUSER (-SASUSER /sasdata/sasuser )

/*
* sasv9_usermods.cfg
*
*   This config file extends options set in sasv9.cfg.  Place your site-specific
*   options in this file.  Any options included in this file are common across
*   all server components in this application server.
*
*   Do NOT modify the sasv9.cfg file.
*
*/

-WORK /saswork
-UTILLOC /sasutil -SASUSER /sasdata/sasuser
-VALIDVARNAME any
-BUFSIZE = 64k
-UBUFSIZE = 64k
-IBUFSIZE = 32767
-BUFNO = 10
-UBUFNO = 10
-IBUFNO = 10
-ALIGNSASIOFILES
-MEMSIZE = 512M
-SORTSIZE = 256M

The thing I don't understand is why this entry is not in our production environment. The file in prod is very similar except for that entry (not sure why it is not on the next line, but I have also tried dropping the entry to the next line). In Test, we have a directory called sasdata/sasuser but can't see this in Prod. I can see a sasusers directory in both environments, but only the directory in production has user profiles.

I hope this all makes sense. Look forward to hearing from you.

Thanks,

Vinnie

[jakarman]

Vinnie, There is a lot there that makes sense.

1/ Your personal account in the prod-system is "vellis" (pid=19071220) and is that account running the spawned process for connect (pid=25821360) . That sasels process is coming from your SAS/connectproces (Pid 18481230 - 25821360)  

It is not the internal SASConmectUser@saspw. As you have proved this, where is the account "vellis" coming from in production?  

As you are saying to do a login with that internal account, the result is something different. why? how? What are we missing?

2/ The metadata server log is as you have given are describing correct actions and the mentioned users are coming in.   

You changed the portnumbers being the suffix 1-=series to 2-series. This is smart and sensible as very recognizable also for firewalls/networking domains.

In production there is process started, no immediate close. In test it is closed immediate. We already are knowing is, the log is having that same information too.      

3/  The local generic account "sas" is running the connectspawner (pid 5636288) and is that doing since may for prod. The test also running (pid=11337962) since 9 apr.

The connect spawner port are nicely there 7551 for the prod machine and 7552 for the test machine.

Within these startups a logfile is mentioned (ConnectSpawner_saspd1cts.log ConnectSpawner_saststcts.log) is there information in? when not you could exchange that temporary for the logconfig.trace.xml (intention debugging) at your test-machine,  restart the Connectspawner and do some tests.

I am missing your attempts to connect using port 7551(prod) 7552(test) directly with a signon command in the classic approach.

I am also missing the results for query-ing the SAS metadata with a signon command and than doing the Serverv.....

As you change port numbers and a lot more I am convinced you did a clean configuration trying to achieve a similar installation. 

Running a process since May, that is in the future in his moment (10 apr 2014). It is telling that process was never restarted since last year.

Hopefully nobody did change anything on that machine. All security information in a Unix approach is cached at startup and this cache is never refreshed.

Imagine a security change having impact on your processes. You will only notice that as you need to restart them. Never mind spawners are not very critical suffering on this.

4/ Starting that ConnectServer.sh script you are seeing the handshake of a Connect-session start. That SAS process is responding with a Portnumber that is used in the to be port-port communication of SAS/connect sessions.

Your note 18527 is of the old time (v8) but is valid for a possible root-cause in this case a failing SASUSER location.

The usermods file somebody added changed by hand, normally it is left empty. The name is "usermods" intentionally.

The Work must be a valid location and accessible to all sas-users it is something like /tmp of Unix. /tmp is setup with the access rights 3777 (gid-bit and sticky bit) for valid reasons. What is your access to /saswork?

The SASUSER should be a valid location for your users. Normally that is: "/home/<user>". You definition is strange in this config-file.

For my head. It is possible to start this ConnectServer.sh with options so you can use that in terminal mode. have an X11 server started, your DISPLAY environment being set and use parameters  -terminal -dmr (and ? ) with that script.  It will not give you that port-number but a normal SAS session.

----

Something different to check. SAS/connect (and much more) is dependent on special security configuration. I expect you have needed something with PAM (new machine) and several special files. 15231 - SAS® modules that must have the setuid bit set to root in the UNIX environment. These files make it possible to do that low level (sensitive) change user actions.                

[Vinnie_MOH]

Hi Jaap,

Thanks for replying.

Yes, my personal account is vellis. These are also my AD credentials. An account exists in Metadata which uses a Back-End configuration to authenticate with LDAP.

I have also created internal accounts in both environments for myself, and have tried running the autoexec in prod, which runs successfully, but fails in test.

Prod:

Test:

10   /*password = _Prompt_;*/

11   options metaserver="saststmds.moh.govt.nz" /*10.130.21.141*/
12
13       metaport=8562
14       metarepository=Foundation
15       metauser="Vinnie Ellis@saspw"
16       metapass=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
17
18          ;
19    signon server="SASApp - Connect Server";

ERROR: ERROR: No logins found.
ERROR: Retrieving login information based on AuthenticationDomain from the SAS Metadata Server
       failed.
ERROR: Unable to retrieve the SAS/CONNECT server definition "SASAPP - CONNECT SERVER".
ERROR: Remote signon to SASAPP - CONNECT SERVER canceled.
20   /*password = _Prompt_;*/

This is what I see in the ConnectSpawner_saststcts.log. I am pretty sure I have checked this previously but haven't seen these Permission Denied error which are concerning. I have checked the internet and forums but haven't been able to find anything useful regarding this error.

08.45.58.99: Copyright (C) 2002-2011, SAS Institute Inc., Cary, NC, USA. All Rights Reserved.
08.45.58.99: SAS Job Spawner for Open Systems version 9.3
08.46.00.79: ERROR: The tcpSockWrite call failed. The system error is 'Broken pipe'.
08.46.00.82: ERROR: The tcpSockWrite call failed. The system error is 'Broken pipe'.
09.59.47.42: ERROR: Launch Failed. Cannot execute program "/opt/sas/sas93/config/Lev2/SASApp/ConnectServer/ConnectServer.sh", error = 13 (Permission denied).
09.59.47.42: ERROR: Search permission is denied for a directory or file is not an executable file
09.59.47.42: Error spawning child process, use -logfile option to capture details.
10.21.44.62: ERROR: Launch Failed. Cannot execute program "/opt/sas/sas93/config/Lev2/SASApp/ConnectServer/ConnectServer.sh", error = 13 (Permission denied).
10.21.44.62: ERROR: Search permission is denied for a directory or file is not an executable file
10.21.44.62: Error spawning child process, use -logfile option to capture details.
08.45.45.41: ERROR: Launch Failed. Cannot execute program "/opt/sas/sas93/config/Lev2/SASApp/ConnectServer/ConnectServer.sh", error = 13 (Permission denied).
08.45.45.41: ERROR: Search permission is denied for a directory or file is not an executable file
08.45.45.41: Error spawning child process, use -logfile option to capture details.

I have tried running the signon command in the classic approach but get the following errors for both environments...

4    OPTIONS COMAMID=TCP;
5    %LET MYNODE=saspd1mds.moh.govt.nz 7551;
6    SIGNON MYNODE;
NOTE: Remote signon to MYNODE commencing (SAS Release 9.03.01M1P110211).
ERROR: A communication subsystem partner link setup request failure has occurred.
ERROR: The connection was refused.
ERROR: Remote signon to MYNODE canceled.
7    OPTIONS COMAMID=TCP;
8    %LET MYNODE=saststmds.moh.govt.nz 7552;
9    SIGNON MYNODE;
NOTE: Remote signon to MYNODE commencing (SAS Release 9.03.01M1P110211).
ERROR: A communication subsystem partner link setup request failure has occurred.
ERROR: The connection was refused.
ERROR: Remote signon to MYNODE canceled.

I'm not the best with SAS code so am a little unsure how to use serverv with the classic signon command with serverv.

I did try it using my autoexec and got the following...

50   options metaserver="saststmds.moh.govt.nz" /*10.130.21.141*/
51
52       metaport=8562
53       metarepository=Foundation
54       metauser="Vinnie Ellis@saspw"
55       metapass=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
56
57          ;
58    signon serverv="SASApp - Connect Server";
NOTE: Server=                SASApp - Connect Server
      Remote Session ID=     local
      ServerComponentID=     A5586FBU.AV00000E
      Remote Host=           saststcts.moh.govt.nz
      Communication Protocol=TCP
      Port=                  7552
      Prompt= No
      SpawnerCommand= /opt/sas/sas93/config/Lev2/SASApp/ConnectServer/ConnectServer.sh
      AuthDomain= DefaultAuth
      LoginName= Login.sasdemo.77
      Wait= Yes
      SignonWait= Yes
      Status= Yes
      Notify= No
59   /*password = _Prompt_;*/

I will have a read of the link you sent regarding the security configuration for UNIX environments now, to see if that may help.

Thanks again,

Vinnie