topic Re: connectivity issues with EG2 and EG4 in SAS Enterprise Guide

connectivity issues with EG2 and EG4

deleted_user — Tue, 26 Feb 2008 08:19:09 GMT

I seem to have problems with EG4 sessions that are disconnected after some time, which I dont recall on EG2. I wonder if the way EG4 connects to the server tier is different from the way EG2 connects, or whether EG4 keeps a persistent connection across the network even after the script compilation has ended.

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 17:54:35 GMT

We are having this exact same problem. EG disconnects from the server if it remains idle for too long or even if submitted code is taking more than an hour to run. I've spoken to several folks at SAS and they swear it is not anything EG is doing but is instead probably due to some kind of firewall on the server that disconnects after a period of time. But my IT people swear it is not due to any firewall or other settings on the server side. We don't know where to turn now. Any suggestions?

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 18:20:10 GMT

I am under the impression that EG only communicates to the sas server under certain conditions \ to perform logging on both server-client end \ to authenticate the user credentials during login \ to send instructions to run data and proc steps \ to update the client when the server has completed all steps \ when the data is routed through EG because of SAS-access without passthrough.

Given the above, I suspect that perhaps the EG2 and EG4 client might act differently at the network layer due to authentication issues or logging issues. I'll be doing a packet capture some time to do a comparsion between the 2 clients.

I am not too concerned about sessions getting disconnected if it only resulted in usability inconveniences, but a disconnected session also means the workspace is being terminated prematurely and your session output is lost too (saving the output automatically isnt feasible for us due to disk space constraints)

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 18:26:56 GMT

The IT people are lieing to you. NOT intentionally, though.
It's an idle login timeout thing. Since this is a non-terminal remote session, there is no terminal activity, so the OS (tcp/ip?) times out.

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 18:33:18 GMT

I noticed this happens more often if connecting (from EG4) to a unix SAS server and hardly with a wintel unix box. If this were the case, shouldnt I notice the same problem (of sessions being terminated or disconnected) with EG2?

Once the session terminates, the workspace data is gone and thats alot of wastage. Is there anything to mitigate the problem or to keep the session alive then?

thanks for your advice on this one,

-Josh

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 18:44:12 GMT

It is a common security practice to put a timeout on an account, especially in the Unix world. The way to eliminate the problem is for the sysadmin to remove the timeout (I don't remember how this is done). Corporate security policies may not allow this. In which case, higher up managers need to battle this out. What is more important, the timeout/security, or getting their reports and not wasting people's time.

Usually, the Unix admins are too busy and take the easy way out to do things. If there is absolutely no need for the SAS user community to have to have a telnet session on the server for shell script generation, then it is possible, with a little more work, to allow the login, but to disallow a shell session (can't telnet, rlogin, rsh, ssh, etc. into the box). This is similar in windows to not allowing a remote desktop connection, while still allowing connections (shares, application access, etc.).

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 19:20:22 GMT

I disabled the firewall and the problem seemed to have been resolved for now. But why is it pegged to the unix pam credentials? I had always assumed that once you've set how EG performs authentication, it only involves how EG checks for username and password during the inital login, but not for the rest of the session. Does the server-client session require EG to re-authenticate itself at the server end (albeit silently and with cached credential details)?

I wonder if this is something that could be resolved using a common metadata server that centralises the login credentials of all the wintel and unix boxes. Is metadata server able to obtain the credential details from AD too? Message was edited by: Joshua

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 19:56:36 GMT

You are forgetting system layering.

EG and SAS ride on top of an OS and underlying systems.

EG is running on a workstation, only.
EG uses SAS's Integration Technologies (IT) to start and connect to a remote SAS session.
BUT, that remote thing is on a remote box, a separate piece of hardware from the workstation, with a different OS between the box and SAS. Part of IT services is connecting to the OS on the box, not just SAS. That is the authentication method that you specified for EG, but, if the OS/box has a timeout feature to it, it is outside the control of SAS, and since it is between SAS and the box, it can override anything SAS may want to do.

All processing on a box is controlled by the OS kernel's scheduler. All modern OS's are pre-emptive, which means that the OS can take control of the processor away from a user process (the scheduler can take control away from most any other process), whenever it wants to. The scheduler's all have prioritization schemes, all of which mean that some higher priority processes get to run on the processor before others with lower priority. The security features will run at a higher priority and can check on stuff and terminate/kill things in-spite of what those things may think they can do.

This layering is generally refered to as "layer" or "ring".
Some of these layers/rings are hardwired into processors for security/control reasons.
The lowest layer used to be layer 1 where the kernel lives.
Now the lowest layer is "ring 0" where a "hypervisor" lives to control virtual machines/servers. A guest OS then is rooted in ring 1, with User/application programs running in rings 3 or 4 or some such. Interrupt handlers live either in ring 1 or ring 2. Device drivers usually are in layer 2 or 3. Other OS services may run in layers 2, 3, or 4, depending on criticality. After that, there are the priority schemes of the scheduler itself. Again, interrupt handlers, besides running in a lower ring, also generally have a "higher" priority. So that they can get their work done, reading from disk, interacting with the network adapter, etc.

While EG and SAS may be talking to each other, that communication is contained within layers of networking protocols:
1) the physical electronics and physics of transmitting/receiving the data from one device to another (switch port to NIC port on the server).
2) the data link layer that controls the meaning/grouping of the ordered 1's and 0's decoded by the electronics into bytes of data.
3) the network/IP layer that manages the sending and receiving of packets of data from one box to another (why there is an ip address).
4) The session layer (TCP/UDP) that packetizes the data and controls the quality of service of the communications (ordering, connectedness, successful reception, etc.)

The OSI model has 7 layers, I remember the presentation and application layers, but forget one of the layers to make all 7.

The TCP/IP stack is contained within the OS and there maybe/are security attachments/hooks that are controlled within the OS below SAS. A local software firewall is such a hook/attachment that has priority/control over any application's (SAS's) communications. Even a hardware firewall can sit between the workstation and the server, and interfere with SAS's functioning, outside of SAS's control/configuratino.

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 20:17:50 GMT

That makes sense. Though I had assumed (perhaps wrongly) that the communication between the EG to the SAS IT component of the server is at an application tier which is unfettered by typical pam constraints (e.g. denied login after certain number of failed logins). Is the EG session connecting to SAS IT actually running off a shell session instead of an instance thats encapsulated by the application process? That would explain why and how EG is limited by shell login policies.

Given that many customers run in a non-heterogeneous environment, I would hope to see some info on this (from sas) in their knowledge base. As it is, I am now doing alot of guess work and using a packet analyser to infer whats happening at the network layer. I am sure this is something other sas customers have experienced and possibly resolved or mitigated in their own sites, which was why I had hoped to glean different ideas through the forum null

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 20:41:53 GMT

Yes, it is a shell session.
IT's spawner is actually a shell script process that issues the "sas ..." command.
.profile for the user applies, and is necessary because the base directory for "sas" has to be in PATH, or you have to fully qualify the path in the spawner. Also, a proper SAS installation creates some additional environment variables for SAS's use.

This is why you can make use of a local sasVx.cfg in the user's home directory and SAS_USER_CONFIG and SAS_SYSTEM_CONFIG environment variables for custom configuration settings.

The beauty of Unix here (I don't know about Windows) is that if you have an LDAP tool that has integrated with AD to control user login authentication, it is possible have the same home directory for all users, and thus one customizable sasVx.cfg file (or divide them into groups with common home directories for each group).

This sasVx.cfg file would then have the startup options you care about for your server, and you don't have to touch/edit the base sasVx.cfg file in the SAS home directory (preventing buggering it up by accident). If the local one is deleted, it is easier to recreate the variation than the original. Also, you don't have to copy the original.

Use of a symbolic link would allow having one common sasVx.cfg appear in each group's home directory, that is unalterable by them.

When creating the logic server, the "Files" property for "Initial folder for file navigation" should be set to this home directory to keep the users from being able to do any system mischief from within EG. Don't be lazy and let them have "system root" access. Message was edited by: Chuck

Re: connectivity issues with EG2 and EG4

deleted_user — Fri, 07 Mar 2008 21:01:09 GMT

If thats the case, then that would explain why the EG session is terminated pre-maturely. However, we noticed that the problem of pre-mature terminations was easily solved by disabling the firewall (which isnt much of a solution). Hence, its hard to guess whether the firewall or the unix user login policy kicked in first. I was wondering if using a metadata server might be an easier way to centralise login credentials for all my wintel and unix sas boxes, but this might not fully mitigate the session-time out problem too.

If it were caused by the shell session ending after a long idle time, then I'd wonder why the same problem does not surface when using EG2 to connect to the same box.

From what I hear from SAS support, the EG client also connects to the server periodically when it attempts to perform logging. But I am not sure if this logging actually stops once all proc and data steps are concluded

Re: connectivity issues with EG2 and EG4

deleted_user — Mon, 10 Mar 2008 00:51:33 GMT

Just of curiosity, are your user sessions all running off temporary workspaces in the WORK library? If the problem of session time out cannot be resolved, I wonder if setting the user's session to run off a permanent folder will mean that the session will (complete the steps before it times out) or simply leave whatever work needs to be done in the permanent location so that even if the session dies, the resulting data is persistent and outlasts the session

Re: connectivity issues with EG2 and EG4

deleted_user — Mon, 10 Mar 2008 07:28:58 GMT

I finally got around doing a packet dump and I noticed a few things which I wonder might be the common traffic seen between EG (client) and SAS(server with SAS IT)

* keep alive packets being sent every 5 minutes from client to server (NBSS procotol);
* echo request from server to client (smb);
* echo response from client to server (smb);
* server sending info to client providing info on the data mounted paths and permissions (smb);
* the client sending to the server at src port1991 (stun-p2) to dest port: 445 (microsoft-ds);

I am surprised this is the case, since I would assume that the smb information should be between the SAS server and the data tier. But at the least, I am able to have a better idea whats happening at the network layer now

Re: connectivity issues with EG2 and EG4

ChrisHemedinger — Thu, 13 Mar 2008 01:56:22 GMT

Wow, I'm very impressed with the amount of network brainpower you all have poured into this thread.

From an EG application point of view, I can tell you this:

- EG (the app) does not send keep-alive signals to the server when the app is idle. If a lower layer does that, I'm not aware of it.

- While a server job is running, EG polls the server every so often (30 seconds I think, or maybe as few as 5 seconds) to collect any log output. You can watch this happen if you have a verbose SAS program that runs for a while and you open the log window while it's working. EG collects and updates the log within the project.

Chris

Re: connectivity issues with EG2 and EG4

deleted_user — Thu, 13 Mar 2008 03:12:45 GMT

I came from a computational mathematics background (where we would dwell on subjects such as parallel programming, e.g. MPI in C, and algorithm optimisation), so its as much a personal interest as it is an obligation to master all the dominant technologies in each work stint. Theres a lot that has been said in the white papers for NT and win2003, but not much yet on what can or needs to be done for EG4, SAS 9.1.3, and to optimise data paths.

Thats one thing I find comforting when dealing with Oracle, because they are comfortable as a solution provider and systems integrator given the various domains acquired by Oracle. They'd be able to tell you how to tune the file system, the physical and logical architecture to obtain optimal and reliable performance, and also anticipate the various concerns customers have with regards to deployment.

I dont suppose many customers have the luxury of purchasing a SAS server purely for SIT/UAT/development, so alot of tuning can only be performed on the production box. This gives customers like us little luxury to perform iterative tests to resolve usability or systems integration problems we face on a day to day basis. The only route is to observe the server and client and infer certain conclusions. Not the most productive way, but its a start

Re: connectivity issues with EG2 and EG4

deleted_user — Thu, 13 Mar 2008 13:13:06 GMT

As Chris@SAS said I too am impressed by the brainpower in this tread. If anyone can solve the problem you guys can. But since I am a research psychologist by training who found himself in the role of SAS Administrator I have to admit that I am having difficulty following you and applying what has been said to solving my disconnect problem.

From what I gather you all agree that EG only communicates to the SAS server under certain conditions and is running on a workstation only, thus it is not responsible for severing the session with the server. You all seem to agree that there is something on the server that is causing the disconnect and making me lose all of my session output. Is that correct? By the way is anyone else out there having this disconnect problem?

Much of your conversation though seems to be centered around solutions in a UNIX environment. I'm am working on a Windows server, so I'm am trying to get solutions for that environment. I read that Joshua disabled the firewall and the problem seemed to have been resolved for him. My company's IT folks say that while there is an active firewall between the server and the outside world, the firewall that is between the server and my client is already disabled. Is there something else there related to firewalls that is being missed?

Chuck mentioned that it is a common security practice to put a timeout on server account. However the IT folks here showed me that they do not have these timeouts active. On the server, under "Computer Management" and under "Session Properties" the "Idle session limit" and all other limits are set to "Never." It seemed pretty clear cut. Is there something being missed?

Any help would be valued. Thanks.

Re: connectivity issues with EG2 and EG4

deleted_user — Thu, 13 Mar 2008 13:49:26 GMT

Theres a simple way to prove that the firewall isnt part of the problem. Run one EG session connecting to a server in the same subnet, and another connecting across the firewall to another server. With the right controls and careful observation, I think it would be easy to infer how and why EG performs the way it is (after a few iterations), even if its a black box to us.

Given that many of us are familiar with OR, I find it ironic that as mathematicans we spend effort trying to optimise business decisions and mathematical computations, yet end up troubled by system and application layer bottlenecks.
-
I suspect the way EG connects to windows and to unix differs. I am not sure why that is the case, but I think that the login session could be disconnected by the client, the server and by the network layer. So there are 3 segments we'll need to look into Message was edited by: Joshua

Re: connectivity issues with EG2 and EG4

deleted_user — Thu, 13 Mar 2008 14:21:54 GMT

Ok, different bias for a new audience (not Joshua).

Are you sure it is timing out?

Start up EG, new project, select a large dataset on a remote server so that it is brought into the project. Wait a long time (after the suspected time out) and see if you can scroll through the dataset. DO NOT scroll through the dataset on initially bringing it into the project, as EG caches some of the information, and so could give a false indication. Also, after the long wait, and after the scrolling test, can you select another data set?

Re: connectivity issues with EG2 and EG4

deleted_user — Thu, 13 Mar 2008 17:50:07 GMT

Chuck, I tried your test of bringing in a dataset, waiting until the disconnect, and seeing if I could scroll through the dataset.

After the disconnection I could not scroll through the dataset. The error was "The object invoked has disconnected from the client." I could not call a new dataset in either. Error included "A call to the metadata repository has failed."

That seems to be a timing out issue. Do you have another idea in mind?

Re: connectivity issues with EG2 and EG4

deleted_user — Thu, 13 Mar 2008 19:07:35 GMT

I am running you through some troubleshooting.

Ok, good, that confirms that there is definitely a timeout issue.
I didn't want to assume, because for all I knew, you were running code that was aborting/abending or hitting an endsas statement.

Next step, when the OS guys showed you the timeout values being set to "never" was that on the server that holds the metadata repository, or the application/SAS server or both? Or, is SAS and the metadata repository on the same server?