SAS Temp Files and Windows Storage Performance

[SASExpo]

I am the Windows Admin supporting a SAS environment.

User profiles are written to a remote share on a high performance 6070 HPE SAN on vmWare 7.

I have reviewed this guide SAS Storage Best Practices - Paper Template (sas.com)

The issues we're having seem related to i/o but I wanted to get more confirmation that the guidance in the PDF above also applies to TMP file processing and storage i/o requirements.

When the below errors are encountered the entire SMB service crashes and the share is dropped, forcing a reboot.

Does high performance storage i/o also apply to TMP files for end user profile reading and writing? I am not a SAS user or SME so I know these are rudimentary questions. This issue is wreaking havoc on our end users. Good money was spent on high performance storage. But, perhaps there are additional LUN configuration for r/w for tmp files that should be done, but I assumed that is for actual SAS data and not end user Windows tmp files.

Thank you for your help!

[JuanS_OCS]

Hi there, excellent question!

Yes, the high performance I/O (thoughput) applies as well, and, in fact, specially, to the temporary files in SASWORK and UTILOC.

What is important to mention is that SASWORK and UTILOC are not recommended to be mounted in a SMB mount/share!

Regardless of the excellent perfomance you have behind at your shared storage level, SMB is not a protocol you want to use for this kind of purpose.

It is simple: either keep it local storage/SSDs, with a (proper) LUN (as you mentioned) or a SAS supported & tested Distributed File System such as GPFS, Lustre, Veritas, NetApp, etc. If you can keep it local with Premium SSDs, the best.

I also notice you are using Windows. Please mind that, generally speaking, there is more support for this kind of configurations on Linux or AIX or windows, from a driver development point of view.  Of course the vendors will always try to do their best. In some cases the drivers work flawlessly, but in many cases i have seen and keep seeing, there are workarounds (such as installing a Linux substrate/Cywin) and that is rarely flawless. 

[SASExpo]

Thank you.

Can you verify if these sample path tmp files are the type of files affected by the poor performance?

[JuanS_OCS]

Potentially, and depending on the usage of SAS EG, yes, they could be some of the culprits. 

Based on what you mention, chances are high that those are a big part of the issue.

Please mind, it is very hard to say without knowing better your architecture, infrastructure and actual usage. 

I would recommend some consulting hours from SAS Professional Services or a trusted SAS partner who can cover deeply the architecture details at the technical and functional level.

[Sajid01]

Hello @SASExpo 
From your posts what I understand is that (1) user profiles are being stored on a remote SMB server (2) The tmp files in the users %APPDATA% folder is crashing the SMB server. (3) Your question is whether the i/o guidelines apply to the user data too.

I am using that  SAS is installed on the server and SAS EG on the users/client  machine
As far as my understanding goes the i/o guidelines apply to the i/o throughput of executing  SAS code  on the server.(.)
SAS EG is Windows IDE for SAS and the temp files that you have shown are those created at the client end and not by the running SAS code. In my experience these tmp files have no long term value.
Typically a SAS EG user has a number of sessions open and along with the code, there may be windows with logs, open reports  and SAS datasets.

These temp files may be outcome of this action at the user level.  At the end of the session or the day the tmp files should go away OR may be safely removed.  (Something like tmp files created when somebody opens for example a word document for editing).
Admins typically do not have any role in the users workflow and in your scenario, as an admin, I would recommend keeping the users profile locally on the users machine.

[JuanS_OCS]

Agreed with the last comment of Sajid: specially on Windows you do not put your user profiles on a share, but local.
That’s why there is a Roaming area in the Windows profiles!

[Patrick]

Just adding to what @JuanS_OCS already wrote:

- There is an I/O Test Utility that allows you to measure throughput for read/write of files the way SAS does it. Usage Note 51659: Tool to test I/O throughput on Microsoft Windows platforms

- The following link found in the System Requirements for a SAS9.4 Foundation installation:  Best Practices for Configuring Your I/O Subsystem for SAS®9 Applications 

From what you shared it's highly likely that you need to repoint the SAS WORK/UTILLOC space to a location with better throughput.

@Sajid01 

Admins typically do not have any role in the users workflow and in your scenario, as an admin, I would recommend keeping the users profile locally on the users machine.

These ...\EGTEMP\... paths is where EG initiated SAS sessions typically store WORK and UTILLOC files. Read/Write to this locations happens from the SAS Server side (not client) and that's where one needs good I/O. What you propose would only apply for a locally installed SAS ("PC SAS").

[JuanS_OCS]

Adding to Patrick’s comment, running the IoTest tool is always a good idea for performance. However!
1. This will not test creating thousands of files, but a few very large files
2. As you want to test the performance for 50-70 users, you would need to run it with at least 50 sessions… I doubt even that server has enough RAM or enough storage.

In any case, please do read well the instructions of that tool.

[Kurt_Bremser]

I suspect that your users create more file handles on the server than the server can handle concurrently. After all, each EG session creates multiple (and I mean a lot) temporary files and keeps them open.

What is your SMB backend? A Samba server running on UNIX, or Windows?

[SASExpo]

Our users open a RDP session to a Windows terminal server that hosts the SAS client software, their user profiles for Windows are mapped to a share via group policy.

I am beginning to think that this is not ideal for the hundreds of temp files being generated per user per second, over the network to this SMB share.

Also, can anyone confirm that the architecture of using a RDS terminal server with the SAS components installed is definitely not ideal for 50 to 70 users at a time? 🙂

I know the answer to that - but confirm I'm not crazy telling mgmt that this is a poor decision to expect good performance on ONE server. 🙂

I suspect using only one server saves money? I'm not a SAS guy so I have no idea about that.

TLDR; our users RDP to a server that maps Windows profiles to a remote share (\\userprofiles\jondough) and I suspect this is a terrible idea for so many users on one server hosting the SAS client. Its no wonder the SMBServer service is choking.

[Sajid01]

Hello @SASExpo 
The general practice is to install the SAS client (SAS EG) on the user's local machine and the SAS   Server software   on the remote server. User's profiles are typically local to their machines.
With the arrangement you have SAS EG on a remote machine and all 50-70 users starting their session on the same machine would cause what you are facing. Definitely not an ideal scenario.

[JuanS_OCS]

You are not crazy and I think we can confirm that indeed.

This being said a Terminal Server for 50-70 concurrent users, regardless of this issue, is killing a fly with a cannon, aside incredibly expensive!

Do you know the saying divide and conquer?

Coming back to the issue, SMB is not the way to go for this design, I think we stablished that.

Leaving to you if you would go back to the drawing board and design (I would), here is a piece of extra knowledge:

In SAS EG it’s possible to change configurations under the file SEGuide.exe.config and tell it to place the temporary files in a different location like this:

<configuration>
<configSections>
<section name="Environment" type="SAS.EG.Configuration.EnvironmentSection, SAS.EG.Utilities"/>
</configSections>
<Environment>
<Variables>
<add name="TEMP" value="D:\MyTempFiles"/>
</Variables>
</Environment>
</configuration>

However, is up to you how you will do that for each user, since placing it for All users is just a formula for disaster.

[SASExpo]

I assume this is not for general Windows temp files, but specifically for SAS temp. My issue is Windows temp files as stated in the original and proceeding posts - although if you're saying that this option could redirect SAS specific temp files to a local drive, that would be a game changer for us - but we're still in a horrible architecture dilemma that I inherited - many users RDPing to 1 server using the SAS client.
Thanks!

[JuanS_OCS]

Yes and no. I don’t think Windows by default creates as many temp files that quick… however there might be either Windows configurations (eg Audit) or other non-SAS applications which might behave similarly and lead to similar situations, if that makes sense.

I feel you for inheriting that. Best of luck!!

[SASExpo]

So this is a per user config contained in a unique file independently for each user?