topic Re: Is changing work directory actually to change the HOME directory? in Administration and Deployment

Is changing work directory actually to change the HOME directory?

fierceanalytics — Tue, 25 Aug 2020 16:26:57 GMT

Hello,

I wrote a nice do loop that has 6 layers of macros to test HPFOREST. I know the output, be it log or html, will accumulate to 'endanger' the size limit capped on the HOME/myuserID directory. So I wrote "printto to..." and "ods html path..., file=".

The long running job,after one night running, is working great. Only that one html file still looms bigger and bigger > 500MB, but another HTML file is on that non-home directory also bigger. My admin emailed me to delete that html. I told him I cannot kill that long job. That model has 7000 variations looking great....

The admin said he cannot relocate my HOME directory. I figured otherwise. I think I did once long time ago, but forgot details.

Google shows this blog https://blogs.sas.com/content/sgf/2018/05/18/how-to-change-your-working-directory-for-sas-with-the-dlgcdir-data-step-function/.... Changing working directory of SAS, as we did often in Python and R.

My question: is changing work directory actually to relocate HOME? if not, please how to relocate HOME, links, tips,....?

I think probably we need to go to Management Console with super admin rights. Thanks

Re: Is changing work directory actually to change the HOME directory?

Reeza — Tue, 25 Aug 2020 16:48:15 GMT

Do you need the ODS HTML output? Otherwise why not turn it off entirely?
A 500MB file seems unmanageable for many reasons.

I think you need to change the default location using the ideas here:
https://blogs.sas.com/content/sasdummy/2018/08/28/sas-current-directory/

Re: Is changing work directory actually to change the HOME directory?

Tom — Tue, 25 Aug 2020 17:14:27 GMT

Also how are you running SAS? Are you using some front end, like SAS/Studio or Enterprise Guide.

If you are running regular old SAS (base?foundation?Display Manager?) are you running on a Windows or Unix host?

In general when running a program from the command line the default location for the LOG and LISTING files is the current directory. You can change that using the -log or -print option on the command line that runs sas.

So example this sequeence of commands will move to you project directory (on some disk with real space allocated to it) and then run the SAS program named myprogram.sas and create the log file as myprogram.log. Any normal listing output should goto myprogram.lst (or perhaps myprogram.lis if you are somehow still running VAX/VMS operating system).

>cd /myproject_folder
>sas myprogram

Re: Is changing work directory actually to change the HOME directory?

fierceanalytics — Tue, 25 Aug 2020 17:57:21 GMT

HTML output critical. I am puzzled
because ods html redirects was on top but hpforest still sent diagnostics to Home. All the other macro, graphics away from home. Does procedure have to go HOME? I don’t see merit forcing that. Thanks

Re: Is changing work directory actually to change the HOME directory?

Reeza — Tue, 25 Aug 2020 18:24:41 GMT

Are you at least limiting it to what you need? Why not save the output to data sets instead?

Re: Is changing work directory actually to change the HOME directory?

fierceanalytics — Tue, 25 Aug 2020 18:50:21 GMT

Yes I have saved and computed my differential KS statistics in datasets. I still need html, though. I have cut out those unwanted ods output objects but variables importance.. must be there. If HOME is capped at 1GB, I feel like I am forced to sacrifice analytics. My iPhone has 64GB, it does even care if I fill up 40 GB, but modeling work is enterprise computing. I can split macro to 4 brackets to run sequentially but why? Cutting combination is not very straightforward And I have to keep notes.

Re: Is changing work directory actually to change the HOME directory?

Ksharp — Wed, 26 Aug 2020 12:53:14 GMT

The following code could check WORK pathname .

proc options option=work value;
run;

Re: Is changing work directory actually to change the HOME directory?

fierceanalytics — Wed, 26 Aug 2020 14:40:58 GMT

Thank you. This wakes up my old memory of writing lines like "options work= ;" to redirect. So, no Work is NOT Home. Work is where we process jobs like 'proc sort'. It is by design not to store permanent files.

So I modify my original question: how to relocate or redirect HOME directory?

I searched the internet and found many tips on how to relocate Work. I don't have Work directory problem.

The notion that log, sas, output, html files should be at different directory from data sets is understood. But the default installation size, which appears to be ~1GB?/user in 2020 (since when my HOME hits 400MB my admin called me, several times, and the total size he showed 92% collectively over many users are not very big), is apparently outdated. Now I recalled ~10 years ? I got similar calls for loading too much at HOME. I recall the offending total at that time was ~300MB when we mostly were not using html or ODS output in general. Now some procedures, in my case, HPFOREST, is replicating huge html at HOME while following my redirection of ods html path and proc printto and while the log files were successfully redirected as prescribed. The practical trouble is most enterprise installation follows default setting.

So, where is the simple 'one-stop-shop' manual on how to relocate HOME directory? Thank you.

Re: Is changing work directory actually to change the HOME directory?

Kurt_Bremser — Wed, 26 Aug 2020 14:58:30 GMT

To do something about the HOME directory, you need to work with your server administrator.

For one, because he/she's the one with the necessary permissions, and second, if I found a user creating big files where he/she is not supposed to, there's going to be a very substantive talk-to (on the server I am responsible for, it's not possible, anyway; see "one").

I am with you that 1 GB is not much nowadays; our current quota in /home is 3 GB per user. But you see there IS a limit, and it's not THAT large.

I would get in contact with SAS technical support to check what is actually using up all that space. I also question the value of HTML data of this size, as it will cause a problem for the user agent which tries to display that; browsers are already VERY memory-hungry with considerably smaller HTML pages.

And I am with you that there should be a means to redirect that traffic to a place where the limits are less restrictive, like WORK if your quota there is set to a sufficiently large value. And that is also a thing where SAS TS will be your best bet.

Re: Is changing work directory actually to change the HOME directory?

fierceanalytics — Wed, 26 Aug 2020 17:38:37 GMT

Thank you, Kurt.

1. No, I don't manage SAS administrators. Needless to say I don't have super, server rights....

2. Most, if not all, SAS administrators, my opinion (and you may differ), do not have tradition to reach out to SAS support.

3. In the case where they do like to work things out, which is this case I have, I need to feed them with sufficient materials for them to know what to ask for when contacting SAS. Most admin staff do not actually know SAS. They approach SAS like it is just another ...

I will focus on why HPFOREST procedure defies that redirection. I did some test and found only when the procedure reaches certain loops when the HTML starts to mirror-duplicate along side the non-HOME html creation. This is entirely different subject.

Thank you.

Jia

Re: Is changing work directory actually to change the HOME directory?

Tom — Thu, 27 Aug 2020 02:47:22 GMT

It is not clear to me what your actual problem is from your description. Its sounds like your program is writing files where you don't have room for them.

I seriously doubt you need to make any changes to your SAS setup. You just need to fix your program that is calling the HPFOREST to tell it to direct all of its output to a place where there is room. (or figure out how to have it produce less output).

You could open your own support ticket with SAS support to have them help you understand what files HPFOREST is creating and how to control where it creates them.

Re: Is changing work directory actually to change the HOME directory?

Kurt_Bremser — Thu, 27 Aug 2020 08:25:14 GMT

@fierceanalytics wrote:

2. Most, if not all, SAS administrators, my opinion (and you may differ), do not have tradition to reach out to SAS support.

They need to realize that one of the duties (and I use this word in a very serious manner here!) of a SAS administrator is to be the mediator between their end users and SAS. If a problem is encountered that they cannot solve by themselves (or find a sufficient answer why it can't be solved, so a work around is needed), they have to get into contact with SAS TS. Your organization pays a sh*tload of money for the SAS license, and access to TS is a main part of that, so not using it is a kind of negligence.

Your organization should have a IT helpdesk, so use that to create a ticket for your problem. That is the way problems are handled where I work.

If your pathway to SAS is defective, feel free to contact SAS TS on your own: all you need is your site number, which can be found by running PROC SETINIT, and technical details (SAS version, operating system, ...). You can open a track via web (go to www.sas.com, and from the menu, select Support - Support Services - Technical Support).

If your admins are mainly server administrators that do not feel responsible for anything up from the operating system level, then request that a dedicated SAS administrator is put in place; this might even be one of the end-users (like you) who develops the necessary skills (SAS provides courses for this, and your operating system vendor for the OS - see Maxim 15).

Re: Is changing work directory actually to change the HOME directory?

Nigel_Pain — Thu, 27 Aug 2020 08:40:19 GMT

I can't add enough likes to your response @Kurt_Bremser. I have been a sysadmin and a SAS admin for over 20 years. I have to say, I couldn't have survived without the help of TS, especially over the last 5-10 years. They are 3rd level support for us, and for any sensibly-minded SAS admin.

Re: Is changing work directory actually to change the HOME directory?

fierceanalytics — Thu, 27 Aug 2020 14:10:30 GMT

Well, the focus and the root of my post are not really whether SAS TS is useful or not. The key is : if you have 12 TS tickets open today, would you rather to have 8 opened instead? If now you know, by changing package default setting, is going to help reduce the traffic to TS queue. When a TS ticket is open, somebody has already spent time on the issue. And more people will spend more time on it. In 2020, I don't have to tell you opportunity cost of your time, 15-30 minutes if not more.

Another different subject with the same theme that I ran into, a lot: default memory setting with Enterprise Miner. 8GB RAM? That was 2 years ago. The admin told me it was not possible to change that. Two months later SAS TS walked him through to raise it to 20GB, nice? yes. TS helpful? sure. Only that it past the time I needed it. I had to abandon EM just for that sake. That was not a judgement call on EM or on SAS in general. This is driving a car: you have to get on whichever road allows you through on the moment for the project, with praise or complaint, or both.

Re: Is changing work directory actually to change the HOME directory?

fierceanalytics — Thu, 27 Aug 2020 14:15:56 GMT

You points are well taken. Sometimes， users draft to describe the issue. The admin copies into their contact with TS to get things done. We do work hand-in-hand.

Re: Is changing work directory actually to change the HOME directory?

JuanS_OCS — Thu, 27 Aug 2020 14:53:08 GMT

Hello @fierceanalytics ,

I feel your frustration. I think your SAS application needs indeed to be updated, or redesigned. Mostly at Infra level.

It seems to me as either your system administrators are not at the front lines, or the issues you raise via 1st line support do not arrive them.

There was a discussion long time ago that I sometimes revisit, it was interesting: https://communities.sas.com/t5/Administration-and-Deployment/Moving-SASUSER-to-a-Network-Share/td-p/198730

To me the issue is not very complicated. Indeed WORK it is not probably your HOME folder. HOME it is probably the SASUSER variable, or SASINITIALFOLDER. To get out of any doubt, you can run "%put _ALL_;" and find the values of WORK, SASUSER and SASINITIALFOLDER folders.

Once this is clear, you can confront this challenge from the SAS Application point of view, of from the Infra point of view.

From the SAS application point of view:

https://support.sas.com/kb/41/806.html

https://communities.sas.com/t5/SAS-Enterprise-Guide/redirecting-sasuser-default-path/td-p/182239

But this will work only on the following SAS sessions, not the running one. I am not aware of a way to change those folders dynamically within the session itself. It is also probably not a good idea to do so (risk of corrupting system)

Edit: I just happen to believe to recall an ancient method I used in the past to modify session variables on the fly, but cannot recall the details. CLIST and TSO commands. Does it ring a bell? In any case, I would not use it careless without the right support if never used it before.

https://support.sas.com/kb/40/451.html

http://v8doc.sas.com/sashtml/os390/z1-hfs.htm

http://www.math.wpi.edu/saspdf/os390/1117c01.pdf

https://documentation.sas.com/?docsetId=hosto390&docsetTarget=p0m1fwogu1px51n16i7cfy7zugzk.htm&docsetVersion=9.4&locale=en

https://v8doc.sas.com/sashtml/os390/z0216812.htm

From the Infra point of view:

- the easiest solution would be to increase your HOME quota. It can be done on the fly, and with no impact for running sessions.

- it is also possible to extend the disks/volumes on the fly, but I do not think this is your issue.

- a good solution, but your admins need to be extremely careful with (and aligned with you), is to create symbolic links (symlinks) to the folder that is getting full, pointing to another location with enough space. It can be also be done on the fly, with a "but". The files must be fully in sync with the new location right before the creation of the symlink.

All in all, I think that your issue lies closer to your system administrators and service desk than to the SAS application itself, or SAS Technical Support. I also think that you need someone to review the requirements of your solution at least every year or two, to check if needs to be re-designed.

I loved the comparison of SAS to a car. Keep it maintained and you should be happy with it.

Re: Is changing work directory actually to change the HOME directory?

Kurt_Bremser — Thu, 27 Aug 2020 17:05:03 GMT

@fierceanalytics wrote:

The admin told me it was not possible to change that. Two months later SAS TS walked him through to raise it to 20GB, nice? yes.

Such people have a name. It's Mordac.

Re: Is changing work directory actually to change the HOME directory?

JuanS_OCS — Fri, 28 Aug 2020 07:15:05 GMT

So true!