Architecting, installing and maintaining your SAS environment

SAS WORK and UTIL usage

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 13
Accepted Solution

SAS WORK and UTIL usage

Hello all,

I'm currently trying to figure out the best, fastest and most efficent way to monitor our SAS Work and Util directories. I have a few ideas, stemming from a manually run stored process, a batch job running ever 30 mins or so producing a report.

 

I was just interested in how other admins keep on top of these areas

 

I'm currently on SAS 9.4 GRID on REHL (santiago)

 

Any ideas or help will be appreciated.


Accepted Solutions
Solution
‎02-16-2017 03:22 AM
Trusted Advisor
Posts: 1,141

Re: SAS WORK and UTIL usage

Yeah, I read your post in your LinkedIn profile. It's going to be quite an achievement Smiley Happy

 

If the 100+  users will have direct exposure to the metrics, then you are done I suppose. For the others (I guess there are hundreds of other users), you can still query the ESM db and send some metrics by email. Or to expose their team leaders to the ESM GUI.

View solution in original post


All Replies
Trusted Advisor
Posts: 1,141

Re: SAS WORK and UTIL usage

[ Edited ]

Hello @sasadmin_tlove,

 

For the basic coverage, I would recommend you to monitor logical folders by one of your company's monitoring tools, such as BMC Patrol, SCOM, Nagios or anything else.

 

If you have SAS 9.4, you could be using SAS Environment Manager.

 

There is another great 3rd Party product named ESM (Enterprise Session Monitor - https://boemskats.com/esm/ ) that can give you the required details.

 

Having great experiences with both of them, keeping history and having visibility on the usage on Live. Awsome thing from the ESM tool is that you can find and drill down the information to a node or even job level.

 

 

Occasional Contributor
Posts: 13

Re: SAS WORK and UTIL usage

Hi Juan,

Thanks for the comment Smiley Happy

 

Yes we are one of the fortunate people to have ESM. and i spend pretty much most my time in it. its like nothing else on the market.

 

While ESM is a cut above anything else i've use, i need something that i can push output to users, something which can make them aware of their consumption - in the past we have users consumer over 1TB of space. Now in out new Linux environment there is half of that, and its only when we get "Sort Execution Failure" in jobs or workspace sessions that it too late.

 

 

Super User
Posts: 6,936

Re: SAS WORK and UTIL usage


sasadmin_tlove wrote:

Hi Juan,

Thanks for the comment Smiley Happy

 

Yes we are one of the fortunate people to have ESM. and i spend pretty much most my time in it. its like nothing else on the market.

 

While ESM is a cut above anything else i've use, i need something that i can push output to users, something which can make them aware of their consumption - in the past we have users consumer over 1TB of space. Now in out new Linux environment there is half of that, and its only when we get "Sort Execution Failure" in jobs or workspace sessions that it too late.

 

 


To this I only have three answers:

Quotas

Quotas

Quotas

Be proactive instead of reactive.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Occasional Contributor
Posts: 13

Re: SAS WORK and UTIL usage

Hi Kurt, I agree - however the problem there is for our Linux team in IT and the complex nature of how all the mount points are setup for each node (8 in total) and also- but that will still allow the workspace to grow then fail.

 

 

Super User
Posts: 6,936

Re: SAS WORK and UTIL usage


sasadmin_tlove wrote:

Hi Kurt, I agree - however the problem there is for our Linux team in IT and the complex nature of how all the mount points are setup for each node (8 in total) and also- but that will still allow the workspace to grow then fail.

 

 


The more complex a setup becomes, the more important it is to be proactive. Every person-hour you spend now in setting up a self-hardened environment will save you person-weeks to person-months or even person-years in the future.

 

Being lazy now will cost you big time, believe me. I'm speaking from experience.

 

My workspace never fails for all users, only for single ones, when they overrun their quota. Which is usually a moment where I then show them the mistakes in their code(s).

 

One problem is that the Linux admins often come "up" from the PC side of things instead of "down" from the mainframe/midrange/proprietary UNIX side, where certain types of thinking and behaviour have been drummed into people's brains over decades.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Occasional Contributor
Posts: 13

Re: SAS WORK and UTIL usage

I hear what you are saying Kurt, I heed your warning.

 

For me i have to put a lot of trust in our 3rd party IT company as i'm very new to linux (about 6 months experience) but due to spending so much time in back of this server, and being the one who help build the platform when it was on windows, i'm already starting to educate our offshore 3rd party IT company in a few things.

 

I very much agree with you, on the windows side we had over 15 different hard limit quotas in place and the entire ETL folder structure was controlled using the AD.

 

Luckily we have a plan, SAS are coming in soon and have setup workshops to hopefully work with our off shore 3rd party IT company, and get the environemnt tuned and fit for purpose. SAS have been amazing in this space before.

 

 

Trusted Advisor
Posts: 1,141

Re: SAS WORK and UTIL usage

Hi @sasadmin_tlove,

 

agreed, I love ESM Smiley Happy

 

Well, if you got ESM, then you already have all the information. Why not to query the ESM database to send reports to the users? You could even generate a little SAS Datamart with all the relevant information you want. Automation is great.

 

Occasional Contributor
Posts: 13

Re: SAS WORK and UTIL usage

Whats exciting is we role out ESM to our wider team on the 28th - so this will be 100+ users viewing the entire server usage in real time...

 

cant wait Smiley Happy

Solution
‎02-16-2017 03:22 AM
Trusted Advisor
Posts: 1,141

Re: SAS WORK and UTIL usage

Yeah, I read your post in your LinkedIn profile. It's going to be quite an achievement Smiley Happy

 

If the 100+  users will have direct exposure to the metrics, then you are done I suppose. For the others (I guess there are hundreds of other users), you can still query the ESM db and send some metrics by email. Or to expose their team leaders to the ESM GUI.

Super User
Posts: 6,936

Re: SAS WORK and UTIL usage

[ Edited ]

We run AIX and have quotas enabled on our WORK filesystem. Every ten minutes, this script is executed:

export OUTFILE=/temp/repquota.txt
echo datum `date +%d%b%Y:%H:%M:%S` > $OUTFILE
for FILESYS in fs1 fs2 fs3
do
  echo $FILESYS >> $OUTFILE
  repquota -u $FILESYS >> $OUTFILE
done

The program that receives this is

data quotarep;
infile '/temp/repquota.txt' dsd truncover;
format
  zeitpunkt datetime18.
  filesystem $30.
;
retain zeitpunkt filesystem;
input
  @  1 user    $8. @;
if user ne ' ' and user ne 'User' then do; /* keine Überschriftszeile */
  if substr(user,1,5) eq 'datum' then do; /* Datumszeile */
    input @ 6 zeitpunkt datetime18.;
    datum = datepart(zeitpunkt);
    call symput('ja', substr(put(year(datum), z4.),3,2));
    call symput('mo', put(month(datum), z2.));
    call symput('ta', put(day(datum), z2.));
  end;
  else if substr(user,1,1) eq '/' then do; /* Zeile mit Filesystem */
    input @ 1 filesystem;
  end;
  else do; /* einzelne Datenzeile */
    input
      @ 12 bquote  $1.
      @ 13 iquote  $1.
      @ 14 nblocks 9.
      @ 23 blsoft  9.
      @ 32 blhard  9.
      @ 51 ninode  9.
      @ 60 ilsoft  9.
      @ 69 ilhard  9.
    ;
    output;
  end;
end;
else do; /* Überschriftszeile überspringen */
input;
end;
drop datum;
run;

proc append data=quotarep base=sasuser.qrep&ja&mo&ta;
run;

proc sort data=sasuser.qrep&ja&mo&ta nodupkey;
by zeitpunkt filesystem user;
run;

proc copy in=sasuser out=admin memtype=data;
select qrep&ja&mo&ta;
run;

/* Makrovariablen für Vortag setzen */
data _null_;
gesdatum = input('20' !! "&ja" !! "&mo" !! "&ta", yymmdd8.) - 1;
call symput('gja', substr(put(year(gesdatum), z4.),3,2));
call symput('gmo', put(month(gesdatum), z2.));
call symput('gta', put(day(gesdatum), z2.));
run;

%macro aufbereitung(jj,mm,tt,fs,tag);

data qrep;
set admin.qrep&jj&mm&tt;
where filesystem="&fs";
run;

proc sort data=qrep;
by user zeitpunkt;
run;

data
  steigerung (keep=user anfang ende absolut relativ)
  verbrauch (keep=user ende blsoft);
;
set qrep;
by user;
retain anfang ende;
if first.user then
  anfang=nblocks;
if last.user then do;
  ende=nblocks;
  absolut = ende - anfang;
  if anfang ne 0 then
    relativ = (ende / anfang - 1) * 100;
  if absolut gt 0 then output steigerung;
  if ende gt 1000000 then output verbrauch;
end;
run;

proc sort data=steigerung;
by descending absolut;
run;

proc sort data=verbrauch;
by descending ende;
run;

data _null_;
set steigerung end=file_ende;
file "$HOME/public_html/statrep/steigerung_&tag..html";
if _n_ = 1 then do;
put '<HTML>';
put '<HEAD>';
put '<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">';
put '<LINK rel="stylesheet" type="text/css" href="statrep.css">';
put '<LINK rel="SHORTCUT ICON" href="/icons/allianz.ico">';
put '<TITLE>Steigerung im Platzverbrauch</TITLE>';
put '</HEAD>';
put '<BODY>';
put '<H1><IMG src="/allianz_logo.gif"><BR>Steigerung in ' "&fs</H1><HR><BR>";
put '<CENTER>';
put '<TABLE border=1 summary="Steigerung" class="Table" rules=BOX';
put 'frame=BOX style=" Text-align:CENTER;">';
put '<THEAD><TR>';
put '<TD class="Header" style=" Text-align:LEFT;">User</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">Anfangsbestand</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">Endbestand</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">absolute Steigerung</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">relative Steigerung</TD>';
put '</TR></THEAD>';
put '<TBODY>';
end;
put '<TR>';
put '<TD class="RowHeader" style=" Text-align:LEFT;">';
put user;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put anfang commax12.;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put ende commax12.;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put absolut commax12.;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put relativ 10.5;
put '</TD>';
put '</TR>';
if file_ende then do;
put '</TBODY>';
put '</TABLE>';
put '</CENTER>';
put '</BODY>';
put '</HTML>';
end;

run;

%mend;

%aufbereitung(&ja,&mo,&ta,/home,heute);

%aufbereitung(&gja,&gmo,&gta,/home,gestern);

Apologize for the German names.

Aside from the already created HTML, the resulting datasets allow analysis back over quite some time.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Occasional Contributor
Posts: 13

Re: SAS WORK and UTIL usage

Thanks for sharing this with me, i'll be more than happy to take a look.

 

Regards

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 11 replies
  • 263 views
  • 14 likes
  • 3 in conversation