Hello all,
I'm currently trying to figure out the best, fastest and most efficent way to monitor our SAS Work and Util directories. I have a few ideas, stemming from a manually run stored process, a batch job running ever 30 mins or so producing a report.
I was just interested in how other admins keep on top of these areas
I'm currently on SAS 9.4 GRID on REHL (santiago)
Any ideas or help will be appreciated.
Yeah, I read your post in your LinkedIn profile. It's going to be quite an achievement 🙂
If the 100+ users will have direct exposure to the metrics, then you are done I suppose. For the others (I guess there are hundreds of other users), you can still query the ESM db and send some metrics by email. Or to expose their team leaders to the ESM GUI.
Hello @sasadmin_tlove,
For the basic coverage, I would recommend you to monitor logical folders by one of your company's monitoring tools, such as BMC Patrol, SCOM, Nagios or anything else.
If you have SAS 9.4, you could be using SAS Environment Manager.
There is another great 3rd Party product named ESM (Enterprise Session Monitor - https://boemskats.com/esm/ ) that can give you the required details.
Having great experiences with both of them, keeping history and having visibility on the usage on Live. Awsome thing from the ESM tool is that you can find and drill down the information to a node or even job level.
Hi Juan,
Thanks for the comment 🙂
Yes we are one of the fortunate people to have ESM. and i spend pretty much most my time in it. its like nothing else on the market.
While ESM is a cut above anything else i've use, i need something that i can push output to users, something which can make them aware of their consumption - in the past we have users consumer over 1TB of space. Now in out new Linux environment there is half of that, and its only when we get "Sort Execution Failure" in jobs or workspace sessions that it too late.
@sasadmin_tlove wrote:
Hi Juan,
Thanks for the comment 🙂
Yes we are one of the fortunate people to have ESM. and i spend pretty much most my time in it. its like nothing else on the market.
While ESM is a cut above anything else i've use, i need something that i can push output to users, something which can make them aware of their consumption - in the past we have users consumer over 1TB of space. Now in out new Linux environment there is half of that, and its only when we get "Sort Execution Failure" in jobs or workspace sessions that it too late.
To this I only have three answers:
Quotas
Quotas
Quotas
Be proactive instead of reactive.
Hi Kurt, I agree - however the problem there is for our Linux team in IT and the complex nature of how all the mount points are setup for each node (8 in total) and also- but that will still allow the workspace to grow then fail.
@sasadmin_tlove wrote:
Hi Kurt, I agree - however the problem there is for our Linux team in IT and the complex nature of how all the mount points are setup for each node (8 in total) and also- but that will still allow the workspace to grow then fail.
The more complex a setup becomes, the more important it is to be proactive. Every person-hour you spend now in setting up a self-hardened environment will save you person-weeks to person-months or even person-years in the future.
Being lazy now will cost you big time, believe me. I'm speaking from experience.
My workspace never fails for all users, only for single ones, when they overrun their quota. Which is usually a moment where I then show them the mistakes in their code(s).
One problem is that the Linux admins often come "up" from the PC side of things instead of "down" from the mainframe/midrange/proprietary UNIX side, where certain types of thinking and behaviour have been drummed into people's brains over decades.
I hear what you are saying Kurt, I heed your warning.
For me i have to put a lot of trust in our 3rd party IT company as i'm very new to linux (about 6 months experience) but due to spending so much time in back of this server, and being the one who help build the platform when it was on windows, i'm already starting to educate our offshore 3rd party IT company in a few things.
I very much agree with you, on the windows side we had over 15 different hard limit quotas in place and the entire ETL folder structure was controlled using the AD.
Luckily we have a plan, SAS are coming in soon and have setup workshops to hopefully work with our off shore 3rd party IT company, and get the environemnt tuned and fit for purpose. SAS have been amazing in this space before.
Hi @sasadmin_tlove,
agreed, I love ESM 🙂
Well, if you got ESM, then you already have all the information. Why not to query the ESM database to send reports to the users? You could even generate a little SAS Datamart with all the relevant information you want. Automation is great.
Whats exciting is we role out ESM to our wider team on the 28th - so this will be 100+ users viewing the entire server usage in real time...
cant wait 🙂
Yeah, I read your post in your LinkedIn profile. It's going to be quite an achievement 🙂
If the 100+ users will have direct exposure to the metrics, then you are done I suppose. For the others (I guess there are hundreds of other users), you can still query the ESM db and send some metrics by email. Or to expose their team leaders to the ESM GUI.
We run AIX and have quotas enabled on our WORK filesystem. Every ten minutes, this script is executed:
export OUTFILE=/temp/repquota.txt echo datum `date +%d%b%Y:%H:%M:%S` > $OUTFILE for FILESYS in fs1 fs2 fs3 do echo $FILESYS >> $OUTFILE repquota -u $FILESYS >> $OUTFILE done
The program that receives this is
data quotarep;
infile '/temp/repquota.txt' dsd truncover;
format
zeitpunkt datetime18.
filesystem $30.
;
retain zeitpunkt filesystem;
input
@ 1 user $8. @;
if user ne ' ' and user ne 'User' then do; /* keine Überschriftszeile */
if substr(user,1,5) eq 'datum' then do; /* Datumszeile */
input @ 6 zeitpunkt datetime18.;
datum = datepart(zeitpunkt);
call symput('ja', substr(put(year(datum), z4.),3,2));
call symput('mo', put(month(datum), z2.));
call symput('ta', put(day(datum), z2.));
end;
else if substr(user,1,1) eq '/' then do; /* Zeile mit Filesystem */
input @ 1 filesystem;
end;
else do; /* einzelne Datenzeile */
input
@ 12 bquote $1.
@ 13 iquote $1.
@ 14 nblocks 9.
@ 23 blsoft 9.
@ 32 blhard 9.
@ 51 ninode 9.
@ 60 ilsoft 9.
@ 69 ilhard 9.
;
output;
end;
end;
else do; /* Überschriftszeile überspringen */
input;
end;
drop datum;
run;
proc append data=quotarep base=sasuser.qrep&ja&mo&ta;
run;
proc sort data=sasuser.qrep&ja&mo&ta nodupkey;
by zeitpunkt filesystem user;
run;
proc copy in=sasuser out=admin memtype=data;
select qrep&ja&mo&ta;
run;
/* Makrovariablen für Vortag setzen */
data _null_;
gesdatum = input('20' !! "&ja" !! "&mo" !! "&ta", yymmdd8.) - 1;
call symput('gja', substr(put(year(gesdatum), z4.),3,2));
call symput('gmo', put(month(gesdatum), z2.));
call symput('gta', put(day(gesdatum), z2.));
run;
%macro aufbereitung(jj,mm,tt,fs,tag);
data qrep;
set admin.qrep&jj&mm&tt;
where filesystem="&fs";
run;
proc sort data=qrep;
by user zeitpunkt;
run;
data
steigerung (keep=user anfang ende absolut relativ)
verbrauch (keep=user ende blsoft);
;
set qrep;
by user;
retain anfang ende;
if first.user then
anfang=nblocks;
if last.user then do;
ende=nblocks;
absolut = ende - anfang;
if anfang ne 0 then
relativ = (ende / anfang - 1) * 100;
if absolut gt 0 then output steigerung;
if ende gt 1000000 then output verbrauch;
end;
run;
proc sort data=steigerung;
by descending absolut;
run;
proc sort data=verbrauch;
by descending ende;
run;
data _null_;
set steigerung end=file_ende;
file "$HOME/public_html/statrep/steigerung_&tag..html";
if _n_ = 1 then do;
put '<HTML>';
put '<HEAD>';
put '<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">';
put '<LINK rel="stylesheet" type="text/css" href="statrep.css">';
put '<LINK rel="SHORTCUT ICON" href="/icons/allianz.ico">';
put '<TITLE>Steigerung im Platzverbrauch</TITLE>';
put '</HEAD>';
put '<BODY>';
put '<H1><IMG src="/allianz_logo.gif"><BR>Steigerung in ' "&fs</H1><HR><BR>";
put '<CENTER>';
put '<TABLE border=1 summary="Steigerung" class="Table" rules=BOX';
put 'frame=BOX style=" Text-align:CENTER;">';
put '<THEAD><TR>';
put '<TD class="Header" style=" Text-align:LEFT;">User</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">Anfangsbestand</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">Endbestand</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">absolute Steigerung</TD>';
put '<TD class="Header" style=" Text-align:RIGHT;">relative Steigerung</TD>';
put '</TR></THEAD>';
put '<TBODY>';
end;
put '<TR>';
put '<TD class="RowHeader" style=" Text-align:LEFT;">';
put user;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put anfang commax12.;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put ende commax12.;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put absolut commax12.;
put '</TD>';
put '<TD class="Data" style=" Text-align:RIGHT;">';
put relativ 10.5;
put '</TD>';
put '</TR>';
if file_ende then do;
put '</TBODY>';
put '</TABLE>';
put '</CENTER>';
put '</BODY>';
put '</HTML>';
end;
run;
%mend;
%aufbereitung(&ja,&mo,&ta,/home,heute);
%aufbereitung(&gja,&gmo,>a,/home,gestern);
Apologize for the German names.
Aside from the already created HTML, the resulting datasets allow analysis back over quite some time.
Thanks for sharing this with me, i'll be more than happy to take a look.
Regards
The SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment.
SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.
Find more tutorials on the SAS Users YouTube channel.