Email from EG when bandwidth exceeded

[heleenw]

Hi all,
Hope I can describe my idea in a clear way...
So today I was thinking I might convince EG to send me an email, if in a certain column usually I see values like seven, eight or perhaps every once in a while six or seldom a nine; and suddenly I would be seeing 40 or 100 or 0 ... that would be weird; it'd be something worth looking into; so the bandwidth in which I expect my stuff to be sort of gets 'exceeded' . So then I want to be mailed about that.

Now I know how to configure the email part, given I would still need to somehow obtain emailserverdetails etc. - but I need help to specify exactly when I want to be mailed.

I suppose I need an EG process flow? Is that correct?

Some data set import is probably step a ; and then an E-mail sending step is step c .
But what do I put in between as step b... ?

Like <10 or so, but I would want it to not refer to absolute values... In the sense that it could work, but would be better if it looked at what average values usually come through or so... then determine deviation from the means...?
Just thinking out loud now, I am no statistician 😞
Can anyone help on part b please or is my trail of thought faulty... or perhaps, is EG not the right tooling for this idea,... or is the idea not feasible or are there dependances ?

[ballardw]

If you don't want to report with hard coded values then you will need to do something to establish boundaries for when to report.

If you are worried about extremes that occur intermittently you might look at the RANGE statistic for a time period. That is basically the largest value minus the smallest. So if the range is "large" you might have an extreme.

Another thing to look at might be Standard deviation but it is not as sensitive to one or two values if you have 1000s. But a single value 2 or 3 standard deviations from the mean might be an outlier.

You might do some investigation on a "large" set of data (several months perhaps) just to learn about the distributions of your data values to determine some likely basic rules and provide parameters.

Proc Means asking for mean, min, max, std, mode, median and maybe the Q1 and Q3 quantiles should show you a bit about your current data

Do you have some sort of change seasonally like temperature going down in winter and increasing in summer? If so you should create the summaries by groups that might reflect that.

What goes into your data makes a difference on what may be useful to look at. I worked with a lot of instrument data related to weather and soil conditions once upon a time to help diagnose possible instrument issues. So I had seasonality to set some bounds for "reasonable" values; some instruments had natural ranges such as humidity (really cannot go below 0 or over 100 but some conditions would make the instruments report that), or barometric pressure (0 would mean there is no local atmosphere). Others I looked at rates of change (air temperature is unlikely to go from 10 to 100 degrees in one hour).

The idea of what to check will tell you how to make the report. Generally a set of values, which would be  in a data set, and rules on how to compare those values to your data flow. 

How to report? Provide some actual rules you come up with after looking at your data. 

[heleenw]

Hello Ballard

Thanks so much.

Sounds like a good idea to try to go for RANGE after having explored with
PROC MEANS. But I am in need of a bit of coding help.

I can do step a, define libname and tell EG which dataset to look at and
which column, by myself.

I can also do step c, define the needed parameters for the mail-sending to
work, by myself.

But how do I go about the step b, do you perhaps, from your weather coding
experience, have an example for me about the RANGE statement where I can
see this live in action, because I am unsure of how this would work?

Basically I think amounts that are far too low or far too high, extremes as
you call it, are indeed what I am looking for and what I would like SAS to
mail me about ...

If I set the email for instance to occur weekly, so if the range were small
usually weekly I would not get a mail ; but one week suddenly a big range
turns up, it is weird and I would know it already in the beginning of that
week...

Had you, or anyone else with these weather instruments, already had
anything like this coded so I could view an example ... or shall I just go
ahead and try to sort this out on my own and test? Just IF RANGE bigger
than x Then send mail ; IF RANGE smaller than x Then do nothing or is this
too simple?

Great help thus far by the way.

[jimbarbour]

Well, you would need to define what trigger levels would cause an email to be generated, but if you keep some kind of history of the average value of the columns you're interested in, that shouldn't be too difficult.

I find it easier to just use regular SAS to send emails rather than using SAS EG.  Let me know if you need sample code for this.

What you would need to do, and there are other ways to do this, but this is what came to mind, is something along the lines of:

1.  First you would run a Data step that compares the current maximum value to the average value and if that value is exceeded, then it sets a return code.

2.  Second, you would code a step that executes conditionally -- if the return code is a certain value.

At a high level, it might look like this:

%LET	Return_Code				=	0;

DATA	_NULL_
	SET	&Lib..Avg_Value_History;

	IF	Curr_Max_Value_Col_12	>	Hist_Avg_Max_Value_Col_12	THEN
		CALL	SYMPUTX	('ReturnCode', 4, 'G');
RUN;

%IF	%BQUOTE(&Return_Code)		=	4	%THEN
	%DO;
		%Send_Alert_Email;
	%END;

Jim

[heleenw]

Hi Jim
Thanks for your kind reply & offer.
After having played around with it a little I find that I do actually stand a better chance of understanding this when I can see some sample code from your weather-measurement practice (and possibly a quick word with it)... could you post something or send to me privately, or...?

[jimbarbour]

Below is some code from one of the jobs in my area.  I'm not sure who wrote this; I might have done it differently, but hopefully it's sufficient as an example.

Some notes:

• You need to have a TO, a FROM, and a SUBJECT on the filename.  Your system may have some defaults set up, but generally these should be present.  You may also optionally include a CC and a BCC on the file name.
• The critical thing on the filename statement is to code EMAIL after the filename.
• In your Data step, you should include a FILE statement that references the email Filename.
• Notice in the below example that the email Data step is executed conditionally.  If &SysErr is > 0 then the email is not sent.  In your case, you probably would set some kind of limits that define what your normal range of values is and then send the email if the current value falls outside that range.  You could set these limits in a Data step that also includes an email filename (that might be easier in your case) or you can execute a macro as in the below example.

Please ask questions if you have any.

%macro gen_email(Email_Address_List, Sender, Environment, Table, Nbr);
	FILENAME Mailbox EMAIL
		to		=	("&Email_Address_List")
		from	=	"&Sender"
		subject	=	"&environment Table Lock Alert! Please close *** &Table *** ASAP. (alert #&Nbr)"
		;

	%if &syserr	>	0 %then
		%do;
			%put &Err1  occurred previous to the email step;
		%end;
	%else 
		%do;
			/*email that no errors*/
			DATA _NULL_;
				FILE Mailbox;
				put "/********************  automated message - do not reply to sender  ********************/";
				put " ";
				put "Team,";
				put " ";
				put "&environment Table Lock Alert! Please close ASAP:";
				put " ";
				put "                    &Table ";
				put " ";
				put "This message will be sent every 3 minutes until table is closed.";
				put " ";
				put "This is alert #&Nbr..";
			RUN;
		%end;
%mend;

%gen_email(Some_Email@XXX.com, another_email@XXX.com, Development, Acct04p, 12);

Jim

P.S.  The above assumes that you have something like the below in your SASV9.cfg file.  In the below, you would replace "YYYY.XXX.org" with whatever server your organization uses for email.

/* set email options */
-EMAILSYS SMTP
-EMAILHOST YYYY.XXX.org
-EMAILPORT 25

If you're not sure what those values are, you can run the following, and the values will be written to the log.

PROC	OPTIONS	OPTION=(EmailSys EmailHost EmailPort);
RUN;

If the above are not in your SASv9.cfg, you can add that code to your cfg file, if you have access.  If you don't have access, you can set those values inside your SAS program like so:

OPTION	EmailSys=SMTP;
OPTION	EmailHost=YYYY.XXX.org;
OPTION	EmailPort=25;

[SASKiwi · Accepted solution]

IMHO, you typically use email if you want to let others know about a problem. If it is only for you why not just create an exception or tracking report in a folder where you can just go and view it? Sending an email sounds like an unnecessary complication.

[jimbarbour]

@SASKiwi, 

Not that I disagree entirely with you, but for some processes, if they're critical enough -- or if they so seldom have problems that you might forget to check such a folder, I actually send an email to my smart phone which pops up as a text message.  That way, I can jump on a  critical problem or get notified of something that so rarely occurs that I might otherwise miss it.  

Each to his or her own, I suppose, but that's been helpful to me, particularly the critical issues.

Jim

[AlanC]

First of all, I think email is fine to use. Everyone likes their own means of communication and email works well. So does SMS and other means. Email/SMS show up while you are anywhere which is nice.

That out of the way, when I send email I use a service. Our solution involves our own services but it is easy to use SMS or email services (ex. MailChimp using proc http. Look here: SAS Help Center: Syntax: PROC HTTP PROC HTTP Statement

You will probably be using a POST command so follow that. 

Also, there are other services other than MailChimp. Pick whichever one works for you and do a  REST call. REST makes everything easy w/o having to reinvent the world. There will also be REST calls for SMS.