topic I just want to make sure I am thinking of this correctly in New SAS User

I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 15:30:45 GMT

I have to examine if the dataset has any anomalies:

Would you find this using proc univariate?

Examine if there are any association among the variables

Would I use proc ttest for this?

I just want to make sure I am thinking of this correctly?

Re: I just want to make sure I am thinking of this correctly

PaigeMiller — Thu, 06 May 2021 15:37:50 GMT

@sportguy3089 wrote:

I have to examine if the dataset has any anomalies:

Would you find this using proc univariate?

UNIVARIATE will find some anomalies, but others it may not find. What types of "anomalies" could be expected?

Examine if there are any association among the variables

Would I use proc ttest for this?

PROC CORR is a way to find some types of associations among the variables, but it will not find other types of associations among the variables. More information about the variables in the data set (and whether or not they are numeric or character) is needed. More information about what "association" means would be helpful.

Re: I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 15:38:27 GMT

I am trying to find info on bank marketing. Banks often market their financial products through phone calls. Assume that you are a data mining expert who works in the marketing campaign team at a bank. You are tasked with analyzing their marketing data to better understand future customers.

Re: I just want to make sure I am thinking of this correctly

PaigeMiller — Thu, 06 May 2021 15:48:05 GMT

Yes, but what "anomalies" would you expect to find? What do you mean by "association"? Are your variables all numeric, or are some of them character?

Re: I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 15:50:34 GMT

I got an example of what I got. What would the outlier be? would it be student's t, sign, and signed rank? I just want to make sure I thinking of this correctly.

Re: I just want to make sure I am thinking of this correctly

PaigeMiller — Thu, 06 May 2021 15:52:31 GMT

My questions remain unanswered. I cannot help you unless you address the questions I asked. The questions cannot be addressed by SAS output, you must define the terms you are using.

Re: I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 15:52:35 GMT

It should variable due to being variables in the data set.

Re: I just want to make sure I am thinking of this correctly

sbxkoenk — Thu, 06 May 2021 15:53:59 GMT

Hello,

What you want to do is fitting propensity-to-buy models on (historical) marketing data.

You can then profile typical customers of financial product XYZ and call the false positives (those with a very high probability to have the product XYZ but they haven't got it yet).

With base SAS you will not make it. You need to dive into statistical modelling, binary classification to be more specific. You need SAS/STAT for that and even better Enterprise Miner (SAS 9.4) or Model Studio (Visual Data Mining and Machine Learning in VIYA).

By the way, if you speak about anomalies: you need to specify if you want to detect univariate outliers or multivariate (case-wise) outliers. A multi-variate outlier will possibly not be detected with univariate detection techniques. But, you're right, in your modelling, univariate and multivariate outliers should be removed (don't be too harsh!!) or smoothed.

Good luck,

Koen

Re: I just want to make sure I am thinking of this correctly

sbxkoenk — Thu, 06 May 2021 16:01:48 GMT

Hello,

You are not thinking about univariate outlier detection correctly.

The values Student's t, Sign, and Signed Rank are values of the test-statistic for your hypothesis test (and your hypothesis tests whether the mean equals zero). Testing if the mean equals zero has nothing to do with univariate outlier detection.

Go to blogs.sas.com and search for outlier. You will get multiple useful hits!

https://blogs.sas.com/content/?s=outlier

Koen

Re: I just want to make sure I am thinking of this correctly

Reeza — Thu, 06 May 2021 16:03:56 GMT

Anomalies do not have a specific technical definition, they're defined in context of the data. For example for a sales of $0.01 cents to a client would be an anomaly and likely a bad one. A sales of $2B to a customer may also be an anomaly but a good one. How you deal with an anomaly depends on what you're trying to do, I do a lot of work in fraud detection and outliers for fraud are very specific and defined, ie flag all customers who billed in the top 0.01 % in drug billings and another that's flag all providers who work with only 1 group. Anomalies are not just based on the data.

For a purely numerical analysis point of view, for numeric variables you may want to trim the bottom 1/5/10% depending on your data. For categorical variables you'd likely do frequency counts and combine smaller groups into more equally sized groups.

Re: I just want to make sure I am thinking of this correctly

Reeza — Thu, 06 May 2021 16:04:32 GMT

And as a quick FYI - the free SAS Statistics 1 course will cover a lot of these topics and will probably save you time in the long run.

Re: I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 16:42:44 GMT

In the example the outlier would be .6 and 1048 correct?

Re: I just want to make sure I am thinking of this correctly

mkeintz — Thu, 06 May 2021 16:56:20 GMT

@sportguy3089 wrote:

I am trying to find info on bank marketing. Banks often market their financial products through phone calls. Assume that you are a data mining expert who works in the marketing campaign team at a bank. You are tasked with analyzing their marketing data to better understand future customers.

If "you assume that you are a data mining expert", then you need to know the actual attributes of the marketing data values maintained by the bank before you can apply your expertise. Which variables are (1) nominal, (2) ordinal, (3) interval? As others have pointed out, identifying anomalies would not be done the same way for each type. And you can't rely on a single measure to be suitable for determining associations between each of these variable types.

And of course, a little SME (subject matter expertise) would be helpful in determining where to look first. Since you refer to phone call marketing, I presume one would look at day-of-week, time-of-day, length-of-call, content of the marketing message, who answered (if available), whether the phone number is cellular or landline, demographics of the telephone area/exchange (when calling landlines), as well as the criterion variables (signing up for a service, calling back, maybe length of call etc.).

Re: I just want to make sure I am thinking of this correctly

Reeza — Thu, 06 May 2021 17:15:49 GMT

@sportguy3089 wrote:

In the example the outlier would be .6 and 1048 correct?

No. Why is 0.6 and outlier and not 0.9, or 1? they're only 0.6 apart when your scale goes to 1000?

It depends on context, you cannot analyze data without understanding it.

If this was age for example, all values shown would be outliers. Unless maybe it was age in months 🙂

If I had nothing else but the data shown, I'd be somewhat OK calling anything over the 900 mark an outlier. But what if the next value in the series prior to 728 was 500 and 728 was an outlier? Even if you wanted just a numerical analysis this wouldn't be enough. You'd also need to look at the distribution of the data via the histograms.

@sportguy3089 wrote:

In the example the outlier would be .6 and 1048 correct?

Re: I just want to make sure I am thinking of this correctly

sbxkoenk — Thu, 06 May 2021 17:18:43 GMT

Hello,

The very lowest and very highest value(s) that you correctly identified are not necessarily outliers (let alone anomalies). Outliers are not necessarily anomalies and vice versa as pointed out by someone above. Outliers are a numerical concept while anomalies can be found with subject matter expertise alone. Mostly outliers and anomalies coincide of course.

Univariate outlier detection looks at the whole distribution of the variable to find out which observations can be marked as outliers. The bottom or top 1% observations (or the extreme observations as labelled by PROC UNIVARIATE) are not necessarily outlying. An extreme value can be as probable as the mean value (in a uniform distribution for example).

Look at the blogs I referred to above to find outliers.

After reading of the blogs let us know if you need more assistance.

Koen

Re: I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 17:38:07 GMT

I wanted to see if this will work?

proc univariate data=sashelp.shoes PLOT;

run;

To see the output and see if I see outliers?

Re: I just want to make sure I am thinking of this correctly

Reeza — Thu, 06 May 2021 18:05:18 GMT

What's your definition of an outlier? Explicitly? You first need to determine your logical criteria and then implementation is easy. The code is the easy part of this work, deciding what/how to do things is the hard part.

An explicit definition is something like:
Any values in the top and bottom 1 percentile is considered an outlier.

Now for percentiles, you also need to consider how you want to handle tied values.
Note that you would make a rule for each variable, not a broad rule as above for all variables. That's a very broad approach, kinda like trying to sharpen a pencil with an axe.

Now what are you planning to do once you've identified these outliers? Remove them?

You may actually get better answers if you explain what you're trying to do overall. What's the context of the analysis problem you're facing that makes you think you have outliers in your data? Are outliers good or bad or both? Do you want to identify them for what reason?

Randomly happened to walk through a review of call center analysis earlier this morning with my analysts. We found calls that were less than a minute which are likely disconnects/wrong numbers so we decided to remove those for example. It usually takes a few minutes to authenticate and verify a user so those are not valid for the analysis we're doing (first call resolution, how many customer service calls were resolved on the first call). Longer calls, like 5+ are our complex calls s those are filtered out for separate analysis and further text analytics.

Re: I just want to make sure I am thinking of this correctly

sportguy3089 — Thu, 06 May 2021 18:14:21 GMT

Just to see if there are outliers that is it 🙂

Re: I just want to make sure I am thinking of this correctly

Reeza — Thu, 06 May 2021 18:16:31 GMT

What's your definition of an outlier? Explicitly?

Re: I just want to make sure I am thinking of this correctly

Reeza — Thu, 06 May 2021 18:17:59 GMT

Not trying to be a pain in the ass, but if you can't answer that question you definitely cannot code the solution because the requirements are unclear.