Help using Base SAS procedures

How to count outliers

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 12
Accepted Solution

How to count outliers

[ Edited ]

Hi All,

I have a program to genarate a data set from normal distribution with outliers.

I belive the following program genarate a data set with 5% of outliers.

 

 %let N = 100;

data CN(keep=x); call streaminit(12345);

do i = 1 to &N; if rand("Normal", 100, 16) then

x = rand("Normal", 100, 4); else

x = rand("Normal"); output; end; run;(Ref: Simulation data with SAS, By Rick Wicklin).

 

I want to estimate the propotion of the outliers to make sure my data set genarate  5% of outliers.

If any body have any idea , please help me.

 

 


Accepted Solutions
Solution
‎11-20-2015 10:28 PM
Respected Advisor
Posts: 4,659

Re: How to count outliers

I added a random number to the dataset (rnd) for the sole purpose of ordering outliers randomly within the dataset. The sort operation creates that random order. If you don't care about the position of your outliers in the dataset, you can simply do

 


%let N=100;
%let outlierPct=5;

data CN(keep=x); 
call streaminit(12345);
outlierNb = round(&N.*&outlierPct./100);
do i = 1 to outlierNb; 
    x = rand("Normal",100,4);
    output;
    end;
do i = outlierNb+1 to &N.; 
    x = rand("Normal");
    output;
    end;
run;
PG

View solution in original post


All Replies
Super Contributor
Posts: 275

Re: How to count outliers

In order to count 5% of outliers, you first have to calculate 97.5% and 2.5% quantile, x value which is greater than 97.5% quantile or less than 2.5% quantile is identified as outlier, then count frequence. 

 

%let N = 100;

data CN(keep=x);

call streaminit(12345);

do i = 1 to &N;

if rand("Normal", 100, 16) then

x = rand("Normal", 100, 4); else

x = rand("Normal");

output;

end;

run;

 


proc univariate data=cn;
var x;
output out=outlier pctlpts=2.5 97.5 pctlpre=x pctlname=pct25 pct975;
run;

 

data want;
set cn;
if _n_=1 then set outlier;
if x>xpct975 or x<xpct25 then flag=1;
else flag=0;
run;


proc freq data=want;
table flag;
run;

Trusted Advisor
Posts: 1,115

Re: How to count outliers

It looks like you want to contaminate a sample from a standard normal distribution with a sample from a N(100, 4) distribution, which is reasonable and would make it fairly easy to detect the outliers. However, your IF condition is satisfied with probability 1, so that all x-values will come from the N(100, 4) distribution. Of course, if you defined outliers as @slchen suggests, you would find (four) outliers even in the set {1, 2, 3, ..., 99, 100} and you wouldn't need contamination.

 

Occasional Contributor
Posts: 12

Re: How to count outliers

Thank you very much for both of your answers (slchen and rd).What I want to simulate data from normal distribution with outliers. It does not need N(100,4) but normal distribution with any mean and s.d. I am checking the effect of outliers for study. So I want to simulate the outliers with several pecentage like 5%, 10% 15% etc...I greately appriciate any suggestions. 

SAS Super FREQ
Posts: 3,488

Re: How to count outliers

The contaminated normal distribution is a specific two-component mixture distribution. The article "Generate a random sample from a mixture distribution" discusses simulting data from a mixture distribution in SAS.  The example uses three components, but can be modified to produce a contaminated normal.  See also Chapter 7 of Simulating Data with SAS.

Occasional Contributor
Posts: 12

Re: How to count outliers

Thank you so much Rick...
Respected Advisor
Posts: 4,659

Re: How to count outliers

[ Edited ]

To get exactly the number of outliers that you specify, randomly interspersed in your data you could do something like this:

 

%let N=100;
%let outlierPct=5;

data CN(keep=x rnd); 
call streaminit(12345);
outlierNb = round(&N.*&outlierPct./100);
do i = 1 to outlierNb; 
    x = rand("Normal",100,4);
    rnd = rand("UNIFORM");
    output;
    end;
do i = outlierNb+1 to &N.; 
    x = rand("Normal");
    rnd = rand("UNIFORM");
    output;
    end;
run;

proc sort data=CN out=CN(drop=rnd); by rnd; run;
PG
Occasional Contributor
Posts: 12

Re: How to count outliers

Thank you very much PG. I have a question. Why did you sort the data at the end? Thank you again.

Solution
‎11-20-2015 10:28 PM
Respected Advisor
Posts: 4,659

Re: How to count outliers

I added a random number to the dataset (rnd) for the sole purpose of ordering outliers randomly within the dataset. The sort operation creates that random order. If you don't care about the position of your outliers in the dataset, you can simply do

 


%let N=100;
%let outlierPct=5;

data CN(keep=x); 
call streaminit(12345);
outlierNb = round(&N.*&outlierPct./100);
do i = 1 to outlierNb; 
    x = rand("Normal",100,4);
    output;
    end;
do i = outlierNb+1 to &N.; 
    x = rand("Normal");
    output;
    end;
run;
PG
Occasional Contributor
Posts: 12

Re: How to count outliers

Great! Thank you very much for your quick reply and great help PG.
Regular Learner
Posts: 1

Re: How to count outliers

Hi,

I found this programe(genarate outliers) is very helpful. I would like to write this program in SAS IML. If you can write this in IML programe, it will greately appriciated. Thank you in advance.

 

SAS Super FREQ
Posts: 3,488

Re: How to count outliers

See my earlier comment in which I refer to the article "Generate a random sample from a mixture distribution." 

 

proc iml;
call randseed(12345);
N = 100;                            /* sample size */
k = ceil(0.05*N);                   /* 5% of sample */
x = j(N, 1); 
call randgen(x, "Normal", 0, 10);   /* sample from N(0, 10) */
z = j(k, 1);
call randgen(z, "Normal", 0, 100);  /* contamination from N(0, 100) */
idx = sample(1:N, k, "NoReplace");  /* k random elements */
x[idx] = z;                         /* overwrite with contaminated values */
🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 11 replies
  • 663 views
  • 0 likes
  • 6 in conversation