BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
kamal1
Fluorite | Level 6

Hi All,

I have a program to genarate a data set from normal distribution with outliers.

I belive the following program genarate a data set with 5% of outliers.

 

 %let N = 100;

data CN(keep=x); call streaminit(12345);

do i = 1 to &N; if rand("Normal", 100, 16) then

x = rand("Normal", 100, 4); else

x = rand("Normal"); output; end; run;(Ref: Simulation data with SAS, By Rick Wicklin).

 

I want to estimate the propotion of the outliers to make sure my data set genarate  5% of outliers.

If any body have any idea , please help me.

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
PGStats
Opal | Level 21

I added a random number to the dataset (rnd) for the sole purpose of ordering outliers randomly within the dataset. The sort operation creates that random order. If you don't care about the position of your outliers in the dataset, you can simply do

 


%let N=100;
%let outlierPct=5;

data CN(keep=x); 
call streaminit(12345);
outlierNb = round(&N.*&outlierPct./100);
do i = 1 to outlierNb; 
    x = rand("Normal",100,4);
    output;
    end;
do i = outlierNb+1 to &N.; 
    x = rand("Normal");
    output;
    end;
run;
PG

View solution in original post

11 REPLIES 11
slchen
Lapis Lazuli | Level 10

In order to count 5% of outliers, you first have to calculate 97.5% and 2.5% quantile, x value which is greater than 97.5% quantile or less than 2.5% quantile is identified as outlier, then count frequence. 

 

%let N = 100;

data CN(keep=x);

call streaminit(12345);

do i = 1 to &N;

if rand("Normal", 100, 16) then

x = rand("Normal", 100, 4); else

x = rand("Normal");

output;

end;

run;

 


proc univariate data=cn;
var x;
output out=outlier pctlpts=2.5 97.5 pctlpre=x pctlname=pct25 pct975;
run;

 

data want;
set cn;
if _n_=1 then set outlier;
if x>xpct975 or x<xpct25 then flag=1;
else flag=0;
run;


proc freq data=want;
table flag;
run;

FreelanceReinh
Jade | Level 19

It looks like you want to contaminate a sample from a standard normal distribution with a sample from a N(100, 4) distribution, which is reasonable and would make it fairly easy to detect the outliers. However, your IF condition is satisfied with probability 1, so that all x-values will come from the N(100, 4) distribution. Of course, if you defined outliers as @slchen suggests, you would find (four) outliers even in the set {1, 2, 3, ..., 99, 100} and you wouldn't need contamination.

 

kamal1
Fluorite | Level 6

Thank you very much for both of your answers (slchen and rd).What I want to simulate data from normal distribution with outliers. It does not need N(100,4) but normal distribution with any mean and s.d. I am checking the effect of outliers for study. So I want to simulate the outliers with several pecentage like 5%, 10% 15% etc...I greately appriciate any suggestions. 

Rick_SAS
SAS Super FREQ

The contaminated normal distribution is a specific two-component mixture distribution. The article "Generate a random sample from a mixture distribution" discusses simulting data from a mixture distribution in SAS.  The example uses three components, but can be modified to produce a contaminated normal.  See also Chapter 7 of Simulating Data with SAS.

kamal1
Fluorite | Level 6
Thank you so much Rick...
PGStats
Opal | Level 21

To get exactly the number of outliers that you specify, randomly interspersed in your data you could do something like this:

 

%let N=100;
%let outlierPct=5;

data CN(keep=x rnd); 
call streaminit(12345);
outlierNb = round(&N.*&outlierPct./100);
do i = 1 to outlierNb; 
    x = rand("Normal",100,4);
    rnd = rand("UNIFORM");
    output;
    end;
do i = outlierNb+1 to &N.; 
    x = rand("Normal");
    rnd = rand("UNIFORM");
    output;
    end;
run;

proc sort data=CN out=CN(drop=rnd); by rnd; run;
PG
kamal1
Fluorite | Level 6

Thank you very much PG. I have a question. Why did you sort the data at the end? Thank you again.

PGStats
Opal | Level 21

I added a random number to the dataset (rnd) for the sole purpose of ordering outliers randomly within the dataset. The sort operation creates that random order. If you don't care about the position of your outliers in the dataset, you can simply do

 


%let N=100;
%let outlierPct=5;

data CN(keep=x); 
call streaminit(12345);
outlierNb = round(&N.*&outlierPct./100);
do i = 1 to outlierNb; 
    x = rand("Normal",100,4);
    output;
    end;
do i = outlierNb+1 to &N.; 
    x = rand("Normal");
    output;
    end;
run;
PG
kamal1
Fluorite | Level 6
Great! Thank you very much for your quick reply and great help PG.
save
Calcite | Level 5

Hi,

I found this programe(genarate outliers) is very helpful. I would like to write this program in SAS IML. If you can write this in IML programe, it will greately appriciated. Thank you in advance.

 

Rick_SAS
SAS Super FREQ

See my earlier comment in which I refer to the article "Generate a random sample from a mixture distribution." 

 

proc iml;
call randseed(12345);
N = 100;                            /* sample size */
k = ceil(0.05*N);                   /* 5% of sample */
x = j(N, 1); 
call randgen(x, "Normal", 0, 10);   /* sample from N(0, 10) */
z = j(k, 1);
call randgen(z, "Normal", 0, 100);  /* contamination from N(0, 100) */
idx = sample(1:N, k, "NoReplace");  /* k random elements */
x[idx] = z;                         /* overwrite with contaminated values */

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 11 replies
  • 2407 views
  • 0 likes
  • 6 in conversation