BookmarkSubscribeRSS Feed
PaalNavestad
Pyrite | Level 9

Hi I'm using the dataPreprocess.outlier action set in CAS. Using the IQR method i get an error message "ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations."

I've tried several settings of the tolerance and maxiterations without any success. Since I've never seen this before I can't really create a simple sharable data set.

 

The data I'm running with is residuals from a simpel exponential smooth done with proc TSmodel. 

 

Changing to method zscore does run the process smoothly.

 

Does anybody have an idea of why and hat can be done to fix.

Thanks Pål N

8 REPLIES 8
acordes
Rhodochrosite | Level 12

Show your CAS action code. 

I've been running into many of these errors with CAS actions (nevertheless I love them) and the log often does not point you in the right direction. It's not self-explanatory

PaalNavestad
Pyrite | Level 9

Thanks here is the code. Yes I agree the actionset is overall good. Just something about the data. I'm checking for outliers on data coming from a simple exponential forecast from TSModel. There is 469 different by groups all with approx. 880 observations. We run similar workflows on other data and have had no challenges.

proc cas;
	session casauto;
		dataPreprocess.outlier /                          
	      table={name="forecast"
	 			caslib="casuser"
				groupby={"unique_id"}
				}
		inputs={"Error"}
		method= "IQR"
		treatment = "trim"
		outVarsNamePrefix=""
	    outVarsNameSuffix="excl_Outl"
		countOutliers="True"
	    arguments={ scaleMultiplier=1.5 } 
		copyVars = {"unique_id" , "ACTUAL", "PREDICT", "ERROR", "prdate"}
		casout = {caslib="casuser" name="new_withoutl" replace= true}
		casOutOutlierInformation={caslib="casuser" name="OutlierStat" replace=True}
		;
run;

The log writes as this

84 proc cas;
85 session casauto;
86 dataPreprocess.outlier /
87 table={name="forecast"
88 caslib="casuser"
89 groupby={"unique_id"}
90 }
91 inputs={"Error"}
92 method= "IQR"
93 treatment = "trim"
94 outVarsNamePrefix=""
95 outVarsNameSuffix="excl_Outl"
96 countOutliers="True"
97 arguments={ scaleMultiplier=1.5 }
98 copyVars = {"unique_id" , "ACTUAL", "PREDICT", "ERROR", "prdate"}
99 casout = {caslib="casuser" name="new_withoutl" replace= true}
100 casOutOutlierInformation={caslib="casuser" name="OutlierStat" replace=True}
101 ;
102 run;
NOTE: Active Session now casauto.
ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations.
ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations.
ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations.
ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations.
ERROR: The action stopped due to errors.
103
 

Just switching to zscore as the outlier method everything works.

 

 

acordes
Rhodochrosite | Level 12

I've applied your code to my data making only the necessary changes to get it run. 

I cannot reproduce the error. 

I've tried with a variable which is normally distributed within each group. 

And I've tried with a variable having a constant value over all observations. 

It works for me. 

 

Have you tried with a different variable?

 

pic.pngpic1.png

PaalNavestad
Pyrite | Level 9

Yes, and from the data I have a challenge with I've run a smaller subset of data and it runs. In the data there are 4 groups that has missing residuals removeing these did not help, and one of them runs. I also have approx. 50 groups that are constant, again removing does not work and running one or two of these does run.

 

Strange also that zscore runs which I think is a more complex method.

 

At the moment I'm not that concerned about the method I'm running, I needed the structure to go further. 

 

I think I will report as a bug in the track system, unless omebody from SAS picks tis up with asuggestion.

 

Thanks for your interest.

 

taiphe
SAS Employee

The error message is from Percentile action. It happens when the parameter percentileTolerance is too small or the percentileMaxIterations is low. You can try adjust the parameter values to obtain the percentile result. Try this:

percentileTolerance=0.1

percentileMaxIterations=20

PaalNavestad
Pyrite | Level 9

Thanks taiphe, still do not work. I do not have time to test this using the percentiles method directly. I discussed with local support and create a track so I can attach data and so that SAS can try and see why.

 

It seems to me that it might have something to do with the number of groups.

 

And this error is not stopping progress for me. 

 

Pål N

 

PaalNavestad
Pyrite | Level 9

Problem reported as a track. Track number: 7613580754

 

taiphe
SAS Employee

Thanks for the info! I will forward the ticket to our percentile action support.

sas-innovate-2024.png

Don't miss out on SAS Innovate - Register now for the FREE Livestream!

Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.

 

Register now!

How to choose a machine learning algorithm

Use this tutorial as a handy guide to weigh the pros and cons of these commonly used machine learning algorithms.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 8 replies
  • 1251 views
  • 3 likes
  • 3 in conversation