topic Re: Outlier method in CAS. The calculation of percentiles did not converge in SAS Data Science

Outlier method in CAS. The calculation of percentiles did not converge

PaalNavestad — Wed, 18 May 2022 08:57:30 GMT

Hi I'm using the dataPreprocess.outlier action set in CAS. Using the IQR method i get an error message "ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations."

I've tried several settings of the tolerance and maxiterations without any success. Since I've never seen this before I can't really create a simple sharable data set.

The data I'm running with is residuals from a simpel exponential smooth done with proc TSmodel.

Changing to method zscore does run the process smoothly.

Does anybody have an idea of why and hat can be done to fix.

Thanks Pål N

Re: Outlier method in CAS. The calculation of percentiles did not converge

acordes — Wed, 18 May 2022 14:26:16 GMT

Show your CAS action code.

I've been running into many of these errors with CAS actions (nevertheless I love them) and the log often does not point you in the right direction. It's not self-explanatory

Re: Outlier method in CAS. The calculation of percentiles did not converge

PaalNavestad — Thu, 19 May 2022 06:48:19 GMT

Thanks here is the code. Yes I agree the actionset is overall good. Just something about the data. I'm checking for outliers on data coming from a simple exponential forecast from TSModel. There is 469 different by groups all with approx. 880 observations. We run similar workflows on other data and have had no challenges.

proc cas;
	session casauto;
		dataPreprocess.outlier /                          
	      table={name="forecast"
	 			caslib="casuser"
				groupby={"unique_id"}
				}
		inputs={"Error"}
		method= "IQR"
		treatment = "trim"
		outVarsNamePrefix=""
	    outVarsNameSuffix="excl_Outl"
		countOutliers="True"
	    arguments={ scaleMultiplier=1.5 } 
		copyVars = {"unique_id" , "ACTUAL", "PREDICT", "ERROR", "prdate"}
		casout = {caslib="casuser" name="new_withoutl" replace= true}
		casOutOutlierInformation={caslib="casuser" name="OutlierStat" replace=True}
		;
run;

The log writes as this

84 proc cas;

85 session casauto;

86 dataPreprocess.outlier /

87 table={name="forecast"

88 caslib="casuser"

89 groupby={"unique_id"}

90 }

91 inputs={"Error"}

92 method= "IQR"

93 treatment = "trim"

94 outVarsNamePrefix=""

95 outVarsNameSuffix="excl_Outl"

96 countOutliers="True"

97 arguments={ scaleMultiplier=1.5 }

98 copyVars = {"unique_id" , "ACTUAL", "PREDICT", "ERROR", "prdate"}

99 casout = {caslib="casuser" name="new_withoutl" replace= true}

100 casOutOutlierInformation={caslib="casuser" name="OutlierStat" replace=True}

101 ;

102 run;

NOTE: Active Session now casauto.

ERROR: The calculation of percentiles did not converge. You might want to adjust the tolerance or the maximum number of iterations.

ERROR: The action stopped due to errors.

103

Just switching to zscore as the outlier method everything works.

Re: Outlier method in CAS. The calculation of percentiles did not converge

acordes — Thu, 19 May 2022 08:12:17 GMT

I've applied your code to my data making only the necessary changes to get it run.

I cannot reproduce the error.

I've tried with a variable which is normally distributed within each group.

And I've tried with a variable having a constant value over all observations.

It works for me.

Have you tried with a different variable?

Re: Outlier method in CAS. The calculation of percentiles did not converge

PaalNavestad — Thu, 19 May 2022 08:52:57 GMT

Yes, and from the data I have a challenge with I've run a smaller subset of data and it runs. In the data there are 4 groups that has missing residuals removeing these did not help, and one of them runs. I also have approx. 50 groups that are constant, again removing does not work and running one or two of these does run.

Strange also that zscore runs which I think is a more complex method.

At the moment I'm not that concerned about the method I'm running, I needed the structure to go further.

I think I will report as a bug in the track system, unless omebody from SAS picks tis up with asuggestion.

Thanks for your interest.

Re: Outlier method in CAS. The calculation of percentiles did not converge

taiphe — Thu, 19 May 2022 15:43:30 GMT

The error message is from Percentile action. It happens when the parameter percentileTolerance is too small or the percentileMaxIterations is low. You can try adjust the parameter values to obtain the percentile result. Try this:

percentileTolerance=0.1

percentileMaxIterations=20

Re: Outlier method in CAS. The calculation of percentiles did not converge

PaalNavestad — Fri, 20 May 2022 06:20:32 GMT

Thanks taiphe, still do not work. I do not have time to test this using the percentiles method directly. I discussed with local support and create a track so I can attach data and so that SAS can try and see why.

It seems to me that it might have something to do with the number of groups.

And this error is not stopping progress for me.

Pål N

Re: Outlier method in CAS. The calculation of percentiles did not converge

PaalNavestad — Fri, 20 May 2022 14:16:24 GMT

Problem reported as a track. Track number: 7613580754

Re: Outlier method in CAS. The calculation of percentiles did not converge

taiphe — Fri, 20 May 2022 15:45:17 GMT

Thanks for the info! I will forward the ticket to our percentile action support.