BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
peter_sjogarde
Fluorite | Level 6

I am getting a warning message when running Proc quantreg:

 

"WARNING: Convergence was not attained. Duality gap 4.3888576E-8 could not be reduced further. Reduce the required accuracy or try a smaller step length with the KAPPA= option."

 

I do not understand this message. I guess that reducing the accuracy only takes the message away but does not change anything. I do not understand what reducing the step length with the KAPPA option does. Does it change the outcome or just the efficiency?

 

Is there an easy sollution to this problem? I would be greatful if someone could explain this warning message. Is it a severe warning?

1 ACCEPTED SOLUTION

Accepted Solutions
Rick_SAS
SAS Super FREQ

I am not a  world-class expert in this area, but from what you've told me I think it isn't surprising that the model for lower quantiles does not converge. An optimization algorithm will often "not converge" if there is not a unique optimum, and it sounds like this is the case here.

 

One option is to give up on constructing a model for the lower quantiles. The model does not make sense for the data.

 

Another option is to change the data by adding a  small amount of random noise to the response variable. This would to overcome the dupicated value problem. Read the following articles that discuss quantiles for and GOF tests for rounded data. I also include an article about the smooth  bootstrap, which shows how adding small random values can sometimes work around problems caused by rounded data:

 

- Binning data by quantiles? Beware of rounded values

- Goodness-of-fit tests: a cautionary tale

- The smooth bootstrap method in SAS

View solution in original post

12 REPLIES 12
Rick_SAS
SAS Super FREQ

Please post the SAS   code that generates the error.  Also read the documentation for the Interior Point method and the KAPPA= option.  The error message appears to tell you the solution: decrease the step size.  If you are using the defalt value of kappa ( who knows? No  code posted!) you can try

proc quantreg algorithm=interior(kappa=0.8);

...

run;

 

or try ALGORITHM=IPM

peter_sjogarde
Fluorite | Level 6

Thank you for your answer. I am afraid that I do not fully understand the description of the Interior point algorithm. How would reducing the Kappa option effect the results?

 

Here is the code:

 

proc quantreg data=t_quantreg;
model cP = var1 var2 var3 var4
/ quantile= 0.10 to 0.90 by 0.10 
PLOT=QUANTPLOT;
run;
Rick_SAS
SAS Super FREQ

Did you try it? Did it work?

 

KAPPA is related to a step size during the optimization phase. When you optimize a nonlinear function, the algorithm starts with a guess and then takes steps that (hopefully) improve the guess. If the step size is too large, the algorithm might step over the optimum or bounce around without converging. If the step size is too small, the optimization will take a long time and many iterations to converge.

peter_sjogarde
Fluorite | Level 6
Thank you for the explanation. Unfortunately it did not work. I tried setting the Kappa option as low as 0.1. I guess it has to do with the skewed distribution of my data. I only get a warning for quantiles 0.1-0.5. What does the value actually say? How important is the accuracy?
Rick_SAS
SAS Super FREQ

What happens if you try a different algorithm for estimating the parameters? For example, ALGORITHM=IPM or ALGORITHM=SMOOTH?

peter_sjogarde
Fluorite | Level 6

The smooth algorithm is not efficient enough. I stopped this procedure after about 14 hours. The ipm algorithm is not accepted. An error message says that the procedure expects interior, simplex or smooth. 

Rick_SAS
SAS Super FREQ

Sounds like you are running an old release of SAS. 

 

I would be interested in knowing whether you have many duplicate rounded values for the response variable. It could be the the nature of your data does not permit discrimination between the lower quantiles.  For example, in the data Y={0, 0, 0, 0, 0, 0, 1, 2, 2, 3, 3}, the quantiles for 0.1, 0.2, 0.3, 0.4, and 0.5 are all the same (0). Consequently, there is no way to distinguish between those quantiles. Can you post a histogram of your response variable?

 

Also, how many observations in the data?

peter_sjogarde
Fluorite | Level 6

It is likely the case that proc quantreg cannot discriminate between quantiles. My data is skewed, have a lot of duplicates since it is derived from integers. The distribution looks like this.

 

dist1.png

And zooming in to the values between 1 and 5:

 

dist2.png

I have about 9.5 million observations. 

Rick_SAS
SAS Super FREQ

I am not a  world-class expert in this area, but from what you've told me I think it isn't surprising that the model for lower quantiles does not converge. An optimization algorithm will often "not converge" if there is not a unique optimum, and it sounds like this is the case here.

 

One option is to give up on constructing a model for the lower quantiles. The model does not make sense for the data.

 

Another option is to change the data by adding a  small amount of random noise to the response variable. This would to overcome the dupicated value problem. Read the following articles that discuss quantiles for and GOF tests for rounded data. I also include an article about the smooth  bootstrap, which shows how adding small random values can sometimes work around problems caused by rounded data:

 

- Binning data by quantiles? Beware of rounded values

- Goodness-of-fit tests: a cautionary tale

- The smooth bootstrap method in SAS

peter_sjogarde
Fluorite | Level 6

Thank you Rick! Now I understand the problem and can proceed with my work. One last question; Do you know how the procedure deals with the problem? Since I get a result it has delt with the problem somehome. In the data Y={0,1,2,2,3,4,5,6,7,8}. The first decile is 0 and the second decile 1 but what about decile 3 and 4, are the values arbitrarily assigned?

Rick_SAS
SAS Super FREQ

As discussed in the "Binning data by quantiles" article, the 30th pctl is 2 and the 40th pctl is also 2.  Quantiles are not unique when you have repeated values.

peter_sjogarde
Fluorite | Level 6
Thank you very much for the help on this issue!

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 12 replies
  • 2863 views
  • 1 like
  • 2 in conversation