turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

Find a Community

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Convergence warning when running proc quantreg

Topic Options

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

12-20-2016 05:47 AM

I am getting a warning message when running Proc quantreg:

"WARNING: Convergence was not attained. Duality gap 4.3888576E-8 could not be reduced further. Reduce the required accuracy or try a smaller step length with the KAPPA= option."

I do not understand this message. I guess that reducing the accuracy only takes the message away but does not change anything. I do not understand what reducing the step length with the KAPPA option does. Does it change the outcome or just the efficiency?

Is there an easy sollution to this problem? I would be greatful if someone could explain this warning message. Is it a severe warning?

Accepted Solutions

Solution

12-30-2016
03:09 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-30-2016 07:13 AM

I am not a world-class expert in this area, but from what you've told me I think it isn't surprising that the model for lower quantiles does not converge. An optimization algorithm will often "not converge" if there is not a unique optimum, and it sounds like this is the case here.

One option is to give up on constructing a model for the lower quantiles. The model does not make sense for the data.

Another option is to change the data by adding a small amount of random noise to the response variable. This would to overcome the dupicated value problem. Read the following articles that discuss quantiles for and GOF tests for rounded data. I also include an article about the smooth bootstrap, which shows how adding small random values can sometimes work around problems caused by rounded data:

- Binning data by quantiles? Beware of rounded values

All Replies

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-20-2016 04:13 PM - edited 12-20-2016 04:18 PM

Please post the SAS code that generates the error. Also read the documentation for the Interior Point method and the KAPPA= option. The error message appears to tell you the solution: decrease the step size. If you are using the defalt value of kappa ( who knows? No code posted!) you can try

proc quantreg algorithm=interior(kappa=0.8);

...

run;

or try ALGORITHM=IPM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

12-22-2016 03:29 AM - edited 12-22-2016 03:32 AM

Thank you for your answer. I am afraid that I do not fully understand the description of the Interior point algorithm. How would reducing the Kappa option effect the results?

Here is the code:

proc quantreg data=t_quantreg; model cP = var1 var2 var3 var4 / quantile= 0.10 to 0.90 by 0.10 PLOT=QUANTPLOT; run;

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-25-2016 07:06 AM

Did you try it? Did it work?

KAPPA is related to a step size during the optimization phase. When you optimize a nonlinear function, the algorithm starts with a guess and then takes steps that (hopefully) improve the guess. If the step size is too large, the algorithm might step over the optimum or bounce around without converging. If the step size is too small, the optimization will take a long time and many iterations to converge.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-26-2016 09:49 AM

Thank you for the explanation. Unfortunately it did not work. I tried setting the Kappa option as low as 0.1. I guess it has to do with the skewed distribution of my data. I only get a warning for quantiles 0.1-0.5. What does the value actually say? How important is the accuracy?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-26-2016 11:01 AM

What happens if you try a different algorithm for estimating the parameters? For example, ALGORITHM=IPM or ALGORITHM=SMOOTH?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

12-28-2016 05:24 AM

The smooth algorithm is not efficient enough. I stopped this procedure after about 14 hours. The ipm algorithm is not accepted. An error message says that the procedure expects interior, simplex or smooth.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-28-2016 09:36 AM

Sounds like you are running an old release of SAS.

I would be interested in knowing whether you have many duplicate rounded values for the response variable. It could be the the nature of your data does not permit discrimination between the lower quantiles. For example, in the data Y={0, 0, 0, 0, 0, 0, 1, 2, 2, 3, 3}, the quantiles for 0.1, 0.2, 0.3, 0.4, and 0.5 are all the same (0). Consequently, there is no way to distinguish between those quantiles. Can you post a histogram of your response variable?

Also, how many observations in the data?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

12-30-2016 02:32 AM

It is likely the case that proc quantreg cannot discriminate between quantiles. My data is skewed, have a lot of duplicates since it is derived from integers. The distribution looks like this.

And zooming in to the values between 1 and 5:

I have about 9.5 million observations.

Solution

12-30-2016
03:09 PM

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-30-2016 07:13 AM

I am not a world-class expert in this area, but from what you've told me I think it isn't surprising that the model for lower quantiles does not converge. An optimization algorithm will often "not converge" if there is not a unique optimum, and it sounds like this is the case here.

One option is to give up on constructing a model for the lower quantiles. The model does not make sense for the data.

Another option is to change the data by adding a small amount of random noise to the response variable. This would to overcome the dupicated value problem. Read the following articles that discuss quantiles for and GOF tests for rounded data. I also include an article about the smooth bootstrap, which shows how adding small random values can sometimes work around problems caused by rounded data:

- Binning data by quantiles? Beware of rounded values

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to Rick_SAS

12-30-2016 08:18 AM

Thank you Rick! Now I understand the problem and can proceed with my work. One last question; Do you know how the procedure deals with the problem? Since I get a result it has delt with the problem somehome. In the data Y={0,1,2,2,3,4,5,6,7,8}. The first decile is 0 and the second decile 1 but what about decile 3 and 4, are the values arbitrarily assigned?

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-30-2016 09:57 AM

As discussed in the "Binning data by quantiles" article, the 30th pctl is 2 and the 40th pctl is also 2. Quantiles are not unique when you have repeated values.

- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

Posted in reply to peter_sjogarde

12-30-2016 03:10 PM

Thank you very much for the help on this issue!