topic Re: There is a bug in the code in SAS/IML Software and Matrix Computations

There is a bug in the code

Alberto_Alvarez — Thu, 14 Jun 2018 10:53:16 GMT

Dear SAS Users,

I use the code below. What I am trying to do is to run sum equations simultaneously so that some of the observations (volatility, value of the firm) are determined simultaneously. For this, I employ the code below. However, what I get is that there is a bug that I am failing to identify. In particular, I receive the following mistake:

matrix z has not been set to a value 
no data set is currently open for output
File WORK.MYDATA2.DATA does not exist

I attach code (main - SASnew) and a data sample ('"sample" together with the "Winsorize_Macro" - that I used as a macro for winsorization purposes). Please, advise me this issue.

Re: There is a bug in the code

Ksharp — Fri, 20 Apr 2018 14:00:50 GMT

If you want winsorized data . Here is the code I wrote before.

data have;
 do i=1 to 100;
  a=ceil(ranuni(1)*100);
  b=ceil(ranuni(2)*100);
  output;
 end;
 drop i;
run;


%let low=0.05 ;
%let high=0.95 ;

proc iml;
use have;
read all var _num_ into x[c=vname];
close have;
call qntl(q,x,{&low ,&high});

do i=1 to ncol(x);
 x[loc(x[,i]<q[1,i]),i]=q[1,i];
 x[loc(x[,i]>q[2,i]),i]=q[2,i];
end;

create want from x[c=vname];
append from x;
close want;

quit;

Re: There is a bug in the code

Alberto_Alvarez — Fri, 20 Apr 2018 14:02:59 GMT

Thank you very much! I will follow your code. However, the problem is in the main code (SASnew), that results in a bug stated above. Maybe you have an idea of how I can fix it.

Re: There is a bug in the code

Ksharp — Sat, 21 Apr 2018 10:06:10 GMT

Sorry. I have no time to go through all these code.

Since it is IML code, Maybe @Rick_SAS would give you some advice .

Re: There is a bug in the code

Rick_SAS — Sat, 21 Apr 2018 10:43:02 GMT

You need to examine more of the log, Is the call to QUAD failing? The error says that the z variable (the integral of "FUN" on the interval ylow || yhigh is not assigned a value.

Here is how to resolve this issue. You need to extract the code that has the problem from all the other code.:

1. Print the values of the TEST data set.

2. In the PROC IML code, hard-code the values of the B vector. For example the start of the IML segment might become

csho = 4.2; t_debt = 3.86; ... ylow = 1.23]; /* values from TEST data */

3. The PROC IML step can then be run independently and you can debug the issue.

Re: There is a bug in the code

Rick_SAS — Sat, 21 Apr 2018 17:29:50 GMT

By the way, although your problem has nothing to do with Winsorization, I would caution you to always verify that the code you are using to Winsorize data is correct. I have seen many SAS programs that claim to Winsorize data, but many of them are wrong. The correct way to Winsorize data is explained in the article "How to Winsorize data in SAS," which includes an example. (If you have missing values in your data, be doubly wary!)

To verify whether the code you are using correctly Winsorizes data, you can run the following check. The call to PROC UNIVARIATE computes the Winsorized means of the variables X1 and X2. The call to PROC MEANS computes the means of the Winsorized data. If you do not get the same answers, then the code that Winsorized the data is not correct.

/* Winsorized means of the original data */
proc univariate data=OrigData Winsor=0.05;
   var x1 x2;
   ods select WinsorizedMeans;
run;

/* means of the Winsorized data */
proc means data=WinsorizedData mean;
   var x1 x2;
run;

I would also urge you to consider whether Winsorizing data is an appropriate way to handle extreme values in the data. Please read my article "Winsorization: The good, the bad, and the ugly"

Re: There is a bug in the code

Rick_SAS — Tue, 24 Apr 2018 10:14:28 GMT

I looked at this further. Instead of focusing on the integral, you need to focus on the function (the integrand) first. You are attempting to integrate a function that is essentially zero (numerically) on its domain. The following statements evaluate the function at the min, mid, and max values of the domain. The function returns zero for each location. If you try to plot the function, you will discover that it evaluates to zero on a uniform grid of points on its domain.

x1 = fun(ylow);
x2 = fun(mid); 
x3 = fun(yhigh);
print x1 x2 x3;    /* prints 0 0 0 */

tt = do(yhigh, ylow, (ylow-yhigh)/200);
yy = j(1, ncol(tt), .);
do i = 1 to ncol(tt);
   yy[i] = fun(tt[i]);
end;
print (range(yy));   /* prints 0 */
call series(tt, yy); /* shows straight line at y=0 */

Re: There is a bug in the code

Alberto_Alvarez — Tue, 24 Apr 2018 17:09:37 GMT

Thank you very much for your help. The main problem that I face is the following:

Convergence could not be attained over the subinterval

Can you advise me on this issue?

Re: There is a bug in the code

Rick_SAS — Tue, 24 Apr 2018 17:23:32 GMT

Yes. I have given you two pieces of advice:

1. Isolate the problem into a single, self-contained IML program.

2. Debug and visualize the integrand first before you worry about the integrand.

When you have done these two things, the problem should be apparent. If not, post the reduced and simplified example.