I have no idea on the solution to this specific question, but would like to recommend the only monograph I have found on building overdispersion models with SAS: Amazon.com: Overdispersion Models in SAS: 9781607648819: Morel PhD, Jorge G., Neerchal PhD, Nagaraj: Books. I have read part of it and am not sure whether it contains the answer to your specific question. But I am sure it is a good monograph for those who wish to have a systematic understanding on this field. ... View more

Although not solved yet, I have made some progress on the question I raised in the original post and would like to share my findings. (1) Theoretical issues: the non-zero part of my dependent variable follows a lognormal distribution. Theoretical discussions revolving the maximum likelihood estimation of these models can be found in Amazon.com: Generalized Linear Models for Categorical and Continuous Limited Dependent Variables (Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences): 9781032477466: Smithson, Michael, Merkle, Edgar C.: Books. Moreover, heteroscedasticity of limited dependent variable models is an problem I encountered and named by the authors of this book as an issue extensively discussed therein. Truncated and Censored Samples: Theory and Applications (Statistics: A Series of Textbooks and Monographs) by A. Clifford Cohen (1991-05-23): A. Clifford Cohen: Amazon.com: Books also contains a brief summary of maximum likelihood estimators of limited dependent variable models whose dependent variable follows a lognormal distribution. (2) Software implementation: Amazon.com: Maximum Likelihood Estimation and Inference: With Examples in R, SAS and ADMB: 9780470094822: Millar, Russell B.: Books is a monograph on maximum likelihood estimation (MLE) with special emphasis on software implementation, including issues of conducting MLE with SAS. ... View more

Now I see where the problem seems to be. The entire but the last few entires of the vector of results (i.e., the CDF's I wish to calculate) are in fact 1. For instance, one of the calculations is as follows: a=1817.9997; b=38.043169; gamma=cdf('GAMMA',a,b); But given that I had pre-emptively filled the vector gamma with a batch of 1's with the J function, I was not able to discover that SAS had already conducted calculations on some of the entires. Setting the default values to 0 or missing helps me make this discovery. But the last few entries of up and p contained negative values, which is why SAS kept reporting errors. I will go through my code and search for reasons of the appearance of negative values. ... View more

That's interesting. My code exactly looks like the one you provided but it still kept failing. I will copy my log once again. ERROR: (execution) Invalid argument to function. operation : CDF at line 180 column 9 operands : *LIT1009, _TEM1001, _TEM1002 *LIT1009 1 row 1 col (character, size 5) GAMMA _TEM1001 1 row 1 col (numeric) -2.326E19 _TEM1002 1 row 1 col (numeric) -4.868E17 statement : ASSIGN at line 180 column 1 I looked at the entries of up and p and found minuscule numbers at the end of the two vectors (e.g.,1.4066E19). Is it because of the presence of the letter "E" that causes the CDF function to falsely identify it as a character variable and the consequent failure of the code? ... View more

I have just finished reading the book A SAS/IML Companion for Linear Models | SpringerLink from cover to cover. That is the only IML textbook I have read till now. I had known that you had written an IML book. But by the time I was selecting which book to read, I was not in a dire need for the software. I learned it because I wished to have more "knowledge (or technology) reserve" in response to demanding requests that may be proposed by reviewers after submission of my research papers to journals. Given that your book was thicker, I chose the book with less pages. ... View more

Ah, yes... That's a mistake that shouldn't had been make... But the problem lingers after I rearranged the sequence of codes. The error-reporting log was similar to the one I posted earlier. It still said there was an invalid argument to function. So, ..., why is the case now? ... View more

I read your blog and found that given there was an extra Γ(p) term to divide after I compute the incomplete gamma function, the whole thing I had been calculating was nothing but the cumulative distribution function (CDF) of Γ(up,p). I tried the numerical integration method by the QUAD routine first and found out that for small values of p, the QUAD routine failed potentially as a result of lack of specification of the SCALE= option. I therefore read your blog and found out the conclusion I gave at the beginning of this thread. Still, I encountered a problem when I was actually computing the CDF's. I directly applied the CDF function element-wise and found an error. The code looked like this: proc iml; Gamma=j(nrow(p),1,1); up = {3, 4, 5}; p = {1.1, 2.2, 0.8}; do i=1 to nrow(p); Gamma[i]=cdf('gamma',up[i],p[i]); end; print p up Gamma; quit; The code failed. The log read: ERROR: (execution) Invalid argument to function. operation : CDF at line 358 column 9 operands : *LIT1008, _TEM1001, _TEM1002 *LIT1008 1 row 1 col (character, size 5) gamma _TEM1001 1 row 1 col (numeric) -2.326E19 _TEM1002 1 row 1 col (numeric) -4.868E17 statement : ASSIGN at line 358 column 1 Why is it the case? ... View more

@Tom wrote: Poor example, why would you want to define the same function 10 times? The original code I composed revolved around the incomplete gamma function. As the integrand varied from iteration to iteration and I was unaware of the correct usage of the GLOBAL option of the START statement before I consulted @Rick_SAS, I tried to use the do loop to define the function differently in each loop, but I failed and therefore deleted the original code. I did not want to spend time on recalling and typing the real code I used before, so I generated the same function again and again. I think this is not a major concern to focus on. The real major concern is that building modules are not supported in the DO loop, an issue that seems to have been overlooked by IML textbooks, as least the one I referred to. Therefore, I wrote down my experience to alert to readers who may be willing to adopt a similar approach. @Tom wrote: However if you did need to build something were the definition of a function changes then just use some type of code generation.  Such as macro code or call execute() or writing the code to a file and using %INCLUDE. @Rick_SAS gave an example of the GLOBAL option. I think that is good enough. I had just read a blog by @Rick_SAS (Macros and loops in the SAS/IML language - The DO Loop) in which he said that he did not support using macros in SAS/IML that much. Neither do I. I am not that familiar with macros. ... View more

Thank you, Rick, for your rapid, detailed and helpful reply! It takes me time to digest the information you provided in your thread. However, there is one issue that I can point out: @Rick_SAS wrote: > Given that a do loop is not acceptable in a module of SAS/IML, what should I do next? This statement is false. What you should do is use the IML language instead of macro loops. An IML program rarely requires a macro loop because the language supports loops and arrays. I made a mistake in my post. The correct message to convey is "building modules are not supported in a do loop". For instance, the following code makes SAS report an error in the log: proc iml; do i=1 to 10; start fun(u); y=u**2; return(y); finish; end; quit; I will post my questions if there are any after reading your reply carefully. Thanks! ... View more

Thank you for your further explanation, including the history of change in SAS arguments! I previously proposed several suggestions on improving SAS that I thought was necessary in the SAS Product Suggestions block of SAS Community. You can see from my personal profile page that I have the "Now that's an idea" badge. However, none of the suggestions were adopted, making me a bit frustrated. So I may not propose my suggestion this time. But thank you for suggesting that I propose a suggestion anyway! ... View more

Thank you for your further explanation. @Tom wrote: Note that the CALL EXECUTE() actually works better in PROC IML than it does in a data step.  At least when using it to generate PROC IML statements.  That is because the generated code is run immediately rather than being stacked up to execute later.  When using CALL EXECUTE() in a data step the code generated does not run until the data step used to generate it finishes.   The || operator in the data step language is the STRING concatenation operator.  Your code is concatenating strings with numbers.   That will require the data step to convert the number into a string. When the data step does this it uses the BEST12. format for the number.  In your example code call execute ('data a'||x||'=2;run;'); that will cause it to generated invalid SAS code like: data a 1=2; run; You should either use the PUT() function, or the PUTN() function like the one that PROC IML will require, to first convert the number into a string.    Or use the new CAT... series of functions that SAS added recently.  Those will automatically remove the leading spaces when they autoconvert numbers.   So perhaps something like: call execute(catx('data',cats('a',x),';set a; where x=',x,';run;')); Which will generate a data steps like these to run after the current data step finishes. data a1 ;set a; where x=1 ;run; data a2 ;set a; where x=2 ;run; data a3 ;set a; where x=3 ;run;   Yes, I found myself wrong on this code. An error is reported when I ran my code. The correct way to do this is to transform the numerical variable represented by x into character ones. In addition, the "=2" is incorrect. The correct code should be like this: data _null_; set a; call execute ('data a'||cats(x)||';a=2;run;'); run; Other string functions like the ones you used can also be employed. ... View more

Now that my first question is resolved, I turn to the second. I am not sure of what your sentence of "I am not sure why when they added the ability to use name lists to reference datasets they did not also add a similar feature to the IML commands that reference matrixes." means. I am not a native English speaker. Please forgive me for my lack of fluency in English to understand this sentence. ... View more

    @Tom wrote: The || operator is not what you want in the IML code (also not what you want in your data step example either but let's ignore that). No, that's exactly what I want. The "||" operator works perfectly in CALL EXECUTE routine of the DATA step. Try run the codes I provided a few hours ago and verify that this does generate datasets named a1, a2, a3, ..., a35. It is both relatively simple and efficient. So I wish to go on using it in IML. That is why I originally mentioned it in my post and repetitively referred to it in my various replies. As for your codes provided... Yes, it does solve my problem with the help of string functions. I really appreciate your help. But I still grumble about the lack of simpler codes like the CALL EXECUTE code I composed above. ... View more