Thank you for your explantaion on the theoretical and practical details! I have learnt a lot! ... View more

I agree with @Ksharp's solution. Fisher's exact test is extremely computationally intensive. Moreover, a frustrating fact is that you may insist in running the code by letting your computer alone while the computer may end up reporting a lack of memory after an extremely long period of time (e.g., 7 days) and nothing else. Still, if you are still curious and just want to try out Fisher's exact test, then you first have to augment the available memory of your SAS session. By default, a memory of 2GB is assigned to SAS. This is far from enough for doing such a Fisher's exact test. Therefore, the first thing you should do is to open a folder on your computer, and then type the following code in its address bar: Place where your SAS program is installed -memsize 1T Like this: C:\Program Files\SASHome\SASFoundation\9.4\sas.exe -memsize 1T "C:\Program Files\SASHome\SASFoundation\9.4\sas.exe" stands for the directory of my SAS software. " -memsize 1T" stands for provision of 1TB of memory to a SAS session that will appear once you press the "Enter" button on your keyboard. Run Fisher's exact test on this very SAS session instead of those invoked by other means (e.g., via the Start Menu) in that those SAS sessions are still automatically assigned a memory of size 2GB. A final remark is that if your computer is not computationally powerful, then it is better to resort to the Monte Carlo-simulated Fisher's exact test as suggested by @Ksharp. ... View more

Now back to my question. @SteveDenham wrote: So given the definition of left censoring that PROC SEVERITY uses, your response value could potentially be negative. Zero and negative values aren't supported by several of the interesting distributions available to you in SEVERITY. Would those values be meaningful, or even observable? (I only ask as I don't know what the response variable is). If the variables are not observable, then consider that the left truncation approach has some appeal. In fact, my data is censored at zero in that it contains cases with y=0. However, my data somehow resembles insurance reimbursement data in that negative values are meaningless and therefore can never be observed. They are only present in the imaginary latent variable. To summarize, my data contains cases at the censoring threshold. They are not truncated. I know that the same collection of statistical tools (e.g.,  Tobit models) can be applied to such data regardless on their condition on being censored or truncated, but I beg to point out that I am still skeptical on whether arbitrarily designate censored data as truncated would lead to correct results. ... View more

Thank you for your reply! @SteveDenham wrote: So given the definition of left censoring that PROC SEVERITY uses, your response value could potentially be negative. Zero and negative values aren't supported by several of the interesting distributions available to you in SEVERITY. Would those values be meaningful, or even observable? (I only ask as I don't know what the response variable is). In fact, PROC SEVERITY adopts a latent variable modeling paradigm for dealing with censoring data. For instance, if I am the manager of an insurance company and wish to find out issues associated with reimbursement, then I plan to build a linear regression model with the amount of reimbursement (term it "y" here) as the dependent variable and several variables (term them "x1", "x2", ... "xn" here). It is easy to find out that y is not less than 0 (non-negative), essentially violating the assumption of multiple regression where the dependent variable can take any real value. The latent variable modeling paradigm assumes that y is in fact a partially observed variable of a latent one, which is called y* here. In the example here, the amount of reimbursement is assumed to be closely associated with y* such that if y* is ≥0, then y=y*; if y* is <0, then y=0. The unobserved variable y* is a latent variable because it is not fully observed. On the other hand, if the insurance company does have records where y=0, then the manifest variable y is called censored. However, if the insurance company only contains records with reimbursement, namely all subjects without reimbursement are not documented in the dataset, then the record only contains cases with y>0 (no equal sign here). In this case, the manifest variable is called truncated. Of note, the latent variable modeling paradigm can be readily applied to the case of truncation. In fact, you can use the same collection of statistical tools built on the latent variable assumption to handle censoring and truncation. However, the researcher himself/herself has to be fully aware of whether his/her data is truncated or censored so as to correctly report and interpret their results, as truncation and censoring are essentially different concepts. Before I end this thread, I would like to point out that the latent variable modeling paradigm is not the only approach for modeling censored or truncated data. Other modeling paradigms include the two-part model approach and the hurdle model approach. They are, however, not supported by PROC SEVERITY. For a concise yet comprehensive and therefore excellent review, see Two-Part Models for Zero-Modified Count and Semicontinuous Data | SpringerLink. ... View more

Thank you for taking your precious time to read the documentation! I really appreciate it! As I have said in the original question, my data is censored instead of truncated in that data with values equal zero, the threshold, can be observed in my dataset. However, there is one thing regarding your reply that I cannot be sure of and therefore have to further consult on it: @SteveDenham wrote: Is your data left censored or left truncated? The way I read the documentation, left truncation means the result is observed only if Y > T where T is the truncation threshold. Then the documentation  defines left censoring if it is known that the magnitude is Y<= C. That may have some effect on the CDF estimates. I suspect that the use of a small value for the cutpoint may then have a different effect, especially for the candidate distributions that are not defined for Y=0.  I am not sure if the "different effect" of the final quoted sentence refers to what I intended to ask. My original question was that I was supposed to explictly define the censoring threshold, namely C, as zero, but was forced to add a tiny yet positive value to it, e.g., C+0.001, by the software. I was unsure whether this would cause the parameter estimates to be incorrect. Now that you mentioned truncation in your thread as well, I am not sure if you meant to express that confounding truncation and censoring would lead to incorrect results. @SteveDenham wrote: I would be tempted to add a small value to all the observations, and then set the cutoff at that value, just to see what happens.   SteveDenham Anyway, regardless of the message you wished to deliver, I think your suggestion is a nice choice for the original question I raised. I had not thought about trying approaches like that before reading your response. Thank you! ... View more

I am also interested in building this kind of model. Could you please explain why there it might be inappropriate to build such models for continuous distributions in more detail? Thank you! ... View more

I continued reading the documentation of PROC SEVERITY yesterday and found one of the examples in it (SAS Help Center: Example 29.3 Defining a Model for Mixed-Tail Distributions)  explictly pointed out the second phenomenon you mentioned, namely the presence of extreme values. However, it was also explicly stated there that these values should not be regarded as outliers and hence discarded. So I am afraid that @Ronein should embark on a more complicated analysis instead of simply deleting the extreme values. The good news for @Ronein is a code suitable for this purpose is readily available, saving a lot of work. By the way, the documentation also contains an example of building finite mixture models with PROC SEVERITY, so relevant codes are also readily available there. ... View more

Hello, there. I found your post while searching for something on the lognormal distribution in the community. I read from your profile that you have not been here for more than four years. In addition, this question was raised more than 10 years ago. I am not sure if your problem has been solved and if you need my answer for the time being. But I think somebody else may need it and am therefore here to offer my viewpoint on your problem. I think ANOVA is a suitable choice in terms of estimating the group means. But forming confidence intervals is not so intuitive under your setting. Why not try the accelerated failure time (AFT) model and put the group indicators as the only independent variables in your model? The AFT model is very inclusive in the sense that the lognormal distribution is only one of the popular distributions that can be modeled. Methods of inferences with respect to the AFT model, including the construction of confidence intervals of the dependent variable, may be more comprehensively studied than the log-transformed version of ANOVA you employed. I think you can have a try. I have also found an article that might solve your problem under another paradigm. You may take a look. Inferences on the means of lognormal distributions using generalized p-values and generalized confidence intervals - ScienceDirect ... View more

Thank you for sharing your experience on modeling such data as they are not only useful for @Ronein, but also for me. There are attitional thoughts I come up with after reading your post. I would like to share them with @Ronein and other friends of this community. In contrast to yours, my perspective is a statistical one. (1) Methods of handling missing data might be needed, especially when the proportion of subjects that satisfy the first exclusion criterion proposed by @SASKiwi is not negligible. (2) While excluding extreme values is a choice, the extreme value themselves might be of scientific or commerical interest. In fact, there is a class of statistical models specifically tailored for extreme value modeling. See, for instance, An Introduction to Statistical Modeling of Extreme Values: 9781852334598: Medicine & Health Science Books @ Amazon.com. However, your data is complicated by the presence of zero-censoring and extreme value modeling methods that take this factor into account might not be present for the time being. You can search on the web to see if I am right on this issue. (3) Based upon the experience of @SASKiwi, it is useful to include customer type as one of the independent variables of your statistical model. Moreover, it is also possible that your customers' wealth falls into none of the known distributions but is rather a blend mixture of them (e.g., 40% of them follow a truncated normal distribution and 60% of them follow a truncated lognormal distribution). In this case, a finite mixture model might be more suitable. An aside for this is that modeling finite mixture models of zero-censored data is also supported in PROC SEVERITY, but you might have to specify the distribution manually via PROC FCMP. Another option for building finite mixture models in SAS is PROC FMM. ... View more

I think it is up to you to figure out the aspect of data mining. For your question, the "distribution" of customer wealth is a very broad field. Aside from the statistics @Ksharp has pointed out and plots that @Ksharp has drawn, I think checking whether the kind of distribution changes over time might be of interest. For instance, at 2013, the wealth may follow a truncated normal distribution while in 2022 the wealth may follow a truncated lognormal distribution. As you have explictly pointed out the zero-inflation (or zero-censoring) property of your data, I warmly welcome you to join the cohort of PROC SEVERITY users. I first heard about it from @Rick_SAS a few years ago. @Rick_SAS told me that it is a procedure suitable for modeling insurance reimbursement data. I happen to be modeling the same kind of data as you are dealing with and am learning PROC SEVERITY as well. I have found out that it is a fantastic tool for modeling such data in the following senses: (1) many forms of distributions, including those less commonly applied in everyday statistics (e.g., Pareto, lognormal), are built in the procedure and readily accessible for modeling; (2) modeling data with censoring and truncation is supported by merely assigning a few options in the procedure; (3) you can specify your own distribution with the FCMP procedure if you find out that the built-in distributions still do not match your requirement; (4) goodness-of-fit statistics can be readily calculated for your comparison among many alternative choices of distribution; (5) it is suitable for both descriptive and modeling purposes as you can specify the regressors for the scale parameter via the SCALEMODEL statement or omit that statement to let SAS simply calculate the estimated parameters of the distribution you specify (e.g., the μ and σ of the lognormal distribution). So why not have a try? There are also other alternative procedures that might be suitable. (1) The HPSEVERITY procedure. This is a high-performance version of the SEVERITY procedure. I am still not aware of the other differences between them. (2) The QLIM procedure. It is suitable for modeling semi-continuous data like yours, but it only supports modeling data whose non-zero portion follow a normal or logistic distribution while PROC SEVERITY supports much more. By the way, as I am also learning PROC SEVERITY and modeling zero-censored semi-continuous data, I am running into a problem that you may also encounter in your upcoming data analysis process. Please kindly share your viewpoints on my problem if you wish: Modeling zero-censored semi-continuous data with PROC SEVERITY - SAS Support Communities. Thank you! ... View more

Thank you for answering my question! But I am not sure whether problems related to modeling instead of the software itself is appropriate to be raised to the technical support assistants, who essentially did the job of statistical consultency if I really forwarded my question. In addition, I myself have previously contacted SAS's technical support assistants before and I think that their responses to my questions were rather slow. Moreover, as they typically communicate with me via e-mail, my very few encounters with them all ended up in their non-response, namely they simply stopped replying to my e-mail after briefly replying to me for a couple of times. In all, I thank you for pointing out this modality of seeking for help but will stay here and wait for somebody to answer for a while. I have chatted with many knowledgable and kind friends on this forum. ... View more

I saw your post above that your dataset contained missing data and you were to repetitively go through the modeling process on each imputed sample. An informal approach I can think of is to use the Firth's method and exact logistic regression on one (or several) of the imputed dataset and see how the results differ. If this tentative modeling process does not find any big difference between them, then we might speculate that the differences in each of the unanalyzed imputed dataset is not big either. Therefore, the combined estimate, which, according to Rubin's rules, is in fact the arithmetic mean of the regression coefficients calculated from each imputed sample, is not significantly affected by the selection of estimating method (i.e., Firth's method or exact logistic regression). ... View more

Thank you so much, for your detailed and patient response! I tried searching for papers comparing the methods online yet ended up finding little useful information. Your concise comparison is very informative and fruitful. Based upon my findings online, I think a field of further research on this issue is the empirical (i.e., simulation-based) and theoretical (i.e., mathematics-based) comparison of the accuracy and (or) efficiency of the estimators formed by the two methods. You mentioned that exact logistic regression is a computationally extensive method, yet this deficiency can be (partially) overcome by supercomputers that are becoming increasingly accessible to industries. Therefore, the comparison of the two methods is becoming an issue worth detailed investigation. I hope more statisticians can read my appeal and realize it. ... View more

I think this phenomenon is largely caused by the small sample size and its resultant wide confidence interval. You can try the exact logistic regression method as suggested by @StatDave to see if it is helpful in its alleviation, but I am pessimistic on whether the mere change of method will have any impact at all. The best solution to gaining a narrower confidence interval and a resultant less extreme OR or HR value is to increase your sample size. If that is impossible, then I think you should report the results as they are. ... View more