In a nutshell, @hewei2005 wrote: 1. Does GLMSELECT LASSO by default, assume response variable is continuous and approximately normally distributed?  the assumption of the statistical model depends on the model, not the method of parameter estimation. Therefore, if you are building linear regression models, the normality assumption is required regardless of whether you employ LASSO or not. @hewei2005 wrote: 2. A key assumption of traditional linear regression is that the residuals (the differences between the observed and predicted values) are normally distributed. This allows for statistical inference and hypothesis testing. Can we relax this assumption when doing LASSO and how to implement a NON-normal distribution of error in GLMSELECT (if the answer to question 1 is that GLMSELECT do assume normal distribution)?   Thank you.  No. However, if your residual does not follow a normal distribution, then (1) transformation of the dependent variable into normality via Box-Cox transformation or (2) resort to the generalized linear model (GLM) if the dependent variable follows certain distributions that can be modeled by GLM. Please note that LASSO can also be applied in GLM's. ... View more

@Dylan1 wrote: I was of the view that given the proportional hazard assumption has failed when I used the Cox. The Royston parmar was the closest option which doesnt have a number of assumptions as compared to the other models. The proportional hazards assumption is also needed in the the Royston-Parmar model, according to this source:   ... View more

Thank you so much for your very detailed summarization of your experiences in dealing with real-world datasets! I think your response deserves much more than a mere like, but unforunately, that is all I can offer. Still, now that you have invested plenty of your time on composing these words, I strongly suggest that you go a step further and transform them into a paper that may eventually appear in SAS User Group or academic journals like Journal of Statistical Software. Your experience is invaluable for both SAS users and beyond. @ballardw wrote: @Season wrote: Does importing the file with the DATA step necessitates specification of the name and informat of each of every variable in the CSV? That would be a very formidable job as I have possibly thousands of columns in all. Check on the assigned informat for your problem variables. If they were read as character but should be dates that is an indication that you may need to create new variables by parsing the values. Check on your national language settings (NLS) to see what order dates are read. OR if you see lots of invalid data messages involving those variables it is one indicator that the order may be different than your NLS and override to read as character and parse. Still, I would like to further consult on NLS. I searched on the web and saw Microsoft having a webpage on NLS, but with a slightly different meaning- national language service. I am not sure if the two NLS's are the same, but anyway, could you please brief introduce what national language settings is and what impact does it have on importing datasets into statistical softwares like SAS? I used to think that the mere difference in the language of the interface and log of SAS does not really have an impact on its core capabilities such as loading and editing datasets. @ballardw wrote: Any data source that may have "thousands of columns" and doesn't provide documentation as to content of the file, such as expected lengths of character variables and layouts of date, time or datetime values needs to be considered with great suspicion. Without documentation how do you know what anything represents? Finally, I would like to make a clarification. Your reminder is of great value and I thank you for it. However, the datasets I use does have a documentation. In fact, it is huge as it has a description for every variable therein. Still, only descriptions of variables instead of their intriacacies are documented. In other words, for a given column, I only know that it stands for a date with a particular meaning but do not know that it can take multiple formats like "12/16/15" and "12 16 15". Only when I imported it into SAS did I realize this issue. ... View more

Thank you so much for your reminder as well as your code! I will try modifying the DATA step codes as recommended. ... View more

@Kathryn_SAS wrote: The documentation link that I sent is specifically for delimited files - csv, tab, dlm - so it does not include statements/options that are not available for you to use for your CSV files. Thank you for your reminder! I should had taken a second look. @Kathryn_SAS wrote: Another thought would be that you can run PROC IMPORT because it uses DATA step code behind the scenes. Then you could grab that code either from the log or using Run -> Recall Last Submit and then you could make adjustments to that DATA step code as others have suggested. You just wouldn't have to start from scratch. Thank you for your valuable piece of information! So that makes DATA step-based import of CSV files easier. Starting from scratch is so painful and boring for a data analyst who wish to concentrate on data analysis instead of such dull, monotonic work. ... View more

Thanks for your response. But I beg to point out that the options listed in SAS Help are not very informative as many of those listed there are not available for CSV's. ... View more

@Tom wrote: What do you mean by varying?  Do you mean that it changes from file to file?  Or do you mean it changes within a single file? I mean "within a single file". @Tom wrote: Your example is not actually a problem for SAS to read (no idea if it is problem for PROC IMPORT to guess how to handle). data test; infile cards dsd truncover; input date :ddmmyy10.; format date yymmdd10.; cards; "16/12/20" "16 12 20" ; Obs date 1 2020-12-16 2 2020-12-16   Yes, I did try PROC IMPORT. Does importing the file with the DATA step necessitates specification of the name and informat of each of every variable in the CSV? That would be a very formidable job as I have possibly thousands of columns in all. ... View more

Thanks for the input! Yes, you are right. Space serves as the delimiter in none of the three informats you mentioned in your post. @ballardw wrote: However that is best when the years are 4 digits. With 2 digit years you have to pray that the order of values entered does match the national language settings.  I am also rather apprehensive on this issue. With no further information in my dataset, it is likely that I end up designating the digits with the wrong meaning. However, as dates are not the primary concern for the time being, I have to focus on the goal of successful import right now and think about the issue of dates when conditions are permitted. ... View more

Thank you so much! That is a very handy option. By the way, in addition to the GUESSINGROWS statement, are there any useful options and/or statements available in importing CSV's? As I have said in my original question, many of the options and/or statements of PROC IMPORT are not available for CSV's, which is frustrating. ... View more

Thank you! Could you please provide some hint on how to construct temporary variables? Hopefully this does not entail the specification of each and every column in the CSV file like I see in many SAS codes importing CSV's. I have too many columns to specify them one by one. ... View more

I saw you repetitively raise the same question in the forum. I think that the absence of response from the community signifies that maybe nobody in the forum really knows how to solve your problem in SAS. Therefore, I suggest that you contact the SAS support team by telephone or e-mail or resort to other statistical softwares. ... View more

Thanks for sharing. One thing I would like to further comment on is the functional form of the time-dependent variables. As has been shown in the preceding SAS code, the time-dependent variable is assumed to be linear. However, it should be noted that in theory, nonlinear versions of time-dependent variables are also permitted in Cox models. See, for instance Using fractional polynomials and restricted cubic splines to model non‐proportional hazards or time‐varying covariate effects in the Cox regression model - Austin - 2022 - Statistics in Medicine - Wiley Online Library for more on the theoretical intricacies and SAS codes of nonlinear time-dependent variables. Still, the linear time-dependent variable is the most widely used functional form in practice. ... View more

Take a look at 33307 - Scoring (computing predicted values) for new observations or a validation data set. Special thanks are given to @StatDave for introducing me this note. ... View more