topic Solving least squares problem with sparse matrix in SAS/IML Software and Matrix Computations https://communities.sas.com/t5/SAS-IML-Software-and-Matrix/Solving-least-squares-problem-with-sparse-matrix/m-p/187961#M1912 <HTML><HEAD></HEAD><BODY><P>This question is related to a question I posted in the SAS/STAT forum (<A __default_attr="207873" __jive_macro_name="message" class="jive_macro jive_macro_message" href="https://communities.sas.com/" modifiedtitle="true" title="Re: Regression with large number of fixed effects in a sparse matrix"></A>).</P><P></P><P>I have a large and sparse design matrix of ones and zeros (about 450k rows, 2500 columns). I'm trying to use this to run a regression, but SAS/STAT procedures are taking too long. (I ran proc hpmixed and it had not produced any results after 24 hours.)</P><P></P><P>If I read the data into Matlab as a sparse matrix, I can run a QR decomposition and invert the resulting matrix in less than 10 seconds. Is it possible to take this approach in IML? I feel like I must be missing something because it doesn't seem like this is a hard problem to solve, so I thought SAS would handle everything on the fly, but it seems that I'm going wrong somewhere.</P><P></P><P>Thanks for any advice.</P></BODY></HTML> Mon, 21 Apr 2014 19:27:59 GMT stoffprof 2014-04-21T19:27:59Z Solving least squares problem with sparse matrix https://communities.sas.com/t5/SAS-IML-Software-and-Matrix/Solving-least-squares-problem-with-sparse-matrix/m-p/187961#M1912 <HTML><HEAD></HEAD><BODY><P>This question is related to a question I posted in the SAS/STAT forum (<A __default_attr="207873" __jive_macro_name="message" class="jive_macro jive_macro_message" href="https://communities.sas.com/" modifiedtitle="true" title="Re: Regression with large number of fixed effects in a sparse matrix"></A>).</P><P></P><P>I have a large and sparse design matrix of ones and zeros (about 450k rows, 2500 columns). I'm trying to use this to run a regression, but SAS/STAT procedures are taking too long. (I ran proc hpmixed and it had not produced any results after 24 hours.)</P><P></P><P>If I read the data into Matlab as a sparse matrix, I can run a QR decomposition and invert the resulting matrix in less than 10 seconds. Is it possible to take this approach in IML? I feel like I must be missing something because it doesn't seem like this is a hard problem to solve, so I thought SAS would handle everything on the fly, but it seems that I'm going wrong somewhere.</P><P></P><P>Thanks for any advice.</P></BODY></HTML> Mon, 21 Apr 2014 19:27:59 GMT https://communities.sas.com/t5/SAS-IML-Software-and-Matrix/Solving-least-squares-problem-with-sparse-matrix/m-p/187961#M1912 stoffprof 2014-04-21T19:27:59Z Re: Solving least squares problem with sparse matrix https://communities.sas.com/t5/SAS-IML-Software-and-Matrix/Solving-least-squares-problem-with-sparse-matrix/m-p/187962#M1913 <HTML><HEAD></HEAD><BODY><P>In your MATLAB example, if X = QR, then R and Q are dense 2500x2500 matrices, So I guess you are asking whether IML has a sparse QR decomposition?&nbsp; No, it does not.</P></BODY></HTML> Mon, 21 Apr 2014 20:56:08 GMT https://communities.sas.com/t5/SAS-IML-Software-and-Matrix/Solving-least-squares-problem-with-sparse-matrix/m-p/187962#M1913 Rick_SAS 2014-04-21T20:56:08Z