How to improve execution speed of program in Base SAS?

[RobF]

Thanks Jaap, much appreciated. I'll put DS2 and the advanced data mgmt functions FriedEgg used on my reading list!

[Kurt_Bremser]

One final note: keep in mind that the best optimizer is still the package of brain & eyeballs, aka human being. As friedegg has displayed once again.

[RobF]

Yes, agreed.

The weird thing that still nags me is that my logistic regression code (and the Poisson regression code which is very similar) converges to a solution which is close but not exactly the same as the glmnet solution. I expected that my code would either be way off the mark or dead on, not really close but still off a bit.

So my algorithm is still missing something - it's hard to say without delving into the Fortran code that Jerome Friedman wrote.

[Kurt_Bremser]

You may experience a precision problem. SAS uses 8 bytes real to store numbers (52 bit mantissa), the FORTRAN/R code may work with extended precision or employ 64 bit integer math where possible.

[RobF]

Good point - but would that translate into a parameter estimate being off by more than a few percentage pts?

[jakarman]

This is one of the rare occasions I disagree with Kurt.   The Floating point routines are part of the processor. All programming languages are using that same processor and by that are having the same limitations. As you are using advanced multiplication / division that is the best place to have them used. The way it works is standardized.  Double-precision floating-point format - Wikipedia, the free encyclopedia

There dedicated routines optimize for scalar.matrix processing in generic libraries having limited checks around them and there are routines full of checks. That is making a difference in speed. SAS is not having the option of using the GPU for calculations with those optimized for imaging you have a real difference in precision not always being told .