DATA Step, Macro, Functions and more

SAS/R: Benchmarks - Microsoft R Open(MKL) versus Vanilla R Covariance timings

Reply
Valued Guide
Posts: 505

SAS/R: Benchmarks - Microsoft R Open(MKL) versus Vanilla R Covariance timings

SAS/R: Benchmarks - Microsoft R Open(MKL) versus Vanilla R Covariance timings

I am having difficulty with the SAS timing of 262.17. A response
to this post on the SAS-L claimed that the IML code below ran in about 5 seconds.
What I need is someone to run the SAS IML code below and post the
result.

Microsoft R includes a multi-core math language package

Run on Dell T7400(2008) dual XEON(3gz) no hyperthreading,
64gb ram(ddr2), dual rais 0 arrays on separate channels

inspired by
https://goo.gl/glD4Vs
http://stackoverflow.com/questions/39941842/computation-of-large-correlation-for-genomic-data-using-bigcor-function

Other than Cholesky decomposition for Matrix Inversion. Computing a cross product X prime X is
critical for most statistical analysis

Below are timings for the covariance of a a 63480 by 1803 matrix.


MIcrosoft R with MLK and 6 cores              5.44 seconds
IML                                         262.17 seconds  (2015 Dell I5 laptop DDR3 and hyperthreading took 176 seconds)
Vanilla R 3.2.4 (crossproduct function)     284.77 seconds
Vanilla R 3.2.4 (tcrossproduct function)    216305 (killed did not complete)

MIcrosoft R with MLK and 3 cores 8.11 seconds
MIcrosoft R with MLK and 4 cores 6.84
MIcrosoft R with MLK and 5 cores 6.09
MIcrosoft R with MLK and 6 cores 5.44
MIcrosoft R with MLK and 7 cores 5.20
MIcrosoft R with MLK and 8 cores 5.14

It is notable that MIcrosoft R with MLK does not scale beyond about 5 cores.
Much better to partition the problem over 1000(200 5 core jobs)?

*   *   ***          ****          *   *  *      *   *
** **  *   *         *   *         ** **  *      *  *
* * *   *            *   *         * * *  *      * *
*   *    *           ****          *   *  *      **
*   *     *          * *           *   *  *      * *
*   *  *   *         *  *          *   *  *      *  *
*   *   ***          *   *         *   *  *****  *   *

#! MSaRaMLK ;

* Microsoft R;
%utl_submit_msr64('
if(require("RevoUtilsMath")){
  setMKLthreads(5)
};
m <-  63480;
n <-  1803;
A <- matrix (runif (m*n),m,n);
nrow(A);
ncol(A);
A<-scale(A, scale = FALSE);
system.time (B <- crossprod(A)/(nrow(A)-1));
B[1000:1005,500:505];
');


> if(require("RevoUtilsMath")){  setMKLthreads(5)};m <-  63480;
n <-  1803;A <- matrix (runif (m*n),m,n);nrow(A);ncol(A);
A<-scale(A, scale = FALSE);system.time (B <- crossp
rod(A)/(nrow(A)-1));B[1000:1005,500:505];
[1] 63480
[1] 1803
   user  system elapsed
  30.03    0.30    6.09
              [,1]          [,2]          [,3]          [,4]          [,5]
[1,]  0.0001457259  0.0003179404  4.706374e-04 -2.511438e-04  3.223882e-04
[2,] -0.0003414748  0.0002328494  1.912994e-04  2.633961e-04 -1.929325e-04
[3,]  0.0001358837 -0.0004326236  6.505861e-05 -1.532928e-04 -5.720538e-04
[4,]  0.0002039385  0.0008505185  1.294216e-03 -9.633184e-05  2.294814e-04
[5,] -0.0001477482 -0.0004466242 -2.424956e-04  5.761261e-05  6.838813e-05
[6,]  0.0001911581 -0.0004942158 -1.878654e-04  8.138017e-06 -1.317634e-04
              [,6]
[1,] -0.0003329041
[2,]  0.0004478483
[3,]  0.0001240792
[4,]  0.0002072300
[5,]  0.0004954472
[6,]  0.0001689799
>

NOTE: 20 records were read from the infile RUT.
      The minimum record length was 2.
      The maximum record length was 212.
NOTE: DATA statement used (Total process time):
      real time           20.35 seconds
      user cpu time       0.01 seconds
      system cpu time     0.04 seconds
      memory              260.93k
      OS Memory           18932.00k
      Timestamp           10/12/2016 07:02:34 PM
      Step Count                        169  Switch Count  0

*****  *   *  *
  *    ** **  *
  *    * * *  *
  *    *   *  *
  *    *   *  *
  *    *   *  *
*****  *   *  *****

#! IML ;


proc iml;
seed = 123456;
a = uniform( j(63480,1803, seed) );
C = A - A[:,];          /* center the matrix */
t1=time();
d = C`*C / (nrow(C)-1); /* cross product and division; assumes nonmissing */
t2=time()-t1;
print t2;
quit;

    T2

  262.166

NOTE: IML Ready
894   seed = 123456;
895   a = uniform( j(63480,1803, seed) );
896   C = A - A[:,];
896 !                         /* center the matrix */
897   t1=time();
898   d = C`*C / (nrow(C)-1);
898 !                         /* cross product and division; assumes nonmissing */
899   t2=time()-t1;
900   print t2;
901   quit;
NOTE: Exiting IML.
NOTE: PROCEDURE IML used (Total process time):
      real time           4:45.75
      user cpu time       17:17.42
      system cpu time     1.17 seconds
      memory              2708311.68k
      OS Memory           2726860.00k
      Timestamp           10/12/2016 04:28:30 PM
      Step Count                        132  Switch Count  0


****           ***           ***             *
*   *         *   *         *   *           **
*   *             *             *          * *
****            **            **          *****
* *               *          *               *
*  *          *   *         *                *
*   *          ***          *****            *

#! Ra3a2a4 ;

%utl_submit_r64('
m <-  63480;
n <-  1803 ;
A <- matrix (runif (m*n),m,n);
nrow(A);
ncol(A);
system.time(A<-scale(A, scale = FALSE));
system.time(B <- crossprod(A)/(nrow(A)-1));
B[1000:1005,500:505];
');


> m <-  63480;n <-  1803 ;A <- matrix (runif (m*n),m,n);nrow(A);
ncol(A);system.time(A<-scale(A, scale = FALSE));
system.time(B <- crossprod(A)/(nrow(A)-1));b[1000:1005,500:
505];
[1] 63480
[1] 1803
   user  system elapsed
  12.79    0.51   13.31
   user  system elapsed
 284.71    0.03  284.77


NOTE: 7 records were read from the infile RUT.
      The minimum record length was 8.
      The maximum record length was 176.
NOTE: DATA statement used (Total process time):
      real time           5:06.36
      user cpu time       0.01 seconds
      system cpu time     0.04 seconds
      memory              260.93k
      OS Memory           18932.00k
      Timestamp           10/12/2016 07:12:20 PM
      Step Count                        183  Switch Count  0

Ask a Question
Discussion stats
  • 0 replies
  • 291 views
  • 0 likes
  • 1 in conversation