Base SAS + SAS/CONNECT - A simple method to generate load on any number of licensed cores
Views
5,177
Before diving into the detail my thanks to my colleague Mark Thomas who helped me with MP Connect code and to colleagues in Technical Support and R&D for raising this as a topic of interest. From time to time, customers ask SAS how to determine if SAS 9.4 is able to use all the cores available to it. This is sometimes as a result of their own testing which appears to be inconclusive e.g. a customer had over 12+ cores available to them but in their testing they were only seeing a handful of cores being leveraged. So what was going on? Well here are some reminders when it comes to Base SAS:
- For quick explanation about threads and SAS Foundation, read What Is Threading Technology in SAS?
- Many of the SAS Foundation procs and a big chunk of datastep were written to execute in single threaded mode
- However there are some procs and data step that are multithreaded either by default or by setting some options. For more information read Threading in Base SAS.
- For some procs that can use multiple threads, they exploit by-group processing to enable each thread to focus on a particular by-group (or multiple by-groups depending on how many groups and how many threads are available)
- SAS Foundation licensing can be cores based
- Demonstrating multi-threading using multi-threaded procs may be limited by the constraints of the underlying storage being able to provide the data in a timely manner
- For a clear definition of features which MP Connect (which is part of SAS/CONNECT) brings read MP CONNECT.
So with all this in mind, the question of whether users can generate SAS jobs that don't rely on sufficient data to generate enough CPU load, to occupy as many cores in a symmetric multiprocessing (SMP), server springs to mind. There are probably multiple solutions to this, but one that sprang to my mind was the generation of prime numbers. Is there a sufficient number of prime numbers to find within a set of numbers e.g. 1 - 10000, that would generate sufficient load so that a user/observer could see generating across all cores (or as many cores as needed). BUT we also have to ensure that the number is not too large that the test takes hours to complete. So we want to have some code which to some degree is inefficient but not too inefficient.
With help from Mark and his knowledge of MP Connect and macros, this is the code we came up with. In essence for this example we generate 10 distinct sessions (which equates to 10 of threads out of a possible 88, and yes this is a hyperthreaded host, so 88 threads = 44 cores) and sought to find the prime numbers available in the first 100,000 numbers (1-100,000) and write them to SAS datasets. If you're wondering there 9592 but our code generates 9591 into a table (can you guess which one it leaves out?). As mentioned earlier the code is meant to be inefficient but not too inefficient. So in our approach, we deliberately exclude all even numbers from the test. For greater efficiencies you can take a look these resources to get you started:
- How many prime numbers?
- Using SAS/IML to sieve prime numbers by Rick Wicklin
- SAS/CONNECT -> MP CONNECT documentation
- Running SAS programs in parallel using SAS/CONNECT by Leonid Batkhan
The code below is what we used for our tests:
options sascmd=”!SASCMD” fullstimer symbolgen;
/* Set the ‘num” macro variable to the number of sessions you want to create */
%let num=10 ;
/* this loop initiates sessions and runs the code to generate prime numbers */
%macro loop;
%local i ;
%do i=1 %to &num
%let session=sess&i. ;
%let &session.=myddfai.sas.com;
signon &session.;
%let testds=test&i. ;
%syslput testds=&testds.;
rsubmit &session. wait=no ;
data &testds.;
do i = 1 to 100000 by 2;
L=0;
do j=1 to i;
k=0;
if (mod(i , j)= 0) then do;
k+1;
a=mod(i , j);
end;
if K=1 then L+1;
end;
If L=2 then output;
end;
run;
endrsubmit;
%end ;
%mend loop;
%loop ;
/* macro to generate a string of session names */
%macro gensess;
%do i=1 %to &num
sess&i.
%end ;
%mend gensess ;
listtask _all_ ;
waitfor _all_ %gensess;
/* macro to download datasets generated in each session */
%macro download;
%local i ;
%do i=1 %to &num
%let testds=test&i. ;
%syslput testds=&testds. /remote=sess&i.;
rsubmit sess&i. wait=no ;
proc download data=work.&testds. out=work.&testds.;
run;
endrsubmit ;
%end ;
%mend download;
%download ;
/* macro to signoff each of the connect sessions */
%macro signoff;
%local i ;
%do i=1 %to &num
signoff sess&i. ;
%end ;
%mend signoff;
%signoff ;
/* proc datasets lib=work ; */
/* delete test1-test10 ; */
/* run; */
Below is the log that it generates, when we look at the first 100,000 numbers. The interesting takeaway is that on the this specific server, it takes roughly between 53-56 seconds to create the tables at the server side.
Using the linux utility "htop" us to look at the demand on each core and if needed we can take a screenshot (as we did below). We did not test this using SAS Environment Manager, but there's a good chance that in Viya 3.5 you will see the spikes in CPU activity.
1 %studio_hide_wrapper;
82 options sascmd=”!SASCMD” fullstimer symbolgen;
83
84 /* Set the ‘num” macro variable to the number of sessions you want to create */
85 %let num=10 ;
86
87 /* this loop initiates sessions and runs the code to generate prime numbers */
88 %macro loop;
89
90 %local i ;
91
92 %do i=1 %to #
93
94 %let session=sess&i. ;
95 %let &session.=myddfai.sas.com;
96 signon &session.;
97
98 %let testds=test&i. ;
99 %syslput testds=&testds.;
100 rsubmit &session. wait=no ;
101
102 data &testds.;
103 do i = 1 to 100000 by 2;
104 L=0;
105 do j=1 to i;
106 k=0;
107 if (mod(i , j)= 0) then do;
108 k+1;
109 a=mod(i , j);
110 end;
111 if K=1 then L+1;
112 end;
113 If L=2 then output;
114 end;
115 run;
116
117 endrsubmit;
118 %end ;
119 %mend loop;
120
121 %loop ;
SYMBOLGEN: Macro variable NUM resolves to 10
SYMBOLGEN: Macro variable I resolves to 1
SYMBOLGEN: Macro variable SESSION resolves to sess1
SYMBOLGEN: Macro variable SESSION resolves to sess1
NOTE: Remote signon to SESS1 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.04 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS1 complete.
SYMBOLGEN: Macro variable I resolves to 1
SYMBOLGEN: Macro variable TESTDS resolves to test1
SYMBOLGEN: Macro variable SESSION resolves to sess1
NOTE: Background remote submit to SESS1 in progress.
SYMBOLGEN: Macro variable I resolves to 2
SYMBOLGEN: Macro variable SESSION resolves to sess2
SYMBOLGEN: Macro variable SESSION resolves to sess2
NOTE: Remote signon to SESS2 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.05 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS2 complete.
SYMBOLGEN: Macro variable I resolves to 2
SYMBOLGEN: Macro variable TESTDS resolves to test2
SYMBOLGEN: Macro variable SESSION resolves to sess2
NOTE: Background remote submit to SESS2 in progress.
SYMBOLGEN: Macro variable I resolves to 3
SYMBOLGEN: Macro variable SESSION resolves to sess3
SYMBOLGEN: Macro variable SESSION resolves to sess3
NOTE: Remote signon to SESS3 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.05 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS3 complete.
SYMBOLGEN: Macro variable I resolves to 3
SYMBOLGEN: Macro variable TESTDS resolves to test3
SYMBOLGEN: Macro variable SESSION resolves to sess3
NOTE: Background remote submit to SESS3 in progress.
SYMBOLGEN: Macro variable I resolves to 4
SYMBOLGEN: Macro variable SESSION resolves to sess4
SYMBOLGEN: Macro variable SESSION resolves to sess4
NOTE: Remote signon to SESS4 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.05 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS4 complete.
SYMBOLGEN: Macro variable I resolves to 4
SYMBOLGEN: Macro variable TESTDS resolves to test4
SYMBOLGEN: Macro variable SESSION resolves to sess4
NOTE: Background remote submit to SESS4 in progress.
SYMBOLGEN: Macro variable I resolves to 5
SYMBOLGEN: Macro variable SESSION resolves to sess5
SYMBOLGEN: Macro variable SESSION resolves to sess5
NOTE: Remote signon to SESS5 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.04 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS5 complete.
SYMBOLGEN: Macro variable I resolves to 5
SYMBOLGEN: Macro variable TESTDS resolves to test5
SYMBOLGEN: Macro variable SESSION resolves to sess5
NOTE: Background remote submit to SESS5 in progress.
SYMBOLGEN: Macro variable I resolves to 6
SYMBOLGEN: Macro variable SESSION resolves to sess6
SYMBOLGEN: Macro variable SESSION resolves to sess6
NOTE: Remote signon to SESS6 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.07 seconds
cpu time 0.06 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS6 complete.
SYMBOLGEN: Macro variable I resolves to 6
SYMBOLGEN: Macro variable TESTDS resolves to test6
SYMBOLGEN: Macro variable SESSION resolves to sess6
NOTE: Background remote submit to SESS6 in progress.
SYMBOLGEN: Macro variable I resolves to 7
SYMBOLGEN: Macro variable SESSION resolves to sess7
SYMBOLGEN: Macro variable SESSION resolves to sess7
NOTE: Remote signon to SESS7 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.06 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS7 complete.
SYMBOLGEN: Macro variable I resolves to 7
SYMBOLGEN: Macro variable TESTDS resolves to test7
SYMBOLGEN: Macro variable SESSION resolves to sess7
NOTE: Background remote submit to SESS7 in progress.
SYMBOLGEN: Macro variable I resolves to 8
SYMBOLGEN: Macro variable SESSION resolves to sess8
SYMBOLGEN: Macro variable SESSION resolves to sess8
NOTE: Remote signon to SESS8 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.05 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS8 complete.
SYMBOLGEN: Macro variable I resolves to 8
SYMBOLGEN: Macro variable TESTDS resolves to test8
SYMBOLGEN: Macro variable SESSION resolves to sess8
NOTE: Background remote submit to SESS8 in progress.
SYMBOLGEN: Macro variable I resolves to 9
SYMBOLGEN: Macro variable SESSION resolves to sess9
SYMBOLGEN: Macro variable SESSION resolves to sess9
NOTE: Remote signon to SESS9 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.05 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS9 complete.
SYMBOLGEN: Macro variable I resolves to 9
SYMBOLGEN: Macro variable TESTDS resolves to test9
SYMBOLGEN: Macro variable SESSION resolves to sess9
NOTE: Background remote submit to SESS9 in progress.
SYMBOLGEN: Macro variable I resolves to 10
SYMBOLGEN: Macro variable SESSION resolves to sess10
SYMBOLGEN: Macro variable SESSION resolves to sess10
NOTE: Remote signon to SESS10 commencing (SAS Release V.03.05M0P110619).
NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA.
NOTE: SAS (r) Proprietary Software V.03.05 (TS5M0 MBCS3170)
Licensed to GEL Viya 3.5- ImplWkshp – 19w47 – Ship – Shared, Site 70180938.
NOTE: This session is executing on the Linux 4.1.12-124.19.2.el6uek.x86_64 (LIN X64) platform.
NOTE: Analytical products:
SAS/STAT 15.1
SAS/ETS 15.1
SAS/OR 15.1
SAS/IML 15.1
SAS/QC 15.1
NOTE: Additional host information:
Linux LIN X64 4.1.12-124.19.2.el6uek.x86_64 #2 SMP Fri Sep 14 08:59:12 PDT 2018 x86_64 Oracle Linux Server release 6.10
NOTE: SAS initialization used:
real time 0.06 seconds
cpu time 0.06 seconds
NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/compsrv/default/autoexec.sas.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to SESS10 complete.
SYMBOLGEN: Macro variable I resolves to 10
SYMBOLGEN: Macro variable TESTDS resolves to test10
SYMBOLGEN: Macro variable SESSION resolves to sess10
NOTE: Background remote submit to SESS10 in progress.
122
123 /* macro to generate a string of session names */
124 %macro gensess;
125 %do i=1 %to #
126 sess&i.
127 %end ;
128 %mend gensess ;
“SESS10” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS9” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS8” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS7” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS6” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS5” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS4” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS3” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS2” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
“SESS1” ————–
Type: SAS/CONNECT Process
State: RUNNING ASYNCHRONOUSLY
129
130 listtask _all_ ;
131 waitfor _all_ %gensess;
SYMBOLGEN: Macro variable NUM resolves to 10
SYMBOLGEN: Macro variable I resolves to 1
SYMBOLGEN: Macro variable I resolves to 2
SYMBOLGEN: Macro variable I resolves to 3
SYMBOLGEN: Macro variable I resolves to 4
SYMBOLGEN: Macro variable I resolves to 5
SYMBOLGEN: Macro variable I resolves to 6
SYMBOLGEN: Macro variable I resolves to 7
SYMBOLGEN: Macro variable I resolves to 8
SYMBOLGEN: Macro variable I resolves to 9
SYMBOLGEN: Macro variable I resolves to 10
132
133
134 /* macro to download datasets generated in each session */
135 %macro download;
136 %local i ;
137
138 %do i=1 %to #
139
140 %let testds=test&i. ;
141
142 %syslput testds=&testds. /remote=sess&i.;
143 rsubmit sess&i. wait=no ;
144 proc download data=work.&testds. out=work.&testds.;
145 run;
146 endrsubmit ;
147 %end ;
148 %mend download;
149
150 %download ;
SYMBOLGEN: Macro variable NUM resolves to 10
SYMBOLGEN: Macro variable I resolves to 1
SYMBOLGEN: Macro variable TESTDS resolves to test1
SYMBOLGEN: Macro variable I resolves to 1
SYMBOLGEN: Macro variable I resolves to 1
SYMBOLGEN: Macro variable I resolves to 2
SYMBOLGEN: Macro variable TESTDS resolves to test2
SYMBOLGEN: Macro variable I resolves to 2
SYMBOLGEN: Macro variable I resolves to 2
SYMBOLGEN: Macro variable I resolves to 3
SYMBOLGEN: Macro variable TESTDS resolves to test3
SYMBOLGEN: Macro variable I resolves to 3
SYMBOLGEN: Macro variable I resolves to 3
SYMBOLGEN: Macro variable I resolves to 4
SYMBOLGEN: Macro variable TESTDS resolves to test4
SYMBOLGEN: Macro variable I resolves to 4
SYMBOLGEN: Macro variable I resolves to 4
SYMBOLGEN: Macro variable I resolves to 5
SYMBOLGEN: Macro variable TESTDS resolves to test5
SYMBOLGEN: Macro variable I resolves to 5
SYMBOLGEN: Macro variable I resolves to 5
SYMBOLGEN: Macro variable I resolves to 6
SYMBOLGEN: Macro variable TESTDS resolves to test6
SYMBOLGEN: Macro variable I resolves to 6
SYMBOLGEN: Macro variable I resolves to 6
SYMBOLGEN: Macro variable I resolves to 7
SYMBOLGEN: Macro variable TESTDS resolves to test7
SYMBOLGEN: Macro variable I resolves to 7
SYMBOLGEN: Macro variable I resolves to 7
SYMBOLGEN: Macro variable I resolves to 8
SYMBOLGEN: Macro variable TESTDS resolves to test8
SYMBOLGEN: Macro variable I resolves to 8
SYMBOLGEN: Macro variable I resolves to 8
SYMBOLGEN: Macro variable I resolves to 9
SYMBOLGEN: Macro variable TESTDS resolves to test9
SYMBOLGEN: Macro variable I resolves to 9
SYMBOLGEN: Macro variable I resolves to 9
SYMBOLGEN: Macro variable I resolves to 10
SYMBOLGEN: Macro variable TESTDS resolves to test10
SYMBOLGEN: Macro variable I resolves to 10
SYMBOLGEN: Macro variable I resolves to 10
151
152
153 /* macro to signoff each of the connect sessions */
154 %macro signoff;
155
156 %local i ;
157
158 %do i=1 %to #
159 signoff sess&i. ;
160 %end ;
161 %mend signoff;
162
163 %signoff ;
SYMBOLGEN: Macro variable NUM resolves to 10
SYMBOLGEN: Macro variable I resolves to 1
NOTE: Remote submit to SESS1 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST1 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 55.61 seconds
cpu time 55.60 seconds
3 The SAS System Sunday, 21 March 2021 14:06:24
NOTE: Remote submit to SESS1 complete.
NOTE: Background remote submit to SESS1 in progress.
NOTE: Remote submit to SESS1 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST1 to out=WORK.TEST1
NOTE: 383640 bytes were transferred at 19158950 bytes/second.
NOTE: The data set WORK.TEST1 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST1 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.02 seconds
cpu time 0.00 seconds
NOTE: Remote submit to SESS1 complete.
NOTE: Remote signoff from SESS1 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 57.45 seconds
cpu time 55.71 seconds
NOTE: Remote signoff from SESS1 complete.
SYMBOLGEN: Macro variable I resolves to 2
NOTE: Remote submit to SESS2 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST2 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 55.26 seconds
cpu time 55.25 seconds
3 The SAS System Sunday, 21 March 2021 14:06:24
NOTE: Remote submit to SESS2 complete.
NOTE: Background remote submit to SESS2 in progress.
NOTE: Remote submit to SESS2 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST2 to out=WORK.TEST2
NOTE: 383640 bytes were transferred at 5682803 bytes/second.
NOTE: The data set WORK.TEST2 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST2 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.07 seconds
cpu time 0.02 seconds
NOTE: Remote submit to SESS2 complete.
NOTE: Remote signoff from SESS2 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 57.34 seconds
cpu time 55.37 seconds
NOTE: Remote signoff from SESS2 complete.
SYMBOLGEN: Macro variable I resolves to 3
NOTE: Remote submit to SESS3 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST3 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 54.93 seconds
cpu time 54.92 seconds
3 The SAS System Sunday, 21 March 2021 14:06:24
NOTE: Remote submit to SESS3 complete.
NOTE: Background remote submit to SESS3 in progress.
NOTE: Remote submit to SESS3 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST3 to out=WORK.TEST3
NOTE: 383640 bytes were transferred at 5913536 bytes/second.
NOTE: The data set WORK.TEST3 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST3 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.07 seconds
cpu time 0.01 seconds
NOTE: Remote submit to SESS3 complete.
NOTE: Remote signoff from SESS3 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 57.21 seconds
cpu time 55.04 seconds
NOTE: Remote signoff from SESS3 complete.
SYMBOLGEN: Macro variable I resolves to 4
NOTE: Remote submit to SESS4 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST4 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 55.48 seconds
cpu time 55.47 seconds
4 The SAS System Sunday, 21 March 2021 14:06:25
NOTE: Remote submit to SESS4 complete.
NOTE: Background remote submit to SESS4 in progress.
NOTE: Remote submit to SESS4 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST4 to out=WORK.TEST4
NOTE: 383640 bytes were transferred at 6157076 bytes/second.
NOTE: The data set WORK.TEST4 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST4 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.07 seconds
cpu time 0.01 seconds
NOTE: Remote submit to SESS4 complete.
NOTE: Remote signoff from SESS4 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 57.08 seconds
cpu time 55.59 seconds
NOTE: Remote signoff from SESS4 complete.
SYMBOLGEN: Macro variable I resolves to 5
NOTE: Remote submit to SESS5 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST5 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 54.91 seconds
cpu time 54.90 seconds
4 The SAS System Sunday, 21 March 2021 14:06:24
NOTE: Remote submit to SESS5 complete.
NOTE: Background remote submit to SESS5 in progress.
NOTE: Remote submit to SESS5 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST5 to out=WORK.TEST5
NOTE: 383640 bytes were transferred at 4911086 bytes/second.
NOTE: The data set WORK.TEST5 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST5 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.09 seconds
cpu time 0.02 seconds
NOTE: Remote submit to SESS5 complete.
NOTE: Remote signoff from SESS5 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 56.95 seconds
cpu time 54.99 seconds
NOTE: Remote signoff from SESS5 complete.
SYMBOLGEN: Macro variable I resolves to 6
NOTE: Remote submit to SESS6 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST6 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 55.57 seconds
cpu time 55.54 seconds
4 The SAS System Sunday, 21 March 2021 14:06:25
NOTE: Remote submit to SESS6 complete.
NOTE: Background remote submit to SESS6 in progress.
NOTE: Remote submit to SESS6 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST6 to out=WORK.TEST6
NOTE: 383640 bytes were transferred at 4664201 bytes/second.
NOTE: The data set WORK.TEST6 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST6 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.08 seconds
cpu time 0.00 seconds
NOTE: Remote submit to SESS6 complete.
NOTE: Remote signoff from SESS6 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 56.83 seconds
cpu time 55.66 seconds
NOTE: Remote signoff from SESS6 complete.
SYMBOLGEN: Macro variable I resolves to 7
NOTE: Remote submit to SESS7 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST7 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 55.23 seconds
cpu time 55.22 seconds
4 The SAS System Sunday, 21 March 2021 14:06:25
NOTE: Remote submit to SESS7 complete.
NOTE: Background remote submit to SESS7 in progress.
NOTE: Remote submit to SESS7 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST7 to out=WORK.TEST7
NOTE: 383640 bytes were transferred at 4030212 bytes/second.
NOTE: The data set WORK.TEST7 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST7 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.10 seconds
cpu time 0.00 seconds
NOTE: Remote submit to SESS7 complete.
NOTE: Remote signoff from SESS7 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 56.69 seconds
cpu time 55.34 seconds
NOTE: Remote signoff from SESS7 complete.
SYMBOLGEN: Macro variable I resolves to 8
NOTE: Remote submit to SESS8 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST8 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 54.88 seconds
cpu time 54.86 seconds
4 The SAS System Sunday, 21 March 2021 14:06:25
NOTE: Remote submit to SESS8 complete.
NOTE: Background remote submit to SESS8 in progress.
NOTE: Remote submit to SESS8 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST8 to out=WORK.TEST8
NOTE: 383640 bytes were transferred at 4171902 bytes/second.
NOTE: The data set WORK.TEST8 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST8 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.10 seconds
cpu time 0.01 seconds
NOTE: Remote submit to SESS8 complete.
NOTE: Remote signoff from SESS8 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 56.56 seconds
cpu time 54.99 seconds
NOTE: Remote signoff from SESS8 complete.
SYMBOLGEN: Macro variable I resolves to 9
NOTE: Remote submit to SESS9 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST9 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 54.85 seconds
cpu time 54.84 seconds
4 The SAS System Sunday, 21 March 2021 14:06:25
NOTE: Remote submit to SESS9 complete.
NOTE: Background remote submit to SESS9 in progress.
NOTE: Remote submit to SESS9 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST9 to out=WORK.TEST9
NOTE: 383640 bytes were transferred at 3724369 bytes/second.
NOTE: The data set WORK.TEST9 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST9 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.11 seconds
cpu time 0.02 seconds
NOTE: Remote submit to SESS9 complete.
NOTE: Remote signoff from SESS9 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 56.43 seconds
cpu time 54.96 seconds
NOTE: Remote signoff from SESS9 complete.
SYMBOLGEN: Macro variable I resolves to 10
NOTE: Remote submit to SESS10 commencing.
1 data &testds.;
2 do i = 1 to 100000 by 2;
3 L=0;
4 do j=1 to i;
5 k=0;
6 if (mod(i , j)= 0) then do;
7 k+1;
8 a=mod(i , j);
9 end;
10 if K=1 then L+1;
11 end;
12 If L=2 then output;
13 end;
14 run;
NOTE: The data set WORK.TEST10 has 9591 observations and 5 variables.
NOTE: DATA statement used (Total process time):
real time 55.09 seconds
cpu time 55.08 seconds
4 The SAS System Sunday, 21 March 2021 14:06:26
NOTE: Remote submit to SESS10 complete.
NOTE: Background remote submit to SESS10 in progress.
NOTE: Remote submit to SESS10 commencing.
15 proc download data=work.&testds. out=work.&testds.;
16 run;
NOTE: Download in progress from data=WORK.TEST10 to out=WORK.TEST10
NOTE: 383640 bytes were transferred at 3675059 bytes/second.
NOTE: The data set WORK.TEST10 has 9591 observations and 5 variables.
NOTE: Downloaded 9591 observations of 5 variables.
NOTE: The data set WORK.TEST10 has 9591 observations and 5 variables.
NOTE: PROCEDURE DOWNLOAD used (Total process time):
real time 0.11 seconds
cpu time 0.00 seconds
NOTE: Remote submit to SESS10 complete.
NOTE: Remote signoff from SESS10 commencing.
NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414
NOTE: The SAS System used:
real time 56.30 seconds
cpu time 55.20 seconds
NOTE: Remote signoff from SESS10 complete.
164
165
166
167 /* proc datasets lib=work ; */
168 /* delete test1-test5 ; */
169 /* run; */
170
171
172 %studio_hide_wrapper;
SYMBOLGEN: Macro variable _STUDIONOTES resolves to
SYMBOLGEN: Macro variable _STUDIOSOURCE resolves to
SYMBOLGEN: Macro variable _STUDIOSTIMER resolves to
SYMBOLGEN: Macro variable GRAPHTERM resolves to GOPTIONS NOACCESSIBLE;
SYMBOLGEN: Macro variable _STUDIONOTES resolves to NOTES
SYMBOLGEN: Macro variable _STUDIOSOURCE resolves to SOURCE
SYMBOLGEN: Macro variable _STUDIOSTIMER resolves to STIMER
183
184
What does the CPU load look like for 10 sessions when run against a machine with 88 available threads (44 hyperthreaded cores)? Here's a screenshot when using the Htop command.
Select the image to see a larger version.
Mobile users: To view the image, select the "Full" version at the bottom of the page.
Mobile users: To view the image, select the "Full" version at the bottom of the page.
Some final thoughts and comments:
- Stress testing CPUs outside of SAS can be done with readily available Linux utilities like "Stress" which are part of the RHEL/CentOS EPEL repository
- Care should be taken when running any of these tests to ensure no other users are currently running workloads and the tests take a matter of minutes and not hours
- When running such tests, users must consider whether they are staying within SAS licensing terms and conditions
- Having done this in Base SAS, it would be an interesting exercise to see how this could be done in CAS
- The possibility of using performing the same test in SAS Studio or Enterprise Guide without the need for SAS/CONNECT is something that is also of future interest
Well that's all folks. I hope you found it interesting and I'm sure that that there will be some of you who can come up with additional ways to load an SMP using Base SAS code. Feel free to share what you know in the comments section below.
Thanks, Simon
