I'm new to DS2, but learning. We would like to use DS2 to cut down on processing time for certain processes we run. From some simple testing, I'm finding a DS2 threaded process runs slower and takes almost twice as long than if I use PROC SQL. This is the case even though I can see that DS2 uses several thread processes. I find this quite puzzling since I had hoped DS2 would run faster. Any input / comments why DS2 is running slower? Query and logs below.
/* proc sql for comparing to ds2 thread step below -- basically copying a compressed dataset */ proc sql; create table lib1.ds2_test_procsql as select segment, region, state, MBR_ENRLMNT_CNT from lib1.final;
NOTE: Table LIB1.DS2_TEST_PROCSQL created, with 35811639 rows and 4 columns.
NOTE: PROCEDURE SQL used (Total process time): real time 2:04.90 user cpu time 54.25 seconds system cpu time 7.44 seconds memory 6290.06k OS Memory 28840.00k Timestamp 05/17/2018 08:43:26 AM Step Count 39 Switch Count 284 Page Faults 0 Page Reclaims 163 Page Swaps 0 Voluntary Context Switches 3487 Involuntary Context Switches 33598 Block Input Operations 0 Block Output Operations 0
/* ds2 thread steps */ proc ds2; thread newton/overwrite=yes; dcl double y count; dcl bigint thisThread; drop count y thisThread;
method run(); set {select segment, region, state, MBR_ENRLMNT_CNT from lib1.final}; thisThread= _threadid_; count+1; end; method term(); put '**Thread' _threadid_ 'processed' count 'rows:'; end; endthread; run; quit;
proc ds2; data lib1.ds2_test_thread / overwrite=yes; dcl thread newton frac;
method run(); set from frac threads=4; end; enddata; run; quit;
**Thread 1 processed 4086810 rows: **Thread 0 processed 4037670 rows: **Thread 3 processed 13699140 rows: **Thread 2 processed 13988019 rows: NOTE: Execution succeeded. 35811639 rows affected.
NOTE: PROCEDURE DS2 used (Total process time): real time 3:47.06 user cpu time 1:49.49 system cpu time 15.59 seconds memory 5952.46k OS Memory 28336.00k Timestamp 05/17/2018 08:47:14 AM Step Count 42 Switch Count 66 Page Faults 0 Page Reclaims 1620 Page Swaps 0 Voluntary Context Switches 56263 Involuntary Context Switches 64441 Block Input Operations 0 Block Output Operations 0
... View more
