DATA Step, Macro, Functions and more

Using parallel procesing (threads, cpucount)

Reply
Frequent Contributor
Posts: 122

Using parallel procesing (threads, cpucount)

Hi,

I am trying to use parallel processing in order to know the way to get better performance.(SAS EG 6.1)

I am doing a proc means over a 50 millions (no many fields) datasets.

I don't know much about parallel processing but I thinks the default option may be threads=YES. I suppose that because real time < cpu time (the half more or less), then I suppose that the procedure is executed with paralelization.

Is there anyway of seeing if my process runs parallel (log, option...)??

I have tried to modify the parallization options using:

OPTIONS THREADS=YES CPUCOUNT=8;

I have tested with 2,4,6 and 8 CPUs and the exec times remains very similar.

Is there bay other way to change paralellization options?, is this usefull or SAS by default run parallel when needed.

Any advice will be greatly apreciatted.

My code:

options fullstimer;

OPTIONS THREADS=YES CPUCOUNT=8;

options mlogic mprint mtrace symbolgen;

%let fecha_ini = %sysevalf(%sysfunc(date()) - 20);

%put fecha_ini = &fecha_ini;

data ventas(drop = i);

do i = 1 to 50000000;

cod_articulo = byte(80 + round(10*ranuni(1)))||put(round(ranuni(1)*100),z4.);

cod_centro = byte(65 + round(2*ranuni(1)))||put(round(ranuni(1)*5),z2.);

hc_venta = round(20*ranuni(1),.01);

fc_venta = &fecha_ini. + round(10*ranuni(1)) ;

format fc_venta ddmmyyn8.;

output;

end;

run;

proc means data=ventas noprint nway;

class cod_centro cod_articulo;

var hc_venta;

output out=ag_ventas(drop=_type_ _freq_)

sum(hc_venta) = suma_ventas ;

run;

/*

By default (no user option):

real time 2.29 segundos

user cpu time 5.57 segundos

cpucount=4

real time 2.29 segundos

user cpu time 5.57 segundos

cpucount=6

real time 2.30 segundos

user cpu time 5.54 segundos

cpucount=8

real time 2.31 segundos

user cpu time 5.58 segundos

*/

Super User
Posts: 5,424

Re: Using parallel procesing (threads, cpucount)

Posted in reply to juanvg1972

Trying to answer strictly:

"Is there anyway of seeing if my process runs parallel (log, option...)?"

You already noticed the first way, by comparing real with CPU time. This is not however 100% reliable, so by setting the option MSGLEVEL=I, you will get a NOTE in the log that tells that the processing is using multiple threads:

NOTE: Multiple concurrent threads will be used to summarize data.

"Is there bay other way to change paralellization options?"

Besides CPUCOUNT, not to my knowledge. I guess SAS uses threads when appropriate. However, you could if got access to OS tool be able to monitor your SAS process, but I can't really see the point if you are not having serious problems.

If I read behind the lines, you wish to optimize your code. Besides threads, summarization needs memory, so increasing MEMSIZE/SORTSIZE  might help.

You said you are dong this over 50 millions datasets. DO you mean 50 million row datasets (not actually 50 M datasets, right - otherwise you have a serious structure problem Smiley Happy ). So if you wish to run multiple MEANS, do that in parallel, each wit CPUCOUNT=4. DO this using either MP CONNECT, or by having them executed in seperate SAS sessions.

Data never sleeps
Frequent Contributor
Posts: 122

Re: Using parallel procesing (threads, cpucount)

Thanks for youre help Linus,

Obviously I have 50 miilons row dataset not 50 millions datasets.

I can't see the numbers of threads of my process in my log, although I have used MSGLEVEL=I as you said.


Here my log:

NOTE: Writing TAGSETS.SASREPORT13(EGSR) Body file: EGSR

21        

22         GOPTIONS ACCESSIBLE;

23         options msglevel=I;

24        

25         proc means data=ventas noprint nway;

26         class cod_centro cod_articulo;

27         var hc_venta;

28         output out=ag_ventas(drop=_type_ _freq_)

29         sum(hc_venta) = suma_ventas ;

30         run;

NOTE: Multiple concurrent threads will be used to summarize data.

NOTE: There were 50000000 observations read from the data set WORK.VENTAS.

NOTE: The data set WORK.AG_VENTAS has 11110 observations and 3 variables.

NOTE: PROCEDIMIENTO MEANS used (Total process time):

      real time           10.06 segundos

      user cpu time       24.60 segundos

      system cpu time     0.64 segundos

      memory              10951.64k

      OS Memory           30928.00k

      Timestamp           05/07/2015 09:57:58 p.m.

      Step Count                        9  Switch Count  97

      Page Faults                       0

      Page Reclaims                     2887

      Page Swaps                        0

      Voluntary Context Switches        34630

      Involuntary Context Switches      17563

      Block Input Operations            0

      Block Output Operations           776

Trusted Advisor
Posts: 3,212

Re: Using parallel procesing (threads, cpucount)

Posted in reply to juanvg1972

Using parallel processing is one of the many techniques to improve turn around times for processes.

As a rule of thumb do that in this order of precedence:

1/ look at your used algorithm. Doing anything that is not necessary needed is waste/

    Understand the possible bottlenecks.

2/ choose of possible available solutions like already programmed ones the one that is most  appropriate.

    The difficulty is getting to know the ones you do not know without losing too much time.

3/ re-evaluate sas options.  It should be part of the installation/configuration but there can be missed a lot.

4/ re-evalutate operating system and hardware options it should be part of the design and set up but also there a lot could have gone wrong?

With proc means you are IO-bound as that is the bottleneck you cannot go faster as one pass for all. The cpucount / threading will not help at SAS only the operating system will do some thing as IO can be done delayed/predicted (caching). Base SAS(R) 9.4 Procedures Guide, Third Edition (proc means threading on input data).

When you would have many smaller datasets and the weighted options is functionally acceptable. you could imaging to pre-calculated on all those datasets generating a weighted variable with intermediate results. and then combine those results (less data) use the weight. That is similar to a map/reduce approach.

---->-- ja karman --<-----
PROC Star
Posts: 1,759

Re: Using parallel procesing (threads, cpucount)

Posted in reply to juanvg1972

Elapse is 2.5 times CPU, so you have at least 3 threads.

Since I see you are using unix, the easiest is to look at the processes spawned using for example the ps command, and count the sas processes. A more comprehensive way is to look at how the value of CPUCOUNT affects performance for your process.

Here is an example where the number of CPUs and the number of class variables vary:

This chart is taken from my book:

High-Performance SAS Coding: Christian Graffeuille: 9781512397499: Amazon.com: Books

High-Performance SAS Coding: Polychrome: Christian Graffeuille: 9781514362310: Amazon.com: Books


Trusted Advisor
Posts: 3,212

Re: Using parallel procesing (threads, cpucount)

Chris, There are major differences between threads and forks. It something fundamental for system concepts.
A nice descritpion of that is at http://www.geekride.com/fork-forking-vs-threading-thread-linux-kernel/  . The forking is the one the spawner does, that is creating new visisble sas sessions.
The OS session manager (workload balancing) will see sessions as the major anchors points (one kind of  table) wihtin each sessions threads being run.  By the sharing nature of threads that is less likely being well possible.  http://www.thegeekstuff.com/2011/04/ps-command-examples/  (see the pid/tid difference at example 6)

Yes you can see some effects as the more spread used of cpu's instead a very skewed (single) usage. In a multi-users environment with many users active and having many cpu's that is different than a desktops single user approach.  For fun the OS geek is assuming the application bein the SAS-system.   SAS applications however are built on the SAS-system. The confusion is that SAS developers are not seeing that much of the OS-level like threading/forking. However designing efficient code they shoud understand the basics of those.  

---->-- ja karman --<-----
Ask a Question
Discussion stats
  • 5 replies
  • 1086 views
  • 2 likes
  • 4 in conversation