topic Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency in SAS Programming

SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

mandan414 — Mon, 01 Feb 2021 18:15:10 GMT

Hello,

I'm operating cross tabulations like below:

proc surveyfreq data=mydata varmethod=BRR;

tables Var1 * Var2 * Var3

Var1 * Var2 * Var4

Var1 * Var2 * Var5

………………

……………..

Var1 * Var2 * Var9 / row cl nofreq nowt nocellpercent nototal;

repweights BSW1-BSW1000;

weight weight_var;

run;

Var1 has 5 values, var2 has 2 values and var3 or var 4...… can have from 3 to 6 values. SAS operation is really slow for this. I tried only one table (tables Var1 * Var2 * Var3 ) with 2 bootstrap weight (BSW1-BSW2) and it took SAS 1.5mins to complete it. Any suggestions?

Thank you so much.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

ballardw — Mon, 01 Feb 2021 18:40:08 GMT

How many records are involved in this data set?

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

mandan414 — Mon, 01 Feb 2021 18:42:35 GMT

Above 100,000 records. Thanks!

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

Reeza — Mon, 01 Feb 2021 18:46:05 GMT

Is your table local or on a network drive? Have you tried loading the file with the SASFILE command?
I'm assuming you have decent RAM as well, if you're using SAS University Edition the default settings are low.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

mandan414 — Mon, 01 Feb 2021 18:50:38 GMT

Reeza. My SAS is on network drive. It's for my work and I must use the network and work computer, no other choice. Thanks.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

SASKiwi — Mon, 01 Feb 2021 19:15:01 GMT

Where does your SAS WORK library point to? I'm hoping it is your local drive. If so run a test using data copied to WORK to see if that improves performance.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

mandan414 — Tue, 02 Feb 2021 17:47:23 GMT

SAS work library is on the network. Thanks.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

Reeza — Tue, 02 Feb 2021 21:13:25 GMT

Not to be a pest and repeat myself again, but have you tried SASFILE?
I would assume a bootstrap option would require many reads so it may help.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

andreas_lds — Wed, 03 Feb 2021 06:13:58 GMT

@mandan414 wrote:

SAS work library is on the network. Thanks.

This is ok, if - and only if - sas runs on a server. If sas is installed on your computer, having work on a network share is, kindly put, a really bad idea.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

mandan414 — Wed, 03 Feb 2021 12:35:05 GMT

Hi Reeza,

I tried it, it's super! Much much faster! Although, I still don't know what SASFILE exactly does. I'm going to study about it. Thank you!

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

mandan414 — Wed, 03 Feb 2021 12:37:43 GMT

I don't quite understand it. Could you explain it a little more? Thank you for your time.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

andreas_lds — Wed, 03 Feb 2021 14:20:57 GMT

If you have a local installation of sas any network-traffic will, most likely, slow down the execution of you sas job. So using local drives is highly recommended.

Re: SAS slow operation for Multiple cross-tabulations for bootstrap-based frequency

Reeza — Wed, 03 Feb 2021 16:49:47 GMT

It loads the data into RAM so it's not being re-read constantly so it should speed things up significantly when there are multiple reads of the same file and you're using it for data intensive calculations.