topic Re: Optimize PROC MCMC software/hardware environment in Statistical Procedures

Optimize PROC MCMC software/hardware environment

Haris — Wed, 26 Jan 2022 19:05:20 GMT

Need to run multiple parallel MCMC processes. Models currently take between 8 and 21 hours to complete one-by-one. Looking for any advice in terms of software OPTIONS settings such as THREADS, MEMSIZE, etc.

Also hardware: what should I prioritize -- get more or faster RAM, single thread CPU speed, more cores/threads, more virtual memory, faster I/O as in would upgrading from SATA to M2 interface.

Any help or references much appreciated.

Re: Optimize PROC MCMC software/hardware environment

SASKiwi — Wed, 26 Jan 2022 21:16:17 GMT

Is this on a PC or on a remote SAS server? With PCs parallel processing is not really an option if the process itself already chews up most resources.

Re: Optimize PROC MCMC software/hardware environment

Haris — Thu, 27 Jan 2022 01:24:46 GMT

Thanks for your reply, SASKiwi,

The answer is 'in between'. I am working on a Virtual Machine with Local SAS installation. During preliminary testing, I was able to fire off three simultaneous MCMC threads that, initially, took up close to 70% of CPU power but eventually settled at just over 40%. Looks like the CPU can handle double that.

RAM behaved very strangely: even though I have 64GB dynamically allocated RAM, SAS grabbed to 8GB on the start of the first process and did not go up from there? Not sure why not. The server can let me get up to 256GB.

I don't know much about the I/O other than the server runs SSD. This is an interim solution and, if needed, we can build a dedicated machine to handle the analytics.

Can you work with this? I'd love to hear any input I can get. Any additional questions I can answer to facilitate the exchange? Thanks again.

Re: Optimize PROC MCMC software/hardware environment

Haris — Thu, 27 Jan 2022 01:32:17 GMT

BTW, if helpful, single models took 8.5 hours to run. When running three at the same time, the execution time increased to 9.5 hours for each but total start-to-finish time was clearly saved in concert.