"we run into slow I/O" - am not sure how I can provide any information on that, there are thousands of hardware/software setups. Consult with your IT group to analyse the reasons behind it.
500gb doesn't sound that much.
"especially if we are processing it with temporary datasets" - what do you mean? SAS code you have written being run writing to temporary datasets? If so look at the code you are writing. Whilst there will always be a certain amount of resources needed to process that data (and do note that R loads directly into memory so is limited to your memory, but will be quicker), its more likely to how the data is being processed which will impact speed. Excessive use of macro, ignoring Base SAS and by group processing, Excel thinking (all of which you can see plenty of examples on this forum) often cause lots of processing, read/writes etc. which consume resources. Start by looking at the log, see which procedures/processes take up most time/resource, then refactor them.
... View more