topic Speeding up SAS in SAS Procedures

Speeding up SAS

vomer — Thu, 23 Feb 2012 17:54:55 GMT

Hi guys,

I have some fairly large data sets that are created on a monthly basis. I was wondering if you have any tips on data compression techniqes to speed up the time SAS takes to create these data sets?

Thanks for the tips!

Speeding up SAS

sassharp — Thu, 23 Feb 2012 18:07:49 GMT

Instead of using proc sql by libname reference go through Oracle or ODBMS pass through. try to use dbkey in proc sql. It drastically increases the join times.

Speeding up SAS

sassharp — Thu, 23 Feb 2012 18:08:59 GMT

just paste some code here. there are legends they help you tune up your code.

Speeding up SAS

Astounding — Thu, 23 Feb 2012 18:21:04 GMT

vomer,

You're asking two different questions. Compression affects disk space usage. As a general rule, this does not speed up programs but rather slows them down slightly. What are you trying to accomplish?

Speeding up SAS

vomer — Thu, 23 Feb 2012 18:30:02 GMT

Mainly looking to speed up the code run time. Wondering if thewe are any tricks that everyone here has used to speed things up in general.

Speeding up SAS

FriedEgg — Thu, 23 Feb 2012 18:31:02 GMT

Astounding,

Since the slower computing operation is typically I/O compression often does speed up programs by allowing the disk to push the data faster at the cost of increased CPU usage. It all depends where your bottlenecking in performance.

vomer,

Please provide additional information about how these files are created currently on a monthly basis and any system information you know or feel comfortable sharing, especially OS, available storage types, cpu counts, ram, etc..

What is the nature of the data, does it have natural segmentation for the analysis you intend to do with it? Why do you feel this program is running slow? What is it's current performance metrics and where would you want it to be?

Increasing performance is a very large question to ask so vaguely.

Speeding up SAS

Tom — Thu, 23 Feb 2012 20:50:33 GMT

Summarize as early in the process as possible.

Speeding up SAS

art297 — Thu, 23 Feb 2012 23:23:05 GMT

Like others have said, seeing what your data and data needs are, as well as environment, would have to be known in order for anyone to suggest valid possibilities.

Tom mentioned summarizing as early as possible. I agree, but would even back up a step before that. Only import and/or keep the data that are really necessary.

Speeding up SAS

twocanbazza — Fri, 24 Feb 2012 00:28:46 GMT

In My experience compressing has actually speed up the process, it really depends on where the bottle neck is... Ie if you have slow disk, writing/reading as little as possible helps...

Barry

Re: Speeding up SAS

LinusH — Fri, 24 Feb 2012 08:42:11 GMT

If we look at compression as such, if your data is mainly character, try the ordinary CHAR method.

If you have lots of numerical data, try BINARY.

If you have lots of numerical integers, you can try to specify other length than the default 8.

As Barry says, it deponds on your system and data, if compression will help processing time. The penalty for compression is more CPU cycles.

My experience is that you need at least 50% compression to gain shorter processing time.

Hoe long is your run time anyway? Do you have indexes defined?

Depending on the requirements, some re-modelling of the data can sometimes help.

Re: Speeding up SAS

ballardw — Fri, 24 Feb 2012 18:53:37 GMT

If I don't want to babysit the program while running I'd be tempted to schedule it to run while I'm out of the office, say overnight if possible. Then everything is done when I come in the morning.