topic Re: improving running time of the program. in SAS Programming

improving running time of the program.

irena_g — Tue, 24 Nov 2020 11:01:39 GMT

Hello,

We work with big files. They are more 8000000 rows and more than 2000 columns. Our program runs for 4 hours. We want to improve the running time of the program.

I changed all merge section to sql code. It’s don’t help me very much. My program now runs for 3:45 hours.
I tried using simple data set statements and used SPDE files and indexes. I removed all proc sort statements--now my program runs for 3:25 hours.

May be you have another idea how can I improve the running time of the program.

Many thanks,

Irena

Re: improving running time of the program.

PeterClemmensen — Tue, 24 Nov 2020 11:05:21 GMT

This is a huge topic and depends largely on what your code does.

Post your code and let ud have a look.

Re: improving running time of the program.

irena_g — Tue, 24 Nov 2020 11:35:02 GMT

@PeterClemmensen Thanks

Unfortunately, I can't to post the program code. The program consists mainly of the merges.

Re: improving running time of the program.

PeterClemmensen — Tue, 24 Nov 2020 11:38:23 GMT

Do you sort the data prior to merging?

Re: improving running time of the program.

andreas_lds — Tue, 24 Nov 2020 11:44:38 GMT

@irena_g wrote:

@PeterClemmensen Thanks

Unfortunately, I can't to post the program code. The program consists mainly of the merges.

Without seeing the code and the log it is hardly possible to suggest something improving the runtime.

When calling proc sort and the data is already sorted, using the option "presorted" will reduce the time proc sort needs.
Maybe the high running time is caused by the hardware involved. Can you post some details on the hardware you have?

Re: improving running time of the program.

Astounding — Tue, 24 Nov 2020 14:26:58 GMT

As others have noted, "can't see the program" + "can't see the data" = "can only guess"

So here's my guess.

You need to fix the data. For example, it takes longer to process 200 characters than it takes to process one character. So if (for example) GENDER should be "M" or "F" but it is actually defined as 200 characters long, you need to fix that first. There's a lot of work involved to do this, starting with PROC CONTENTS to see the actual lengths being used.

Re: improving running time of the program.

ballardw — Tue, 24 Nov 2020 15:12:28 GMT

TAGSORT can reduce the temporary disk space used for sorts on large datasets improving performance.

If you think that your variable names give away too much information ("can't share code") then your names contain information that belongs in variables and not in the names and may be the reason you have too many variables. Plus processing information in names means that you likely have other data structure issues that negatively impact performance.

Re: improving running time of the program.

irena_g — Wed, 25 Nov 2020 04:57:22 GMT

@PeterClemmensen In spde files i use options BYSORT=YES so Ican skip proc sort.