BookmarkSubscribeRSS Feed
mattstat
Calcite | Level 5
I am updating some old code which previously took 14 hours to improve efficiency. I'm trying to use a Proc Summary with a Class statement instead of sorting the dataset first, followed by a Proc Summary. When I take a small sample of the dataset this Proc Summary works fine, but with the entire dataset it ran for 10 hours before I halted it. Based on the past code I would only expect about a 2-3 hour run time.

When I broke the proc summary, this the run time:

NOTE: PROCEDURE SUMMARY used:
real time 8:07:47.26
cpu time 19:56.10

Any ideas?
2 REPLIES 2
Cynthia_sas
SAS Super FREQ
This is really a question for Tech Support. They are in the best position to look at your stored process code and give you some ideas of how to improve performance -- there may be a problem in the data, in the configuration of the servers used or the network or anyplace in between.

To find out how to contact Tech Support, refer to:
http://support.sas.com/techsup/contact/index.html

cynthia
deleted_user
Not applicable
I think you're going the wrong way with this change. Your subject mentions "memory issue", which suggests you know there is a memory problem. I would turn on step monitoring with [ Options FullSTimer; ] which will give you memory usage as well.

When you summarise using a by statement, the summary data for each stratum is written out at the end of each group, and the data are processed in a linear sequence. If you use a class statement on unsorted data, SAS has to hold a table of stats for each level that isn't written until the end of the table. Memory usage is very much higher, and for large data sets can cause the step to fail when memory is exhausted.

So, I would plan to stay with the BY statement, but look further back in the code to see if the source data could have been generated in sorted order to save the sort step. Sorting data is expensive to the process, but not sorting prior to large table summarisations can be even more expensive.

Kind regards

David

SAS INNOVATE 2024

Innovate_SAS_Blue.png

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Get the $99 certification deal.jpg

 

 

Back in the Classroom!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 721 views
  • 0 likes
  • 3 in conversation