BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
Steelers_In_DC
Barite | Level 11

I have some legacy code that takes a very long time to run, on join is > 8 hours.  When I break it down it has many or and multiple filters:

 

proc sql;

create table have as

select a.*,b.*

from table 1 a,

         table 2 b

where a.id = b.id

            and blah

      or where a.id = b.id

          and blah blah blah

 

 

I've broken this apart into 6 different queries.  I concatenate at the end in a dataset and it runs so much faster I'm shocked.  From hours to minutes.  I can't find much on using or like this other than it's not efficient.  Is there any rules/laws/best practices/documentation somewhere that details this?

 

I'd like to have more than my test to take back to the original developers to get them to change their ways.  Thank You in advance

 

1 ACCEPTED SOLUTION

Accepted Solutions
Tom
Super User Tom
Super User

Assuming you meant something like:

where (condition1)
   or (condition2)
   or (condition3)
....

Then the issue is probably related to the size of the data and whether or not indexes can be used.

 

If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.

 

Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables.  Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.

View solution in original post

2 REPLIES 2
Tom
Super User Tom
Super User

Assuming you meant something like:

where (condition1)
   or (condition2)
   or (condition3)
....

Then the issue is probably related to the size of the data and whether or not indexes can be used.

 

If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.

 

Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables.  Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.

awesome_opossum
Obsidian | Level 7
Principally speaking though, proc sql is multi-threaded, so you can also ramp up your processors. SAS uses 4 by default, but if your computer/server is faster, you can increase it. However, not all SAS procs are multi-threaded, just SQL happens to be one.

sas-innovate-2024.png

 

Secure your spot at the must-attend AI and analytics event of 2024: SAS Innovate 2024! Get ready for a jam-packed agenda featuring workshops, super demos, breakout sessions, roundtables, inspiring keynotes and incredible networking events.

 

Register by March 1 to snag the Early Bird rate of just $695! Don't miss out on this exclusive offer. 

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 330 views
  • 2 likes
  • 3 in conversation