BookmarkSubscribeRSS Feed
☑ This topic is solved. Need further help from the community? Please sign in and ask a new question.
Steelers_In_DC
Barite | Level 11

I have some legacy code that takes a very long time to run, on join is > 8 hours.  When I break it down it has many or and multiple filters:

 

proc sql;

create table have as

select a.*,b.*

from table 1 a,

         table 2 b

where a.id = b.id

            and blah

      or where a.id = b.id

          and blah blah blah

 

 

I've broken this apart into 6 different queries.  I concatenate at the end in a dataset and it runs so much faster I'm shocked.  From hours to minutes.  I can't find much on using or like this other than it's not efficient.  Is there any rules/laws/best practices/documentation somewhere that details this?

 

I'd like to have more than my test to take back to the original developers to get them to change their ways.  Thank You in advance

 

1 ACCEPTED SOLUTION

Accepted Solutions
Tom
Super User Tom
Super User

Assuming you meant something like:

where (condition1)
   or (condition2)
   or (condition3)
....

Then the issue is probably related to the size of the data and whether or not indexes can be used.

 

If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.

 

Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables.  Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.

View solution in original post

2 REPLIES 2
Tom
Super User Tom
Super User

Assuming you meant something like:

where (condition1)
   or (condition2)
   or (condition3)
....

Then the issue is probably related to the size of the data and whether or not indexes can be used.

 

If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.

 

Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables.  Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.

awesome_opossum
Obsidian | Level 7
Principally speaking though, proc sql is multi-threaded, so you can also ramp up your processors. SAS uses 4 by default, but if your computer/server is faster, you can increase it. However, not all SAS procs are multi-threaded, just SQL happens to be one.

SAS INNOVATE 2024

Innovate_SAS_Blue.png

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Get the $99 certification deal.jpg

 

 

Back in the Classroom!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 2 replies
  • 181 views
  • 2 likes
  • 3 in conversation