I have some legacy code that takes a very long time to run, on join is > 8 hours. When I break it down it has many or and multiple filters:
proc sql;
create table have as
select a.*,b.*
from table 1 a,
table 2 b
where a.id = b.id
and blah
or where a.id = b.id
and blah blah blah
I've broken this apart into 6 different queries. I concatenate at the end in a dataset and it runs so much faster I'm shocked. From hours to minutes. I can't find much on using or like this other than it's not efficient. Is there any rules/laws/best practices/documentation somewhere that details this?
I'd like to have more than my test to take back to the original developers to get them to change their ways. Thank You in advance
Assuming you meant something like:
where (condition1)
or (condition2)
or (condition3)
....
Then the issue is probably related to the size of the data and whether or not indexes can be used.
If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.
Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables. Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.
Assuming you meant something like:
where (condition1)
or (condition2)
or (condition3)
....
Then the issue is probably related to the size of the data and whether or not indexes can be used.
If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.
Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables. Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.