I have some legacy code that takes a very long time to run, on join is > 8 hours. When I break it down it has many or and multiple filters:
proc sql;
create table have as
select a.*,b.*
from table 1 a,
table 2 b
where a.id = b.id
and blah
or where a.id = b.id
and blah blah blah
I've broken this apart into 6 different queries. I concatenate at the end in a dataset and it runs so much faster I'm shocked. From hours to minutes. I can't find much on using or like this other than it's not efficient. Is there any rules/laws/best practices/documentation somewhere that details this?
I'd like to have more than my test to take back to the original developers to get them to change their ways. Thank You in advance
Assuming you meant something like:
where (condition1)
or (condition2)
or (condition3)
....
Then the issue is probably related to the size of the data and whether or not indexes can be used.
If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.
Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables. Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.
Assuming you meant something like:
where (condition1)
or (condition2)
or (condition3)
....
Then the issue is probably related to the size of the data and whether or not indexes can be used.
If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.
Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables. Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.
Find more tutorials on the SAS Users YouTube channel.