proc sql using or

[Steelers_In_DC]

I have some legacy code that takes a very long time to run, on join is > 8 hours.  When I break it down it has many or and multiple filters:

proc sql;

create table have as

select a.*,b.*

from table 1 a,

         table 2 b

where a.id = b.id

            and blah

      or where a.id = b.id

          and blah blah blah

I've broken this apart into 6 different queries.  I concatenate at the end in a dataset and it runs so much faster I'm shocked.  From hours to minutes.  I can't find much on using or like this other than it's not efficient.  Is there any rules/laws/best practices/documentation somewhere that details this?

I'd like to have more than my test to take back to the original developers to get them to change their ways.  Thank You in advance

[Tom · Accepted solution]

Assuming you meant something like:

where (condition1)
   or (condition2)
   or (condition3)
....

Then the issue is probably related to the size of the data and whether or not indexes can be used.

If you just have a simple condition involving equality test and AND operators then the indexes can be used to only join the observations that meet the criteria.

Once you start adding in OR conditions that is impossible for the SQL execution plan to take into account so it is probably reverting to doing full table scans for both input tables.  Which can result in a hugh intermediate file that might require a lot of DISK I/O to create and re-read.

[awesome_opossum]

Principally speaking though, proc sql is multi-threaded, so you can also ramp up your processors. SAS uses 4 by default, but if your computer/server is faster, you can increase it. However, not all SAS procs are multi-threaded, just SQL happens to be one.