topic Re: Proc SQL with left join clause so long in execution. in SAS Procedures

Proc SQL with left join clause so long in execution.

KOUPGANG — Wed, 06 Jul 2016 16:58:11 GMT

Hi all,

Please i think i need help here.

I Have a proc sql code running for more than 4 hours to give me an output.

the dataset on the left (Base0) has only 1 million rows, but Base2 is a remote dataset in a global repository.

PLEASE why does it takes so long to execute (4 hours) ?

How can it be optimized ?

NB Note than the key variable doesnt have the same structure in both dataset.

Thanks

proc sql;

create table lib1.Base1 as

select a.*, b.DW_DATE_KEY, b.SERVICE_CLASS,

b.TOTAL_VOICE_AMT,b.TOTAL_DURATION,

b.NB_CALLS, b.NB_CALLS_ONNET, b.NB_CALLS_OFFNET,b.NB_CALLS_INTERNATIONAL,

b.VOICE_AMT_ONNET,b.VOICE_AMT_OFFNET,b.VOICE_AMT_INTERNATIONAL,

b.VOICE_DUREE_ONNET,b.VOICE_DUREE_OFFNET , b.VOICE_DUREE_INTERNATIONAL,

b.MA_VOICE_USED, b.BONUS_VOICE_USED

from lib1.Base0 as a

left join lib2.Base1(where =(DW_DATE_KEY >= 20160516 and DW_DATE_KEY <=20160630)) as b

on a.msisdn =(input(b.msisdn, best12.)-237000000000);

quit;

Re: Proc SQL with left join clause so long in execution.

Doc_Duke — Wed, 06 Jul 2016 18:51:14 GMT

A guess that should help.

Copy the subset of lib2 to WORK and do the ID transform during the copy. The left join may be making multiple passes through lib2 and input is not the most efficient function in SAS.

Re: Proc SQL with left join clause so long in execution.

Kurt_Bremser — Wed, 06 Jul 2016 18:54:07 GMT

I'd fetch the base2 table to a local SAS dataset in a separate step. In that step, do the calculation for your key. Then sort per the key, which should make things much easier for the SQL step.

You might still find that proc sort and a data step merge are faster. If you have a many-to many relationship requiring a cartesian join, you must use SQL, of course.

Re: Proc SQL with left join clause so long in execution.

KOUPGANG — Thu, 07 Jul 2016 08:31:52 GMT

Thanks a lot. Its faster whith Base2 dataset in Work library.

Re: Proc SQL with left join clause so long in execution.

LinusH — Thu, 07 Jul 2016 08:43:46 GMT

Options msglevel = i sastrace = ',,,d' sastraceloc = saslog;
Helps you understand what gets passed to the underlying dbms.
Try to avoid function calls in filter/join criterias.
How many records does base2 have?
How many records typically meet the date fire criteria?
Any indexes on dw_date_key or msisdn?