topic Re: Question about using %include in a macro to include another macro in SAS Programming

Question about using %include in a macro to include another macro

Doxastic — Fri, 18 Jul 2014 21:31:43 GMT

Very often when I use %include within a macro, to invoke another macro or a set of macros, when I execute the main macro stops running at the %include.

So, for instance. If my macro looked like:

%macro Test();

Data Output; set input;

Var="TEST";

run;

%include "C:\user\Sub_Macros"

Data Final_output; set Sub_Macros_Input;

Var2="Finished";

run;

%mend;

The data step for "Final_output" won't run. The macro just ends at the %include.... it's really annoying.

Re: Question about using %include in a macro to include another macro

Tom — Fri, 18 Jul 2014 21:35:28 GMT

In that case it is the missing semi-colon that is making the data step not run.

Did it also complain that it could not find the files named DATE and FINAL_OUTPUT ?

Re: Question about using %include in a macro to include another macro

Tom — Fri, 18 Jul 2014 21:38:02 GMT

There is no need to define a macro inside the definition of another macro. You are just asking the computer to do extra work since now it has to re-define the macro every time you run the outer macro.

%macro Test();

Data Output; set input;

Var="TEST";

run;

* Call SUB_MACRO ;

%SUB_MACRO;

Data Final_output; set Sub_Macros_Input;

Var2="Finished";

run;

%mend;

* Include definition of SUB_MACRO ;

%include "C:\user\Sub_Macros";

* Call main macro;

%test;

Re: Question about using %include in a macro to include another macro

Doxastic — Fri, 18 Jul 2014 21:38:17 GMT

Adding the semicolon doesn't help....

It just stop executing after the %include. There's no errors, it just stops.

Re: Question about using %include in a macro to include another macro

Tom — Fri, 18 Jul 2014 21:41:28 GMT

Unbalanced quotes?

Or worse the include file includes and ENDSAS statement?

Re: Question about using %include in a macro to include another macro

Reeza — Fri, 18 Jul 2014 21:42:57 GMT

Does the %include run on its own? Without being inside a macro?

Re: Question about using %include in a macro to include another macro

Doxastic — Fri, 18 Jul 2014 21:46:54 GMT

I do place macros within macros all the time. I don't like 2000 lines in a macro.... It's just confusing. And there will often be pieces of a macro that themselves can be macro-tized for use in a different macro or just code in general.

I use SAS to write ETLs on flat files. So I have really long SAS programs. Putting everything in one macro would make some macros confusingly long....

On top of that, they won't be dynamic anymore. Here's how I'm using %include (this time):

%macro Import(FilePath);

%get_filenames(&FilePath.);

Data rename_var(keep= Campaign_Names); set campaign_file_names; Campaign_Names=VAR1; run;

data Camp_Total; set rename_var; count+1; run;

data Camp_Macro_Names(keep=Var); set Camp_Total;

Var=cat('%let campy',count,'=',Campaign_Names,';');

run;

data _null_;

set Camp_Macro_Names;

FILE "E:\Shared\Progressive\Campaigns\Specify Campaigns_&today..txt" DLM=',';

PUT Var;

run;

%include "E:\Shared\Progressive\Campaigns\Specify Campaigns_&today..txt";

proc import datafile="&FilePath.\&&campy&i...csv"

out=tempy&i

dbms=dlm replace;

delimiter=',';

getnames = no;

datarow=1;

run;

proc sql;

create table camp&i as

select a.* from tempy&i a

where var1 ne .

order by var2;

quit;

proc datasets; delete tempy&i; run; quit;

%end;

data campaigns_rerun(keep= HH_id airing_date live_offset Network daypart); set

%read2

;

rename var6 = HH_id;

rename var4 = airing_date;

rename var7 = live_offset;

rename var16 = Network;

rename var20 = daypart;

run;

%end;

%mend Import;

Everything after the %include won't run. SAS stops executing after the %include and it doesn't error out.

Re: Question about using %include in a macro to include another macro

Doxastic — Fri, 18 Jul 2014 21:49:17 GMT

I have no idea if %include has ENDSAS built into it.... nor do I know how to figure that out.

Re: Question about using %include in a macro to include another macro

Doxastic — Fri, 18 Jul 2014 21:50:06 GMT

Yes. %include runs on its own. It also runs inside the macro. SAS stops executing AFTER the %include. So it runs that piece, it just stops after.

Re: Question about using %include in a macro to include another macro

Tom — Fri, 18 Jul 2014 21:59:00 GMT

Sounds like a data issue. You are using the values of a data step variable to write %LET statements. Most likely the content of the file contains something that should be macro quoted.

Note that you are not including any macro definitions here. You are including some lines of code with %LET statements to define some macro variables.

And not even lines of code from some other code file, but lines of code generated from data.

Note that you can replace the four data steps and the %INCLUDE with a single data _NULL_ that uses CALL SYMPUTX to create the macro variables.

data _null_ ;

set campaign_file_names;

call symputx(cats('campy',_n_),var1);

run;

There are more issues after that also as you seem to want to loop of these macro variables since you later reference &&CAMPY&I but there is no macro logic to do the looping.

Also you are calling a macro in the middle of a SET statement. Perhaps you meant to reference a macro variable instead at that point?

Re: Question about using %include in a macro to include another macro

Doxastic — Fri, 18 Jul 2014 22:03:47 GMT

I'll try that again... but that's what I first tried and I couldn't get it working. I'll get back in a bit to let you know if it works.

Re: Question about using %include in a macro to include another macro

Tom — Fri, 18 Jul 2014 23:07:33 GMT

Looks like you just want to read all of the CSV files in a folder. You seem to be assuming they all have the same format, so why not just write the data step to read them instead of letting PROC IMPORT guess what is in the file?

From looking at your program it looks like something like this should work.

%macro Import(FilePath);

data campaigns_rerun ;

length filename fname $256 ;

infile "&FilePath\*.csv" dsd filename=fname truncover ;

input skip1 @;

if skip1=. then delete ;

filename = fname ;

length skip2-skip19 $200 airing_date 8 HH_id $10 live_offset $10

Network $20 daypart $10

;

informat airing_date anydtdte. ;

format airing_date date9.;

input skip2 skip3 airing_date skip5 HH_id live_offset

skip8-skip15 Netowrk skip17-skip19 daypart

;

drop skip: ;

run;

%mend Import;

Re: Question about using %include in a macro to include another macro

Doxastic — Mon, 21 Jul 2014 15:12:25 GMT

Thank you all for your help, btw!

@Tom: I didn't think to do it how you are doing it. And honestly, I don't really understand it. If I can get it working this way, I'd like to keep it just because I get it.

And I don't think the problem is with the %include anymore.... I have no idea. I thought it was the %include because it's happened before. But when I got your symput code working, and macro-tized it, it failed again. Here's the code as it is now:

%macro Import_Campaigns(FilePath, ncampaigns, Data_type);

%get_filenames(&FilePath.); /* Gets file names from a directory and puts them into a SAS dataset */

%Create_Mac_Var(campy, memname); /* Tom's symput code in macro form ("campy" and "Var1" are macro variables)

/******************* This is where thing seem to be going wrong. Either in the step above or the step below.... idk */

%if &Data_type.="csv" %then %do i=1 %to &ncampaigns.;

proc import datafile="&FilePath.\&&campy&i.."

out=tempy&i

dbms=dlm replace;

delimiter=',';

getnames = no;

datarow=1;

run;

proc sql;

create table campy&i as

select a.* from tempy&i a

where var1 ne .

order by var2;

quit;

proc datasets; delete tempy&i; run; quit;

%end;

%if &Data_type.="pipe" %then %do i=&start. %to &stop.;

proc import datafile="&FilePath.\&&camp&i...txt"

out=tempy&i

dbms=dlm replace;

delimiter='|';

getnames = no;

datarow=4;

run;

proc sql;

create table campy&i as

select a.* from tempy&i a

where var1 ne .

order by var2;

quit;

proc datasets; delete tempy&i; run; quit;

%end;

%if &Data_type.="csv" %then %do;

data campaigns_rerun(keep= HH_id airing_date live_offset Network daypart); set

%read2

;

rename var6 = HH_id;

rename var4 = airing_date;

rename var7 = live_offset;

rename var16 = Network;

rename var20 = daypart;

run;

%end;

%if &Data_type.="pipe" %then %do;

data campaigns_rerun(keep= HH_id airing_date live_offset Network daypart); set

%read2

;

rename var1 = HH_id;

rename var2 = airing_date;

rename var3 = live_offset;

rename var4 = Network;

rename var5 = daypart;

run;

%end;

%mend Import_Campaigns;

And here's the log for that entire length of code.....

23588 %Import_Campaigns(E:\Shared\Campaigns\temp\csv, 5, csv);

SYMBOLGEN: Macro variable FILEPATH resolves to E:\Shared\Campaigns\temp\csv

SYMBOLGEN: Macro variable LOCATION resolves to E:\Shared\Campaigns\temp\csv

MPRINT(GET_FILENAMES): filename _dir_ "E:\Shared\Progressive\Campaigns\temp\csv";

MPRINT(GET_FILENAMES): data campaign_file_names(keep=memname);

MPRINT(GET_FILENAMES): handle=dopen( '_dir_' );

MPRINT(GET_FILENAMES): if handle > 0 then do;

MPRINT(GET_FILENAMES): count=dnum(handle);

MPRINT(GET_FILENAMES): do i=1 to count;

MPRINT(GET_FILENAMES): memname=dread(handle,i);

MPRINT(GET_FILENAMES): output campaign_file_names;

MPRINT(GET_FILENAMES): end;

MPRINT(GET_FILENAMES): rc=dclose(handle);

MPRINT(GET_FILENAMES): run;

NOTE: The data set WORK.CAMPAIGN_FILE_NAMES has 5 observations and 1 variables.

NOTE: Compressing data set WORK.CAMPAIGN_FILE_NAMES increased size by 100.00 percent.

Compressed is 2 pages; un-compressed would require 1 pages.

NOTE: DATA statement used (Total process time):

real time 0.00 seconds

cpu time 0.00 seconds

MPRINT(GET_FILENAMES): filename _dir_ clear;

NOTE: Fileref _DIR_ has been deassigned.

MPRINT(IMPORT_CAMPAIGNS): ;

MPRINT(IMPORT_CAMPAIGNS): data _null_ ;

MPRINT(IMPORT_CAMPAIGNS): set campaign_file_names;

MPRINT(IMPORT_CAMPAIGNS): call symputx(cats('campy',_n_),memname);

MPRINT(IMPORT_CAMPAIGNS): run;

NOTE: There were 5 observations read from the data set WORK.CAMPAIGN_FILE_NAMES.

NOTE: DATA statement used (Total process time):

real time 0.00 seconds

cpu time 0.00 seconds

SYMBOLGEN: Macro variable DATA_TYPE resolves to csv

Re: Question about using %include in a macro to include another macro

Doxastic — Mon, 21 Jul 2014 15:18:24 GMT

Also, in the end this is what I'd like the macro to look like:

%macro Import_Campaigns(FilePath, ncampaigns, Data_type);

%get_filenames(&FilePath.);

%Create_Mac_Var(campy, memname);

%if &Data_type.="csv" %then %do; %readcampaigns(1,&ncampaigns.,csv); %end; /* readcampaigns would be the last bit of the main macro, itself macro-tized. */

%if &Data_type.="pipe" %then %do; %readcampaigns(1,&ncampaigns.,pipe); %end;

%mend Import_Campaigns;

See how much easier this is to read and understand? It also makes it so I can use all 3 of those macros in other places really easily.

This is why I use macros within macros.

Re: Question about using %include in a macro to include another macro

RW9 — Mon, 21 Jul 2014 15:24:39 GMT

I would personally put your if statements in quotes and trim them e.g.:

%if %trim(&data_type)="csv". Whats happening is the if statement is not true and hence none of the other code operates.

As Tom previously mentioned however, you don't need to do a lot of that code. If you look at a datastep with an infile statement you can have multiple delimiters (there are examples on here of that). The other code as well can also be trimmed. Am leaving for the day, however if its still open I add some code.

Re: Question about using %include in a macro to include another macro

Reeza — Mon, 21 Jul 2014 15:30:18 GMT

There's nothing wrong with this structure, but you need to make sure each component/module works on its own first and then the whole.

My other issue is you assume what the output files are named, which isn't a convention I like to use as it makes it harder to reuse components across different programs.

Also, if you are trying to read all files from a folder, SAS has much more efficient ways of doing it than this as Tom has pointed out. Its really worth learning how to do it as your code will be cleaner and much easier to debug in the future.

Re: Question about using %include in a macro to include another macro

Doxastic — Mon, 21 Jul 2014 15:44:36 GMT

This sounds interesting but I definitely don't understand how you'd do that

Re: Question about using %include in a macro to include another macro

jakarman — Mon, 21 Jul 2014 15:45:42 GMT

The path to hell is paved with good intentions. @doxastic the intention a good one as you have shown the wanted goal to achieve.
The coding habits to achieve that is not that good. Macro-codes should be easy to read pieces and all easy to debug ones.

You can define a bunch of subset macros while using just one master macro. The trick is to define all those in the only source code named as "master" like:

source member (autocall or include) master:

/* comment describing your macro */

%macro master ..... ;

...

%mend;

/* comment used dedicated subroutines */

%macro subblock1 ..... ;

...

%mend;

%macro subblock2 ..... ;

...

%mend;

And several will complain that doing it this way they cannot find all interactions of macro-s and code.

Working on this you can easily check and debug modifications.
One missed quoted ; or other nesting language element will cause your described troublesome behavior.

The macro language processor does the preprocessing somewhat different as the datastep/proc language. In rare situations having unexpected behavior.

Remember the macro references do not need the semicolon as that has a meaning at the other level. Macro quoting is an other area of language conflict solving.
The scope of macrovariables can be a pitfall, this is an automatic process with some conventions.

The generic advice don not use defining macro-s within macro-s (some exceptions exist) however using macro-s within macro-s is allowed practice.

Re: Question about using %include in a macro to include another macro

Tom — Mon, 21 Jul 2014 15:49:43 GMT

So the issue is that the data in the list of files that you are importing contains characters that you would have to quote using macro quoting functions like %QUOTE, %BQUOTE etc if you typed them into you program file. So when you used them to generate %LET statements they caused trouble. Now that you are using them to generate other SAS code you are seeing the problems in that code.

In particular you are using

datafile="&FilePath.\&&campy&i.."

and

create table campy&i as

So if the name of the source file contains some strange characters you might have trouble with the DATAFILE= or if more likely with using the name as the name of SAS table.

SAS tablenames can by up to 32 characters, digits and underscores and must not start with a digit.

Note that your loops for CSV and PIPE are not consistent with each. Since the only difference between the two is the value used for the DELIMITER in the PROC IMPORT you can simplify your logic.

%do i=1 %to &ncampaigns.;

proc import datafile="&FilePath.\&&campy&i.."

out=tempy&i

dbms=dlm replace;

getnames = no;

datarow=1;

%if &Data_type ="csv" %then %do;

delimiter=',';

%end;

%if &Data_type ="pipe" %then %do;

delimiter='|';

%end;

run;

proc sql;

create table campy&i as

select a.* from tempy&i a

where var1 ne .

order by var2

;

drop table tempy&i ;

quit;

%end;

Re: Question about using %include in a macro to include another macro

Reeza — Mon, 21 Jul 2014 15:50:17 GMT

You use a wild card in your infile statement.

For example

infile 'C:\localdata\*.txt';

points to all files in the folder C:\localdata that end with the extension .txt.

If you want the file name as well, the filename option can be added on to provide the filename.

If you're file has headers in each row, you'll want to ignore those.

Try running Tom's code and see if it works. It's the correct solution to your problem.