Apparent symbolic reference IN_BESTAND not resolved. (Base SAS 9.4)

[Longimanus]

Hi there! I'm having a problem and it drives me nuts. 😉  I hope somebody can help me understanding this. 

What is it I like to do: 

What I show you is a part of a larger program. I just pulled out the statements that bug me. The "real" program does (or should do) the following: 

1. It should place all the filenames (with path) in a given forlder in a table called x_dirlist.

     (eg: /opt/dir1/dir2/filename_yymmdd.skv.gpg)

2. My program should then for each file that qualifies run a macro that will: 

    a. Input var for the marco is INPUT_BESTAND

    b. Decrypt the file (file name comes in with a .skv.gpg extention)  

    c. Removed the .gpg extention from the filename (in INPUT_BESTAND) end place the result in a new macro variable IN_BESTAND. The result should be:  /opt/dir1/dir2/filename_yymmdd.skv

    d. Konvert the file (som values need to be anonymized) that now should reside in IN_BESTAND (ie. with ony the .skv extention) 

    e. Write it back to the same folder. 

Below two examples I played with. One works, the other don't. In the example that doesn't work I removed a lot of statements (the decrypt, convertion etc) that are not relevant for the problem I try to understand. 

This works: (the put statements are just inserted so I can see what's in the variables): 

%let dir_path = opt/app/sasdata;

%macro test_it(macro_input);

                   %let input_bestand = &dir_path./&macro_input.; 

                   %put &input_bestand.;

                             data _null_;

                                call symput('in_bestand',(tranwrd("&input_bestand.",'.gpg','    '))); 

                             run;

                     %put &in_bestand.;

%mend;

%test_it(menno.skv.gpg);

%test_it(maximilian.skv.gpg);

LOG:

53   %let dir_path = opt/app/sasdata;

54   %macro test_it(macro_input);

55          %let input_bestand = &dir_path./&macro_input.;

56          %put &input_bestand.;

57       data _null_;

58       call symput('in_bestand',(tranwrd("&input_bestand.",'.gpg','    ')));

59       run;

60       %put &in_bestand.;

61   %mend;

62   %test_it(menno.skv.gpg);

opt/app/sasdata/menno.skv.gpg

NOTE: DATA statement used (Total process time):

      real time           0.00 seconds

      cpu time            0.00 seconds

opt/app/sasdata/menno.skv

63   %test_it(maximilian.skv.gpg);

opt/app/sasdata/maximilian.skv.gpg

NOTE: DATA statement used (Total process time):

      real time           0.00 seconds

      cpu time            0.00 seconds

opt/app/sasdata/maximilian.skv

The following example code does not work for macro variable IN_BESTAND :

%macro get_dirlist;

filename dirlist pipe "ls -p &dir_path. | grep -v /";

data x_dirlist;

                             length name $100;

                             infile dirlist;

                             input name $;

                             run;

%mend;

%macro convert_file(macro_input);

                             %let input_bestand = &dir_path./&macro_input.; 

                             %put &input_bestand.;

                             data _null_;

                                  call symput('in_bestand',(tranwrd("&input_bestand.",'.gpg','    '))); 

                              run;

                             %put &in_bestand.;

%mend convert_file;

%let incountry = fi;

%let yy = 2015;

%let dir_path = /opt/app/sas94/config/Lev3/SASApp/Data/coredata/delivery/&incountry./freja/STAGED/&yy.;

%get_dirlist;

Data _null_;

 Set x_dirlist;

    if upcase(substr(name,1,5)) = 'FI03H' then do; 

         Execmacr = '%convert_file('||strip(name)||')';      

         Call execute(execmacr);

    end;

Run;

So instead of calling the macro manually everytime with a new file name ....:

%convert_file(/opt/dir1/dir2/filename_170131.skv); 

%convert_file(/opt/dir1/dir2/filename_170228.skv); 

etc.

... I go through a table where the file names reside and call the macro for each file that qualifies. 

LOG:

MLOGIC(CONVERT_FILE):  Beginning execution.

MLOGIC(CONVERT_FILE):  Parameter MACRO_INPUT has value FI03HO_02_2015-09-16_2015-09-30_l305_s1_012.skv.gpg

MLOGIC(CONVERT_FILE):  %LET (variable name is INPUT_BESTAND)

SYMBOLGEN:  Macro variable DIR_PATH resolves to

            /opt/app/sas94/config/Lev3/SASApp/Data/coredata/delivery/&incountry./freja/STAGED/&yy.

SYMBOLGEN:  Macro variable INCOUNTRY resolves to fi

SYMBOLGEN:  Macro variable YY resolves to 2015

SYMBOLGEN:  Macro variable MACRO_INPUT resolves to FI03HO_02_2015-09-16_2015-09-30_l305_s1_012.skv.gpg

MLOGIC(CONVERT_FILE):  %PUT &input_bestand.

SYMBOLGEN:  Macro variable INPUT_BESTAND resolves to

            /opt/app/sas94/config/Lev3/SASApp/Data/coredata/delivery/fi/freja/STAGED/2015/FI03HO_02_2015-09-16_2015-09-30_l305_s1_01

            2.skv.gpg

/opt/app/sas94/config/Lev3/SASApp/Data/coredata/delivery/fi/freja/STAGED/2015/FI03HO_02_2015-09-16_2015-09-30_l305_s1_012.skv.gpg

MPRINT(CONVERT_FILE):   data _null_;

SYMBOLGEN:  Macro variable INPUT_BESTAND resolves to

            /opt/app/sas94/config/Lev3/SASApp/Data/coredata/delivery/fi/freja/STAGED/2015/FI03HO_02_2015-09-16_2015-09-30_l305_s1_01

            2.skv.gpg

MPRINT(CONVERT_FILE):   call

symput('in_bestand',(tranwrd("/opt/app/sas94/config/Lev3/SASApp/Data/coredata/delivery/fi/freja/STAGED/2015/FI03HO_02_2015-09-16_201

5-09-30_l305_s1_012.skv.gpg",'.gpg','    ')));

MPRINT(CONVERT_FILE):   run;

MLOGIC(CONVERT_FILE):  %PUT &in_bestand.

WARNING: Apparent symbolic reference IN_BESTAND not resolved.

&in_bestand.

MLOGIC(CONVERT_FILE):  Ending execution.

 I see the same construction here ... but still it doesn't work. I just call the same macro a number of time - once for each file. 

Where is my "brain-fart"? 😉 

Best regards, 

Longimanus 😄 

[Kurt_Bremser]

What you are facing is a peculiarity of the interaction of call execute with the macro processor.

When you call execute a macro that has macro triggers in it, those can immediately be resolved; base SAS code created by the macro, OTOH, needs to wait until the data step that uses call execute has finished. So the call symput becomes active long after the macro itself created the code for the data step, and the macro can't see the result of the call symput, as the data step has not yet run. You will see that in the log, the "apparent symbolic reference" messages appear long before the NOTEs from the data steps.

This can be a place where using %sysfunc(tranwrd( instead of the internal data _null_ step might be the solution. But be aware that other operations involving base SAS code within the macro might still need to be reworked.

[Longimanus]

Thank you for your explanation and time! I will try to read it over and let it sink in. 🙂 Best regards! Menno 🙂

[BrunoMueller]

As @Kurt_Bremser metioned there is a timing issue.

I recommend this approach, instaed of using CALL EXECUTE, write your macro calls to a file and then after the step generating the calls use the %INCLUDE to run the code, see also code sample below.

%macro someTest(nobs);
  %put NOTE: running &sysmacroname as %sysfunc(time(), time12.3) &=nobs;
  proc print data=sashelp.class(obs=&nobs);
  run;
%mend;

filename xpgm temp;
data _null_;
  do i = 1 to 2;
    macroCall = cats('%someTest(', i, ')');
    putlog "NOTE: " macroCall=;
    file xpgm;
    put macroCall;
  end;
run;

%include xpgm / source2;
filename xpgm clear;

[Longimanus]

Hey @BrunoMueller! 🙂 What is that 'source2' doing? I removed it and running the pgm gave the same result. Just curious. Thanx for this solution. I will use that. Maybe not here - since @Astounding also came with an option - but I have (am getting) an understanding of what you all wrote. Thanx a lot!  

[Kurt_Bremser]

source2 directs the inclusion (or non-inclusion) of code from secondary sources (%include) into the log. With source2, the code will.be added to the log.

Basically, this is a system option; adding /source2 to the %include statement overrides any options nosource2; that is active, for the duration of this include.

[Longimanus]

Thank you! 🙂 I got that now.

[Astounding]

CALL EXECUTE has some unusual behavior.  It runs the indicated code as soon as it can.  Since CALL EXECUTE appears in a DATA step, that means:

• Any generated SAS statements (such as data_null_; call symput...) have to wait until the currenct DATA step is over.
• But any macro statements (%let, %put) run immediately, without waiting for the current DATA step to end.

The solution would be to mask the macro call so that it appears to SAS that the CALL EXECUTEd statements must wait until the current DATA step is over.  %NRSTR is the usual tool for the job:

Execmacr = '%nrstr(%convert_file('||strip(name)||'))';   

I'm pretty sure I got the pieces in the right places, but you would have to test it to confirm.

[Longimanus]

Thank you for your explanation and time! I will try to read it over and let it sink in. 🙂 Best regards! Longimanus

[Tom]

If you are going to use CALL EXECUTE() to submit macro calls to run after your current data step finishes you should get in the habit of use %NRSTR() function to suppress the execution of the macro during the execution of the CALL EXECUTE() statement.  Personally I like to just wrap the macro name itself in the %NRSTR() and leave any parameters unquoted.

data _null_;
  set x_dirlist;
  if upcase(substr(name,1,5)) = 'FI03H';
  call execute(cats('%nrstr(%convert_file)(',name,')'));
run;

Which will result in code like this being stacked up to execute.

%nrstr(%convert_file)(FIO3H_170131.skv)
%nrstr(%convert_file)(FIO3H_170228.skv)

You can also just avoid CALL EXECUTE() and use the data step to write the code to a file and then use %INCLUDE to run. Then there is no need for the %NRSTR() macro functions. This process also has other advantages over CALL EXECUTE() because you can more easily debug the logic of the code generation. Plus use can take advantage of the features of the PUT statement to help in generating code that will make the SAS log clearer.  For example if your source dataset has variables with names that match the parameters for the macro you generating the call for you can use the VAR= syntax on the PUT statement write both the parameter name and its value.

For example if you have a macro name like this with three parameters.

%macro printit(dsn=,bylist=,varlist=);
proc print data=&dsn;
  by &bylist ;
  var &varlist;
run;
%mend printit;

And you had a dataset with the variables DSN, BYLIST and VARLIST then you could generate a series of call using code like this.

filename code temp;
data _null_;
  file code;
  set print_list;
  put '%printit(' dsn= ',' bylist= ',' varlist= ')' ;
run;
%include code / source2 ;