Understanding macro compiling

[JacquesR]

In this thread:

https://communities.sas.com/t5/SAS-Programming/Macro-quoting-for-a-url/m-p/643575

I was asking about constructing a url to be fed to a macro that uses PROC HTTP to get a response from an API, and then uses several data steps to process that response. Only, the total process consisted of over 16 thousand API calls (i.e., the macro was called 16,000+ times).

One thing I am failing to understand is this.

I had hoped, in the interest of clean, error free code, to test the result of the API call, and if it was successful, then to proceed to the data steps:

        DATA _NULL_;
            SET VOLUMES.ROOT;
            CALL SYMPUT('GetOutcome',status);
        RUN;
        %IF &GetOutcome EQ success %THEN %DO;

However, what I find is that &GetOutcome is not being reprocessed each time the macro is being called:

DATA _NULL_;
    SET MonitorsAndDates;
    URL=CATS("'https://api.....com/api/monitor?auth=",SYMGET('Auth'),%NRSTR('&id='),Monitor,%NRSTR('&start='),PUT(StartDate,E8601DA10.),%NRSTR('&end='),PUT(EndDate,E8601DA10.),"'");
    CALL EXECUTE('%GetVolume(MonitorLabel='||MonitorLabel||',N='||_N_||',URL='||URL||');');
RUN;

Instead, the very first time I run the code, I get the error that GetOutcome has not yet been assigned. Thereafter, it is 'success' (sans quotes) for each and every call, regardless of what the result actually was.

So this help file page:

https://support.sas.com/documentation/cdl/en/mcrolref/61885/HTML/default/viewer.htm#a000210266.htm

indicates that CALL SYMPUT "resolves variables at program execution instead of macro execution" so that is to be expected (I get that now).

For the same reason, the url (which I was previously constructing in the macro, and not in the Data _Null_ step for CALL EXECUTE) was also not changing from iteration to iteration.

The same help file states that:

"SYMPUT assigns values produced in a DATA step to macro variables during program execution, but the SYMGET function returns values of macro variables to the program during program execution."

My question, though, is how do I test the result of my API call, and then run all the following steps, if I cannot use CALL SYMPUT?

Perhaps I need someone to explain the intricacies of how SAS macros are compiled and what changes when the are called from CALL EXECUTE?

Thanks

[RichardDeVen]

Not seeing the macro source code, so I would recommend stacking all the CALL EXECUTES  by wrapping the macro invocation in %NRSTR

DATA _NULL_;
    SET MonitorsAndDates;
    URL=CATS("'https://api.....com/api/monitor?auth=",SYMGET('Auth'),%NRSTR('&id='),Monitor,%NRSTR('&start='),PUT(StartDate,E8601DA10.),%NRSTR('&end='),PUT(EndDate,E8601DA10.),"'");

length statement $2000;
statement = '%nrstr('
|| '%GetVolume(MonitorLabel='||MonitorLabel||',N='||_N_||',URL='||URL||');'
|| ');';

    CALL EXECUTE(statement);
RUN;

[Kurt_Bremser]

When you push code that contains macro triggers into the execution queue by using call execute, the macro triggers are resolved immediately, but all data or procedure step code has to wait until the data step calling execute has finished.

To prevent this, you use the %nrstr() around your macro call. The macro triggers are not resolved, but put into the execution queue without the masking, and are resolved in order once the calling data step has finished.

[JacquesR]

@Kurt_Bremser , @RichardDeVen 

Thanks for the comments.

I think I need to post more of the code, I thought what I had shown would be enough, but I was clearly mistaken.

%MACRO GetVolume(MonitorLabel=,N=,URL=);
       /* Use the API call in &URL to get a JSON result */
        FILENAME Vol TEMP;
        PROC HTTP 
            URL=&URL
            METHOD="GET"
            OUT=Vol;
        RUN;
        LIBNAME VOLUMES JSON FILEREF=Vol;
      /* Test whether the returned result was successful */
        DATA _NULL_;
            SET VOLUMES.ROOT;
            CALL SYMPUT('GetOutcome',status);
        RUN;
     /* If successful, then take out the datasets parsed from the JSON result and append them to the existing larger datasets */
        %IF &GetOutcome EQ success %THEN %DO;
            /* Store the JSON result so that it need not be downloaded again, if the datasets are to be rebuilt */
            data STUDYDAT.API_result&N;
            infile Vol length=len lrecl=32767;
            input line $varying32767. len;
             line = strip(line);
             if len>0;
            run;
            PROC EXPORT
                DATA=STUDYDAT.API_result&N
                OUTFILE="&OutputPath.API_result&N"
                DBMS=CSV
                REPLACE;
            RUN;
            /* Now get each individual dataset from the result */
            DATA WORK.ROOT; SET VOLUMES.ROOT; RUN;
            DATA WORK.ALLDATA; SET VOLUMES.ALLDATA; RUN;
            DATA WORK.DATA; SET VOLUMES.DATA; RUN;
            DATA WORK.DATA_INFO; SET VOLUMES.DATA_INFO; RUN;
            /* Append these datasets to the existing 'compilation' datasets */
            PROC DATASETS NOPRINT LIBRARY=STUDYDAT;
                APPEND BASE=AllData DATA=WORK.ALLDATA FORCE;
                APPEND BASE=Root DATA=WORK.ROOT FORCE;
                APPEND BASE=Data DATA=WORK.DATA FORCE;
                APPEND BASE=Data_info DATA=WORK.DATA_INFO FORCE;
            QUIT;                                        
        %END;

    %MEND GetVolume;
DATA _NULL_;
    SET MonitorsAndDates;
    URL=CATS("'https://api.....com/api/.../countries?auth=",SYMGET('Auth'),%NRSTR('&id='),Monitor,%NRSTR('&start='),PUT(StartDate,E8601DA10.),%NRSTR('&end='),PUT(EndDate,E8601DA10.),"'");
    CALL EXECUTE('%GetVolume(MonitorLabel='||MonitorLabel||',N='||_N_||',URL='||URL||');');
RUN;

In short, this has nothing to do with the CALL execute. That is only because I am doing so many API calls.

But for each API call (which is done in the beginning of the macro), I need to analyse the result, and if it was successful, then get the result into datasets.

Does that clarify?

[Kurt_Bremser]

You have a macro statement in your macro:

%IF &GetOutcome EQ success %THEN %DO;

When submitting a macro in call execute without masking it, this macro statement will be executed immediately, long before the code that sets GetOutCome is run.

Use %nrstr to mask the whole macro call:

CALL EXECUTE('%nrstr(%GetVolume(MonitorLabel='||MonitorLabel||',N='||_N_||',URL='||URL||'));');

[Tom]

Your original post had enough information to show that your macro will have trouble when run via CALL EXECUTE().  Because you are setting a macro variable via execution of SAS statements (in your case the call syputx() function) and then using the value of that macro variable to drive logic or code generation you will have a timing problem.  The use of the %NRSTR() around the macro invocation will cause CALL EXECUTE to delay actually running the macro until it pulls the macro call back off the stack of code it generated.

To see the difference just look at the lines in the SAS log with the + at the beginning that CALL EXECUTE generated.  Your current CALL EXECUTE() will generate lines like:

+ FILENAME Vol TEMP;
+ PROC HTTP ...

If you change the call to delay the execution of the macro like this:

CALL EXECUTE('%nrstr(%GetVolume)(MonitorLabel='||MonitorLabel||',N='||_N_||',URL='||URL||');');

then the lines in the log will look like:

+ %GetVolume(MonitorLabel= ...

[s_lassen]

Rather than messing round with CALL EXECUTE, I would recommend writing to a temporary file and %INCLUDE that: 

filename tempsas temp:
data _null_;
  if _N_=1 then do; 
    set volumes.root;
    if status ne 'success' then stop;
  end;
  SET MonitorsAndDates;
  URL=CATS("'https://api.....com/api/monitor?auth=",SYMGET('Auth'),%NRSTR('&id='),Monitor,%NRSTR('&start='),PUT(StartDate,E8601DA10.),%NRSTR('&end='),PUT(EndDate,E8601DA10.),"'");
  file tempsas;
  put   '%GetVolume(MonitorLabel=' MonitorLabel ',N=' _N_ ',URL=' URL ');';
run;

Then take a look at the TEMPSAS temporary file, see if the macro calls look right. If they do, submit the first call and see if the result looks right. If it does, add this line to your code:

%include tempsas/source2;

I also took the CALL SYMPUT and macro stuff out, you can just do the whole thing in a single data step as shown.