Odd behavior with attempted PROC HTTP loop when 502 Bad Gateway message occurs

[texasmfp]

I have a series of simple PROC HTTP statements.  I've noticed that, if one of the calls gets a Note:  502 Bad Gateway Note (rather than the Note: 200 OK) that the program runs to completion and I get missing data.  So I devised a loop using this macro (https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/proc/n1ietrp9bhfq65n1px6nxl6vp9df.htm#n1ietrp...) but repeated the http statement instead of aborting.

%macro prochttp_check_return(code);
%if %symexist(SYS_PROCHTTP_STATUS_CODE) ne 1 %then %do;
  %put ERROR: Expected &code., but a response was not received from the HTTP Procedure;
  %get_CPI;%end;
  
%else %do;
  %if &SYS_PROCHTTP_STATUS_CODE. ne &code. %then %do;
   %put ERROR: Expected &code., but received &SYS_PROCHTTP_STATUS_CODE.
 &SYS_PROCHTTP_STATUS_PHRASE.;
   %get_CPI;%end;
%end;
%mend;

%macro get_CPI;
filename out5 "e:\sas data\out5.xml";
proc http
url='http://dataservices.imf.org/REST/SDMX_xml.svc/CompactData/IFS/M.IN.PCPI_IX.?startPeriod=2019&endPeriod=2024'
method="get" out=out5;
run;

%prochttp_check_return(200);

%mend get_CPI;

%get_CPI;

But, when I forced an error (by adding an extra character to the url - adding an extra dot "xml..scv"), it loops exactly 10 times, then gives a 200 OK note.  But a check of the file shows that it is  not OK, the call failed.

5539  %macro prochttp_check_return(code);
5540  %if %symexist(SYS_PROCHTTP_STATUS_CODE) ne 1 %then %do;
5541    %put ERROR: Expected &code., but a response was not received from the HTTP Procedure;
5542    %get_CPI;%end;
5543
5544  %else %do;
5545    %if &SYS_PROCHTTP_STATUS_CODE. ne &code. %then %do;
5546     %put ERROR: Expected &code., but received &SYS_PROCHTTP_STATUS_CODE.
5547   &SYS_PROCHTTP_STATUS_PHRASE.;
5548     %get_CPI;%end;
5549  %end;
5550  %mend;
5551
5552  %macro get_CPI;
5553  filename out5 "e:\sas data\out5.xml";
5554  proc http
5555  url='http://dataservices.imf.org/REST/SDMX_xml..svc/CompactData/IFS/M.IN.PCPI_IX.?startPeriod=2019&endPeriod=2024'
5556  method="get" out=out5;
5557  run;
5558
5559
5560  %prochttp_check_return(200);
5561  %mend get_CPI;
5562  %get_CPI;

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.26 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.26 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.25 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.25 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.27 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.27 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.26 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.25 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.27 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.25 seconds
      cpu time            0.00 seconds

NOTE: 404 Not Found

ERROR: Expected 200, but received 404  Not Found

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.30 seconds
      cpu time            0.03 seconds

NOTE: 200 OK

When I did get a 502 Bad Gateway note, there was no 10x repeat like above.  Instead it went to a 200 OK note, and yet again, the call was bad in spite of the OK note. 

2152  %macro prochttp_check_return(code);
2153  %if %symexist(SYS_PROCHTTP_STATUS_CODE) ne 1 %then %do;
2154    %put ERROR: Expected &code., but a response was not received from the HTTP Procedure;
2155    %get_CPI;%end;
2156
2157  %else %do;
2158    %if &SYS_PROCHTTP_STATUS_CODE. ne &code. %then %do;
2159     %put ERROR: Expected &code., but received &SYS_PROCHTTP_STATUS_CODE.
2160   &SYS_PROCHTTP_STATUS_PHRASE.;
2161     %get_CPI;%end;
2162  %end;
2163  %mend;
2164
2165  %macro get_CPI;
2166  filename out5 "e:\sas data\out5.xml";
2167  proc http
2168  url='http://dataservices.imf.org/REST/SDMX_xml.svc/CompactData/IFS/M.IN.PCPI_IX.?startPeriod=2019&endPeriod=2024'
2169  method="get" out=out5;
2170  run;
2171  %mend get_CPI;
2172
2173  %prochttp_check_return(200);
ERROR: Expected 200, but received 502  Bad Gateway

NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.57 seconds
      cpu time            0.00 seconds

NOTE: 200 OK

Is my code wrong or am I not understanding something about how the Return Codes work?  Thanks

[Quentin · Accepted solution]

Your code looks fine.  I probably wouldn't code it that way (with recursion), you could do it with just one macro and a loop, but I don't think that's the problem.

I could replicate the problem without a macro, just by submitting the broken URL (extra dots) ten times fast:

filename out5 "Q:\junk\out5.xml";
proc http
url='http://dataservices.imf.org/REST/SDMX_xml..svc/CompactData/IFS/M.IN.PCPI_IX.?startPeriod=2019&endPeriod=2024'
method="get" out=out5;
run;

It looks like the gateway gets mad and maybe thinks there is a DOS attack.  It returns a file, so you get the 200 ok response, but if you look in the file, it's not the data, it's a message saying you've exceeded their request limit:

				<h1>DataServices - Rate limit exceeded</h1>
				<br>
				Your requests have exceeded the rate limit. Please consider lowering your request rate and try again later.
				<br>
				More information on the request rate limit can be found <a href="https://datahelp.imf.org/knowledgebase/articles/630877-data-services">here</a>.

I guess if you need multiple requests, you might add the sleep() function to sleep a few seconds before repeating the request, and see if that helps.

In below macro I sleep for 1 second after each failed try.  It tries 20 times and fails every time, without exceeding their rate limit.

%macro get_cpi();
  filename out5 "Q:\junk\out5.xml";

  %local i SYS_PROCHTTP_STATUS_CODE;

  %do i=1 %to 20 ;
    proc http
    url='http://dataservices.imf.org/REST/SDMX_xml..svc/CompactData/IFS/M.IN.PCPI_IX.?startPeriod=2019&endPeriod=2024'
    method="get" out=out5;
    run;

    %put &=i &=SYS_PROCHTTP_STATUS_CODE &=SYS_PROCHTTP_STATUS_PHRASE ;

    %if &SYS_PROCHTTP_STATUS_CODE=200 %then %return ;

    data _null_ ;
      rc=sleep(1) ;
    run ;
  %end ;
  %put ERROR: tried 20 times and could not get data!  ;
%mend ;


%get_cpi()

Just realized that approach probably won't help with your second example, where you got a bad gateway response.  I would look at the returned file in that example.  It looks to me like if the server returns an xml file, PROC HTTP was successful so returns 200.  PROC HTTP doesn't know if the content of the file is an error message or the data you want.  You might actually have to parse the XML file to see if it is data, or an error message, and then use that to decide whether or not to try again.

[texasmfp]

Thanks. It took a while to replicate the Bade Gateway Note, but that macro works. Mine also works. The difference is that mine goes on forever, while yours exits after 20x. Time will tell which is best suited. Thanks

[Patrick]

To add to what @Quentin wrote: Just to be on the safe side you could also consider to add the CLEAR_CACHE argument to your Proc HTTP request.

I've used in production code very similar syntax to yours with a call sleep() of a second between retries and this worked and hasn't caused any issues in the last two years.

[texasmfp]

Thanks, I have added that