topic Re: REST API PROC HTTP failing for dynamic IP address in SAS Programming

REST API PROC HTTP failing for dynamic IP address

Jim_Cooper_hmsa — Sat, 27 Jun 2020 02:24:56 GMT

I am trying to automate the download and analysis of The Covid Tracking Project data that is made avalable on the Internet. I was able to get the IP address white listed by our network security team and the program worked once. What I discovered over the lat couple of weeks is that the Web site used multiple IP addresses and sometimes the code works, but most times it does not because it connects through a different IP address.

Is there a SAS workaround or has someone run into this problem before? I don't think that I can ask for a white list to dynamic list of IP addresses since I don;t know the complete list of addresses that might be used.

Program Log.

81 * Test Internet connection for downloading;

82 %macro prochttp_check_return(code);

83 %if %symexist(SYS_PROCHTTP_STATUS_CODE) ne 1 %then %do;

84 %put ERROR: Expected &code., but a response was not received from

85 the HTTP Procedure;

86 %abort;

87 %end;

88 %else %do;

89 %if &SYS_PROCHTTP_STATUS_CODE. ne &code. %then %do;

90 %put ERROR: Expected &code., but received &SYS_PROCHTTP_STATUS_CODE.

91 &SYS_PROCHTTP_STATUS_PHRASE.;

92 %abort;%end;

93 %end;

94 %mend;

96 proc http url="https://covidtracking.com/api/v1/states/daily.csv";

97 run;

ERROR: Error connecting to 104.248.78.24:443. (The connection has timed out.)

ERROR: Unable to establish connection to covidtracking.com.

ERROR: Unable to connect to Web server.

Re: REST API PROC HTTP failing for dynamic IP address

s_lassen — Sat, 27 Jun 2020 06:41:43 GMT

It seems a bit strange to me that it is the IP address that gets white-listed, can't you ask them to white-list the URL?

Another way to access the file could just be to use a filename, e.g.:

filename covid url 'https://covidtracking.com/api/v1/states/daily.csv';

proc import datafile=covid out=test dbms=csv replace;
run;

I am not sure whether this will be any different, though.

Re: REST API PROC HTTP failing for dynamic IP address

Jim_Cooper_hmsa — Mon, 29 Jun 2020 16:33:55 GMT

Thank you for your alternate solution of using the filename URL as a better alternative. I am not sure why our security team white lists IP addresses and not URLs. Because we are a hospital system, we tend to be more conservative in our approach to Internet access. Interestingly, both approaches worked when I tested your solution this morning. When I refreshed the table to demonstrate the process is repeatable, both approaches failed to connect to the source. I am going to operate under the assumption that this is not a SAS solvable problem and that I need to work with our security team and the website administrator to understand and resolve the issue.