topic Re: Web scraping PROC HTTP: NOTE: 404 Not Found in SAS Programming

Web scraping PROC HTTP: NOTE: 404 Not Found

dcortell — Thu, 08 Jun 2023 11:43:07 GMT

Hi experts. I'm using the following example:

filename src temp;
proc http
url="https://www.sas.com/en/whitepapers/artificial-intelligence-banking-risk-management-110277.br.html#formsuccess?utm_source=linkedin&utm_medium=paid-social&utm_campaign=rsk-gen-emea&utm_content=50931-lklgf-english"
out=src;
run;

The scraping produce the following note:

73 filename src temp;
74 proc http
75 method="get"
76 url="https://www.sas.com/en/whitepapers/artificial-intelligence-banking-risk-management-110277.br.html#formsuccess?utm_so
76 ! urce=linkedin&utm_medium=paid-social&utm_campaign=rsk-gen-emea&utm_content=50931-lklgf-english"
WARNING: Apparent symbolic reference UTM_MEDIUM not resolved.
WARNING: Apparent symbolic reference UTM_CAMPAIGN not resolved.
WARNING: Apparent symbolic reference UTM_CONTENT not resolved.
77 out=src;
78 run;

NOTE: 404 Not Found
NOTE: PROCEDURE HTTP ha utilizado (Tiempo de proceso total):
real time 0.11 seconds
cpu time

However, the URL is reachable and it has an HTML structure that should be possible to scrape

Any idea about why the scraping is failing here?

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

LinusH — Thu, 08 Jun 2023 12:05:56 GMT

Try to use single qoutes for the url option, unless you are actually trying to use SAS macro variables

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

Tom — Thu, 08 Jun 2023 12:44:53 GMT

I suspect that the webserver that SAS is using to host that page is not playing well with PROC HTTP.

Since it is a SAS site why not open a SAS Support ticket and let them debug their own site.

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

ChrisHemedinger — Thu, 08 Jun 2023 13:15:22 GMT

I suspect your environment is working through a proxy server and you might need to specify PROXY= options.

filename resp temp; proc http url="https://www.sas.com/en/whitepapers/artificial-intelligence-banking-risk-management-110277.br.html" out=resp; run; data _null_; infile resp; input ; put _infile_ ; run;

But what you trying to do here? This page is a gate to a whitepaper, but it won't get you the paper itself -- that's a download process with a PDF.

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

LinusH — Thu, 08 Jun 2023 13:20:43 GMT

If works for me from SAS Viya Learing Ed:

82   filename src temp;
83   proc http
84   url='https://www.sas.com/en/whitepapers/artificial-intelligence-banking-risk-management-110277.br.html#formsuccess?utm_source=l
84 ! inkedin&utm_medium=paid-social&utm_campaign=rsk-gen-emea&utm_content=50931-lklgf-english'
85   out=src;
86   run;
NOTE: PROCEDURE HTTP used (Total process time):
      real time           0.42 seconds
      cpu time            0.03 seconds

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

dcortell — Thu, 08 Jun 2023 13:32:40 GMT

Hi Chris, testing proc http under different conditions

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

ChrisHemedinger — Thu, 08 Jun 2023 13:35:20 GMT

Here are some tips for checking that PROC HTTP is usable with your internet access in your SAS session.

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

dcortell — Thu, 08 Jun 2023 13:50:29 GMT

Running the test code getting the following error:

92 /* Tell SAS to parse the JSON response */

93 libname stream JSON fileref=resp;

NOTE: JSON data is only read once. To read the JSON again, reassign the JSON LIBNAME.

ERROR: JSON no válido en input cerca de la línea 1 columna 1: Encountered an illegal character.

ERROR: Error in the LIBNAME statement.

95 title "JSON library structure";

96 proc datasets lib=stream;

ERROR: Libref STREAM is not assigned.

97 quit;

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

dcortell — Thu, 08 Jun 2023 13:53:45 GMT

EDIT: Runned for a second iteration, the code works fine and no error is generated

Re: Web scraping PROC HTTP: NOTE: 404 Not Found

dcortell — Thu, 08 Jun 2023 15:19:18 GMT

In addition to Chris test, If I cut the url at the ".html" part and provide it, the scrape works fine:

%let url='https://www.sas.com/en/whitepapers/artificial-intelligence-banking-risk-management-110277.br.html';

filename src temp;
proc http
url=&url
out=src;
run;

data _null_;
infile src;
input;
list;
run;

I believe then it could make sense just to make similar URLs being trimmed at the ".HTML" part, if that allow the scraping, but still not sure why for some folks the full URL scrape works fine while on mine no