topic Re: read url fail in SAS Programming

read url fail

GeorgeSAS — Sat, 06 May 2017 05:33:47 GMT

.I have problem to read the url,please help.

Thanks!

FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
DATA SOURCE2;
FORMAT WEBPAGE $1000.;
INFILE SOURCE LRECL=32767 DELIMITER=">";
INPUT WEBPAGE $ @@;
RUN;

Re: read url fail

Shmuel — Sat, 06 May 2017 05:59:54 GMT

What problem do you have ?

Please post your log.

Re: read url fail

ArtC — Sat, 06 May 2017 06:30:16 GMT

Why are you using the %STR function?

Re: read url fail

art297 — Sat, 06 May 2017 18:04:08 GMT

Your code reads the url just fine, but it doesn't do anything to parse it. e.g., if you wanted the dates and titles of the presentations, the following works for all but one of the presentations. The one it misses contains a non-ASCII character:

FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
DATA SOURCE2 (drop=junk);
  informat junk $80.;
  informat published $8.;
  informat title $100.;
  INFILE SOURCE LRECL=32767 DELIMITER=">";
  INPUT @"<spam class='meta'"
  published &/////junk title &;
  title=tranwrd(scan(title,1,'<'),'&#39;',"'");
RUN;

Art, CEO, AnalystFinder.com

Re: read url fail

GeorgeSAS — Sat, 06 May 2017 18:52:56 GMT

Here is the SAS log error notes:

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*.
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>>
NOTE: <<< HTTP/1.1 404 Not Found
NOTE: <<< Age: 0

NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline'
'unsafe-eval' https:; style-src 'unsafe-inline' 'self' https:; default-src
'self' https: data: blob:; report-uri
https://error-collector.ted.com/?context=csp-report

NOTE: <<< Content-Type: text/html; charset=utf-8

NOTE: <<< Date: Sat, 06 May 2017 18:51:10 GMT

NOTE: <<< Server: nginx

NOTE: <<< Set-Cookie: _nu=1494096670.957; Expires=Thu, 05 May 2022 18:51:10 GMT;
Path=/

NOTE: <<< Set-Cookie: _abby=Lvv8yWxjLHErXBP; Expires=Thu, 05 May 2022 18:51:10 GMT;
Path=/; Domain=.ted.com

NOTE: <<< Status: 404 Not Found

NOTE: <<< X-Served-By: e02; w12

NOTE: <<< Content-Length: 4123

NOTE: <<< Connection: Close

NOTE: <<<

ERROR: Invalid reply received from the HTTP server. Use the debug option for more info.
NOTE: The SAS System stopped processing this step because of errors
Thanks

Re: read url fail

GeorgeSAS — Sat, 06 May 2017 18:54:55 GMT

Thank you art,

After I run your code here is SAS log:

23 FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
24 DATA SOURCE2 (drop=junk);
25 informat junk $80.;
26 informat published $8.;
27 informat title $100.;
28 INFILE SOURCE LRECL=32767 DELIMITER=">";
29 INPUT @"<spam class='meta'"
30 published &/////junk title &;
31 title=tranwrd(scan(title,1,'<'),''',"'");
32 RUN;

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*.
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>>
NOTE: <<< HTTP/1.1 404 Not Found
NOTE: <<< Age: 0

NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline'
'unsafe-eval' https:; style-src 'unsafe-inline' 'self' https:; default-src
'self' https: data: blob:; report-uri
https://error-collector.ted.com/?context=csp-report

NOTE: <<< Content-Type: text/html; charset=utf-8

NOTE: <<< Date: Sat, 06 May 2017 18:54:06 GMT

NOTE: <<< Server: nginx

NOTE: <<< Set-Cookie: _nu=1494096846.592; Expires=Thu, 05 May 2022 18:54:06 GMT;
Path=/

NOTE: <<< Set-Cookie: _abby=vFKhWaq8GMJMGiy; Expires=Thu, 05 May 2022 18:54:06 GMT;
Path=/; Domain=.ted.com

NOTE: <<< Status: 404 Not Found

NOTE: <<< X-Served-By: e12; w12

NOTE: <<< Content-Length: 4123

NOTE: <<< Connection: Close

NOTE: <<<

ERROR: Invalid reply received from the HTTP server. Use the debug option for more info.
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.SOURCE2 may be incomplete. When this step was stopped
there were 0 observations and 2 variables.
WARNING: Data set WORK.SOURCE2 was not replaced because this step was stopped

Thanks

Re: read url fail

Kurt_Bremser — Sat, 06 May 2017 19:38:00 GMT

Could it be that the site doesn't like SAS/URL as a user agent and responds with a 404?

Re: read url fail

art297 — Sat, 06 May 2017 19:45:55 GMT

A major difference between my log and your log:

Yours: GET /talks/quick-list HTTP/1.0

Mine: GET /talks/quick-list HTTP/1.1

Is it possible that you're behind a firewall that doesn't support http/1.1 requests?

Art, CEO, AnalystFinder.com

Re: read url fail

GeorgeSAS — Sat, 06 May 2017 20:05:34 GMT

If I change URL to https://www.ted.com/talks it will be ok. But that is not the url I wanted.

Re: read url fail

Patrick — Sun, 07 May 2017 00:38:28 GMT

Hi @GeorgeSAS

The code @art297 posted works for me as well. See below:

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>> 
NOTE: <<< HTTP/1.1 200 OK
NOTE: <<< Age: 0
NOTE: <<< Cache-Control: max-age=0, public, s-maxage=30
NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline' 'unsafe-eval' https:; style-src 'unsafe-inline' 'self' 
      https:; default-src 'self' https: data: blob:; report-uri https://error-collector.ted.com/?context=csp-report
NOTE: <<< Content-Type: text/html; charset=utf-8
NOTE: <<< Date: Sun, 07 May 2017 00:29:18 GMT
NOTE: <<< ETag: W/"bc10c663abfefcaba0aa45f32b8efbe2"
NOTE: <<< Server: nginx
NOTE: <<< Set-Cookie: _nu=1494116958.827; Expires=Fri, 06 May 2022 00:29:18 GMT; Path=/
NOTE: <<< Set-Cookie: _abby=7OCod3AeLqPqydy; Expires=Fri, 06 May 2022 00:29:18 GMT; Path=/; Domain=.ted.com
NOTE: <<< Status: 200 OK
NOTE: <<< X-Content-Type-Options: nosniff
2                                                          The SAS System                                  10:29 Sunday, May 7, 2017

NOTE: <<< X-Served-By: e11; o11
NOTE: <<< X-XSS-Protection: 1; mode=block
NOTE: <<< Connection: Close
NOTE: <<< 
NOTE: The infile SOURCE is:
      Filename=https://www.ted.com/talks/quick-list,
      Local Host Name=<user name>,
      Local Host IP addr=<ip address>,
      Service Hostname Name=www.ted.com,
      Service IP addr=<ip address>,Service Name=N/A,
      Service Portno=443,Lrecl=32767,Recfm=Variable

NOTE: 1847 records were read from the infile SOURCE.