read url fail

[GeorgeSAS]

.I have problem  to read the url,please help.

Thanks!

.

FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
DATA SOURCE2;
FORMAT WEBPAGE $1000.;
INFILE SOURCE LRECL=32767 DELIMITER=">";
INPUT WEBPAGE $ @@;
RUN;

[Shmuel]

What problem do you have ?

Please post your log.

[GeorgeSAS]

Here is the SAS log error notes:

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*.
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>>
NOTE: <<< HTTP/1.1 404 Not Found
NOTE: <<< Age: 0

NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline'
'unsafe-eval' https:; style-src 'unsafe-inline' 'self' https:; default-src
'self' https: data: blob:; report-uri
https://error-collector.ted.com/?context=csp-report

NOTE: <<< Content-Type: text/html; charset=utf-8

NOTE: <<< Date: Sat, 06 May 2017 18:51:10 GMT

NOTE: <<< Server: nginx

NOTE: <<< Set-Cookie: _nu=1494096670.957; Expires=Thu, 05 May 2022 18:51:10 GMT;
Path=/

NOTE: <<< Set-Cookie: _abby=Lvv8yWxjLHErXBP; Expires=Thu, 05 May 2022 18:51:10 GMT;
Path=/; Domain=.ted.com

NOTE: <<< Status: 404 Not Found

NOTE: <<< X-Served-By: e02; w12

NOTE: <<< Content-Length: 4123

NOTE: <<< Connection: Close

NOTE: <<<

ERROR: Invalid reply received from the HTTP server. Use the debug option for more info.
NOTE: The SAS System stopped processing this step because of errors
Thanks

[ArtC]

Why are you using the %STR function?

[art297]

Your code reads the url just fine, but it doesn't do anything to parse it. e.g., if you wanted the dates and titles of the presentations, the following works for all but one of the presentations. The one it misses contains a non-ASCII character:

FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
DATA SOURCE2 (drop=junk);
  informat junk $80.;
  informat published $8.;
  informat title $100.;
  INFILE SOURCE LRECL=32767 DELIMITER=">";
  INPUT @"<spam class='meta'"
  published &/////junk title &;
  title=tranwrd(scan(title,1,'<'),'&#39;',"'");
RUN;

Art, CEO, AnalystFinder.com

[GeorgeSAS]

Thank you art,

After I run your code here is SAS log:

23 FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
24 DATA SOURCE2 (drop=junk);
25 informat junk $80.;
26 informat published $8.;
27 informat title $100.;
28 INFILE SOURCE LRECL=32767 DELIMITER=">";
29 INPUT @"<spam class='meta'"
30 published &/////junk title &;
31 title=tranwrd(scan(title,1,'<'),'&#39;',"'");
32 RUN;

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*.
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>>
NOTE: <<< HTTP/1.1 404 Not Found
NOTE: <<< Age: 0

NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline'
'unsafe-eval' https:; style-src 'unsafe-inline' 'self' https:; default-src
'self' https: data: blob:; report-uri
https://error-collector.ted.com/?context=csp-report

NOTE: <<< Content-Type: text/html; charset=utf-8

NOTE: <<< Date: Sat, 06 May 2017 18:54:06 GMT

NOTE: <<< Server: nginx

NOTE: <<< Set-Cookie: _nu=1494096846.592; Expires=Thu, 05 May 2022 18:54:06 GMT;
Path=/

NOTE: <<< Set-Cookie: _abby=vFKhWaq8GMJMGiy; Expires=Thu, 05 May 2022 18:54:06 GMT;
Path=/; Domain=.ted.com

NOTE: <<< Status: 404 Not Found

NOTE: <<< X-Served-By: e12; w12

NOTE: <<< Content-Length: 4123

NOTE: <<< Connection: Close

NOTE: <<<

ERROR: Invalid reply received from the HTTP server. Use the debug option for more info.
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.SOURCE2 may be incomplete. When this step was stopped
there were 0 observations and 2 variables.
WARNING: Data set WORK.SOURCE2 was not replaced because this step was stopped


Thanks

[Kurt_Bremser]

Could it be that the site doesn't like SAS/URL as a user agent and responds with a 404?

[art297]

 A major difference between my log and your log:

Yours: GET /talks/quick-list HTTP/1.0

Mine: GET /talks/quick-list HTTP/1.1

Is it possible that you're behind a firewall that doesn't support http/1.1 requests?

Art, CEO, AnalystFinder.com

[GeorgeSAS]

If I change URL to https://www.ted.com/talks it will be ok. But that is not the url I wanted.

[Patrick]

Hi @GeorgeSAS

The code @art297 posted works for me as well. See below:

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>> 
NOTE: <<< HTTP/1.1 200 OK
NOTE: <<< Age: 0
NOTE: <<< Cache-Control: max-age=0, public, s-maxage=30
NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline' 'unsafe-eval' https:; style-src 'unsafe-inline' 'self' 
      https:; default-src 'self' https: data: blob:; report-uri https://error-collector.ted.com/?context=csp-report
NOTE: <<< Content-Type: text/html; charset=utf-8
NOTE: <<< Date: Sun, 07 May 2017 00:29:18 GMT
NOTE: <<< ETag: W/"bc10c663abfefcaba0aa45f32b8efbe2"
NOTE: <<< Server: nginx
NOTE: <<< Set-Cookie: _nu=1494116958.827; Expires=Fri, 06 May 2022 00:29:18 GMT; Path=/
NOTE: <<< Set-Cookie: _abby=7OCod3AeLqPqydy; Expires=Fri, 06 May 2022 00:29:18 GMT; Path=/; Domain=.ted.com
NOTE: <<< Status: 200 OK
NOTE: <<< X-Content-Type-Options: nosniff
2                                                          The SAS System                                  10:29 Sunday, May 7, 2017

NOTE: <<< X-Served-By: e11; o11
NOTE: <<< X-XSS-Protection: 1; mode=block
NOTE: <<< Connection: Close
NOTE: <<< 
NOTE: The infile SOURCE is:
      Filename=https://www.ted.com/talks/quick-list,
      Local Host Name=<user name>,
      Local Host IP addr=<ip address>,
      Service Hostname Name=www.ted.com,
      Service IP addr=<ip address>,Service Name=N/A,
      Service Portno=443,Lrecl=32767,Recfm=Variable

NOTE: 1847 records were read from the infile SOURCE.