read url fail

Reply
Regular Contributor
Posts: 241

read url fail

.I have problem  to read the url,please help.

 

Thanks!

.

FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
DATA SOURCE2;
FORMAT WEBPAGE $1000.;
INFILE SOURCE LRECL=32767 DELIMITER=">";
INPUT WEBPAGE $ @@;
RUN;
Trusted Advisor
Posts: 1,566

Re: read url fail

Posted in reply to GeorgeSAS

What problem do you have ?

Please post your log.

Regular Contributor
Posts: 241

Re: read url fail

Here is the SAS log error notes:

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*.
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>>
NOTE: <<< HTTP/1.1 404 Not Found
NOTE: <<< Age: 0

NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline'
'unsafe-eval' https:; style-src 'unsafe-inline' 'self' https:; default-src
'self' https: data: blob:; report-uri
https://error-collector.ted.com/?context=csp-report

NOTE: <<< Content-Type: text/html; charset=utf-8

NOTE: <<< Date: Sat, 06 May 2017 18:51:10 GMT

NOTE: <<< Server: nginx

NOTE: <<< Set-Cookie: _nu=1494096670.957; Expires=Thu, 05 May 2022 18:51:10 GMT;
Path=/

NOTE: <<< Set-Cookie: _abby=Lvv8yWxjLHErXBP; Expires=Thu, 05 May 2022 18:51:10 GMT;
Path=/; Domain=.ted.com

NOTE: <<< Status: 404 Not Found

NOTE: <<< X-Served-By: e02; w12

NOTE: <<< Content-Length: 4123

NOTE: <<< Connection: Close

NOTE: <<<

ERROR: Invalid reply received from the HTTP server. Use the debug option for more info.
NOTE: The SAS System stopped processing this step because of errors
Thanks
Valued Guide
Posts: 634

Re: read url fail

Posted in reply to GeorgeSAS
Why are you using the %STR function?
PROC Star
Posts: 7,474

Re: read url fail

Posted in reply to GeorgeSAS

Your code reads the url just fine, but it doesn't do anything to parse it. e.g., if you wanted the dates and titles of the presentations, the following works for all but one of the presentations. The one it misses contains a non-ASCII character:

 

FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
DATA SOURCE2 (drop=junk);
  informat junk $80.;
  informat published $8.;
  informat title $100.;
  INFILE SOURCE LRECL=32767 DELIMITER=">";
  INPUT @"<spam class='meta'"
  published &/////junk title &;
  title=tranwrd(scan(title,1,'<'),'&#39;',"'");
RUN;

Art, CEO, AnalystFinder.com

 

Regular Contributor
Posts: 241

Re: read url fail

Thank you art,

After I run your code here is SAS log:

23 FILENAME SOURCE URL "%STR(https://www.ted.com/talks/quick-list)" DEBUG;
24 DATA SOURCE2 (drop=junk);
25 informat junk $80.;
26 informat published $8.;
27 informat title $100.;
28 INFILE SOURCE LRECL=32767 DELIMITER=">";
29 INPUT @"<spam class='meta'"
30 published &/////junk title &;
31 title=tranwrd(scan(title,1,'<'),'&#39;',"'");
32 RUN;

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*.
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>>
NOTE: <<< HTTP/1.1 404 Not Found
NOTE: <<< Age: 0

NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline'
'unsafe-eval' https:; style-src 'unsafe-inline' 'self' https:; default-src
'self' https: data: blob:; report-uri
https://error-collector.ted.com/?context=csp-report

NOTE: <<< Content-Type: text/html; charset=utf-8

NOTE: <<< Date: Sat, 06 May 2017 18:54:06 GMT

NOTE: <<< Server: nginx

NOTE: <<< Set-Cookie: _nu=1494096846.592; Expires=Thu, 05 May 2022 18:54:06 GMT;
Path=/

NOTE: <<< Set-Cookie: _abby=vFKhWaq8GMJMGiy; Expires=Thu, 05 May 2022 18:54:06 GMT;
Path=/; Domain=.ted.com

NOTE: <<< Status: 404 Not Found

NOTE: <<< X-Served-By: e12; w12

NOTE: <<< Content-Length: 4123

NOTE: <<< Connection: Close

NOTE: <<<

ERROR: Invalid reply received from the HTTP server. Use the debug option for more info.
NOTE: The SAS System stopped processing this step because of errors.
WARNING: The data set WORK.SOURCE2 may be incomplete. When this step was stopped
there were 0 observations and 2 variables.
WARNING: Data set WORK.SOURCE2 was not replaced because this step was stopped


Thanks
Super User
Posts: 7,782

Re: read url fail

Posted in reply to GeorgeSAS

Could it be that the site doesn't like SAS/URL as a user agent and responds with a 404?

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
PROC Star
Posts: 7,474

Re: read url fail

Posted in reply to GeorgeSAS

 A major difference between my log and your log:

 

Yours: GET /talks/quick-list HTTP/1.0

Mine: GET /talks/quick-list HTTP/1.1

 

Is it possible that you're behind a firewall that doesn't support http/1.1 requests?

 

Art, CEO, AnalystFinder.com

 

Regular Contributor
Posts: 241

Re: read url fail

[ Edited ]

If I change URL to https://www.ted.com/talks it will be ok. But that is not the url I wanted.

Respected Advisor
Posts: 4,173

Re: read url fail

Posted in reply to GeorgeSAS

Hi @GeorgeSAS

The code @art297 posted works for me as well. See below:

NOTE: >>> GET /talks/quick-list HTTP/1.0
NOTE: >>> Host: www.ted.com:443
NOTE: >>> Accept: */*
NOTE: >>> Accept-Language: en
NOTE: >>> Accept-Charset: iso-8859-1,*,utf-8
NOTE: >>> User-Agent: SAS/URL
NOTE: >>> 
NOTE: <<< HTTP/1.1 200 OK
NOTE: <<< Age: 0
NOTE: <<< Cache-Control: max-age=0, public, s-maxage=30
NOTE: <<< Content-Security-Policy-Report-Only: script-src 'unsafe-inline' 'unsafe-eval' https:; style-src 'unsafe-inline' 'self' 
      https:; default-src 'self' https: data: blob:; report-uri https://error-collector.ted.com/?context=csp-report
NOTE: <<< Content-Type: text/html; charset=utf-8
NOTE: <<< Date: Sun, 07 May 2017 00:29:18 GMT
NOTE: <<< ETag: W/"bc10c663abfefcaba0aa45f32b8efbe2"
NOTE: <<< Server: nginx
NOTE: <<< Set-Cookie: _nu=1494116958.827; Expires=Fri, 06 May 2022 00:29:18 GMT; Path=/
NOTE: <<< Set-Cookie: _abby=7OCod3AeLqPqydy; Expires=Fri, 06 May 2022 00:29:18 GMT; Path=/; Domain=.ted.com
NOTE: <<< Status: 200 OK
NOTE: <<< X-Content-Type-Options: nosniff
2                                                          The SAS System                                  10:29 Sunday, May 7, 2017

NOTE: <<< X-Served-By: e11; o11
NOTE: <<< X-XSS-Protection: 1; mode=block
NOTE: <<< Connection: Close
NOTE: <<< 
NOTE: The infile SOURCE is:
      Filename=https://www.ted.com/talks/quick-list,
      Local Host Name=<user name>,
      Local Host IP addr=<ip address>,
      Service Hostname Name=www.ted.com,
      Service IP addr=<ip address>,Service Name=N/A,
      Service Portno=443,Lrecl=32767,Recfm=Variable

NOTE: 1847 records were read from the infile SOURCE.
Ask a Question
Discussion stats
  • 9 replies
  • 241 views
  • 1 like
  • 6 in conversation