topic PROC HTTP error in SAS Procedures

PROC HTTP error

Murali — Thu, 22 Sep 2011 21:29:02 GMT

Hello All,

I am trying to query textual comments data from the youtube api using proc http with the below code. However, i am getting a http error in the log and my request is not getting processed. I have referenced for related documentation why it is failing but other than the description of the error at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html, i couldn't figure out the issue. I have tried using my youtube login credentials in the webusername and webpassword parameters but it didnt work. Appreciate your help !

ERROR: java.io.IOException: Server returned HTTP response code: 415 for URL:

http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1

Below is the code i used, this is similar to the code that is available at http://sww.sas.com/toolpool/pool/Twitter_Web_Application/index.html which was used for retrieving twitter data.

GOPTIONS ACCESSIBLE;

/* this is where we will store the HTML output */

filename odsout "C:\Inetpub\wwwroot\youtube";

ods listing close;

/*The code to connect to twitter using SAS looks like this: */

filename REQUEST "C:\temp\blank.txt"; /*make sure to create this blank txt file before running the code*/
filename SchOut "c:\temp\Video_Meta.xml";

/* url is the location we pass to proc http */

%let url = "http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1";
%put &url;

/*connect to youtube */

proc http
    in=REQUEST
    out=SchOut
    url=&url
    method="get"
    proxyhost="inetgw.unx.sas.com"
    proxyport=80
    webusername='' /* not required here */
    webpassword='' /* not required here */
    ;
run;

Thanks,

Murali

PROC HTTP error

FriedEgg — Thu, 22 Sep 2011 22:51:39 GMT

Do you actually have access to the proxyhost you are attempting to connect through?

If all you are trying to go is download an xml form you could (if you are using unix) just use wget. I assume windows has a similar tool but it does not come to mind at the moment.

x wget &url > /temp/schout.xml;

EDIT: you can get wget for windows here: http://gnuwin32.sourceforge.net/packages/wget.htm

PROC HTTP error

Murali — Fri, 23 Sep 2011 13:39:17 GMT

Thanks for your help ! I believe I have the access required, because when i try the same proxy server and port for twitter api, I can get it work. I have some third party free softwares currently which I am able to use to directly download the data from twitter in .csv or xml formats. However, I would like to be able to use SAS instead to showcase the capabilities even for data extraction through the standard api interfaces.

PROC HTTP error

Tom — Fri, 23 Sep 2011 14:49:35 GMT

On my Unix box I have to explicitely add the port number to the URL to read it from SAS code. Try adding :80 after the hostname.

http://gdata.youtube.com:80/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1

PROC HTTP error

Murali — Fri, 23 Sep 2011 15:00:48 GMT

I tried it, didn't work

Re: PROC HTTP error

FriedEgg — Fri, 23 Sep 2011 15:21:43 GMT

I still am shakey as to why you are connecting through the proxy host?

Why not just use the filename statement for url access?

filename yt url 'http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1';

data _null_;

infile yt length=len;

input record $varying32000. len;

put record $varying32000. len;

run;

libname ytube xml xmlfileref=yt;

data ytube;

set ytube.entry;

run;

PROC HTTP error

Murali — Fri, 23 Sep 2011 16:45:22 GMT

Thank you for the help ! I ran this and it works perfectly fine excepting some missing details like author id of the comment posted. A colleague of mine also helped me solve the issue with proc http by just removing the proxy host and port details from the code. Thanks a lot again

PROC HTTP error

FriedEgg — Fri, 23 Sep 2011 17:17:09 GMT

Yes, it seemed like the likely cause of the issues with proc http was the proxy options you were using. The variables will be better defined not using the defualt xmlmapper file and instead definind your own the fits the data apporiately.

http://support.sas.com/documentation/cdl/en/engxml/62845/HTML/default/viewer.htm#a002484837.htm

PROC HTTP error

Murali — Fri, 23 Sep 2011 17:22:09 GMT

Yes, I am using xmlmapper for this. Thanks!

PROC HTTP error

art297 — Fri, 23 Sep 2011 17:49:24 GMT

Muali,

If you got it to run with your original code would you mind posting the code that you finally used? I'm wondering as I haven't been able to get proc http to do it yet (other than using FriedEgg's suggested code).

Art

PROC HTTP error

Murali — Fri, 23 Sep 2011 18:28:27 GMT

Hello Art,

Here is the code.

%let url = "http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2%nrstr(&max)-results=50%nrstr(&start)-index=1";

%put &url;

filename test "< --path-- >.xml";

proc http

/* in=REQUEST */

out=test

url=&url

method="get"

/* proxyhost="inetgw.unx.sas.com" */

/* proxyport=80 */

/* webusername=''*/

/* webpassword=''*/

;

run;

I am yet to do the mapper, will post the rest of the code when i'm done with that.

Thanks,

Murali

PROC HTTP error

art297 — Fri, 23 Sep 2011 18:33:15 GMT

Much appreciated! That worked for me as well!

Re: PROC HTTP error

FriedEgg — Fri, 23 Sep 2011 19:54:42 GMT

filename in temp;

filename out temp;

data null;

file in;

input;

put infile;

datalines;

pzI4D6dyp_o/comments?v=2%nrstr(&max)-results=50%nrstr(&start)-index=1

;

proc http in=in out=out url='http://gdata.youtube.com/feeds/api/videos'

method='get'

run;

libname ytube xml xmlfileref=out;

data ytube;

set ytube.entry;

run;

Re: PROC HTTP error

jeeth79usa — Sat, 07 Apr 2012 22:11:07 GMT

Hello All,

I need help with the similar situation. Thanks to you all for your postings. It was very informative.

I was wondering on how to handle the proc http, if my URL is a variable in a data set called HTTP. I have used the url access method, but I was getting an error saying invalid server response. I need to run all the url links one by one and get the values stored on the server (some name), and send those values to the name variable back to my work data set. I was thinking to export the file to the temp folder on unix, and then use the

filename in temp;

filename out temp;

Please advise.

I have used your method here:

My environment is UNIX:

proc sql noprint;

select count(*) into :httpcnt

from work.URL_Test where substr(http,1,5)='http:';

quit;

%put &httpcnt;

data _null_;

set Test;

call symput("cpe",trim(variable));

call symput("date_time",trim(variable));

run;

%let url = "http://host:80abcdetcetc%nrstr(&cpe)%nrstr(%&)incidentDate=%nrstr(&date_time)";

filename http_get "/test/test/http.csv";

%macro get_http;

/*%do i=1 %to &httpcnt;*/

proc http

out=http_get

url=&url

method="get" ;

run;

/*%end;*/

%mend;

%get_http;

Btw, This method worked for me when I took just a single link from the variable to test its out. It worked great, and created forllowing valuesin the out=http_get.csv file.

Please let me know if its possible to get only "name I am looking for"? are there any options to remove the tags?? I need to run all the links, and include the results to my data set (in name variable). Thanks a lot in advance.

*****http_get.csv file.

<html>

<head>

<title>adcdef</title>

</head>

<body>

<h1>name I am looking for</h1>

</body>

</html>