PROC HTTP error

[Murali]

Hello All,

I am trying to query textual comments data from the youtube api using proc http with the below code. However, i am getting a http error in the log and my request is not getting processed. I have referenced for related documentation why it is failing but other than the description of the error at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html, i couldn't figure out the issue.  I have tried using my youtube login credentials in the webusername and webpassword parameters but it didnt work. Appreciate your help !

ERROR: java.io.IOException: Server returned HTTP response code: 415 for URL:

http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1

Below is the code i used, this is similar to the code that is available at http://sww.sas.com/toolpool/pool/Twitter_Web_Application/index.html which was used for retrieving twitter data.

GOPTIONS ACCESSIBLE;

/* this is where we will store the HTML output */

filename odsout "C:\Inetpub\wwwroot\youtube";

ods listing close;

/*The code to connect to twitter using SAS looks like this: */

filename REQUEST "C:\temp\blank.txt"; /*make sure to create this blank txt file before running the code*/
filename SchOut "c:\temp\Video_Meta.xml"; 

/* url is the location we pass to proc http */

%let url = "http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1";
%put &url;

/*connect to youtube */

proc http
    in=REQUEST
    out=SchOut
    url=&url
    method="get"
    proxyhost="inetgw.unx.sas.com"
    proxyport=80
    webusername='' /* not required here */
    webpassword='' /* not required here */
    ;
run; 

Thanks,

Murali

[FriedEgg]

Do you actually have access to the proxyhost you are attempting to connect through?

If all you are trying to go is download an xml form you could (if you are using unix) just use wget.  I assume windows has a similar tool but it does not come to mind at the moment.

x wget &url > /temp/schout.xml;

EDIT: you can get wget for windows here: http://gnuwin32.sourceforge.net/packages/wget.htm

[Murali]

Thanks for your help ! I believe I have the access required, because when i try the same proxy server and port for twitter api, I can get it work. I have some third party free softwares currently which I am able to use to directly download the data from twitter in .csv or xml formats. However, I would like to be able to use SAS instead to showcase the capabilities even for data extraction through the standard api interfaces.

[Tom]

On my Unix box I have to explicitely add the port number to the URL to read it from SAS code.  Try adding :80 after the hostname.

http://gdata.youtube.com:80/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1

[Murali]

I tried it, didn't work

[FriedEgg · Accepted solution]

I still am shakey as to why you are connecting through the proxy host?

Why not just use the filename statement for url access?

filename yt url 'http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1';

data _null_;

infile yt length=len;

input record $varying32000. len;

put record $varying32000. len;

run;

libname ytube xml xmlfileref=yt;

data ytube;

set ytube.entry;

run;

[Murali]

Thank you for the help ! I ran this and it works perfectly fine excepting some missing details like author id of the comment posted. A colleague of mine also helped me solve the issue with proc http by just removing the proxy host and port details from the code. Thanks a lot again

[FriedEgg]

Yes, it seemed like the likely cause of the issues with proc http was the proxy options you were using.  The variables will be better defined not using the defualt xmlmapper file and instead definind your own the fits the data apporiately.

http://support.sas.com/documentation/cdl/en/engxml/62845/HTML/default/viewer.htm#a002484837.htm

[Murali]

Yes, I am using xmlmapper for this. Thanks!

[art297]

Muali,

If you got it to run with your original code would you mind posting the code that you finally used?  I'm wondering as I haven't been able to get proc http to do it yet (other than using FriedEgg's suggested code).

Art

[Murali]

Hello Art,

Here is the code.

%let url = "http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2%nrstr(&max)-results=50%nrstr(&sta...";

%put &url;

filename test "< --path-- >.xml";

proc http

/*  in=REQUEST */

    out=test

    url=&url

    method="get"

/*    proxyhost="inetgw.unx.sas.com" */

/*    proxyport=80 */

/*    webusername=''*/

/*    webpassword=''*/

    ;

run; 

I am yet to do the mapper, will post the rest of the code when i'm done with that.

Thanks,

Murali

[art297]

Much appreciated!  That worked for me as well!

[FriedEgg]

filename in temp;

filename out temp;

data null;

   file in;

   input;

   put infile;

   datalines;

pzI4D6dyp_o/comments?v=2%nrstr(&max)-results=50%nrstr(&start)-index=1

;

proc http in=in out=out url='http://gdata.youtube.com/feeds/api/videos'

  method='get'

run;

libname ytube xml xmlfileref=out;

data ytube;

set ytube.entry;

run;

[jeeth79usa]

Hello All,

I need help with the similar situation. Thanks to you all for your postings. It was very informative.

I was wondering on how to handle the proc http, if my URL is a variable in a data set called HTTP. I have used the url access method, but I was getting an error saying invalid server response. I need to run all the url links one by one and get the values stored on the server (some name), and send those values to the name variable back to my work data set. I was thinking to export the file to the temp folder on unix, and then use the

filename in temp;

filename out temp;

Please advise.

I have used your method here:

My environment is UNIX:

proc sql noprint;

select count(*) into :httpcnt

from work.URL_Test where substr(http,1,5)='http:';

quit;

%put &httpcnt;

data _null_;

  set Test;

call symput("cpe",trim(variable));

call symput("date_time",trim(variable));

run;

%let url = "http://host:80abcdetcetc%nrstr(&cpe)%nrstr(%&)incidentDate=%nrstr(&date_time)";

filename http_get "/test/test/http.csv";

%macro get_http;

/*%do i=1 %to &httpcnt;*/

proc http

    out=http_get

    url=&url

    method="get" ;

run;

/*%end;*/

%mend;

%get_http;

Btw, This method worked for me when I took just a single link from the variable to test its out. It worked great, and created forllowing valuesin the out=http_get.csv file.

Please let me know if its possible to get only "name I am looking for"? are there any options to remove the tags?? I need to run all the links, and include the results to my data set (in name variable). Thanks a lot in advance.

*****http_get.csv file.

<html>

<head>

<title>adcdef</title>

</head>

<body>

<h1>name I am looking for</h1>

</body>

</html>