BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
Murali
SAS Employee

Hello All,

I am trying to query textual comments data from the youtube api using proc http with the below code. However, i am getting a http error in the log and my request is not getting processed. I have referenced for related documentation why it is failing but other than the description of the error at http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html, i couldn't figure out the issue.  I have tried using my youtube login credentials in the webusername and webpassword parameters but it didnt work. Appreciate your help !

ERROR: java.io.IOException: Server returned HTTP response code: 415 for URL:

http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1

Below is the code i used, this is similar to the code that is available at http://sww.sas.com/toolpool/pool/Twitter_Web_Application/index.html which was used for retrieving twitter data.

GOPTIONS ACCESSIBLE;

/* this is where we will store the HTML output */

filename odsout "C:\Inetpub\wwwroot\youtube";

ods listing close;

/*The code to connect to twitter using SAS looks like this: */

filename REQUEST "C:\temp\blank.txt"; /*make sure to create this blank txt file before running the code*/
filename SchOut "c:\temp\Video_Meta.xml"; 

/* url is the location we pass to proc http */

%let url = "http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1";
%put &url;

/*connect to youtube */

proc http
    in=REQUEST
    out=SchOut
    url=&url
    method="get"
    proxyhost="inetgw.unx.sas.com"
    proxyport=80
    webusername='' /* not required here */
    webpassword='' /* not required here */
    ;
run; 

Thanks,

Murali

1 ACCEPTED SOLUTION

Accepted Solutions
FriedEgg
SAS Employee

I still am shakey as to why you are connecting through the proxy host?

Why not just use the filename statement for url access?

filename yt url 'http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1';

data _null_;

infile yt length=len;

input record $varying32000. len;

put record $varying32000. len;

run;

libname ytube xml xmlfileref=yt;

data ytube;

set ytube.entry;

run;

View solution in original post

13 REPLIES 13
FriedEgg
SAS Employee

Do you actually have access to the proxyhost you are attempting to connect through?

If all you are trying to go is download an xml form you could (if you are using unix) just use wget.  I assume windows has a similar tool but it does not come to mind at the moment.

x wget &url > /temp/schout.xml;

EDIT: you can get wget for windows here: http://gnuwin32.sourceforge.net/packages/wget.htm

Murali
SAS Employee

Thanks for your help ! I believe I have the access required, because when i try the same proxy server and port for twitter api, I can get it work. I have some third party free softwares currently which I am able to use to directly download the data from twitter in .csv or xml formats. However, I would like to be able to use SAS instead to showcase the capabilities even for data extraction through the standard api interfaces.

Tom
Super User Tom
Super User

On my Unix box I have to explicitely add the port number to the URL to read it from SAS code.  Try adding :80 after the hostname.

http://gdata.youtube.com:80/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1

Murali
SAS Employee

I tried it, didn't work Smiley Sad

FriedEgg
SAS Employee

I still am shakey as to why you are connecting through the proxy host?

Why not just use the filename statement for url access?

filename yt url 'http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2&max-results=50&start-index=1';

data _null_;

infile yt length=len;

input record $varying32000. len;

put record $varying32000. len;

run;

libname ytube xml xmlfileref=yt;

data ytube;

set ytube.entry;

run;

Murali
SAS Employee

Thank you for the help ! I ran this and it works perfectly fine excepting some missing details like author id of the comment posted. A colleague of mine also helped me solve the issue with proc http by just removing the proxy host and port details from the code. Thanks a lot again Smiley Happy

FriedEgg
SAS Employee

Yes, it seemed like the likely cause of the issues with proc http was the proxy options you were using.  The variables will be better defined not using the defualt xmlmapper file and instead definind your own the fits the data apporiately.

http://support.sas.com/documentation/cdl/en/engxml/62845/HTML/default/viewer.htm#a002484837.htm

Murali
SAS Employee

Yes, I am using xmlmapper for this. Thanks!

art297
Opal | Level 21

Muali,

If you got it to run with your original code would you mind posting the code that you finally used?  I'm wondering as I haven't been able to get proc http to do it yet (other than using FriedEgg's suggested code).

Art

Murali
SAS Employee

Hello Art,

Here is the code.

%let url = "http://gdata.youtube.com/feeds/api/videos/pzI4D6dyp_o/comments?v=2%nrstr(&max)-results=50%nrstr(&sta...";

%put &url;

filename test "< --path-- >.xml";

proc http

/*  in=REQUEST */

    out=test

    url=&url

    method="get"

/*    proxyhost="inetgw.unx.sas.com" */

/*    proxyport=80 */

/*    webusername=''*/

/*    webpassword=''*/

    ;

run

I am yet to do the mapper, will post the rest of the code when i'm done with that.

Thanks,

Murali

art297
Opal | Level 21

Much appreciated!  That worked for me as well!

FriedEgg
SAS Employee

filename in temp;

filename out temp;

data null;

   file in;

   input;

   put infile;

   datalines;

pzI4D6dyp_o/comments?v=2%nrstr(&max)-results=50%nrstr(&start)-index=1

;

proc http in=in out=out url='http://gdata.youtube.com/feeds/api/videos'

  method='get'

run;

libname ytube xml xmlfileref=out;

data ytube;

set ytube.entry;

run;

jeeth79usa
Calcite | Level 5

Hello All,

I need help with the similar situation. Thanks to you all for your postings. It was very informative.

I was wondering on how to handle the proc http, if my URL is a variable in a data set called HTTP. I have used the url access method, but I was getting an error saying invalid server response. I need to run all the url links one by one and get the values stored on the server (some name), and send those values to the name variable back to my work data set. I was thinking to export the file to the temp folder on unix, and then use the

filename in temp;

filename out temp;

Please advise.

I have used your method here:

My environment is UNIX:

proc sql noprint;

select count(*) into :httpcnt

from work.URL_Test where substr(http,1,5)='http:';

quit;

%put &httpcnt;

data _null_;

  set Test;

call symput("cpe",trim(variable));

call symput("date_time",trim(variable));

run;

%let url = "http://host:80abcdetcetc%nrstr(&cpe)%nrstr(%&)incidentDate=%nrstr(&date_time)";

filename http_get "/test/test/http.csv";

%macro get_http;

/*%do i=1 %to &httpcnt;*/

proc http

    out=http_get

    url=&url

    method="get" ;

run;

/*%end;*/

%mend;

%get_http;

Btw, This method worked for me when I took just a single link from the variable to test its out. It worked great, and created forllowing valuesin the out=http_get.csv file.

Please let me know if its possible to get only "name I am looking for"? are there any options to remove the tags?? I need to run all the links, and include the results to my data set (in name variable). Thanks a lot in advance.

*****http_get.csv file.

<html>

<head>

<title>adcdef</title>

</head>

<body>

<h1>name I am looking for</h1>

</body>

</html>

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 13 replies
  • 11152 views
  • 4 likes
  • 5 in conversation