downloading data from a webpage

[ilikesas]

hi, 

is there a way to download information froma webpage into sas?

fo example, if I go to a youtube video: 

Kitten Meets Computer

https://www.youtube.com/watch?v=kQ1L39be1e0

is it possible to download say the number of views and the number of likes at any given moment?

thank you!

[Reeza]

HTML is text. 

Go to the page in question and view the page source. See if you can find the elements in the page source. If you can find it then yes you can, and then you can determine how to parse the file to get the information of interest. 

[ilikesas]

Hi Reeza,

On the web page I right click - inspect on the views count and got that element in html. But when I try to parse it in sas I have trouble. Is there some documentation on how to parse html elements?

Thank you!

[Reeza]

Not really, but as I mentioned it's a text file, you process it the same way. You can search using FIND/INDEX functions or PRX functions.

Also, the solution from Tom Kari, in your other question, had samples of how that was occurring.  

SAS isn't a good webscraping tool...it can be used, but there are better tools out there...Import.IO is free, either web or desktop based and has a free version. It can be scripted so that you have results run and available on a regular basis. 

[LinusH]

Perhaps this could be of interest?

https://developers.google.com/youtube/2.0/developers_guide_protocol_video_entries

[ilikesas]

Hi LinusH,

thanks for the link!

In the meanwhile I just tried to do some code which will extract the number of views of the youtube video Kitten Meets Computer:

data kitten;

      length view $32767;

      filename the_vid url "https://www.youtube.com/watch?v=kQ1L39be1e0";

      infile the_vid lrecl=32767;

      input;

      view = _infile_;

run;

data kitten2;

set kitten;

IF

(INDEX(view,'watch-view-count') = 0)  THEN DELETE;

run;

data kitten3;

set kitten2;

place=index(view,'watch-view-count');

like=substr(view,place+18,5);

drop place;

run;

Just in the kitten3 I should do the part "like=substr(view,place+18,5);"  dynamic because once the number of views reaches 10,000 the 5 will become 6 etc. 

[Reeza]

Also, post a screenshot or sample of the text you're trying to parse.

It's hard to make suggestions otherwise.

[TomKari]

This is a really nice application for regular expressions. Try this code instead of your data steps for kitten2 and kitten3:

data kitten2;
retain re;
if _n_ = 1
then re=prxparse('#(<div class="watch-view-count">)([\d,]+)(</div>)#');
set kitten;
if prxmatch(re, view)
then do;
  view_count = input(prxposn(re, 2, view), comma20.);
  output;
end;
run;

The regular expression (assignment to re) has three parts, each in a set of parentheses:

First is the literal "<div class="watch-view-count">", and third is the literal "</div>"

Second, represented by "[\d,]+", looks for one or more repetitions of a digit or a blank.

PRXPOSN returns the string found as the second part, as a character variable, which is converted to a number by the INPUT function.

Tom

[Reeza]

Instead of substr look at other functions such as scan. Also, how did you calculate 18? If you used index or find you can dynamically calculate the amount.

[ilikesas]

hi,

I actually counted the number of spaces, I know, not the most efficient way...

I tried meanwhile scraping other web pages. I tried to scrap the friends that liked a post that I uploaded on my faceboof, but it didn't work --> sas didn't download any of the html elements associated with the opening sliding window with the people who liked. 

I also saw an example from the paper that you suggested that scraps job postings from the Edmonton career website. There the actual scraping is done with Perl, and the parsing with SAS, so I guess that for more complicated wed scraping SAS is not the best program.

I also saw import.io and tried it online and it seemed to be a bit too limited because it chooses itself what to scrap, and sometimes it didn't scrap anything.

Thank you!

[Reeza]

---

@ilikesas wrote:

hi,

I actually counted the number of spaces, I know, not the most efficient way...

I tried meanwhile scraping other web pages. I tried to scrap the friends that liked a post that I uploaded on my faceboof, but it didn't work --> sas didn't download any of the html elements associated with the opening sliding window with the 

I also saw import.io and tried it online and it seemed to be a bit too limited because it chooses itself what to scrap, and sometimes it didn't scrap anything.

Thank you!

---

1. Yes, change that to a scan or find and you'll probably be able to make it dynamic

2. For FB you definitely need to go through the API, same with Twitter. Here's a SAS paper that tried it years ago, and most likely won't work today : http://www.sascommunity.org/wiki/Social_Networking_and_SAS:_Running_PROCs_on_Your_Facebook_Friends

3. Import.Io -> the user can select the fields to parse. In general download app is better than the web one.