topic downloading data from a webpage in SAS Data Management

downloading data from a webpage

ilikesas — Sun, 14 Feb 2016 02:05:42 GMT

hi,

is there a way to download information froma webpage into sas?

fo example, if I go to a youtube video:

Kitten Meets Computer

https://www.youtube.com/watch?v=kQ1L39be1e0

is it possible to download say the number of views and the number of likes at any given moment?

thank you!

Re: downloading data from a webpage

Reeza — Sun, 14 Feb 2016 03:11:00 GMT

HTML is text.

Go to the page in question and view the page source. See if you can find the elements in the page source. If you can find it then yes you can, and then you can determine how to parse the file to get the information of interest.

Re: downloading data from a webpage

ilikesas — Sun, 14 Feb 2016 04:03:01 GMT

Hi Reeza,

On the web page I right click - inspect on the views count and got that element in html. But when I try to parse it in sas I have trouble. Is there some documentation on how to parse html elements?

Thank you!

Re: downloading data from a webpage

Reeza — Sun, 14 Feb 2016 04:05:53 GMT

Not really, but as I mentioned it's a text file, you process it the same way. You can search using FIND/INDEX functions or PRX functions.

Also, the solution from Tom Kari, in your other question, had samples of how that was occurring.

SAS isn't a good webscraping tool...it can be used, but there are better tools out there...Import.IO is free, either web or desktop based and has a free version. It can be scripted so that you have results run and available on a regular basis.

Re: downloading data from a webpage

LinusH — Sun, 14 Feb 2016 11:02:18 GMT

Perhaps this could be of interest?

https://developers.google.com/youtube/2.0/developers_guide_protocol_video_entries

Re: downloading data from a webpage

ilikesas — Sun, 14 Feb 2016 15:39:28 GMT

Hi LinusH,

thanks for the link!

In the meanwhile I just tried to do some code which will extract the number of views of the youtube video Kitten Meets Computer:

data kitten;

length view $32767;

filename the_vid url "https://www.youtube.com/watch?v=kQ1L39be1e0";

infile the_vid lrecl=32767;

input;

view = _infile_;

run;

data kitten2;

set kitten;

(INDEX(view,'watch-view-count') = 0) THEN DELETE;

run;

data kitten3;

set kitten2;

place=index(view,'watch-view-count');

like=substr(view,place+18,5);

drop place;

run;

Just in the kitten3 I should do the part "like=substr(view,place+18,5);" dynamic because once the number of views reaches 10,000 the 5 will become 6 etc.

Re: downloading data from a webpage

Reeza — Sun, 14 Feb 2016 17:16:26 GMT

Instead of substr look at other functions such as scan. Also, how did you calculate 18? If you used index or find you can dynamically calculate the amount.

Re: downloading data from a webpage

Reeza — Sun, 14 Feb 2016 18:03:56 GMT

Also, post a screenshot or sample of the text you're trying to parse.

It's hard to make suggestions otherwise.

Re: downloading data from a webpage

ilikesas — Sun, 14 Feb 2016 18:45:32 GMT

hi,

I actually counted the number of spaces, I know, not the most efficient way...

I tried meanwhile scraping other web pages. I tried to scrap the friends that liked a post that I uploaded on my faceboof, but it didn't work --> sas didn't download any of the html elements associated with the opening sliding window with the people who liked.

I also saw an example from the paper that you suggested that scraps job postings from the Edmonton career website. There the actual scraping is done with Perl, and the parsing with SAS, so I guess that for more complicated wed scraping SAS is not the best program.

I also saw import.io and tried it online and it seemed to be a bit too limited because it chooses itself what to scrap, and sometimes it didn't scrap anything.

Thank you!

Re: downloading data from a webpage

Reeza — Sun, 14 Feb 2016 18:52:02 GMT

@ilikesas wrote:

hi,

I actually counted the number of spaces, I know, not the most efficient way...

I tried meanwhile scraping other web pages. I tried to scrap the friends that liked a post that I uploaded on my faceboof, but it didn't work --> sas didn't download any of the html elements associated with the opening sliding window with the

I also saw import.io and tried it online and it seemed to be a bit too limited because it chooses itself what to scrap, and sometimes it didn't scrap anything.

Thank you!

1. Yes, change that to a scan or find and you'll probably be able to make it dynamic

2. For FB you definitely need to go through the API, same with Twitter. Here's a SAS paper that tried it years ago, and most likely won't work today : http://www.sascommunity.org/wiki/Social_Networking_and_SAS:_Running_PROCs_on_Your_Facebook_Friends

3. Import.Io -> the user can select the fields to parse. In general download app is better than the web one.

Re: downloading data from a webpage

TomKari — Tue, 16 Feb 2016 22:50:39 GMT

This is a really nice application for regular expressions. Try this code instead of your data steps for kitten2 and kitten3:

data kitten2;
retain re;
if _n_ = 1
then re=prxparse('#(<div class="watch-view-count">)([\d,]+)(</div>)#');
set kitten;
if prxmatch(re, view)
then do;
view_count = input(prxposn(re, 2, view), comma20.);
output;
end;
run;

The regular expression (assignment to re) has three parts, each in a set of parentheses:

First is the literal "<div class="watch-view-count">", and third is the literal "</div>"

Second, represented by "[\d,]+", looks for one or more repetitions of a digit or a blank.

PRXPOSN returns the string found as the second part, as a character variable, which is converted to a number by the INPUT function.

Tom