BookmarkSubscribeRSS Feed
ilikesas
Barite | Level 11

hi, 

 

is there a way to download information froma webpage into sas?

 

fo example, if I go to a youtube video: 

Kitten Meets Computer

https://www.youtube.com/watch?v=kQ1L39be1e0

 

is it possible to download say the number of views and the number of likes at any given moment?

 

thank you!

10 REPLIES 10
Reeza
Super User

HTML is text. 

Go to the page in question and view the page source. See if you can find the elements in the page source. If you can find it then yes you can, and then you can determine how to parse the file to get the information of interest. 

 

 

ilikesas
Barite | Level 11

Hi Reeza,

 

On the web page I right click - inspect on the views count and got that element in html. But when I try to parse it in sas I have trouble. Is there some documentation on how to parse html elements?

 

Thank you!

Reeza
Super User

Not really, but as I mentioned it's a text file, you process it the same way. You can search using FIND/INDEX functions or PRX functions.

 

Also, the solution from Tom Kari, in your other question, had samples of how that was occurring.  

 

SAS isn't a good webscraping tool...it can be used, but there are better tools out there...Import.IO is free, either web or desktop based and has a free version. It can be scripted so that you have results run and available on a regular basis. 

 

 

LinusH
Tourmaline | Level 20
Perhaps this could be of interest?

https://developers.google.com/youtube/2.0/developers_guide_protocol_video_entries
Data never sleeps
ilikesas
Barite | Level 11

Hi LinusH,

 

thanks for the link!

 

In the meanwhile I just tried to do some code which will extract the number of views of the youtube video Kitten Meets Computer:

 

data kitten;

      length view $32767;

      filename the_vid url "https://www.youtube.com/watch?v=kQ1L39be1e0";

      infile the_vid lrecl=32767;

      input;

      view = _infile_;

run;

 

 

data kitten2;

set kitten;

IF

(INDEX(view,'watch-view-count') = 0)  THEN DELETE;

run;

 

data kitten3;

set kitten2;

place=index(view,'watch-view-count');

like=substr(view,place+18,5);

drop place;

run;

 

Just in the kitten3 I should do the part "like=substr(view,place+18,5);"  dynamic because once the number of views reaches 10,000 the 5 will become 6 etc. 

Reeza
Super User

Also, post a screenshot or sample of the text you're trying to parse.

It's hard to make suggestions otherwise.

TomKari
Onyx | Level 15

This is a really nice application for regular expressions. Try this code instead of your data steps for kitten2 and kitten3:

 

data kitten2;
retain re;
if _n_ = 1
then re=prxparse('#(<div class="watch-view-count">)([\d,]+)(</div>)#');
set kitten;
if prxmatch(re, view)
then do;
  view_count = input(prxposn(re, 2, view), comma20.);
  output;
end;
run;

 

The regular expression (assignment to re) has three parts, each in a set of parentheses:

First is the literal "<div class="watch-view-count">", and third is the literal "</div>"

Second, represented by "[\d,]+", looks for one or more repetitions of a digit or a blank.

 

PRXPOSN returns the string found as the second part, as a character variable, which is converted to a number by the INPUT function.

 

Tom

Reeza
Super User
Instead of substr look at other functions such as scan. Also, how did you calculate 18? If you used index or find you can dynamically calculate the amount.
ilikesas
Barite | Level 11

hi,

I actually counted the number of spaces, I know, not the most efficient way...

 

I tried meanwhile scraping other web pages. I tried to scrap the friends that liked a post that I uploaded on my faceboof, but it didn't work --> sas didn't download any of the html elements associated with the opening sliding window with the people who liked. 

I also saw an example from the paper that you suggested that scraps job postings from the Edmonton career website. There the actual scraping is done with Perl, and the parsing with SAS, so I guess that for more complicated wed scraping SAS is not the best program.

 

I also saw import.io and tried it online and it seemed to be a bit too limited because it chooses itself what to scrap, and sometimes it didn't scrap anything.

 

Thank you!

Reeza
Super User

@ilikesas wrote:

hi,

I actually counted the number of spaces, I know, not the most efficient way...

 

I tried meanwhile scraping other web pages. I tried to scrap the friends that liked a post that I uploaded on my faceboof, but it didn't work --> sas didn't download any of the html elements associated with the opening sliding window with the 

 

I also saw import.io and tried it online and it seemed to be a bit too limited because it chooses itself what to scrap, and sometimes it didn't scrap anything.

 

Thank you!


 

1. Yes, change that to a scan or find and you'll probably be able to make it dynamic

2. For FB you definitely need to go through the API, same with Twitter. Here's a SAS paper that tried it years ago, and most likely won't work today : http://www.sascommunity.org/wiki/Social_Networking_and_SAS:_Running_PROCs_on_Your_Facebook_Friends

3. Import.Io -> the user can select the fields to parse. In general download app is better than the web one.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 10 replies
  • 1478 views
  • 2 likes
  • 4 in conversation