topic Re: webscraping links from a web page in SAS Programming

webscraping links from a web page

ilikesas — Fri, 09 Dec 2016 02:01:47 GMT

Hi,

I am reading Rick Wicklin's SAS blogs, and what I would like to do is to scrape the full list of his posts, otherwise there are too many pages (currently 79) and it is tedious to go over each page and see what posts have been posted.

The approach that I undertook is to go to a certain page and scrape the lines that end with .html (this is how the links end) and which are confined between quotes. Here is the code I tried:

filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/2";
data blog_links(keep=link);
length link $200;
infile rick length=len lrecl=32767;
input line $varying32767. len;
if find(line,".html") then do;
link=scan(line,2,'"');
output;
end;
run;

But I didn't get any links at all. Could you please help me?

Thank you!

Re: webscraping links from a web page

Damien_Mather — Fri, 09 Dec 2016 02:24:42 GMT

not sure I follow the logic behind

link=scan(line,2,'"');

do any or all links on those pages begin at position 2 and finish with double quotes?

Re: webscraping links from a web page

ilikesas — Fri, 09 Dec 2016 02:57:17 GMT

The links start and finish with quatation marks. As for the position, I thought that I had to put the number 2 because html is of the form <a ... and therefore I thought that it starts with position 2. But then I counted the position of the "h" in the "http..." and it turned out to be 8 and when I put 8 instead of 2 I did actually get the links!

But besides the links I get lines that don't contain at all the "html". Why is that so?

Thanks!

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 03:09:30 GMT

How about this one. It seems that each page contain 10 blogs in Rick's blog website.

filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/2";
data blog_links;
infile rick length=len lrecl=32767;
input line $varying32767. len;
p=find(line,'<h2 class="entry-title"><a href="http://blogs.sas.com/content/iml/');
if p then do;
 x=substr(line,p);
 link=scan(x,4,'"');
 title=substr(scan(x,6,'"'),14); 
output;
end;
keep link title;
run;

Re: webscraping links from a web page

ilikesas — Fri, 09 Dec 2016 03:16:19 GMT

Thanks Ksharp! Your code gives me only the links and the titles ina neat way!

In a logic continuation of the question, is it possible to get all the links from page 1 up to the last page?

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 03:51:36 GMT

Make a macro to replcae 1 2 3 4.........

filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/&n";

I think it is easy for you .

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 04:11:38 GMT


%macro rick(n);
filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/&n";
data blog_links_&n;
infile rick length=len lrecl=32767;
input line $varying32767. len;
p=find(line,'<h2 class="entry-title"><a href="http://blogs.sas.com/content/iml/');
if p then do;
 x=substr(line,p);
 link=scan(x,4,'"');
 title=substr(scan(x,6,'"'),14); 
output;
end;
keep link title;
run;
%mend;

data _null_;
 do n=1 to 4;
  call execute(cats('%rick(',n,')'));
 end;
run;

data rick_blogs;
 set blog_links_:;
run;

Re: webscraping links from a web page

ilikesas — Fri, 09 Dec 2016 04:19:29 GMT

Wow, thanks Ksharp for the macro! Its actually an insight to me that macro variables could be used in the "filename".

In your code you do the macro for page 1 to n. But suppsoe that you want to get all the pages but don't know how many pages there are, so is it possible to do the macro incrementally from pages 1 up to the point when a certain page doesn't exist and that's when the macro should stop?

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 04:45:09 GMT

You can make it as a very big number .

do n=1 to 400;

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 04:51:38 GMT

This would look better.


%macro rick(n);
filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/&n";
data blog_links_&n;
infile rick length=len lrecl=32767;
input line $varying32767. len;
p=find(line,'<h2 class="entry-title"><a href="http://blogs.sas.com/content/iml/');
if p then do;
 x=substr(line,p);
 link=scan(x,4,'"');
 title=htmldecode(substr(scan(x,6,'"'),14)); 
output;
end;
keep link title;
run;
%mend;

data _null_;
 do n=1 to 4;
  call execute(cats('%rick(',n,')'));
 end;
run;

data rick_blogs;
 set blog_links_:;
run;
proc print;run;

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 05:13:54 GMT

OK. Here is:


%macro rick;
%do n=78 %to 82;
filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/&n";
data blog_links_&n;
infile rick length=len lrecl=32767;
input line $varying32767. len;
p=find(line,'<h2 class="entry-title"><a href="http://blogs.sas.com/content/iml/');
if p then do;
 x=substr(line,p);
 link=scan(x,4,'"');
 title=htmldecode(substr(scan(x,6,'"'),14)); 
output;
end;
keep link title;
run;

proc sql;
create table _null_ as
 select * from  blog_links_&n;
quit;

%if &sqlobs=0 %then %ABORT;

%end;

%mend;

%rick

data rick_blogs;
 set blog_links_:;
run;
proc print;run;

Re: webscraping links from a web page

Ksharp — Fri, 09 Dec 2016 05:29:08 GMT


%macro rick;
%do n=78 %to 82;
filename rick url "http://blogs.sas.com/content/iml/author/rickwicklin/page/&n";
data blog_links_&n;
infile rick length=len lrecl=32767;
input line $varying32767. len;
p=find(line,'<h2 class="entry-title"><a href="http://blogs.sas.com/content/iml/');
if p then do;
 x=substr(line,p);
 link=scan(x,4,'"');
 title=htmldecode(substr(scan(x,6,'"'),14)); 
output;
end;
keep link title;
run;

proc sql;
create table _null_ as
 select * from  blog_links_&n;
quit;

%if &sqlobs=0 %then %return ;

%end;

%mend;

%rick

data rick_blogs;
 set blog_links_:;
run;
proc print;run;

Re: webscraping links from a web page

ilikesas — Sat, 10 Dec 2016 04:59:48 GMT

Thanks Ksharp for the solution!

I just have a question - why do you use the "htmldecode" to get the title, and why do you use the substr with position # 14 if the scan(x,6,' " ') already should (theoretically) give the title?

Thanks!

Re: webscraping links from a web page

Ksharp — Sat, 10 Dec 2016 10:43:01 GMT

In Rick's bolg title ,there are something like < .... these html source code.
I use htmldecode() to translate <: into < .

substr(,,14) is to get blog title, you want why ? check html source code, (right click --> View Page Source  in Chrome)
you will know why I code this .

Re: webscraping links from a web page

ilikesas — Sat, 10 Dec 2016 17:50:51 GMT

When I right click and select Inspect element I see the html code.

I think that I begin to understand. I tried to get the title in a similar way as the link: title = scan(x,6,' " ') --> but here I got "Permalink to TITLE" (becasue this entire string is between the 5th and 6th delimiter), and the length of the substring "Permalink to " is in fact 13, so by doing the substr from position 14 you get rid of that substring.

But then I did title = substr(scan(x,6,' " '),14) and got the title, so what does htmldecode exactly do? I searched on Google and the only thing that I found about htmldecode is that it decodes html text, for example it transforms &lt to <. So what is htmldecode doing in this case? You said that it translated <: into <, but I don't quite understand what this means. Could you please clarify to me?

Thanks!

Re: webscraping links from a web page

Ksharp — Sun, 11 Dec 2016 04:13:12 GMT

Yes. you are right. If you don't use HTMLDECODE() ,you gonna see some strange character like < .......... in blogs' title

use htmldecode() can translate them into normal character like < ......

or instead, you could try title = scan(x,9,' " ');

For example:

title="Permalink to Ahh, that&#039;s smooth! Anti-aliasing in SAS statistical graphics"

----htmldecode()--->

Ahh, that's smooth! Anti-aliasing in SAS statistical graphics

Re: webscraping links from a web page

ilikesas — Sun, 11 Dec 2016 04:47:39 GMT

OK, I see now, if I don't use htmldecode function than the ' in that's will appear as '

and the reason why htmldecode wasn't used for getting the link is because in the link symbols such as ' don't appear. But to be safe, I guess that if I want to extract text then I should always use the htldecode function.

Thanks Ksharp for your guidance!

Re: webscraping links from a web page

Ksharp — Sun, 11 Dec 2016 11:46:49 GMT

Another way is using:

 title=htmldecode(substr(scan(x,6,'"'),14)); 

----->

 title=scan(scan(x,9,'"'),1,'<>'); 


Both could get the same thing .

Re: webscraping links from a web page

ChrisHemedinger — Sun, 11 Dec 2016 16:45:20 GMT

I love this topic for so many reasons -- it's an innovative use for SAS to pull data from the web, and you all are obviously fans of one of our top blog authors, @Rick_SAS.

Aside from scraping HTML pages, there is another approach you can use: pull the XML-based RSS feed. These XML-based representations are more like regular data, and the XML libname engine can read it as such. Downside: the RSS feeds show only the more recent blogs. You could pull data from this point forward and build a list, but it might be hard to go back into the history. Here's the the DO Loop RSS: http://feeds.feedburner.com/TheDoLoop.

At SAS, we also slice and dice our blog data, but we have an advantage -- we can point SAS directly at our blog database. I've written a paper about our process here.

If you simply want the data (and aren't as invested in the journey of parsing HTML), there is one more thing you can try: just ask us! Here -- I've attached a CSV file of all of The DO Loop posts to-date combined with the tag terms. If you want the larger inventory of technical blogs (The SAS Dummy, Graphically Speaking, The Learning Post, SAS Users) -- I can easily add to it. Want this on a regular basis? I can try make that happen -- I'll have to think about the best way to automate it...

Re: webscraping links from a web page

ilikesas — Sun, 11 Dec 2016 22:02:53 GMT

Hi Chris,

It is in fact very interesting to me to know how to parse data from the internet. SAS is great at analyzing data, but you have to have the data in the first place, and the internet seems like an untapped source!

As for the blogs, it would be great if there were some sort of a table of contents of the authors' blogs like this newcomers can see what was posted before, and I personally found (and still keep searching for) a lot of valuable information!

Maybe you could even post Ksharp's code on the blog in order to let readers learn about webscraping and exploring the blog contents?

Thanks!