Building models with SAS Enterprise Miner, SAS Factory Miner, SAS Visual Data Mining and Machine Learning or just with programming

Searching a website for key words using SAS

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 5
Accepted Solution

Searching a website for key words using SAS

Good afternoon, I am task to write a SAS program to search a website and the links within the website for key words and phases. I wrote the program but it is only searching the webpage and not the entire website. I would also want it to spit out the link that they find the key words. I have gotten this:

 

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";

data LAW_achive;

*length chapter $200;

infile LAW length=len lrecl=32767;

input line $varying32767. len;

uline= upcase(line);

if find(uline,"chapter") and find(uline,"LAW") or find(uline,"SCHOOL")or find(uline,"PUBLIC")

or find(uline,"BOARD")or find(uline,"EDUCATION")or find(uline,"AUTHORITY")or find(uline,"MEMBER")

then do;

chapter=scan(line,2,'"');

output;

end;

run;

 

Is it possible to help me cleaning up my program and help me with the website search part. Please look at the link to see the why it woulf be beneficial to search not only the webpage but the entire website making the first the starting point.


Accepted Solutions
Solution
‎10-26-2016 08:11 AM
Super User
Posts: 10,020

Re: Searching a website for key words using SAS

HoHo. That is really not easy. Firstly get those URL ,after that call execute them.

 

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";
data have(keep=xx) url(keep=url);
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output have;
 if prxmatch('/CHAPTER\s+\d+/i',x) then do;
  temp=scan(x,2,'"');
  url=cats('http://delcode.delaware.gov/sessionlaws/ga148/',temp);
  if not missing(temp) then output url;
 end;
end;
run;


%macro chapter(url=);
filename LAW url "&url";
data %scan(&url,-2,%str(/.))(keep=xx) ;
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output;
end;
run;
%mend;

data _null_;
 set url;
 call execute(cats('%chapter(url=',url,')'));
run;

View solution in original post


All Replies
Solution
‎10-26-2016 08:11 AM
Super User
Posts: 10,020

Re: Searching a website for key words using SAS

HoHo. That is really not easy. Firstly get those URL ,after that call execute them.

 

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";
data have(keep=xx) url(keep=url);
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output have;
 if prxmatch('/CHAPTER\s+\d+/i',x) then do;
  temp=scan(x,2,'"');
  url=cats('http://delcode.delaware.gov/sessionlaws/ga148/',temp);
  if not missing(temp) then output url;
 end;
end;
run;


%macro chapter(url=);
filename LAW url "&url";
data %scan(&url,-2,%str(/.))(keep=xx) ;
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output;
end;
run;
%mend;

data _null_;
 set url;
 call execute(cats('%chapter(url=',url,')'));
run;
Super User
Posts: 10,020

Re: Searching a website for key words using SAS

I just realize you need add the following into code  to avoid truncate problem.

 

length xx url $ 20000;
☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 2 replies
  • 378 views
  • 2 likes
  • 2 in conversation