topic Re: Searching a website for key words using SAS in SAS Data Science

Searching a website for key words using SAS

kevin12 — Tue, 25 Oct 2016 20:16:16 GMT

Good afternoon, I am task to write a SAS program to search a website and the links within the website for key words and phases. I wrote the program but it is only searching the webpage and not the entire website. I would also want it to spit out the link that they find the key words. I have gotten this:

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";

data LAW_achive;

*length chapter $200;

infile LAW length=len lrecl=32767;

input line $varying32767. len;

uline= upcase(line);

if find(uline,"chapter") and find(uline,"LAW") or find(uline,"SCHOOL")or find(uline,"PUBLIC")

or find(uline,"BOARD")or find(uline,"EDUCATION")or find(uline,"AUTHORITY")or find(uline,"MEMBER")

then do;

chapter=scan(line,2,'"');

output;

end;

run;

Is it possible to help me cleaning up my program and help me with the website search part. Please look at the link to see the why it woulf be beneficial to search not only the webpage but the entire website making the first the starting point.

Re: Searching a website for key words using SAS

Ksharp — Wed, 26 Oct 2016 05:04:30 GMT

HoHo. That is really not easy. Firstly get those URL ,after that call execute them.

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";
data have(keep=xx) url(keep=url);
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output have;
 if prxmatch('/CHAPTER\s+\d+/i',x) then do;
  temp=scan(x,2,'"');
  url=cats('http://delcode.delaware.gov/sessionlaws/ga148/',temp);
  if not missing(temp) then output url;
 end;
end;
run;


%macro chapter(url=);
filename LAW url "&url";
data %scan(&url,-2,%str(/.))(keep=xx) ;
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output;
end;
run;
%mend;

data _null_;
 set url;
 call execute(cats('%chapter(url=',url,')'));
run;

Re: Searching a website for key words using SAS

Ksharp — Wed, 26 Oct 2016 05:15:26 GMT

I just realize you need add the following into code to avoid truncate problem.

length xx url $ 20000;