Solved: Searching a website for key words using SAS

kevin12 · Posted 10-25-2016 04:16 PM

Good afternoon, I am task to write a SAS program to search a website and the links within the website for key words and phases. I wrote the program but it is only searching the webpage and not the entire website. I would also want it to spit out the link that they find the key words. I have gotten this:

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";

data LAW_achive;

*length chapter $200;

infile LAW length=len lrecl=32767;

input line $varying32767. len;

uline= upcase(line);

if find(uline,"chapter") and find(uline,"LAW") or find(uline,"SCHOOL")or find(uline,"PUBLIC")

or find(uline,"BOARD")or find(uline,"EDUCATION")or find(uline,"AUTHORITY")or find(uline,"MEMBER")

then do;

chapter=scan(line,2,'"');

output;

end;

run;

Is it possible to help me cleaning up my program and help me with the website search part. Please look at the link to see the why it woulf be beneficial to search not only the webpage but the entire website making the first the starting point.

Ksharp · Posted 10-26-2016 01:04 AM

HoHo. That is really not easy. Firstly get those URL ,after that call execute them.

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";
data have(keep=xx) url(keep=url);
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output have;
 if prxmatch('/CHAPTER\s+\d+/i',x) then do;
  temp=scan(x,2,'"');
  url=cats('http://delcode.delaware.gov/sessionlaws/ga148/',temp);
  if not missing(temp) then output url;
 end;
end;
run;


%macro chapter(url=);
filename LAW url "&url";
data %scan(&url,-2,%str(/.))(keep=xx) ;
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output;
end;
run;
%mend;

data _null_;
 set url;
 call execute(cats('%chapter(url=',url,')'));
run;

View solution in original post

Ksharp · Posted 10-26-2016 01:04 AM

HoHo. That is really not easy. Firstly get those URL ,after that call execute them.

filename LAW url "http://delcode.delaware.gov/sessionlaws/ga148/index.shtml";
data have(keep=xx) url(keep=url);
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output have;
 if prxmatch('/CHAPTER\s+\d+/i',x) then do;
  temp=scan(x,2,'"');
  url=cats('http://delcode.delaware.gov/sessionlaws/ga148/',temp);
  if not missing(temp) then output url;
 end;
end;
run;


%macro chapter(url=);
filename LAW url "&url";
data %scan(&url,-2,%str(/.))(keep=xx) ;
infile law length=len lrecl=32767;
input  x $varying32767. len;
retain flag;
if x =: '<body>' then flag=1;
if flag then do;
 xx=prxchange('s/\<[^\<\>]+\>//',-1,x);
 if not prxmatch('/^\s+$/',xx) then output;
end;
run;
%mend;

data _null_;
 set url;
 call execute(cats('%chapter(url=',url,')'));
run;

Ksharp · Posted 10-26-2016 01:15 AM

I just realize you need add the following into code to avoid truncate problem.

length xx url $ 20000;

Searching a website for key words using SAS

Re: Searching a website for key words using SAS

Re: Searching a website for key words using SAS

Re: Searching a website for key words using SAS