Help using Base SAS procedures

accessing files on the internet

Accepted Solution Solved
Reply
Frequent Contributor
Posts: 77
Accepted Solution

accessing files on the internet

I have a series of html files I need to parse they are of the form: http://shakespeare.mit.edu//full.html where each directory has a html file called full.html ....how can I iterate through and read these files?


Accepted Solutions
Solution
‎02-17-2014 10:57 PM
SAS Super FREQ
Posts: 8,743

Re: accessing files on the internet

Hi:

  I get a 404 Page Not Found when I try your URL. I think it's wrong anyway, because I don't think a URL can have double // in anyplace except after the http://, but even if I try this: http://shakespeare.mit.edu/full.html it doesn't work.

  However, if I go down to the page for All's Well that End's Well at this URL (http://shakespeare.mit.edu/allswell/allswell.1.1.html), I can read the HTML file with the URL engine, however, what comes back is just the HTML on the page (all the tags and text), as you can see in the screen shot of my output window. It would be up to you to write a program to parse the tags, if, for example, you wanted to identify the speeches or make a list of characters.

  The index.html file for the site, has the list of files that you can link to, but you would have to get to each of these directories to further grab the HTML on each site. This is shown below in the section of code that reads the INDEX.HTML file for the ALLSWELL subdirectory on the site.

  I suppose you could automate everything and iterate with a SAS Macro program, but you first have to get everything working for 1 play before you can automate this for all the plays. And I'm not sure the point of just READING the full HTML of a Shakespeare play with SAS. What is the end result you are looking for??

Cynthia

** go to main index.html page;
filename main url 'http://shakespeare.mit.edu/index.html'
         url debug;
     
ods _all_ close;
title 'INDEX.HTML for Shakespeare site';
ods listing;
   data _null_;
     infile main;
     file print;
     input;
     put _infile_;
   run;
        
** read index.html to find how the plays are organized;
** every play has a separate index.html file in a separate directory;
filename alltop url
   'http://shakespeare.mit.edu/allswell/index.html'
  debug;
       
title 'INDEX.HTML for Allswell play';
ods listing;
   data _null_;
     infile alltop;
     file print;
     input;
     put _infile_;
   run;

    

** read the first section of the Alls Well that Ends Well site;
filename allswell url
   'http://shakespeare.mit.edu/allswell/allswell.1.1.html'
  debug;
       
title 'Alls Well that Ends Well section 1.1'; 
ods _all_ close;
ods listing;
   data _null_;
     infile allswell;
     file print;
     input;
     put _infile_;
   run;

** or read the FULL play;

filename allsfull url

   'http://shakespeare.mit.edu/allswell/full.html'

  debug;

    

title 'Alls Well that Ends Well full play '; 

ods _all_ close;

ods listing;

   data _null_;

     infile allsfull;

     file print;

     input;

     put _infile_;

   run;

View solution in original post


All Replies
Super User
Posts: 17,819

Re: accessing files on the internet

Bad link, but you can look into PROC HTTP for starters.

Frequent Contributor
Posts: 77

Re: accessing files on the internet

Solution
‎02-17-2014 10:57 PM
SAS Super FREQ
Posts: 8,743

Re: accessing files on the internet

Hi:

  I get a 404 Page Not Found when I try your URL. I think it's wrong anyway, because I don't think a URL can have double // in anyplace except after the http://, but even if I try this: http://shakespeare.mit.edu/full.html it doesn't work.

  However, if I go down to the page for All's Well that End's Well at this URL (http://shakespeare.mit.edu/allswell/allswell.1.1.html), I can read the HTML file with the URL engine, however, what comes back is just the HTML on the page (all the tags and text), as you can see in the screen shot of my output window. It would be up to you to write a program to parse the tags, if, for example, you wanted to identify the speeches or make a list of characters.

  The index.html file for the site, has the list of files that you can link to, but you would have to get to each of these directories to further grab the HTML on each site. This is shown below in the section of code that reads the INDEX.HTML file for the ALLSWELL subdirectory on the site.

  I suppose you could automate everything and iterate with a SAS Macro program, but you first have to get everything working for 1 play before you can automate this for all the plays. And I'm not sure the point of just READING the full HTML of a Shakespeare play with SAS. What is the end result you are looking for??

Cynthia

** go to main index.html page;
filename main url 'http://shakespeare.mit.edu/index.html'
         url debug;
     
ods _all_ close;
title 'INDEX.HTML for Shakespeare site';
ods listing;
   data _null_;
     infile main;
     file print;
     input;
     put _infile_;
   run;
        
** read index.html to find how the plays are organized;
** every play has a separate index.html file in a separate directory;
filename alltop url
   'http://shakespeare.mit.edu/allswell/index.html'
  debug;
       
title 'INDEX.HTML for Allswell play';
ods listing;
   data _null_;
     infile alltop;
     file print;
     input;
     put _infile_;
   run;

    

** read the first section of the Alls Well that Ends Well site;
filename allswell url
   'http://shakespeare.mit.edu/allswell/allswell.1.1.html'
  debug;
       
title 'Alls Well that Ends Well section 1.1'; 
ods _all_ close;
ods listing;
   data _null_;
     infile allswell;
     file print;
     input;
     put _infile_;
   run;

** or read the FULL play;

filename allsfull url

   'http://shakespeare.mit.edu/allswell/full.html'

  debug;

    

title 'Alls Well that Ends Well full play '; 

ods _all_ close;

ods listing;

   data _null_;

     infile allsfull;

     file print;

     input;

     put _infile_;

   run;

Frequent Contributor
Posts: 77

Re: accessing files on the internet

hey cool thanks

Frequent Contributor
Posts: 77

Re: accessing files on the internet

one other thing how would I check each line looking for "index.html"

line in this line

<a href="allswell/index.html">


identify that it has index.html in it then strip out


allswell/index.html  so that i can use that to build


'http://shakespeare.mit.edu/allswell/full.html'





☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 303 views
  • 0 likes
  • 3 in conversation