topic Re: web scraping in SAS Programming

web scraping

jojozheng — Thu, 20 Sep 2018 17:53:19 GMT

Hi everyone,

I need to scrap data from this website https://www.addresses.com/zipcodes

I need to scrap all US states zip codes, I don't want to copy&past 56 times by manual. Anyone knows how to to do it? Thank you.

I want all cloumns in following page, but I don't want to manually do it. If I do it manually, I need to click states to search over 50 times...

Re: web scraping

danielduval0 — Thu, 20 Sep 2018 19:35:26 GMT

Unfortunately, SAS will not be your best tool for web scraping.

I recommend using something like the Scraper Chrome add-on : https://chrome.google.com/webstore/detail/scraper/mbigbapnjcgaffohmbkdlecaccepngjd?hl=en

This allows you to highlight any table on a webpage, right-click and hit "scrape", and you can then export the table as a CSV file. You can then import that CSV file into SAS and begin analyzing the data.

You can also pip install the BeautifulSoup4 package in Python if you would like to automatically web scrape certain pages, however it can sometimes be extremely tedious and if you are simply trying to scrape some data and perform some analyses on it, I recommend using something like the Scraper Chrome add-on.

SAS unfortunately is not the best tool to web scrape data. For example, the following code scrapes only the zip code portion of the table you wanted to scrape from https://www.addresses.com/zip-codes-by-state/alabama :

filename HTMLtext "C:\Users\daduva\Desktop\WebScrape\HTML_Output.txt";


proc http
	method='GET'
	url="https://www.addresses.com/zip-codes-by-state/alabama"
	out=HTMLtext;
run;

data zipcodes;
infile "C:\Users\daduva\Desktop\WebScrape\HTML_Output.txt" TRUNCOVER dlm='"';
input @'<a href="/zip-code-lookup/' zipcode :$30. @;
run;

proc sort data=zipcodes NODUP;
by zipcode;
run;

I hope this post was helpful.

Re: web scraping

JasonDiVirgilio — Thu, 20 Sep 2018 19:42:13 GMT

Maybe scraping a website is not what you want to do...have you tried the Zipcode dataset?

proc print data=sashelp.zipcode;
where statecode='AL';
run;

...or other SAS-provided resources https://support.sas.com/downloads/download.htm?did=122546

Re: web scraping

jojozheng — Thu, 20 Sep 2018 22:37:51 GMT

Thank you! I tried the code, it seems like it can catch a part of data, but not all of them. But thank you so much for your help!

Re: web scraping

jojozheng — Thu, 20 Sep 2018 22:43:53 GMT

Thank you for this useful tip! It seems like SASHELP is a useful folder, not just sample data.