topic Re: web scraping in SAS Programming

web scraping

Mokalake — Fri, 10 May 2019 09:27:10 GMT

Good day,

Is there a way in which i can web scrap with sas enterprise guide?

Here is what i need to achieve " suppose i have a column field with word phrases that i want to pass to google chrome and maybe extract the contents of their home page. E.G i have the word SAS and i want the contents of the ABOUT US from the first link"

Thank you.

Re: web scraping

Ksharp — Fri, 10 May 2019 13:28:45 GMT

Maybe you need help from the third side software cURL .

Re: web scraping

ybolduc — Fri, 10 May 2019 13:37:39 GMT

Hi Mokalake,

SAS may not be the best tool to do Web Scraping the way you want to do it. But if you really way do this with SAS, you will need to use PROC HTTP to download the page. Once you have the page downloaded, the best would be to use Regular Expressions to find the information that you are looking for. You will likely need to use SAS Macros to wrap your logic and make iterate properly.

Note that both PROC HTTP and Regular Expressions are fairly advanced skills in SAS.

Thanks,

Re: web scraping

Reeza — Fri, 10 May 2019 15:20:28 GMT

You can and it's been done and I've done some of this. You can search on lexjansen.com for some full papers on the topic or Chris Hemidinger has a blog post on it as well. There are some posts on here that also asked this question and have solved answers that may help you. I'd highly recommend starting there.

https://blogs.sas.com/content/sasdummy/2017/12/04/scrape-web-page-data/

BUT...it's possible your system may be locked down and may not let you do some of these things. EG can often be locked down pretty hard.

Re: web scraping

Mokalake — Mon, 13 May 2019 07:22:05 GMT

Thank you. I will have a look at it.