topic Re: Parsing HTML to return JSON file from list in SAS Programming

Parsing HTML to return JSON file from list

A_SAS_Man — Fri, 22 Jul 2022 16:00:45 GMT

I am attempting to access publicly available data files stored in JSON files which are stored in a list on a webpage (https://transparency-in-coverage.uhc.com/undefined). I am first getting the html for the page this way:

filename source temp;
proc http
	URL="https://transparency-in-coverage.uhc.com/undefined"
	out=source
	method="Get";
run;

After I do this I am trying to return all the JSON file names so that I can then iterate through them with a macro. But after inspecting the HTML returned from proc http, it appears to be only one line of html. it does not appear to include all of the available code I can see if I'm using google chrome developer tools to inspect the web pages html directly. This makes it impossible to correctly parse the text from the page to find the files. Is there something I'm doing wrong or something different about how this site is set up that doesn't allow the full html to be returned?

Re: Parsing HTML to return JSON file from list

ChrisHemedinger — Fri, 22 Jul 2022 17:00:44 GMT

You're not doing anything wrong, but this data is not part of the source in this page. It looks like the page calls a service on a cloud-hosted database to bring in the list of files. Unless you can find another way to get to the same list, I don't think you'll be able to dynamically load/process from this page.

I can see the action by looking at the network tab of the browser developer tools. But no simple way to simulate the same call using PROC HTTP without inside knowledge or a trusted connection.

Re: Parsing HTML to return JSON file from list

ballardw — Fri, 22 Jul 2022 17:09:31 GMT

I find this message from the URL when loading in my brower of interest:

PLEASE NOTE: Due to the number and size of files being posted it can take up to 5 minutes for the page to load.

Re: Parsing HTML to return JSON file from list

A_SAS_Man — Fri, 22 Jul 2022 17:28:40 GMT

Thank you for taking a look. I don't have a ton of experience with html so maybe this question is silly. But why am I able to navigate to the list html in developer tools and see the file names but not return it via proc http? Maybe that is what you were getting at with your last response but I didn't quite follow.

Re: Parsing HTML to return JSON file from list

A_SAS_Man — Fri, 22 Jul 2022 17:28:06 GMT

It does sometimes take a minute to load but should eventually.

Re: Parsing HTML to return JSON file from list

ChrisHemedinger — Fri, 22 Jul 2022 17:57:41 GMT

When loading the page in the browser, the site sends calls this external service and asynchronously populates the list and updates the DOM, or the HTML. As it updates the HTML the browser re-renders for you.

When using a console or command method like PROC HTTP or cURL, you don't get that async-populate experience. So, no list.