BookmarkSubscribeRSS Feed
A_SAS_Man
Pyrite | Level 9

I am attempting to access publicly available data files stored in JSON files which are stored in a list on a webpage (https://transparency-in-coverage.uhc.com/undefined). I am first getting the html for the page this way:

 

filename source temp;
proc http
	URL="https://transparency-in-coverage.uhc.com/undefined"
	out=source
	method="Get";
run;

After I do this I am trying to return all the JSON file names so that I can then iterate through them with a macro. But after inspecting the HTML returned from proc http, it appears to be only one line of html. it does not appear to include all of the available code I can see if I'm using google chrome developer tools to inspect the web pages html directly. This makes it impossible to correctly parse the text from the page to find the files. Is there something I'm doing wrong or something different about how this site is set up that doesn't allow the full html to be returned?

5 REPLIES 5
ChrisHemedinger
Community Manager

You're not doing anything wrong, but this data is not part of the source in this page. It looks like the page calls a service on a cloud-hosted database to bring in the list of files. Unless you can find another way to get to the same list, I don't think you'll be able to dynamically load/process from this page.

 

I can see the action by looking at the network tab of the browser developer tools. But no simple way to simulate the same call using PROC HTTP without inside knowledge or a trusted connection.

Learn from the Experts! Check out the huge catalog of free sessions in the Ask the Expert webinar series.
A_SAS_Man
Pyrite | Level 9

Thank you for taking a look. I don't have a ton of experience with html so maybe this question is silly. But why am I able to navigate to the list html in developer tools and see the file names but not return it via proc http? Maybe that is what you were getting at with your last response but I didn't quite follow.

ChrisHemedinger
Community Manager

When loading the page in the browser, the site sends calls this external service and asynchronously populates the list and updates the DOM, or the HTML. As it updates the HTML the browser re-renders for you.

 

When using a console or command method like PROC HTTP or cURL, you don't get that async-populate experience. So, no list.

Learn from the Experts! Check out the huge catalog of free sessions in the Ask the Expert webinar series.
ballardw
Super User

I find this message from the URL when loading in my brower of interest:

PLEASE NOTE: Due to the number and size of files being posted it can take up to 5 minutes for the page to load.

 

 

A_SAS_Man
Pyrite | Level 9
It does sometimes take a minute to load but should eventually.

SAS Innovate 2025: Save the Date

 SAS Innovate 2025 is scheduled for May 6-9 in Orlando, FL. Sign up to be first to learn about the agenda and registration!

Save the date!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

SAS Training: Just a Click Away

 Ready to level-up your skills? Choose your own adventure.

Browse our catalog!

Discussion stats
  • 5 replies
  • 798 views
  • 2 likes
  • 3 in conversation