Watch this Ask the Expert session on using SAS® to access data and APIs from the web.
Watch the webinar
Join SAS expert Chris Hemedinger as he demonstrates how to use specialized statements and techniques within SAS to pull data from live internet sources using your SAS programs with no additional process. No need for a separate script in Python or cURL – SAS can do it all. Access data from any API, including services like Google Analytics, GitHub, Microsoft365 and more. During this webinar, you will learn:
- Ways to use PROC HTTP as a web client to download data or use REST APIs.
- How to process API results of any type in SAS: JSON, ZIP files and more.
- To test APIs outside of SAS and debug your SAS programs that use APIs.
We're working on compiling the questions from the Q&A segment -- watch this space! The slides from the webinar are attached.
Recommended Resources
How to translate your cURL command into SAS code
How to test PROC HTTP and the JSON library engine
How to scrape data from a web page using SAS
Using SAS to access Microsoft 365
The ABCs of PROC HTTP
SAS Programming Certification Fast Track
Q&A from the webinar
I use APIs that require credentials, but I don't want to store these in my program files. How can I manage this?
My practice is to store the credentials in files that are local to my process and readable only to my account. They are not in my SAS code itself. This is an important security step. Read this article for specific techniques.
Some of the APIs that I use won't return all of the data I need in a single call. Can I use SAS code to "page" through the data to retrieve all values?
Yes. Usually I use SAS macro code to loop through pages or I use CALL EXECUTE for a data-driven approach. Some API paging is through URL parameters ("...&page=1" on the URL, for example) and some is more dynamic, where an API response will include a new URL to use to get the next "page" of results. For the latter, you would check for this when examining the JSON data, and then loop through to call for the next page of data.
My company uses a proxy gateway (firewall) to control internet traffic. Can I still use PROC HTTP to access data from the Web?
PROC HTTP has a collection of PROXY* options to specify a proxy host, port, and credentials if you need to get through a firewall/gateway.
How do you connect SAP data to SAS?
The best way is SAS/ACCESS to SAP HANA (or R/3) to provide a natural approach for accessing SAP data. However, SAP also offers APIs that you could use -- the process for getting credentials/authentication is probably similar to other APIs. Some SAP APIs use the SOAP protocol, and SAS has PROC SOAP that you could use for that.
We are pulling data from a private company which we are paying to get their data. Does PROC HTTP support Asynchronous API calls?
No. A PROC HTTP step will "block" until the job finishes. Some services have a protocol to begin a request and then poll for when it completes. In this case, you would capture the token provided when you initiate the request, and then loop (perhaps with wait/sleep steps) to poll for when the operation is complete, then fetch the data.
Is SAS Studio in SAS Viya capable of async programming?
By "async" I assume you mean can we use SAS code to call a service and then react to an event/signal when the service call is complete. SAS code doesn't support that natively. But you can use SAS Viya as a player in an integrated app that uses SAS with other services to provide a responsive experience. The SAS Job Execution Service or Job Execution Web App can be used to create async-style experiences for end users.
How do you connect historian data set to SAS?
Do you mean from a data historian like OSI/Pi? SAS Event Stream Processing provides a number of connectors to data services. These will be the most robust method for bringing this data to your SAS analytics.
Are there options to pull down data that is in a password-protected location (e.g., S&P data)?
Yes, PROC HTTP supports WEBUSERNAME and WEBUSERPASSWORD to allow user/password authentication to web services.
Will the SAS encrypted passwords work on external sites?
Yes. SAS will manage that part for you if you encode with PROC PWENCODE. SAS will decode and send the password "across the wire" in any unencoded form so be aware of that. Best to use HTTPS and pass credentials in the body of the message, not the URL.
HTTPBIN is in clear text not https?
See https://httpbin.org/ for information about this service. It supports HTTPS.
Should I be seeing the HTTPBIN output?
This depends on the methods you use -- see https://httpbin.org/ for details about what's available.
If you use a global.sas file to store usernames and passwords, and reference them using a %global statement, instead of typing in actual usernames and passwords into the code, will PROC HTTP resolve the macros?
Yes as long as they are in double quotes. My practice is to store the credentials in files that are local to my process and readable only to my account. They are not in my SAS code itself. This is an important security step. Read this article for specific techniques.
Can PROC HTTP download file from SharePoint?
Yes. Answering in the context of Microsoft 365. Microsoft has an API called Microsoft Graph API. After you go through the process of getting an authorization code and access token, you can access those sources that you need to access in SharePoint, Teams, OneDrive, etc. See blog post and video in SAS Support Communities.
Is there a way to list the contents of a zip file; from the code it looks like the user would need to know the name of txt file within the zip file?
You can use FILENAME ZIP to list the contents of a zip file as well as extract the content out. If it’s a text file, you can read it in directly. If it’s a binary file, we would copy that file out of the zip file and move it somewhere else and operate it on it from there. You can also use it to create zip files. See examples in my blog: https://blogs.sas.com/content/sasdummy/2017/09/08/filename-zip-details/
Is there any reference to what our administrators have to do to open up the ability to use the http method?
PROC HTTP is part of Base SAS and isn't restricted. However, you might have network limitations for calling out to internet services from your SAS environment. See the notes about "proxy gateway" above.
When posting, is there an issue with sites with reCaptcha?
If Captcha is implemented, you won’t be able to bypass it using PROC HTTP. (It's checking that you "aren't a robot" and PROC HTTP is, well, sort of a robot.) Be looking for ways that you can negotiate with the website that don’t rely on Captcha. Sometimes it may require authentication to bypass that check.
Is it possible to GET/POST to SAS Webservers? For example, I want to run VA Report distribute job using PROC HTTP?
Yes. SAS Visual Analytics has a robust list of APIs that are supported. You can use these techniques to call SAS APIs as well. Authentication becomes easier if running the code in the same SAS Viya environment that hosts the reports. Some examples on developer.sas.com. If you have some particular you’re trying to do, please post to the developer community and others can comment.
If we have a login page, and then a few pages in between the file location that have selections (i.e. do you agree to this, click a link before you start downloading the files), how would that work?
Some sites (like my example with the CMS data) provide a nice web page to allow to make selections, and then eventually lead you to a URL you can use as an endpoint. Capture that URL once and then you can use it in the future. However, other sites use cookies and tokens to enforce your login/agreement within a session, and it's more difficult to automate that.
Does the JSON libname statement work on ANY website with a JSON response?
Yes, if the JSON is a well-formed JSON. Just a caution that not all JSON responses generate nice, neat tables.
Can you connect to social media APIs?
Yes, in general. However, the process for authenticating/registering as an app can be complex, and using an automated tool might not be within the Terms and Conditions of the social media service. Most prominent social media services have resources for developers that you can use to get information on API calls.
Can we read R files?
SAS provides several ways to integrate with R. SAS/IML is one method that allows you to read R data and use R packages.
How do you get the connect requirements for an API, e.g., the syntax to connect to the API?
These will vary on the API. Some APIs provide thorough documentation with examples in multiple technologies. While most don't address SAS, cURL is often used. It's simple to translate cURL examples to PROC HTTP.
Can you use PROC HTTP with services which use Single Sign-On authentication?
Not directly, no. You would need to use an authentication method that's different from the SSO approach (such as using an account specifically for API access). Some services allow you to generate an API or webhook token using your SSO account, and then you can apply that for API use. (Example: GitHub allows you to create an personal access token for use with the GitHub API.)
Can you speak to how you use CURL: What it is and how you use it for API code development?
See this article about cURL and PROC HTTP.
How does one best handle errors like HTTP return codes? Eg. Does PROC HTTP set & syscc or do I have to parse the response headers?
PROC HTTP sets two automatic macro variables for you: SYS_PROCHTTP_STATUS_CODE and SYS_PROCHTTP_STATUS_PHRASE. See this topic for examples of how to use.
We have survey data collected in an online application, Qualtrics. What info is needed to pull the data from Qualtrics into SAS? Not sure where to start.
I'm not familiar with the Qualtrics API, but this has been discussed on the community.
Is PROC HTTP supported on SAS base 9.4? Could VPNs affect its ability to work?
Yes, PROC HTTP was introduced in SAS 9.2 and continues to add new features through the latest version of SAS Viya. VPN affects the network traffic from your local machine to the internet, usually via your corporate network. It can affect PROC HTTP if you're using it from a local machine. But keep in mind PROC HTTP is a communication from your SAS session to whatever service you're calling -- so if SAS is on your corporate network, VPN wouldn't play a role.
Great presentation, this question is probably out of scope for this presentation, and if so, no worries. Looking for a way to screen capture a part of a webpage and then email or save as a file.
Not with PROC HTTP -- you need a tool that emulates/automates a web browser to render a page. There are some testing automation tools that do this.
Sometimes a JSON response has several levels within a field. For example, a field could be labeled "person" and within that field you'll find name, age, etc. Does the JSON handle those situations well?
Yes, the JSON libname engine will create multiple tables and relationships to reflect nested structures within the JSON response. The engine will create "ordinal" keys that allow you to remerge the data into the structures you need.
Can we use PROC HTTP for graphql?
Yes. Graphsql is a style of API where you form a query in JSON with the information you seek, and the response contains the data in the template you supply. You can then use the JSON libname engine to sift through the data, but the response is usually not a clean relational table. So it requires a little bit more manipulation with DATA step to pull the values you need. I've used this approach with the GitHub API and with Google's Pagespeed Insights API.
Does SAS support key pair (private/public) authentication or kerberos authentication than username/password?
Kerberos or NTLM can be used via the AUTH_NEGOTIATE option. SSH keys are supported for some SAS features like Git integration and Secure FTP (SFTP), but not for HTTP (as far as I'm aware).
Can you comment on Google Drive?
Google Drive can be accessed using Google APIs. I don't have a specific example for this but the process is similar to that for Google Analytics. For document sharing and reading publicly shared documents, you can use PROC HTTP with the download directive on the sharing link. Example:
filename _data "%sysfunc(getoption(work))/streaming.sas7bdat";
proc http
out=_data
url="https://drive.google.com/u/0/uc?id=0BwSh7LOTCPQ5MmlJZFdOOFJhOHc%str(&)export=download";
run;
proc datasets lib=work;
contents data=streaming;
quit;
Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.
Click image to register for webinar