Data scraping - java script form has me stuck

[BigD]

Hello,

I've been using SAS to scrape a lot of "easy" web sites. Now I have one where I have to submit a location (province) into a form then read in the information. To further complicate matters, the return results are listed say 20 at a time, so I need to span the result pages. The proc http documentations doesn't really help. Anyone have a basic working example I can build on.

Thanks,

Bruce

[mich1]

​Here is an example of a complex API transaction using SAS and PROC HTTP

/*POST A REQUEST FOR SURVEY DATA*/

/****************************************************************************/

/* Qualtrics API Request to get export of survey response data

/* To compare to Python go to https://api.qualtrics.com/docs/response-exports

/****************************************************************************/

filename rsp temp;  

proc http

url="https://yourdatacenter.qualtrics.com/API/v3/responseexports/"

method= "POST"

in='{"format" : "csv","surveyId" : "yoursurveyID"}'

out=rsp;

headers

"X-API-Token"="yourAPItoken"

"Content-Type"="application/json"

;

run; 

libname RSP JSON fileref=rsp;

/*Check Status of request*/

filename dwnld temp;

proc http

url="https://yourdatacenter.qualtrics.com/API/v3/responseexports/yourresultID" /*Check json from rsp for result*/

method= "GET"

out=dwnld;

headers 

"X-API-Token"="yourAPItoken";

run;

libname dwnld JSON fileref=dwnld;

/*Download request - You can view json dwnld for status and filename (same as yourresultID above)*/

filename dwnld '/yourPathHere/GOTDATA';

proc http

url="https://yourdatacenter.qualtrics.com/API/v3/responseexports/yourresultID/file"

method= "GET"

out=dwnld;

headers 

"X-API-Token"="yourAPItoken";

run;