topic Re: Cannot load more than 43 observations with proc http using sas ondemand. in SAS Studio

Cannot load more than 43 observations with proc http using sas ondemand.

jhirschb — Sat, 08 Apr 2023 04:16:28 GMT

The code below only loads 43 lines of the web-site data to scrape.

filename output temp;  * Setup a scratch file ; 

%Let sub = Airport%20West;

	proc http url="http://reiv.com.au/market-insights/suburb/&sub" method="get"
		out=output;
	run;
*
	Create a data set called first with one variable for the contents of the 
	webpage called value
;
	data first;
		length suburb $ 20;
		suburb = "&sub";
		suburb = tranwrd(suburb, "%20", " ");
		length value $30000;
		infile output _infile_ = line;
		input @1 value $30000.;
		linenum = _n_;
		output;
	label 
		value = webpage code 
		suburb = "Name of suburb with %20 replaced by space"
		linenum = Line number of code;
	run;

How do I get it to set aside sufficient space for up to 1000 lines of code from this website?

Thanks, Joe

Re: Cannot load more than 43 observations with proc http using sas ondemand.

Kurt_Bremser — Sat, 08 Apr 2023 11:58:19 GMT

You get the whole HTML content (the last observation contains the </html> tag), and it seems what you get is a script supposed to actually fetch the data, something I don't know if it's doable with PROC HTTP.

Re: Cannot load more than 43 observations with proc http using sas ondemand.

Kurt_Bremser — Sat, 08 Apr 2023 12:13:52 GMT

After trying the URL in Safari, you first get one of those "I am not a robot" pages, and after verification are routed to the actual data.

You should get in touch with the operators of the website, if they provide a dedicated download method for their data.

Re: Cannot load more than 43 observations with proc http using sas ondemand.

jhirschb — Sun, 09 Apr 2023 00:37:30 GMT

This is not the case when using SAS on machines that have it installed. Do you figure it is a location problem since our local machines are on a vpn located in Melbourne?
Where does the SAS ondemand login from?

Thanks, Joe

Re: Cannot load more than 43 observations with proc http using sas ondemand.

Kurt_Bremser — Sun, 09 Apr 2023 07:03:01 GMT

@jhirschb wrote:
This is not the case when using SAS on machines that have it installed.

Once you visited the page "manually" from such a computer, the "I am not a robot" script is circumvented because the necessary cookie is present.

Visit the page from your PC, save the HTML, then upload it to On Demand, so you can play around with it there.