Scraping dynamic forms from ASP.net applications is difficult and SAS does not really have a lot of the tools you need. Also eventually when the sites you are scraping replace their encryption keys nothing will work anymore. With the large number of scrapes you are trying to accomplish I would recommend using an outside tool built specifically for what you are trying to do. If you did want to do this in SAS what you need to do is decode what exactly the javascript call __doPostBack is doing. It is probably performing a new post. Hopefully it still uses the same viewstate and evenvalidation pieces. The process would be to make a call to the initial search results. Gather your data, and check for a javascript link to a subsequent page. If it exists make a new call to the subsequent datagrid location, and so on, in a loop. Best of luck, if I were you I would move outside of SAS to perform these heavy scraping tasks (which you should probably confirm the legality of).
... View more
