topic Re: Download xml files from multipe urls in SAS Programming

Download xml files from multipe urls

92568466 — Thu, 17 Mar 2022 23:29:15 GMT

I have list of urls of xmls that I need to download. I've the urls in csv (but can covert to any). I'm wondering there's a better way to download all xmls at once from all urls than doing it one by one. I'm currently using the following program to download one by one.

filename out "c:\temp\want.xml";
 
proc http
 url='https://s3.amazonaws.com/irs-form-990/201723199349204207_public.xml'
 method="get" out=out;
run;

Thank you in advance.

Re: Download xml files from multipe urls

Tom — Fri, 18 Mar 2022 01:38:59 GMT

Make a SAS dataset that has two character variables.

URL for the location you want pull from

FILE for the name where you want to save the result.

Then just use a data step to generate a PROC HTTP statement for each observation:

filename code temp;
data _null_;
  set LIST ;
  file code;
  put  'proc http method="get"'
     /  ' url='  url :$quote.
     /  ' out=' file :$quote.
     / ';run;'
  ;
run;

And you can use %INCLUDE to run them.

%include code / source2;

Re: Download xml files from multipe urls

92568466 — Fri, 18 Mar 2022 05:30:26 GMT

Hi,

I'm not sure what you meant by 'file'. I created a variable named file with the possible names. I tried to run your code, but it didn't work. I got the following error:

"ERROR 22-322: Expecting a name.
ERROR 200-322: The symbol is not recognized and will be ignored."

Can you advice on this? I've uploaded data here.

Re: Download xml files from multipe urls

Kurt_Bremser — Fri, 18 Mar 2022 06:37:47 GMT

Please post the complete (code and messages) log of the failing step into a window opened with this button:

Re: Download xml files from multipe urls

92568466 — Fri, 18 Mar 2022 07:13:25 GMT

I'm using the following code, as posted by Tom:

filename code temp;

data _null_;
set test1;
file code;
put 'proc http method="get"'
/ ' url=' url :$quote.
/ ' out=' file :$quote.
/ ';run;'
;
run;

%include code / source2;

Attached are datasets and the log file that shows the error. My objective is to get xmls from the urls (stored in the data set 'test1') and download all to a local folder.

Thanks in advance.

Re: Download xml files from multipe urls

Kurt_Bremser — Fri, 18 Mar 2022 07:24:09 GMT

You need to also include the FILENAME in the created code, as the OUT= option accepts only file references, not filenames.

filename code temp;
data _null_;
  set LIST ;
  file code;
  put  '
    filename out ' file :$quote. ';'
    / 'proc http method="get"'
     /  ' url='  url :$quote.
     /  ' out=out'
     / ';run;'
  ;
run;

Re: Download xml files from multipe urls

92568466 — Wed, 23 Mar 2022 17:43:25 GMT

Hello @Kurt_Bremser, appreciate your help. The code works great. I'm however running into a related problem that I've posted in as another question. I'm wondering if you can look into that as well.

https://communities.sas.com/t5/SAS-Programming/Download-correct-xml/m-p/803612

Thank you.