Download a unix zipped file from the web

Reply
Contributor
Posts: 42

Download a unix zipped file from the web

I'm trying to download a zip file from the web.

Anything someone can point me to?

filename fnma url "https://loanperformancedata.fanniemae.com/lppub/index.html#Portfolio/Acquisition_2000Q1.txt.gz";

data _null_;

    infile fnma recfm=n;

    input;

    file _fnma;

    put _infile_;

run;

ERROR: SSL Error: Missing CA trust list_.

ERROR: Insufficient authorization to access /opt/sas/eg/Lev1/SASApp/_fnma.dat.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: DATA statement used (Total process time):

      real time           2.65 seconds

      cpu time            0.00 seconds

Valued Guide
Posts: 3,208

Re: Download a unix zipped file from the web

There are two errors.

- the SSL error is telling something about your url   when ignored and you can still read this data

- Your file/filename where you want to store that zip-file is missing.

  It is getting to a default generated name. _fnma.dat at a location, your wrongly set home location on a Unix server.

  Your Eguide session is running a Workspace server but those settings are incorrect. See: 50345 - Changing the current working directory for the SAS® Workspace Server 

 

You can bypass the second error by coding: file _fnma "<your location>/Acquistion_2000Q1.txt.gz" ;

The first error is more difficult to pinpoint. It could be a ssl url mismatch or that you need to use webdav access. SAS(R) 9.3 Statements: Reference (WebDAV file)

---->-- ja karman --<-----
Contributor
Posts: 42

Re: Download a unix zipped file from the web

Is there an option I need to specify to ignore the SSL error?

I tried webdav - no luck there.

Valued Guide
Posts: 3,208

Re: Download a unix zipped file from the web

I realy don not know if this is possible https on Unix with url access.  SAS is making some references to the companion Unix: SAS(R) 9.4 Companion for UNIX Environments, Third Edition (filename). At that chapter https is missing with the url fileaccess. As https is often combined with secured data user/password the WebDAV is making more sense.

Checking your link you posted, it is aksing for a userid and password. Where did you solve that in your code?

When all is failing, it could be a part of the tcp/ip stack is missing not configured correctly.

It would not be the first time SAS is far, too far, up with the market evolvments. I have seen similar things with email access and much more.

You are in an exceptional situation having open access to internet of the Unix machine. Mostly it is closed behind firewalls by that isolating sensitive data from the angry outer world. Connecting to data often is done by executing that on an other machine and than do a (s)FTP/upload to the server.

---->-- ja karman --<-----
Contributor
Posts: 42

Re: Download a unix zipped file from the web

I'm getting pretty close but something is going wrong with the download. The file size is slightly smaller. I'm not able to unzip the file and when I try to read the zip file directly I'm getting this error.

.

options mlogic sslcalistloc="/ma/scott/fhlmc/fhlmc.cert"; /* location of certificate file */

filename fhlmc url "https://freddiemac.embs.com/FLoan/Data/historical_data1_Q11999.zip" debug;

data _null_;

    infile fhlmc recfm=n;

    input;

    file "/ma/scott/fhlmc/data/historical_data1_Q11999.zip";

    put _infile_;

run;

SYMBOLGEN:  Macro variable LIB resolves to /ma/scott/fhlmc/data/

SYMBOLGEN:  Macro variable AS_OF_QTR_YEAR resolves to Q11999

MPRINT(ORIG):   FileName zip Saszipam "/ma/scott/fhlmc/data/historical_data1_Q11999.zip";

MPRINT(ORIG):   data origfile;

SYMBOLGEN:  Macro variable AS_OF_QTR_YEAR resolves to Q11999

MPRINT(ORIG):   infile zip(historical_data1_Q11999.txt) dsd dlm='|' missover firstobs=2 lrecl=32767;

MPRINT(ORIG):   input fico : 8. dt_first_pi : 8. flag_fthb : $1. dt_matr : 8. cd_msa : 8. mi_pct : 8. cnt_units : 8. occpy_sts :

$1. cltv : 8. dti : 8. orig_upb : 8. ltv : 8. int_rt : 8. channel : $1. ppmt_pnlty : $1. prod_type : $5. st : $2. prop_type : $2.

zipcode : 8. id_loan : $16. loan_purpose : $5. orig_loan_term : 8. cnt_borr : $2. seller_name : $30. servicer_name : $30. ;

MPRINT(ORIG):   run;

2                                                          The SAS System                                09:07 Monday, June 16, 2014

ERROR: The central directory is invalid in /ma/scott/fhlmc/data/historical_data1_Q11999.zip.

WARNING: End of file.

NOTE: The SAS System stopped processing this step because of errors.

WARNING: The data set WORK.ORIGFILE may be incomplete.  When this step was stopped there were 0 observations and 25 variables.

WARNING: Data set WORK.ORIGFILE was not replaced because this step was stopped.

NOTE: DATA statement used (Total process time):

      real time           0.00 seconds

      cpu time            0.01 seconds

Contributor
Posts: 42

Re: Download a unix zipped file from the web

Interesting I just found this syntax and it appears to work. I'm able to read from the files now. Still need to figure out why the file size is smaller though.

filename d1 "/pm/ma/scott/fhlmc/data/historical_data1_Q11999.zip";

proc http

method='GET'

url="https://freddiemac.embs.com/FLoan/Data/historical_data1_Q11999.zip"

out=d1;

run;

Super User
Posts: 6,971

Re: Download a unix zipped file from the web

You obviously have a macro involved in this, but did not post the macro code and the macro call.

The log output clearly does not belong to the data _null_ step.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 6,971

Re: Download a unix zipped file from the web

You may want to consult this:

http://support.sas.com/documentation/cdl/en/secref/62092/PDF/default/secref.pdf

about setting up SSL, it dleas with the CA trust list.

OTOH, why don't you try to do the downloads externally (ie with curl), and then read from the downloaded files?

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Valued Guide
Posts: 3,208

Re: Download a unix zipped file from the web

The difference in filesize is telling there is some conversion done, not real binary.

In the first approach the input is having recfm=  but the output not.

I am learning as you find proc http. That one is calling a Web page directly. As that one is also converting the file it could be a failure on the content type identification.

This is the CT parameter to influence.

---->-- ja karman --<-----
Ask a Question
Discussion stats
  • 8 replies
  • 899 views
  • 3 likes
  • 3 in conversation