topic Re: Accessing a URL with the ZIP engine in SAS Programming

Accessing a URL with the ZIP engine

PGStats — Tue, 03 Feb 2015 15:18:12 GMT

Hi All,

Can I access the contents of this file (a .csv) :

http://donnees.ville.montreal.qc.ca/storage/f/2013-10-05T17%3A49%3A08.194Z/l2p-801-15052013-leplateau-mont-royal.zip

for example, with the ZIP engine directly, without downloading it first?

I'm using SAS 9.4 on Windows 7.

Re: Accessing a URL with the ZIP engine

sbb — Wed, 04 Feb 2015 01:02:31 GMT

The contents of the file (referenced by the URL) is a zip-archive container, with a "CSV" file inside, which must be processed after retrieving from the foreign/remote site. Explore SAS 9.4 with the FILENAME ZIP engine, possibly, after retrieving the container file, using SAS FILENAME, PROC DOWNLOAD, possibly.

Scott Barry

SBBWorks, Inc.

Re: Accessing a URL with the ZIP engine

FriedEgg — Wed, 04 Feb 2015 01:22:30 GMT

You can do it with a pipe (which will be system dependent). The following would be an example for linux.

filename x pipe 'wget -qO- http://donnees.ville.montreal.qc.ca/storage/f/2013-10-05T17%3A49%3A08.194Z/l2p-801-15052013-leplateau-mont-royal.zip | funzip';

Re: Accessing a URL with the ZIP engine

PGStats — Wed, 04 Feb 2015 02:49:44 GMT

Thank you both.

I was naively hoping to wrap the URL into a Windows UNC, somehow, and use that as the 'external-file' physical name in the FILENAME statement. That way I could access the contents of the zip archive directly. Just a dream...

Re: Accessing a URL with the ZIP engine

FriedEgg — Wed, 04 Feb 2015 04:06:20 GMT

In windows you could try something like the following:

filename x pipe 'cmd /C "pushd \\myshare\folder\unc && C:\PROGRA~1\7-Zip\7z.exe x -so foo.zip 2> _garbage.txt && del _garbage.txt && popd"';

Re: Accessing a URL with the ZIP engine

Ksharp — Wed, 04 Feb 2015 06:26:28 GMT

PG,

This could not possibly be done in one step .

firstly proc http + TEMP engine download this ZIP file ,then use filename's ZIP engine.

Xia Keshan

Re: Accessing a URL with the ZIP engine

PGStats — Wed, 04 Feb 2015 18:30:57 GMT

Thank you Matt,

I cannot use your suggestion unless I understand what it does. Would you please explain the OS commands feeding that pipe? Can foo.zip be a URL?

Re: Accessing a URL with the ZIP engine

FriedEgg — Wed, 04 Feb 2015 18:56:51 GMT

You said you are wanting to use UNC path, not URL, very different.

cmd /c

- starts a new instance of the command interpreter and carries out the following command string and stops

http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/cmd.mspx?mfr=true

pushd \\myshare\folder\unc

- assigns a UNC path a temporary drive letter and changes directory to this location

https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/pushd.mspx?mfr=true

C:\PROGRA~1\7-Zip\7z.exe x -so foo.zip 2> _garbage.txt

- 7-zip is a open source file archiver that has the ability to output the contents of a ZIP archive to STDOUT

7-Zip

Where:

x - denotes to eXtract from the archive

-so - write contents of archive to STDOUT

foo.zip - is the name of the file contained in the UNC path assigned with PUSHD

2> _garbage.txt - this redirects the other messages produced by 7z.exe to a junk file, redirecting to NUL doesn't work, for some reason

del _garbage.txt

- Delete the file we had to create with 7z.exe to get the STDOUT write to work

https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/del.mspx?mfr=true

popd

- Removes the temporary drive-letter assignment created with pushd and returns to the original working directory

https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/popd.mspx?mfr=true

The double ampersand (&&) command separator strings all these commands together, with error checking (only running one after the previous runs without error

http://support.microsoft.com/kb/279253

If you are wanting to use a proper URL in Windows it would be a different set of commands and different tools.

Re: Accessing a URL with the ZIP engine

FriedEgg — Wed, 04 Feb 2015 19:03:54 GMT

If you want to use URL then:

Get Wget for Windows

Get bsdtar for Windows with LibArchive for Windows

filename x pipe 'wget -qO- http://donnees.ville.montreal.qc.ca/storage/f/2013-10-05T17%3A49%3A08.194Z/l2p-801-15052013-leplateau-mont-royal.zip | bsdtar -xvf-';

Where:

wget downloads the zipfile

-q sets the wget command to be quiet

-O- sets the output to be directed to STDOUT

bsdtar reads the zip file from the pipe and extracts the contents, again to STDOUT

-x extract

-v tells bsdtar to operate verbosely

-f- indicates to operate on the file coming from STDIN

Re: Accessing a URL with the ZIP engine

sbb — Wed, 04 Feb 2015 19:08:19 GMT

Recommend native SAS function/facility here with FILENAME / FTP engine to retrieve the file, followed by SAS ODS PACKAGE to extract from the zip-archive.

Re: Accessing a URL with the ZIP engine

PGStats — Wed, 04 Feb 2015 19:12:28 GMT

Thank you Matt, this should be very helpful to me and hopefully to others too!

Re: Accessing a URL with the ZIP engine

PGStats — Wed, 04 Feb 2015 19:14:29 GMT

Thank you , I will try. - PG

Re: Accessing a URL with the ZIP engine

FriedEgg — Wed, 04 Feb 2015 19:19:43 GMT

It's not something I'd really recommend, but's it certainly possible to do.

Re: Accessing a URL with the ZIP engine

PGStats — Wed, 04 Feb 2015 19:39:26 GMT

I have to ask! What would you really recommend? - PG

Re: Accessing a URL with the ZIP engine

FriedEgg — Wed, 04 Feb 2015 19:52:00 GMT

It would depend on the specific situation, but 's recommendation would be on the list:

Use PROC HTTP to download the file, probably to a TEMP file if I didn't care to keep it.

Use the FILENAME statement with the ZIP access method to gain access to the internal file

Re: Accessing a URL with the ZIP engine

PGStats — Wed, 04 Feb 2015 20:13:28 GMT

I'll try that first then. Thanks again Matt. - PG