topic Re: connect files from a web server in SAS Programming

connect files from a web server

JuanaPrada — Sat, 11 May 2024 11:29:41 GMT

Hi all - ¿How can I connect files from a web server to sas viya? I want to automatically capture the log files in .gz, and then process them (ex: domain.com-2024-04-11.gz).

Re: connect files from a web server

Kurt_Bremser — Sat, 11 May 2024 12:05:23 GMT

PROC HTTP as well as FILENAME URL can be used to download/access web resources.

Re: connect files from a web server

JuanaPrada — Sat, 11 May 2024 18:22:39 GMT

Firstly, I would like to express my gratitude for your response Kurt. I believe it’s a suitable solution. Now, I’m curious about how I can unzip the .gz files (e.g., domain.com-2024-04-11.gz) on SAS Viya. These files are actually .txt, but I would like to automate the loading and decompression process for further processing. I understand that you’re proposing to connect to the server and then assemble a bash script to unzip the files, right?

Re: connect files from a web server

Kurt_Bremser — Sat, 11 May 2024 21:09:26 GMT

First, I would define a file reference with FILENAME URL. Then I would use that reference in a DATA step in the INFILE statement with the GZIP option. This should perform the download and uncompress in one action.

Re: connect files from a web server

Sajid01 — Sun, 12 May 2024 13:03:17 GMT

Hello @JuanaPrada
In a typical webserver log files are not exposed over the web. The appropriate way would be to use scp.

If you are using Viya 4, one approach would be scp using your jump box (if that is permissible ) and copy to appropriate folder in /viya-share .
However if it is Viya 3.x scp would be much simpler process.

For exact details applicable do consult the system admin on the web server side and do include you Viya Admin in planning and implementation.

Re: connect files from a web server

JuanaPrada — Tue, 14 May 2024 13:31:11 GMT

The solution I have obtained was to create a simple bash task using linux command-line tools to establish the ssh connection, copy the files from the server to the local folder (scp) and then unzip them (gunzip). Later I created the connection for data import in SAS Viya. The first option through FILENAME URL is not possible because "The access method "SOCKET is disabled when SAS is in the lockdown state".

Re: connect files from a web server

Tom — Tue, 14 May 2024 15:33:37 GMT

Note that once you have local files you should be able to skip the gunzip step. SAS can read gzipped text files directly by just using the ZIP fileref engine with the GZIP option.

In normal SAS code that would look something like:

filename mylog zip 'mylog.log.gz' gzip;

or if writing your own data step you can skip the FILENAME statement and reference the file directly in the INFILE statement adding the ZIP engine and GZIP option there instead. Like this:

infile 'mylog.log.gz' zip gzip;

Re: connect files from a web server

JuanaPrada — Thu, 16 May 2024 20:14:41 GMT

Dear Tom,

I sincerely appreciate your value observation. Upon thorough examination and careful consideration, I have concluded that the most prudent and effective approach would be to PROC LIBRARY step because depends on several factors, such as file size, access frequency, processing needs, and execution environment. For small files and occasional access, INFILE is the simplest and most efficient option. For medium or large files, or when complex processing is required, PROC ZIP or PROC LIBRARY are better options. If flexible file access or control over processing is needed, the SAS Viya REST API or SAS Event Stream may be more suitable alternatives. In other words, to summarize:

Factor	infile	proc zip	proc library	SAS Viya REST API	SAS Event Stream
File size	Small	Medium/Large	Medium/Large	Any	Any
Access frequency	Occasional	Frequent	Frequent	Variable	Frequent
Processing needs	Basic	Complex	Variable	Complex	Complex
Execution environment	SAS Studio, SAS Enterprise Guide	SAS Studio, SAS Enterprise Guide	SAS Studio, SAS Enterprise Guide	SAS Viya	SAS Viya