connect files from a web server

[JuanaPrada]

Hi all - ¿How can I connect files from a web server to sas viya? I want to automatically capture the log files in .gz, and then process them (ex: domain.com-2024-04-11.gz).

[Kurt_Bremser]

PROC HTTP as well as FILENAME URL can be used to download/access web resources.

[JuanaPrada]

Firstly, I would like to express my gratitude for your response Kurt. I believe it’s a suitable solution. Now, I’m curious about how I can unzip the .gz files (e.g., domain.com-2024-04-11.gz) on SAS Viya. These files are actually .txt, but I would like to automate the loading and decompression process for further processing. I understand that you’re proposing to connect to the server and then assemble a bash script to unzip the files, right?

[Kurt_Bremser]

First, I would define a file reference with FILENAME URL. Then I would use that reference in a DATA step in the INFILE statement with the GZIP option. This should perform the download and uncompress in one action.

[Sajid01 · Accepted solution]

Hello @JuanaPrada 
In a typical webserver log files are not exposed over the web. The appropriate way would be to use scp.

If you are using Viya 4, one approach would be scp using your jump box (if that is permissible ) and copy to appropriate folder in /viya-share .
However if it is Viya 3.x scp would be much simpler process.

For exact details applicable do consult the system admin on the web server side and do include you Viya Admin in planning and implementation.

[JuanaPrada]

The solution I have obtained was to create a simple bash task using linux command-line tools to establish the ssh connection, copy the files from the server to the local folder (scp) and then unzip them (gunzip). Later I created the connection for data import in SAS Viya. The first option through FILENAME URL is not possible because "The access method "SOCKET is disabled when SAS is in the lockdown state".

[Tom]

Note that once you have local files you should be able to skip the gunzip step.  SAS can read gzipped text files directly by just using the ZIP fileref engine with the GZIP option.

In normal SAS code that would look something like:

filename mylog zip 'mylog.log.gz' gzip;

or if writing your own data step you can skip the FILENAME statement and reference the file directly in the INFILE statement adding the ZIP engine and GZIP option there instead. Like this:

infile 'mylog.log.gz' zip gzip;

[JuanaPrada]

Dear Tom,

I sincerely appreciate your value observation. Upon thorough examination and careful consideration, I have concluded that the most prudent and effective approach would be to PROC LIBRARY step because depends on several factors, such as file size, access frequency, processing needs, and execution environment. For small files and occasional access, INFILE is the simplest and most efficient option. For medium or large files, or when complex processing is required, PROC ZIP or PROC LIBRARY are better options. If flexible file access or control over processing is needed, the SAS Viya REST API or SAS Event Stream may be more suitable alternatives. In other words, to summarize:

Factor	infile	proc zip	proc library	SAS Viya REST API	SAS Event Stream
File size	Small	Medium/Large	Medium/Large	Any	Any
Access frequency	Occasional	Frequent	Frequent	Variable	Frequent
Processing needs	Basic	Complex	Variable	Complex	Complex
Execution environment	SAS Studio, SAS Enterprise Guide	SAS Studio, SAS Enterprise Guide	SAS Studio, SAS Enterprise Guide	SAS Viya	SAS Viya