How to read 4GB+ log file?

Reply
New Contributor
Posts: 4

How to read 4GB+ log file?

A SAS program I ran on PuTTY created a log file that is 4GB+. I would like to be able to read some of its contents. Its obviously too large to download from my server to my local computer. I can't open it directly on the server because, again, it is too large. How can I split this log file into many smaller files or reduce its file size so that I can read some of its contents? Thanks for any suggestions.

Super User
Posts: 19,167

Re: How to read 4GB+ log file?

If you want to split the file you can pass the split command to your server using the appropriate OS command

If Unix:

ie split into 100 files:

split -l 100 my_log_file my_file

You could also consider running it through a log parser as well.

SAS Employee
Posts: 1

Re: How to read 4GB+ log file?

What happens when you try to open the log file in vi?

To transfer - zip, ftp and unzip.

1. tar -czvf filename.tar.gz filename

2. use ftp to transfer (winscp, filezilla)

3. unzip using 7-zip, winzip

I am assuming you want to transfer the file from Unix to Windows desktop.

N/A
Posts: 1

Re: How to read 4GB+ log file?

Treat the log file as any external text file.  Create metadata to the log file as a New ...External File...Fixed Width...and point the external file location to where your log file is .  Define logical record length to 256 and select 'Pad column values with blank'.  In the Column Definition, add 1 character column of length 256,  begin position 1 end position 256.     When you click the "Refresh" button, you should be able to see  a few records (just as a test).  Do not worry about setting informat.  Once you have the metadata setup, you can view it (open it) as any other external file or table.

ji
N/A
Posts: 1

Re: How to read 4GB+ log file?

what I did before is to read the log file into a SAS dataset as a normal text file, Then you can do all the SAS magic to it. Here is a reference code


data Logfile;

infile "/###/###.log" truncover dsd lrecl = 32767;

length column1 $255;

input @1 column1 $255.;

run;

if it is 4GB, make sure you use options compress = yes.

Super User
Posts: 9,875

Re: How to read 4GB+ log file?

You can use command MORE to check the log file at the first a couple of lines .

N/A
Posts: 1

Re: How to read 4GB+ log file?

Text file size can be drastically reduced with gzip and the -9 option: gzip -9 myfile

Then transfer it to your computer, and use the pipe option of the filename statement:

filename mylog pipe "gzip -dc myfile.gz";

If you're running on Windows, you can use 7zip, it has a command line interface usable within SAS via with pipe.

I do not have SAS on Windows so you may need to adapt the command line with something like

filename mylog pipe "%COMSPEC% /c 'c:\Program Files\7zip\7z mylog.gz'";

Be sure to output only what you need or you will create a huge table.

Another solution might be to filter your log file on the server with some grep commands

Super User
Posts: 3,237

Re: How to read 4GB+ log file?

Also I have successfully used the FSLIST procedure or command with large text files. This procedure is part of Base SAS, but can only be used in an interactive SAS session on the server (via a remote login), not remotely via a PC client like EG.

Ask a Question
Discussion stats
  • 7 replies
  • 752 views
  • 0 likes
  • 8 in conversation