SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Turning Log Data into Dataset

Reply
Contributor jei
Contributor
Posts: 74

Turning Log Data into Dataset

Hi,

 

I have logs from machines which is in a txt file format and I would like to convert it into SAS dataset or in a tabular form. I have SAS Data Integration and SAS Enterprise Guide in our environment. May I know how it will be possible?

 

Thank you!

PROC Star
Posts: 1,167

Re: Turning Log Data into Dataset

It depends on how sophisticated you want to be. At the most basic, this will work (untested, you'll need to check to see what the longest log records are, I'm assuming 256):

 

data LogData;

length LogRecord $256;

infile "/dir1/dir2/SASlog.log" lrecl=32767;

input;

LogRecord = _infile_;

run;

 

You can then get into all sorts of sophisticated fiddles. Try googling "SAS log scraper", and that should give you a ton of ideas.

 

Tom

Super User
Posts: 7,866

Re: Turning Log Data into Dataset

I guess you would have a hard time creating the import step with just a point-and-click interface. You most probably won't get around writing the code yourself, in a code node either in DI Studio or EG.

 

I have done such things, reading performance reports from mainframe into SAS, or from the log of the SAS/SHARE server or the webserver. To give you hints or code snippets, we would need an example of the log.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
PROC Star
Posts: 1,167

Re: Turning Log Data into Dataset

Posted in reply to KurtBremser

Thanks, @KurtBremser.

 

I hadn't thought of trying it using the point and click options. I just gave it a quick test, and assuming all you want is one variable with the entire contents of the log line, the point and click import task works beautifully. I just defined the data as tab-delimited, and I don't believe there are ever any tabs in SAS logs.

 

Tom

Contributor jei
Contributor
Posts: 74

Re: Turning Log Data into Dataset

Posted in reply to KurtBremser

Hi @KurtBremser,

 

I masked the log details of the attached samle file but still the format is the same. Hope you can give me a hint to transform it into SAS dataset.

 

Thank you!

Super User
Posts: 7,866

Re: Turning Log Data into Dataset

[ Edited ]

Try the following code. I added comments to describe what it does. Note that my detection of lines broken by a CRLF in the text might need modification.

data want;
/* The infile comes from DOS and uses CRLF and the pipe character as delimiter */
infile '$HOME/sascommunity/sample_log.txt' lrecl=500 termstr=CRLF dlm='|' truncover;
input f1 :$1. dt_str :$27. textstr :$200.;
if f1 ne ' '; /* filters emtpy lines */
dt = input(substr(dt_str,1,23),e8601dt23.3);
tz = substr(dt_str,25,3);
/* you might want to correct the timestamp according to the timezone, with intnx() */
format dt e8601dt23.3; /* ISO-standard display */
/* now let's test if we have a continuation line caused by a spurious CRLF in the text */
input teststr $200. @1 @@; /* hold the line pointer and reset the column to 1 */
if substr(teststr,1,1) = '08'x
then do;
  textstr = trim(textstr) !! teststr;
  input; /* finish reading the continuation line */
end;
drop dt_str teststr;
run;

PS I edited line 11 of your file before importing as it contains a typo in the year (22017).

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 5,441

Re: Turning Log Data into Dataset

Anything is possible, especially when you have SAS.
The more that you know about the log files the better. Perhaps there's a file specification?
If it's a common machine source chances are that someone have experience with this task.
If you have DI Studio that's the preferred tool if this will be a continuous setup.
Data never sleeps
Ask a Question
Discussion stats
  • 6 replies
  • 295 views
  • 0 likes
  • 4 in conversation