topic How to read data beyond 32767 characters and keep them in a single variable? in SAS Programming

How to read data beyond 32767 characters and keep them in a single variable?

Chery — Wed, 09 Nov 2016 17:23:11 GMT

I am reading data from sas log file row by row and would like to concatinate all rows into single variable.

Can someone let me know the approach that i can read data beyond 32767 character and keep it in dataset variable for my further processing.

Re: How to read data beyond 32767 characters and keep them in a single variable?

BrunoMueller — Wed, 09 Nov 2016 17:36:56 GMT

The max length for a char variable in a SAS dataset is 32767. So there is not way to store the contents of a log file into one variable.

If you say more about what you want to do, there might be different solutions.

Bruno

Re: How to read data beyond 32767 characters and keep them in a single variable?

Chery — Wed, 09 Nov 2016 18:37:22 GMT

Thanks Bruno for your quick respone.

Can you advise how to handle this situation?. If we split into two variables and later concatenate them and generate a external file (.csv) with more than 32k will it possible?.

Our requirement is to generate a external csv file with all the values in a single variable might fall beyond 32 K length.

Pleaes advise.

Thanks in advance,

Cheers...

Re: How to read data beyond 32767 characters and keep them in a single variable?

Kurt_Bremser — Wed, 09 Nov 2016 19:02:00 GMT

Forget it. Log files are in another order of magnitude. Ours routinely reach megabyte sizes.

If you want to create a database of logs, put path/filenames into the dataset and create a report with links to the files.

Re: How to read data beyond 32767 characters and keep them in a single variable?

ballardw — Wed, 09 Nov 2016 20:07:40 GMT

Ugly but you may be able to fool things depending on operating systems involved. For instance the in Windows if you create a csv file by default each record ends in a Carriage-Return Line-Feed pair of characters. You can, when writing data to a file you can specify a different record terminatior using the TERMSTR option on the file statement. So if you use TERMSTR=LF for instance it misses the carriage return and a WINDOWS system will think it is part of the same record though it will appear on a different line.

But I am having a hard time finding a valid reason to stuff that much text into a single CSV column.

Note, if you are intendent to add other variables the above may not be viable.

And what kind of processing?

Re: How to read data beyond 32767 characters and keep them in a single variable?

Chery — Wed, 09 Nov 2016 20:37:12 GMT

Hi Kurt,

Thanks for your advise. Let me explain the requirement.

1. Read sas logs into sas system and create dataset

2. While reading into sas system I am splitting the log into blocks based on cpu time... Means I am reading a log from starting position to till it reaches to cpu time as I am considering as a block. It contains the the atual code and and related information as well. For ex. if the log contains 4 times cpu time , my code will split the log into 4 blocks.( starting position to till cpu time as a block). In this situation some times log maybe huge and have length more than 32k ... I am facing issue here.

3. Need to analyze each block and need to populate some values like block,number of observations, target table , source table ..etc..

Please advise if you have any solution for this.

Re: How to read data beyond 32767 characters and keep them in a single variable?

ballardw — Wed, 09 Nov 2016 21:08:59 GMT

If your aim is to track application reponse and such you might look into the ARM facility.

Re: How to read data beyond 32767 characters and keep them in a single variable?

Kurt_Bremser — Thu, 10 Nov 2016 08:45:54 GMT

@Chery wrote:

Hi Kurt,

Thanks for your advise. Let me explain the requirement.

1. Read sas logs into sas system and create dataset

2. While reading into sas system I am splitting the log into blocks based on cpu time... Means I am reading a log from starting position to till it reaches to cpu time as I am considering as a block. It contains the the atual code and and related information as well. For ex. if the log contains 4 times cpu time , my code will split the log into 4 blocks.( starting position to till cpu time as a block). In this situation some times log maybe huge and have length more than 32k ... I am facing issue here.

3. Need to analyze each block and need to populate some values like block,number of observations, target table , source table ..etc..

Please advise if you have any solution for this.

I'd rather read the log line by line and save the data everytime a step boundary appears.

Basically collect data into RETAINed variables, and once a proc or data statement appears, write the record out and reset the variables to empty/missing to start a new cycle.

Or you could simply only parse the NOTE: lines that conclude every step.

After that you can cumulate statistics for a given time window.