DATA Step, Macro, Functions and more

How to read data beyond 32767 characters and keep them in a single variable?

Reply
Occasional Contributor
Posts: 5

How to read data beyond 32767 characters and keep them in a single variable?

I am reading data from sas log file row by row and would like to concatinate all rows into single variable.

Can someone let me know the approach that i can read data beyond 32767 character and keep it in dataset variable for my further processing.

SAS Super FREQ
Posts: 683

Re: How to read data beyond 32767 characters and keep them in a single variable?

The max length for a char variable in a SAS dataset is 32767. So there is not way to store the contents of a log file into one variable.

 

If you say more about what you want to do, there might be different solutions.

 

Bruno

Occasional Contributor
Posts: 5

Re: How to read data beyond 32767 characters and keep them in a single variable?

Thanks Bruno for your quick respone.

 

Can you advise how to handle this situation?. If we split into two variables and later concatenate them and generate a external file (.csv) with more than 32k will it possible?.

 

Our requirement is to generate a external csv file with all the values in a single variable might fall beyond 32 K length.

 

Pleaes advise.

 

Thanks in advance,

Cheers...

Super User
Posts: 6,936

Re: How to read data beyond 32767 characters and keep them in a single variable?

Forget it. Log files are in another order of magnitude. Ours routinely reach megabyte sizes.

If you want to create a database of logs, put path/filenames into the dataset and create a report with links to the files.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Occasional Contributor
Posts: 5

Re: How to read data beyond 32767 characters and keep them in a single variable?

Hi Kurt,

 

Thanks for your advise. Let me explain the requirement.

 

1. Read sas logs into sas system and create dataset

2. While reading into sas system I am splitting the log into blocks based on cpu time... Means I am reading a log from starting position to till it reaches to cpu time as I am considering as a block. It contains the the atual code and and related information as well. For ex. if the     log contains 4 times cpu time , my code will split the log into 4 blocks.( starting position to till cpu time as a block). In this situation some times log maybe huge and have length more than 32k ... I am facing issue here.

3. Need to analyze each block and need to populate some values like block,number of observations, target table , source table ..etc..

 

Please advise if you have any solution for this.

 

 

Super User
Posts: 10,497

Re: How to read data beyond 32767 characters and keep them in a single variable?

If your aim is to track application reponse and such you might look into the ARM facility.

Super User
Posts: 6,936

Re: How to read data beyond 32767 characters and keep them in a single variable?


Chery wrote:

Hi Kurt,

 

Thanks for your advise. Let me explain the requirement.

 

1. Read sas logs into sas system and create dataset

2. While reading into sas system I am splitting the log into blocks based on cpu time... Means I am reading a log from starting position to till it reaches to cpu time as I am considering as a block. It contains the the atual code and and related information as well. For ex. if the     log contains 4 times cpu time , my code will split the log into 4 blocks.( starting position to till cpu time as a block). In this situation some times log maybe huge and have length more than 32k ... I am facing issue here.

3. Need to analyze each block and need to populate some values like block,number of observations, target table , source table ..etc..

 

Please advise if you have any solution for this.

 

 


I'd rather read the log line by line and save the data everytime a step boundary appears.

Basically collect data into RETAINed variables, and once a proc or data statement appears, write the record out and reset the variables to empty/missing to start a new cycle.

Or you could simply only parse the NOTE: lines that conclude every step.

After that you can cumulate statistics for a given time window.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 10,497

Re: How to read data beyond 32767 characters and keep them in a single variable?

Ugly but you may be able to fool things depending on operating systems involved. For instance the in Windows if you create a csv file by default each record ends in a Carriage-Return Line-Feed pair of characters. You can, when writing data to a file you can specify a different record terminatior using the TERMSTR option on the file statement. So if you use TERMSTR=LF for instance it misses the carriage return and a WINDOWS system will think it is part of the same record though it will appear on a different line.

 

But I am having a hard time finding a valid reason to stuff that much text into a single CSV column.

 

Note, if you are intendent to add other variables the above may not be viable.

 

And what kind of processing?

Ask a Question
Discussion stats
  • 7 replies
  • 242 views
  • 2 likes
  • 4 in conversation