DATA Step, Macro, Functions and more

How to read data beyond 32767 characters and keep them in a single variable?

Reply
Occasional Contributor
Posts: 5

How to read data beyond 32767 characters and keep them in a single variable?

I am reading data from sas log file row by row and would like to concatinate all rows into single variable.

Can someone let me know the approach that i can read data beyond 32767 character and keep it in dataset variable for my further processing.

SAS Super FREQ
Posts: 708

Re: How to read data beyond 32767 characters and keep them in a single variable?

The max length for a char variable in a SAS dataset is 32767. So there is not way to store the contents of a log file into one variable.

 

If you say more about what you want to do, there might be different solutions.

 

Bruno

Occasional Contributor
Posts: 5

Re: How to read data beyond 32767 characters and keep them in a single variable?

Posted in reply to Bruno_SAS

Thanks Bruno for your quick respone.

 

Can you advise how to handle this situation?. If we split into two variables and later concatenate them and generate a external file (.csv) with more than 32k will it possible?.

 

Our requirement is to generate a external csv file with all the values in a single variable might fall beyond 32 K length.

 

Pleaes advise.

 

Thanks in advance,

Cheers...

Super User
Posts: 7,771

Re: How to read data beyond 32767 characters and keep them in a single variable?

Forget it. Log files are in another order of magnitude. Ours routinely reach megabyte sizes.

If you want to create a database of logs, put path/filenames into the dataset and create a report with links to the files.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Occasional Contributor
Posts: 5

Re: How to read data beyond 32767 characters and keep them in a single variable?

Posted in reply to KurtBremser

Hi Kurt,

 

Thanks for your advise. Let me explain the requirement.

 

1. Read sas logs into sas system and create dataset

2. While reading into sas system I am splitting the log into blocks based on cpu time... Means I am reading a log from starting position to till it reaches to cpu time as I am considering as a block. It contains the the atual code and and related information as well. For ex. if the     log contains 4 times cpu time , my code will split the log into 4 blocks.( starting position to till cpu time as a block). In this situation some times log maybe huge and have length more than 32k ... I am facing issue here.

3. Need to analyze each block and need to populate some values like block,number of observations, target table , source table ..etc..

 

Please advise if you have any solution for this.

 

 

Super User
Posts: 11,343

Re: How to read data beyond 32767 characters and keep them in a single variable?

If your aim is to track application reponse and such you might look into the ARM facility.

Super User
Posts: 7,771

Re: How to read data beyond 32767 characters and keep them in a single variable?


Chery wrote:

Hi Kurt,

 

Thanks for your advise. Let me explain the requirement.

 

1. Read sas logs into sas system and create dataset

2. While reading into sas system I am splitting the log into blocks based on cpu time... Means I am reading a log from starting position to till it reaches to cpu time as I am considering as a block. It contains the the atual code and and related information as well. For ex. if the     log contains 4 times cpu time , my code will split the log into 4 blocks.( starting position to till cpu time as a block). In this situation some times log maybe huge and have length more than 32k ... I am facing issue here.

3. Need to analyze each block and need to populate some values like block,number of observations, target table , source table ..etc..

 

Please advise if you have any solution for this.

 

 


I'd rather read the log line by line and save the data everytime a step boundary appears.

Basically collect data into RETAINed variables, and once a proc or data statement appears, write the record out and reset the variables to empty/missing to start a new cycle.

Or you could simply only parse the NOTE: lines that conclude every step.

After that you can cumulate statistics for a given time window.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 11,343

Re: How to read data beyond 32767 characters and keep them in a single variable?

Ugly but you may be able to fool things depending on operating systems involved. For instance the in Windows if you create a csv file by default each record ends in a Carriage-Return Line-Feed pair of characters. You can, when writing data to a file you can specify a different record terminatior using the TERMSTR option on the file statement. So if you use TERMSTR=LF for instance it misses the carriage return and a WINDOWS system will think it is part of the same record though it will appear on a different line.

 

But I am having a hard time finding a valid reason to stuff that much text into a single CSV column.

 

Note, if you are intendent to add other variables the above may not be viable.

 

And what kind of processing?

Ask a Question
Discussion stats
  • 7 replies
  • 270 views
  • 2 likes
  • 4 in conversation