Importing very long records

[FatCaptain]

I'm looking to import several text files which have a single observation and a single varaible.

The variables are between 14,000 and 22,000 characters in length (it's a data stream that I'm wanting to parse in SAS).

I can only get the text file to import if I specify the exact length of the character string, e.g.

     lrecl=14,382 (in the infile statement)

     informat datvar $14382. ;

     format datvar $14382. ;

     input datvar $ 1-14382 ;

Whilst this is fine for a single file, It's not possible to do this manual task for all of the files.

Please note that the datastream contains spaces so I'm having to specify it as a fixed width file.

Is there any way of determining the length of the variable and loading this into a macro variable for use in the data step?

Alternatively, has anyone written a json parser I could borrow!

Cheers,

Paul.

[data_null__]

The input statement is a very powerful parser.

With regards to reading single record.  Try something like this...

data name;

   infile fileref lrecl=32767 length=len;

   input line $varying32767. len;

   run;   

or you can assign the value of _INFILE_ to a character variable.

data name;

   length line 32767;

   infile fileref lrecl=32767;

   input;

   line = _infile_;

   run;

You can use a value less than 32767 if you like.

I'll google json I don't know about that.

[Tom · Accepted solution]

Just add the TRUNCOVER option to your INFILE statement.

See this recent thread.

http://communities.sas.com/message/13199#13199

[Peter_C]

FatCaptain

sounds like you might be planning to parse these data after they are loaded.

Have a look at some of the examples of the input statement.

I think that you might discover it to be even more powerful that the many parsing functions available in a data step.

You could also read the data files as multiple pieces of, say, 64 bytes, like

data shorts;

   infile 'your file' recfm=F lrecl=64 pad ;

   input short $char64. ;

run ;

[FatCaptain]

Thanks Tom, I knew the answer was going to be a simple one! I was confusing missover with truncover and going nowhere.

Thanks also Data _null_ and Peter.C.

I never thought about using the input statement to parse the character string. Looking at some of the options available, it will allow me to do most of the work before I use data steps to pretty things up.

I have to admit that I'm too used to wizards handling data imports for me. Looks like this is as good an opportunity to get to grips with the finer points of coding imports instead.

Cheers,

Paul.