DATA Step, Macro, Functions and more

Importing very long records

Accepted Solution Solved
Reply
Contributor
Posts: 29
Accepted Solution

Importing very long records

I'm looking to import several text files which have a single observation and a single varaible.

The variables are between 14,000 and 22,000 characters in length (it's a data stream that I'm wanting to parse in SAS).

I can only get the text file to import if I specify the exact length of the character string, e.g.

     lrecl=14,382 (in the infile statement)

     informat datvar $14382. ;

     format datvar $14382. ;

     input datvar $ 1-14382 ;

Whilst this is fine for a single file, It's not possible to do this manual task for all of the files.

Please note that the datastream contains spaces so I'm having to specify it as a fixed width file.

Is there any way of determining the length of the variable and loading this into a macro variable for use in the data step?

Alternatively, has anyone written a json parser I could borrow!

Cheers,

Paul.


Accepted Solutions
Solution
‎08-07-2011 08:44 PM
Super User
Super User
Posts: 7,050

Re: Importing very long records

Posted in reply to FatCaptain

Just add the TRUNCOVER option to your INFILE statement.

See this recent thread.

http://communities.sas.com/message/13199#13199

View solution in original post


All Replies
Respected Advisor
Posts: 3,799

Re: Importing very long records

Posted in reply to FatCaptain

The input statement is a very powerful parser.

With regards to reading single record.  Try something like this...

data name;

   infile fileref lrecl=32767 length=len;

   input line $varying32767. len;

   run;   

or you can assign the value of _INFILE_ to a character variable.

data name;

   length line 32767;

   infile fileref lrecl=32767;

   input;

   line = _infile_;

   run;

You can use a value less than 32767 if you like.

I'll google json I don't know about that.

Solution
‎08-07-2011 08:44 PM
Super User
Super User
Posts: 7,050

Re: Importing very long records

Posted in reply to FatCaptain

Just add the TRUNCOVER option to your INFILE statement.

See this recent thread.

http://communities.sas.com/message/13199#13199

Valued Guide
Posts: 2,177

Importing very long records

Posted in reply to FatCaptain

FatCaptain

sounds like you might be planning to parse these data after they are loaded.

Have a look at some of the examples of the input statement.

I think that you might discover it to be even more powerful that the many parsing functions available in a data step.

You could also read the data files as multiple pieces of, say, 64 bytes, like

data shorts;

   infile 'your file' recfm=F lrecl=64 pad ;

   input short $char64. ;

run ;

Contributor
Posts: 29

Importing very long records

Posted in reply to FatCaptain

Thanks Tom, I knew the answer was going to be a simple one! I was confusing missover with truncover and going nowhere.

Thanks also Data _null_ and Peter.C.

I never thought about using the input statement to parse the character string. Looking at some of the options available, it will allow me to do most of the work before I use data steps to pretty things up.

I have to admit that I'm too used to wizards handling data imports for me. Looks like this is as good an opportunity to get to grips with the finer points of coding imports instead.

Cheers,

Paul.

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 484 views
  • 6 likes
  • 4 in conversation