Help using Base SAS procedures

INPUT - How to read a single line which is longer than 32767 characters?

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 12
Accepted Solution

INPUT - How to read a single line which is longer than 32767 characters?

A bit of history

Yesterday I asked this question, about reading a JSON file which had arbitrary carriage returns in it, and reading it into one variable. This was because the random new lines were messing with my INPUT statement.

I'm having a separate issue with the same file which I thought I'd raise separately.

My new problem

Now that I've sanitised the file to remove the random new lines, I have a text file containing JSON which has a single line of text whose length exceeds 32767 characters (worth pointing out that even before I removed the random new lines, there were still lines longer than 32767 characters!)

When I try to read this line in, my INPUT step only processes records until the 32767th character, then ends, so I'm left with a fraction of the observations that I should have.

I assume this is either due to the maximum length of the input buffer or the fact that I've set the LRECL to its maximum value (32767)?

What I'd like to do

Ideally I'd like to specify a record delimiter string (e.g. "},{") which could split my massive, massive line to treat each JSON object (which are wrapped in curly brackets) individually. Is this possible? Any solution which will allow me to read this whole file would work though.

Any help appreciated.

<<Sample file attached>>

Attachment

Accepted Solutions
Solution
‎05-09-2014 12:21 PM
Occasional Contributor
Posts: 12

Re: INPUT - How to read a single line which is longer than 32767 characters?

Aha! My assertion that 32767 was the maximum length of LRECL was incorrect. Thanks to this post, I've realised I could bump the size of the _infile_ to whatever I want, and this has stopped my issues.

Thank you @Tom !

View solution in original post


All Replies
Super User
Posts: 9,682

Re: INPUT - How to read a single line which is longer than 32767 characters?

Could you use PROC JSON to get data. or just use JAVA code to get it .

Occasional Contributor
Posts: 12

Re: INPUT - How to read a single line which is longer than 32767 characters?

Ksharp - As I understand it, PROC JSON is only for outputting JSON, not consuming it. To be honest it is looking more and more like I might have to use Java.

I take it it's just not possible to specify a "record delimiter" then?

Valued Guide
Posts: 3,208

Re: INPUT - How to read a single line which is longer than 32767 characters?

I did post some links on reading Json files in your first thread.

They were presentations on forums

---->-- ja karman --<-----
Occasional Contributor
Posts: 12

Re: INPUT - How to read a single line which is longer than 32767 characters?

Jaap - Thanks, I did check those out but couldn't immediately get them working, and wanted to see if I could tweak the parsing code I already had before scrapping it. I do appreciate you linking them, and will probably have to use that approach the more I look into it!

Solution
‎05-09-2014 12:21 PM
Occasional Contributor
Posts: 12

Re: INPUT - How to read a single line which is longer than 32767 characters?

Aha! My assertion that 32767 was the maximum length of LRECL was incorrect. Thanks to this post, I've realised I could bump the size of the _infile_ to whatever I want, and this has stopped my issues.

Thank you @Tom !

Valued Guide
Posts: 3,208

Re: INPUT - How to read a single line which is longer than 32767 characters?

I did include some links on reading Json files in your first thread.

They were sessions with papers presented on forum events.

The 32k limit was I already mentioned of a string variable.

With sting handling you should be able to manage that.

Make a pointer to what is scanned and shorten.trim that part.

---->-- ja karman --<-----
Super User
Posts: 9,682

Re: INPUT - How to read a single line which is longer than 32767 characters?

I understand what TOM means. but you also can define this file as a stream file by RECFM=n . and sometime you need to consider to handle JASON object.but for your example. it seems that it is easy to get.

filename x 'c:\temp\test.json';
data want;
 infile x lrecl=3456677 dsd;
 input 
 @'"ID":' id : $40.   
 @'"startTime":'  startTime
 @'"endTime":'  endTime
 @'"name":'  name :$40.
 @'"content":'  content :$40.
@@;
run;
data want1;
 infile x recfm=n dlm='{[",:';
 input x : $100. @@;
run;


Xia Keshan

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 7 replies
  • 1567 views
  • 3 likes
  • 3 in conversation