Problem with variable length greater 32767

[Ksharp]

NO. actually according to SAS9.2 documentation(dictionary), it would be almost 2G ,

Sorry.Chris

using a file may do a trick.

[ChrisNZ]

Mmm, weird.

The documentation for the LRECL system option states 32kb max, but the doc for the LRECL filename option states 1Gb max for windows and unix.

I missed that change. I wonder why the discrepancy between the 2 lengths. That discrepancy sure threw me off in any case.

No need for streaming then, all good.

Sorry for misleading you DJDaniel.

Thanks for the update KSharp.

Mmm weird again: It looks like sas did an half-baked enhancement. lrecl is longer but then variable _infile_ causes errors.

The following code should be valid but throws an error. What's going on sas?

data T (compress=yes);

  infile TEST lrecl=500000 pad missover termstr=CRLF dlm=';' dsd;

  input;

L=length(_infile_);

run;

yields:

ERROR: The LRECL / LINESIZE for infile TEST exceeds the maximum allowable length for an _INFILE_ or _INFILE_= variable (32,767).

The DATA STEP will not be executed.

so all is not so clear it seems.

[DanielKaiser]

Thanks Ksharp,

that seems to be the right way.

I understand most of it.

But what do i need to change to use an infile not instead of pasting my lines in to the code?

I have 45 flatfile, each has about 4000 lines...

[Ksharp]

Your data looks very complex. You 'd better post a file .

[DanielKaiser]

Here is testfile: https://www.dropbox.com/s/8snrc298se4f2pt/testdata.txt

[Ksharp]

OK.

1) copy your data firstly.

2)replace : with = by using notepad++ or  other text editor.

and due to your complicated situation, you need to deep process data more .

data have;
 infile 'c:\temp\testdata.txt' firstobs=3  lrecl=200000 expandtabs ;
 input (host ports status  index seq os) (= $400.);
run;

Ksharp

[ChrisNZ]

This reads the file you posted directly (no prior editing needed). No long record there though.

data OUT(compress=yes);

infile 'c:\router.txt' lrecl=400 pad missover dlm='09'x;

input  @'Host:' HOST : $200. @'Status:' STATUS : $4. ;

if HOST ne '';

if STATUS='Up' then

   input @'Ports:' PORTS : $200. @'OS:' OS : $200. @'Seq Index:' SEQINDEX : $200. @'Seq:' SEQ : $200. ;

run;

[DanielKaiser]

Thanks Chris, Thanks Ksharp,

chris your output looks good but does not solve my problem that the maximum length of the Ports Variable is at 32k. but in some of the files i have a Lenght of that string thats much longer.

[DanielKaiser]

A combination of your Code and the code of Achilles seems interessting.

[ChrisNZ]

I dont see how Achille's code can work. Variable _infile_ cannot exceed 32k in length.

[ChrisNZ]

Why didn't you upload a file which includes the data that causes you problems?

[DanielKaiser]

You'r right that would have been better,

in this file there are two observations with more than 32767characters

https://www.dropbox.com/s/svyo7c62gg8kdo8/textdaten.txt

[Ksharp]

OK. It waste me lots of time. As I mentioned before, you need process it more after that.

data _null_;
file 'c:\temp\x.txt' lrecl=200000;
infile 'c:\temp\textdaten.txt' lrecl=200000  recfm=n;
input x $char1. @@;     
retain max n;
if x=':' then do;x='=';     n=0;end;
 else if x=',' then do;
                      n+1; max=max(max,n) ;
                           name= cats('p',n,'=');
                           put +(-1) ' ' name @;
                           call symputx('max',max);
                           return;
                         end;
put +(-1) x @;
 if _n_ eq 50000 then stop;
run;
%put &max;

data have;
 infile 'c:\temp\x.txt'  lrecl=200000  firstobs=3  expandtabs ;
 input (host Status ports p1 - p&max state os index seq ) (= $2000.) ;
 if _n_ eq 500 then stop;
run;

Ksharp

[DanielKaiser]

Ohh... it works. much thanks to you both for your time and your knowledge.

It was a really big help to me und i learned a lot too.

[ChrisNZ]

data OUT(compress=yes);

  infile 'c:\router2.txt' lrecl=250000 pad missover dlm='09'x;

  length LASTSTR $40.;

  drop LASTSTR PORTNB LASTCOMMAPOS L;

  array PORTS[8] $32767. ;

  input  @'Host:' HOST : $200. @'Status:' STATUS : $4. ;

  if HOST ne '';

  if STATUS='Up' then

     input @'Ports:'         PORTS1   : $32767.

           @'OS:'            OS       : $200.

           @'Seq Index:'     SEQINDEX : $200.

           @'Seq:'           SEQ      : $200.

           @'Ignored State:' IGNORED  : $20. @;

  L=length(PORTS1);

  PORTNB=1;

  do while (L > 32700) ;                                    * string full, read more;

    LASTCOMMAPOS=find(PORTS[PORTNB],',',-32767);            * look for last comma;

    substr(PORTS[PORTNB],LASTCOMMAPOS+1)='';                * remove last item as it may be truncated;

    LASTSTR=substr(PORTS[PORTNB],length(PORTS[PORTNB])-40); * get string-end;             

    PORTNB+1;                                               * read into next ports variabl

    input @1 @(strip(LASTSTR)) PORTS[PORTNB] : $32767. @;   * read data after string-end;

    L=length(PORTS[PORTNB]);                                * see if new string still full;

end;

run;

Similar to the previous code with extra looping at the end to read the long port string.

Ksharp's code puts each port in a separate variable, this one reads in bulk adn should be much faster.

You'll have to modify the loop to read in smaller chunks if you want.

And please use this to learn and experiment, don't use it blindly.