topic Re: read a text file character by character on the fly in SAS Programming

read a text file character by character on the fly

janpeter — Wed, 12 Jun 2019 13:55:32 GMT

Hi,

im trying to read a large non delimited text file into a dataset. Im using PC SAS 9.4.

The file is the document.xml part from a docx file. It does not seem to be possible to read it into one variable and one row in a dataset since it is too large (>300kb). I was thinking that one way of doing this is to pre process the file by reading it character by character and add a CR (carriage return) every time i see a '>'. Then output it and re-read it by using proc import with CR ('0D0A'x) as a delimiter.

Can this be done. If yes then how?

Jan

p.s. note that reading the file with a XML libname is not useful here.

Re: read a text file character by character on the fly

Reeza — Wed, 12 Jun 2019 15:30:51 GMT

What happens if you do try to read it in as one string?

data test;
infile 'path to xml' lrecl=32000;
input;
length x $32000.;
x=_infile_;
run;

Re: read a text file character by character on the fly

Tom — Wed, 12 Jun 2019 15:53:05 GMT

Not sure if it will help but here is how to do what you asked.

filename in 'document.xml';
filename out 'document_fixed.xml';

data _null_;
  infile in recfm=n;
  file out recfm=n;
  input char $char1. ;
  put char $char1. ;
  if char='>' then put '0D'x;
run;

I don't think that UTF-8 (or other multibyte character sets) would make any difference.

Re: read a text file character by character on the fly

Reeza — Wed, 12 Jun 2019 16:14:29 GMT

If you remind me later, I did this parsing out a Word doc before. If that's what you're trying to do by the way, I ended up using a python library instead (called from SAS) that worked very well to parse the contents from word documents.But I can dig out the code I had been working with at least.

Re: read a text file character by character on the fly

janpeter — Wed, 12 Jun 2019 17:06:17 GMT

Hi Reeza,

that certainly did the trick. Thanks a lot!

Yes. I was considering Python as an option but prefer to keep it all in SAS.

Jan

Re: read a text file character by character on the fly

janpeter — Wed, 12 Jun 2019 17:11:01 GMT

Sorry i was too fast. I of course meant to write Tom

Thanks Tom!!!

😉

Re: read a text file character by character on the fly

data_null__ — Wed, 12 Jun 2019 19:38:05 GMT

@Tom wrote:

Not sure if it will help but here is how to do what you asked.
filename in 'document.xml';
filename out 'document_fixed.xml';

data _null_;
  infile in recfm=n;
  file out recfm=n;
  input char $char1. ;
  put char $char1. ;
  if char='>' then put '0D'x;
run;
I don't think that UTF-8 (or other multibyte character sets) would make any difference.

I wonder if it would be faster to read the file as if were fixed length and apply TRANSLATE function to _INFILE_.

Re: read a text file character by character on the fly

Tom — Wed, 12 Jun 2019 19:42:17 GMT

Perhaps with TRANWRD() function since this is inserting an extra character.