topic Re: Reading a whole text file in a single row in SAS dataset in SAS Programming

Reading a whole text file in a single row in SAS dataset

ojaro — Mon, 09 May 2022 16:34:38 GMT

Hi all,

I am trying to read a text file that has line breaks. I would like to read the whole file in a single row in SAS dataset. Any pointers to how I could achieve this?

My current code is as follows:

data work.base64;
   length text $32767;
   retain text '';
   infile b64img;
   input;
   text=_infile_;
run;

but the above creates a data set that has a row per line from the source file.

Many thanks in advance,

Olli

Re: Reading a whole text file in a single row in SAS dataset

ErikLund_Jensen — Mon, 09 May 2022 18:45:47 GMT

Hi @ojaro

As long as the whole file will fit into a 32767 byte string, it should be possible, if the file is read as binary stream data (record format = N).

The line breaks are "outside" the file content and are not read, so a line is followed by the next with no delimiters.

filename b64img "c:\temp\xif.txt";
data work.base64;
  infile ind recfm=N lrecl=32767;
  input text $32767.;
run;

Re: Reading a whole text file in a single row in SAS dataset

ojaro — Mon, 09 May 2022 19:06:34 GMT

Thanks @ErikLund_Jensen

The files are likely to be larger than 32767 characters, since they are base64 encoded images.

I am planning to use the content from the file as an input for another text file I am going to use as a JSON payload for proc http. Do you think I could somehow append the text directly to a another file without the character limit? If this is possible, I somehow need to append the whole file in one (not row by row), since the base64 data must go in the JSON as an attribute value, so must be between quotes and with no line breaks.

Re: Reading a whole text file in a single row in SAS dataset

Reeza — Mon, 09 May 2022 19:14:13 GMT

How are you creating your JSON file?

There may be a way to still create the file with having the data on different rows.

Re: Reading a whole text file in a single row in SAS dataset

ojaro — Mon, 09 May 2022 19:19:36 GMT

Hi @Reeza

This is my code so far, but it does not work, since it will end up adding quotes around every line in the image file. Also, if I move the quotes to previous and next data step, I end up with a line break after between the quotes and the base64 image data, which messes up the processing at server side for the payload.

data _null_;
    file payload;
    text = '{"represent": {"algorithmOptions": ["ROC_STANDARD_REPRESENTATION", "ROC_STANDARD_DETECTION", "ROC_ANALYTICS"], "minQuality": -4.0, "k": -1, "minSize": 20, "falseDetectionRate": 0.02, "image":';
    put text+(-1)@;
run;

data _null_;
	file payload mod;
    length text $32767;
    retain text '';
    infile base64img;
	input;
	put '"' _infile_ '"';
run;

data _null_;
    file payload mod;
    text = '}}';
    put text+(-1)@;
run;

Re: Reading a whole text file in a single row in SAS dataset

Reeza — Mon, 09 May 2022 19:24:51 GMT

Ok, if you're doing a data step to create JSON you can definitely create a single item without spaces/line breaks. Instead of asking to read the file as a single line, what's your actual issue that makes you think you require this.

Instead of infile I would read the file in first and then use a SET statement to access that data set and then write the image data.

Re: Reading a whole text file in a single row in SAS dataset

ojaro — Mon, 09 May 2022 19:39:21 GMT

Thanks@Reeza

Well, I did try a version like this as well:

data _null_;
	file payload;
	
	text = '{"request": {"algorithmId": 512, "minQuality": -4.0, "minSize": 20, "falseDetectionRate": 0.02, "k": -1, "image": ';
	put text+(-1)@; 
run;

data work.base64;
   length text $32767;
   retain text '';
   infile b64img;
   input;
   text=_infile_;
run;

data _null_;
	set work.base64;
	file payload mod;
	textf = cats('"',text,'"');
	put textf+(-1)@;
run;

data _null_;
	file payload mod;
	
	text = '}}';
	put text+(-1)@; 
run;

But, I am not quite sure how I need to modify the following, so that it appends the all lines of base64 data within the quotes:

data _null_;
	set work.base64;
	file payload mod;
	textf = cats('"',text,'"');
	put textf+(-1)@;
run;

Any pointers?

Re: Reading a whole text file in a single row in SAS dataset

ErikLund_Jensen — Tue, 10 May 2022 08:01:23 GMT

Hi @ojaro

It should be possible to change your code into one data step. I cannot find time do make a working example until later today, but the principle would be:

First put the leading JSON + double quote to the outfile,
Then read the infile one byte at a time and write it to the outfile until end-of-file is reached,
And then put the finishing double quote + JSON.

Re: Reading a whole text file in a single row in SAS dataset

ojaro — Tue, 10 May 2022 09:57:36 GMT

Thanks @ErikLund_Jensen

I ended up writing some Python to achieve a desired result:

proc fcmp;
declare object py(python);
submit into py;
import json

def writePayload(payload_fn,base64_fn):
    "Output: write_result"
    write_result=-1

    base64_file = open(base64_fn, "r")
    base64_data=base64_file.read().replace("\n", "")
    base64_file.close()

    payload_data='{"represent": {"algorithmOptions": ["ROC_STANDARD_REPRESENTATION", "ROC_STANDARD_DETECTION", "ROC_ANALYTICS"], "minQuality": -4.0, "k": -1, "minSize": 20, "falseDetectionRate": 0.02, "image":"'
    payload_data=payload_data + base64_data
    payload_data=payload_data + '"}}'
    payload_file = open(payload_fn, 'w')
    payload_file.write(payload_data)
    payload_file.close()

    return write_result
endsubmit;
rc = py.publish();
rc = py.call("writePayload",&payload_fn.,&base64_fn.);
write_result = py.results["write_result"];
run;

However, if there is a pure SAS way to do this, I would be still interested in knowing how.

Re: Reading a whole text file in a single row in SAS dataset

Reeza — Tue, 10 May 2022 15:23:11 GMT


data _null_;
	set work.base64 end=eof;
	file payload mod;
       *quote for start of line;
        if _n_=1 then put('"');
       *data written to file;
        put trim(text)+(-1)@;
       *write end quotes;
       if eof then  put('"');
run;

Something like this, roughly?