BookmarkSubscribeRSS Feed
ojaro
SAS Employee

Hi all,

I am trying to read a text file that has line breaks. I would like to read the whole file in a single row in SAS dataset. Any pointers to how I could achieve this?

My current code is as follows:

 

data work.base64;
   length text $32767;
   retain text '';
   infile b64img;
   input;
   text=_infile_;
run;

but the above creates a data set that has a row per line from the source file.

Many thanks in advance,

Olli

9 REPLIES 9
ErikLund_Jensen
Rhodochrosite | Level 12

Hi @ojaro 

 

As long as the whole file will fit into a 32767 byte string, it should be possible, if the file is read as binary stream data (record format = N).

The line breaks are "outside" the file content and are not read, so a line is followed by the next with no delimiters.

filename b64img "c:\temp\xif.txt";
data work.base64;
  infile ind recfm=N lrecl=32767;
  input text $32767.;
run;

 

 

ojaro
SAS Employee

Thanks @ErikLund_Jensen 

The files are likely to be larger than 32767 characters, since they are base64 encoded images.

I am planning to use the content from the file as an input for another text file I am going to use as a JSON payload for proc http. Do you think I could somehow append the text directly to a another file without the character limit? If this is possible, I somehow need to append the whole file in one (not row by row), since the base64 data must go in the JSON as an attribute value, so must be between quotes and with no line breaks.

Reeza
Super User
How are you creating your JSON file?

There may be a way to still create the file with having the data on different rows.

ojaro
SAS Employee

Hi @Reeza 

 

This is my code so far, but it does not work, since it will end up adding quotes around every line in the image file. Also, if I move the quotes to previous and next data step, I end up with a line break after between the quotes and the base64 image data, which messes up the processing at server side for the payload.

 

data _null_;
    file payload;
    text = '{"represent": {"algorithmOptions": ["ROC_STANDARD_REPRESENTATION", "ROC_STANDARD_DETECTION", "ROC_ANALYTICS"], "minQuality": -4.0, "k": -1, "minSize": 20, "falseDetectionRate": 0.02, "image":';
    put text+(-1)@;
run;

data _null_;
	file payload mod;
    length text $32767;
    retain text '';
    infile base64img;
	input;
	put '"' _infile_ '"';
run;

data _null_;
    file payload mod;
    text = '}}';
    put text+(-1)@;
run;
Reeza
Super User

Ok, if you're doing a data step to create JSON you can definitely create a single item without spaces/line breaks. Instead of asking to read the file as a single line, what's your actual issue that makes you think you require this.

Instead of infile I would read the file in first and then use a SET statement to access that data set and then write the image data.

ojaro
SAS Employee

Thanks@Reeza 

 

Well, I did try a version like this as well:

 

data _null_;
	file payload;
	
	text = '{"request": {"algorithmId": 512, "minQuality": -4.0, "minSize": 20, "falseDetectionRate": 0.02, "k": -1, "image": ';
	put text+(-1)@; 
run;

data work.base64;
   length text $32767;
   retain text '';
   infile b64img;
   input;
   text=_infile_;
run;

data _null_;
	set work.base64;
	file payload mod;
	textf = cats('"',text,'"');
	put textf+(-1)@;
run;

data _null_;
	file payload mod;
	
	text = '}}';
	put text+(-1)@; 
run;

But, I am not quite sure how I need to modify the following, so that it appends the all lines of base64 data within the quotes:

 

data _null_;
	set work.base64;
	file payload mod;
	textf = cats('"',text,'"');
	put textf+(-1)@;
run;

Any pointers?

ErikLund_Jensen
Rhodochrosite | Level 12

Hi @ojaro 

 

It should be possible to change your code into one data step. I cannot find time do make a working example until later today, but the principle would be:

 

  • First put the leading JSON + double quote to the outfile,
  • Then read the infile one byte at a time and write it to the outfile until end-of-file is reached,
  • And then put the finishing double quote + JSON. 

 

ojaro
SAS Employee

Thanks @ErikLund_Jensen 

 

I ended up writing some Python to achieve a desired result:

proc fcmp;
declare object py(python);
submit into py;
import json

def writePayload(payload_fn,base64_fn):
    "Output: write_result"
    write_result=-1

    base64_file = open(base64_fn, "r")
    base64_data=base64_file.read().replace("\n", "")
    base64_file.close()

    payload_data='{"represent": {"algorithmOptions": ["ROC_STANDARD_REPRESENTATION", "ROC_STANDARD_DETECTION", "ROC_ANALYTICS"], "minQuality": -4.0, "k": -1, "minSize": 20, "falseDetectionRate": 0.02, "image":"'
    payload_data=payload_data + base64_data
    payload_data=payload_data + '"}}'
    payload_file = open(payload_fn, 'w')
    payload_file.write(payload_data)
    payload_file.close()

    return write_result
endsubmit;
rc = py.publish();
rc = py.call("writePayload",&payload_fn.,&base64_fn.);
write_result = py.results["write_result"];
run;

However, if there is a pure SAS way to do this, I would be still interested in knowing how.

Reeza
Super User

data _null_;
	set work.base64 end=eof;
	file payload mod;
       *quote for start of line;
        if _n_=1 then put('"');
       *data written to file;
        put trim(text)+(-1)@;
       *write end quotes;
       if eof then  put('"');
run;

Something like this, roughly?

Ready to join fellow brilliant minds for the SAS Hackathon?

Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.

Register today!
How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 9 replies
  • 3597 views
  • 3 likes
  • 3 in conversation