Filename size limitations

[BillJones]

Hello Everyone,

I'm calling cURL in SAS to perform POST requests. My code runs fine except the post output (json file) gets truncated when I submit larger requests. The lrecl of the post output is 1098304 even though I set both the --max-filesize option in cURL and lrecl in SAS equal to 100M.  I'm running SAS 9.3 on Windows 7 Professional.  Note I need the ability to download larger json files, so simply limiting the size of the output isn't an option.  Also, when I submit the cURL request from the Windows command line, I get the entire file, no truncation.  So this appears to be a SAS/Windows issue.  Again, total record length is limited to 1098304.  Is there some other option that I should tweak to increase the size of the total record length?  Perhaps buffersize?  Any suggestions would be greatly appreciated.  Thanks so much.

Code:

filename data pipe "curl -d @C:\Users\Bill\Desktop\curl.txt -H ""Authorization: OAuth oauth_consumer_key=""""&oauth_consumer_key"""",oauth_token=""""&oauth_token"""",oauth_signature_method=""""&oauth_signature_method."""",

  oauth_signature=""""&oauth_signature."""",oauth_timestamp=""""&oauth_timestamp."""",oauth_nonce=""""&oauth_nonce."""",oauth_version=""""&oauth_version.""""""

  POST -k &url. --max-filesize 100000000" lrecl=100000000;

SAS log excerpt::

NOTE: 1 record was read from the infile DATA.

      The minimum record length was 1098304.

      The maximum record length was 1098304.

      One or more lines were truncated.

[Kurt_Bremser]

How do you actually read the line? (what's your input statement). And think about using the truncover option on the infile statement.

[BillJones]

Kurt,

Thanks for your thoughts.  My input statement is below.  Currently, I'm putting the output to the log, so I can view it.  I could be wrong, but I don't believe the input statement is the issue.  It appears that the data gets truncated when I execute the filename statement.  Note I've examined the output from the filename statement and the total record length is just below 1098304, which is the limit SAS imposes even though I specify lrecl=100M. 

data _null_;

  infile data truncover;

  input;

  put _infile_;

run;

[Kurt_Bremser · Accepted solution]

My next idea would be to store the curl output into a file and then try to read that with SAS. This could end up giving you a clue about further options needed. Since the whole "file" seems to be a stream without record boundaries at all, you would probably need to read it with recfm=n, which is not valid in a filename pipe.

[BillJones]

Kurt,

Your idea works!  I use the x command to call cURL and save the output directly to a text file.  This circumvents any restrictions in SAS.  I can then use proc groovy to parse the data (json file).  Thanks for your help.

-Bill

Code:

x "curl -d @C:\Users\Bill\Desktop\curl.txt -H ""Authorization: OAuth oauth_consumer_key=""""&oauth_consumer_key"""",oauth_token=""""&oauth_token"""",oauth_signature_method=""""&oauth_signature_method."""",
  oauth_signature=""""&oauth_signature."""",oauth_timestamp=""""&oauth_timestamp."""",oauth_nonce=""""&oauth_nonce."""",oauth_version=""""&oauth_version.""""""
  -k &url. -o C:\json.txt";

[FriedEgg]

@BillJones,

If you are going to parse the file with PROC GROOVY, you might as well call the web service with it as well and avoid using cURL and writing the file.  You could probably also read the pipe as a stream in the data step to parse it with SAS.

[BillJones]

@FriedEgg,

Thanks for the suggestions.  By web service do you mean use java via proc groovy to submit a request to the api?   Would this be faster than using cURL?  The reason I use cURL is that I can submit 3 requests simultaneously.

-Bill

[FriedEgg]

If you license SAS/CONNECT then you can execute the multiple calls simultaneously, or spawn multiple instances of SAS either in batch or with systask/x statements, etc....  It also makes the process more transportable.  Far as efficiency, I couldn't necessarily say, but there are fewer I/O operations by avoid the write and read of the JSON file on disk.