Help using Base SAS procedures

Filename size limitations

Accepted Solution Solved
Reply
Frequent Contributor
Posts: 92
Accepted Solution

Filename size limitations

Hello Everyone,

I'm calling cURL in SAS to perform POST requests. My code runs fine except the post output (json file) gets truncated when I submit larger requests. The lrecl of the post output is 1098304 even though I set both the --max-filesize option in cURL and lrecl in SAS equal to 100M.  I'm running SAS 9.3 on Windows 7 Professional.  Note I need the ability to download larger json files, so simply limiting the size of the output isn't an option.  Also, when I submit the cURL request from the Windows command line, I get the entire file, no truncation.  So this appears to be a SAS/Windows issue.  Again, total record length is limited to 1098304.  Is there some other option that I should tweak to increase the size of the total record length?  Perhaps buffersize?  Any suggestions would be greatly appreciated.  Thanks so much.


Code:

filename data pipe "curl -d @C:\Users\Bill\Desktop\curl.txt -H ""Authorization: OAuth oauth_consumer_key=""""&oauth_consumer_key"""",oauth_token=""""&oauth_token"""",oauth_signature_method=""""&oauth_signature_method."""",

  oauth_signature=""""&oauth_signature."""",oauth_timestamp=""""&oauth_timestamp."""",oauth_nonce=""""&oauth_nonce."""",oauth_version=""""&oauth_version.""""""

  POST -k &url. --max-filesize 100000000" lrecl=100000000;

SAS log excerpt::

NOTE: 1 record was read from the infile DATA.

      The minimum record length was 1098304.

      The maximum record length was 1098304.

      One or more lines were truncated.


Accepted Solutions
Solution
‎03-09-2015 03:13 AM
Super User
Posts: 7,762

Re: Filename size limitations

Posted in reply to BillJones

My next idea would be to store the curl output into a file and then try to read that with SAS. This could end up giving you a clue about further options needed. Since the whole "file" seems to be a stream without record boundaries at all, you would probably need to read it with recfm=n, which is not valid in a filename pipe.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers

View solution in original post


All Replies
Super User
Posts: 7,762

Re: Filename size limitations

Posted in reply to BillJones

How do you actually read the line? (what's your input statement). And think about using the truncover option on the infile statement.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Frequent Contributor
Posts: 92

Re: Filename size limitations

Posted in reply to KurtBremser

Kurt,

Thanks for your thoughts.  My input statement is below.  Currently, I'm putting the output to the log, so I can view it.  I could be wrong, but I don't believe the input statement is the issue.  It appears that the data gets truncated when I execute the filename statement.  Note I've examined the output from the filename statement and the total record length is just below 1098304, which is the limit SAS imposes even though I specify lrecl=100M. 


data _null_;

  infile data truncover;

  input;

  put _infile_;

run;

Solution
‎03-09-2015 03:13 AM
Super User
Posts: 7,762

Re: Filename size limitations

Posted in reply to BillJones

My next idea would be to store the curl output into a file and then try to read that with SAS. This could end up giving you a clue about further options needed. Since the whole "file" seems to be a stream without record boundaries at all, you would probably need to read it with recfm=n, which is not valid in a filename pipe.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Frequent Contributor
Posts: 92

Re: Filename size limitations

Posted in reply to KurtBremser

Kurt,

Your idea works!  I use the x command to call cURL and save the output directly to a text file.  This circumvents any restrictions in SAS.  I can then use proc groovy to parse the data (json file).  Thanks for your help.

-Bill

Code:

x "curl -d @C:\Users\Bill\Desktop\curl.txt -H ""Authorization: OAuth oauth_consumer_key=""""&oauth_consumer_key"""",oauth_token=""""&oauth_token"""",oauth_signature_method=""""&oauth_signature_method."""",
  oauth_signature=""""&oauth_signature."""",oauth_timestamp=""""&oauth_timestamp."""",oauth_nonce=""""&oauth_nonce."""",oauth_version=""""&oauth_version.""""""
  -k &url. -o C:\json.txt";

Trusted Advisor
Posts: 1,301

Re: Filename size limitations

Posted in reply to BillJones

@BillJones,

If you are going to parse the file with PROC GROOVY, you might as well call the web service with it as well and avoid using cURL and writing the file.  You could probably also read the pipe as a stream in the data step to parse it with SAS.

Frequent Contributor
Posts: 92

Re: Filename size limitations

@FriedEgg,

Thanks for the suggestions.  By web service do you mean use java via proc groovy to submit a request to the api?   Would this be faster than using cURL?  The reason I use cURL is that I can submit 3 requests simultaneously.

-Bill

Trusted Advisor
Posts: 1,301

Re: Filename size limitations

Posted in reply to BillJones

If you license SAS/CONNECT then you can execute the multiple calls simultaneously, or spawn multiple instances of SAS either in batch or with systask/x statements, etc....  It also makes the process more transportable.  Far as efficiency, I couldn't necessarily say, but there are fewer I/O operations by avoid the write and read of the JSON file on disk.

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 7 replies
  • 523 views
  • 3 likes
  • 3 in conversation