[PROC HTTP] Truncated response on one very long row

[Pierre2]

Hi all,

I am trying to make POST requests to a webservice that returns some XML content.

To do this, I am using PROC HTTP. Until this point, everything works. I get a response.

proc http in=POST 
out=RESPONSE  headerout=RESP_H ct="application/xml"
url="https://****" 
method="POST" 
/*http_tokenauth*/
WEBUSERNAME="&username" WEBPASSWORD="&password" AUTH_BASIC;
run;

The problem is that the response from this webservice is always stacked on one row. The resulting row/file can be up to 800 Mb large... For instance, this is what the response header says:

HTTP/1.1 200 
Content-Type: application/xml; charset=UTF-8
Content-Length: 665459054

If I now look at the XML file produced by the 'out=RESPONSE' , PROC HTTP seems to truncate the one and only row from the response to 32767 characters, which is I guess linked to the max value of the LRECL option.

If I was doing for instance a GET request through a filename and a data step, I would use something like below, however this is not an option here, since I really need to make a POST request...

filename URL "https://...."
DATA TOTO;
  infile URL recfm=n lrecl=32767;
  input txt: $32767. @@;
run;

What can I do to have the full response written? I could not find any option in the documentation in that regard.

Current version: 9.04.01M3P062415

Thanks in advance for your time!

[ChrisNZ]

@ChrisHemedinger is the source of knowledge for proc http.

[ChrisHemedinger]

I'm surprised that you're finding the response file truncated -- PROC HTTP should be able to download the entire thing...even if it takes some time.

However, reading the file with one large line using DATA step has its own challenges, but there are ways to accomplish that.  Can you verify that the response is indeed cut off?  PROC HTTP now has a useful DEBUG option that can help.

[Pierre2]

Unfortunately the debug option arrived right after 9.4M3 :(.

However I am pretty sure that PROC HTTP is indeed downloading the whole thing because, as you said, it takes a few minutes for it to run. It takes about the same time when I perform the very same POST via other software like cURL or SOAP UI (the difference being that the full file is written...).

So to me the problem seems to be more the way PROC HTTP writes the response...

[ChrisHemedinger]

I feel certain that I've used PROC HTTP to download and work with large files -- for example, a large PDF or ZIP file.  LRECL is a max of 32767.  But you might be able to assign the filename with RECFM=N and basically write as a binary stream.

With such a large XML file (unwieldy, really), I don't think there is much hope of using the XML libname engine to parse -- but it doesn't sound like you're pursuing that.

Chris

[Pierre2]

The problem I believe is more that the whole file is contained only on one row. I managed to download very large files as well in the past, but they were all spread on multiple rows.

I will try to work with RECFM=N, thanks.

Surprisingly enough, the xmlv2 libname engine manages to parse these big files very well. The content of the files is not too highly dimensional though, I think it helps.

[ChrisNZ]

If nothing else works, you can also try to introduce a fake end-of record character to split the data into bits that the data step can save in string variables.

Maybe something not too common (but still appearing at least once every 32k characters) like 'k' or maybe 'x'. See here for frequencies.

Use option TERMSTR= for this.

[Pierre2]

So, I tried to play with different filename options, but nothing seems to help unfortunately. I really believe that whatever is behind the output part of PROC HTTP simply does not handle properly a response with rows longer than 32767 characters...

@ChrisNZ Thanks, but the problem here is that I can't use a data step, since I need to make a POST request. To my knowledge this is only possible via PROC HTTP 

[ChrisNZ]

> Thanks, but the problem here is that I can't use a data step,

True. I started under the assumption TERMSTR could be used on the FILENAME statement, but it is for INFILE only.

[ChrisHemedinger]

Let's bring in @JosephHenry to see if he has time to comment/investigate.

[Pierre2]

Hi,

Any news on this topic? in the meantime I looked for alternatives but with no success 😞

[JosephHenry]

I have not been able to reproduce this.

I have tried with a 128 MB minified XML file, and proc http download the file byte for byte. no truncation.

A few questions.

1.) How are you creating your output fileref?

2.) what encoding is the source and target files?

3.) How are you seeing this truncation? Have you done a manual diff of the file download by proc http and curl?

[Pierre2]

I think that the problem really resides in the fact the the response is stacked only on one row (eg no carriage return or new line characters).

1) Here is my full call, including the filerefs:

filename POST "&g_USER_BASE.\...\POST_IN.xml";
filename RESPONSE "&g_USER_BASE.\...\response_FULL.xml";
filename RESP_H "&g_USER_BASE.\...\response_HEADER.xml";

proc http in=POST 
out=RESPONSE  headerout=RESP_H ct="application/xml"
url="https://.../wsrest/extended/full" 
method="POST" 
/*http_tokenauth*/
WEBUSERNAME="&username" WEBPASSWORD="&password" AUTH_BASIC;
run;

2) Both are UTF-8

3) I can see the truncation by several ways.

 - using cURL I obtain a very large and valid XML file stacked on one row. This is what notepad++ tells me: 

- The response header (both using SAS or cURL give me the same information (see first post)

- The output created by PROC HTTP is also on one line, but only 32676 characters (which is not a random number...).

Also, this response's xml structure is completely messed up (cut right in the middle of a tag for instance), so the resulting XML file is obviously not valid.

[JosephHenry]

I dont think this has anything to do with the data being on only one line. 

PROC HTTP simply writes the bytes that it is given to the output file, it does not care about end of lines or anything like that.

There does seem to be something weird going on though. If you are willing, you can email me at Joseph.Henry@sas.com and I could help you setup some debug logging that could help me diagnose the problem.

[Pierre2 · Accepted solution]

Hi all,

As I just told Joseph via e-mail, I realized that the file was actually fully written on the SAS server, but it was displayed as cut in the EG text editor, which led me to believe the response was not correctly written.

I guess that the XML mapper error I have with the libname engine is thus just a matter of the file being too big as Chris was mentioning earlier...