Help using Base SAS procedures

multipart/form-data REST API POST in PROC HTTP failing

Accepted Solution Solved
Reply
New Contributor
Posts: 4
Accepted Solution

multipart/form-data REST API POST in PROC HTTP failing

Hello everyone,

 

I am having some trouble sending a POST request to a REST API that requires content-type 'multipart/form-data' and some parameters that point to a path containing a JSON file and a type. The cURL equivalent (that works today using Base SAS and an X command calling a shell script) looks like this:

 

curl -X POST -H 'X-API-TOKEN: tokenwenthere' -F 'contacts=@\\path\to\upload\Contact_Upload.json;type=application/json' 'https://co1.qualtrics.com/API/v3/mailinglists/ML_mailinglistid/contactimports' -o '\\path\to\response\response.txt'

However, I want to move this process to Enterprise Guide, which in my instance is connected to a SAS server where cURL doesn't work. With SAS 9.4M3's enhancements to PROC HTTP, I should be able to do this within the procedure. But I am having trouble getting the form data that the call needs as a parameter to work with the procedure. Below is the code that I am running:

 

filename resp2 '\\path\to\apiresponse\response.txt';
filename test '\\path\to\headerout\test.txt';

proc http method="POST"
	url="&link"
	in='contacts=@\\Path\to\upload\Contact_Upload.json;type=application/json'
	ct="multipart/form-data"
	headerout=test
	out=resp2;
	headers
		"X-API-TOKEN"="tokenwenthere";

run;

When I run this code, I get nothing from the API as a response, but the headerout does give me some information:

 

HTTP/1.1 413 Request Entity Too Large
Request-Time: 1
Content-Length: 0
Date: Tue, 17 Jan 2017 02:20:20 GMT
Connection: close

 

On the API documentation, it mentions that a 413 error could be a result of a malformed multipart/form-data request. Which is why I am suspecting that there is a problem within the HTTP procedure with how it handles multipart/form-data requests.

 

Has anyone run in to this problem and can help?


Accepted Solutions
Solution
‎01-19-2017 03:15 PM
SAS Employee
Posts: 7

Re: multipart/form-data REST API POST in PROC HTTP failing

[ Edited ]

Editor's note: See the final working code in this message later in the thread.

 

Like Chris said, unfortunately proc http does not yet natively support sending multipart data.

 

You can pretty easily use the datastep to format a document that follows the mutipart format.

 

For your example you would do somthing like this:

 

filename resp2 '\\path\to\apiresponse\response.txt';
filename test '\\path\to\headerout\test.txt';
filename in TEMP;
/* Set the boundary identifier. It does not matter what this is, as long as its unique */
%let boundary=thisisauniqueid; 

data _null_;
infile test end=eof;
file in;

/* for each file we are sending, we need to add some special headers at the beginning*/
if _n_ = 1 then
 do;
  put "--&boundary."; /* This separates each data piece as a separate entity. Must start with -- */
  put 'Content-Disposition: form-data; name="file1"; filename="file1.txt"'; /* This is the content identifier */
  put "Content-Type: text/plain"; /* This is the acutaly content type of the entity in the multi-part form*/
  put ; /* Must end with a CRLF signaling that what comes next is the actual entity */
end;
input;
put _infile_; /* add the actual file to be sent*/
/* the end of the multipart blob needs to be terminated */
if eof then
 do;
   put ; /* Must have a CRLF*/
   put "--&boundary--"; /* must start and end with --*/
 end;
run;

/* now that the file is created, we can send it to our server*/
proc http url="&link."
 in=in
 ct="multipart/form-data; boundary=&boundary."
 out=out;
 run;

If you are sending multiple files, the code is pretty similar, but you would need to do multiple datasteps.

 

The first datastep you would be identical to what is above, except you would leave off the EOF bit.

 

for Subsequent files, you would need to add the "mod" option for your file and also add a CRLF before the boundary. so just do a put;

 

The file would then just need to be terminated like it was ealier. This would be pretty easy to wrap into a few macros. If you need a full example of sending multiple files, just let me know.

 

Hope this helps.

View solution in original post


All Replies
Community Manager
Posts: 2,757

Re: multipart/form-data REST API POST in PROC HTTP failing

According to the PROC HTTP developer, the procedure does not (yet) support multipart form data.   You can achieve what you want (I think) by using DATA step to create the multipart form as a file, and then use the IN= parameter to point to that fileref.

 

I'm sorry I don't have an example.  I've asked the developer if he can come up with one -- if that happens, we'll post back.

Solution
‎01-19-2017 03:15 PM
SAS Employee
Posts: 7

Re: multipart/form-data REST API POST in PROC HTTP failing

[ Edited ]

Editor's note: See the final working code in this message later in the thread.

 

Like Chris said, unfortunately proc http does not yet natively support sending multipart data.

 

You can pretty easily use the datastep to format a document that follows the mutipart format.

 

For your example you would do somthing like this:

 

filename resp2 '\\path\to\apiresponse\response.txt';
filename test '\\path\to\headerout\test.txt';
filename in TEMP;
/* Set the boundary identifier. It does not matter what this is, as long as its unique */
%let boundary=thisisauniqueid; 

data _null_;
infile test end=eof;
file in;

/* for each file we are sending, we need to add some special headers at the beginning*/
if _n_ = 1 then
 do;
  put "--&boundary."; /* This separates each data piece as a separate entity. Must start with -- */
  put 'Content-Disposition: form-data; name="file1"; filename="file1.txt"'; /* This is the content identifier */
  put "Content-Type: text/plain"; /* This is the acutaly content type of the entity in the multi-part form*/
  put ; /* Must end with a CRLF signaling that what comes next is the actual entity */
end;
input;
put _infile_; /* add the actual file to be sent*/
/* the end of the multipart blob needs to be terminated */
if eof then
 do;
   put ; /* Must have a CRLF*/
   put "--&boundary--"; /* must start and end with --*/
 end;
run;

/* now that the file is created, we can send it to our server*/
proc http url="&link."
 in=in
 ct="multipart/form-data; boundary=&boundary."
 out=out;
 run;

If you are sending multiple files, the code is pretty similar, but you would need to do multiple datasteps.

 

The first datastep you would be identical to what is above, except you would leave off the EOF bit.

 

for Subsequent files, you would need to add the "mod" option for your file and also add a CRLF before the boundary. so just do a put;

 

The file would then just need to be terminated like it was ealier. This would be pretty easy to wrap into a few macros. If you need a full example of sending multiple files, just let me know.

 

Hope this helps.

Community Manager
Posts: 2,757

Re: multipart/form-data REST API POST in PROC HTTP failing

Thanks to @JosephHenry for the response!  I'll add one tip: you can get SAS to create a unique ID for you:

 

%let boundary=%sysfunc(uuidgen());
New Contributor
Posts: 4

Re: multipart/form-data REST API POST in PROC HTTP failing

Thanks so much for your help! It was very useful, and I am glad to know that form data submission is not natively supported.

 

Unfortunately, the solution provided did not work. I am getting the same 413 error that I had gotten previously. The form data that I need to submit to the API server is:

"contacts=\\path\to\contacts.contacts.json; type=application\json", not the file that contains the upload information. So this is the form data that the API requires as parameters.

 

Is there a way to use a similar process to be able to format these two parameters in the in= statement so that proc http sends them over in a multipart/form-data format that the API can recognize?

 

Thanks again for your help!

Community Manager
Posts: 2,757

Re: multipart/form-data REST API POST in PROC HTTP failing

You're using this API, right?

 

It's new to me, but my guess is that the cURL example uses the -F with the @ notation so you can reference a local file with your contacts (JSON format).  When you run cURL, the tool reads the content of that local file and packages it up for transport to the API.  That is, the API server won't be able to see/read your local JSON file directly, right?  

 

With the DATA step approach shown by @JosephHenry, you're basically doing that step that cURL would have done for you.  You would use DATA step to read the local JSON file, and echo its contents and sandwich it between the content headers/metadata, then use that as the IN= on your HTTP POST.  Don't forget the METHOD="POST" (in your original code, but wasn't in Joseph's example).

 

 

SAS Employee
Posts: 7

Re: multipart/form-data REST API POST in PROC HTTP failing

If you run the cURL command with the option --trace and copy me the output, I could modify the example to match.


New Contributor
Posts: 4

Re: multipart/form-data REST API POST in PROC HTTP failing

[ Edited ]

Happily, that will not be necessary! I was able to get it to work with a couple of small modifications. I will share them below:

 

@ChrisHemedinger's response about what the data _null_ step was doing in relation to cURL was the big aha moment. At that point, we needed to translate both the path parameter and the type parameter that was being passed in to the form to something that worked within the data _null_ framework that you presented. So, the first thing I did was took Chris's suggestion and incorporated that into the boundary macro variable:

/*Set up the form parameters*/

%let boundary=%sysfunc(uuidgen());

Then, the next important part was to make sure that the name passed through the block of metadata matched the name of the parameter that the API was looking for and that there was a matching file name. Since the data _null_ was doing the unpacking for me, I just needed to make sure that the data had the same parameter name that the API was looking for. 

data _null_;
infile Contact /*This represents the json file containing the contacts and was defined in an earlier fileref*/ 
end=eof;
file in;

if _n_ = 1 then do;
	put "--&boundary.";
	put 'Content-Disposition: form-data; name="contacts"; filename="contact_upload.json"';
	put 'Content-Type: application/json';
	put ;
end;
input;
put _infile_;
put ;
if eof then do;
	put ;
	put "--&boundary.--";
end;
run;

 

And then I needed to specify the type. You had already put the header that specified the content type in the program, it just needed to be changed to a different type.

 

With that it works! Thank you both so much, this will be a huge time saver! For those interested, below is the full code:

 

filename resp2 '\\path\to\APIoutput\response.txt';
filename in TEMP;

/*Set up the form parameters*/

%let boundary=%sysfunc(uuidgen());

data _null_;
infile Contact /*This represents the json file containing the upload data and was defined in an earlier fileref*/ 
end=eof;
file in;

if _n_ = 1 then do;
	put "--&boundary.";
	put 'Content-Disposition: form-data; name="contacts"; filename="contact_upload.json"';
	put 'Content-Type: application/json';
	put ;
end;
input;
put _infile_;
put ;
if eof then do;
	put ;
	put "--&boundary.--";
end;
run;

proc http method="POST"
	url="&link"
	in=in
	ct="multipart/form-data;boundary=&boundary."
	out=resp2;
	headers
		"X-API-TOKEN"="tokenwenthere";

run;

 

Community Manager
Posts: 2,757

Re: multipart/form-data REST API POST in PROC HTTP failing

Great job team!  And thanks @KTanner for posting the final solution back into the thread.

 

High fives all around!

☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 8 replies
  • 1625 views
  • 10 likes
  • 3 in conversation