topic Re: using http methods in SAS ds2 in SAS Programming

using http methods in SAS ds2

noffer — Fri, 12 Jan 2024 21:23:49 GMT

Hi, I have a working ds2 prototype that does http posts to a service. Here's the working code:

    /* HTTP POST in DS2, requests sent iteratively */
	proc ds2;
	data _null_;
		method init();
		end;
		method call_service(varchar(256) consUrlBase, varchar(256) uuid, varchar(256) brand, varchar(100) definition, varchar(2048) auth_token);
			filename response temp;
			filename request temp;
			dcl varchar(2050) authHeader;
			dcl varchar(1024) consUrl;
			dcl varchar(255) consPart jsonBody;
    		dcl package http h();
			dcl varchar(1024) body;
			dcl int status;
			dcl varchar(1024) contentType;

			
			/* set a couple variables*/

			/* use http package to call service */
			h.createPostMethod(consUrl);
			h.setOAuthToken(authHeader);
			/* set a couple headers */
			h.setRequestContentType('application/json; charset=utf-8');
			h.setRequestBodyAsString(jsonBody);
			h.executeMethod();
			status = h.getStatusCode();
			if status ^= 201 then do;
      			put 'ERROR: ' status ' response calling ' consUrl;
			end;
			h.delete();
			end;
		method run(); /* run step executes for each row in dataset */
			set work_dir.&dsname._&currdate.;
			call_service(cons_url_base, UUID, brnd_name, definition, auth_token);
			end;
		method term();
			put 'Done with ds2 logic!';
			end;
	enddata;
	run;

So the code executes and I get the 201 response I'm expecting. Except I'm also getting the following warning...

WARNING: Warning reported by DS2 package d2http:
WARNING: Aborted execution of HTTP POST method with URL:

After 500 messages (or so it seems), I get:

ERROR: Maximum number of error messages that can be recorded has been exceeded.

I'll routinely process thousands of records, so looking for a way to suppress the warning as it is causing the program to fail.

I'm also slightly suspicious of the dcl package http h(); ... maybe that should just be declared globally? I'm not seeing any performance improvement by moving it. (real time was slower, cpu time was less) I'm just following the sample code found here and other posts, so open to any guidance.

Re: using http methods in SAS ds2

Tom — Sat, 13 Jan 2024 16:53:15 GMT

What is the error message you are getting?

Re: using http methods in SAS ds2

noffer — Mon, 15 Jan 2024 14:41:44 GMT

That's the fun part. I'm getting warnings, but at some point its hitting some limit, and displaying the message I've included.

Re: using http methods in SAS ds2

Tom — Mon, 15 Jan 2024 16:31:42 GMT

Is there any reason why you are using DS2 for this?

Why do you seem to be making a new HTTP object for each observation? Can't you reuse a single object? Are you trying to have DS2 make multiple HTTP requests at the same time?

Re: using http methods in SAS ds2

noffer — Mon, 15 Jan 2024 18:08:59 GMT

Using ds2 as this will move into a multi-threaded approach like here. Yes, I thought to move the instantiation of the http object to a global var, but doesn't seem to make a performance difference.

And why not use a proc http? That will make all calls at once, so if I have to make 10k calls, I wouldn't want to make all calls at once.

Re: using http methods in SAS ds2

Patrick — Tue, 16 Jan 2024 00:34:12 GMT

Are you sure your server can handle that many calls in a short time or that the server doesn't qualify these calls as a potential DoS attack and blocks you?

It might be worth considering to "combine" the calls and then sort out the response file.

Re: using http methods in SAS ds2

noffer — Tue, 16 Jan 2024 15:59:05 GMT

Thanks for you reply Patrick. The web service currently processes 10-20k daily in bulk batches. Sometimes a couple hundred thousand, which is why there's some motivation to streamline the process. The current process, written years ago dumps the data to a file, which is then picked up by an external program that iterates through the file. The ask is to move the logic into a multi-threaded SAS proc. It doesn't help much if it throws an error after 500 calls. I've opened a ticket with SAS support to see if there's any options that they can offer.

Re: using http methods in SAS ds2

Tom — Tue, 16 Jan 2024 16:05:43 GMT

With such volume it is probably worth exploring a different path.

Can't you just have them create a report that runs directly on the data without the HTTP interface?

That would be much less burden on everyone (and every system too).

Re: using http methods in SAS ds2

noffer — Tue, 16 Jan 2024 20:58:37 GMT

Solution was to add the following just above the h.delete() ....

h.abortMethod();