Re: JSON input in proc http

SGSP · Posted 05-10-2021 10:05 AM

Hello,

I've searched around for a similar problem, so if I've missed something, please let me know.

I'm trying to make proc http work with an restAPI. I am able to do it via curl and Python, so the API itself should be fine. Since this is a local web-service, it'll be hard to replicate the error, but let me know if there's anything I can do to make the description more thorough.

I'm using SAS EG 8.1. This is what I've tried so far:

Attempt number 1:

filename json_in "C:\Temp\json_in.json";
data _null_;
file json_in;
put '{';
put '"valueDate": "2021-05-10T04:32:34.394Z",';
put '"inputSpecification": "TESTINP",';
put '"name": "FOO",';
put '"parameters": [';
put '{';
put '"name": "Parameter1",';
put '"title": "My Parameter",';
put '"value": "coverstat1"';
put '}';
put '],';
put '"configuration": "TESTCONF",';
put '"scenarios": [';
put '"Scen1"';
put '],';
put '"outputDestination": "Database"';
put '}';
run;
filename out "C:\temp\out.txt";
proc http url="http://blah/api/v2/Batches" 
METHOD="POST"
in="C:\temp\json_in.json"
ct="application/json"
out=out;
headers "accept"="application/json" 
/*"Content-Type"="application/json"*/;
run;

Attempt number 2:

proc http METHOD="POST" 
url="http://blah/api/v2/Batches" 
ct="application/json"
in="{ ""valueDate"": ""2021-05-10T04:32:34.394Z"", ""inputSpecification"": ""TESTINP"", ""name"": ""FOO"", ""configuration"": ""TESTCONF"", ""scenarios"": {[""SCEN1""]}, ""parameters"": [{""name"": ""Parameter1"", ""title"": ""MyParameter"", ""value"": ""coverstat1""}]"
ct="application/json"
out=out;
headers "accept"="application/json";
run;

Both of these attempt yield the same error from the API:

{"detail":"Value cannot be null.\r\nParameter name: source"}

Apparently this has to do with the parameter-input being multi-dimensional. Is there any way of handling this in SAS?

For completeness, this is the Python-code that works:

import requests

params = {
  "valueDate": "2021-05-10T04:32:34.394Z",
  "inputSpecification": "TESTINP",
  "name": "FOO",
  "parameters": [
    {
      "name": "Parameter1",
      "title": "My Parameter",
      "value": "coverstat1"
    }
  ],
  "configuration": "TESTCONF",
  "scenarios": [
    "SCEN1"
  ],
  "outputDestination": "Database"
}
url = 'http://blah/api/v2/Batches'
headers = {'accept': 'application/json'}
response = requests.post(url, json=params, headers=headers)

print(response.json())

I know it's not ideal, that the webservice can't be accessed by people here, but hopefully I've done enough to describe the problem.

Thanks in advance.

ChrisHemedinger · Posted 05-10-2021 10:45 AM

The IN= accepts either a fileref (no quotes, not a literal filename) or the content of the payload you want to POST. So your first example should be:

proc http url="http://blah/api/v2/Batches" 
 METHOD="POST"
 in=json_in
 ct="application/json"
 out=out;
 headers "accept"="application/json" 
run;

Learn from the Experts! Check out the huge catalog of free sessions in the Ask the Expert webinar series.

SGSP · Posted 05-10-2021 12:05 PM

Thanks for your reply. I did try with the direct reference to json_in as well. Still the same error.

Oligolas · Posted 05-10-2021 10:50 AM

Hi,

the code itself seems to work; when I'm using url="http://httpbin.org/put" I get no errors.

correction: url="http://httpbin.org/post"

________________________
- Cheers -

SGSP · Posted 05-11-2021 02:08 AM

This is weird. When I do the same thing, I get the following reply:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>

I'm guessing this also has to do with the multi-dimensional parameters input. Could it be something to do with the version I'm running? (SAS EG 8.1)

Kurt_Bremser · Posted 05-11-2021 04:42 AM

That is a message you get from the website. The particular URL does not allow a PUT method for the HTTP request. It may be that you use the URL of a page where the actual PUT request is triggered through a button on that page.

Maxims of Maximally Efficient SAS Programmers
How to convert datasets to data steps
The macro for direct download as ZIP
How to post code
Please vote for Provide Sequential Search Capability for Hash Objects
How to deal with locked files on UNIX

Oligolas · Posted 05-11-2021 05:01 AM

Sorry my bad the URL for test should be

url="http://httpbin.org/post"

________________________
- Cheers -

SGSP · Posted 05-11-2021 08:48 AM

Thanks for showing me that website, I'm not used to working with SAS or APIs, and that'll be a big help.

I still haven't found a solution, however, but I have a bit more information. When I run the following

filename out "\\sas00750\analyse3\AKF\Data\Personlige\SIG\out.txt";
filename json_in "\\sas00750\analyse3\AKF\Data\Personlige\SIG\json_in.json";
data _null_;
file json_in recfm=f lrecl=1;
put '{"valueDate": "2021-05-10T04:32:34.394Z", "inputSpecification": "TESTINP", "name": "FOO", "parameters": [{"name": "Parameter1", "title": "My Parameter", "value": "coverstat1"}], "configuration": "TESTCONF", "scenarios": ["SCEN1"], "outputDestination": "Database"}';
run;
proc http METHOD="POST" 
url="http://httpbin.org/post" 
ct="application/json"
in=json_in 
ct="application/json"
out=out;
headers "accept"="application/json";
run;

I get this response:

{
  "args": {}, 
  "data": "{\"valueDate\": \"2021-05-10T04:32:34.394Z\", \"inputSpecification\": \"TESTINP\", \"name\": \"FOO\", \"parameters\": [{\"name\": \"Parameter1\", \"title\": \"My Parameter\", \"value\": \"coverstat1\"}], \"configuration\": \"TESTCONF\", \"scenarios\": [\"SCEN1\"], \"outputDestination\": \"Database\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "application/json", 
    "Content-Length": "263", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "SAS/9", 
    "X-Amzn-Trace-Id": "Root=1-609a7799-4b92fc8f27fb3ea812533140"
  }, 
  "json": {
    "configuration": "TESTCONF", 
    "inputSpecification": "TESTINP", 
    "name": "FOO", 
    "outputDestination": "Database", 
    "parameters": [
      {
        "name": "Parameter1", 
        "title": "My Parameter", 
        "value": "coverstat1"
      }
    ], 
    "scenarios": [
      "SCEN1"
    ], 
    "valueDate": "2021-05-10T04:32:34.394Z"
  }, 
  "origin": "", 
  "url": "http://httpbin.org/post"
}

However, when I run the Python code that works (see original post) I get this:

{
  'args': {}, 
  'data': '{"valueDate": "2021-05-10T04:32:34.394Z", "inputSpecification": "TESTINP", "name": "FOO", "parameters": [{"name": "Parameter1", "title": "My Parameter", "value": "coverstat1"}], "configuration": "TESTCONF", "scenarios": ["SCEN1"], "outputDestination": "Database"}', 
  'files': {}, 
  'form': {}, 
  'headers': {
    'Accept': 'application/json', 
    'Accept-Encoding': 'gzip, deflate', 
    'Content-Length': '263', 
    'Content-Type': 'application/json', 
    'Host': 'httpbin.org', 
    'User-Agent': 'python-requests/2.25.1', 
    'X-Amzn-Trace-Id': 'Root=1-609a7993-516acbb91c5c1f902ee96668'
  }, 
  'json': {
    'configuration': 'TESTCONF', 
    'inputSpecification': 'TESTINP', 
    'name': 'FOO', 
    'outputDestination': 'Database', 
    'parameters': [
      {
        'name': 'Parameter1', 
        'title': 'My Parameter', 
        'value': 'coverstat1'
      }
    ], 
    'scenarios': [
      'SCEN1'
    ], 
    'valueDate': '2021-05-10T04:32:34.394Z'
  }, 
  'origin': '', 
  'url': 'http://httpbin.org/post'
}

Note: When I replace the single quotes with double quotes and vice versa, in the SAS program, then it doesn't seem like httpbin interprets it as json.

I'm wondering if it could be an encoding issue, since the python request includes 'Accept-Encoding': 'gzip, deflate'? Is there a similar option to set in SAS?

Thanks for your help.

ChrisHemedinger · Posted 05-11-2021 09:05 AM

You already have a HEADERS statement, and you could add to that:

headers "accept"="application/json"
        "Accept-Encoding"="gzip, deflate";

Learn from the Experts! Check out the huge catalog of free sessions in the Ask the Expert webinar series.

SGSP · Posted 05-11-2021 09:59 AM

Thanks 🙂

This unfortunately doesn't solve the problem. But now the only remaining difference seems to be the double quotes/single quotes. Perhaps the application only accepts the single quotes. Does anyone have an idea as to why the response from httpbin is in single quotes when I use Python or curl and in double quotes when I use SAS?

Oligolas · Posted 05-11-2021 10:12 AM

JSON is standardized to use double-quotes, to the degree that this would not be considered standard ...

________________________
- Cheers -

SGSP · Posted 05-11-2021 10:47 AM

That's true, but what confuses me is that the response from httpbin is in single-quotes when I use Python or curl, even though my json input is in double-quotes. Then, when I use SAS, the response is in double-quotes.

Oligolas · Posted 05-11-2021 11:50 AM

It's confusing to me too

________________________
- Cheers -

Tom · Posted 05-11-2021 12:50 PM

I don't see any parameter named "source" in your JSON.

Are you sure you are sending the same JSON as you send with the other methods?

SGSP · Posted 05-11-2021 01:23 PM

If I have to be completely honest, I’m not entirely sure about anything. I’m a bit of a novice with both SAS and APIs.

But going by the responses from httpbin that I poster earlier, it seems like the only difference between the response for the Python request and the response for the SAS request is the quotation marks. At least after I added the header.

SAS Innovate 2025: Save the Date

SAS Training: Just a Click Away