BookmarkSubscribeRSS Feed
gcardozo
Calcite | Level 5

Hi experts,

 

I'm pulling some data from a REST API using proc HTTP and saving the response into a JSON file, I'm trying to parse this file using many libname, datasteps etc to build a generalized parser.

 

I already have a python code that uses pandas and parses the same json file.

I'm attaching the response file and the python file, does anyone have any idea how to "translate"  this python code into SAS? or use libname correctly.

 

Thanks for the help

9 REPLIES 9
Reeza
Super User
What version of SAS do you have? Latest version is SAS 9.4TS1M6.
gcardozo
Calcite | Level 5

Yes, I'm in SAS 9.4

Reeza
Super User
The exact version is important, LIBNAME JSON is new.

You can find it in the log via:
proc product_status;run;
gcardozo
Calcite | Level 5
For Base SAS Software ...
    Custom version information: 9.4_M6
    Image version information: 9.04.01M6P110718
 For SAS/STAT ...
    Custom version information: 15.1
 For SAS/ETS ...
    Custom version information: 15.1
 For SAS/IML ...
    Custom version information: 15.1
 For High Performance Suite ...
    Custom version information: 2.2_M7
 For SAS/ACCESS Interface to PC Files ...
    Custom version information: 9.4_M6
 NOTE: PROCEDURE PRODUCT_STATUS used (Total process time):
       real time           0.01 seconds
       cpu time            0.02 seconds
Tom
Super User Tom
Super User

SAS can already read JSON files.

%let fname=ReportResponse.txt;
filename map temp;
filename resp "&path/&fname";
libname resp json jsonmap=map automap=reuse ;
proc copy inlib=resp outlib=work;
run;
NOTE: Copying RESP.ALLDATA to WORK.ALLDATA (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 817 observations read from the data set RESP.ALLDATA.
NOTE: The data set WORK.ALLDATA has 817 observations and 11 variables.
NOTE: Copying RESP.ATTRIBUTES_FORMS to WORK.ATTRIBUTES_FORMS (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 3 observations read from the data set RESP.ATTRIBUTES_FORMS.
NOTE: The data set WORK.ATTRIBUTES_FORMS has 3 observations and 6 variables.
NOTE: Copying RESP.CHILDREN_CHILDREN to WORK.CHILDREN_CHILDREN (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 34 observations read from the data set RESP.CHILDREN_CHILDREN.
NOTE: The data set WORK.CHILDREN_CHILDREN has 34 observations and 3 variables.
NOTE: Copying RESP.CHILDREN_ELEMENT to WORK.CHILDREN_ELEMENT (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 8 observations read from the data set RESP.CHILDREN_ELEMENT.
NOTE: The data set WORK.CHILDREN_ELEMENT has 8 observations and 5 variables.
NOTE: Copying RESP.CHILDREN_ELEMENT2 to WORK.CHILDREN_ELEMENT2 (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 34 observations read from the data set RESP.CHILDREN_ELEMENT2.
NOTE: The data set WORK.CHILDREN_ELEMENT2 has 34 observations and 5 variables.
NOTE: Copying RESP.DATA_PAGING to WORK.DATA_PAGING (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.DATA_PAGING.
NOTE: The data set WORK.DATA_PAGING has 1 observations and 8 variables.
NOTE: Copying RESP.DATA_ROOT to WORK.DATA_ROOT (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.DATA_ROOT.
NOTE: The data set WORK.DATA_ROOT has 1 observations and 3 variables.
NOTE: Copying RESP.DEFINITION_ATTRIBUTES to WORK.DEFINITION_ATTRIBUTES (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 2 observations read from the data set RESP.DEFINITION_ATTRIBUTES.
NOTE: The data set WORK.DEFINITION_ATTRIBUTES has 2 observations and 5 variables.
NOTE: Copying RESP.DEFINITION_METRICS to WORK.DEFINITION_METRICS (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 2 observations read from the data set RESP.DEFINITION_METRICS.
NOTE: The data set WORK.DEFINITION_METRICS has 2 observations and 8 variables.
NOTE: Copying RESP.DEFINITION_SORTING to WORK.DEFINITION_SORTING (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 2 observations read from the data set RESP.DEFINITION_SORTING.
NOTE: The data set WORK.DEFINITION_SORTING has 2 observations and 4 variables.
NOTE: Copying RESP.DEFINITION_THRESHOLDS to WORK.DEFINITION_THRESHOLDS (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 2 observations read from the data set RESP.DEFINITION_THRESHOLDS.
NOTE: The data set WORK.DEFINITION_THRESHOLDS has 2 observations and 6 variables.
NOTE: Copying RESP.ELEMENT_FORMVALUES to WORK.ELEMENT_FORMVALUES (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 8 observations read from the data set RESP.ELEMENT_FORMVALUES.
NOTE: The data set WORK.ELEMENT_FORMVALUES has 8 observations and 3 variables.
NOTE: Copying RESP.ELEMENT_FORMVALUES2 to WORK.ELEMENT_FORMVALUES2 (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 34 observations read from the data set RESP.ELEMENT_FORMVALUES2.
NOTE: The data set WORK.ELEMENT_FORMVALUES2 has 34 observations and 4 variables.
NOTE: Copying RESP.METRICS_NUMBERFORMATTING to WORK.METRICS_NUMBERFORMATTING (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 2 observations read from the data set RESP.METRICS_NUMBERFORMATTING.
NOTE: The data set WORK.METRICS_NUMBERFORMATTING has 2 observations and 9 variables.
NOTE: Copying RESP.METRICS_RANK_BY_REGION to WORK.METRICS_RANK_BY_REGION (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 34 observations read from the data set RESP.METRICS_RANK_BY_REGION.
NOTE: The data set WORK.METRICS_RANK_BY_REGION has 34 observations and 5 variables.
NOTE: Copying RESP.METRICS_SALES to WORK.METRICS_SALES (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 34 observations read from the data set RESP.METRICS_SALES.
NOTE: The data set WORK.METRICS_SALES has 34 observations and 6 variables.
NOTE: Copying RESP.METRICS_THRESHOLDS to WORK.METRICS_THRESHOLDS (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.METRICS_THRESHOLDS.
NOTE: The data set WORK.METRICS_THRESHOLDS has 1 observations and 4 variables.
NOTE: Copying RESP.ROOT to WORK.ROOT (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.ROOT.
NOTE: The data set WORK.ROOT has 1 observations and 5 variables.
NOTE: Copying RESP.ROOT_CHILDREN to WORK.ROOT_CHILDREN (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 8 observations read from the data set RESP.ROOT_CHILDREN.
NOTE: The data set WORK.ROOT_CHILDREN has 8 observations and 4 variables.
NOTE: Copying RESP.SORTING_ATTRIBUTE to WORK.SORTING_ATTRIBUTE (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.SORTING_ATTRIBUTE.
NOTE: The data set WORK.SORTING_ATTRIBUTE has 1 observations and 4 variables.
NOTE: Copying RESP.SORTING_FORM to WORK.SORTING_FORM (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.SORTING_FORM.
NOTE: The data set WORK.SORTING_FORM has 1 observations and 4 variables.
NOTE: Copying RESP.SORTING_METRIC to WORK.SORTING_METRIC (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.SORTING_METRIC.
NOTE: The data set WORK.SORTING_METRIC has 1 observations and 4 variables.
NOTE: Copying RESP.SOURCE to WORK.SOURCE (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 1 observations read from the data set RESP.SOURCE.
NOTE: The data set WORK.SOURCE has 1 observations and 3 variables.
NOTE: Copying RESP.THRESHOLDS_FORMAT to WORK.THRESHOLDS_FORMAT (memtype=DATA).
NOTE: BUFSIZE is not cloned when copying across different engines. System Option for BUFSIZE was used.
NOTE: There were 2 observations read from the data set RESP.THRESHOLDS_FORMAT.
NOTE: The data set WORK.THRESHOLDS_FORMAT has 2 observations and 4 variables.
NOTE: PROCEDURE COPY used (Total process time):
      real time           0.37 seconds
      cpu time            0.17 seconds

 

gcardozo
Calcite | Level 5

It works partially, it does not parse in the desired format.

I need to extract attributes and it's values and then metrics to transform in a dataset where I can work with.

 

I've attached the desired result from the json

Tom
Super User Tom
Super User

It might be possible to craft your own map file to pass to the JSON engine.

Or write code to re-merge the tables that the default method creates.

 

But then it is not longer "generic".  

Reeza
Super User
Flattening a JSON is a non trivial task. You'll need to build customized processes to get what you need here, similar to what you did in the python file. Usually you can just merge the tables from the LIBNAME approach instead of parsing it out manually in a data step. I believe DS2 can also work, but then its more similar to your python code. You could also just call your python program from SAS and read in the results if you have something working and have that option.
Tom
Super User Tom
Super User

Really hard to follow what the question is when you hide the details in attachments.

It looks like you just want to extract these "fields" from the JSON data.

 

Region|Employee|Employee2|Sales|Rank_by_Region
Northeast|Sawyer|Leanne|$2,411,912|1
Northeast|Kelly|Laura|$2,350,720|2

Not sure if you do it with JSON map but just pulling together some of the fields found by the generic mapping I can get something like your table. Not sure if it is pulling the fields you want and whether it is pulling ALL of the values you want.

filename map2 temp;
data _null_;
 file map2;
 put '
{
  "DATASETS": [
    {
      "DSNAME": "WANT",
      "TABLEPATH": "/root/result/data/root/children/children/metrics/Rank by Region",
      "VARIABLES": [
        {
          "NAME": "Region",
          "TYPE": "CHARACTER",
          "PATH": "/root/result/data/root/children/element/name",
          "CURRENT_LENGTH": 12,
          "OPTIONS" : ["RETAIN"]
        },
        {
          "NAME": "Last_Name",
          "TYPE": "CHARACTER",
          "PATH": "/root/result/data/root/children/children/element/formValues/Last Name",
          "CURRENT_LENGTH": 11
        },
        {
          "NAME": "First_Name",
          "TYPE": "CHARACTER",
          "PATH": "/root/result/data/root/children/children/element/formValues/First Name",
          "CURRENT_LENGTH": 8
        },
        {
          "NAME": "Sales",
          "TYPE": "CHARACTER",
          "PATH": "/root/result/data/root/children/children/metrics/Sales/fv",
          "CURRENT_LENGTH": 10
        },
        {
          "NAME": "Rank",
          "TYPE": "NUMERIC",
          "PATH": "/root/result/data/root/children/children/metrics/Rank by Region/rv"
        }
      ]
    }
  ]
}
';
run;
%let path=c:\downloads;
%let fname=ReportResponse.txt;
filename resp "&path/&fname";
libname resp json map=map2 ;
options nocenter ls=132 ps=100; run;
proc print data=resp.want;
run;
                                           First_
     Obs    Region          Last_Name      Name          Sales       Rank

       1    Northeast       Sawyer         Leanne      $2,411,912      1
       2    Northeast       Kelly          Laura       $2,350,720      2
       3    Northeast       Yager          Beth        $2,303,847      3
       4    Northeast       De Le Torre    Sandra      $607,895        4
       5    Northeast       Kieferson      Jack        $584,933        5
       6    Northeast       Sonder         Melanie     $295,108        6
       7    Mid-Atlantic    Bernstein      Lawrence    $1,060,632      1
       8    Mid-Atlantic    Folks          Adrienne    $1,047,776      2
       9    Mid-Atlantic    Hollywood      Robert      $1,026,874      3
      10    Mid-Atlantic    Brown          Vernon      $331,735        4
      11    Mid-Atlantic    Corcoran       Peter       $325,147        5
      12    Mid-Atlantic    Ingles         Walter      $229,439        6
      13    Mid-Atlantic    Smith          Thomas      $221,379        7
      14    Mid-Atlantic    Young          Sarah       $209,634        8
      15    Southeast       Strome         Fred        $595,372        1
      16    Southeast       Lynch          Sam         $592,471        2
      17    Southeast       McClain        Sean        $531,371        3
      18    Southeast       Benner         Ian         $520,737        4
      19    Central         Torrison       Mary        $1,690,350      1
      20    Central         Gale           Loren       $1,669,290      2
      21    Central         Ellerkamp      Nancy       $847,227        3
      22    Central         Zemlicka       George      $822,500        4
      23    South           Pierce         Charles     $2,084,241      1
      24    South           Nelson         Arthur      $1,654,297      2
      25    South           Conner         Beatrice    $1,650,742      3
      26    Northwest       Gedot          Harriet     $739,741        1
      27    Northwest       Hall           David       $513,213        2
      28    Northwest       Becker         Kyle        $508,234        3
      29    Southwest       Bates          Michael     $1,068,907      1
      30    Southwest       Bell           Caitlin     $1,040,481      2
      31    Southwest       Hunt           Matthew     $731,413        3
      32    Southwest       Johnson        Andrew      $445,052        4
      33    Southwest       Schafer        Rose        $408,280        5
      34    Web             Walker         Robert      $3,902,762      1

Note that you can also tell it how to process the fields. For example you might want SALES to be numeric instead of character.

        {
          "NAME": "Sales",
          "TYPE": "NUMERIC",
          "PATH": "/root/result/data/root/children/children/metrics/Sales/fv",
          "INFORMAT" : [ "COMMA", 32, 0 ],
          "FORMAT": ["DOLLAR",10,0]
        }

 

Ready to join fellow brilliant minds for the SAS Hackathon?

Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.

Register today!
Mastering the WHERE Clause in PROC SQL

SAS' Charu Shankar shares her PROC SQL expertise by showing you how to master the WHERE clause using real winter weather data.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 9 replies
  • 1546 views
  • 3 likes
  • 3 in conversation