Improve Your Relationships: With a REST API in SAS Viya 3.5
Help your customers improve their data traceability and demonstrate transparency, compliance and adherence to regulations. The Relationship REST API is to SAS Lineage what a made-to-measure garment is to a ready-to-wear. Learn how to use this REST API to extract lineage information, create, update and delete objects and links. As a bonus, you will get a piece of SAS code that you can use with any SAS Viya 3.5 REST APIs.
In 60 seconds
Lineage in Viya 3.5 presents itself as a sequence of objects related one to another. You might want to create relationships between two objects, because you have expert information. In this post you will learn how to 'connect the dots' using the relationships REST APIs.
You need to know how the Viya relationship REST API works, if you want to:
- Extract information from SAS Lineage in a table (compliance reporting)
- Create / update a link between two objects that have a relationship
- Delete obsolete relationships
Connect the dots
Say for a reason you want to relate two objects from SAS Lineage:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Dots connected
References and Relationships of an object
You search for an object in SAS Lineage in SAS Viya:
Viya Relationships REST API “speak”:
The reference is an object, like a piece of fabric in a garment. It presents most typically as a CAS table (loaded in memory), a physical table, a SASHDAT file (memory-mapped), a CASLIB, an analytical model, data plan, etc.
The relationship is the link between the two, the stitch. The link can have several types (see Annex):
To use the Viya Relationship REST APIs you need typically the:
- ResourceURI of the object or Object URI. The name is part of it:
- Reference id of each of the objects
- Relationship id of their link
Object URI
Each object has in Viya an URI (or Universal Resource Identifier). For example, if you would search for an object Gatedemo301_LOAN, Separate Equivalents and look at the Object URI:
Relationship REST API URL
Behind SAS Lineage you will find a database called the ‘relationship service’. Suppose you want to retrieve all the objects containing ‘301_LOAN’ in their name (ResourceURI).
To call a Viya REST API, you would need a URL In my example the server is 'sasviya01.race.sas.com':
http://sasviya01.race.sas.com/relationships/relationships/?filter= contains(resourceUri,"301_LOAN")&limit=100
Plum = server
Green = Relationship service
Blue=Request body, parameters and filters
Request types: GET = Query, POST = create, PUT = update, DELETE = remove relationships or references.
Pre-requisites
SAS Viya is very open, but when it comes to exposing its REST APIs, it has many locks. To unlock them, you need tokens. Tokens act as electronic keys. The tokens tell Viya you are who you say you are and you have the authorizations and the rights to access the services.
I've read a few articles on the topic and tried many things, but I will give you a simple process to follow and a good piece of code I got from one of my colleagues (it will simplify your life). You need:
- Appropriate Rights
- Consul token
- OAuth access token
- Register Viya Client Id and secret
- Viya access token and refresh token
Administrator Tasks:
Appropriate Rights
Your administrator must grant the group or the user using the REST API: Read, Create, Update, Delete permissions (SAS Environment Manager > Rules screen filtered for / relationships). The user running the code in our example is in the Data Builders group.
Consul Token
Ask your administrator to log into the Viya system on sasviya01 and navigate to:
cd /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/consul/default/
Display the contents of the client.token file:
sudo more client.token
You obtained the client token.
6fd548xz-d899-4ye5-9fd6-62111f1xy7y5
User Tasks
Obtain an OAuth Access Token for registration, register a SAS Viya Client ID and secret
This is the piece of SAS code you will need if you work with SAS Viya REST APIs (customize the macros to your environment):
* Create a SAS Viya Client ID;
*******************************;
* credits: Vincent DelGobbo;
/* This is a one-time operation to create a Client ID that you use
when accessing Viya services. Refer to the documentation:
Obtain an OAuth Access Token to Register a New Client ID
Obtain an Access Token Using Password Credentials*/
/*Initialize Macros*/
options ls=max nodate;
ods _all_ close;
* Set the base URI for service calls;
%let BASE_URI=http://sasviya01.race.sas.com; *customize to your server;
* Specify the new Client ID name;
*! Name must not be registered already - no spaces;
%let CLIENT_ID=ViyaRel;
* Specify the secret for the new Client ID;
%let CLIENT_SECRET=DG;
* Specify the consul token;
%let CONSUL_TOKEN=6fd548xz-d899-4ye5-9fd6-62111f1xy7y5;
*Check macro variables;
%put &CLIENT_ID;
%put &CLIENT_SECRET;
* FILEREFs for the response and the response headers;
filename resp temp;
filename resp_hdr temp;
* FILEREFs for the response and the response headers;
filename resp temp;
filename resp_hdr temp;
* Get the access token to use to create the Client ID;
proc http url="&BASE_URI/SASLogon/oauth/clients/consul?callback=false%nrstr(&serviceId)=&CLIENT_ID"
method='post' out=resp headerout=resp_hdr headerout_overwrite verbose;
debug level=3;
headers 'X-Consul-Token'="&CONSUL_TOKEN";
run;quit;
*;
* Get the access token from the JSON data and
* store it in the ACCESS_TOKEN macro variable.
*;
libname tokens json fileref=resp;
proc sql noprint;
select access_token into:ACCESS_TOKEN from tokens.root;
quit;
* FILEREFs for the request, the response, and the response headers;
filename req temp recfm=n;
filename resp temp;
filename resp_hdr temp;
* Create the request body;
data _null_;
file req;
put '{' @;
put '"client_id": ' """&CLIENT_ID""" ',' @;
put '"client_secret": ' """&CLIENT_SECRET""" ',' @;
put '"redirect_uri": "http://127.0.0.1",' @;
put '"scope": ["openid", "*"],' @;
put '"resource_ids": "none",' @;
put '"authorities": ["uaa.resource", "sasapp"],' @;
put '"authorized_grant_types": ["password", "client_credentials", "authorization_code", "refresh_token"],' @;
put '"autoapprove": "true",' @;
put '"access_token_validity": 43200,' @;
put '"use-sessions": "true"' @;
put '}' @;
run;
* Register the Client ID;
proc http url="&BASE_URI/SASLogon/oauth/clients" method='post'
ct='application/json; charset=utf-8' in=req out=resp headerout=resp_hdr
verbose;
debug level=3;
headers 'Authorization'="Bearer &ACCESS_TOKEN";
run;quit;
Open the generated dataset TOKENS.ALLDATA :
Access Token and Refresh Token
Almost there! To update the relations or the reference, you need to get your access token. Without it you can still query relationships (perform GET requests) but cannot modify:
******************************************
*Use the Client ID to Get an Access Token;
******************************************
*Submit this code once to get the access token or repeat if your access token has expired.;
options ls=max nodate;
ods _all_ close;
* Set the base URI for service calls;
%let BASE_URI=http://sasviya01.race.sas.com;
* Specify the username and password;
%let USERNAME=gatedemoYYY; *replace YYY with user;
%let PASSWORD=********;
*MUST replace with an ACTUAL PASSWORD!;
* Specify the Client ID name and secret;
* Specify the new Client ID name;
*! Name must be registered above - no spaces;
%let CLIENT_ID=ViyaRel;
* Specify the secret for the new Client ID;
%let CLIENT_SECRET=DG;
* FILEREFs for the response and the response headers;
filename resp temp;
filename resp_hdr temp;
* Get access and refresh tokens in JSON format;
proc http url="&BASE_URI/SASLogon/oauth/token" method='post'
in="grant_type=password%nrstr(&username=)&USERNAME%nrstr(&password=)&PASSWORD"
username="&CLIENT_ID" password="&CLIENT_SECRET" out=resp auth_basic verbose;
debug level=3;
run;quit;
*;
* Get the access token from the JSON data and
* store it in the ACCESS_TOKEN macro variable.
*;
libname tokens json "%sysfunc(pathname(resp))";
proc sql noprint;
select access_token into:ACCESS_TOKEN from tokens.root; quit;
Go once again to TOKENS.ALLDATA and confirm your group or user is in the scope. Your access-token resides in a variable and the scope is now Data Builders, sasusers. You can now use the REST API according to your rights in SAS Environment Manager.
Relationship REST API in Action
Create a relationship between two objects (STITCH)
As mentioned above, you want to create a link between two objects, a “stitch” between:
Object 1 URI : /dataTables/dataSources/cas~fs~cas-shared-default~fs~CASUSER(gatedemo301)/tables/GATEDEMO301_LOAN
Object 2 URI: /dataSources/providers/cas/sources/cas-shared-default~fs~CASUSER(gatedemo301)
- You need the reference id of object 1
- You need the reference id of object 2
- Prepare a json request with the resourceURI and the references of both objects
Get reference id of object 1
The code will query the relationships REST API for the object indicated and return a value in a macro variable obj1ref.
Replace the location with your folder and OBJ1URI in with the resourceURI of object 1:
* Get object1 reference based on URI;
%let location=/home/gatedemo301/;
filename respa "&location/get_ref_a.json";
filename resphdra "&location/get_ref_a.txt";
%let OBJ1URI=/dataTables/dataSources/cas~fs~cas-shared-default~fs~CASUSER(gatedemo301)/tables/GATEDEMO301_LOAN;
proc http url="&BASE_URI/relationships/relationships/?resourceUri=&OBJ1URI"
method='get'
out=respa headerout=resphdra headerout_overwrite verbose;
debug level=3;
headers 'Authorization'="Bearer &ACCESS_TOKEN";
run;quit;
* Get the reference id from the file and store it in a file:
*;
libname obj1 json "%sysfunc(pathname(respa))";
proc sql ;
select referenceId into:obj1ref from obj1.items; quit;
Submit and check the log for: 200 = success HTTP/1.1 200
Note: if you get a message in the log, error=”invalid token” re-run Access token and Refresh Token code above.
Get reference id of object 2
Replace the OBJ2URI in with the resourceURI of object 2:
* Get object2 reference based on URI;
filename respb "&location/get_ref_b.json";
filename resphdrb "&location/get_ref_b.txt";
%let OBJ2URI=/casManagement/servers/cas-shared-default/caslibs/CASUSER(gatedemo301)/tables/GATEDEMO301_I_LOAN;
proc http url="&BASE_URI/relationships/relationships/?resourceUri=&OBJ2URI"
method='get'
out=respb headerout=resphdrb headerout_overwrite verbose;
debug level=3;
headers 'Authorization'="Bearer &ACCESS_TOKEN";
run;quit;
* Get the reference id from the file and store it in a file:
*;
libname obj2 json "%sysfunc(pathname(respb))";
proc sql ;
select referenceId into:obj2ref from obj2.items; quit;
%put &obj1ref "," &obj2ref "," &OBJ1URI "," &OBJ2URI;
Submit and check the log for: 200 = success HTTP/1.1 200
Note: if you get a message in the log, error=”invalid token” re-run Access token and Refresh Token code above.
Prepare a json request to “stitch”
To create a relationship, a file must be sent to the Relationship API. See example below (the documentation is quite clear):
{
"referenceId": "d8ce9854-714d-4cfa-9e9b-625b8b81c07a",
"relatedReferenceId": "0162bc36-0a29-4877-b850-fb41ba024ac3",
"relatedResourceUri": "/dataSources/providers/cas/sources/cas-shared-default~fs~CASUSER(gatedemo301)",
"resourceUri": "/dataTables/dataSources/cas~fs~cas-shared-default~fs~CASUSER(gatedemo301)/tables/GATEDEMO301_LOAN",
"source": "/dataTables/dataSources/cas~fs~cas-shared-default~fs~CASUSER(gatedemo301)/tables/GATEDEMO301_LOAN",
"type": "Dependent",
"version": 0
}
Write a POST request:
* POST relationships;
* files used;
filename resps "&location/post_rels.json";
filename resphdrs "&location/post_rels.txt";
filename in "&location/post_rel_content.json";
* Create a dataset;
data x;
obj1ref=strip("&obj1ref.");
obj2ref=strip("&obj2ref.");
obj1URI=strip("&OBJ1URI.");
obj2URI=strip("&OBJ2URI.");
run;quit;
* Create a file using the variables in the dataset;
data _null;
set work.x end=eof;
file in;
put '{"referenceId": "' obj1ref +(-1) '", "relatedReferenceId": "' obj2ref +(-1) '", "relatedResourceUri": "' obj2URI +(-1) '", "resourceUri": "' obj1URI +(-1) '",
"source": "' obj1URI +(-1) '", "type": "Dependent", "version": 0';
EOF:
if (eof) then do;
put '}';
end;
run;
* POST relationships;
proc http url="&BASE_URI/relationships/relationships"
method='post'
in=in ct='application/vnd.sas.relationship+json' out=resps headerout=resphdrs headerout_overwrite verbose;
headers 'Authorization'="Bearer &ACCESS_TOKEN" ;
debug level=3;
run;quit;
Check the log for a 201 = created HTTP/1.1 201
Note: if you get a message in the log, error=”invalid token” re-run Access token and Refresh Token code above
< HTTP/1.1 401 < Date: Fri, 29 Nov 2019 01:55:12 GMT < Server: Apache/2.4 < Pragma: no-cache < WWW-Authenticate: Bearer realm="oauth", error="invalid_token",...
Check the result:
In SAS Lineage, refresh one of the stitched object: Bingo, the relationship was CREATED.
Update a relationship
To update a relationship, e.g.: change the type from ‘Dependent’ to ‘Equivalent’ you need to write a PUT request similar with the POST.
Delete an object
This might come handy as SAS Lineage does not automatically delete objects when they no longer exist.
Let’s say you want to delete
- run the get reference id of object 1 code above and change the OBJ1URI to the object URI you want to delete : /dataTables/dataSources/cas~fs~cas-shared-default~fs~CASUSER(gatedemo301)/tables/GATEDEMO301_LOAN?sessionId=98b8f427-614c-504e-b809-34c274c33d71
- replace in the code below your object reference id and the file location. You need to know the referenceId of the object.
* Delete the object reference based on reference URI;
filename resp5 "&location/del_ref5.json";
filename resphdr5 "&location/del_ref5.txt";
proc http url="&BASE_URI/relationships/references/&obj1ref"
method='delete'
out=resp5 headerout=resphdr5 headerout_overwrite verbose;
debug level=3;
headers 'Authorization'="Bearer &ACCESS_TOKEN";
run;quit;
Submit and check the log: 204 = deleted. HTTP/1.1 204
Conclusions
You have now a very good overview of the steps to use a SAS Viya REST API programmatically. You can use the Relationship REST API to extract information from SAS Lineage, update it and correct it.
Acknowledgements:
Big thank you: Vincent DelGobbo, for the Viya tokens SAS code.
REST API documentation
- Wonderful resource for SAS Viya REST API
Authentication
- OAuth Access Token, Client ID, Access Token
- Publish a Decision to MAS - Authenticate to SAS Viya: Register a Client ID
Want to Learn More about Viya 3.5?
- DevOps Applied to SAS Viya 3.5: Run a SAS Program with a Jenkins Pipeline
- DevOps Applied to SAS Viya 3.5: Top Git Commands with Examples
- Two Simple Ways to Import Local Files with Python in CAS (Viya 3.5)
- Six Ways to Import a Local File into CAS (SAS Viya 3.5)
- Query Performance? Use a CAS Star Schema in Viya 3.5
- Automatic Data Loading with a CAS Data Source Star Schema in Viya 3.5
- Go With the Job Flow in SAS Viya 3.5
- How to Load Images in SAS Viya 3.5
Thank you for your time reading this post. Please comment and share your experience with the Viya Lineage and REST APIs to help others.
You will find this in the properties when you click on an object in the /SASLineage application
