LIBNAME META engine replacement in VIYA

[olegbogdanov]

Hey experts. 

My customer migrate large SAS 9.4 BI to VIYA. They have hundreds of scheduled jobs and actually also hundreds of libraries. 

In 9.4 they used META engine in code in libname to avoid preassign of hundred libraries in Batch context. I have dig in SAS documents and have got understanding, that there are no META engine any more in VIYA.

But have anyone found good workaround to keep the amount of assign libraries reasonable during launch of scheduled job. 

I see following options: 

- group jobs by subject (or by users groups) , so that Batch context assign only subset of libraries. So useing multiple contexts (if it possible - I do not have VIYA to test). 

- hard code libname statements to code  (ugly, unsecure..) 

- I have got impression, that You can assign libraries in autoexec of context, but how to assign it in batch server code. 

Do I miss some key hint ? I haven't learned systematically VIYA . 

Thanks in advance 

/oleg 

[SASKiwi]

Maybe this post is helpful: https://communities.sas.com/t5/SAS-Communities-Library/Managing-Connections-to-Data-Sources-and-SAS-...

You are correct in that there is no metadata server in SAS Viya 4, hence no meta engine.

[olegbogdanov]

Thanks. I have seen this post. This is not exact solution, but looks like it is really only way. So I needed confirmation, that there are no other ways to solve this crossword.
/oleg

[Patrick]

The referenced article shows how to define a library in Viya so it becomes accessible to any selected user (also a batch user) but is not pre-assigned. Isn't that similar to a library metadata definition without pre-assignment in SAS9.4?

The article then also shows how to assign such a library in code using the LIBRARYDEFINITION= LIBNAME Statement Option which looks very similar to using the libname meta engine in SAS 9.4 code.

Where I see a challenge which needs some further thinking is how to migrate such code from one environment to the next because the source-definition-URI is likely environment specific. TBC: I do expect the uri to change but would need testing/confirmation.

[olegbogdanov]

Thanks Patric!

Yes. that looks very much like META engine in 9.4. And I hope, that if datasource are created, this ugly 😊 hexadecimal link is permanent. 

I do not have VIYA to test currently, but from practical viewpoint.

Lets suppose, that I am in process of writing code  in Studio. Now I need to add this LIBRARYDEFINITION to code. Where I can see this URI. According to my experience with VIYA ( Intelligent Decision), typically these K8S ID-s were visible in object (datasource)  properties.  So, can developer copy this URI from datasource properties. Or have he change to Environment manager. 

Replace in bulk is different story, but I guess, that as this is just one-time  task, it can be solved in some way with python code over all code files. (if URI-s are permanent and I have these already). 

Or via REST call ... 

Thanks again - I see light at the end of tunnel .....

/oleg

[Patrick]

@olegbogdanov I'm myself still on the steep Viya learning path so many things are still on the level of "I believe, I'd expect" without being certain.

I would expect that the hex part of a liburi is fixed for a library definition but will differ between environments for the same libref/path - similar to a metadata id  under SAS9.4.

So far I couldn't find anything in the docu that would allow to assign such a library definition based on defined libref or path.

This article shows how to use the Viya Rest api to retrieve all library definitions from an environment.

Right now my thinking is that I probably would replace the libname meta definition in the SAS9.4 code with a macro call for the library assignment (likely with the libref as input parameter). This would allow for code that certainly won't need change when promoting to higher environments.

The Rest call from the article should return all the information needed to then actually assign the library - I'd be doing this within the macro as well as the actual library assignment.   ....and should the REST call require permissions not available to any user that needs to do such an assignment then there could also be some regular job for the REST call populating some SAS table with the info to which all users get read access - and that's then what you're using in the macro (which again is an advantage to implement via a centralized macro as then you can change or enhance the approach in a single place if and as needed).

For any article with links to the docu: Make sure that you always switch to the latest (or your version) of Viya because things are evolving and enhancing quickly with every single version.

And last but not least: You need Viya training. It's a big change from SAS9.4 

[olegbogdanov · Accepted solution]

Thanks for thorough answer. 

central macro was also one option I have considered. Based on table, which is regularily updated by some SAS code with REST call. 

All these tricks need testing as details are not exposed typically in documentation.  

I think that I have now all info to present to my colleagues as advice. 

Thanks again and have a nice afternoon. 

 /oleg

[GuidoG]

Thank you @olegbogdanov for raising the question and thank you all for your input.  
If it helps someone please find my first dynamic solution to assign libraries in batch code in the manner to avoid hardcoded LIBNAME statements

%let lib_ref = SASHELP;  /* As for example */

%let SERVICESBASEURL=%sysfunc(getoption(servicesbaseurl));

/* Assign a temporary file to store the response */
filename resp temp;

proc http
  url="&SERVICESBASEURL./dataSources/providers/Compute/sourceDefinitions?filter=eq(defaultLibref,%27&lib_ref.%27)"
  method="GET"
  oauth_bearer=sas_services
  out=resp;
  headers
    "Accept"="application/json";
run;

libname respjson json fileref=resp;
    
/* Check if the dataset exists */
%if %sysfunc(exist(respjson.items_links)) %then %do;
  /* assign library definition value to macro variable*/
  data _null_;
    set respjson.items_links;
    where ordinal_items = 1 and rel = 'self'; /* choose the first match */
    call symputx('lib_def', uri);
  run;
%end;
%else %do;
  %put --------------------;
  %put WARNING: Library &lib_ref. was not found.;
  %put --------------------;
  %return;
%end;

libname respjson clear;

/*  Actual LIBNAME statement */
libname &lib_ref. libdef="&lib_def.";