topic Re: Reading a metadata tree in SAS Data Management

Reading a metadata tree

LaurieF — Wed, 26 Jun 2019 20:43:14 GMT

When we move from dev to test (and to prod), our Hadoop and CAS libraries and jobs have to be manually patched because the text strings contained within the library references frustratingly don't remap to the test properties and associations, like it would with a real DBMS. (Old man shouting at clouds)

The library bit is relatively easy, because I can read through all the "omsobj: SASLibrary?@Id contains '.'" entries where the engine is CAS, HADOOP or IMPALA, then walk my way though the associated data and set the attributes and properties and patch them. The execution time is virtually instantaneous, and works well.

But the jobs (and it's mainly the CAS ones that are causing me grief) are proving problematic. I have three or so metadata folders whose uris I can identify with metadata_pathobj. I can determine the number of members of each tree, using metadata_getnasn with Members as the association. In fact do n = 1 by 1 while(metadata_getnasn(_uri, 'Members', n, member_uri) > 0) loops the expected 20 times (for the particular folder I'm initially interested in. But nothing shows up in member_uri.

What am I missing? Am I using the wrong association? How do I walk through a tree? (If you'll excuse the expression)

Re: Reading a metadata tree

Patrick — Thu, 27 Jun 2019 09:33:49 GMT

@LaurieF

Just as an opinion: You normally don't have that many metadata library definitions so setting up environment specific paths and eventually also configuration options is a one time task. I'm used to add such things to a migration instructions document.

You don't have to change any jobs as these should be using the library definition from the environment they are linked to.

Re: Reading a metadata tree

LaurieF — Thu, 27 Jun 2019 23:59:58 GMT

I'm afraid you've missed the point. I was talking about how to read members in a tree.

The other problem, which I've already solved, was to show that I do know how to navigate my way through metadata functions. In that particular case we have 20-30 entries, and each one previously had to be updated every time we did a migration - the import function did not update them to the correct values. This is beyond a manual process and has no part in a migration document.

Re: Reading a metadata tree

Rama_V — Fri, 28 Jun 2019 06:04:20 GMT

Previously, to understand metadata tree I used Base SAS software "metadata browser" should be there in the menu. Hope this blog explains enough about that.

https://blogs.sas.com/content/sgf/2013/02/06/two-methods-for-editing-sas-metadata/

Re: Reading a metadata tree

ErikLund_Jensen — Fri, 28 Jun 2019 18:09:18 GMT

Hi @LaurieF

Could you please post the code that is not working. I guess your problem might be that you have not declared a length for the member_uri variable and initiated it to blank before the loop, but it is difficult to say without acces to the code.

Re: Reading a metadata tree

LaurieF — Sun, 30 Jun 2019 20:24:38 GMT

Here you go:

data _null_;
length id folder_uri job_uri member_uri uri $ 20
       type prop value $ 256;
length name _uri location _location $200;
call missing(name, _uri, location, _location, uri);
call missing(id, type, folder_uri, job_uri, member_uri, prop, value);
rc = metadata_pathobj(' ', '/DIP Data Management/5000 Processes/10 Raw Ingestion', 'Folder', type, _uri);
do n = 1 by 1 while(metadata_getnasn(_uri, 'Members', n, member_uri) > 0);
   put n= _uri= member_uri=;
   end;
run;

The output I get shows that it's nearly there - but not quite. I know the number of members, but the uri returned from getnasn is the folder's one.

 n=1 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=2 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=3 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=4 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=5 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=6 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=7 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=8 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=9 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=10 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=11 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=12 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=13 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=14 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
 n=15 _uri=A5IKCEC7.AA00008A member_uri=OMSOBJ:Job\A5IKCEC7.
etc...

Re: Reading a metadata tree

LaurieF — Sun, 30 Jun 2019 20:27:55 GMT

Sadly I haven't been able to get Global Hosting to switch it on for me...

Re: Reading a metadata tree

Patrick — Thu, 04 Jul 2019 09:39:46 GMT

@LaurieF

Just in case you haven't resolved this yet:

Your code works for me. You just need to increase the length for member_uri so the returned string doesn't get truncated.

"but the uri returned from getnasn is the folder's one"

Look again at the log you've posted. That's only the repository ID. After that the string is truncated.

Re: Reading a metadata tree

LaurieF — Thu, 04 Jul 2019 09:51:23 GMT

Really? I'm an idiot! Thanks very much.

I can't wait to get to work tomorrow morning to try it out.

Re: Reading a metadata tree

LaurieF — Thu, 04 Jul 2019 19:53:25 GMT

And it works - well spotted. I owe you a beer.

Now, down to examining the stuff about each job. Seats upright, table stowed, ready to extend this metaphor too far...

Re: Reading a metadata tree

AllanBowe — Thu, 02 Apr 2020 11:10:33 GMT

I added a macro to the macrocore library that performs this very function.

To execute:

/* compile macro */
filename mc url "https://raw.githubusercontent.com/macropeople/macrocore/master/meta/mm_tree.sas";
%inc mc;

/* execute */
%mm_tree(root=/my/path, outds=iwantthisdataset)