Help – going through the motions, but still confused with Metadata concepts and practicalities

[mad_scientist]

Hi all,

I’ve read the NESUG 2007 paper 'Metadata 101: A Beginner’s Guide to Table-Driven Applications Programming', which describes use-cases where changing data needs to be efficiently managed without refactoring code, and it makes sense.  I’ve used the DICTIONARY library and SASHELP successfully.

I’ve been able to register tables in Metadata through SAS mgmt. console, and then successfully referenced those tables in dynamic prompts within stored processes.  Seems like a long winded process, but if it enables dynamic prompts, then it's valuable.

But I’m still confused about the whole metadata thing !!  My questions are:

1  As a programmer, what are the benefits of registering tables in Metadata ? I thought all SAS table metadata is available through DICTIONARY/SASHELP, which doesn’t require registering the table...

2  What actual macros/ PROCs / statements do I use to get the above benefits that I can’t get through DICTIONARY/SASHELP ?

3 What’s the point of the SAS mgmt. console – why aren’t all tables automatically registered in Metadata ?

Many thanks.

[jakarman]

Yes mad_scientist for a scientist programmer this kind of approach with metadata seems weird.

There you have your first answer you are a "scientist programmer" not an "ET: programmer" or a "systems programmer" (that last is one is not about programming more integrating).

1/ An ETL programmer can code the process only doing click-click for metadata associations SAS Data Integration Studio .

The SAS dictionary/sashelp only covers SAS datasets not RDBMS external or flat files.

Building the ETL process by metadata as associations is a new level on that by wich you can do impact-analyses on questions related to data-table changes

This is the area of science of data modeling Comparison of data modeling tools - Wikipedia, the free encyclopedia

2/ There are no SAS procs that benefits from SAS metatadata. The benefits are coming from the data governance Data governance - Wikipedia, the free encyclopedia

SAS is doing some weird things with their metadata. They are positioning it as a tool that can overrule the OS security and policies or that one that are present at an External DBMS. IMO a bad approach as violating the data governance (standard of good practice)  .

It is nice you have registered tables and got dynamic prompts working. With those dynamic prompts there is an failure in the implementation by SAS. You will not notice that unless you want to spread that prompt to be used by others using different data-content. The dynamic prompt is opening an app-server to retrieve the data and will use the first found appserver that allows access tot that. Not necessary the same appserver you assigned to the prompt.
The prompt and all information how to process is metadata driven.

By teh way using the SAS.dictionary helps views can be a bad idea having (many) external connections. As it access all those connections retrieving all kind of info every time you are using that. (slow and much overhead..

The revival of proc delete is argumented by that.

3/ SMC (mgmt console) as synonym for the SAS metadata is humbug.  It is faked sales argument.

The SAS metadata is composed of a lot of SAS-datasets (memory cached) and there are many ways to access that SAS-metadata.

You could have update those tables by Eguide a batch job etc. there are JAVA .NET and SA batch interfaces (data step).
No you do not want to register all tables in SAS metadata, why should you?  Ever thought on all those intermediate tables in SASwork you own simulated SASuser.

Do you really want to become a DI programmer instead of science programmer?
               

[mad_scientist]

Thanks Jaap, I'm very grateful for your response.

You've pointed out that Metadata primarily benefits data modelers who are working across both SAS and RDBMS.  You go on to describe the issues you see with how SAS have designed SAS Metadata.  Finally you note that SAS Metadata is actually just a set of SAS datasets that can be accessed lots of ways, not just through SAS Mgmt Console.

[MichelleHomes]

Hi mad_scientist,

I did a high-level presentation on metadata at a SAS Western Australian user group meeting a couple of years ago that you might find helpful. I gave the analogy that metadata is like the blood traveling through your body... https://www.metacoda.com/en/2013/07/metadata-wasup-presentation/ It is the ability to share information between other components in the SAS Intelligence platform.

There is more to metadata than tables, libraries and macro definitions/stored process prompts. As described at SAS(R) 9.4 Intelligence Platform: Overview - SAS Metadata Server, "the SAS Metadata Server is a multi-user server that enables users to read metadata from and write metadata to one or more SAS Metadata Repositories. This server is a centralized resource for storing, managing, and delivering metadata for all SAS applications across the enterprise." The page outlines what type of content is stored in metadata. Please note this list is a partial list as doesn't include objects from SAS solutions and newer products such as SAS Visual Analytics Explorations, Reports, Queries etc... Having a VA Report stored in a centralized metadata repository governed by a SAS security metadata model means those users who have access to it can view it in a SAS client they have access to, such as SAS Visual Analytics Designer, SAS Mobile BI app, SAS Enterprise Guide, SAS Add-in for Microsoft Office clients.

In looking at your questions

1  As a programmer, what are the benefits of registering tables in Metadata ? I thought all SAS table metadata is available through DICTIONARY/SASHELP, which doesn’t require registering the table...

Yes SAS table metadata is available through dictionary tables however the ability to have lineage or an understanding of what reports/stored processes etc use a particular column or table is what having a table registered in metadata provides. Also the ability to have security on the table.

2  What actual macros/ PROCs / statements do I use to get the above benefits that I can’t get through DICTIONARY/SASHELP ?

If you are after technical information about how to use SAS metadata programmatically there are a number of ways. This 2008 SAS Global Forum paper might give you some ideas - Metadata for SAS 9 Programmers

3 What’s the point of the SAS mgmt. console – why aren’t all tables automatically registered in Metadata ?

SAS Management Console and also the SAS Environment Manager are clients which allow you to manage the SAS metadata stored in the SAS metadata repository. This repository stores logical data representations of items such as libraries, tables, information maps and cubes thus ensuring central control over the quality and consistency of data definitions and business rules. The repository also stores information about system resources such as servers, the users who access data and metadata, and the rules that govern who can access what.

Note it is a logical representation which may be different to the physical.

Hope this helps...

Kind Regards,

Michelle

[mad_scientist]

Thanks MichelleH, I'm very grateful for your response.

Summarising: You've given some insights into the breadth and depth of SAS Metadata covering more than tables, libraries and macro definitions/prompts.  DICTIONARY doesn't track where the data has come from or where it is used.   You can also 'have security' on the table (or other resource).  The paper you referenced Metadata for SAS 9 Programmers outlines different ways of accessing Metadata.  The SAS Mgmt console is a way to have ‘central control’ of data definitions and business rules.

My response

The paper you referenced Metadata for SAS 9 Programmers outlines different ways of accessing Metadata.  The second method listed, through datasteps, makes the most sense to me.  But still more questions :

A          Good to know that Metadata is pretty much all-inclusive… I guess I’m trying to get my head around 1 or 2 solid reasons to voluntarily use Metadata before trying to explore everything.  As well as table definitions, the idea of centrally storing business rules appeals.  But when I looked in SAS Mgmt Console, I couldn’t see any reference to Business Rules.  I found some references online to SAS Business Rule Manager, but it looks like a different application to SAS Mgmt Console… shouldn’t the SAS Management Console be a central place to do everything about Metadata ?

B          I noted your reference to ‘central control’ of definitions – does that mean that only a small set of programmers  should have access to the SAS Mgmt Console, and everyone else is just consuming the Metadata ?  I'm trying to understand who uses Metadata now...

C          The ability to look upstream (lineage) and downstream (uses) of a dataset make sense to me.  I tried looking in SAS Management Console… there were a few references to ‘BI lineage’ but I couldn’t see any content (shouldn't it be automatic ?), and couldn’t see anything about where a dataset is used…. Could you please advise / point me to practical information on these kinds of things ?

D      Finally, is there much real evidence that SAS Metadata ‘works’ organisationally ?  By this I mean all the above seems like a good idea, logical enough, smart people have thought it through etc, but is it all too complex for real organisations to take advantage of ?  Is there any actual data to show productivity of business users/organisations with and without SAS Metadata ?

Cheers.

[MichelleHomes]

Hi,

Further to Jaap's reply, the SAS Management Console isn't the only SAS client that can update SAS metadata, changes can also be made in SAS Visual Analytics Administrator, DI Studio etc.

In terms of metadata relationship you may want to look at a recent blog post that describes some reporting tools for this - http://blogs.sas.com/content/sgf/2015/03/04/relationships-are-easy-for-sas-objects

I'd recommend reading the SAS(R) 9.4 Intelligence Platform: Overview document to get an understanding of the tiers of the SAS platform, it's architecture, clients, metadata and security.

Kind Regards,

Michelle

[jakarman]

A      SAS isn't the only one ambracing the idea of metadata. That concept is really old.  An other name is for example Integrated Data Dictionary. Metadata standards - Wikipedia, the free encyclopedia and Data dictionary - Wikipedia, the free encyclopedia.

Seeing the SMC the same as the Metadata is a terrible mistake. It is only an example of a tool accessing that and the SMC has only a narrow view on that part.

You found the "Business Rule manager" you should have found "Data integration Studio"  SAS Data Integration Studio.   When you would review SAS-VA you will see the SMC is bypassed but still metadata is being used.

B      That DI studio tools is a special one for Developers as they alos could use Eguide. There is also SAS OLAP Cube Studio and SAS OLAP Server when the part is only cubes (subset functions in DI) and SAS Information Map Studio What is building a dedicated layer around datasets in a very special way as the concept of OLAP is failing with row-security level access. It can bypass the ETL steps but at the cost of resource overhead.    

Every modern SAS Solution is adding his own metadata controls into the SAS metadata. For Enterprise Miner there are several additional controls for operations and Model management.

One part of the SAS metadata is about user groups and their access control. That is about identity management and the security with all those roles. It is similar to LDAP (Unix) or AD (Windows).

The question who is using Metadata?  That is the whole of all people involved to IT processes.

The question who is using SAS Metadata?  That should be the same people doing those IT processes being responsible for those different tasks.  The problem you are seeing it is not aligned like that.

C      Looking upstream and downstream into data. That is part of the ETL (data modeling) with the SAS tool DI (impact analyses). SAS(R) Data Integration Studio 4.9: User's Guide . Same parts of that can be found in other tools.  

There are many other tools doing the same like DI Comparison of data modeling tools - Wikipedia, the free encyclopedia. A common question is how to integrate those. There are standards for those       

D     SAS metadata works as metadata concepts are existing and heavily in use for 30 years (or more). SAS is just a follower with that coming very late (since 2006). They combined several older tools having each their own metadata into an centralized concept. 

Is working with metadata complicated? No you are doing this all the time.       

Is working with centralized metadata complicated? Of course cooperation with others not doing it all yourself that is difficult. Integration of all kind of tools and all kind of processed that is complicated. That is how a enterpise works.

[mad_scientist]

Thankyou both for your further explanations.

I met a colleague who has able to give me some worked examples and step me through things, and I'll keep reading up on all the material out there.

Cheers.