Change management with SAS platform - DI, metadata - any experience with "Continues Delivery"??

[HrZiller]

hi all,

I am working on a large project with +40 developers within many different technoligy areas. The SAS Data Integration platform using DI Studio is used for the sourcing data, remodelling and the creating a datamodel being used by teams with a more "devops" approach to development processes. 

In our team we are leaning towards the more traditional dev-test-prod approach, using partiel promotion - and (unfortunately) a bit to many manual processes involved in the promotion. We are looking at how this can be optimized.

My question is:

Our downstream dependent teams are leaning towards a continues delivery approach, with a release management proces that is based on Github and at present seems very automated/script based. They can change "versions" very fast - due to the fact they are using code-based tools. So here it is: has anybody had experiences they can share on how a SAS DI+metadata environment can work efficiently with continues delivery.

At present we have approx. 6-700 DI jobs in the environment - this number is going slowly up.

All the information I have found regarding th use of release management tools and SAS is based on a *.SAS file approach - not using DI studio and metadata. We have considered not using metadata going info production - but using the LSF scheduling tool is giving us some dependencies of having metadata.

Thanks

Jan

[RahulG]

Are you looking for solution where in you can keep versions of SAS DI jobs instead of keeping version of *.sas file?

SAS DI have change mangement but it is only to have check in and check out feature with No versioning system.

OR 

Are you looking something for finding object dependency in SAS metadata?

[HrZiller]

We are actually using subversion, and check-out/check-in when develuping in DI - so it is not this what I am looking for.

It is the handling of the concept of "versions" (which does not really exist in metadata)  and coordinating the traditional approach:

dev

test

prod

example: We are promotion a version:

dev - v1

promotion to test v1

promotion to prod v1

Now v1 is in production

Meanwhile we get new business requirements - which means that we start on v2:

Dev v2

promoted to test V2

but our downstream teams are still working on the V1 version and not ready for v2 yet.

This is the complicated by the fact that their promotion processes are based on the approach: continues delivery/deployment - so they might be working in many version in many environments: dev, test, demo, training, preprod, prod etc.... and the add versions... 

So the complexity is increased many times. At present our thoughts are that the downstream teams must handle this internally by housekeeping our data delivery in the amount of versions they require. This is not optimal since our development phases very fast can get put of sync.

So I am looking for thoughs and experiences in a similar setup.

regards

Jan 

[Patrick]

@HrZiller

What you describe appears to become a more common scenario for projects using an agile approach. Especially when it comes to parallel development the risks can become substantial and version management is crucial.

As you write versioning tools work best for textual files.

The one site I know of which uses agile with even substantially more jobs than you that appears to have things under control uses DIS only in DEV. They only promote the .sas files to higher environments which then also allows to use versioning tools in a standard way. As far as I know this site implemented scripts to change the few environment specific code bits in DIS generated SAS code.

You don't need SAS Metadata deployment objects for LSF scheduling. I know of a case where job dependencies got defined in an Excel sheet which then serves as source to generate the LSF flows.

[HrZiller]

hi Patrick,

thanks for your thoughts. The more I think this through the more it seems a requirement for promoting pure SAS code rather than the full metadata incl. redeploy since the SPK packages are bit more complex to deal with when respect to track changes.

Could be interesting to hear from SAS customers that are seeing this in their development environments.

regards

Jan

PS: not SAS employee any more - can not change the "SAS employee" label for my account.

[LinusH]

If you are using the target data for other SAS applications I strongly recommend promoting metadata. At some point data structures will start to differ. Also you use metadata for authorization and structuring of content. And of course impact analysis from source to end user reports etc.