BookmarkSubscribeRSS Feed
HrZiller
Obsidian | Level 7

hi all,

 

I am working on a large project with +40 developers within many different technoligy areas. The SAS Data Integration platform using DI Studio is used for the sourcing data, remodelling and the creating a datamodel being used by teams with a more "devops" approach to development processes. 

In our team we are leaning towards the more traditional dev-test-prod approach, using partiel promotion - and (unfortunately) a bit to many manual processes involved in the promotion. We are looking at how this can be optimized.

 

My question is:

Our downstream dependent teams are leaning towards a continues delivery approach, with a release management proces that is based on Github and at present seems very automated/script based. They can change "versions" very fast - due to the fact they are using code-based tools. So here it is: has anybody had experiences they can share on how a SAS DI+metadata environment can work efficiently with continues delivery.

 

At present we have approx. 6-700 DI jobs in the environment - this number is going slowly up.

 

All the information I have found regarding th use of release management tools and SAS is based on a *.SAS file approach - not using DI studio and metadata. We have considered not using metadata going info production - but using the LSF scheduling tool is giving us some dependencies of having metadata.

 

Thanks

Jan

5 REPLIES 5
RahulG
Barite | Level 11

Are you looking for solution where in you can keep versions of SAS DI jobs instead of keeping version of *.sas file?

SAS DI have change mangement but it is only to have check in and check out feature with No versioning system.

 

OR 

 

Are you looking something for finding object dependency in SAS metadata?

 

HrZiller
Obsidian | Level 7

We are actually using subversion, and check-out/check-in when develuping in DI - so it is not this what I am looking for.

 

It is the handling of the concept of "versions" (which does not really exist in metadata)  and coordinating the traditional approach:

dev

test

prod

 

example: We are promotion a version:

dev - v1

promotion to test v1

promotion to prod v1

 

Now v1 is in production

 

Meanwhile we get new business requirements - which means that we start on v2:

Dev v2

promoted to test V2

but our downstream teams are still working on the V1 version and not ready for v2 yet.

 

This is the complicated by the fact that their promotion processes are based on the approach: continues delivery/deployment - so they might be working in many version in many environments: dev, test, demo, training, preprod, prod etc.... and the add versions... 

 

So the complexity is increased many times. At present our thoughts are that the downstream teams must handle this internally by housekeeping our data delivery in the amount of versions they require. This is not optimal since our development phases very fast can get put of sync.

 

So I am looking for thoughs and experiences in a similar setup.

 

regards

Jan 

Patrick
Opal | Level 21

@HrZiller

What you describe appears to become a more common scenario for projects using an agile approach. Especially when it comes to parallel development the risks can become substantial and version management is crucial.

 

As you write versioning tools work best for textual files.

 

The one site I know of which uses agile with even substantially more jobs than you that appears to have things under control uses DIS only in DEV. They only promote the .sas files to higher environments which then also allows to use versioning tools in a standard way. As far as I know this site implemented scripts to change the few environment specific code bits in DIS generated SAS code.

 

You don't need SAS Metadata deployment objects for LSF scheduling. I know of a case where job dependencies got defined in an Excel sheet which then serves as source to generate the LSF flows.

HrZiller
Obsidian | Level 7

hi Patrick,

 

thanks for your thoughts. The more I think this through the more it seems a requirement for promoting pure SAS code rather than the full metadata incl. redeploy since the SPK packages are bit more complex to deal with when respect to track changes.

 

Could be interesting to hear from SAS customers that are seeing this in their development environments.

 

regards

Jan

 

PS: not SAS employee any more - can not change the "SAS employee" label for my account.

LinusH
Tourmaline | Level 20
If you are using the target data for other SAS applications I strongly recommend promoting metadata. At some point data structures will start to differ. Also you use metadata for authorization and structuring of content. And of course impact analysis from source to end user reports etc.
Data never sleeps

SAS Innovate 2025: Register Now

Registration is now open for SAS Innovate 2025 , our biggest and most exciting global event of the year! Join us in Orlando, FL, May 6-9.
Sign up by Dec. 31 to get the 2024 rate of just $495.
Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 1899 views
  • 0 likes
  • 4 in conversation