Source Control Options for DI Studio

[JohnJPS]

Is anyone aware of better options for Source Control, for DI Studio in particular?  The existing approach seems like a tacked-on afterthought, and compared to something like Visual Studio is quite a letdown.  Have any vendors written add-ons or anything, or do people just manually work with what's there?

[LinusH]

Hi John,
If you can briefly tell us what options you have evaluated, and your conclusions (and your requirements) it would easier to tell what might be better.
But generally speaking I pretty much share your opinion. On the other hand ETL metadata IMO is hard to manage the same way as other programming environments where you usually strive for isolation, whereas in DW "everything" is related.

[JohnJPS]

Our use-case is such that DI Studio is used for data prep within the context of a given project, so there is no broader ETL concern beyond the scope of that given project.

For what we're doing now: we support Subversion, so we can utilize the DI Studio tools menu to point at a given Subversion repository, then right-click to archive anything in particular to that repository. Combining that with check-out/check-in works, but is not very smooth.

Want: they already have check-out/check-in... if that tied into the source-control repository directly, it would be so much easier: every time you check something back in, you provide a check-in comment and it gets a new version automatically. For any of those micro-versions, we generally don't care what they are. But at any given time (say, release time), we should have the ability to tag everything under a given hierarchy (say, a project) with a tag, which would tie the version of each component in the hierarchy at the time of tag to the given tag. So basically, everything is always source controlled, and tagged with a "release tag" for any given release. This means the production system only needs to keep track of "which release" it is.  Rollbacks to a previous release should be allowed by providing the tag and the system should roll everything back to the check-in version corresponding to the provided tag.  This is all common practice, and should be supported by any decent source control platform (TFS, GIT, SVN, perhaps others).

Options evaluated... nothing really. I'm not aware of any solutions other than the CVS/SVN tabs in the DI Studio options; and then archiving there. Thus this question fishing for 3rd party addons/plugins/extension. 🙂

Comments appreciated: Metadata seems to be history with SAS Viya, no DI Studio analog as yet in SAS Viya... any comments about the future of DI Studio are welcome also. If it's simply "going away" and if we should perhaps prepare for that possibility, it would be nice to know.

[AngusLooney]

So, as an “old school” DI person, I’m not comelette au fair with source control tools like SVN and GIT, but I do have the SVN module running on my DI install on my PC, and have dabbled with that a bit.

Back to the “old school” - for me, a lot of this comes back to the promotion processes and ceremonies. The Dev/Test/Prod pattern is power, especially when you think of the artefacts involved in the promotion process itself, the exported SPK that carry new and changed objects between environments or levels.

Thinking about the lifecycle of code over time, those SPKs do capture all changes and versions over time, albeit only at a “finished enough to promote” level of granularity.

So, establishing a strong regime to retain and document all these SPKs represents an opportunity to be in a position of control over changes over time, and to be able to roll back, through the catalogue of SPKs.

Couple this with a well considered approach to the folder structure, at ”source level” and separating out utility/toolbox objects, and I think you do end up in a strong position with respect to overall control.

That said, it isn’t the same as running GIT over a corpus of text based source code, but then, DIS isn’t a corpus of text based source code.

That said, there was a paper a few years back about using GIT to manage text based items in SAS environments, like the autocall macro area, configuration files. And I guess, something like this could be considered as an additional step, run over the deployed code directory, if only to create a view of the changes over time, at a deployed job level.

Yes, Data Prep in Viya isn’t currently addressing the same range of use cases as DIS currently does, sure. My view is that we will see DIS in play for quite a while yet, certainly any existing sites are likely to persist, as the depth of functionality in DIS will take quite a while to refactor. There’s over a decade of elaboration and finessing vested in DIS.