topic Re: Decomposition and Orchestration in DI Studio and LSF. in SAS Data Management

Decomposition and Orchestration in DI Studio and LSF.

AngusLooney — Tue, 11 Dec 2018 21:28:58 GMT

As part of the UK and Ireland SAS User Group - https://www.meetup.com/SUGUKI/ - I volunteered to give talk on " Decomposition and Orchestration in DI Studio and LSF".

Posting the slides here as well in case anyone finds them useful, thought or comment provoking - or perhaps to kick off an discussion.

The main point I was hoping to make was the features provided by LSF or equivalent offerings should be viewed as an intrinsic part of the toolset, and that they promote, support and enable the adoption of fundemental approaches to how ETL processes are designed, built and operated. "Zen and the Art of DI Studio" - almost.

It's an independant user group, so the slides are my own thoughts/ideas/opinions, in no way "SAS official" or anything - hence the non-corporate typography.

Enjoy.

Re: Decomposition and Orchestration in DI Studio and LSF.

LaurieF — Wed, 12 Dec 2018 19:38:11 GMT

That's bloody excellent. Thank you. I've come to the same, or similar, conclusion from a slightly different direction: a DI job should do one thing (have one target), and that makes it easy to put into an LSF stream. LSF can then do its magic by parallelising as much as possible, and we can all go off to the pub.

Re: Decomposition and Orchestration in DI Studio and LSF.

AngusLooney — Wed, 12 Dec 2018 20:15:54 GMT

Absolutely. If this was useful, there are a couple of other decks (and a video) from me hereabouts on related topics, not to blow my own trumpet...

Re: Decomposition and Orchestration in DI Studio and LSF.

KumarT_SAS — Fri, 11 Jan 2019 05:29:10 GMT

Interesting and a great process to combine the mix of data management tools. If we look at SAS Grid Manager with LSF, we are looking at a more advanced version of the scheduling process, parallelism and most importantly load balancing. Just LSF with SAS DI Studio, definitely looks lean and less complex for most of the repeatable data management jobs.

Re: Decomposition and Orchestration in DI Studio and LSF.

AngusLooney — Wed, 06 Mar 2019 10:26:45 GMT

Single server LSF is just the same "LSF Grid" stuff as a full grid, just on a single server.