Discussion of transition from SAS9 to Viya. Call for input.

[ErikLund_Jensen]

Hi community

As we are approaching the retirement date for SAS9, the time has come to start planning a migration to Viya. We don't know much about Viya, but we see three bigger obstacles that I want to discuss with the community, and I hope this discussion will ease our minds, though I fear the worst.

SAS Information Delivery Portal

SAS9M8 is available now with support expected until 2028, so we thought that we had 5 years left to find out what to do. But it is a nasty surprise to see that the Information Delivery Portal is retired with this release. M7 is supported until 2025, so we have a couple of years to find a replacement for our current reports.

They are all built with use of javascript to provide selectboxes to control the content of displayed tables and graphs, and we use a framework supplied by SAS to generate the scripts. It works with special DI Studio transformations that generate the javascript and write it to the web server, but I am not sure if this framework is "official" SAS or something developed in our local SAS company. So I wonder if or how a similar functionality could be obtained in SAS Visual analytics.

Converting SAS programs to run under SAS Viya

There is a lot of information available on this topic, but it seems that converting even a simple job will be a time-consuming task, and we have many jobs doing things I never saw addressed, like interacting with Linus to use character conversion routines, retrieve folder content and other information from Linux, interact with LSF Process Manager, get content from FTP servers, run bash scripts, using macro loops, fcmp-functions etc. So how are these non-standard tasks implemented in Viya, and what to do if they are not?

We have about 6.100 jobs with at least 30.000 transformations built in SAS Data integration Studio and besides about 200 SAS programs executed by CRON/bash scripts, and we have no idea about the total time that should be allocated to the whole conversion.

Deployment and Scheduling

We run a SAS Data Warehouse with more more than 700 LSF Flows containing more than 6.000 scheduled jobs in the daily batch. At least 600 of the flows depend on previous flows, and a dependence chain can be up to 20 flows long. The scheduling works so any flow is held until all previous flows on the daily run list are finished, and then be either triggered to be run by LSF or omitted if a previous flow is failed. This is the basics, there is other funcionality included.

We deploy the jobs from SAS Data Integration Studio, where jobs are organized so one folder with jobs corresponds to one flow. Flows are created and jobs added in SAS Management Console, and they are scheduled to LSF as "Manually" without any triggers or depencencies defined. It is not necessary, because our home-built scheduler (SAS code) does the rest. It determines the flow dependencies by extracting information from SAS Metadata, and it monitors job and flow results  and triggers or excludes flows depending on previous results. Everything is written to a control database, and a web application is used to follow the current batch status. We have used this scheduler for 12 years, and it works with minimal intervention, so it takes a few hours per month to keep the the whole batch up to date and running, including rerunning of flows after errors.

Now we try to figure out how we can get a similar functionality in SAS Viya. It seems that Viya is not built with a batch environment in mind, and from what I have read I gather that it is a long and complicated process to deploy a SAS Studio job as program code and create a script to execute it, and then what? It seems that the concept "Flow" and functionality comparable to LSF is missing in Viys, no scheduler to run the scripts with respect for dynamically derived dependencies, and no metadata to extract dependency information from. We are not alone in the world with a setup that depends on traditional batch processing, so somebody must have a setup to handle that in viya. The question is how time-consuming the tasks with defining scheduling details, running and rerunning jobs, maintaining of correct execution order, monitoring batch progress etc. is, i.e. how many extra hands will be needed in the daily work.

When it comes to development, we use about 3 minutes to deploy a new/changed job from DI Studio and add it to a flow. We suspect the process will be a much more complicated and time-consuming process in Viya, and it might require a lot of additional work to set the scheduling up too, depending on the solution to we find to handle that. We have at least 50 new/changed/obsolete jobs in at least 10 flows per week, often several times more, and we have no idea about the extra workload we can expect.

Today, We are 5 persons in a team to run a Linux cluster + metadata servers and web servers, get data from external sources, manage authorizations, manage client program, promote jobs to production, schedule jobs, monitor the daily batch, handle odd tasks and help users. We support 25 ETL developers and more than 100 EG Users. We cannot expect to cope with this without extra help. A team of extra persons is probably needed for the migration process, but we fear that we also will need several extra persons to handle the daily routine afterwards with a decent level of quality and quick response.

We need an estimate, so we can prepare our management, but more than anything we need a basic understanding of how we are going to do things in the future, down to tools and techniques. Our SAS consultants has a tendency to become uneasy and vague when these questions come up, so I look forward to any comments, thank you.

[SASKiwi]

Congratulations on an excellent post.

We too are considering what our future SAS roadmap will be. Initially we looked at a purely SAS Viya transition but that didn't stack up. You don't mention whether you've been looking at cloud-based or on-premises Viya. We feel that with Viya, SAS's architectural complexity reaches new heights. Unless your IT shop is clued up on cloud-hosting, Kubernetes, MS Azure, AWS, blah, blah blah....we think Software As A Service (SAAS) is definitely worth considering. And SAS offers this.

To cut a long story short we are now considering a hybrid transition which will involve both SAS 9.4 M8 and SAS Viya 4 (both on-cloud). We will use 9.4 for ETL and Viya for our presentation layer which is SAS Visual Analytics. That means we can continue to use SAS EG, SMC, LSF as our main SAS interfaces, can avoid reengineering all of our ETL, retain our metadata-based security model and so on. But we get to dip our toes into the water with Viya for VA dashboards and reporting and get to experiment with CAS libraries and other Viya goodies.

To summarise, I've heard other SAS customers are choosing to retain SAS 9.4 as well as using Viya for similar reasons to us. You ease the migration pain while not being pressured to switch over to Viya all at once. With all due respect, I regard Viya 4 as still an immature platform that is best eased into rather than go "big bang". I'd like to see more GUIs instead of all of the command-line interfaces and tools that seem to have proliferated. Command line interfaces are so last century...

Hope this gives you food for thought. AFAIK, SAS VA is supposed to be the eventual successor to SAS Info Delivery Portals - see link. I suggest you work with SAS in that regard to see if it fits your requirements.  

[ErikLund_Jensen]

@SASKiwi 

Thank you for this interesting input. 

We share your opinion that Viya has a very complex architecture, and our IT already struggles with Linux Grid, as it is basically a big Windows installation. I am afraid that cloud hosting isn't an option due to GDPR-rules, and I fear that in-house hosting will be a challence for our IT Shop, as it already struggles with Linux Grid, Kerberos etc. I also think that we need to continue with SAS9 as long as possible. It will take years to rebuild out actual batchjobs, whether they should be converted to Viya or something else, so the sheer size of the task forces us to choose a stepwise migration. Besides, the user organisation will also need a long time span do a stepwise migration from SAS Enterprise Guide to SAS Studio or whatever.

I had hoped for responses from people that are not only planning a migration, but in the process, so they had some answers regarding conversion time and a new way of handling promotion/scheduling/batch, so I wonder if I have choosen a wrong forum for this post. It would also be interesting to get some comments from SAS representatives. It is possible that nobody read my post, but it is worrying to think that the silence is caused by a lack of good answers.

[Quentin]

I also think this is a very important topic for the SAS community, and hope to see more input from both admins and developers about their migration plans, or migration experiences.

I found this related thread interesting, where someone was basically asking "why will this transition take longer than I would hope?" https://communities.sas.com/t5/Administration-and-Deployment/Input-on-others-moving-SAS-devs-from-Ba... .

As a developer, I like the idea of running Viya and SAS 9 environments in parallel for a year.  Yes, I know that Viya has a SAS 9 compute server inside of it, so most of my code will not need to change.  But from what little I've read, even as a developer there are many significant differences between a SAS 9 environment and a Viya environment, which will require time to learn, before thinking about migrating jobs/stored processes/etc, and then validating.  When I first worked on a SAS 9 BI environment, it probably took me a year to get up to speed with basic metadata security, stored processes, and stored process web app.  It was also my introduction to EG.  I'm guessing my personal transition to Viya would also take about a year.

[Al_CUA]

Just to add, there will likely be licensing implications with running SAS 9.4 and Viya. Viya also uses a subscription-based rather than a Core based model. 

As mentioned above, moving to a microservices approach with Viya, particularly on-prem, will be a steep learning curve. Yes, GDPR will be a pain. There are an increasing number of countries with GDPR-like rules too.

On the positive side, the underlying improvements will likely come thick and fast, with the underlying technologies mainly being popular open-source projects. 

In terms of support it looks like SAS will follow the LTS vs stable approach. 

[SASKiwi]

 @ErikLund_Jensen - Looks like you are not the only one with Information Delivery Portal issues. Have you seen this? https://communities.sas.com/t5/Administration-and-Deployment/SAS-9-4-M8-is-now-available/m-p/859299#...

I'd suggest engaging with your SAS account manager regarding the lack of an obvious upgrade path for Information Delivery Portal. 

[Quentin]

If I were tasked with migrating a suite of Information Data Portal reports to Viya, I would probably reach out to @AllanBowe to discuss his company's approach for developing SAS web apps that run on both SAS 9 and Viya.

[ErikLund_Jensen]

Hi @Quentin @SASKiwi @Al_CUA 

Thanks for your input so far, both comments and links.

What we know

We are forced to implement a new solution due to the retirement of SAS9, but we will in any case need SAS9 for years to come because the replacement of 6000+ production jobs deployed from DI Studio combined with building a scheduler to handle the daily batch will be a time-consuming process.

SAS Data Integration Studio
Replacement of SAS DI is a big issue. It seems that the SAS9 Compute Server in Viya can handle many of the challenges with conversion of ETL jobs, at least  @Al_CUA to redeveloping everything in e.g. Python or Microsoft products. This is an important observation because we have a management that are open to other solutions beside SAS.

SAS Enterprise Guide
We have about 100 EG users. In my understanding SAS Studio will be the only programming interface to Viya, so EG will also be discontinued with SAS9. A lot of EG projects are used as production jobs in workflows, so they need to be converted. I assume that a conversion to SAS Studio is more or less straightforward, but anything else will be a challenge.

SAS Information Delivery Portal.
Most reports can be handled in SAS Visual Analytics or any other BI solution, but we have several which can be considered applications, as they include data collection, automatic selection of content to the current user etc. They are built by external SAS consultants, and we have no ideas about what to do with those, but we could always get somebody to develop web applications instead, and it would probably be a minor expense compared to the rest.

SAS Visual Analytics
We use SAS VA today in a separate server Linux cluster. I don't know if there is a full Viya platform there also, but in any case we don't use it as a programming platform today. There is nothing between the writing of data to CAS libnames from our SAS9 environment and the building of reports on these data, except for maintenance of CAS libnames. For other reasons, our management is looking into a possible replacement for SAS VA, but this might be irrelevant if we already have a Viya platform and ends up with choosing Viya as the future platform instead of SAS DI and SAS EG.

Would anybody know if VA always comes with a full Viya that are open to use as a programming platform under the VA licence? 

Daily operation
Job promotion, scheduling and batch execution is the remaining - and in my position the most important - issue. We don't know how to handle the concept of separate environments for development and production, controlled promotion and job scheduling in Viya (or any other product). We also don't know how a replacement for our current batch setup could be established. Is is - as described in a previous post - based on actual metedata content, so execution order is established based on dependencies extracted from metadata, so releasing, omitting or rerunning jobs in case of errors is fully automated. 

Would anybody have links to papers adressing batch operations in a Viya environment?

[Quentin]

I think it's important to say that SAS 9 is not being retired.  Yes, IDP is being retired, but SAS 9 lives on.  And there is no end-of-life date for SAS 9.  If someone has been told an end-of-life date for SAS 9 from a reliable source, please correct me.  Yes, SAS as a company clearly would like to push the community completely to Viya, but strategies change.  If SAS goes public, or is acquired, there could be a significant change in strategy.  And SAS has announced their plan to go public.

In terms of DI jobs, I think it's safe to assume that SAS code can be migrated to Viya and adapted.  There is a Viya 3.5 environment in my company.  They created DI jobs using visual DI objects/transformations. When they were exploring migrating to Viya 4, they were disappointed to learn that many of the transformations they used did not exist in Viya 4.  Thus the SAS migration tools would convert those unavailable transformations into user-written SAS code nodes.  For me, this would not be a problem, as my DI jobs consist only of a single user-written code node.  But most DI Studio developers tend to use the visual transformations, and try to avoid mixing in user-written code.  The inability to convert their existing DI jobs into Viya 4 jobs without user-written code nodes has resulted in them exploring alternative ETL tools.  I think the thought was that since they didn't like the migration path, they would need to recreate the jobs, and then might as well explore options.

[ChrisHemedinger]

Great discussion here, and I'm grateful for the good input from the community experts. I'll chime in with a couple of clarification points:

• @Quentin is correct - SAS 9 is not "being retired". SAS 9.4M8 was just released and this extends the standard support commitment from SAS. There is no "end of life" deadline -- many customers rely on SAS 9.4 and we expect that to continue even as more of them adopt SAS Viya. But you are also correct in that some solutions/applications are not currently supported in Maint 8 and those products have a documented migration path (ex: SAS Customer Intelligence 6->SAS CI 360).
• GDPR (and data privacy rules in general) are concerns that are accommodated in SAS solutions and the various cloud-hosting offerings from providers like AWS, Azure and others. All of these providers offer options that comply with regional government regulations and security standards. Of course, moving to cloud in a way that complies with the rules and enforces a "privacy by default" ethos -- that's a project that requires planning and change management in order to be successful, independent of SAS or any specific vendor. See the SAS Trust Center resource for more information about how SAS works with these practices.

[ErikLund_Jensen]

@Quentin @ChrisHemedinger 

Thanks for clarifying (and to Chris: thanks for your guest appearance at our local Danish SAS Platform Meeting this month - very interesting and informative. Do you have a graph or table showing posts/country).

I am aware that M8 is available now, so V9 isn't going into immediate retirement. But our SAS representative has informed us about M8 being the last maintenance update with end of support in 2028. In our organization - a big municipal government - it may take years to come to an agreement, which combined with the time needed for migration/redevelopment makes me think that planning should start now. 

Regarding GDPR, the obstacle is the Court of Justice of the European Union's judgement in the Schrems II case, which rules out cloud solutions hosted by US owned companies. Links:  https://www.eff.org/deeplinks/2020/07/eu-court-again-rules-nsa-spying-makes-us-companies-inadequate... 

https://www.europarl.europa.eu/RegData/etudes/ATAG/2020/652073/EPRS_ATA(2020)652073_EN.pdf 

Our DPO has the final word, and there is a discussion right now on continued use of Office365 in our organiszation.

So far, nobody has given any comments on batch processing in a Viya environment. Are we alone in the world with an automated metadata-driven solution, while everyone else sees job processing as a separate activity where dependencies and triggers are manually maintained in e.g. LSF?

[Patrick]

Thanks for sharing this ruling and the impact on cloud hosting.

I'd be rather surprised if companies like Microsoft and Amazon wouldn't come-up with a solution to meet GDPR requirements to stay in the EU market. May-be that's the motivation for below.

With M8 standard support ends in 2028. But even if there shouldn't be an extension or another maintenance release that doesn't mean you'd be in an unsupported state after 2028 - the support level then just changes to "limited support". You can see under this link that SAS Tech Support even lists SAS8.2 as still supported.

As far as I know limited support means that no new hotfixes will get created - but after 5 years only very "exotic" bugs should remain for which there isn't a workaround. 

We can't predict the future but based on decades of past experiences it's hard to imagine that SAS will just stop to support "older" versions. 

I agree with you that the DI migration path to Viya still seems to be "lacking" - at least as far as I know. There are more and more matching transformations on the Viya side but it doesn't feel things are fully there yet. I'm sure they will at one point but with the current Viya version it's eventually still about a co-existence between SAS9.4 for DIS and most of the rest under Viya that meets real life requirements the best.  ...that's just my personal opinion and I'm currently not fully up-to-speed with Viya 4 so I can be wrong.

[Quentin]

---

@ErikLund_Jensen wrote:

I am aware that M8 is available now, so V9 isn't going into immediate retirement. But our SAS representative has informed us about M8 being the last maintenance update with end of support in 2028. In our organization - a big municipal government - it may take years to come to an agreement, which combined with the time needed for migration/redevelopment makes me think that planning should start now. 

Thanks @ErikLund_Jensen  that is very big news.  Certainly if SAS has decided that M8 will be the last SAS 9 release, many enterprises will need to start migration planning now.

Given the magnitude of that SAS 9 roadmap decision, and the magnitude of the decision to share the roadmap with customers, I can imagine a few possibilities:

• Your SAS rep knows the SAS 9 roadmap, and is telling customers information that SAS would probably prefer they not share (at risk of scaring customers)
• Your SAS rep is mistaken about the SAS 9 roadmap (or the SAS 9 roadmap has not been defined)
• Your SAS rep is using the threat of losing SAS 9 full support as a sales strategy to push you toward Viya.

Hopefully we will see an official communication from SAS this year about the roadmap for SAS 9.  If not, hopefully there will be unofficial communication in the hallways of UG conferences etc.

I think we were several years into M7 before hints of forthcoming M8 started to appear in documentation. If there is risk of M8 being the last SAS 9 release, I don't think businesses will wait 2-3 years to learn the roadmap. 

I've spent the past year fighting IT at my company, telling them that there was no end-of-life date for SAS 9.  They don't want to go to Viya, and don't really want to keep the SAS 9 BI server.  If they are hearing from the SAS rep that M8 will be the last SAS 9 release, then I would understand why they are pushing alternatives (snowflake, data bricks) so hard.  Ugh. 

[ChrisHemedinger]

I am not aware of any official decision about "no more maintenance releases" (or otherwise extension of support) beyond M8. While there is not another one scheduled at the moment, SAS has always been responsive to customers and their needs on this front.

At the same time, SAS continues to improve SAS Viya to make it easier and "less of a lift" for customers to adopt. I imagine this will play out as a "meet in the middle" approach: SAS 9.4 continues as needed, and SAS Viya becomes more accessible with faster migration paths. We've seen this to-date in SAS Analytics Pro for SAS Viya (which you can install on a desktop and run via Docker) and in SAS Viya on Microsoft Azure Marketplace (which gives you full Viya in just about an hour with a pay-as-you-go model).

[Al_CUA]

Hi Erik

You may have come across these already but here is some material on promoting between environments, use a REST API.
https://www.youtube.com/watch?v=jDhj2qIBTSU

And this is for the scheduling, Apache Airflow.
https://youtu.be/z1zfSbM-CJo?t=451

btw, IBM LSF is still around. 
SAS 9.4 Support for Other Third-Party Software Products | SAS Support