topic Re: TO UAT OR NOT TO UAT in Administration and Deployment

TO UAT OR NOT TO UAT

stewart_Jardine — Thu, 13 Oct 2022 15:02:20 GMT

Hi all

basically looking to settle an argument, IT bods are 100% indignant that we need 2 enviroments, 1 UAT and 1 production... I personally disagree as users only ever use prod and security profiles and model deployment put enough controls in place. People seem to have VERY STRONG feelings one way or another so..... what's your take?

Re: TO UAT OR NOT TO UAT

Jlochoa — Thu, 13 Oct 2022 15:35:46 GMT

It makes sense to have UAT environment in order to ensure SAS Hotfixes are applied correctly to address an issue. After the Hotfix is verify in the UAT environment, there is confidence that the same Hotfix will be successfully deployed in the PROD environment, reducing the risk of taking down the PROD environment.

Re: TO UAT OR NOT TO UAT

Sajid01 — Fri, 14 Oct 2022 00:53:08 GMT

It is the industry standard practice to have three environment namely development or dev, QA or (Quality assurance or pre production , UAT) and Production.
The order of testing would be Dev /QA/Production.
The QA and Production environment are identical in every aspect. Whatever is to be done on production has to be mandatorily tested on the QA. When that action (executing a job, applying patches or OS updates etc) runs successfully and error free on the QA environment then only it is to be promoted to production. If it fails then it goes back to the development stage.
What your IT team is insisting is perfectly reasonable and the industry standard practice. This recommendation needs to be implemented on an immediate basis without any delay.

Re: TO UAT OR NOT TO UAT

Patrick — Fri, 14 Oct 2022 01:13:47 GMT

And to add to what @Sajid01 wrote:

If you're doing anything with some business importance/criticality in Prod then you also need a 4th Disaster Recovery (DR) environment.

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 01:58:38 GMT

Thanks for the reply, really appreciate you taking the time. I should have mentioned that we are talking about SAS VIYA on a SAS hosted cloud and managed by SAS cloud services, so they will be applying all of the patches and hot fixes

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 02:05:03 GMT

Hi, thanks for taking the time to reply, I think what I'm really struggling with is the fact that in this small company we have 50 SAS users and none of them have ever worked anywhere that has a SAS setup with multiple environments.

I am not trying to argue a point here, I am genuinely interested in hearing opinions and rationales but I don't understand why you would have those environments for any other reason than "that's what you do". It's a hosted service so we don't need to test patches and updates, my 50 users and all of their managers have no appetite to move to separate development and production environments... I may be being completely ignorant, but I can't understand it

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 02:05:45 GMT

Re: TO UAT OR NOT TO UAT

Sajid01 — Fri, 14 Oct 2022 02:45:46 GMT

The next best thing then is to approach SAS itself.
Ask your Account representative to approach SAS with the question.

Re: TO UAT OR NOT TO UAT

Patrick — Fri, 14 Oct 2022 05:40:27 GMT

@stewart_Jardine

What's right for your company depends on your target operating model and what risks and outages/level of availability you can accept. Sure, you can work with a single environment as long as you've got a good backup & recovery strategy in place.

I would assume once the environment and SAS usage matures also your company will have a need for multiple environments - like for example if you start using SAS for things like Fraud or Risk that's regulated and audited.

Re: TO UAT OR NOT TO UAT

Tom — Fri, 14 Oct 2022 03:50:39 GMT

Just to throw a monkey wench (spanner?) into the discussion

I find it works much better to have a dev/test/prod lifecycle for SAS code development on the "production environment". That way your SAS code testing does not need to worry that the version of SAS is different. That the physical enviroment is different.

Leave the DR and system testing of new versions of SAS to IT team. But don't let them force you to use some substandard instance of SAS to do your coding and development.

Re: TO UAT OR NOT TO UAT

Reeza — Fri, 14 Oct 2022 04:30:14 GMT

Questions to help answer:

Will you be okay if the environment is down for up to a week, while IT debugs issues with a patch/fix?
Will you be okay if you lose work?
Is there any privacy concerns with the data or applications you work with?
Who will be maintaining the system, yourself or the IT mods?
Is there a SDLC or version control system in place? Or will be there be one?

50 SAS users is actually a decent sized analytical environment. Considering each analyst typically costs around $200 a day (rough estimate, use your own) that's $10,000 a day in labour costs per day using the system.

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 04:32:48 GMT

Very much my thinking on the matter

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 04:38:27 GMT

1.sas hosted service agreement guarantees recovery in 48 hours
2. Obviously not, but we have daily backups
3. You would need to clarify that, internal or external? Internal no concerns, external definately concerns but can't see how UAT would impact that
4. SAS cloud services
5. We plan on using model manager for this

Re: TO UAT OR NOT TO UAT

Reeza — Fri, 14 Oct 2022 04:46:49 GMT

So a DEV/UAT environment would allow your IT team to test any changes to applications and how they impact SAS without it affecting your workload. For that reason alone I'd agree they need at least 2 environments. UAT is primarily for the IT folks to use, end users rarely use the environments except for end user testing so just make sure you have a good test suite set up for yourself.

There are sometimes bugs in the SAS systems, any application systems. Assuming things will be entirely bug free isn't realistic. As someone on Microsoft 365 the number of bugs we've seen with it is impressive. They usually get fixed soon, but there's bugs for sure. And I know they follow a DEV/UAT environment so who knows what they catch in that environment.

Model manager is more MLOps from what I understand, not sure it will do the full data engineering steps stored as well. I would guess you're going to need an SDLC for that. You also need the processes to supplement Model Manager but I assume you're aware of that.

50 users is a smaller environment but not small by any means. It's worth working towards the SDLC and full cycle. As someone currently architecting a data analysis system for a company (not SAS based) we're going with a Dev/UAT/Prod system but my Data Scientists will not be using Dev/UAT, that's mostly for the IT team and data team to ensure their processes are correct.

Re: TO UAT OR NOT TO UAT

Patrick — Fri, 14 Oct 2022 05:49:37 GMT

@Tom Agree with you for pure SAS script environments but the moment you start using SAS solutions or have critical applications that require high availability things change.

Another reason for environment separation can be that you don't want development teams having access to unmasked PII data and ideally ensure that this separation is implemented on a network level.

The OP also mentions that this is a SAS hosted Cloud environment. Afaik SAS Cloud hosting doesn't support configurations with multiple Lev's.

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 06:17:14 GMT

I appreciate that but our IT team isn't going to be doing anything, it's a hosted and managed service, does that not make a difference?

Re: TO UAT OR NOT TO UAT

Patrick — Fri, 14 Oct 2022 09:02:52 GMT

@stewart_Jardine wrote:
I appreciate that but our IT team isn't going to be doing anything, it's a hosted and managed service, does that not make a difference?

Yes, it does... especially in regards that you have less influence when patches and upgrades you need get applied. It's easier to make your on-prem admin team work outside of business hours and during weekends than it is with a hosted service unless you have a contractual agreement in place for such weekend work (for which you certainly would have to pay a premium).

Re: TO UAT OR NOT TO UAT

Patrick — Fri, 14 Oct 2022 09:16:56 GMT

@Reeza wrote:

......

50 SAS users is actually a decent sized analytical environment. Considering each analyst typically costs around $200 a day (rough estimate, use your own) that's $10,000 a day in labour costs per day using the system.

@Reeza

When I read above initially I was surprised how low the number is. But guess that's the daily average with weekends and leave included.

And after a quick check on https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410020901 guessing that the actual number would still be higher given the industry, location and education level of the average SAS analyst.

Re: TO UAT OR NOT TO UAT

stewart_Jardine — Fri, 14 Oct 2022 09:33:57 GMT

sorry, yes I understand that it makes a difference to when patches are applied and the costs but that has all been agreed and contracted. I meant would that change your opinion on the need for UAT, given that we are not doing any of the patches and upgrades

Re: TO UAT OR NOT TO UAT

Quentin — Fri, 14 Oct 2022 12:28:36 GMT

It's hard to argue against DEV/UAT/PROD.

That said, I'm an analyst / BI developer I've never worked in a place that had these environments. But I really wish I could. Without 3 environments, I have to work on PROD, and make folder branches for DEV/TEST/PROD for my little SAS apps.

What's the down side of this, from the perspective of a developer?

When the server is patched or upgraded, I can't test my code before. That's very scary. I have no input into when a server is patched, and sometimes no advance notice. One day soon (I hope) I'll log on, and M8 will magically be there.
Since my DEV/TEST/PROD stuff is all running one one box, I have to be very careful that jobs running in each 'environment' don't collide, or try to access the same resource at the same time.
Because everyone is doing development work on a production box, it's always possible for someone else's development job to bring the production server to its knees over night, and then all the production jobs fail.
When I make SAS web reports, I need to manage role-based access so that only developers have access to the /DEV and /TEST directories, and users only have access to /PROD. Would be much better/easier/faster/safer to do that at the server/instance level.

I have heard of companies where the SAS developers don't get *any* access to PROD. The work in DEV, test in TEST, and then put in a ticket for IT to promote the job to PROD. Not sure I would love that as a developer, but I can see how it would add a layer of safety in protecting PROD from runaway jobs.

The one benefit of having everything on one box is I don't have to worry about any differences differences between DEV/TEST/PROD environments, as Tom mentioned. But honestly, I'd feel better if I had access to two environments.

That's all from the perspective of a developer wanting to do DEV/TEST/PROD on my own SAS apps. If I were in IT responsible for a SAS environment (or responsible for hiring the contractor who would maintain a SAS environment), I'm sure I'd have even more reason for wanting multiple evnironments.

I would be curious to hear what the SAS hosts are telling you about benefits of having UAT environment or not.