TO UAT OR NOT TO UAT

[Reeza]

So a DEV/UAT environment would allow your IT team to test any changes to applications and how they impact SAS without it affecting your workload. For that reason alone I'd agree they need at least 2 environments. UAT is primarily for the IT folks to use, end users rarely use the environments except for end user testing so just make sure you have a good test suite set up for yourself. 

There are sometimes bugs in the SAS systems, any application systems. Assuming things will be entirely bug free isn't realistic. As someone on Microsoft 365 the number of bugs we've seen with it is impressive. They usually get fixed soon, but there's bugs for sure. And I know they follow a DEV/UAT environment so who knows what they catch in that environment. 

Model manager is more MLOps from what I understand, not sure it will do the full data engineering steps stored as well. I would guess you're going to need an SDLC for that. You also need the processes to supplement Model Manager but I assume you're aware of that. 

50 users is a smaller environment but not small by any means. It's worth working towards the SDLC and full cycle. As someone currently architecting a data analysis system for a company (not SAS based) we're going with a Dev/UAT/Prod system but my Data Scientists will not be using Dev/UAT, that's mostly for the IT team and data team to ensure their processes are correct. 

[stewart_Jardine]

I appreciate that but our IT team isn't going to be doing anything, it's a hosted and managed service, does that not make a difference?

[Patrick]

---

@stewart_Jardine wrote:
I appreciate that but our IT team isn't going to be doing anything, it's a hosted and managed service, does that not make a difference?

---

Yes, it does... especially in regards that you have less influence when patches and upgrades you need get applied. It's easier to make your on-prem admin team work outside of business hours and during weekends than it is with a hosted service unless you have a contractual agreement in place for such weekend work (for which you certainly would have to pay a premium).

[stewart_Jardine]

sorry, yes I understand that it makes a difference to when patches are applied and the costs but that has all been agreed and contracted. I meant would that change your opinion on the need for UAT, given that we are not doing any of the patches and upgrades

[Quentin]

It's hard to argue against DEV/UAT/PROD.

That said, I'm an analyst  / BI developer I've never worked in a place that had these environments.  But I really wish I could.  Without 3 environments, I have to work on PROD, and make folder branches for DEV/TEST/PROD for my little SAS apps.

What's the down side of this, from the perspective of a developer?

• When the server is patched or upgraded, I can't test my code before.  That's very scary.  I have no input into when a server is patched, and sometimes no advance notice.  One day soon (I hope) I'll log on, and M8 will magically be there.
• Since my DEV/TEST/PROD stuff is all running one one box, I have to be very careful that jobs running in each 'environment' don't collide, or try to access the same resource at the same time.
• Because everyone is doing development work on a production box, it's always possible for someone else's development job to bring the production server to its knees over night, and then all the production jobs fail.
• When I make SAS web reports, I need to manage role-based access so that only developers have access to the /DEV and /TEST directories, and users only have access to /PROD.  Would be much better/easier/faster/safer to do that at the server/instance level.

I have heard of companies where the SAS developers don't get *any* access to PROD. The work in DEV, test in TEST, and then put in a ticket for IT to promote the job to PROD.  Not sure I would love that as a developer, but I can see how it would add a layer of safety in protecting PROD from runaway jobs.

The one benefit of having everything on one box is I don't have to worry about any differences differences between DEV/TEST/PROD environments, as Tom mentioned.  But honestly, I'd feel better if I had access to two environments.

That's all from the perspective of a developer wanting to do DEV/TEST/PROD on my own SAS apps.  If I were in IT responsible for a SAS environment (or responsible for hiring the contractor who would maintain a SAS environment), I'm sure I'd have even more reason for wanting multiple evnironments.

I would be curious to hear what the SAS hosts are telling you about benefits of having UAT environment or not.  

[stewart_Jardine]

SAS are very much saying "its up to you you" however our support partner Amadeus have unofficially said that they agree with my rationales and approach for not having a UAT

[Reeza]

Hosted and managed just means the server is owned by someone else and you're contracting out the work. It doesn't change the best practice recommendations of how the work is done, such as DEV/UAT/PROD. If the managed provider doesn't have a UAT/DEV environment they're doing everything in production as well. 

What happens when your DB system is upgraded, how will you test the new DB against SAS? In production?

Patches/Fixes are tested by running it in dev against jobs and ensuring your output is the same. 

Can you work without a UAT, yes. Will it add labour costs, almost certainly. Will the labour costs be more than hosting, most likely. One benefit of cloud, is that UAT can be de-resourced in between usages so it's not a permanent entity in some ways. It only exists when you need it making it more cost efficient. Assuming you're using a cloud license of SAS that's billing by the resource usage this is worth the labour costs most likely if you do the math.  If you're using a licensing system that's fixed cost per year that's more expensive then. 

[Patrick]

---

@Reeza wrote:

......

50 SAS users is actually a decent sized analytical environment. Considering each analyst typically costs around $200 a day (rough estimate, use your own) that's $10,000 a day in labour costs per day using the system. 

---

@Reeza 

When I read above initially I was surprised how low the number is. But guess that's the daily average with weekends and leave included. 

And after a quick check on https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1410020901 guessing that the actual number would still be higher given the industry, location and education level of the average SAS analyst.

[Sajid01]

Hello @stewart_Jardine 
You are missing the point.
The production server is the one in which there is zero tolerance for disruption. It runs processes essential to running the business and any disruption impacts the company's business.
In realization of this objective another server identical in all respects (You call it by any name UAT/QA etc ) is installed before it to test that the process you are promoting to the production server will run without any issue.

It is a precautionary measure and safety net something like an insurance policy one takes on a car or health insurance The license cost is typically lower than production license like the insurance cost.
Running a production server without UAT is like driving a car without insurance. Whether one likes it or not one has to take an insurance.  Even if there is no accident the driver/owner may be penalized by law enforcement.
There may be a scenario where no work of significance is done or the business has the capacity to absorb the losses , one may go without a UAT.

[stewart_Jardine]

Hi Sajid,

I am genuinely listening and taking everything in, so apologies if I am missing the point. My instant reaction though is it is not driving a car without insurance, it is driving a car without a backup car that you keep just in case. The "insurance" I consider myself to have is a guarantee of 98% availability of my platform from SAS hosted services.  The other thing to say is that a UAT environment is typically set up without the ability to access "real" data or at the very least everything is anonymised, so to go back to the car analogy, its a backup car that cant go on the same roads 🙂

[Reeza · Accepted solution]

---

@stewart_Jardine wrote:

Hi Sajid,

I am genuinely listening and taking everything in, so apologies if I am missing the point. My instant reaction though is it is not driving a car without insurance, it is driving a car without a backup car that you keep just in case. The "insurance" I consider myself to have is a guarantee of 98% availability of my platform from SAS hosted services.  The other thing to say is that a UAT environment is typically set up without the ability to access "real" data or at the very least everything is anonymised, so to go back to the car analogy, its a backup car that cant go on the same roads 🙂

---

98% uptime means you can be down a full week a year. As long as that works for you. 

For the driver the back up car is useless. For the brand new mechanic fixing the car and maintaining the system having a car to practice on before doing it on a real car makes sense. And it's a brand new mechanic because every patch/fix is unique/different so usually something different, plus in my experience the MSPs have high turnover rates so there's less retention of knowledge there anyways. 

[stewart_Jardine]

fair enough, jsut confuses me that we are paying SAS to be our mechanics and we still have to pay for the second car 😜 ... ok I have taken this analogy too far. I do take on board all your points though. Thanks for taking the time

[Reeza]

As mentioned before you're just outsourcing the work, you aren't changing the nature of how systems are managed and run. Without a UAT you're asking the people doing the work to do it without the standard tools/setup.

[stewart_Jardine]

I also don't pay for my local garage to have test cars and tools 😜