Multi-regional SAS GRID

[JuanS_OCS]

Hello,

I would like to openly share this very architectural question in this community of ours:

has anyone here got experience with, or considered, SAS Grid Manager in a multi-regional deployment around the globe?

• Yes? No? Why?
• PROS? CONS?
• In public cloud? On premises?
• And more importantly: what about performance and security? Is it worth or reasonably possible?

I would like to hear from you, and I am sure this would be useful for many people, not only myself.

Thanks in advance for all your insights and thoughts. Most interested.

Best regards,

Juan

[Kurt_Bremser]

My first guess would be that the performance of the grid would be horribly slow, because of the latencies in the internal grid communications.

But that's just what my gut tells me, as I have no experience at all with respect to (SAS) grids.

[JuanS_OCS]

Same here! That is exactly the first thing I thought. 

For the Grid worker nodes, the computing area, I guess it can be more or less workarounded with hardware/software load balancers and queues definition.

However it seems to me as it would be crazy slow for all the interconnections with metadata and middle tiers.

Unless there is a possibility for huge bandwiths, that I am not aware of, or other technologies.

That, for not speaking of the distributed shared file system. For this one, I can suppose it might be able to be workarounded with the right configuration in the metadata name servers and redundancy/load balancing, since I have seen OK results in other smaller distributed GRID Manager configurations. Much closer howerver, so the latency gap is much smaller too.

Thank you @Kurt_Bremser ! Great first response.

[SimonDawson]

This is an interesting topic indeed. What I'm pondering about is a setup with a geographically distributed metadata cluster with metadata server nodes near users and sets of compute nodes near users. Getting the users workspace's launched near them is the easy part. What would be interesting is if there is a way to teach the clients like SAS Enterprise Guide to favour metadata nodes near them then getting their workspace's to also use a close Metadata server node.

That all said I'd say In practice the simplest approach would be to deploy your grid like usual then deploy the clients with presentation virtualisation or a terminal server. I spend my entire day using a client application streamed over a presentation virtualisation technology from Australia to Cary. The experience is perfectly usable. There are no use cases I can think of when using a SAS platform that require a low latency connection. Having users logon remotely to a client that is close to the server seems like a perfectly reasonable way to make a SAS grid available to users across the globe.

Are there any use cases that require really low latency user input that I'm not thinking of?

[JuanS_OCS]

Hello @SimonDawson,

thanks a lot for your thoughts, another very good answer, and even great topics: metadata and clients.

Regarding metadata, I think it would be possible to make clients to connect to a specific metadata, but if the connection goes the other way arround, from object spawner to metadata,I cannot think a way this can be known except by dangerous customizations. Furthermore, the metadata cluster sync would probably make all metadata nodes quite slow.

About the clients, yes, virtualization is the answer, definetely, with unfair regional latencies. 

Well, user input does not have to be a big problem, necessarily, but for SAS EG can be an issue, since input is closely related to output. 

That latency can become very annoying for a user (even frustrating) , for things as the EG tips, the exploration of a dataset, and such.

Same goes for the AJAX functionalities in SAS Studio or others, where client-server iterations is small but constant (thanks to HTML5! Or Flash).

For those AJAX "small issues" you would initially say that not that much important, specially if you have a load balanced web. But that problem can be exponential because of connections to compute and metadata. Or not? 

What do you think about that?

[SimonDawson]

When I mentioned presentation virtualisation I was specifically referring to Citrix. From my experience with using Citrix across a very high latency link (300-400ms) there is no issues with the latency of web apps between the browser and middle tier of whatever apps because they are both close to each other. The only latency you get sometimes is with graphics being rendered and obviously scrolling in certain applications is quite jarring.

I can't speak much to using SAS client applications specifically but I'll speak to my own experience of using another desktop application over a connection to the other side of the globe. In this application I use I predominately edit text and navigate through various screens doing data entry and manipulation. Occasionally I'll need to use web applications over the Citrix connection also. The most interesting thing about this application I'm interacting with is that it responds orders of magnitude faster when using Citrix when navigating between screens when compared with running the application locally. This app is quite chatty to populate the various things on the screen. The only thing I do sometimes notice is there is a bit of latency with operations like cut and paste when you edit text. It takes a little getting used to but once you are used to it like I mentioned before it's perfectly usable.

If I was to think about how I use SAS locally myself I'm confident I could port my workflows to a Citrix connection on the other side of the world and for me it would not make much difference but I guess it depends on your users.

[Kurt_Bremser]

In my experience the IOM bridge is particularly "talkative" and suffers heavily from latencies. While loading a newly created dataset into EG takes about 3 seconds (under heavy load on the server) when I work in the office, it can take up to a minute when working under the same load via VPN from the home office.

So the best bet is to always keep clients that use the IOM bridge "close" to the compute servers. The same might be true for metadata connections.

[JuanS_OCS]

Thank you @SimonDawson, @Kurt_Bremser, for your thoughts about the clients. It is great to have good minds to discuss this and make a point.

Yes, latency does show up when latency is higher and apps are chatty.

That is also what we need to consired regarding the data and the filesystems.

This is something applies for any SAS or non-SAS application.

I am interested into diving deeper about the metadata and middle tier, clustering through this latency. Is it workable?

I also would like to drop this question, probably more suited for SAS colleagues: are you aware of any existing successful implementation of this type?

@MargaretC explores and explains this topic quite well here: https://communities.sas.com/t5/Administration-and-Deployment/Does-It-Matter-Where-the-Various-Compon...

[MargaretC · Accepted solution]

I have spent the day discussing this several SAS Grid deployment experts about the pros and cons of doing this.  The main issue is sharing data between the different regions.  If this is not a requirement, then yes you can setup dedicated SAS Grid nodes in local regions that can only access the data that is locally attached to their dedicated SAS Grid compute node.  This is currently being done successfully by a global insurance firm.

However, if you want to keep certain aspects of SAS Grid (authentication, web services, metadata, ...) in a single location for all the regions to us, then the latency to access the data in this centralized location has caused major performance issues and many upset SAS Grid users.  The latency overhead of accessing data across slow WAN connections make SAS Grid unusable for many customers.

So, with a good understanding of where the data is and setting up the system to keep the data transfer across WANs to be minimal, this can work.

[JuanS_OCS]

Hello @MargaretC ,

thank you very much for your insights and opening those discussions. It is great to receive some thoughts from you, a person that has a good oversight over current working and not working environments, plus all the tests and benchmarks done by SAS in the background.

I wonder how a SAS Grid deployment, with local data, would keep using the main advantages of the Grid solution. I would imagine this kind of deployment would not do much of resource managing, please correctme if I am wrong.

I can also imagine the frustration that can cause a wrong design, it is a very delicate matter, specially if we are talking about a SAS environment that is intended to support a global community of SAS users.

I am now in the process of creating a solution plan, for a similar kind of global company, that would like to unify their SAS environments into a single one. The first question is of course the general design.

The other hovering question I have is, if the solution would be in the cloud or on premises. The customer whom requested the solution plan is now aware that the cloud option is a natural one, but we need to be careful because public clouds have challenges regarding performance. Also, my main concern at this moment is that they have contracted, through a blue chip cloud service provider, the Azure cloud.

I am an old follower of all your, and Tony's great suggestions regarding SAS Grid, Storage, and SAS in the cloud. I see a lot of progress, specially regarding AWS, but I could not see many details for SAS or SAS Grid in Azure, except this paper: https://www.sas.com/content/dam/SAS/support/en/sas-global-forum-proceedings/2018/1866-2018.pdf

My first question, before even asking for recommendation or specifications for SAS Grid in Azure would it be: do you think  Azure is a well fit public cloud provider for SAS Grid?