SAS cluster expansion

[Manoj07k]

Guys , We are approaching physical server end of life. so we have a thought of expanding the cluster by add new node and shift the services from existing server.

We have 9.4 M6 Grid with metadata clustered on 3 nodes , Web on 2 nodes (load balanced) and compute nodes . I just wanted to understand is the idea is feasible and if any supporting documents we have in SAS.

Thanks in advance.

[Sajid01]

As a general rule, if your license permits you should be able to add another node to the cluster.

Alternatively in a three node cluster, two nodes complete a quorum. Shut down one node and replace with a new machine. 
Your specification of your grid architecture is not very clear. If you can mention it clearly we can give a better idea, (Metadata, midtier, grid control and grid nodes - these need to be specified)

[Manoj07k]

Appreciate  your response.

We have below arch now with 11 nodes ( 1 to 11 ) , we want to add nodes 12 to 22 to the cluster and move the binaries and config, since node 1 to 11 is approaching End of life. Once we moved to 12 to 22 we will drop 1 to 11 nodes.

Node Name	Purpose of Node
Node1	Primary Grid Control server Compute
Node2	Secondary Grid Control server Compute
Node3	Master   Metadata
Node4	Failover   Metadata  + Compute
Node5	Failover   Metadata  + Compute
Node6	Web Tier node 1
Node7	Web Tier node 2 ( Load Balanced)  + Compute
Node8	Compute node
Node9	Compute node
Node10	Compute node
Node11	Compute node
Node Name	Purpose of Node
Node12	Primary Grid Control server + Compute
Node13	Secondary Grid Control server + Compute
Node14	Master   Metadata node 1
Node15	Failover   Metadata node 2 + Compute
Node16	Failover   Metadata nod3 + Compute
Node17	Web Tier node 1
Node18	Web Tier node 2 ( Load Balanced) + Compute
Node19	Compute node
Node20	Compute node
Node21	Compute node
Node22	Compute node

We use SAS 9.4 M6 with Linux 7.1. 

Also , we have LSF and SPDS installed in the cluster.

Let me know your thoughts. 

Thanks.

[huntdoug]

As a general rule do not put any other SAS server or share the server but with one metadata server.  The metadata server should have it's own system.  If using a metadata server I would recommend anyone server be as big as all 3.  When the cluster is down 1 node it's really down half - 1 slave and 1 master.  The slaves handle all the work.  The master updates metadata (rw) and the slaves(ro) handle the users connections and read only queries.  So running 1 metadata slave means it does the work of 2 and the master still does the read-write work.  Consider the CPU/Memory/IO that the 3 do.  I'd triple it.  I'd give lots of cpu cores so that the Metadata Server has the most threads to work with as this determines by algorithm how wide that is.  There still can be a bottleneck if you have a user running a long read that may have to finish before the update can be done so that there are no dirty reads.   Honestly a single metadata server node flattens the hierarchy and latency in my mind over a cluster of metadata servers.  A cluster complicates administration.  A cluster is something one must be robust with.

[Sajid01]

Thanks @Manoj07k 

1. Basically you are replacing your existing setup with a new environment. It will be simpler for you to leave the existing arrangement as it is and build new server setup. Once the new one's have been tested and operational the older ones can be decommissioned. Building a new system will help you rationalize your architecture and may help bring down the costs. You have the opportunity to remove any bottlenecks and trouble spots.

2. I see that your metadata and middle tier (Web) are also serving as compute nodes. I would not do this.
3. If I   had the choice, I would prefer to separate the entire setup  into three logical groups each with a distinct role.( Make your own judgement to see if this works for you.)

• Group 1: The Metadata tier :Existing nodes 3,4 and 5. Will function as Metadata Servers only. Remove the compute functionality from these server. They will be clustered.
• Group 2:The middle tier : The existing 6 and 7. I will add one more and remove the compute functionality. Clustered. My understanding is that pure middle tier with no compute activity would not add to licensing cost.(SAS employees please enlighten on this. And Manoj you need to verify this.)
• Group 3: The Compute tier : The Grid Control Servers and the grid nodes. My thinking in a true Grid would not need a secondary grid control server.( May be good to have). If the Grid Control server fails, a grid node should take over.  If I am the decision maker, would do away with the secondary Grid Control server.

4. If you do not want to follow the approach of building a new system, You can do the replacement in phase wise manner . In the first phase have one compute tier, one metadata server and the secondary grid control server. As for the Web tier there are only two , you have to make the judgement..
However this approach has the issue of integrating the newly built servers with existing one's. I would like to avoid this. scenario.

5.My choice would be a new rational system  with just the right number of servers .However  I do not know your requirements and constraints. Make you own judgement.

[Manoj07k]

Thanks @Sajid01 for expressing your thoughts clearly .

I would also love to do a fresh deployment. But we're just thinking of different ideas of how to reduce the timelines and cost of deployment.

As you said, the best approach is to keep meta and mid tier dedicated but that's the project requirements 🙂