In Part 1 and 2, we introduced the concept of elasticity, explored the setup for CAS table rebalancing, and demonstrated how it operates. Now, let’s discuss some important considerations regarding table rebalancing.
What About COPIES?
You might wonder if the COPIES setting is still relevant. It absolutely is—COPIES allows CAS tables to withstand node failures. For example:
- COPIES=1 enables a table to survive one CAS node failure.
- COPIES=2 allows it to survive two CAS node failures.
- And so on.
However, the role of COPIES has evolved slightly with the introduction of automatic table rebalancing.
When automatic CAS table rebalancing is configured for scaling up (using tableRedistUpPolicy), COPIES now represents the number of simultaneous CAS node failures that a table can survive.
For instance, a CAS table with COPIES=1 and tableRedistUpPolicy=REBALANCE will endure a single CAS node failure. Once the failed worker is replaced, the table will rebalance, restoring its COPIES setting—allowing it to withstand another possible failure in the future.
When workers are intentionally added, the COPIES setting for each table is preserved, provided that the rebalancing policy is enabled for scaling up. Similarly, when workers are intentionally removed, the COPIES setting for each table is maintained.
How Long Does Automatic Rebalancing Take?
The time required for adding or removing CAS worker nodes increases with the number of CAS tables set up for rebalancing.
This can significantly affect current users.
Indeed, by default (suspend), “all server activity is paused when data in global tables is being moved to other worker pods. New actions are not started, and new connections are not accepted when data is being moved.”
However, this behavior can be mitigated through various options.
For scaling down operations, you can enable the background scale-down mode with the following setting:
cas.SCALEDOWNMODE = 'SUSPEND' | 'BACKGROUND'
When background mode is active, "each global table is locked, and a copy of the table is created that has data on the worker pods that are not scaled down. When the copy is completed, the original table is deleted and replaced with the copy. During this operation an action can read the locked table, and actions that attempt to alter the locked table have to wait for the copy to complete."
For scaling up operations, set CAS_GLOBAL_TABLE_AUTO_BALANCE to "background" instead of "1" or "true" to achieve similar behavior when adding workers. If "1" or "true" is used, the system will default to suspend mode.
Switching Between MPP and SMP Configurations
Rebalancing of CAS tables is not applicable when converting an SMP CAS server to an MPP CAS server, or vice versa. Both transitions necessitate a CAS restart, which inherently eliminates the need for table rebalancing.
Thanks for reading!
Find more articles from SAS Global Enablement and Learning here.
