Amazon EKS 1.30 no longer assigns a default storage class

[#1]

The Viya platform is supported to run on the major cloud providers. However, that doesn't mean that everything is done the same in each cloud environment. Indeed, the cloud providers can be pretty opinionated about their implementation of resources, including the Kubernetes environment. In particular, Amazon, with the release of version 1.30 of its Elastic Kubernetes Service (or EKS), has recently stopped specifying a default storage class. This has notable implications when deploying the Viya platform to EKS.

Why does Kubernetes have a default storage class?

Kubernetes can be configured to identify a default storage class if you want to. It's there so that when a pod is started, if the containers request storage but don't indicate where to get it from, then Kubernetes will helpfully hook them up with the default storage class.

And for EKS?

Up to version 1.29, Amazon EKS would automatically identify "gp2" as its default storage class. The gp2 storage relied on Amazon's Elastic Block Storage (EBS) as low-cost and reasonably efficient for general purpose storage backed by SSD drives. While the gp2 storage class is still defined and usable in EKS, from 1.30 EKS no longer includes the "default" annotation on the gp2 storage class.

What happens if Kubernetes doesn't identify a default storage class?

Ideally, there's no problem. Pod specifications can define their desired storage, and assuming the resources exist, then they work just fine.

However, in cases where the Pod specification lacks storage definitions for the containers that need storage, then without a default storage class, Kubernetes is unable to help. Those pods then will get hung up and you'll see in their logs and status that they're unable to bind to storage.

And for EKS?

Kubernetes has been evolving quite a lot. One area in particular is in the area of in-tree resources providers moving out of the way in preference for out-of-tree resource providers - that is, moving functionality from inside the Kubernetes project to external, independent projects instead. A while back, Amazon implemented a change in EKS to disable its in-tree "aws-ebs" driver for EBS storage and now requires using the out-of-tree "aws-ebs-csi" driver.

Dropping the default storage class from EKS 1.30 continues this evolutionary process.

How does this affect the Viya platform?

Out of the box, there are two components of the Viya platform that do not specify a storage provider: Redis and OpenSearch.

If you attempt to deploy Viya to a Kubernetes cluster that doesn't specify a default storage class (like EKS 1.30), then those pods are unable to start.

Redis in particular is a key infrastructure service of the Viya platform. There are many other Viya pods that rely on it. If it doesn't achieve a running state, then those pods are unable start up properly either.

Do we have any recommendations?

We certainly do!

First and foremost, the best recommendation is to familiarize yourself with Kubernetes concepts around storage in general and also dive into your chosen cloud provider's implementation. Don't forget to look into the different storage types that Viya services require as well.

Getting a bit more prescriptive, we have three levels of recommendation…

Good

Install the aws-ebs-csi driver to EKS and identify gp2 as the default storage class.

Since version 5.2.0, the IAC project (viya4-iac-aws) helps with that significantly, automatically installing the aws-ebs-csi driver as part of its normal process.

And since version 8.5.1, the IAC project will automatically identify gp2 as the default storage class in an EKS 1.30 cluster for you.

Be wary - the 8.5.0 version of the IAC project supports EKS 1.30, but does not identify any default storage class.

Better

Building on the concepts recommended above, let's go one better: define the gp3 storage and identify it as the default.

The gp3 storage is the next generation of EBS-backed storage for EKS in AWS. Not only is it cheaper than gp2, it's more performant and resilient. Amazon recommends using gp3 anywhere gp2 might be employed.

We still need the aws-ebs-csi driver installed, of course. But we also need to manually declare the gp3 storage class using a simple manifest:

cat << EOF > defineGp3.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
  fsType: ext4
  iopsPerGB: "50" # Optional, sets the baseline performance
  iops: "3000"    # Optional, custom IOPS value (default: 3000)
  throughput: "125" # Optional, custom throughput in MiB/s (default: 125)
EOF

Then we can apply that to the EKS cluster:

kubectl apply -f defineGp3.yaml

Now we just need to identify it to Kubernetes as the default storage class:

kubectl patch storageclass gp3 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

And then validate:

kubectl get sc

With results similar to:

NAME            PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
gp2             kubernetes.io/aws-ebs  Delete          WaitForFirstConsumer   false               
gp3 (default)   ebs.csi.aws.com        Delete          WaitForFirstConsumer   false               

In the future, a later version of the aws-ebs-csi driver might take on the task of defining the gp3 storage class and setting it as the EKS default.

Be wary - Kubernetes will allow you to annotate multiple storage classes as "default". This, of course, is A Very Bad Idea as Kubernetes might then arbitrarily assign any of the "default" storage classes which can lead to unexpected results. Use the kubectl patch command shown above with the is-default-class value set to false to remove the "default" annotation from storage classes until only the desired one remains marked.

Best

Really, we should not rely on blind defaults, especially with a resource as critical as storage. While EBS-based storage is great for many use-cases, there are others where it's exactly the wrong choice. Ultimately, it's up to the Kubernetes administrator to determine if there's a default storage class and, if so, what storage resources are backing it.

So the best thing to do is to configure all services of the Viya platform to identify the storage resources they need (local disk, block storage, NFS shared volumes, high-performance clustered filesystem, and so on).

To help get you there, the Viya order assets include example manifest files (see the SASViyaReadMe file) for customizing the storage for select services. Let's address the two that currently rely on the default storage class:

• For Redis, look to the $deploy/sas-bases/examples/redis/operator/redis-modify-storage.yaml file.
   
• For OpenSearch, look to the $deploy/sas-bases/examples/configure-elasticsearch/internal/storage/storage-class-transformer.yaml file.

Copy those example files to your deployment's site-config directory and modify them to specify the exact storage resources you want those services to use. Changing the storage type after initial deployment is often not supported because there will be persistent volumes with expected data that are no longer referenced. So, this is a change to make when preparing to deploy the Viya platform for the first time.

Next steps

The course SAS Viya 4: Deployment on Amazon Elastic Kubernetes Service provides more information about this topic and many others relevant to deploying the Viya platform to AWS.

Find more articles from SAS Global Enablement and Learning here.