1.  Aim and Use Case The aim of our benchmarking is to determine the best storage option for a BigData Analytical repository accessed by CAS. I wanted to determine which solution would be optimal from the cost/performance point of view. I imagine that the primary way for using the storage is to load data into CAS memory and process it there. After the necessary calculations (exploration, analysis, modeling etc.) the dataset will be saved for future use. No intermittent or intensive back-and-forth transfer will be used.   2.  Setup In total, 7 options and variations will be tested: S3 and NFS on RAID partition (each with 3 file formats: .sashdat, .sashdat with compression and .parquet) as well as SingleStore on EBS and Bottomless. Two test datasets will be used: one simulation an analytical table (wide, common repetitions) and one raw input (larger, more unique observations ).   2.1.               Engines   2.1.1.                       AWS S3   This option relies on Object Storage offered within Amazon Web Services: Simple Storage Service (S3). Its benefits include unlimited storage that doesn’t have to be preallocated and is dynamically resized. The cost is also attractive 0.023 down to 0.004 and data can be easily moved between the tiers to decrease the bill even further. It has to be noted though that there is a charge associated with accessing the data and it’s larger the colder the storage tier. Another benefit is the lack of necessity for any additional software as CAS can access S3 directly.   2.1.2.                       NFS on RAID disks   Ana alternative is to use an in-custer NFS server that relies on throughput-optimized EBS volumes that are connected in a RAID partition. It may result in much higher performance and a slight increase in durability. What is more, if the data is used sparingly reads and writes may take advantage of bursting capabilities of ST1 volumes to achieve even better results limited only by an instances maximum bandwidth to EBS.   The downsides involve at least 1.5 times higher cost per GB compared to baseline hot storage and, more importantly, much more maintenance as well as reliance on components whose support policy may be different. In our test scenario we will be using 4 ST1 EBS volumes 5TB each.   2.1.3.                       SAS Viya with SingleStore (S2)   As a third option let us consider using SingleStore. It uses its very own compression and data access mechanisms but is tightly integrated with SAS CAS for seamless use. It doesn’t rely on sashdat nor parquet so it will be particularly informative to test out this scenario.   It has to be kept in mind that S2 will need its own dedicated hardware as well as licenses. To use unlimited, bottomless storage SingleStore Premium is required which is roughly double the cost of a standard edition. Maintaining SingleStore will be an additional administrative task but not as time consuming as a RAID+NFS scenario. In the test scenario we went contrary to SAS recommendation of using 1/3 of CAS vCPUs and decided to check the performance on full parity (1:1). Consequently we have 1 master node (4 vCPU / 32 GB RAM) and 4 leafs (8/64) with 5TB ST1 EBS each. 8 partition for the database. This will be labeled as S2PVC. Using S3 via Singlestore will be labeled S2BLESS.   2.2.                   Test datasets   2.2.1.                       Analytical Based Table (ABT). It is used to simulate an input table for Machine Learning algorithms as well as exploration tables used typically in reporting. There are a lot of repetitive values in the columns which may help with compression. The code for generating the dataset is attached below:   %macro bigabtset(outname, rows);     %parallelize(&rows., 32);     data casuser.abt&outname.;         set casuser.rozrzucenie;             do i = 1 to _step;             segment = byte(rand("integer", 65, 90)); /* A - Z */             %do i = 1 %to 100;                 var10_&i = %myrand(10);             %end;             %do i = 1 %to 100;                 var100_&i = %myrand(100);             %end;             %do i = 1 %to 100;                 var1k_&i = %myrand(1000);             %end;             %do i = 1 %to 100;                 var10k_&i = %myrand(10000);             %end;             output;         end;         drop i _step;     run; %partition(abt&outname.); %mend;   2.2.2.                       Fact table It is used to simulate a dataset that is an input fact table for further processing. It is loosely based on a test scenario. It has multiple double columns that won’t benefit much from compression. The code for generating the dataset is attached below:       data &lib..meters&outname.;         set casuser.rozrzucenie;         length numer_licznika $18 dzien 8 kierunek $1;         array h{24} 8 h1-h24; /* Array to hold hourly data */             /* Generate data for each meter */         do meter = 1 to _step;             numer_licznika = "000" + put(rand('unifrom')*1e15, z15.);                 /* Generate data for each day */             do i = 0 to &num_days - 1;                 dzien = &start_date + i;                     /* Generate data for each direction */                 %let direction_count = %sysfunc(countw(&directions));                 %do d = 1 %to &direction_count;                     kierunek = "%scan(&directions, &d)";                         /* Generate hourly consumption */                     do hour = 1 to 24;                         h{hour} = rand("uniform") * 10;                     end;                         /* Output the row */                     output;                 %end;             end;         end;         drop i meter hour _step;     run;   3.    Results   3.1.                   Scaling with the number of observations Below are the charts that show the size of the datasets after compression for both of the datasets. The size of files did not differ depending on the underlying storage engine (RAID/S3).   ** NOTE ** The size for S2 is the one reported by the database engine itself. The actual size of the S3 bucket in particular is much greater – up to 500%. We made an assumption that these can be significantly reduces via system settings but not attempted to do so after a quick search.     The charts below show how the time of read and write operations depend on the number of observations.       It appears that it’s safe to conclude that these number scale linearly. It’s an important observation for the future as we can perform benchmarking quicker and using less resources.   3.2.                   Compromise between cost and performance   The chart in this subchapter shows each of the proposed technology on two axis: the vertical one is cost per gigabyte and the horizontal one is the sum of time needed to save and load a dataset. The color of the bubble represents two different datasets. Important assumptions: The cost for S3 assumes 1TB of daily save and load The cost of S2 excludes necessary hardware and licenses! The performance of S2 is measured on a system with CAS vCPU = S2 vCPU. The S2 performance assumes no logs are stored.     We can immediately see that ABT can be efficiently compressed using parquet and SingleStore. These two options seem to be dominant among the alternatives. For the Meters dataset the situation is much more nuanced. We can see much worse performance of the NFSRAID engine with S2BLESS decrasing performance the least.   3.3.                   Operation types: saving and loading   The chart below shows the difference between the save and load operations. Please note the difference in scale for both axes.     A clear distinction between both of the measured operations is visible. It seems that depending on the particular use cases. For example, if a file is created once a month and loaded every day into memory the saving time is not as important. Conversely, if big datasets are created in case they are needed but oftentimes are deleted before being loaded than it's the load time that may be disregarded. An important observation is that compression of sashdat files is an operation that greatly increases the time to save but significantly reduces the load time. On the other hand parquet favour saving time to loading but the difference is not so big.   3.4.                   Fairness check   To make sure all of the tested methods were actually putting the dataset for quick use in the CAS memory we performed an additional check. After loading we ran a quick MDSUMMARY action on all of the applicable variables to make sure that there are no outliers. The results are show below.       As you can see there are three group of observations: NFSRAID.sashdat which was a significant outlier but only when using the ABT files. The outlier status persisted regardless of the file size. I tried correcting it but to no avail. It’s particularly perplexing that this effect is absent in the METERS dataset and they both share the same code. The second group is the compressed sashdat which is a lot slower than the rest of the file types excluding the previously mentioned outlier. This may have to do with the fact that the data can indeed fit without problems into the nodes memory. The result might have been different if the contrary had been true.   4.    Conclusions   There are a number of conclusions that we may draw from the experiment: Both the data size and the time of operations scale linearly in the tested scope. The parquet filetype is not very suitable for preserving factual, unique observations. Compression of sashdat files significantly reduces the time to load a dataset at the expense of saving time and calculations. The number of saves vs. loads should be considered when choosing the storage type as these may vary greatly. SingleStore seems to be a viable option for a mix of ABT and FACT tables and large data volumes and probably when it is already present in the system for other reasons – otherwise a careful calculation of at least the license and additional hardware cost should be performed.   5.    Further testing   A number of topics also seem to be interesting as a hypotheses for future testing: When does buying cheaper storage but larger CAS hardware make sense? Testing SingleStore with recommended and not enlarged hardware vs. CAS. Can compression on the filesystem level improve performance for NFSRAID? How does different RAID types affect the performance and cost? When does burst EBS performance make a difference? Does CAS node size make difference? Should we prioritize the number of nodes or the size? Is linear scaling preserved for files beyond the memory limitations of CAS nodes? ... View more

Aim of the post Thanks to its great connectivity features and licensing providing users with lots of different Data Connector out of the box SAS CAS gives users a plethora of different opportunities to use for storage, especially in the cloud. In this post I’d like to share some experiences in selecting a storage option for a SAS solution. Firstly, I’m going to cover the use case, outline which options were considered and then go through the most important criteria.   Use case My use case was an environment used for supply chain optimization for a retail chain. From the technical point of view SAS was running a complex workload nearly round the clock in three main areas. There is a lot of data preparation that was done in SAS to relieve the strain on the Data Warehouse and minimize the impact of analytical calculations. Apart from that we are running both prediction and optimization model scoring as well as training. Server is on a very tight schedule – once the data is ready it immediately starts to be loaded and transformed before a relatively quick forecast followed by a long and arduous optimization to meet all of the client’s criteria. The hours left before the next batch are used to retrain or fine-tune the models. The whole infrastructure is to be put on Amazon Web Services (AWS) infrastructure. Consequently, SAS Viya will run on AWS EKS. Calculations were done in CAS on multiple workers and orchestrated using 4GL in SAS Compute.   Criteria We need a storage solution that is first and foremost performant while also being cost efficient. Other criteria that can’t be overlooked are durability and security.   Option I – EFS The first and easiest option is to use AWS Elastic File System. Its advantages lie mainly in the simple management: it practically takes care of itself and has many ways to connect it to your EKS cluster.   Setup To setup EFS you need to perform three steps:   Provision cloud resources       resource "aws_efs_file_system" "viya-rwx" { creation_token = "sas-viya-rwx" } resource "aws_efs_mount_target" "rwx-to-workers" { file_system_id = aws_efs_file_system.viya-rwx.id subnet_id = aws_subnet.workers.id }         Add CSI Driver       eksctl create addon --name aws-efs-csi-driver --cluster $cluster --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy --force         While sufficient for a testing scenario you might want to write your own policy for production-grade environments by restricting the “Resource” section of the JSON.   Create a Storageclass       kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com parameters: provisioningMode: efs-ap fileSystemId: {{ efs_filesystem_id }} directoryPerms: "700" gidRangeStart: "1000" # optional gidRangeEnd: "2000" # optional basePath: "/dynamic_provisioning" # optional         Durability Data durability is high and easy to determine by simply checking AWS documentation: Amazon Elastic File System (EFS) | Cloud File Storage | FAQs. Cloud provider guarantees 11 nines over a given year.   Security From the security point of view EFS also delivers. It provides encryption in-transit which is turned on when filesystem is mounted through the CSI driver with default configuration. It also provides encryption at rest with both AWS and customer managed keys (in that scenario keep in mind additional permissions you have to include for the service account). Access to the disk is granted using IAM permissions.   Cost efficiency Every TB costs roughly $300 per month per TB. The cost can be greatly reduced using Infrequent Access, which is less than 1/10th of the cost of Standard. What’s more is that we’d have to pay around $600 for 100MB/s of provisioned throughput or use Bursting Throughput (see Performance chapter).   Management This is a managed service, so no additional management is required.   Performance EFS is capable of providing 3 GiB/s throughput for a cluster of 6 CAS Workers. This is more than enough for our purposes. What also looks promising is using Bursting Throughput mode which scales with the amount of storage and is capable of providing extra performance when under used. This sounds very promising as some of our calculations take place in memory and for those hours bursting credits might accommodate. BT delivers 50 MiBps per TiB of storage baseline with twice as much in bursts. With our 100TB volume we might get truly impressive performance.     Option II – S3   Setup In this point I’d like to walk you through creating an S3 bucket with no public connectivity and used mainly by batch processes, so no user auditing in S3 logs. As in previous example for production environment it is advisable to restrict persmissions for the sas-cas-server service account to specific buckets. To setup it in the simplest way you need to perform these steps: Provision S3 bucket with access control       resource "aws_s3_bucket" "viya_object_storage" {   bucket = "viya-object-storage"   force_destroy = true } resource "aws_s3_bucket_ownership_controls" "viya-ownctr" {   bucket = aws_s3_bucket.viya_object_storage.id   rule {     object_ownership = "ObjectWriter"   } } resource "aws_s3_bucket_acl" "viya-s3acl" {   depends_on = [aws_s3_bucket_ownership_controls.viya-ownctr]   bucket = aws_s3_bucket.viya_object_storage.id   acl    = "private" }       Create private connectivity       resource "aws_vpc_endpoint" "s3endp" {   vpc_id       = aws_vpc.main.id   service_name = "com.amazonaws.${local.region}.s3" } resource "aws_vpc_endpoint_route_table_association" "to_s3_vpc_endpoint" {   route_table_id  = aws_route_table.workers_subn_rtbl.id   vpc_endpoint_id = aws_vpc_endpoint.s3endp.id }       Provide credentials to CAS server       eksctl create iamserviceaccount --cluster=sas-viya --namespace viya --name=sas-cas-server --role-only --role-name=s3fa-for-cas --attach-policy-arn=arn:aws:iam::aws:policy/AmazonS3FullAccess kubectl annotate sa sas-cas-server eks.amazonaws.com/role-arn=arn:aws:iam::<ACCOUNT_NO>:role/s3fa-for-cas       Restart CAS and create a CASLIB caslib S3D datasource=(srctype="s3",     region='eu-north-1',      bucket='viya-object-storage',     objectpath='/'     ) subdirs ;   Durability AWS guarantees durability to be 11 nines over a given year, so the same value as EFS.   Security S3 provides both encryption in transit and at-rest using AWS and customer managed keys. Same remarks as for EFS apply. Access to the disk is granted using IAM permissions.   Cost efficiency AWS charges $0.023 per month for every GiB stored in its Standard Tier dropping to $0.01 for Infrequent Access. This is significantly cheaper than EFS. It has to be notes though that with S3 you also pay per request and in some tiers by GB retrieved.   Management This is a managed service, so no additional management is required.     Option III – NFS   Setup NFS setup will not be provided here as it is provided in kernel space in Viya4-IAC scripts. It has to be noted, however that in that configuration it has to be put on an external EC2 instance or at least on am AMI that has all required kernel modules loaded. It not only bumps the management effort but also cost, as instance size determines storage throughput and consequently we might be forced to pay for computing power we can't really use. To circumvent that I've created a userspace in-cluster NFS solution that can still use fast RAID0 disk array and serve NFS from within the cluster. The implementation of this solution is beyond the scope of this post but don't hesitate to contact me for details.   Durability In this example durability has to be provided by the filesystem driver and it might be a costly one. Since we are aiming for maximum performance we used RAID0 for parallel write and reads which means a discrepancy between disks might lead to data corruption. Disk snapshots also have to take place when no pod is mounting the disks for sake of simultaneity.   Security While we can use data  encryption at rest and rely on customer managed keys it is much more complex with data in transit. As such, we would need another layer to encypt NFS data that is being transmitted between the pods. For many scenarios that might not be a problem but it has to be taken into consideration while evaluating compliance.   Cost efficiency In EBS you pay about $0.10 for every gigabyte when it’s attached to an instance. Consequently, to provision 100 TB of storage not only would we need 64 disks 16 TB each but for a 24/7 installation it would cost us north of $100k a month! And we haven’t even considered the cost of any additional instances for an external NFS...   Conclusion As of now we definitely ruled out using NFS for our storage because while it in theory might provide higher throughput but it would in fact be more than our instances need while also compromising by more management effort and cost. With the current state of affairs it seems reasonable to put more frequently accessed data on the EFS solution while archiving in EFS/S3. ... View more