SAS Viya with SingleStore: Data Flow Concept

[#1]

I’ve written about SAS Viya with SingleStore and showed how we can leverage data updates from SingleStore in near real-time in SAS Visual Analytics. In the coming posts, we will dive in a little bit deeper and understand some of the concepts behind the scenes. First let’s see how data flows from SingleStore to CAS.

1 - Data is NOT loaded in CAS initially

In order to process data in SAS Viya, we usually have to load it into CAS first. This is true for most of SAS visual applications (like SAS Visual Analytics) which work on global tables. This is also true with code (if we put the “transient scope” aside for more simplicity). This means we create a copy of the data into CAS.

With SAS Viya with SingleStore, the SingleStore table is actually not loaded in CAS at load time (when we load it through the UI or with a loadTable CAS action). A few metadata are grabbed from SingleStore (table structure, columns, number of rows, etc.) and the table is then “declared” in CAS. No copy of data is done in CAS and no data movement occurs.

This is depicted in the following figure:

The above mechanism relies on the default value of the backingStore option which is INPLACEPREFERRED. If you set this option to CASDISKCACHE, you will replicate the behavior of traditional CASLIBs (duplicate the data in CAS at load).  

2 - A CAS action triggers some pre-processing in SingleStore

In “traditional” CAS, CAS actions run on CAS data (the copy of the original table loaded in CAS) in the CAS server.

With SAS Viya with SingleStore, the CAS action processing will be distributed between SingleStore and CAS. The goal is to reduce data movement between SingleStore and CAS by preparing a subset of the data in SingleStore. As of today, SingleStore handles:

• Row filtering
• Variable selection
• New calculated columns
  
  

Technically, this is handled by the SAS Embedded Process deployed in the SingleStore cluster.  

3 - Data is streamed to CAS on the fly

Then, with SAS Viya with SingleStore, the result set of the previous step is streamed dynamically from SingleStore to CAS for final computations.

4 - The final computations happen in CAS

Finally, the CAS action processing is much more than just a selection of fields and records and computation of additional columns. It is about simple or complex analytical processing that is performed in CAS on the data that has just been streamed in. The final results are sent to the calling client.

The entire flow description is also available in this video:

Thanks for reading.