Referencing Files Directly on CAS Actions with SinglePass=True

[#1]

We've posted a few times about referencing files directly from CAS actions in CASL code (1,2,3). It's a great way to utilize CAS action functionality in programs without explicitly loading tables to CAS. The referenced file is transiently loaded to CAS and dropped immediately after the requested CAS action is run. For context, when referencing files directly, the CASL code looks like this:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

As you can see, the simple.Summary action references the SASHDAT file directly instead of an explicitly loaded CAS table. SAS7BDAT, CSV, XLSX, and parquet files can also be referenced in this way. Referencing files directly is great for traditional programming where we don't need tables loaded for extended ad hoc analysis and/or reporting. However, there is a more efficient way for most scenarios, setting singlepass=true. With singlepass=true, the files aren't loaded in whole. Their data are read by block as the CAS action requires and are released immediately. This generally makes for faster processing.

For our 500,000 row, 24 column SASHDAT table, setting singlepass=true reduces our total execution time from 2.56 to 1.55 seconds. Examining the log, we see that while setting singlepass=true increased CPU on the summary action, real time for both the load and the summary decreased.

The same pattern exists for the other file types, SAS7BDAT, CSV, XLSX, and parquet. CAS CPU on the summary action increases but overall performance for the action improves.

As the name implies, "singlepass=true" can only be used on CAS actions that perform their processing with a single pass of the input data. Actions that require multiple passes will error as shown below.

To see if a CAS action is eligible for the singlepass=true option setting, check the doc. For example, the option is available for the append action but is not available for the loadTable action. 

Find more articles from SAS Global Enablement and Learning here.