An automated way to test if a SAS data set is encrypted is to use SAS functions to open the data set, try to read a record, and then check for an error message via the SYSMSG function.
Thanks to Thomas Billings for this tip. Read more details in this paper from WUSS 2017: Methods to Identify and Validate SASProprietary and AES Encrypted SAS Data Sets.