A look at genAudit and the data behind the User Activity report

[#1]

In previous GEL posts, we saw how the SAS Operations Infrastructure implements some of the key aspects of Viya's auditing framework. The Operations Infrastructure facilitates the generation of events, which are captured as audit records in the SAS Infrastructure Data Server, and end up in a CAS table that serves as the data source for the User Activity audit report. In this post, we'll take a closer look at genAudit; the task that facilitates the extraction of records from PostgreSQL to load them to CAS.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Viya services publish messages about activity to an exchange in RabbitMQ. Messages about events relating to user activity are published to the Application Exchange (sas.application). The exchange routes messages to queues ready for consumption by subscribers. The audit service is the consumer for the sas.application.audit queue, and it reads messages about user activity auditing and creates new records for those events in PostgreSQL. Specifically, audit records are captured in the resource_audit_entry_data and security_audit_entry_data tables in the audit schema in the SharedServices database.

We've seen in previous posts how records can be surfaced from PostgreSQL using the Command Line Interface. They can also be extracted using REST API calls, which is exactly what genAudit does (we'll also look at some other examples of extracting records using REST API calls in a later post). genAudit is an Operations Infrastructure task responsible for extracting audit records for reports, data plans, CAS management, and CAS access management by calling the audit service. It then dumps the records a to a CSV file (/opt/sas/viya/config/var/cache/auditcli/audit.csv).

genAudit runs every 2 hours by default. It retrieves audit records created since the successful run and adds them to the CSV file. It also purges records older than 7 days from the CSV, and then loads the the CSV data to the AUDIT table in SystemData caslib for use by the User Activity audit report. The genAudit task is one of a set of tasks run by the operations agent (sas-ops-agent) service; specifically, the operations agent server role (ops-agentsrv) is responsible for running genAudit and maintaining its schedule. We can view the list of ops-agentsrv tasks by running:

/opt/sas/viya/home/bin/sas-ops tasks --name "ops-agentsrv"

Task Name                       Description                                                         Frequency
---------                       -----------                                                         ---------
ARMEtl                          Application Response Measurement (ARM) ETL task                     01:18
DatamartEtl                     Datamart incremental etl driver                                     5m0s
DatamartRollOff                 Datamart daily rolloff task                                         02:18
DatamartzipTSV                  Datamart daily ZIP TSV task                                         03:43
EmiSweeper                      Retry publishing any payloads that failed to publish earlier        1h0m0s
FlushReqTask                    Flush agenda (task-list) to permanent storage on request            0s
LogServerReport                 Internal sas-ops-agent log server report                            96h0m0s
LogTaskReport                   Internal sas-ops-agent log task report                              24h0m0s
OpsAgentActivity                Internal sas-ops-agent activity monitor                             2m0s
OpsAgentTaskStatistics          Internal sas-ops-agent task statistics activity monitor             4m0s
genAudit                        Extract audit records. Generate a CSV files for given applications  1h0m0s
registerOpsAgentSvrServiceTask  Register Ops-AgentSvr service task                                  1h0m0s
updateInventory                 Update inventory                                                    01:23

Note that frequency for each task is conveniently displayed in the output (on the far right).

We can also export the task list to a JSON file, where we can view or update the definitions for the tasks. To export, run the following on machine hosting the sas-ops-agentsrv service:

/opt/sas/viya/home/bin/sas-ops-agent export -name "ops-agentsrv" -tasks tasks.json -spoolpath /opt/sas/viya/config/var/lib/evmsvrops/spoolsrv

2019-12-03 17:30:46.318  INFO [xmmain.go:426] [sas-ops-agent] - Server ops-agentsrv log started for service sas-ops-agent version 1.5.68-pioneer date 2019-11-05 14:05:14Z on host intviya02.race.sas.com
2019-12-03 17:30:46.318  INFO [xmmain.go:430] [sas-ops-agent] - Log started for service ops-agentsrv type cluster on host system intviya02.race.sas.com
2019-12-03 17:30:46.318  INFO [xmmain.go:432] [sas-ops-agent] - Executing: ./sas-ops-agent export -name ops-agentsrv -tasks /tmp/tasks.json -spoolpath /opt/sas/viya/config/var/lib/evmsvrops/spoolsrv -force
2019-12-03 17:30:46.318  INFO [xmmain.go:477] [sas-ops-agent] - Using spool path /opt/sas/viya/config/var/lib/evmsvrops/spoolsrv
2019-12-03 17:30:46.319  INFO [xmmain.go:511] [sas-ops-agent] - Active service ID: 21550C8B-6C3D-42CD-903A-D920AD0BF283 service name: ops-agentsrv
2019-12-03 17:30:46.319  INFO [xmmain.go:544] [sas-ops-agent] - Waiting for Consul and Rabbit access
2019-12-03 17:30:46.535  INFO [xmmain.go:559] [sas-ops-agent] - Using Consul base key config
2019-12-03 17:30:55.079  INFO [xmconsul.go:246] [sas-ops-agent] - Consul agenda (task list) for host intviya02.race.sas.com which has service type cluster using role ETL import count 3
2019-12-03 17:30:55.178  INFO [xmconsul.go:345] [sas-ops-agent] - Consul agenda (task list) read 10025 bytes from host intviya02.race.sas.com which has service type cluster using role ETL import count 3
2019-12-03 17:30:55.207  INFO [xmmain.go:875] [sas-ops-agent] - Server program ops-agentsrv ended, exit code 0

Due to a change in the code in SAS Viya 3.5, the spool directory must be specified in the command. The spool directory is the location used for reading and writing persistent task state information. The default spool path for the ops-agentsrv is /opt/sas/viya/config/var/lib/evmsvrops/spoolsrv. The -spoolpath option is not required in SAS Viya 3.4.

If you're trying this for yourself, you'll notice something strange about the tasks.json file after you export. Definitions for most of the ops-agentsrv tasks are there, but there is at least one notable absence; the genAudit task definition. This is due to a defect in the initial 19w47 ship of SAS Viya 3.5. An official hot fix will be available soon; in the meantime, the fix (which consists of a new sas-ops-agent binary file) can be obtained from SAS Technical Support.

The 'fixed' output, will contain the genAudit task definition. It looks something like this:

{
    "version": 1,
    "taskName": "genAudit",
    "description": "Extract audit records. Generate a CSV files for given applications",
    "hostType": "any",
    "runType": "periodic",
    "frequency": "2h0m0s",
    "visibility": "all",
    "controlAccess": "read update",
    "roles": "ALL ETL",
    "maxRunTime": "2h30m0s",
    "timeOutAction": "restart",
    "errorAction": "ignore",
    "command": "ev-genaudit",
    "commandArgs": "-a reports,dataPlans,casManagement,casAccessManagement -l 1000 -d 7",
    "commandType": "ext",
    "publisherType": "none",
    "TaskStatus": "Scheduled"
}

By looking at the command and commandArgs options, we can conclude that the genAudit task effectively runs the following command:

/opt/sas/viya/home/bin/ev-genaudit -a reports,dataPlans,casManagement,casAccessManagement -l 1000 -d 7

where: -a refers to the audit resources to include in the AUDIT table in CAS; -l refers to the batch size of records to process, and; -d refers number of days of data to include.

Be careful not to confuse these with the the properties in the audit service's sas.audit.archive configuration instance, which determine how long records are kept in PostgreSQL and what happens with them afterwards. The properties in the genAudit task definition determine how many records are extracted to the CSV file for uploading to CAS.

The properties in the task definition can be updated and then imported to the ops-agentsrv to modify the task. For example, if we wanted genAudit to run more frequently, we could change frequency to 1h in the tasks.json file, save it, and then run the following:

/opt/sas/viya/home/bin/sas-ops-agent import -name "ops-agentsrv" -tasks tasks.json -spoolpath /opt/sas/viya/config/var/lib/evmsvrops/spoolsrv

The genAudit task runs using the credentials of the SAS install user (sas), and therefore it only extracts those records to which the install user has access.

The data from the CSV file are then added to the AUDIT table, which is the last part of the process. The User Activity report always will contain the data current as of the last genAudit run.

There are situations in which genAudit may not run as expected. For example, be mindful of the issue outlined in SAS Note 64678 (fixed in Viya 3.5), where the token file created by the sas-admin CLI for the SAS install user contains an expired token, creating an issue that prevents genAudit from loading audit data to CAS. If genAudit is not running as expected, try adding the -v option to the ev-genaudit command in the task definition for verbose logging to try to identify what might be happening.

One of the best aspects of Viya's auditing capabilities is its flexibility. Not only is it possible to control what is captured, but also how the data is extracted, processed, archived, and reported on. You can make a copy of and then customise the existing audit reports , or you can create your own, based on the audit data (either the data captured by genAudit or an alternative custom data source).

For more information, refer to the official documentation. 

Thank you for reading. I hope the information provided in this post has been helpful. Please leave a comment below to ask questions or share your own experiences