SAS Viya log-cleanup playbooks

2 Likes

The Operations Infrastructure in SAS Viya has a daily task called LogFileArchive which uses the sas-archive tool to tar and zip log files older than 30 days, and stores the resulting archives in /opt/sas/viya/config/var/log. By default, the ops-agent on each SAS Viya host runs LogFileArchive daily at 0400. Because logs compress well, the resulting log archives are small, so this task frees up disk space well.

But nothing is provided out of the box in SAS Viya to clear down these log archives. SAS Administrators will want to schedule some sort of script to either delete them, or move them elsewhere for long term storage, regularly. Otherwise those log archives will grow over time, and may eventually fill up the disk where /opt/sas/viya/config/var/log lives on one or more hosts.

That would be bad, so in this post, I present some log-cleanup playbooks: a set of ansible playbooks to clean up those log archives. I'll finish by also mentioning other useful log management utilities you may also want to know about, and where you can learn more about them.

Works with Viya-ARK

The SAS Viya Administration Resource Kit (Viya-ARK) is an indispensable source of playbooks and supporting files which help customers prepare for a SAS Viya deployment, and manage SAS Viya afterwards. They are publically available on GitHub, at the link above. At the time of writing, current Viya-ARK tools supporting the latest release of SAS Viya include:

Ansible is the perfect tool for tidying up SAS Viya log archives. It it is easy to write Ansible tasks which act on files on all of your Viya hosts simultaneously, from a single command. The log-cleanup playbooks below follow the style of other Viya-ARK playbooks, so I hope you will find them easy to use.

I have begun working with the team who look after the Viya-ARK project to potentially include these playbooks in it, at a future release, subject to the necessary testing and approval being given. The log-cleanup playbooks presented in this post should be treated as an 'alpha' or pre-release version, tested on CentOS 7.5 and Ansible 2.5.5 to the standard of 'it worked on my machine'. They are not yet tested on other platforms/versions, and may require changes to make them compatible with the other platforms and Ansible versions we support. I will update this post when/if the playbooks are updated or merged into Viya-ARK.

In this post:

Boxes with a green border show bash commands you can type at a command prompt

Boxes with an orange border contain file content you can copy paste and save as a file

Create log-cleanup playbook files

Create the six log-cleanup playbook files, placing them under ~/sas_viya_playbook/viya-ark as follows:

cd ~/sas_viya_playbook

mkdir -p ~/sas_viya_playbook/viya-ark/playbooks/log-cleanup/

File viya-ark/playbooks/log-cleanup/README.md:

# SAS Viya Administration Resource Kit (Viya-ARK) - Log Cleanup Playbooks

## Introduction
The Log Cleanup repository contains a set of playbooks to list, move or delete files, and is configured by default to list, move or delete SAS Viya log archive files. It can be reconfigured to target other files.

## Requirements for Running the Viya Multi-Machine Services Utilities Playbooks
* The Log Cleanup playbooks must be placed under the sas_viya_playbook directory where SAS Viya was deployed. 
  The directory structure of this project must be preserved. 
  For example: ```sas_viya_playbook/viya-ark/playbooks/log-cleanup/```

## View or edit playbook configuration
View or edit log-cleanup-vars.yml in a text editor. This file defines important configuration settings, plus the path(s) and filename pattern(s) of files which are acted on by the log cleanup playbooks, and the path(s) to which they are moved by the log-cleanup-move.yml playbook. The configuration settings and the path_filepattern_moveto_list are explained in detail in that file.

## Running the Playbooks
To find log files (or whatever other files are targeted according to log-cleanup-vars.yml) and display them in Ansible output, execute:
```
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-find.yml
```

To list log files (or whatever other files are targeted according to log-cleanup-vars.yml) and write lists of files found on each SAS Viya host to a tempfile (defaults to /tmp/log-cleanup-list.txt) on the Ansible controller host, execute:
```
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-list.yml
```

To move log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute the playbook below. By default ensure_moveto_directories_exist = no, and the playbook will fail to move files if the moveto directory does not exist on each host, resulting in playbook errors for the relevant task. When this happens, those files which could be moved are moved, and those files which could not be moved (because the moveto directory does not exist) are not moved. If you change ensure_moveto_directories_exist = yes in log-cleanup-vars.yml, then this playbook creates the 'moveto' directory if necessary if does not already exist on each host. The created directories are owned by root:root, with a directory access mode as defined in created_moveto_directory_mode in log-cleanup-vars.yml.
```
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-move.yml
```

To delete log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute:
```
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-delete.yml
```

File viya-ark/playbooks/log-cleanup/log-cleanup-vars.yml:

####################################################################
#### log-cleanup-vars.yml                                       ####
####################################################################
#### Author: SAS Institute Inc.                                 ####
####################################################################

#########################################
# Variables used in playbooks
#########################################
# For tempfile_rootname, choose something which is NOT the
# first part of any filepath/name which is valuable, because
# log-cleanup-list.yml will delete all files beginning with
# this string (i.e. it deletes {{tempfile_rootname}}* on
# the Ansible controller before it creates a new tempfile.
# if you choose a short, generic name, this may delete more
# files than you intend! E.g. /tmp/ is a bad choice, as it
# would cause log-cleanup-list.yml to delete all files in /tmp.
# Worse choices are possible, and we avoid suggesting them here!
# Please choose something nice and unique for tempfile_rootname.
#########################################
# Defaults:
# tempfile_rootname: /tmp/log-cleanup-list
# tempfile_name: "{{tempfile_rootname}}.txt"
#########################################
tempfile_rootname: /tmp/log-cleanup-list
tempfile_name: "{{tempfile_rootname}}.txt"

# Set ensure_moveto_directories_exist to yes to have playbook
# log-cleanup-move.yml create the moveto directories on each
# host, if they do not already exist and a file will be moved
# into them.
# They are created as root:root, with a directory access mode
# as specified in created_moveto_directory_mode.
#########################################
# Defaults:
# ensure_moveto_directories_exist: no
# created_moveto_directory_mode: 0777
#########################################
ensure_moveto_directories_exist: no
created_moveto_directory_mode: 0777

#########################################
# Arrays used in playbooks
#########################################
# path_filepattern_moveto_list
#########################################
# path: a directory to search for files
# pattern: the pattern of files to search for in that directory
# use_regex: whether the pattern is a glob (use_regex: no)
#            or a regular expression (use_regex: yes)
# older_than_days: find only files older than this number of days
# recurse: whether to search subdirectories below the path recursively
# moveto: where to move the files we find to, if we want to move them
#         N.B. moveto directory for each entry should *already exist*
#         on each host where files matching the path and filename
#         pattern could be found.
#########################################
# Examples:
#path_filepattern_moveto_list:
#  - { path: /opt/sas/viya/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log }
#  - { path: /opt/sas/viya/config/var/log/cas/default, pattern: "cas*.log", use_regex: no, older_than_days: 340, recurse: no, moveto: /tmp/cas }
#
# Additional examples for multi-tenant deployments:
#  - { path: /opt/sas/tenantA/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log_tenantA }
#  - { path: /opt/sas/tenantB/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log_tenantB }
#  - { path: /opt/sas/tenantA/config/var/log/cas/default, pattern: "cas*.log", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/cas_tenantA }
#  - { path: /opt/sas/tenantB/config/var/log/cas/default, pattern: "cas*.log", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/cas_tenantB }
#########################################
# Default:
# path_filepattern_moveto_list:
#  - { path: /opt/sas/viya/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log }
#########################################
path_filepattern_moveto_list:
  - { path: /opt/sas/viya/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log }

File viya-ark/playbooks/log-cleanup/log-cleanup-find.yml:

####################################################################
#### log-cleanup-find.yml                                       ####
####################################################################
#### Author: SAS Institute Inc.                                 ####
####################################################################
---
####################################################################
# Tasks to find files matching specified patterns
# This file creates a register called 'output_of_find' which is
# used in the other log-cleanup playbooks.
####################################################################

- hosts: sas-all
  become: yes
  become_user: root
  gather_facts: false
  any_errors_fatal: true
  vars_files:
    - log-cleanup-vars.yml
  tasks:
    - name: Check the accessibility of the hosts
      ping:

    - name: Get lists of files on each host matching each pattern in path_filepattern_moveto_list
      find:
        paths: "{{item.path}}"
        patterns: "{{item.pattern}}"
        use_regex: "{{item.use_regex}}"
        age: "{{item.older_than_days}}d"
        recurse: "{{item.recurse}}"
      register: output_of_find
      with_items: '{{path_filepattern_moveto_list}}'
    
    - name: Display count of files on each host matching each pattern in path_filepattern_moveto_list
      debug:
        msg:
         - 'Found {{item.matched}}/{{item.examined}} files in {{item.item.path}} matching pattern {{item.item.pattern}} older than {{item.item.older_than_days}} days old'
      with_items: '{{output_of_find.results}}'
        
    - name: Display list of files on each host matching each pattern in path_filepattern_moveto_list
      debug:
        msg: "{{item.path}}"
      with_items: "{{output_of_find.results | map(attribute='files') | list}}"

File viya-ark/playbooks/log-cleanup/log-cleanup-list.yml:

####################################################################
#### log-cleanup-list.yml                                       ####
####################################################################
#### Author: SAS Institute Inc.                                 ####
####################################################################
---
####################################################################
# Include tasks to find files matching specified patterns, and store
# the result in a register called output_of_find.
####################################################################

- include: log-cleanup-find.yml

###############################################################
# Tasks to create text files listing matching files - cleanup
###############################################################
      

- hosts: sas-all
  become: yes
  become_user: root
  gather_facts: false
  any_errors_fatal: true
  vars_files:
    - log-cleanup-vars.yml
  tasks:
    
    #- name: Debug results
    #  debug: msg="{{output_of_find.results}}"
    
    - name: Delete temp file on each host if it already exists
      file: path={{tempfile_name}} state=absent

- hosts: 127.0.0.1
  become: yes
  become_user: root
  gather_facts: false
  any_errors_fatal: true
  vars_files:
    - log-cleanup-vars.yml
  tasks:
    - name: Delete temp file on local host if it already exists
      file: path={{item}} state=absent
      with_fileglob: "{{tempfile_rootname}}*"

#####################################################################
# Tasks to create text files listing matching files - make and fetch
#####################################################################
      
- hosts: sas-all
  become: yes
  become_user: root
  gather_facts: false
  any_errors_fatal: true
  vars_files:
    - log-cleanup-vars.yml
  tasks:
    - name: Write file list to output file on each host
      lineinfile: create=yes regexp="NONEXISTENTLINE" dest={{tempfile_name}} line="{{item.path}}" state=present
      with_items: "{{output_of_find.results | map(attribute='files') | list}}"
      
    - name: Fetch output files from each host to store on local host
      fetch:
        src: "{{tempfile_name}}"
        dest: "{{tempfile_rootname}}-{{inventory_hostname}}.txt"
        flat: yes
        fail_on_missing: no

File viya-ark/playbooks/log-cleanup/log-cleanup-move.yml:

####################################################################
#### log-cleanup-move.yml                                       ####
####################################################################
#### Author: SAS Institute Inc.                                 ####
####################################################################
---
####################################################################
# Include tasks to find files matching specified patterns, and store
# the result in a register called output_of_find.
####################################################################

- include: log-cleanup-find.yml

###############################################################
# Move matching files
###############################################################

- hosts: sas-all
  become: yes
  become_user: root
  gather_facts: false
  any_errors_fatal: true
  vars_files:
    - log-cleanup-vars.yml
  tasks:

    #- name: Debug results
    #  debug: msg="{{output_of_find.results}}"

    - name: Print the move command we intend to issue in the next task
      debug: msg="mv {{ item[1].path }} {{item[0].item.moveto}}/{{ item[1].path | basename}}"
      with_subelements:
        - "{{output_of_find.results}}"
        - files
      when: "item[0].matched > 0"
      
    - name: If ensure_moveto_directories_exist == yes, ensure moveto directories exist
      file:
        path: "{{item[0].item.moveto}}"
        state: directory
        mode: "{{created_moveto_directory_mode}}"
      with_subelements:
        - "{{output_of_find.results}}"
        - files
      when: "(item[0].matched > 0) and ensure_moveto_directories_exist"
      
    - name: Move files
      command: mv {{ item[1].path }} {{item[0].item.moveto}}/{{ item[1].path | basename}}
      with_subelements:
        - "{{output_of_find.results}}"
        - files
      when: "item[0].matched > 0"

File viya-ark/playbooks/log-cleanup/log-cleanup-delete.yml:

####################################################################
#### log-cleanup-delete.yml                                     ####
####################################################################
#### Author: SAS Institute Inc.                                 ####
####################################################################
---
####################################################################
# Include tasks to find files matching specified patterns, and store
# the result in a register called output_of_find.
####################################################################

- include: log-cleanup-find.yml

###############################################################
# Delete matching files
###############################################################

- hosts: sas-all
  become: yes
  become_user: root
  gather_facts: false
  any_errors_fatal: true
  vars_files:
    - log-cleanup-vars.yml
  tasks:

    #- name: Debug results
    #  debug: msg="{{output_of_find.results}}"

    - name: List files we intend to delete in the next task
      debug: msg="About to delete {{ item[1].path }}"
      with_subelements:
        - "{{output_of_find.results}}"
        - files
      when: "item[0].matched > 0"
     
    - name: Delete files
      file: path={{ item[1].path }} state=absent
      with_subelements:
        - "{{output_of_find.results}}"
        - files
      when: "item[0].matched > 0"

Configure and run SAS Viya log-cleanup playbooks

When you have all the files in place as above, the next step is to edit viya-ark/playbooks/log-cleanup/log-cleanup-vars.yml as follows:

Review tempfile_rootname and tempfile_name - the defaults for these two variables are probably good for most users
If you want log-cleanup-move to create destination 'moveto' directories if they do not already exist on each host and the playbook finds files to move to that destination directory, set ensure_moveto_directories_exist to yes, and review the created_moveto_directory_mode to choose the access mode for the newly-created directories. By default, this is 0777, which is rwxrwxrwx. However, by default, ensure_moveto_directories_exist is set to no, so log-cleanup-move will error each time it tries to copy a file to a directory that does not exist.
Review and edit the list of file patterns in path_filepattern_moveto_list. By default there is only one pattern, looking for log archive files older than 90 days old in /opt/sas/viya/config/var/log. Change this, or add additional entries to the list to search other directories (e.g. for a multi-tenant deployment) for other file patterns, if you need to. Several examples are given in the comments.

Save any changes you made to viya-ark/playbooks/log-cleanup/log-cleanup-vars.yml. Then run one of the playbooks as follows.

To find log files (or whatever other files are targeted according to log-cleanup-vars.yml) and display them in Ansible output, execute:

ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-find.yml

To list log files (or whatever other files are targeted according to log-cleanup-vars.yml) and write lists of files found on each SAS Viya host to a tempfile (defaults to /tmp/log-cleanup-list.txt), and copy each hosts' tempfile to the Ansible controller host (as /tmp/log-cleanup-list-hostname.txt), execute:

ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-list.yml

To move log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute the playbook below:

ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-move.yml

To delete log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute:

ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-delete.yml

Conclusion

Clearing away old log archives is not something a SAS administrator has to do immediately after deploying a new SAS Viya deployment. It will take time before the volume of old log files becomes a problem. But for some customers, sooner or later ever-growing log archives will become a problem. These log-cleanup playbooks offer a simple and flexible solution to that problem.

Schedule a daily script to call the log-cleanup-move or log-cleanup-delete playbook to help ensure your SAS Viya hosts don't fill their disks with log archive files.