The Operations Infrastructure in SAS Viya has a daily task called LogFileArchive which uses the sas-archive tool to tar and zip log files older than 30 days, and stores the resulting archives in /opt/sas/viya/config/var/log. By default, the ops-agent on each SAS Viya host runs LogFileArchive daily at 0400. Because logs compress well, the resulting log archives are small, so this task frees up disk space well.
But nothing is provided out of the box in SAS Viya to clear down these log archives. SAS Administrators will want to schedule some sort of script to either delete them, or move them elsewhere for long term storage, regularly. Otherwise those log archives will grow over time, and may eventually fill up the disk where /opt/sas/viya/config/var/log lives on one or more hosts.
That would be bad, so in this post, I present some log-cleanup playbooks: a set of ansible playbooks to clean up those log archives. I'll finish by also mentioning other useful log management utilities you may also want to know about, and where you can learn more about them.
The SAS Viya Administration Resource Kit (Viya-ARK) is an indispensable source of playbooks and supporting files which help customers prepare for a SAS Viya deployment, and manage SAS Viya afterwards. They are publically available on GitHub, at the link above. At the time of writing, current Viya-ARK tools supporting the latest release of SAS Viya include:
Ansible is the perfect tool for tidying up SAS Viya log archives. It it is easy to write Ansible tasks which act on files on all of your Viya hosts simultaneously, from a single command. The log-cleanup playbooks below follow the style of other Viya-ARK playbooks, so I hope you will find them easy to use.
I have begun working with the team who look after the Viya-ARK project to potentially include these playbooks in it, at a future release, subject to the necessary testing and approval being given. The log-cleanup playbooks presented in this post should be treated as an 'alpha' or pre-release version, tested on CentOS 7.5 and Ansible 2.5.5 to the standard of 'it worked on my machine'. They are not yet tested on other platforms/versions, and may require changes to make them compatible with the other platforms and Ansible versions we support. I will update this post when/if the playbooks are updated or merged into Viya-ARK.
In this post:
Boxes with a green border show bash commands you can type at a command prompt
Boxes with an orange border contain file content you can copy paste and save as a file
Create the six log-cleanup playbook files, placing them under ~/sas_viya_playbook/viya-ark as follows:
cd ~/sas_viya_playbook mkdir -p ~/sas_viya_playbook/viya-ark/playbooks/log-cleanup/
File viya-ark/playbooks/log-cleanup/README.md:
# SAS Viya Administration Resource Kit (Viya-ARK) - Log Cleanup Playbooks ## Introduction The Log Cleanup repository contains a set of playbooks to list, move or delete files, and is configured by default to list, move or delete SAS Viya log archive files. It can be reconfigured to target other files. ## Requirements for Running the Viya Multi-Machine Services Utilities Playbooks * The Log Cleanup playbooks must be placed under the sas_viya_playbook directory where SAS Viya was deployed. The directory structure of this project must be preserved. For example: ```sas_viya_playbook/viya-ark/playbooks/log-cleanup/``` ## View or edit playbook configuration View or edit log-cleanup-vars.yml in a text editor. This file defines important configuration settings, plus the path(s) and filename pattern(s) of files which are acted on by the log cleanup playbooks, and the path(s) to which they are moved by the log-cleanup-move.yml playbook. The configuration settings and the path_filepattern_moveto_list are explained in detail in that file. ## Running the Playbooks To find log files (or whatever other files are targeted according to log-cleanup-vars.yml) and display them in Ansible output, execute: ``` ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-find.yml ``` To list log files (or whatever other files are targeted according to log-cleanup-vars.yml) and write lists of files found on each SAS Viya host to a tempfile (defaults to /tmp/log-cleanup-list.txt) on the Ansible controller host, execute: ``` ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-list.yml ``` To move log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute the playbook below. By default ensure_moveto_directories_exist = no, and the playbook will fail to move files if the moveto directory does not exist on each host, resulting in playbook errors for the relevant task. When this happens, those files which could be moved are moved, and those files which could not be moved (because the moveto directory does not exist) are not moved. If you change ensure_moveto_directories_exist = yes in log-cleanup-vars.yml, then this playbook creates the 'moveto' directory if necessary if does not already exist on each host. The created directories are owned by root:root, with a directory access mode as defined in created_moveto_directory_mode in log-cleanup-vars.yml. ``` ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-move.yml ``` To delete log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute: ``` ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-delete.yml ```
File viya-ark/playbooks/log-cleanup/log-cleanup-vars.yml:
#################################################################### #### log-cleanup-vars.yml #### #################################################################### #### Author: SAS Institute Inc. #### #################################################################### ######################################### # Variables used in playbooks ######################################### # For tempfile_rootname, choose something which is NOT the # first part of any filepath/name which is valuable, because # log-cleanup-list.yml will delete all files beginning with # this string (i.e. it deletes {{tempfile_rootname}}* on # the Ansible controller before it creates a new tempfile. # if you choose a short, generic name, this may delete more # files than you intend! E.g. /tmp/ is a bad choice, as it # would cause log-cleanup-list.yml to delete all files in /tmp. # Worse choices are possible, and we avoid suggesting them here! # Please choose something nice and unique for tempfile_rootname. ######################################### # Defaults: # tempfile_rootname: /tmp/log-cleanup-list # tempfile_name: "{{tempfile_rootname}}.txt" ######################################### tempfile_rootname: /tmp/log-cleanup-list tempfile_name: "{{tempfile_rootname}}.txt" # Set ensure_moveto_directories_exist to yes to have playbook # log-cleanup-move.yml create the moveto directories on each # host, if they do not already exist and a file will be moved # into them. # They are created as root:root, with a directory access mode # as specified in created_moveto_directory_mode. ######################################### # Defaults: # ensure_moveto_directories_exist: no # created_moveto_directory_mode: 0777 ######################################### ensure_moveto_directories_exist: no created_moveto_directory_mode: 0777 ######################################### # Arrays used in playbooks ######################################### # path_filepattern_moveto_list ######################################### # path: a directory to search for files # pattern: the pattern of files to search for in that directory # use_regex: whether the pattern is a glob (use_regex: no) # or a regular expression (use_regex: yes) # older_than_days: find only files older than this number of days # recurse: whether to search subdirectories below the path recursively # moveto: where to move the files we find to, if we want to move them # N.B. moveto directory for each entry should *already exist* # on each host where files matching the path and filename # pattern could be found. ######################################### # Examples: #path_filepattern_moveto_list: # - { path: /opt/sas/viya/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log } # - { path: /opt/sas/viya/config/var/log/cas/default, pattern: "cas*.log", use_regex: no, older_than_days: 340, recurse: no, moveto: /tmp/cas } # # Additional examples for multi-tenant deployments: # - { path: /opt/sas/tenantA/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log_tenantA } # - { path: /opt/sas/tenantB/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log_tenantB } # - { path: /opt/sas/tenantA/config/var/log/cas/default, pattern: "cas*.log", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/cas_tenantA } # - { path: /opt/sas/tenantB/config/var/log/cas/default, pattern: "cas*.log", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/cas_tenantB } ######################################### # Default: # path_filepattern_moveto_list: # - { path: /opt/sas/viya/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log } ######################################### path_filepattern_moveto_list: - { path: /opt/sas/viya/config/var/log, pattern: "log-*.zip", use_regex: no, older_than_days: 90, recurse: no, moveto: /tmp/log }
File viya-ark/playbooks/log-cleanup/log-cleanup-find.yml:
#################################################################### #### log-cleanup-find.yml #### #################################################################### #### Author: SAS Institute Inc. #### #################################################################### --- #################################################################### # Tasks to find files matching specified patterns # This file creates a register called 'output_of_find' which is # used in the other log-cleanup playbooks. #################################################################### - hosts: sas-all become: yes become_user: root gather_facts: false any_errors_fatal: true vars_files: - log-cleanup-vars.yml tasks: - name: Check the accessibility of the hosts ping: - name: Get lists of files on each host matching each pattern in path_filepattern_moveto_list find: paths: "{{item.path}}" patterns: "{{item.pattern}}" use_regex: "{{item.use_regex}}" age: "{{item.older_than_days}}d" recurse: "{{item.recurse}}" register: output_of_find with_items: '{{path_filepattern_moveto_list}}' - name: Display count of files on each host matching each pattern in path_filepattern_moveto_list debug: msg: - 'Found {{item.matched}}/{{item.examined}} files in {{item.item.path}} matching pattern {{item.item.pattern}} older than {{item.item.older_than_days}} days old' with_items: '{{output_of_find.results}}' - name: Display list of files on each host matching each pattern in path_filepattern_moveto_list debug: msg: "{{item.path}}" with_items: "{{output_of_find.results | map(attribute='files') | list}}"
File viya-ark/playbooks/log-cleanup/log-cleanup-list.yml:
#################################################################### #### log-cleanup-list.yml #### #################################################################### #### Author: SAS Institute Inc. #### #################################################################### --- #################################################################### # Include tasks to find files matching specified patterns, and store # the result in a register called output_of_find. #################################################################### - include: log-cleanup-find.yml ############################################################### # Tasks to create text files listing matching files - cleanup ############################################################### - hosts: sas-all become: yes become_user: root gather_facts: false any_errors_fatal: true vars_files: - log-cleanup-vars.yml tasks: #- name: Debug results # debug: msg="{{output_of_find.results}}" - name: Delete temp file on each host if it already exists file: path={{tempfile_name}} state=absent - hosts: 127.0.0.1 become: yes become_user: root gather_facts: false any_errors_fatal: true vars_files: - log-cleanup-vars.yml tasks: - name: Delete temp file on local host if it already exists file: path={{item}} state=absent with_fileglob: "{{tempfile_rootname}}*" ##################################################################### # Tasks to create text files listing matching files - make and fetch ##################################################################### - hosts: sas-all become: yes become_user: root gather_facts: false any_errors_fatal: true vars_files: - log-cleanup-vars.yml tasks: - name: Write file list to output file on each host lineinfile: create=yes regexp="NONEXISTENTLINE" dest={{tempfile_name}} line="{{item.path}}" state=present with_items: "{{output_of_find.results | map(attribute='files') | list}}" - name: Fetch output files from each host to store on local host fetch: src: "{{tempfile_name}}" dest: "{{tempfile_rootname}}-{{inventory_hostname}}.txt" flat: yes fail_on_missing: no
File viya-ark/playbooks/log-cleanup/log-cleanup-move.yml:
#################################################################### #### log-cleanup-move.yml #### #################################################################### #### Author: SAS Institute Inc. #### #################################################################### --- #################################################################### # Include tasks to find files matching specified patterns, and store # the result in a register called output_of_find. #################################################################### - include: log-cleanup-find.yml ############################################################### # Move matching files ############################################################### - hosts: sas-all become: yes become_user: root gather_facts: false any_errors_fatal: true vars_files: - log-cleanup-vars.yml tasks: #- name: Debug results # debug: msg="{{output_of_find.results}}" - name: Print the move command we intend to issue in the next task debug: msg="mv {{ item[1].path }} {{item[0].item.moveto}}/{{ item[1].path | basename}}" with_subelements: - "{{output_of_find.results}}" - files when: "item[0].matched > 0" - name: If ensure_moveto_directories_exist == yes, ensure moveto directories exist file: path: "{{item[0].item.moveto}}" state: directory mode: "{{created_moveto_directory_mode}}" with_subelements: - "{{output_of_find.results}}" - files when: "(item[0].matched > 0) and ensure_moveto_directories_exist" - name: Move files command: mv {{ item[1].path }} {{item[0].item.moveto}}/{{ item[1].path | basename}} with_subelements: - "{{output_of_find.results}}" - files when: "item[0].matched > 0"
File viya-ark/playbooks/log-cleanup/log-cleanup-delete.yml:
#################################################################### #### log-cleanup-delete.yml #### #################################################################### #### Author: SAS Institute Inc. #### #################################################################### --- #################################################################### # Include tasks to find files matching specified patterns, and store # the result in a register called output_of_find. #################################################################### - include: log-cleanup-find.yml ############################################################### # Delete matching files ############################################################### - hosts: sas-all become: yes become_user: root gather_facts: false any_errors_fatal: true vars_files: - log-cleanup-vars.yml tasks: #- name: Debug results # debug: msg="{{output_of_find.results}}" - name: List files we intend to delete in the next task debug: msg="About to delete {{ item[1].path }}" with_subelements: - "{{output_of_find.results}}" - files when: "item[0].matched > 0" - name: Delete files file: path={{ item[1].path }} state=absent with_subelements: - "{{output_of_find.results}}" - files when: "item[0].matched > 0"
When you have all the files in place as above, the next step is to edit viya-ark/playbooks/log-cleanup/log-cleanup-vars.yml as follows:
Save any changes you made to viya-ark/playbooks/log-cleanup/log-cleanup-vars.yml. Then run one of the playbooks as follows.
To find log files (or whatever other files are targeted according to log-cleanup-vars.yml) and display them in Ansible output, execute:
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-find.yml
To list log files (or whatever other files are targeted according to log-cleanup-vars.yml) and write lists of files found on each SAS Viya host to a tempfile (defaults to /tmp/log-cleanup-list.txt), and copy each hosts' tempfile to the Ansible controller host (as /tmp/log-cleanup-list-hostname.txt), execute:
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-list.yml
To move log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute the playbook below:
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-move.yml
To delete log files (or whatever other files are targeted according to log-cleanup-vars.yml), execute:
ansible-playbook viya-ark/playbooks/log-cleanup/log-cleanup-delete.yml
Clearing away old log archives is not something a SAS administrator has to do immediately after deploying a new SAS Viya deployment. It will take time before the volume of old log files becomes a problem. But for some customers, sooner or later ever-growing log archives will become a problem. These log-cleanup playbooks offer a simple and flexible solution to that problem.
Schedule a daily script to call the log-cleanup-move or log-cleanup-delete playbook to help ensure your SAS Viya hosts don't fill their disks with log archive files.
Scott McCauley wrote about Log Management in SAS Viya 3.3. He described how the ops-agent process on each SAS Viya host in your deployment runs a task called LogFileArchive daily at 04:00, which calls the sas-archive command. This archives files in directories below (by default)/opt/sas/viya/config/var/log, to a single file in that directory. In Scott's post about SAS Viya 3.3, he wrote that this archives log files older than 3 days. In a SAS Viya 3.4 deployment it seems that default has changed to archive (tar and zip) files older than 30 days.
Gerry Nelson recently added the listfiles.py and archivefiles.py tools to our pyviyatools project in Github, and he wrote about why you might want to use them to list and delete files stored by the SAS Viya files service, in his article Where are my Viya files? These tools can be used for cleaning up files held in the SAS Viya files service which similarly could remain there forever if not deleted by something like Gerry's new tools.
The two Python scripts respectively list files stored in the SAS Viya files service, and archive them (i.e. in this context, download them from the files service using a REST API call and save the JSON content as a physical file on the filesystem) and optionally delete the matching files from the files service to free up space in the SAS Infrastructure Data Server database. Note that archive in this sense means something a little different to archive in the sense of compressing them to a tar and zipped file.
See you next time!
I believe in log-cleanup-find.yml and others the line "- host: sas-all" should actually be sas_all as defined in the sas_viya_playbook/inventory.ini.
Is that correct?
If we leave the default of sas-all the log-cleanup-find.yml playbook fails with the following:
PLAYBOOK: log-cleanup-find.yml *********************************************************************************************************************************************************************************************************
1 plays in viya-ark/playbooks/log-cleanup/log-cleanup-find.yml
[WARNING]: Could not match supplied host pattern, ignoring: sas-all
If I change log-cleanup-find.yml to have sas_all it finds all my hosts and completes:
...
ok: [casWorker1] => (item={u'rusr': True, u'uid': 1000, u'rgrp': True, u'xoth': False, u'islnk': False, u'woth': False, u'nlink': 1, u'issock': False, u'mtime': 1677747600.0891082, u'gr_name': u'sas', u'path': u'/opt/sas/viya/config/var/log/log-20230131090000Z.zip', u'xusr': False, u'atime': 1677747600.076109, u'inode': 279195785, u'isgid': False, u'size': 22, u'isdir': False, u'ctime': 1677747600.0891082, u'isblk': False, u'wgrp': False, u'xgrp': False, u'isuid': False, u'dev': 64768, u'roth': True, u'isreg': True, u'isfifo': False, u'mode': u'0644', u'pw_name': u'sas', u'gid': 1000, u'ischr': False, u'wusr': True}) => {
"msg": "/opt/sas/viya/config/var/log/log-20230131090000Z.zip"
}
META: ran handlers
META: ran handlers
PLAY RECAP *****************************************************************************************************************************************************************************************************************************
casWorker1 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
casWorker2 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
deployServices : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
deployTarget : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
If this is correct each one of the .yml files above with sas-all may need to be changed to sas_all instead to match the sas_viya_playbook/inventory.ini file.
This is a very interesting topic and a great explanation. much appreciated.
there is another part most of my customers complain about it. Service Log files themselves cleaning/log rotation.
for instance: in AML/VI orders, we can notice that the indexing service's latest log can reach 50GB, and accordingly to clean this log need to stop clean then start services.
Most customers seek Log rotation automatically configure "no human intervention " Also they don't have an option for a planned restart for SAS every specific period (In a Weekend or so, Frequency)
Join us for SAS Innovate 2025, our biggest and most exciting global event of the year, in Orlando, FL, from May 6-9. Sign up by March 14 for just $795.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.