topic Re: Auto Cleanup script for log files in Administration and Deployment

Auto Cleanup script for log files

sasprofile — Tue, 29 Jun 2021 19:41:41 GMT

Hello friends -

Can any one please help with the log cleanup script

I need your help with the Auto log cleanup script for the maintenance of our sas servers. am totally new to unix scripting. can anyone please assist with the cleanup script.

We need to keep the log files 2 months in the /logs directory and any files older than 2 months has to go to the Backup directory and in the Backup directory files should reside for 6 months and any log files older than 6 months should be deleted.

I want this script should perform this cleanup activity in all servers in different /log file locations of the server.

Would really appreciate if you can help with the script

Thank you all in advance

Re: Auto Cleanup script for log files

gwootton — Tue, 29 Jun 2021 19:50:51 GMT

You could probably do this using a couple find commands, something like:

find <log-path> -type f -mtime +60 -exec mv {} <backup-path> \;
find <backup-path> -type f -mtime +180 -delete

You could put that in a script and schedule it via cron to run regularly. If you run the command without the actions (i.e. no -delete and no -exec mv {} <backup-path \;) it will return the files it would take the action on.

Re: Auto Cleanup script for log files

sasprofile — Tue, 29 Jun 2021 20:38:14 GMT

Thank you gwootton for prompt response..

Actually I found below find commands...but

find /sas/install/config/Lev1/Logs -maxdepth 1 -mtime +60 -type f -exec mv "{}" /backup \;
find /backup -maxdepth 1 -mtime +180 -type f -exec rm "{}" \;

find /sas/install/config/Lev1/Web/WebServer/logs -maxdepth 1 -mtime +60 -type f -exec mv "{}" /sashome/M6_install/config/Lev1/Web/WebServer/logs/backup_062821 \;
find /sashome/M6_install/config/Lev1/Web/WebServer/logs/backup_062821 -maxdepth 1 -mtime +180 -type f -exec rm "{}" \;

find /sas/install/config/Lev1/Web/Logs/SASServer1_1 -maxdepth 1 -mtime +60 -type f -exec mv "{}" /sashome/M6_install/config/Lev1/Web/Logs/SASServer1_1/backup_062821 \;
find /sashome/M6_install/config/Lev1/Web/Logs/SASServer1_1/backup_062821 -maxdepth 1 -mtime +180 -type f -exec rm "{}" \;

but am looking for the script with adding some email messages for each action taking place. find in conjunction with mv or rm. But for every find there must be a communication to sas admin telling what has been done

Script servse the purpose in this case but what is missing the communication and not dealing with the error condition like a situation when there's not enough disk space on the target file system. And also not handling overwriting the file names when moving the file from sources to destinations

I need below three things :

1)need to add comments line for each find command action
2)need to add send email for each action
3) need to add one more condition--- compression of the file after a month
4) also u need to know the size of the file moved from source to destination

and also I need to have a new file name when moving from source to destination

The new name is necessary to avoid overwriting. The new naming convention should reflect the old name + server name + date of the activity,This is necessary since every things goes to same file system And names has to be unique to avoid overwriting

Basically it is a rotation of the log files to compress after a month , followed by moving and then removing

If u do not give new name, it will overwriting in the old file,It us good practice to have any new file with name having time stamp.

That way when we need those files in future we can track them easily

So new file name=old name.time stamp

The whole idea of this practice is that I will be retaining these log file for a year is that i may need them

My opinion is each month. In each folder, compress, tar the previous month's logs

Example; xxxlog-202106.tar.zip at the end of this month

Compression stays in side the source side

After the compression the directory will create a one single file which will be moving after three months

One single file will be Xxxxlog-20320631.targz
And this file will be moved to /log at the end of December
And should have the same name as it is

But should go to the directory for the relevant server inside /log
/log/metadata/xxxlog202106.targz

Would really appreciate your help

Re: Auto Cleanup script for log files

gwootton — Tue, 29 Jun 2021 21:11:05 GMT

It sounds like you have a good idea of what you want your shell script to do, which is a good starting point.

I don't have a sample program that meets all these requirements, but these are all things that can be done in shell scripting.

You can use the date command to create a timestamp, maybe as an environment variable and use that as part of your mv command. You could use output redirection to write some information to a log file for your script, the mail command to send an email. Perhaps instead of using the -exec option of the find command you use find to get a list of files you want to do something with and then for each one write a line to your log, perform an action, etc.

I would recommend reading up on shell scripting and doing a web search on the individual things you are trying to do in that script for more information on different ways to accomplish them.

Re: Auto Cleanup script for log files

sasprofile — Tue, 29 Jun 2021 21:44:12 GMT

Yeah I have some idea about my requirement what the shell script should do, but unfortunately i don't have any idea at all on shell scripting. The find commands also i which sent earlier i found it from google, if anyone can help with my requirement that would be really a big help.

Re: Auto Cleanup script for log files

SASKiwi — Wed, 30 Jun 2021 20:21:57 GMT

Learning how to write shell scripts isn't really a SAS problem so you will probably find better help on websites devoted to this. Surely your company will have IT people skilled at this and who can help you to come up to speed or who you can ask to write the script for you. I know mine does and I'd be asking them to do the shell scripting for me as like you I know very little about it.

Re: Auto Cleanup script for log files

Sajid01 — Thu, 01 Jul 2021 15:32:19 GMT

1. I see that you want a rotation of log. Have a look at the UNIX logrotate utility available is most Unix/Linux distributions.

2.The Unix/Linux administrators should be able to help.

3. I fully understand your not knowing shell scripting. Better to learn but not worry. The expectations from SAS Administrators are a growing unrealistically high these days depending upon the environment without much of a career progression.

Re: Auto Cleanup script for log files

Kurt_Bremser — Tue, 06 Jul 2021 09:34:19 GMT

As soon as you start doing administrator work, Maxim 15 (Know Your Playing Field) turns from a strong recommendation into a requirement.

Knowing at least the basics of bash scripting (creation and use of environment variables, conditional execution, regular expressions, writing to and reading from files, piping, just to name a few) is a must.

You also need to get a grasp on the important UNIX utilities like find, grep, ls, ps. Maxim 6 (Google Is Your Friend) is a big helper in this respect.

Since I do not have to do shell scripting often, I usually need to consult the online documentation for all this myself, so do not be afraid to dig into it.

Re: Auto Cleanup script for log files

ashasupriya — Tue, 26 Jul 2022 13:46:10 GMT

Hello - Did you have luck in completing the script for SAS log purging? I am also looking at options to set up an automatic SAS LOG purging job to clean up the logs periodically on the windows server. Thank you!

Re: Auto Cleanup script for log files

SASKiwi — Tue, 26 Jul 2022 20:15:11 GMT

On Windows, if all you want to do is remove orphan SAS WORK libraries then just schedule the cleanup tool in Windows Task Scheduler: