topic SAS Logs Purge Job/Script in Administration and Deployment

SAS Logs Purge Job/Script

webbm — Thu, 01 Feb 2018 21:16:53 GMT

My environment: Suse Linux Backend, Windows user tools (EG)

Question: I want to set up a nightly script on our Linux server to do maintenance. I'm going to include running the cleanwork utility to get rid of rogue saswork sessions, along with some general housekeeping. I have a list of all the common logging directories, and I'm going through each one to inventory the files that are there, in addition to deciding on a reasonable rule for each directory (ie: delete all old log files over 60 days, etc).

I can tell what I'm looking at with the SAS Server Logs (object spawner, etc) and I feel comfortable making that decision. There are some logs that I am not familiar with and I don't want to "rip the rug" out from under the process, so to speak, if that log is in some way needed. I am mostly wondering about any logging related to Environment Manager, Perf and Audit Logs. Are there any logs that I absolutely shouldn't touch and let them continue to accumulate?

Here are a few specific examples that I'm wondering about:

Lev1/ObjectSpawner/AuditLogs
Lev1/WorkspaceServer/PerfLogs
/Lev1/Web/SASEnvironmentManager/emi-framework/Logs/ acm*, getMetricData*, masterAPMETL*,
/Lev1/Web/SASEnvironmentManager/emi-framework/Archive/logs_processed_date.zip
Lev1/Web/SASEnvironmentManager/emi-client/Logs logDiscovery*, logCollect*,
/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/log/agentstats agentstats*.csv.gz

I could keep going with this, but I think you get the picture. I'm only worried about the locations where there isn't a built-in log cleanup and I'm not sure if it's safe to delete some things. Generally I wouldn't even think to delete anything that obviously didn't have a recurring log with several months of existing old files. I'm not talking about anything current - just old stuff that could free up space.

We are a newer SAS site and I have not used these boards as much as I should, so I hope this is the appropriate place to post this and I hope I am not duplicating an already asked question. I've done several searches and found a few articles, but nothing that answers my specific question.

Thank you!

Re: SAS Logs Purge Job/Script

AndrewHowell — Thu, 01 Feb 2018 23:35:01 GMT

At a former site, we would run a scheduled monthly O/S script which would check a list of log folder locations.

For each log folder:

Compress all log files older than 24 hours into a "monthly-named" zip file.
Once compress, remove those log files.
Remove any compressed files older than 3 months.

So each folder would contain (for example):

todays_log_file(s).log
201712.zip
201711.zip
201710.zip
other (non log) files

That way, admins had access to up to 3 months of logs if required. If older logs were required (unlikely?), revert to the server's filesystem backup.

I don't have a copy of the script here at my new site (different role), but it shouldn't be hard to replicate.

Re: SAS Logs Purge Job/Script

webbm — Fri, 02 Feb 2018 02:50:14 GMT

That's something like what I'm going for. I don't need help with the script - just looking to figure out what's safe to delete more or less. I like the idea of archiving some, then deleting old archives - might be a better way to go than just deleting the files if I'm uncomfortable with deleting some things to start with.

Re: SAS Logs Purge Job/Script

AndrewHowell — Fri, 02 Feb 2018 06:16:28 GMT

Unsure if there's a definitive list - it depends what you have & how often it's used. (May also depend on your logging options.)

My general rule of thumb is to zip any existing log files and remove any more than 24 hours old.

Re: SAS Logs Purge Job/Script

nhvdwalt — Fri, 02 Feb 2018 07:17:33 GMT

@webbm, you're right. There are certain logs that are very easy to spot and maintain, and there other's where you have no clue, especially on the mid-tier side.

From personal experience, I found it's really a learning process that you have to go through. Will you get an inventory of ALL the logs on the system ? I don't believe so. But it's necessarily a problem....

Some considerations:

1.) Run some OS commands that will list files and directories that contain something like "*log*". Yes, it's a fuzzy match.

2.) Then go through all logs and locations and make an inventory. As you go through them, you'll start seeing patterns emerge.

3.) If you are not reasonably sure about something, then rather park it (for now). Some logs are overwritten and some are really so small, it doesn't even warrant housekeeping.

4.) Over time, as you learn more about the system, you can refine your 'log housekeeping inventory'

5.) Also consider that many components in SAS are actually third-party components e.g. gemfire, so you will have to cast you research wider that just the SAS software.

6.) The above will probably carry you 95% of the way, personally I wouldn't worry too much about the other 5% unless it becomes a problem, or if you are looking for work at 2am 🙂

Good luck !

Re: SAS Logs Purge Job/Script

JuanS_OCS — Mon, 05 Feb 2018 13:07:57 GMT

Hello @webbm,

I see perfectly where are you coming from. And I fully subscribe to the great recomendations and experience provided by @AndrewHowell

The SAS auditing, as any other auditing in other systems, consumes space and needs to be archived and clean once they are not directly needed anymore.

I truly think that, first, it would be very useful for you to understand what happens on SAS backstage, before providing a complete answer.

Those components are part of the SAS Environment Manager Service Architecture Framework (link), more specifically from the Environment Manager Agent, ACM and APM.

Understanding SAS Environment Manager Service Management Architecture (link)
Paper 064-2014: Log entries, Events, Performance Measures, and SLAs: Understanding and Managing your SAS® Deployment by Leveraging the SAS® Environment Manager Data Mart (link)

The part you would need to know is that:

SAS Environment Manager agent feeds the SAS Environment Manager server, and this web app will feed the WIP database and a SAS internal process will feed the SAS Environment Manager Datamart. Once the data collected by the agents are stored in WIP, the logs are not needed. Perhaps you would need to keep them archived due to company policies. This would cover the last location from your list.
APM and ACM basically extend the logs from your SAS services, such as Metadata, Object Spawner, Workspace Srv, STP srv, etc. A nightly job will run every night, collecting the data and filling in WIP and the SAS Environment DataMart. Again, once collected, you can handle them safely. This would cover your first locations on your list.

Knowing this, you can create and customise the retention period of your logs in your system. SAS already does this, partially, out-of-the-box for you:

/Lev1/Web/SASEnvironmentManager/emi-framework/Archive/logs_processed_date.zip is an actual archived collection of logs.
SAS will also have a retention period on your audited data. You will want to review and understand well this part, specially if you will clean the original locations of the logs:
- http://support.sas.com/kb/58/599.html
- https://blogs.sas.com/content/sgf/2016/01/26/going-visual-analytics-audit-data-archiving/

Hope it helps.

Kind regards,

Juan