Architecting, installing and maintaining your SAS environment

SAS Logs Purge Job/Script

Reply
Occasional Contributor
Posts: 7

SAS Logs Purge Job/Script

My environment: Suse Linux Backend, Windows user tools (EG)

 

Question:  I want to set up a nightly script on our Linux server to do maintenance.  I'm going to include running the cleanwork utility to get rid of rogue saswork sessions, along with some general housekeeping.  I have a list of all the common logging directories, and I'm going through each one to inventory the files that are there, in addition to deciding on a reasonable rule for each directory (ie: delete all old log files over 60 days, etc).  

 

I can tell what I'm looking at with the SAS Server Logs (object spawner, etc) and I feel comfortable making that decision. There are some logs that I am not familiar with and I don't want to "rip the rug" out from under the process, so to speak, if that log is in some way needed.  I am mostly wondering about any logging related to Environment Manager, Perf and Audit Logs.  Are there any logs that I absolutely shouldn't touch and let them continue to accumulate?

 

Here are a few specific examples that I'm wondering about:

  • Lev1/ObjectSpawner/AuditLogs
  • Lev1/WorkspaceServer/PerfLogs
  • /Lev1/Web/SASEnvironmentManager/emi-framework/Logs/ acm*, getMetricData*, masterAPMETL*,
  • /Lev1/Web/SASEnvironmentManager/emi-framework/Archive/logs_processed_date.zip
  • Lev1/Web/SASEnvironmentManager/emi-client/Logs logDiscovery*, logCollect*, 
  • /Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/log/agentstats agentstats*.csv.gz

I could keep going with this, but I think you get the picture.  I'm only worried about the locations where there isn't a built-in log cleanup and I'm not sure if it's safe to delete some things.  Generally I wouldn't even think to delete anything that obviously didn't have a recurring log with several months of existing old files.  I'm not talking about anything current - just old stuff that could free up space.  

 

We are a newer SAS site and I have not used these boards as much as I should, so I hope this is the appropriate place to post this and I hope I am not duplicating an already asked question.  I've done several searches and found a few articles, but nothing that answers my specific question.  

 

Thank you!

Moderator
Posts: 317

Re: SAS Logs Purge Job/Script

At a former site, we would run a scheduled monthly O/S script which would check a list of log folder locations.

 

For each log folder:

  • Compress all log files older than 24 hours into a "monthly-named" zip file.
  • Once compress, remove those log files.
  • Remove any compressed files older than 3 months.

 

So each folder would contain (for example):

  • todays_log_file(s).log
  • 201712.zip
  • 201711.zip
  • 201710.zip
  • other (non log) files

That way, admins had access to up to 3 months of logs if required. If older logs were required (unlikely?), revert to the server's filesystem backup.

 

I don't have a copy of the script here at my new site (different role), but it shouldn't be hard to replicate.

 

Occasional Contributor
Posts: 7

Re: SAS Logs Purge Job/Script

Posted in reply to AndrewHowell

That's something like what I'm going for. I don't need help with the script - just looking to figure out what's safe to delete more or less. I like the idea of archiving some, then deleting old archives - might be a better way to go than just deleting the files if I'm uncomfortable with deleting some things to start with. 

Moderator
Posts: 317

Re: SAS Logs Purge Job/Script

Unsure if there's a definitive list - it depends what you have & how often it's used. (May also depend on your logging options.)

 

My general rule of thumb is to zip any existing log files and remove any more than 24 hours old.

 

 

SAS Employee
Posts: 296

Re: SAS Logs Purge Job/Script

@webbm, you're right. There are certain logs that are very easy to spot and maintain, and there other's where you have no clue, especially on the mid-tier side.

 

From personal experience, I found it's really a learning process that you have to go through. Will you get an inventory of ALL the logs on the system ? I don't believe so. But it's necessarily a problem....

 

Some considerations:

1.) Run some OS commands that will list files and directories that contain something like "*log*". Yes, it's a fuzzy match.

2.) Then go through all logs and locations and make an inventory. As you go through them, you'll start seeing patterns emerge.

3.) If you are not reasonably sure about something, then rather park it (for now). Some logs are overwritten and some are really so small, it doesn't even warrant housekeeping.

4.) Over time, as you learn more about the system, you can refine your 'log housekeeping inventory'

5.) Also consider that many components in SAS are actually third-party components e.g. gemfire, so you will have to cast you research wider that just the SAS software.

6.) The above will probably carry you 95% of the way, personally I wouldn't worry too much about the other 5% unless it becomes a problem, or if you are looking for work at 2am Smiley Happy

 

Good luck !

 

 

Trusted Advisor
Posts: 1,756

Re: SAS Logs Purge Job/Script

[ Edited ]

Hello @webbm,

 

I see perfectly where are you coming from. And I fully subscribe to the great recomendations and experience provided by @AndrewHowell

 

The SAS auditing, as any other auditing in other systems, consumes space and needs to be archived and clean once they are not directly needed anymore.

 

I truly think that, first, it would be very useful for you to understand what happens on SAS backstage, before providing a complete answer.

 

Those components are part of the SAS Environment Manager Service Architecture Framework (link), more specifically from the Environment Manager Agent, ACM and APM. 

  • Understanding SAS Environment Manager Service Management Architecture (link)
  • Paper 064-2014: Log entries, Events, Performance Measures, and SLAs: Understanding and Managing your SAS® Deployment by Leveraging the SAS® Environment Manager Data Mart (link)

The part you would need to know is that:

  • SAS Environment Manager agent feeds the SAS Environment Manager server, and this web app will feed the WIP database and a SAS internal process will feed the SAS Environment Manager Datamart. Once the data collected by the agents are stored in WIP, the logs are not needed. Perhaps you would need to keep them archived due to company policies. This would cover the last location from your list.
  • APM and ACM basically extend the logs from your SAS services, such as Metadata, Object Spawner, Workspace Srv, STP srv, etc. A nightly job will run every night, collecting the data and filling in WIP and the SAS Environment DataMart. Again, once collected, you can handle them safely. This would cover your first locations on your list.

 

Knowing this, you can create and customise the retention period of your logs in your system. SAS already does this, partially, out-of-the-box for you:

 

 Hope it helps.


Kind regards,

Juan

Ask a Question
Discussion stats
  • 5 replies
  • 243 views
  • 4 likes
  • 4 in conversation