BookmarkSubscribeRSS Feed
drahorg
Obsidian | Level 7

We have deployed Viya 2024.09LTS, additionally logging and monitoring. I need to report on the events when a specific container got an error during the last 7 days. I know that in Opensearch - Dashboards - Log message with Level (Pods/container) - Filter for kube.container=ContainerName and level=error. I have 2 questions if you can help

1. Since there are shown many thousands of pages of these results, how can I export all these to a large Excel or csv/text file so that is easier to filter and look at the results. 

2. is it possible to send any notifications/alerts when let's say a specific word appear in the message for this criteria (eg  Opensearch - Dashboards - Log message with Level (Pods/container) - Filter for kube.container=ContainerName and level=error. if the message contains failure then send an alert and or email). any alerting can be enabled and show in an alerts page? 

 

Thanks a lot for your help!

3 REPLIES 3
gwootton
SAS Super FREQ
The getlogs.py in viya4-monitoring-kubernetes/logging/bin is the easiest way (in my opinion) to pull the relevant lines out.
./getlogs.py -fo csv -l ERROR -c <container-name> -m 10000 -o errors.csv -st 2025-08-04 00:00:00 (7 days ago)
You can also export the results using opensearch dashboards.

For alerting you would need to use Grafana, which I think would require configuring OpenSearch as a data source for Grafana.

Display Log Messages in Grafana Dashboards
https://go.documentation.sas.com/doc/en/obsrvcdc/v_003/obsrvdply/p1bqqaa7r8s06jn1pjexe3ymrckn.htm

You could then build alerts based using the datasource.

Manage Grafana Alerting
https://go.documentation.sas.com/doc/en/obsrvcdc/v_003/obsrvug/n1eslak1oy1dq4n1d9nbtuqefwpb.htm
--
Greg Wootton | Principal Systems Technical Support Engineer
drahorg
Obsidian | Level 7

Gregg, thanks a lot for the link to display log messages to Grafana and also Grafana alerting . 

 

regarding my original question (export the compute - programming container log), have deployed logging/monitoring a year ago when this was under github DAC, so not have the kubernetes github configured for our logging/monitoring.

 

in Dashboards, I filtered for this container and level,  I can export a page at a time, but was not able to export all the events since there are over 2000 pages after applying the filter criteria. is this possible?

 

Thank you for your input!

 

 

gwootton
SAS Super FREQ
I believe both the OpenSearch Dashboards UI and python script have a limit of 10,000 records you can export at a time.
--
Greg Wootton | Principal Systems Technical Support Engineer

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 338 views
  • 1 like
  • 2 in conversation