Capture events from Opensearch and setup alerts

[drahorg]

We have deployed Viya 2024.09LTS, additionally logging and monitoring. I need to report on the events when a specific container got an error during the last 7 days. I know that in Opensearch - Dashboards - Log message with Level (Pods/container) - Filter for kube.container=ContainerName and level=error. I have 2 questions if you can help

1. Since there are shown many thousands of pages of these results, how can I export all these to a large Excel or csv/text file so that is easier to filter and look at the results. 

2. is it possible to send any notifications/alerts when let's say a specific word appear in the message for this criteria (eg  Opensearch - Dashboards - Log message with Level (Pods/container) - Filter for kube.container=ContainerName and level=error. if the message contains failure then send an alert and or email). any alerting can be enabled and show in an alerts page? 

Thanks a lot for your help!

[gwootton]

The getlogs.py in viya4-monitoring-kubernetes/logging/bin is the easiest way (in my opinion) to pull the relevant lines out.
./getlogs.py -fo csv -l ERROR -c <container-name> -m 10000 -o errors.csv -st 2025-08-04 00:00:00 (7 days ago)
You can also export the results using opensearch dashboards.

For alerting you would need to use Grafana, which I think would require configuring OpenSearch as a data source for Grafana.

Display Log Messages in Grafana Dashboards
https://go.documentation.sas.com/doc/en/obsrvcdc/v_003/obsrvdply/p1bqqaa7r8s06jn1pjexe3ymrckn.htm

You could then build alerts based using the datasource.

Manage Grafana Alerting
https://go.documentation.sas.com/doc/en/obsrvcdc/v_003/obsrvug/n1eslak1oy1dq4n1d9nbtuqefwpb.htm

[drahorg]

Gregg, thanks a lot for the link to display log messages to Grafana and also Grafana alerting . 

regarding my original question (export the compute - programming container log), have deployed logging/monitoring a year ago when this was under github DAC, so not have the kubernetes github configured for our logging/monitoring.

in Dashboards, I filtered for this container and level,  I can export a page at a time, but was not able to export all the events since there are over 2000 pages after applying the filter criteria. is this possible?

Thank you for your input!

[gwootton]

I believe both the OpenSearch Dashboards UI and python script have a limit of 10,000 records you can export at a time.